Only do a handshake if we're initiating the connection