Cache remote address in handshake since it might be lost later.