Fix error on sending a message when a server is not connected but the RemoteUser is still held somewhere