Fix possible race condition
When a remote is connected and receives a packet, but is not yet in the remotes dictionary