[#1218] Check which player instance we remove from the connection maps.

Legitimate but slightly sneaky fix for the issue of racing for addition / removal. An alternate fix would be a multimap.