Fixed a synchronization goof where we could get a concurrent modification exception if a connection was removed at just the wrong time during shutdown.