Prevent ConcurrentModification exceptions when removing connections