Always schedule/remove connections on the main run loop.
Don't think there's any need to support an alternative run loop here, and it makes things simpler.