Thread-safe removal of endpoints