Fix locking of removeServer