Make sure a server exists when removing it