Dont remove from server if not connected to one.