Remove old server->ev if it's still in use.