Make sure the async upgrade handler exits