Backport fix for race condition on shutdown