Fix a rare race condition which can occur during shutdown