Only shutdown when fully idle

Okay, we might have been shutting down a little too aggressively.