Fix shutdown even harder

Worked through the happens-before graph a little more formally and found some
(admittedly unlikely) cases where shutdown would fail/panic/hang/whatever. This
version *should* be correct.

@wvanbergen