That trivial race condition thing

we must shutdown the taskmaster *after* the listener stops handing it off
new connections.