Yet another Close() related hang fix.

I'm almost tempted not to use a WaitGroup at this point, though
eventually I'll actually get this correct.