Use reference counting to know when to wait for a destination that's going down.

And latches to know when they are done.

This is gross - I don't like having to count references in addition to
the MSC doing its job. But, we'll see if this helps make things more
stable for the TorqueBox CI build.