SimpleMessenger: Pipe: do not block on takeover while holding global lock

We previously were able to cause deadlocks:
1) Existing pipe is fast_dispatching
2) Replacement incoming pipe is accepted
  *) blocks on stop_and_wait() of existing Pipe
3) External things are blocked on SimpleMessenger::lock() while
  blocking completion of the fast dispatch.

To resolve this, if we detect that an existing Pipe we want to take over is
in the process of fast dispatching, we unlock our locks and wait on it to
finish. Then we go back to the lookup step and retry.

The effect of this should be safe:
1) We are not making any changes to the existing Pipe in new ways
2) We have not registered the new Pipe anywhere
3) We have not sent back any replies based on Messenger state to
   the remote endpoint.

Backport: giant

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
msg/Pipe: inject delay in stop_and_wait

Inject a delay in stop_and_wait.  This will mostly affect the connection
race Pipe takeover code which currently calls stop_and_wait while holding
the msgr->lock.  This should make it easier for a racing fast_dispatch
method to get stuck on a call that (indirectly) needs the msgr lock.
See #9921.

Signed-off-by: Sage Weil <sage@redhat.com>
Merge pull request #2825 from ceph/wip-9921

SimpleMessenger: Pipe: do not block on takeover while holding global locks

Reviewed-by: Sage Weil <sage@redhat.com>