msg/simple: do not stop_and_wait on mark_down

We originally blocked in mark_down for fast dispatch threads
to complete to avoid various races in the code.  Most of these
were in the OSD itself, where we were not prepared to get
messges on connections that had no attached session.  Since
then, the OSD checks have been cleaned up to handle this.
There were other races we were worried about too, but the
details have been lost in the depths of time.

Instead, take the other route: make mark_down never block on
dispatch.  This lets us remove the special case that
was added in order to cope with fast dispatch calling
mark_down on itself.

Now, the only stop_and_wait() user is the shutdown sequence.

Signed-off-by: Sage Weil <sage@redhat.com>
Revert "Objecter: disable fast dispatch of CEPH_MSG_OSD_OPREPLY messages"

This reverts commit 3f23709c474292f9239f77a6cce26309fc86ce29.

We have changed mark_down() behavior so that it no longer blocks on
fast dispatch.  This makes the objecter reply handler safe again.

Fixes: #9598
Signed-off-by: Sage Weil <sage@redhat.com>
Merge pull request #2722 from ceph/wip-9598

msgr: make mark_down not block on fast dispatch

sage-2014-11-11_08:26:01-rados-wip-sage-testing-distro-basic-multi

Reviewed-by: Greg Farnum <greg@inktank.com>