msgr: clear_pipe inside pipe_lock on mark_down_all

Observed a segfault in rebind -> mark_down_all -> clear_pipe -> put that
may have been due to a racing thread clearing the connection_state pointer.
Do the clear_pipe() call under the protection of pipe_lock, as we do in
all other contexts.

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: clear con->pipe inside pipe_lock on mark_down

We need to do this under protection of the pipe_lock.

Signed-off-by: Sage Weil <sage@inktank.com>
msg/Pipe: goto fail_unlocked on early failures in accept()

Instead of duplicating an incomplete cleanup sequence (that does not
clear_pipe()), goto fail_unlocked and do the cleanup in a generic way.
s/rc/r/ while we are here.

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: reaper: make sure pipe has been cleared (under pipe_lock)

All paths to pipe shutdown should have cleared the con->pipe reference
already.  Assert as much.

Also, do it under pipe_lock!

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: clear_pipe+queue reset when replacing lossy connections

We already handle the lossless replacement and lossy fault paths, but
not the lossy replacement.  This fixes an assert(!cleared) in the
reaper.  Adjust comments appropriately.

Signed-off-by: Sage Weil <sage@inktank.com>
Merge pull request #375 from ceph/wip-msgr

misc msgr fixes

Reviewed-by: Greg Farnum <greg@inktank.com>
refactor RecoveryState and RecoveryState::Peering

The goal is to extract the RecoveryState class from the PG class so
that it is easier to maintain, test and reuse in the context of
pluggable placement groups. It reduces the PG lines of code by 30% and
uses about 100 of its methods ( that is 50% of them ).

The PG::RecoveryState sub class is moved to the PGRecoveryState.{cc,h}
files. The PG::RecoveryState::Peering state and its sub-states are
moved to the PGPeering.{cc,h} files.

The PGRecoveryStateInterface abstract class is created (
PGRecoveryStateInterface.h file ) and made a base class of PG. All
references to PG in PGPeering.{cc,h} and PGRecoveryState.{cc,h} are
replaced by PGRecoveryStateInterface. When writing tests, a fixture can
be created with a derived class of PGRecoveryStateInterface and
simulate a rare error condition.

To preserve the scoping of the PG::RecoveryState class and avoid name
clashes, the contents of PGPeering.{cc,h} and PGRecoveryState.{cc,h}
are included in the PGRecoveryState namespace.

Data members previously referenced directly now use
PGRecoveryStateInterface accessors. When possible the accessors are
made const and the calling method is modified accordingly ( using
const_iterator for instance ).

THIS PATCH IS STILL INCOMPLETE. TODO:

* sort PGRecoveryStateInterface.h to group the methods
* cosmetic changes to PGPeering.{cc,h} and PGRecoveryState.{cc,h}
* what about the PGRecoveryState namespace
* what about the class names

Signed-off-by: Loic Dachary <loic@dachary.org>