msgr: take over existing Connection on Pipe replacement

If a new pipe/socket is taking over an existing session, it should also
take over the Connection* associated with the existing session.  Because
we cannot clear existing->connection_state, we just take another reference.

Clean up the comments a bit while we're here.

This affects MDS<->client sessions when reconnecting after a socket fault.
It probably also affects intra-cluster (osd/osd, mds/mds, mon/mon)
sessions as well, but I did not confirm that.

Backport: argonaut
Signed-off-by: Sage Weil <sage@inktank.com>
msgr: use explicit Connection for messages sent to ourself

Move to an explicit Connection for messages sent to ourselves, instead of
using the one on the local_pipe (which we'll remove shortly).

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: use local IncomingQueue instead of Pipe

Simpler, cleaner.  No need for the rest of the Pipe crap.  We just need to
queue messages for ourselves.

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: simplify IncomingQueue creation, pointers

 * create it via DispatchQueue
 * keep pointer to parent DispatchQueue
 * drop now-useless contextual arguments to most methods

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: move Pipe, DispatchQueue into separate files

These don't need to be subclasses of SimpleMessenger.  Separate!

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: uninline ctor and dtor

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: remove obsolete dead path

This hasn't triggered in years.

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: remove unnecessary accept check

We don't need to worry about racing with shutdown here; the cleanup
procedure will stop the accepter thread before cleaning up all the
pipes.

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: simplify checks for queueing connection events

Looking through git history it is not clear exactly how these checks
came to be.  They seem to have grown during the multiple-entity-per-rank
transition a few years back.  I'm not fully convinced they are necessary,
but we will keep them regardless.

Push checks into DispatchQueue and look at the local stop flag to
determine whether these events should be queued.  This moves us away from
the kludgey SimpleMessenger::destination_stopped flag (which will soon
be removed).

Also move the refcount futzing into the DispatchQueue methods.  This makes
the callers much simpler.

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: move dispatch thread into DispatchQueue

The DispatchQueue class now completely owns message delivery.  This is
cleaner and lets us drop the redundant destination_stopped flag from
msgr (DQ has its own stop flag).

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: drop unnecessary (un)locking on queuing connection events

This used to be necessary because the pipe_lock was used when queueing
the pipe in the dispatch queue.  Now that is handled by IncomingQueue's
own lock, so these can be removed.

By no longer dropping the lock, we eliminate a whole category of potential
hard-to-debug races.  (Not that any were observed, but now we dno't need to
worry about them.)

Signed-off-by: Sage Weil <sage@inktank.com>
dispatcher: new 'accept' event type

Create a new event type when we successfully accept a connection.  This is
distinct from the authorizor verification, which may happen for multiple
racing connection attempts.  In contrast, this will only happen on those
that win the race(s).  I don't think this is that important for stateless
servers (OSD, MON), but it is important for the MDS to ensure that it keeps
its Session con reference pointing to the most recently-successful
connection attempt.

Signed-off-by: Sage Weil <sage@inktank.com>
msg/DispatchQueue: queue and deliver accept events

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: queue accept event when pipe is accepted

Queue an event when an incoming connection is accepted.

Signed-off-by: Sage Weil <sage@inktank.com>
mds: fix race in connection accept; fix con replacement

We solve two problems with this patch.  The first is that the messenger
will now reuse an existing session's Connection with a new connection,
which means that we don't want to change session->connection when we
are validating an authorizer.  Instead, set (but do not change) it.

We also want to avoid a race where:

 - mds recovers, replays Sessions with no con's
 - multiple connection attempts for the same session race in the msgr
 - both are authorized, but out of order
 - Session->connection gets set to the losing attempt's Connection*

Instead, we take advantage of an accept event that is called only for
accepted winners.

Signed-off-by: Sage Weil <sage@inktank.com>

fixup
msgr: rework accept() connect_seq/race handling

We change a couple of key things here:

 * If there is a matching connect_seq and the existing connection is in OPEN (or
   STANDBY; same thing + a failure), we send a RETRY_SESSION and ask the peer to
   bump their connect_seq.  This handles the case where there was a race, our
   end successfully opened, but the peer's racing attempt was slowly processed.
 * We always reply with connect_seq + 1.  This handles the above case
   more cleanly, and lets us use the same code path.

Also avoid duplicating the RETRY_SESSION path with a goto.  Beautiful!

Signed-off-by: Sage Weil <sage@inktank.com>
msg/Pipe: get_state_name()

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: move Accepter into separate .cc

Signed-off-by: Sage Weil <sage@inktank.com>
msg/Pipe: move tcp_* functions into Pipe class

This lets us print nice debug prefixes.  It also calls BS on the
Pipe vs tcp.cc separation.

Signed-off-by: Sage Weil <sage@inktank.com>
msg/DispatchQueue: cleanup debug prefix

Signed-off-by: Sage Weil <sage@inktank.com>