msgr: use explicit Connection for messages sent to ourself

Move to an explicit Connection for messages sent to ourselves, instead of
using the one on the local_pipe (which we'll remove shortly).

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: use local IncomingQueue instead of Pipe

Simpler, cleaner.  No need for the rest of the Pipe crap.  We just need to
queue messages for ourselves.

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: simplify IncomingQueue creation, pointers

 * create it via DispatchQueue
 * keep pointer to parent DispatchQueue
 * drop now-useless contextual arguments to most methods

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: move Pipe, DispatchQueue into separate files

These don't need to be subclasses of SimpleMessenger.  Separate!

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: uninline ctor and dtor

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: remove obsolete dead path

This hasn't triggered in years.

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: remove unnecessary accept check

We don't need to worry about racing with shutdown here; the cleanup
procedure will stop the accepter thread before cleaning up all the
pipes.

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: simplify checks for queueing connection events

Looking through git history it is not clear exactly how these checks
came to be.  They seem to have grown during the multiple-entity-per-rank
transition a few years back.  I'm not fully convinced they are necessary,
but we will keep them regardless.

Push checks into DispatchQueue and look at the local stop flag to
determine whether these events should be queued.  This moves us away from
the kludgey SimpleMessenger::destination_stopped flag (which will soon
be removed).

Also move the refcount futzing into the DispatchQueue methods.  This makes
the callers much simpler.

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: move dispatch thread into DispatchQueue

The DispatchQueue class now completely owns message delivery.  This is
cleaner and lets us drop the redundant destination_stopped flag from
msgr (DQ has its own stop flag).

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: drop unnecessary (un)locking on queuing connection events

This used to be necessary because the pipe_lock was used when queueing
the pipe in the dispatch queue.  Now that is handled by IncomingQueue's
own lock, so these can be removed.

By no longer dropping the lock, we eliminate a whole category of potential
hard-to-debug races.  (Not that any were observed, but now we dno't need to
worry about them.)

Signed-off-by: Sage Weil <sage@inktank.com>
dispatcher: new 'accept' event type

Create a new event type when we successfully accept a connection.  This is
distinct from the authorizor verification, which may happen for multiple
racing connection attempts.  In contrast, this will only happen on those
that win the race(s).  I don't think this is that important for stateless
servers (OSD, MON), but it is important for the MDS to ensure that it keeps
its Session con reference pointing to the most recently-successful
connection attempt.

Signed-off-by: Sage Weil <sage@inktank.com>
msg/DispatchQueue: queue and deliver accept events

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: queue accept event when pipe is accepted

Queue an event when an incoming connection is accepted.

Signed-off-by: Sage Weil <sage@inktank.com>
mds: fix race in connection accept

We want to avoid a race where:

 - mds recovers, replays Sessions with no con's
 - multiple connection attempts for the same session race in the msgr
 - both are authorized, but out of order
 - Session->connection gets set to the losing attempt's Connection*

Instead, we take advantage of an accept event that is called only for
accepted winners.

Signed-off-by: Sage Weil <sage@inktank.com>