msgr: move to STANDBY if we replace during accept and then fail

If we replace an existing pipe during accept() and then fail, move to
STANDBY so that our connection state (connect_seq, etc.) is preserved.
Otherwise, we will throw out that information and falsely trigger a
RESETSESSION on the next connection attempt.

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: do not try to reconnect con with CLOSED pipe

If we have a con with a closed pipe, drop the message.  For lossless
sessions, the state will be STANDBY if we should reconnect.  For lossy
sessions, we will end up with CLOSED and we *should* drop the message.

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: fix pipe replacement assert

We may replace an existing pipe in the STANDBY state if the previous
attempt failed during accept() (see previous patches).

This might fix #1378.

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: make D_CONNECT constant non-zero, fix ms_handle_connect() callback

A while ago we inadvertantly broke ms_handle_connect() callbacks because
of a check for m being non-zero in the dispatch_entry() thread.  Adjust the
enums so that they get delivered again.

This fixes hangs when, for example, the ceph tool sends a command, gets a
connection reset, and doesn't get the connect callback to resend after
reconnecting to a new monitor.

Signed-off-by: Sage Weil <sage@inktank.com>
lockdep: enable in common_init

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: move incoming queue to separate class

This extricates the incoming queue and its funky relationship with
DispatchQueue from Pipe and moves it into IncomingQueue.  There is now a
single IncomingQueue attached to each Pipe.  DispatchQueue is now no
longer tied to Pipe.

This modularizes the code a bit better (tho that is still a work in
progress) and (more importantly) will make it possible to move the
incoming messages from one pipe to another in accept().

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: move dispatch_entry into DispatchQueue class

A bit cleaner.

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: preserve incoming message queue when replacing pipes

If we replace an existing pipe with a new one, move the incoming queue
of messages that have not yet been dispatched over to the new Pipe so that
they are not lost.  This prevents messages from being lost.

Alternatively, we could set in_seq = existing->in_seq - existing->in_qlen,
but that would make the other end resend those messages, which is a waste
of bandwidth.

Very easy to reproduce the original bug with 'ms inject socket failures'.

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: use explicit Connection for messages sent to ourself

Move to an explicit Connection for messages sent to ourselves, instead of
using the one on the local_pipe (which we'll remove shortly).

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: use local IncomingQueue instead of Pipe

Simpler, cleaner.  No need for the rest of the Pipe crap.  We just need to
queue messages for ourselves.

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: simplify IncomingQueue creation, pointers

 * create it via DispatchQueue
 * keep pointer to parent DispatchQueue
 * drop now-useless contextual arguments to most methods

Signed-off-by: Sage Weil <sage@inktank.com>