OSD: use the async workqueue to send OSDMap updates on dropped ops

Check whether we actually want to send a map in-line, and if we do, create
a GenContext which does so and put that in the op_gen_wq.

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: don't share_map_incoming() directly from handle_replica_op()

Let the op_tp handle it, or our C_SendMap callback in the op_gen_wq.

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: enable ms_fast_dispatch

We've been setting it up, now this patch actually adds a fast path for osd ops
which bypasses the osd_lock and should not block on any longly held locks. In
addition to the actual ms_fast_dispatch, we take advantage of the fast_notify
functions in order to create a Session for every peer, since that is now the
data structure around which we handle incoming Messages and waitlisting.

Signed-off-by: Samuel Just <sam.just@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: Properly handle mark_down'ing a peer which is doing fast_dispatch

Keep the Session <-> Connection ref around while doing the mark_down, and
grab Session dispatch lock to make sure the Session isn't presently in use,
and remove the Session from the session_waiting_for_map list before
actually clearing the Connection's link to the Session.

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: _share_map_outgoing whenever sending a message to a peer

This ensures that they get new maps before an op which requires them (that
they would then request from the monitor).

Signed-off-by: Greg Farnum <greg@inktank.com>
Messenger/Dispatcher: add a 'fast_preprocess' mechanism

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: maintain the largest epoch received from peers in their Session

We have a problem with fast dispatch: a peer can send us a needed map,
then a subop, and we can queue up the map message in the DispatchQueue
before immediately processing the subop.
osd.0 --> MOSDMap(10,11) --> osd.1
osd.0 --> MOSDSubOp(11)  --> osd.1
      osd.1: DispatchQueue(MOSDMap)
      osd.1: fast_dispatch(MOSDSubOp);
      osd.1: ms_dispatch(MOSDMap);
Whenever we get a message too new for us to process, we must either wait
blindly for a new map to arrive, or preemptively request the new map from
the Monitor. Maintaining a preprocessed largest epoch on the Session lets
us detect the above situation by noting the sent map before we've been
able to incorporate it into the full OSD state. Thus we avoid both bombarding
the monitor with unnecessary traffic and having to make wait time versus
double-work tradeoffs for the common case.

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: request maps in dispatch_op_fast if we don't have them

Signed-off-by: Greg Farnum <greg@inktank.com>
msgr: enable fast_dispatch on local connections

We do two things:
1) Call ms_handle_fast_connect() when setting up the local connection, so
the Dispatcher can set up any state it needs
2) Extend local_delivery() to fast dispatch, by taking a lock if the Message
can_fast_dispatch().

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: do not take pre_publish_lock in send_message_osd_cluster()

It loops back around for local connections and deadlocks, so we use the
map reservation mechanism instead.

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: do not share maps with oneself

Signed-off-by: Greg Farnum <greg@inktank.com>