RWLock: assert pthread function return values

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
PG: constify the init() function params

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD::_share_map_incoming: line wrap debug output

Formatting only.

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
OSD: add handle_osd_map debug output

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: replace handle_pg_scan, handle_pg_backfill with handle_replica_op

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
OSD: remove unused push_wq

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: rename gen_wq, schedule_work, and PG_QueueAsync to include "recovery"

These all hook into the recovery thread pool and need to make that obvious.

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: remove never-activated while loop from send_incremental_map

Signed-off-by: Greg Farnum <greg@inktank.com>
DispatchQueue: factor out pre_dispatch and post_dispatch

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
Messenger,DispatchQueue: add ms_fast_dispatch mechanism

This adds a Dispatcher interface allowing the implementation
to accept ms_fast_dispatch calls for some messages without
going through the DispatchQueue. To support that, we also add
1) new synchronous notifications on connect and accept events
2) a fast_preprocess mechanism

Signed-off-by: Samuel Just <sam.just@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>
msgr: enable fast_dispatch on local connections

We do two things:
1) Call ms_handle_fast_connect() when setting up the local connection, so
the Dispatcher can set up any state it needs
2)Move local_delivery into a separate thread from the sender's. fast_dispatch
makes this entirely necessary since otherwise we're dipping back in to the
Dispatcher while holding whatever locks it held when it sent the Message.

Implementation starts with a thread and a list of messages to process and
proceeds as you'd expect from that.

Signed-off-by: Greg Farnum <greg@inktank.com>
msgr: change the delay queue flushing semantics

Since we're doing fast_dispatch out of the delay queue, we don't want to
flush while holding the pipe lock. Instead, make flush set it up for instant
delivery, and steal the delay queue when replacing pipes. If we're shutting
down a pipe, wait until flushing has completed before doing so.

Signed-off-by: Greg Farnum <greg@inktank.com>
OSDService: add osdmap reservation mechanism

The goal here is to be able to get "reserved" refs
to next_map, and ensure that pgs won't see a newer
map until the ref is "released".  I haven't done
a cute RAII trick here yet...probably not worth
the effort.

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

Conflicts:
	src/osd/OSD.h
Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: add a RWLock pg_map_lock

If we're going to dispatch ops without grabbing the osd lock, we need
something else to protect the pg map (and it'll be a little
contended, so use a read-write lock).

We repurpose the (previously oddly-named) _lookup_lock_pg_with_map_lock_held()
function to refer to the pg_map_lock. handle_pg_query and handle_pg_remove
switch to use that version, because they're holding pg_map_lock already and
we know the PG they're referring to exists.

Signed-off-by: Samuel Just <sam.just@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: pass osdmap to handle_op and handle_replica_op

We need a map to process them, and we don't want to
take the OSD lock to access one. (And we can't just
use the service because we need all processing of
a message to be done with the same map.)

Signed-off-by: Samuel Just <sam.just@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: add session waiting_for_map mechanisms

This will replace the existing waiting_for_osdmap mechanism
with a per-session wait list.

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
OSD: remove wake_all_pg_waiters

We shouldn't need this -- we check the pg waiters list on each
map.

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
OSD: wake_pg_waiters atomically with pg_map update

Also, call enqueue_op directly rather than going back
through the entire dispatch machinery.
Be sure to grab the pg lock under the pg_map_lock in _open_lock_pg() to
preserve necessary lock ordering.

Signed-off-by: Samuel Just <sam.just@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: Push responsibility for grabbing pg_map_lock up to callers of _remove_pg()

The atomicity requirements of other systems prevent us dropping the PG lock
inside that function, and the PG lock is ordered underneath the pg_map_lock.

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: shard heartbeat_lock

heartbeat_need_update must be protected independently in order to avoid
a loop with the pg_map_lock and the PG::_lock.

Signed-off-by: Greg Farnum <greg@inktank.com>