OSD: add a RWLock pg_map_lock

If we're going to dispatch ops without grabbing the osd lock, we need
something else to protect the pg map (and it'll be a little
contended, so use a read-write lock).

We repurpose the (previously oddly-named) _lookup_lock_pg_with_map_lock_held()
function to refer to the pg_map_lock. handle_pg_query and handle_pg_remove
switch to use that version, because they're holding pg_map_lock already and
we know the PG they're referring to exists.

Signed-off-by: Samuel Just <sam.just@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: pass osdmap to handle_op and handle_replica_op

We need a map to process them, and we don't want to
take the OSD lock to access one. (And we can't just
use the service because we need all processing of
a message to be done with the same map.)

Signed-off-by: Samuel Just <sam.just@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: add session waiting_for_map mechanisms

This will replace the existing waiting_for_osdmap mechanism
with a per-session wait list.

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
OSD: remove wake_all_pg_waiters

We shouldn't need this -- we check the pg waiters list on each
map.

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
OSD: wake_pg_waiters atomically with pg_map update

Also, call enqueue_op directly rather than going back
through the entire dispatch machinery.
Be sure to grab the pg lock under the pg_map_lock in _open_lock_pg() to
preserve necessary lock ordering.

Signed-off-by: Samuel Just <sam.just@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: Push responsibility for grabbing pg_map_lock up to callers of _remove_pg()

The atomicity requirements of other systems prevent us dropping the PG lock
inside that function, and the PG lock is ordered underneath the pg_map_lock.

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: shard heartbeat_lock

heartbeat_need_update must be protected independently in order to avoid
a loop with the pg_map_lock and the PG::_lock.

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: protect access to boot_epoch, up_epoch, bind_epoch

We need to access these members in some call chains via fast_dispatch,
where they're otherwise unprotected.

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: protect state member with a Spinlock

This member was previously protected by the osd_lock (although setting
SHUTDOWN was synchronized with the heartbeat lock, too), but we need
to read it for fast dispatch, so protect it under its own lock at all times.

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD::_share_map_incoming: pass osdmap in explicitly

We'll want to be able to use this method without the osd_lock. Note
that we can't do so yet -- we call send_incremental_map, which is not
safe to call unlocked.

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
OSD::send_incremental_map: use service superblock so we can avoid locking osd_lock

TODO: make it actually safe by dealing with build_incremental_map_msg()

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD: use safe params in map-sharing functions

We were previously using unprotected access to OSD members.

Unfortunately, this does not make them completely safe: we are looking up
maps asynchronously from when we got access to the cached map bounds, and
so the OSD could delete a map out from underneath us. Fixing that will
require some kind of map bounds lock. :/

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: use an OSDMapRef& and require the Session* in _share_map_incoming

You can pass in a NULL Session*, but both callers do that; and using
an OSDMapRef& reduces shared_ptr copies.

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: add _should_share_map function

Just copy _share_map_incoming and rip out all the parts that actually
update data structures.

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: simplify _share_map_incoming based on _should_share_map()

Also, remove the bool return code since nobody looks at it.

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: add a Session::sent_epoch_lock

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: change Session handling around _share_map_incoming

Move responsibility for the reference up to _share_map_incoming's caller,
and start using the Session::sent_epoch_lock. This looks a little silly
now, but we're going to split up the decide-to-send-maps and send-maps steps
and don't want to block in the decide-to-send step, so we need some
pretty flexible locking up at this level.

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: refactor handle_op error handling cases

We move our map version-checking code earlier (to dispatch_op) and refactor
our other fail-to-dispatch cases. This is friendlier for the no-lock
message processing we'll use with fast dispatch.

Signed-off-by: Samuel Just <sam.just@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: share map updates in the op_tp threads instead of the main dispatch thread

Sharing maps can require disk accesses and things. We don't want to do that
in our fast path, so do it in OSD::dequeue_op instead of OSD::handle_op. We're
cheating slightly and still doing it in handle_op if no op actually gets queued,
but we're going to put those into a separate work queue next. We'll also be
moving all the functions necessary for this into OSDService so that our completion
struct doesn't need to be a friend to OSD.
To make this easier, we're adding send_map_update and sent_epoch members to
OpRequest.

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: pass a pointer to last_sent_epoch instead of the whole Session

We don't use any other part of the Session, and this interface will
be easier to move out of the OSD class.

Signed-off-by: Greg Farnum <greg@inktank.com>