PG: compound messages must carry epoch_sent for each part

Query and Notify messages include logical messages from multiple
pgs.  Each logical message (pg_query_t and pg_notify_t) now
contains an epoch_sent.

Signed-off-by: Samuel Just <sam.just@inktank.com>
PG: use osd->requeue_ops for ops, pg->queue_for_peering to requeue pg

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD: check for deleting in process_peering_event

Signed-off-by: Samuel Just <sam.just@inktank.com>
PG,OSD: add OSD::queue_for_op, use in PG::queue_op

Signed-off-by: Samuel Just <sam.just@inktank.com>
PG,OSD: fix op checking in pg, take_waiters during ActMap

Signed-off-by: Samuel Just <sam.just@inktank.com>
PG: flush ops by the end of peering without osr.flush

Rather than explicitely flushing the filestore, send a noop through the
filestore at the beginning of peering and, at the end, wait for it to
finish by adding an extra state.

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD: remove superfluous pg get/put around enqueue_op

Signed-off-by: Samuel Just <sam.just@inktank.com>
ReplicatedPG: do not eval_repop if aborted

Signed-off-by: Samuel Just <sam.just@inktank.com>
ReplicatedPG: change ReplicatedPG debug output to match PG

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD: don't assume pending pg removals have flushed

_create_lock_pg might encounter a preexisting pg collection simply
because the removal transaction had not yet completed.

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD: do not lock osd during dequeue_op

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD: push_waiters is no longer used

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD: get map read lock during queue_want_up_thru

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD: do not drop osd_lock in handle_osd_map

PGs have their map updates done in a different thread.  Thus, we no
longer need to grab the pg locks.  activate_map no longer requires
the map_lock in order to allow us to queue events for the pgs.

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD: queue_want_up_thru in process_peering_event

Signed-off-by: Samuel Just <sam.just@inktank.com>
PG,MOSDPGInfo,OSD: use pg_notify_t for Infos since it's compound

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD,PG: Move pg accesible methods, objects to OSDService

In order to clarify data structure locking, PGs will now access
OSDService rather the the OSD directly.  Over time, more structures will
be moved to the OSDService.  osd_lock can no longer be held while pg
locks are held.

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD: _have_pg should return NULL if pg is not in map

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD,PG: clean up pg removal

PG opsequencers will be used for removing a pg.  If the pg is recreated
before the removal is complete, we need the new pg incarnation to be
able to inherit the osr of its predecessor.

Previously, we queued the pg for removal and only rendered it unusable
after the contents were fully removed.  Now, we syncronously remove it
from the map and queue a transaction renaming the collections.  We then
asyncronously clean up those collections.  If the pg is recreated, it
will inherit the same osr until the cleanup is complete ensuring correct
op ordering with respect to the collection rename.

Signed-off-by: Samuel Just <sam.just@inktank.com>
ReplicatedPG: explicitely block on not active for certain ops

Ops and some subops need to wait for active to ensure correct ordering
with respect to peering operations.

Signed-off-by: Samuel Just <sam.just@inktank.com>