OSD,PG: clean up pg removal

PG opsequencers will be used for removing a pg.  If the pg is recreated
before the removal is complete, we need the new pg incarnation to be
able to inherit the osr of its predecessor.

Previously, we queued the pg for removal and only rendered it unusable
after the contents were fully removed.  Now, we syncronously remove it
from the map and queue a transaction renaming the collections.  We then
asyncronously clean up those collections.  If the pg is recreated, it
will inherit the same osr until the cleanup is complete ensuring correct
op ordering with respect to the collection rename.

Signed-off-by: Samuel Just <sam.just@inktank.com>
ReplicatedPG: explicitely block on not active for certain ops

Ops and some subops need to wait for active to ensure correct ordering
with respect to peering operations.

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD,PG: _share_map_outgoing must not require osd_lock

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD: activate_map() in handle_osd_map only when active

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD: map_cache should contain const OSDMap

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD: only do_(notify|info|query) for up osd

pg may have an older map and attempt to notify|info|query on a down
osd.

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD: lock recovery_wq before debug output on finish_recovery_op

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD,PG: clean up _get_or_create_pg and set interval based on msg

Previously, we set last_peering_reset based on the epoch in which the pg
is created.  We now pass the map from the query_epoch to the creation
methods to set based on that.

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD: don't advance_pg() if pg is up-to-date

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD,PG: issue pg removals in line, remove remove_list

Signed-off-by: Samuel Just <sam.just@inktank.com>
CrushWrapper: rmaps don't need to be mutable

Signed-off-by: Samuel Just <sam.just@inktank.com>
CrushWrapper: add locking around crush_do_rule

crush_do_rule uses a cache on the bucket objects.

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD: on pg_removal, project_pg_history to get current interval

First, we don't really want to remove the pg if we can use it.  Second,
there might be messages in the pg peering queue for the next interval.
If one of those happens to be an info request or notify, we would lose
the peering message.

If the message falls in the current interval as determined by the
current osdmap, than we know that any messages currently queued must be
obsolete and can safetly be discarded.

Signed-off-by: Samuel Just <sam.just@inktank.com>
PG: PG now store its own PGPool

Otherwise, we need to syncronize access to the shared PGPool objects.
The wasted memory is probably preferable to syncronization overhead.

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD: bail out of do_recovery if no longer primary and active

Signed-off-by: Samuel Just <sam.just@inktank.com>
common/WorkQueue.h: add BatchWorkQueue

Rather than dispatching one item at a time to process, etc,
BatchWorkQueue dispatches up to a configurable number of
items.

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD: do_(notifies|infos|queries) must now be passed a map

This removes the need to call them from within the osd lock.

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD: peering_wq is now a BatchWorkQueue

process_peering_events now handles multiple pgs at once to better
batch up notifes, etc.

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD: process_peering_event check for new map on each pg

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD: maybe_update_heartbeat_peers, don't print pg

Signed-off-by: Samuel Just <sam.just@inktank.com>