osd_types: add generic HitSet type with bloom and explicit implementations

Track a set of hash values, either explicitly or using a bloom_filter. Hide
the implementation and allow us to transparently encode and decode.

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>
osd/osd_types: add pg_hit_set_{info,history}_t

Signed-off-by: Sage Weil <sage@inktank.com>
osd/osd_types: include pg_hit_set_history_t in pg_info_t

Track metadata about the currently accumulating HitSet as well as
previously archived ones in the pg_info_t.  This will not scale well for
extremely long histories, but does let us avoid explicitly sharing this
metadata during recovery or other normal update activity.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: add hit_set_* parameters to pg_pool_t

Add pool properties to control what type of HitSet we want to use, along with
some (mostly generic) parameters.

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>
mon/OSDMonitor: set hit_set fields

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>
osd/ReplicatedPG: add basic HitSet tracking

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>
osd/ReplicatedPG: use vectorized osd_op outdata for pg ops

This lets us put PGLS in a compound operation.  Nothing does that yet, but
this would allow it.
Despite appearances, this is not a protocol change and does not require
a feature bit for clients: using the osd_ops vector mechanisms store all
the data in the same places as before, it just fills in some of the
already-decoded-but-empty data structures in the MOSDOpReply header.
<Greg note:> We may need a feature bit to let clients know they can send
compound PG ops to OSDs, though? Or maybe we can let it be covered
by supporting hitset ops.

Signed-off-by: Sage Weil <sage@inktank.com>
librados, osd: list and get HitSets via librados

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>
librados: create new ceph_test_rados_api_tier target

Move the dirty/undirty test to it, and add one for HitSets.

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>
osd/ReplicatedPG: put hit_set objects in a configurable namespace

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: trim old hit_set objects on persist

Any time we persist a hit_set object, take the opportunity to remove any
old ones that we don't want any more.

Note that this means if the admin decreases the number of objects to track,
we won't remove them until the next time we persist something.  We also
don't clean up if the HitSet tracking is disabled entirely.

Signed-off-by: Sage Weil <sage@inktank.com>
ceph_test_rados: hit hit_set_{list,get} rados operations

This will do a list, and then get a random HitSet.

Signed-off-by: Sage Weil <sage@inktank.com>
pg_hit_set_info_t: remove unused size, target_size members

Signed-off-by: Greg Farnum <greg@inktank.com>
ReplicatedPG: do not seal() HitSets until we're done with them

We don't want to seal HitSets just because we're writing a
snapshot to disk; it potentially shrinks the in-memory one
we want to keep adding stuff to!

Signed-off-by: Greg Farnum <greg@inktank.com>
HitSet: do not permit users to seal() more than once

As the name implies, you should only be able to seal() once. Add
a sealed member (which we encode) and assert if seal() gets called
more than once.

Signed-off-by: Greg Farnum <greg@inktank.com>
osd/ReplicatedPG: apply log to new HitSet to capture writes after peering

Signed-off-by: Sage Weil <sage@inktank.com>
osd/HitSet: make pg_pool_t and Params operator<< less parenthetical

pool 0 'data' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 8 owner 0 crash_replay_interval 45 hit_set bloom{false_positive_probability: 0, target size: 0, seed: 0} 10s x8

Signed-off-by: Sage Weil <sage@inktank.com>
mon/OSDMonitor: non-zero default bloom fpp

Signed-off-by: Sage Weil <sage@inktank.com>
osd/HitSet: take Params as const ref to avoid confusion about ownership

Signed-off-by: Sage Weil <sage@inktank.com>
osd: prevent zero BloomHitSet fpp

Signed-off-by: Sage Weil <sage@inktank.com>