osd/ReplicatedPG: basic flush and evict agent functionality

This is very basic flush and evict functionality for the tiering agent.

The flush policy is very simple: if we are above the threshold and the
object is dirty, and not super young, flush it.  This is not too braindead
of a policy (although we could clearly do something smarter).

The evict policy is pretty simple: evict the object if it is clean and
we are over our full threshold.  If we are in the middle mode, try to
estimate how cold the object is based on an accumulated histogram of
objects we have examined so far, and decide to evict based on our
position in that histogram relative to our "effort" level.

Caveats:
 * the histograms are not refreshed
 * we aren't taking temperature into consideration yet, although some of
   the infrastructure is there.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: decay tier agent histograms over time

Make decisions based on recent observations of object age distributions,
not all time history.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: test dirty stats in ceph df detail

Signed-off-by: Loic Dachary <loic@dachary.org>
histogram: fix histogram::get_position_micro overflow

Convert the return values to uint64_t

Signed-off-by: Loic Dachary <loic@dachary.org>
osd: add 'notieragent' flag to OSDMap

This will pause tiering agent work.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: observe 'notieragent' osdmap flag

Pause/unpause the agent thread accordingly.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: only enable tier agent when osd is in active state

Signed-off-by: Sage Weil <sage@inktank.com>
osd: add pg_pool_t::get_random_pg_position()

Return a hash position somewhere inside a given pg.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: initial agent to random hash position inside pg

When the agent starts, start at a random offset to ensure we get a more
uniform distribution of attention to all objects in the PG.  Otherwise, we
will disproportionately examine objects at the "beginning" of the PG if we
are interrupted by peering or restarts or some other activity.

Note that if the agent_state is preserved, we do not forget our position,
which is also nice.

We *could* persist this position in the pg_info_t somewhere, but I am not
sure it is worth the effort.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: add slop to agent mode selection

We want to avoid a situation where the agent clicks on and off when the
system hovers around a utilization threshold.  Particularly for trim,
the system can expend a lot of energy doing a minimal amount of work when
the effort level is low.  To avoid this, enable when we are some amount
above the threshold, and do not turn off until we are the same amount below
the target.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: fix evict mode selection for large target

If the target is > 1.0 for some reason (bad configuration, or high slop
value), and we are not yet full, we should be in IDLE mode--not SOME.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: do not flush or evict hitsets

Signed-off-by: Sage Weil <sage@inktank.com>
osd: clear agent state when PG becomes a replica

Signed-off-by: Sage Weil <sage@inktank.com>
ceph_test_rados_api_tier: fix osd pool set json syntax

String, not int.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: do not flush|evict degraded objects

The repop won't work right; we still repair the object before making
any update.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: do not choke on op-less flush OpContexts (from flush)

The agent initiates flush ops that don't have an OpRequest associated
with them.  Make reply_ctx skip the actual reply message instead of
crashing if the flush request gets canceled (e.g., due to a race with
a write).

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: ignore starvation potential when taking write lock during promote

Signed-off-by: Sage Weil <sage@inktank.com>
osd: add OMAP flag to object_info_t

Set a flag if we ever set or update OMAP content on an object.  This gives
us an easy indicator for the cache agent (without actually querying the
ObjectStore) so that we can avoid trying to flush omap to EC pools.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: do not activate agent unless base pool exists

Paranoia.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: do not flush omap objects to an EC base pool

The EC pool does not support omap content.  If the caching/tiering agent
encounters such an object, just skip it.  Use the OMAP object_info_t flag
for this.

Although legacy pools will have objects with omap that do not have this
flag set, no *cache* pools yet exist, so we do not need to worry about the
agent running across legacy content.

Signed-off-by: Sage Weil <sage@inktank.com>