osd/ReplicatedPG: do not flush|evict degraded objects

The repop won't work right; we still repair the object before making
any update.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: do not choke on op-less flush OpContexts (from flush)

The agent initiates flush ops that don't have an OpRequest associated
with them.  Make reply_ctx skip the actual reply message instead of
crashing if the flush request gets canceled (e.g., due to a race with
a write).

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: ignore starvation potential when taking write lock during promote

Signed-off-by: Sage Weil <sage@inktank.com>
osd: add OMAP flag to object_info_t

Set a flag if we ever set or update OMAP content on an object.  This gives
us an easy indicator for the cache agent (without actually querying the
ObjectStore) so that we can avoid trying to flush omap to EC pools.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: do not activate agent unless base pool exists

Paranoia.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: do not flush omap objects to an EC base pool

The EC pool does not support omap content.  If the caching/tiering agent
encounters such an object, just skip it.  Use the OMAP object_info_t flag
for this.

Although legacy pools will have objects with omap that do not have this
flag set, no *cache* pools yet exist, so we do not need to worry about the
agent running across legacy content.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: use reply_ctx in a few cases

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: redirect reads instead of promoting when full

If the cache pool is full, we are processing a read op, and we would
otherwise promote, redirect instead.  This lets us continue to process the
op without blocking or making the cache pool any more full than it is.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: block requests to cache PGs when they are full

If we are full and get a write request to a new object, put the op on a
wait list.  Wake up when the agent frees up some space.

Note that we do not block writes to existing objects.  That would be a
more aggressive strategy, but it is difficult to know up front whether we
will increase the size of the object or not, so we just leave it be.  I
suspect this strategy is "good enough".

Also note that we do not yet prioritize agent attention to PGs that most
need eviction (e.g., those that are full).

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: simplify agent_choose_mode

Use a temp variable.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: schedule agent from a priority queue

We need to focus agent attention on those PGs that most need it.  For
starters, full PGs need immediate attention so that we can unblock IO.
More generally, fuller ones will give us the best payoff in terms of
evicted data vs effort expended finding candidate objects.

Restructure the agent queue with priorities.  Quantize evict_effort so that
PGs do not jump between priorities too frequently.

Signed-off-by: Sage Weil <sage@inktank.com>