osd/ReplicatedPG: skip promote for DELETE

If an op starts with DELETE there is no need to promote the old content
from the base tier.  Note that this only works if the FAILOK flag is
set.  Otherwise, we need to know whether the object existed or not to
return either 0 or -ENOENT.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplcatedPG: maybe_handle_cache style

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: allow osds to issue writes to osds

We asserted that the client was not an OSD years ago when we separated out
the client and cluster networks.  Now, we are about to allow an OSD to
trigger a copy_from on another pool (for cache flush) and the assert can
go away.  We've long since verified that the messages are going out on
the correct interfaces.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: drop unnecessary temp vars in execute_ctx()

Both of these are pulled out of ctx->obs, which is not updated until the
very end; use that instead!

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: be consistent about ctx->obs vs ctx->obc->obs

Just for consistency (ctx->obs =- &ctx->obc->obs).

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: add SKIPRWLOCKS flag

Flush puts us in an conundrum:

 - the flush eventually writes, behaving like a write
 - writes take the write lock at the start
 - to flush, we send copy-from to the base pool, which does a copy-get on
   our object
 - the copy-get is a read, that blocks on the write.

This flag will allow an op to skip the initial locking step.  It will need
to take it later, of course.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: split off finish_ctx from execute_ctx

The second part of execute_ctx() is doing some somewhat generic work to
make the prepared updates in the ctx apply, updating the obc's cached
values.  Factor it out.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: use get_next_version() in finish_promote

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: use finish_ctx for finish_promote

Use the common code here to avoid duplicating this logic.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: set ctx->obc in simple_repop_create

Strangely nobody hss needed this yet, but we will shortly.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: more verbose heading for process_copy_chunk

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: implement cache-flush

Implement a rados operation that will flush a dirty object in the cache
tier by writing it back to the base tier.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: handle dup flushes when object has not changed

If we have an in-progress flush that is still on track to complete, and
get another flush request, piggyback on the existing flush.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: handle dup flushes after a write race

If we

 - start a flush
 - update the object
 - get another flush

then we can cancel the previous flush (we know it will fail) and start a
new attempt.

Signed-off-by: Sage Weil <sage@inktank.com>