osd: discriminate based on connection messenger, not peer type

Replace ->get_source().is_osd() checks and instead see if it is the
cluster_messenger so that we do not confuse ourselves when we get
legit requests from other OSDs on our public interface.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: add objecter_finisher

The Objecter does completions synchronously, but that will run under
the objecter_lock.  We'll want to nest that inside the PG lock to queue
new IOs, which means the completions will need to run async so that
they can retake the pg lock safely.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: drop src_obc.clear() calls

These are all about to go out of scope; no need to clear them
explicitly.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: factor {execute,reply}_ctx() out of do_op()

Separate the processing of an OpContext from the preamble and
allocation, so that we can delay the execution for some ops (like the
COPYFROM operation we're about to add).

Signed-off-by: Sage Weil <sage@inktank.com>
osd, objecter: clean up assert_ver()

Create a separate union in the args and clean up the code a bit so that
this doesn't reuse the (unrelated) watch helpers.  No change in
protocol.

Signed-off-by: Sage Weil <sage@inktank.com>
objecter, librados: add COPYFROM operation

This operation will copy an entire object (data, attrs, omap)
atomically.  If the src_version does not match the source object, or
the source object is updated while the copy is in progress, we will
fail with a suitable error code.  By atomic we mean that it will either
successfully copy the entire object in its entirety or it will fail (and
require no cleanup).

Add to C++ librados API only for now.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: COPYSTART, COPYMORE operations

Add new rados operations to copy all user-visible content for an object
in a simple, safe way.  COPYSTART gets the initial metadata about the
object (size, mtime, attrs) and then however much data (data bytes,
omap) there is room for.  COPYMORE gets more bytes|omap.

Signed-off-by: Sage Weil <sage@inktank.com>
osdc/Objecter: do not steal write buffer

This is rude.  The original motivation was to avoid the std::list copy,
but I do not think it is worth the interface fragility.

Signed-off-by: Sage Weil <sage@inktank.com>
osdc/Objecter: mark input bufferlists const

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: do not requeue if not primary

This saves us a bit of work, since we will discard the op anyway if
we aren't primary (or even if we become primary again before we get to
it).

Signed-off-by: Sage Weil <sage@inktank.com>
osd: ObjectContext: add explicit obc state

Previously we had semi-implicit lock state from the unstable_writes,
readers, writers_waiting, and readers_waiting counters.  Make this
explicitly reflected in a state_t state field and guard all state
transitions (currently in the existing ondisk_*_*lock methods) with
asserts.

We will add new states shortly.

Signed-off-by: Sage Weil <sage@inktank.com>
copyfrom wip
count coypfrom_readsides in obc
only one copyfrom in progress per object at any time
cancel (and possibly requeue) copy ops on_change and on_shutdown
fix copyfrom cleanup
adapt to new copy_get