osd: add a separate messenger for the Objecter

We will give the OSD's Objecter its own messenger so that it does not
interfere with the OSD when it marks things up or down.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: add an Objecter instance

It gets its own lock, timer, and osdmap.

Signed-off-by: Sage Weil <sage@inktank.com>
osdc/Objecter: only request map on startup if epoch == 0

Normal clients have no map and need one to get started.  If we are the
OSD, we will already have one and will get fed maps as they come in.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: feed OSDMaps to the Objecter

Feed every map message we see (that isn't discarded for some other
reason) to the Objecter.  It has the same continuity requirements that
the OSD has, so it should be satisfied with what we get.  It can also
request maps via our MonClient.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: less whitespace

Signed-off-by: Sage Weil <sage@inktank.com>
ceph-osd: rename msgr vars

Signed-off-by: Sage Weil <sage@inktank.com>
osdc/Objecter: allow ops to be canceled

This is useful in general, and specifically will be useful for the
rados COPY operation.

Signed-off-by: Sage Weil <sage@inktank.com>
os/ObjectStore: add bufferlist variant of setattrs

And hopefully we can kill the bufferptr ones someday!

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: add whitespace

Signed-off-by: Sage Weil <sage@inktank.com>
osd: wire Objecter up to its messenger

Feed the Objecter connect and reset events as well as any messages that
come in on its messenger.  That should only be MOSDOp and MOSDOpReply
at this point, since the MonClient is shared with the OSD.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: discriminate based on connection messenger, not peer type

Replace ->get_source().is_osd() checks and instead see if it is the
cluster_messenger so that we do not confuse ourselves when we get
legit requests from other OSDs on our public interface.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: add objecter_finisher

The Objecter does completions synchronously, but that will run under
the objecter_lock.  We'll want to nest that inside the PG lock to queue
new IOs, which means the completions will need to run async so that
they can retake the pg lock safely.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: drop src_obc.clear() calls

These are all about to go out of scope; no need to clear them
explicitly.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: factor {execute,reply}_ctx() out of do_op()

Separate the processing of an OpContext from the preamble and
allocation, so that we can delay the execution for some ops (like the
COPYFROM operation we're about to add).

Signed-off-by: Sage Weil <sage@inktank.com>
osd, objecter: clean up assert_ver()

Create a separate union in the args and clean up the code a bit so that
this doesn't reuse the (unrelated) watch helpers.  No change in
protocol.

Signed-off-by: Sage Weil <sage@inktank.com>
objecter, librados: add COPYFROM operation

This operation will copy an entire object (data, attrs, omap)
atomically.  If the src_version does not match the source object, or
the source object is updated while the copy is in progress, we will
fail with a suitable error code.  By atomic we mean that it will either
successfully copy the entire object in its entirety or it will fail (and
require no cleanup).

Add to C++ librados API only for now.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: COPYSTART, COPYMORE operations

Add new rados operations to copy all user-visible content for an object
in a simple, safe way.  COPYSTART gets the initial metadata about the
object (size, mtime, attrs) and then however much data (data bytes,
omap) there is room for.  COPYMORE gets more bytes|omap.

Signed-off-by: Sage Weil <sage@inktank.com>
osdc/Objecter: do not steal write buffer

This is rude.  The original motivation was to avoid the std::list copy,
but I do not think it is worth the interface fragility.

Signed-off-by: Sage Weil <sage@inktank.com>
osdc/Objecter: mark input bufferlists const

Signed-off-by: Sage Weil <sage@inktank.com>
osd/ReplicatedPG: do not requeue if not primary

This saves us a bit of work, since we will discard the op anyway if
we aren't primary (or even if we become primary again before we get to
it).

Signed-off-by: Sage Weil <sage@inktank.com>