ReplicatedPG: do not meaninglessly fill in the reqid on make_writeable() cloning

This reqid is used to fill in a map that is used for giving clients
the correct version on replayed ops, unless the reqid is blank (in
which case it doesn't go into the map). Indirect clones are not ops
that the client wants to observe and the version is incorrect right now,
so don't fill it in.
Note that this should not have actually caused any buggy behavior, because
the map would have simply been overwritten by the real requested event
a short time later (while still protected by locks and things). But it's
very confusing.

Signed-off-by: Greg Farnum <greg@inktank.com>
MOSDOpReply: more const-correctness

get_map_epoch() is a non-reference getter -- mark it as const!

Signed-off-by: Greg Farnum <greg@inktank.com>
osd: add is_not_empty() function to object_locator_t

Signed-off-by: Greg Farnum <greg@inktank.com>
osd: create a new request_redirect_t

We'll use this so that the OSD can tell the Objecter to redirect a
request to a different object somewhere else.

Signed-off-by: Greg Farnum <greg@inktank.com>
Objecter: rename Op::oid -> Op::base_oid

Analagous to the oloc->base_oloc rename we did in
e2fcad09d94d965867147627b73e99da9454436f, we may specify a different
target name for a redirect. Rename the existing oid to base_oid to
avoid any confusion.

Signed-off-by: Greg Farnum <greg@inktank.com>
Objecter: add an Op::target_oid, and use it when submitting Ops

For now it's just a copy of base_oid, but soon we will allow it
to be overwritten for OSD-driven redirects.

Signed-off-by: Greg Farnum <greg@inktank.com>
MOSDOpReply: add a redirect member

When present, clients must send the request to the location specified
by the redirect (by using the combine_with_locator() function on
request_redirect_t).
A separate mechanism must be used to ensure that clients see and respect
the redirect, as we do not bump up the minimum required version to
decode. We do add an assert that any redirect includes an ENOENT reply
for basic sanity-checking/poison-pilling.

Signed-off-by: Greg Farnum <greg@inktank.com>
Objecter: write a helper function to clean up ops that need to be retried

We have a little block to clean them up if we get back EAGAIN, but it's
actually leaking map references; we will also use this for redirects
from the OSDs.

Signed-off-by: Greg Farnum <greg@inktank.com>
Objecter: follow redirect replies from the OSD

If we get back a redirect reply, we clean up the Op's external references
and re-send using the target_oloc and target_oid. To facilitate this,
recalc_op_target() now only fills them in and overrides them with pool
cache semantics if they're empty.

Keep in mind that this is a pretty simple redirect formula -- the
Objecter will keep following redirects forever if that's what the OSDs
send back. The client is not providing any synchronization right now.

Signed-off-by: Greg Farnum <greg@inktank.com>
osd: implement basic caching policies in ReplicatedPG

Right now these are very basic and aren't as sophisticated as we
want them to end up, but we have a skeleton for where to put the
decision-making logic.

Signed-off-by: Greg Farnum <greg@inktank.com>
test: update cache test since OSDs support redirects

Signed-off-by: Greg Farnum <greg@inktank.com>