osd: create a new request_redirect_t

We'll use this so that the OSD can tell the Objecter to redirect a
request to a different object somewhere else.

Signed-off-by: Greg Farnum <greg@inktank.com>
Objecter: rename Op::oid -> Op::base_oid

Analagous to the oloc->base_oloc rename we did in
e2fcad09d94d965867147627b73e99da9454436f, we may specify a different
target name for a redirect. Rename the existing oid to base_oid to
avoid any confusion.

Signed-off-by: Greg Farnum <greg@inktank.com>
Objecter: add an Op::target_oid, and use it when submitting Ops

For now it's just a copy of base_oid, but soon we will allow it
to be overwritten for OSD-driven redirects.

Signed-off-by: Greg Farnum <greg@inktank.com>
MOSDOpReply: add a redirect member

When present, clients must send the request to the location specified
by the redirect (by using the combine_with_locator() function on
request_redirect_t).
A separate mechanism must be used to ensure that clients see and respect
the redirect, as we do not bump up the minimum required version to
decode.

Signed-off-by: Greg Farnum <greg@inktank.com>
Objecter: write a helper function to clean up ops that need to be retried

We have a little block to clean them up if we get back EAGAIN, but it's
actually leaking map references; we will also use this for redirects
from the OSDs.

Signed-off-by: Greg Farnum <greg@inktank.com>
Objecter: follow redirect replies from the OSD

If we get back a redirect reply, we clean up the Op's external references
and re-send using the target_oloc and target_oid. To facilitate this,
recalc_op_target() now only fills them in and overrides them with pool
cache semantics if they're empty.

Keep in mind that this is a pretty simple redirect formula -- the
Objecter will keep following redirects forever if that's what the OSDs
send back. The client is not providing any synchronization right now.

Signed-off-by: Greg Farnum <greg@inktank.com>
osd: implement basic caching policies in ReplicatedPG

Right now these are very basic and aren't as sophisticated as we
want them to end up, but we have a skeleton for where to put the
decision-making logic.

Signed-off-by: Greg Farnum <greg@inktank.com>
test: update cache test since OSDs support redirects

Signed-off-by: Greg Farnum <greg@inktank.com>