ReplicatedPG: copy: switch out the CopyCallback interface

The tuple was already unwieldy with 4 members; I didn't want to add
more. Instead, create a new CopyResults struct which contains all the
object info and completion data, and pass the retval and a CopyResults*
in the CopyCallbackResults tuple.

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
ReplicatedPG: promote: first draft pass at doing object promotion

This is not yet at all complete -- among other things, it will
retry forever on any object which doesn't exist in the underlying
pool. But it demonstrates the approach reasonably clearly.

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>y
ReplicatedPG: copy: don't return from finish_copyfrom

The return value is meaningless; nothing in this function can fail.

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
ReplicatedPG: promote: move the promote functionality into finish_promote

This way we can have a couple of functions that handle each type of
case, and let the PromoteCallback choose between them. That's much
better than doing it all inline in the callback.

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
ReplicatedPG: promote: add the OpRequest to the Callback

This way we can do stuff to it, and we're about to.

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
ReplicatedPG: promote: handle failed promotes

If we get an error back, reply to the client directly and remove
the op which triggered promotion from our blocked op queue.

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
workunits: check errors propagate on cache pools in caching_redirects.sh

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
workunits: break down cache pool tests to be more precise; expand some

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>