Merge pull request #13 from ceph/wip-full-sync

Bug fixes for full sync and a few other data sync cases
more debugging

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
worker: ignore 404 when removing op_state

It's fine if the gateway deleted it first.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
full data sync: treat 404 from bucket list as success

Since results are paged, the exception may not occur immediately, so
add a new type to distinguish it from other client exceptions, and
catch it at any point during full sync of a bucket.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
sync: use ' ' as the default shard so retries can be stored

The replica log api doesn't allow empty markers, but we may need to
store retries in the replica log even if there are no log entries. Use
' ' as a marker instead, since it is before all possible markers.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
fixup: debug notfound on op_state_remove
worker: check op state for progress on any HTTP error

500 can be caused by an fcgi timeout when the operation still ends up
succeeding. It doesn't hurt to check for op state being in progress in
general in case other error codes happen but don't indicate actual
failure of the copy, since wait_for_object fails immediately if the
op state is not in progress.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>