Getting code readh for handoff

Signed-off-by: Joe Buck <jbbuck@gmail.com>
sync: remove extraneous error checking

These loops all end once a result of any sort is received for a child
process. It will already complete if every worked has a connection
error.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
client: fix spacing

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
client: escape bucket and object names used in urls

Convert to utf8 since urllib doesn't handle unicode well in python 2.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
Refactor sync and fix errors found by pyflakes

Instead of 4 copies of queue and worker creation
and results gathering, just have one, with subclasses
that can add special behavior at the beginning or end.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
Set up full data sync

Get data log markers and store them after all buckets are synced. Also
get bucket-index markers for each bucket, and update them in the
bucket index replica log once syncing is complete. The bucket-index
replica log updates should be moved to happend after each bucket once
the actual object copying is implemented.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
Fix cli to match obj/method names according to sync.py

Signed-off-by: Christophe Courtaut <christophe.courtaut@gmail.com>
Adds data sync capabilities to test server
Merge pull request #1 from kri5/wip-datasync

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
Implement data sync and several other changes that should be separate commits

The larger changes are:
- metadata sync uses the replica log for retrying after errors
- command line usage is simpler, and requires fewer options
- source and destination are gotten from / validated against
  the region map, with unit tests

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
Add forgotten files for running tests
Fix option parsing for destination arg

As mentionned at http://docs.python.org/3.4/library/argparse.html#nargs,
nargs=1 will not provide the default behaviour, but instead will produce
a list of args, and that is not what we want.

This patch uses nargs=None to provide default behaviour instead.

As it is a positionnal argument, we can't use required option.

Signed-off-by: Christophe Courtaut <christophe.courtaut@gmail.com>
Forgot to add unit tests dir as well

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
Merge pull request #3 from kri5/wip-datasync

Fix option parsing for destination arg

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
Rename variable to match option parsing variable name

The data_log_window variable was renamed rgw_data_log_window
in the option parser, but not elsewhere.

This patch fixes it.

Signed-off-by: Christophe Courtaut <christophe.courtaut@gmail.com>
Merge pull request #4 from kri5/wip-datasync

Rename variable to match option parsing variable name

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
Rename variable to match option parsing variable name (again)

The object_timeout variable was renamed object_sync_timeout
in the option parser, but not elsewhere.

This patch fixes it.

Signed-off-by: Christophe Courtaut <christophe.courtaut@gmail.com>
Merge pull request #5 from kri5/wip-datasync

Rename variable to match option parsing variable name (again)

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
Handle errors preparing for an incremental sync

Log the issue, but keep retrying with a delay.
Move continuous incremental sync and preparing retry into
helpers in sync.py.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
reduce default logging level for boto

Log messages about each new connection aren't very useful.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>