radosgw-admin: various multiregion related fixes and changes

Go to the master zone in the master region for radosgw-admin
operations. Trigger metadata sync. Other fixes.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
agent name should also contain port number

otherwise multiple agents per machines will have the same id

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
rgy.py: make log_(data|metadata) configurable

The log_data and log_metadata are made configurable
via the YAML file and default to false
(meaning neither data nor metadata operations are
logged).

Signed-off-by: Joe Buck <jbbuck@gmail.com>
rgw.py: refactor configure

Extend the rados pool configuration options to
specify all pools (if desired).
Also, reordered zone and region configuration
so that they're configure (per client) in
this order: zone, region, set default region

Signed-off-by: Joe Buck <jbbuck@gmail.com>
s3readwrite.py: make user creation optional

Extend the s3readwrite.py task to enable the
creation and deletion of users for the s3readwrite
tests to be independently specified with the default
assumption to both being true.
This is needed for tests that will create a user and
data in one execution and read it in another.

Signed-off-by: Joe Buck <jbbuck@gmail.com>
rgw.py: add a better error message

When pulling region info from the config
structure, if the region info isn't there,
log a more helpful message.

Signed-off-by: Joe Buck <jbbuck@gmail.com>
rgw.py: fix example config

In the example config, the region root and
zone root where pointing to the same pool,
which is not a best practice. Updated the
example to show them pointing to different pools.

Signed-off-by: Joe Buck <jbbuck@gmail.com>
radosgw-agent.py: refactor, enable overrides

Refactored the radosgw-agent.py code so that it
is structured more like existing teuthology
tasks.
Additionally, added code to enable:
using the override field in YAML files,
specifying which radosgw-agent github branch
to use checkout and for the YAML file to
specify one of the following: a full sync,
an incremental sync and the starting of the
test radosgw-agent server (previously the
server was always started by this task).

Signed-off-by: Joe Buck <jbbuck@gmail.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>