common/config: include --cluster in default usage message

Clean it up a bit too.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 157754b3a0191c5ff534a84adbeed88025615898)
rbd.py: increase parent name size limit

64 characters isn't all that long. 4096 ought to be enough for anyone.

Fixes: #6072
Backport: dumpling, cuttlefish
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
(cherry picked from commit 3c0042cde5a12de0f554a16b227ab437c6254ddd)
rgw: fix rgw test to reflect usage change

otherwise src/test/cli/radosgw-admin/help.t fails when running make
check when run after a configure --with-radosgw

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit e50343e4423e20130035c860ba47a0edea876f7c)
common/crc32c: fix #ifdef to be x86_64 specific

Signed-off-by: Sage Weil <sage@inktank.com>
rgw: skip read_policy checks for system_users

A system user should still be able to examine suspended buckets, and
get -ENOENT instead of -EACCESS for a deleted object.

Fixes: #6616
Backport: dumpling
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
(cherry picked from commit ea816c1c2fd47eab647d6fab96c9ca4bfeecd5bb)
cls_log: always return final marker from log_list

There's no reason to restrict returning the marker to the case where
less than the whole log is returned, since there's already a truncated
flag to tell the client what happened.

Giving the client the last marker makes it easy to consume when the
log entries do not contain their own marker. If the last marker is not
returned, the client cannot get the last marker without racing with
updates to the log.

Backport: dumpling
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
(cherry picked from commit e74776f4176470122485a79a4c07e9c12c9fc036)
rgw: include marker and truncated flag in data log list api

Consumers of this api need to know their position in the log. It's
readily available when fetching the log, so return it.  Without the
marker in this call, a client could not easily or efficiently figure
out its position in the log, since it would require getting the global
last marker in the log, and then reading all the log entries.

This would be slow for large logs, and would be subject to races that
would cause potentially very expensive duplicate work.

Returning this atomically while fetching the log entries simplifies
all of this.

Fixes: #6615
Backport: dumpling
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
(cherry picked from commit c275912509255f8bb4c854e181318b45ab0f8564)
rgw: update metadata log list to match data log list

Send the last marker whether the log is truncated in the same format
as data log list, so clients don't have more needless complexity
handling the difference.  Keep bucket index logs the same, since they
contain the marker already, and are not used in exactly the same way
metadata and data logs are.

Backport: dumpling
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
(cherry picked from commit e0e8fb1b2b4a308b2a9317e10c6fd53ad48dbfaf)
rgw: move url escaping to a common place

This is useful outside of the s3 interface. Rename url_escape()
url_encode() for consistency with the exsting common url_decode()
function. This is in preparation for the next commit, which needs
to escape url-unsafe characters in another place.

Backport: dumpling
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
(cherry picked from commit dd308cd481b368f90a64220847b91fc233d92a59)
rgw: escape bucket and object names in StreamReadRequests

This fixes copy operations for objects that contain unsafe characters,
like a newline, which would return a 403 otherwise, since the GET to
the source rgw would be unable to verify the signature on a partially
valid bucket name.

Fixes: #6604
Backport: dumpling
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
(cherry picked from commit ec45b3b88c485140781b23d2c4f582f2cc26ea43)
Formatter: add dump_bool()

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
(cherry picked from commit 28949d5d43beba7cef37cb2f83e3399d978061a6)
formatter: dump_bool dumps unquoted strings

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
(cherry picked from commit ad409f8a6d230e9b1199226a333bb54159c2c910)
rgw: try to create log pool if doesn't exist

When using replica log, if the log pool doesn't exist all operations are
going to fail. Try to create it if doesn't exist.

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
(cherry picked from commit 4216eac0f59af60f60d4ce909b9ace87a7b64ccc)
rgw: quiet down warning message

Fixes: #6123
We don't want to know about failing to read region map info
if it's not found, only if failed on some other error. In
any case it's just a warning.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 055e31359ac977943f014270cc56a1b7786abee3)
rgw: fix leak in RGWMetadataManager::remove()

Backport: dumpling
Fixes: #6445
handler->get() returns a newly allocated obj, need to delete it when
done.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
(cherry picked from commit 8dd7ea3fadc06b5ebb330af41d686d3da155fb5d)
rgw: when failing read from client, return correct error

Fixes: #6214
When getting a failed read from client when putting an object
we returned the wrong value (always 0), which in the chunked-
upload case ended up in assuming that the write was done
successfully.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 0499948aad64c6c66668b43963403fb0cb1a2737)
rgw: don't turn 404 into 400 for the replicalog api

404 is not actually a problem to clients like radosgw-agent, but 400
implies something about the request was incorrect.

Backport: dumpling
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
(cherry picked from commit 306ec7142d699c26ce874b11dc02ccdb3cf296c7)
Merge pull request #805 from ceph/wip-rgw-replica-log-dumpling

backport a bunch of rgw stuff to dumpling

Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>
rgw: Wrap hex_to_num table into class HexTable

Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>
Signed-off-by: Ray Lv <xiangyulv@gmail.com>
(cherry picked from commit 588ed60a8ec1d8db5e096fd8f7b762f2afcf7dd3)
os, osd, tools: Add backportable compatibility checking for sharded objects

OSD
  New CEPH_OSD_FEATURE_INCOMPAT_SHARDS
FileStore
  NEW CEPH_FS_FEATURE_INCOMPAT_SHARDS
  Add FSSuperblock with feature CompatSet in it
  Store sharded_objects state using CompatSet
  Add set_allow_sharded_objects() and get_allow_sharded_objects() to FileStore/ObjectStore
  Add read_superblock()/write_superblock() internal filestore functions
ceph_filestore_dump
  Add OSDsuperblock to export format
  Use CompatSet from OSD code itself in filestore-dump tool
  Always check compatibility of OSD features with on-disk features
  On import verify compatibility of on-disk features with export data
  Bump super_ver due to export format change

Backport: dumpling, cuttlefish

Signed-off-by: David Zafman <david.zafman@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit c6b83180f9f769de27ca7890f5f8ec507ee743ca)

Conflicts:

	src/os/FileStore.cc
	src/os/FileStore.h
	src/osd/OSD.cc

Excluded from cherry-pick:
  Didn't add set_allow_sharded_objects() and get_allow_sharded_objects() to FileStore/ObjectStore
  Didn't add code to check for incomplete transition to sharded objects in ceph-filestore-dump