rgw: fix multipart upload listing

Fixes: #7169

A separate fix has been created for dumpling.

Previously we read the entire list of parts, disregarding the actual
marker and the requested max parts. This fix refactors the way we read
the list of parts (doing it in parts, using marker).

Create new upload-id format that is used to identify uploads with sorted
omap entries. Make sure we're backward compatible and handle correctly
mixed-versions rgw uploads.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
rgw: fix multipart min part size

As part of the fix for wip-7169 it turned out that we removed
min_part_size. Looking back, the original implementation was broken
anyway and didn't do anything. This fixes it and makes it configurable.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
OSD: don't assume we have the pool in handle_pg_create

The pool may have been removed since the creation message
was sent.  Previously, role would end up as -1 and this
path would be avoided.

Fixes: 7190
Introduced in 268ae82ac3fe7b541c450aae168dcf9fa7294508
Signed-off-by: Samuel Just <sam.just@inktank.com>
Merge pull request #1154 from ceph/wip-7169-2

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
Merge pull request #1155 from ceph/wip-7190

OSD: don't assume we have the pool in handle_pg_create
Reviewed-by: Greg Farnum <greg@inktank.com>
OSDMap: fix deepish_copy_from

Start with a shallow copy!

Signed-off-by: Sage Weil <sage@inktank.com>
cls_user: init cls_user_stats fields in ctor

Signed-off-by: Sage Weil <sage@inktank.com>
cls_user: init cls_user_list_buckets_op fields

Signed-off-by: Sage Weil <sage@inktank.com>
civetweb: fix warning

Signed-off-by: Sage Weil <sage@inktank.com>
Add a new KV wrapper GenericObjectMap

Now we already have DBObjectMap which implement ObjectMap and other
interfaces, and ObjectMap.h implied that ObjectMap is used to encapsulates
the FileStore key value store. There exists limitation in current DBObjectMap
implementation, such as lacking of "coll_t" in "key", complicated prefix
hard-coded and inflexible extending.

So in order to provide a more flexible API and clear implementation to wrap KV
store, I copy the origin DBObjectMap and redesign the partial implementation.
Adding "coll_t" argument to all API and export "prefix" to callers. Prefixes
are divided into two parts "INTERN" and "USER". "INTERN" keys used by self to
manage and "USER" keys are managed by callers. Besides above, misc fixes are
imported such as more clear member function name and extendible header
structure.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
docs: Improve CloudStack documentation.

Add a section about creating a CephX user for CloudStack.
docs: Typo in CloudStack user creation
Add KeyValueStore implementation

KeyValueStore is another ObjectStore implementation with FileStore. It
uses KV store wrapper(StripObjectMap) which inherited GenericObjectMap
to implement ObjectStore APIs.

Each object has a header key in KV backend, which encapsulated the metadata
of object such as size, the status of keys. A complete object data maybe spread
around multi keys. The CRUD operation of object need to access the header key
of object to know the details, then the actual data keys will be get.

Now the actual KV backend of KeyValueStore is only LevelDB, more KV backend
(RocksDB, NVM API) will be introduced in the near future.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
Make ObjectStore unit tests shared with KeyValueStore

Via gtest's value-parameterized feature, let the successors of
ObjectStore share the same unit tests.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
Merge pull request #1158 from yuyuyu101/keyvaluestore-dev

Keyvaluestore Support

Reviewed-by: Sage Weil <sage@inktank.com>
keyvaluestore: name to keyvaluetore-dev for now

This helps warn the user that the ondisk format may be subject to change.

Signed-off-by: Sage Weil <sage@inktank.com>
leveldb: increase leveldb defaults (filestore, keyvaluestore)

The FileStore's leveldb currently uses libleveldb's defaults for cache and
write buffer size, which are both 4 MB.  Increase the cache size to 128MB
and the write buffer to 8MB.

Signed-off-by: Sage Weil <sage@inktank.com>