OSDMap: deepish_copy_from()

Make a deep(ish) copy of another OSDMap.  Unfortunatley we can't make the
compiler-generated copy operator/constructors private until c++11.  :(

Signed-off-by: Sage Weil <sage@inktank.com>
OSDMap: fix damaging input osdmap from remove_down_temps

The default copy constructor copies shared_ptrs do vectors that are then
modified by apply_incremental, which means that the const osdmap argument
isn't in fact const.  Fix this by doing a deep(ish) copy.

Fixes: #7060
Signed-off-by: Sage Weil <sage@inktank.com>
OSDMap: use deepish_copy_from in remove_redundant_temporaries

Better not to open-code it.  We do a bit of extra work here, but it's worth
it.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: OSDMonitor: ignore pgtemps from removed pool

There's a window in-between receiving an MOSDPGTemp message from an OSD
and actually handling it that may lead to the pool the pg temps refer to
no longer existing. This may happen if the MOSDPGTemp message is queued
pending dispatching due to an on-going proposal (maybe even the pool
removal).

This patch fixes such behavior in two steps:

1. Check if the pool exists in the osdmap upon preprocessing
 - if pool does not exist in the osdmap, then the pool must have been
   removed prior to handling the message, but after the osd sent it.
 - safe to ignore the pg update
2. If all pg updates in the message have been ignored, ignore the whole
   message.  Otherwise, let prepare handle the rest.

3. Recheck if pool exists in the osdmap upon prepare
 - We may have ignored this pg back in preprocess, but other pgs in the
   message may have led the message to be passed on to prepare; ignore
   pg update once more.
4. Check if pool is pending removal and ignore pg update if so.

We delegate checking the pending value to prepare_pgtemp() because in this
case we should only ignore the update IFF the pending value is in fact
committed.  Otherwise we should retry the message.  prepare_pgtemp() is
the appropriate place to do so.

Fixes: 7116

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
Merge pull request #1137 from ceph/wip-7060

fix #7060 mon scrub problem due to osdmap copying

Reviewed-by: Greg Farnum <greg@inktank.com>
test_rgw_admin_opstate: get it compiled

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
test/Makefile.am: update test_cls_rgw* rules

Need to link with libcls_user_client.a

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Merge pull request #1153 from ceph/wip-7116-joao

osd: OSDMonitor: ignore pgtemps from removed pool

Reviewed-by: Sage Weil <sage@inktank.com>
rgw: fix multipart upload listing

Fixes: #7169

A separate fix has been created for dumpling.

Previously we read the entire list of parts, disregarding the actual
marker and the requested max parts. This fix refactors the way we read
the list of parts (doing it in parts, using marker).

Create new upload-id format that is used to identify uploads with sorted
omap entries. Make sure we're backward compatible and handle correctly
mixed-versions rgw uploads.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Add a virtual interface init, open, create_and_open to KeyValueDB

This will provide an opportunity to set leveldb_* settings in the next
patch.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
Signed-off-by: Sage Weil <sage@inktank.com>
rgw: fix multipart min part size

As part of the fix for wip-7169 it turned out that we removed
min_part_size. Looking back, the original implementation was broken
anyway and didn't do anything. This fixes it and makes it configurable.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
leveldb: add leveldb_* options

The osd_leveldb_* and mon_leveldb_* are now deprecated.  If they are
still used, they override these values.

Signed-off-by: Sage Weil <sage@inktank.com>
OSD: don't assume we have the pool in handle_pg_create

The pool may have been removed since the creation message
was sent.  Previously, role would end up as -1 and this
path would be avoided.

Fixes: 7190
Introduced in 268ae82ac3fe7b541c450aae168dcf9fa7294508
Signed-off-by: Samuel Just <sam.just@inktank.com>
Merge pull request #1154 from ceph/wip-7169-2

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
Merge pull request #1155 from ceph/wip-7190

OSD: don't assume we have the pool in handle_pg_create
Reviewed-by: Greg Farnum <greg@inktank.com>
OSDMap: fix deepish_copy_from

Start with a shallow copy!

Signed-off-by: Sage Weil <sage@inktank.com>
cls_user: init cls_user_stats fields in ctor

Signed-off-by: Sage Weil <sage@inktank.com>
cls_user: init cls_user_list_buckets_op fields

Signed-off-by: Sage Weil <sage@inktank.com>
civetweb: fix warning

Signed-off-by: Sage Weil <sage@inktank.com>
Add a new KV wrapper GenericObjectMap

Now we already have DBObjectMap which implement ObjectMap and other
interfaces, and ObjectMap.h implied that ObjectMap is used to encapsulates
the FileStore key value store. There exists limitation in current DBObjectMap
implementation, such as lacking of "coll_t" in "key", complicated prefix
hard-coded and inflexible extending.

So in order to provide a more flexible API and clear implementation to wrap KV
store, I copy the origin DBObjectMap and redesign the partial implementation.
Adding "coll_t" argument to all API and export "prefix" to callers. Prefixes
are divided into two parts "INTERN" and "USER". "INTERN" keys used by self to
manage and "USER" keys are managed by callers. Besides above, misc fixes are
imported such as more clear member function name and extendible header
structure.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>