rgw: set up curl with CURL_NOSIGNAL

Fixes: #4425
Backport: bobtail
Apparently, libcurl needs that in order to be thread safe. Side
effect is that if libcurl is not compiled with c-ares support,
domain name lookups are not going to time out.
Issue affected keystone.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 88725316ddcfa02ff110e659f7a8131dc1ea2cfc)
ReplicatedPG: don't leak reservation on removal

Fixes: 4431
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 32bf131e0141faf407b5ff993f75f97516b27c12)

Conflicts:

	src/osd/ReplicatedPG.cc
OSD: lock not needed in ~DeletingState()

No further refs to the object can remain at this point.
Furthermore, the callbacks might lock mutexes of their
own.

Backport: bobtail
Fixes: #4378
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit e4bf1bcab159d7c5b720f5da01877c0f67c16d16)
HashIndex: _collection_list_partial must tolerate NULL next

Backport: bobtail
Fixes: #4379
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit ce4432adc67dc2fc06dd21ea08e59d179496bcc6)
PG: check_recovery_sources must happen even if not active

missing_loc/missing_loc_sources also must be cleaned up
if a peer goes down during peering:

1) pg is in GetInfo, acting is [3,1]
2) we find object A on osd [0] in GetInfo
3) 0 goes down, no new peering interval since it is neither up nor
acting, but peer_missing[0] is removed.
4) pg goes active and try to pull A from 0 since missing_loc did not get
cleaned up.

Backport: bobtail
Fixes: #4371
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit de22b186c497ce151217aecf17a8d35cdbf549bb)
osd: improve debug output on snap collections

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 8b05492ca5f1479589bb19c1ce058b0d0988b74f)
osd: unconditionally encode snaps buffer

Previously we would only encode the updated snaps vector for CLONE ops.
This doesn't work for MODIFY ops generated by the snap trimmer, which
may also adjust the clone collections.  It is also possible that other
operations may need to populate this field in the future (e.g.,
LOST_REVERT may, although it currently does not).

Fixes: #4071, and possibly #4051.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 54b6dd924fea3af982f3d729150b6449f318daf2)
osd: include snaps in pg_log_entry_t::dump()

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 715d8717a0e8a08fbe97a3e7d3ffd33aa9529d90)
osd: update snap collections for sub_op_modify log records conditionaly

The only remaining caller is sub_op_modify().  If we do have a non-empty
op transaction, we want to do this update, regardless of what we think
last_backfill is (our notion may be not completely in sync with the
primary).  In particular, our last_backfill may be the same object but
a different snapid, but the primary disagrees and is pushing an op
transaction through.

Instead, update the collections if we have a non-empty transaction.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 31e911b63d326bdd06981ec4029ad71b7479ed70)
OSD: expand_pg_num after pg removes

Otherwise:
1) expand_pg_num removes a splitting pg entry
2) peering thread grabs pg lock and starts split
3) OSD::consume_map grabs pg lock and starts removal

At step 2), we run afoul of the assert(is_splitting)
check in split_pgs.  This way, the would be splitting
pg is marked as removed prior to the splitting state
being updated.

Backport: bobtail
Fixes: #4449
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit f1b031b3cf195cf6df3d3c47c7d606fba63ed4c4)
FileJournal: queue_pos \in [get_top(), header.max_size)

If queue_pos == header.max_size when we create the entry
header magic, the entry will be rejected at get_top() on
replay.

Fixes: #4436
Backport: bobtail
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit de8edb732e3a5ce4471670e43cfe6357ae6a2758)
PG: ignore non MISSING pg query in ReplicaActive

1) Replica sends notify
2) Prior to processing notify, primary queues query to replica
3) Primary processes notify and activates sending MOSDPGLog
to replica.
4) Primary does do_notifies at end of process_peering_events
and sends to Query.
5) Replica sees MOSDPGLog and activates
6) Replica sees Query and asserts.

In the above case, the Replica should simply ignore the old
Query.

Fixes: #4050
Backport: bobtail
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 8222cbc8f35c359a35f8381ad90ff0eed5615dac)
OSD: split temp collection as well

Otherwise, when we eventually remove the temp collection, there might be
objects in the temp collection which were independently pulled into the child
pg collection.  Thus, removing the old stale parent link from its temp
collection also blasts the omap entries and snap mappings for the real child
object.

Backport: bobtail
Fixes: #4452
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
(cherry picked from commit f8d66e87a5c155b027cc6249006b83b4ac9b6c9b)
osd: Add digest of omap for deep-scrub

Add ScrubMap encode/decode v4 message with omap digest
Compute digest of header and key/value.  Use bufferlist
to reflect structure and compute as we go, clearing
bufferlist to reduce memory usage.

Signed-off-by: David Zafman <david.zafman@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 509a93e89f04d7e9393090563cf7be8e0ea53891)
OSD::load_pgs: first scan colls before initing PGs

Backport: bobtail
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 073f58ede2e473af91f76d01679631c169274af7)
osd: fix load_pgs handling of pg dirs without a head

If there is a pgid that passes coll_t::is_pg() but there is no head, we
will populate the pgs map but then fail later when we try to do
read_state.  This is a side-effect of 55f8579.

Take explicit note of _head collections we see, and then warn when we
find stray snap collections.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 1f80a0b576c0af1931f743ad988b6293cbf2d6d9)
osd: fix load_pgs collection handling

On a _TEMP pg, is_pg() would succeed, which meant we weren't actually
hitting the cleanup checks.  Instead, restructure this loop as positive
checks and handle each type of collection we understand.

This fixes _TEMP cleanup.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit b19b6dced85617d594c15631571202aab2f94ae8)