osd: improve debug output on snap collections

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 8b05492ca5f1479589bb19c1ce058b0d0988b74f)
osd: unconditionally encode snaps buffer

Previously we would only encode the updated snaps vector for CLONE ops.
This doesn't work for MODIFY ops generated by the snap trimmer, which
may also adjust the clone collections.  It is also possible that other
operations may need to populate this field in the future (e.g.,
LOST_REVERT may, although it currently does not).

Fixes: #4071, and possibly #4051.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 54b6dd924fea3af982f3d729150b6449f318daf2)
osd: include snaps in pg_log_entry_t::dump()

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 715d8717a0e8a08fbe97a3e7d3ffd33aa9529d90)
osd: update snap collections for sub_op_modify log records conditionaly

The only remaining caller is sub_op_modify().  If we do have a non-empty
op transaction, we want to do this update, regardless of what we think
last_backfill is (our notion may be not completely in sync with the
primary).  In particular, our last_backfill may be the same object but
a different snapid, but the primary disagrees and is pushing an op
transaction through.

Instead, update the collections if we have a non-empty transaction.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 31e911b63d326bdd06981ec4029ad71b7479ed70)
OSD: expand_pg_num after pg removes

Otherwise:
1) expand_pg_num removes a splitting pg entry
2) peering thread grabs pg lock and starts split
3) OSD::consume_map grabs pg lock and starts removal

At step 2), we run afoul of the assert(is_splitting)
check in split_pgs.  This way, the would be splitting
pg is marked as removed prior to the splitting state
being updated.

Backport: bobtail
Fixes: #4449
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit f1b031b3cf195cf6df3d3c47c7d606fba63ed4c4)
FileJournal: queue_pos \in [get_top(), header.max_size)

If queue_pos == header.max_size when we create the entry
header magic, the entry will be rejected at get_top() on
replay.

Fixes: #4436
Backport: bobtail
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit de8edb732e3a5ce4471670e43cfe6357ae6a2758)
PG: ignore non MISSING pg query in ReplicaActive

1) Replica sends notify
2) Prior to processing notify, primary queues query to replica
3) Primary processes notify and activates sending MOSDPGLog
to replica.
4) Primary does do_notifies at end of process_peering_events
and sends to Query.
5) Replica sees MOSDPGLog and activates
6) Replica sees Query and asserts.

In the above case, the Replica should simply ignore the old
Query.

Fixes: #4050
Backport: bobtail
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 8222cbc8f35c359a35f8381ad90ff0eed5615dac)
OSD: split temp collection as well

Otherwise, when we eventually remove the temp collection, there might be
objects in the temp collection which were independently pulled into the child
pg collection.  Thus, removing the old stale parent link from its temp
collection also blasts the omap entries and snap mappings for the real child
object.

Backport: bobtail
Fixes: #4452
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
(cherry picked from commit f8d66e87a5c155b027cc6249006b83b4ac9b6c9b)
osd: Add digest of omap for deep-scrub

Add ScrubMap encode/decode v4 message with omap digest
Compute digest of header and key/value.  Use bufferlist
to reflect structure and compute as we go, clearing
bufferlist to reduce memory usage.

Signed-off-by: David Zafman <david.zafman@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 509a93e89f04d7e9393090563cf7be8e0ea53891)
OSD::load_pgs: first scan colls before initing PGs

Backport: bobtail
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 073f58ede2e473af91f76d01679631c169274af7)
osd: fix load_pgs handling of pg dirs without a head

If there is a pgid that passes coll_t::is_pg() but there is no head, we
will populate the pgs map but then fail later when we try to do
read_state.  This is a side-effect of 55f8579.

Take explicit note of _head collections we see, and then warn when we
find stray snap collections.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 1f80a0b576c0af1931f743ad988b6293cbf2d6d9)
osd: fix load_pgs collection handling

On a _TEMP pg, is_pg() would succeed, which meant we weren't actually
hitting the cleanup checks.  Instead, restructure this loop as positive
checks and handle each type of collection we understand.

This fixes _TEMP cleanup.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit b19b6dced85617d594c15631571202aab2f94ae8)