osd: accumulate authoritative peers during recovery

When PGBackend::be_compare_scrubmaps finds multiple good peers, it only
keeps the last one. This is fine for replication but erasure coding
needs to know all good peers for recovery.

PGBackend::be_compare_scrubmaps is modified to accumulate all good peers
and return them to PGBackend::be_compare_scrubmaps and indirectly to
PG::scrub_compare_maps.

PG::scrub_compare_maps will dispatch the good peers to authmap and
good_peers. In the case of authmap, the data structure is not modified
and only the last good peer is set. The ReplicatedPG::_scrub uses
authmap in a non trivial way and it should probably be modified to use
information from multiple good peers instead of just the last one. This
could be the focus of another change.

The scrubber.authoritative data structure is changed to include a list
of pair<ScrubMap::object, pg_shard_t> instead of a single
pair<ScrubMap::object, pg_shard_t> to pass to PG::repair_object and
allow it to add all good peers to the missing_loc locations if the
primary has a missing object. It could be just a list of pg_shard_t
instead because the ScrubMap::object is not used but makes more sense to
keep both and it will presumably be useful when / if the logic changes.

http://tracker.ceph.com/issues/10017 Fixes: #10017

Signed-off-by: Loic Dachary <loic@dachary.org>
tests: convert osd-scrub-repair to ceph_objectstore_tool

http://tracker.ceph.com/issues/10201 Fixes: #10201

Signed-off-by: Loic Dachary <ldachary@redhat.com>
erasure-code: test repair when file is removed

Add tests for when files disappear from the file system :

  * file is missing from the primary OSD
  * file is missing from an OSD that is not the primary
  * files are missing from two OSDs that are not the primary
  * files are missing from two OSDs, one of which is the primary

http://tracker.ceph.com/issues/10017 Refs: #10017

Signed-off-by: Loic Dachary <loic@dachary.org>