tests: shell functions toolbox dedicated to testing

The osd/osd-test-helpers.sh mon/mon-test-helpers.sh are useful to create
shell based functional tests. Since they were introduced they did not
grow significantly. As more tests are added, it becomes necessary to
have more sophisticated helpers. The ceph-helpers.sh toolbox is a merge
of the existing helpers intended to be easy to maintain and expand.

  * Each function is documented
  * Each function is tested

When adding to a function or looking for a new function, it is unlikely
the developer will find time to explore the entire file and understand
its structure. The documentation, body and test for each function are
grouped together to limit the scope of exploration required from the
casual developer.

The test are designed to not use internal helpers so that they can be
understood in full just by reading the test function. It means a given
test takes longer than it should (i.e. each getters tests run one mon).
In the context of testing this toolbox, simplicity is preferred over
efficiency.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
tests: convert osd-scrub-repair to ceph_objectstore_tool

http://tracker.ceph.com/issues/10201 Fixes: #10201

Signed-off-by: Loic Dachary <ldachary@redhat.com>
osd: accumulate authoritative peers during recovery

When PGBackend::be_compare_scrubmaps finds multiple good peers, it only
keeps the last one. This is fine for replication but erasure coding
needs to know all good peers for recovery.

PGBackend::be_compare_scrubmaps is modified to accumulate all good peers
and return them to PGBackend::be_compare_scrubmaps and indirectly to
PG::scrub_compare_maps.

PG::scrub_compare_maps will dispatch the good peers to authmap and
good_peers. In the case of authmap, the data structure is not modified
and only the last good peer is set. The ReplicatedPG::_scrub uses
authmap in a non trivial way and it should probably be modified to use
information from multiple good peers instead of just the last one. This
could be the focus of another change.

The scrubber.authoritative data structure is changed to include a list
of pair<ScrubMap::object, pg_shard_t> instead of a single
pair<ScrubMap::object, pg_shard_t> to pass to PG::repair_object and
allow it to add all good peers to the missing_loc locations if the
primary has a missing object. It could be just a list of pg_shard_t
instead because the ScrubMap::object is not used but makes more sense to
keep both and it will presumably be useful when / if the logic changes.

http://tracker.ceph.com/issues/10017 Fixes: #10017

Signed-off-by: Loic Dachary <loic@dachary.org>
erasure-code: test repair when file is removed

Add tests for when files disappear from the file system :

  * file is missing from the primary OSD
  * file is missing from an OSD that is not the primary
  * files are missing from two OSDs that are not the primary
  * files are missing from two OSDs, one of which is the primary

http://tracker.ceph.com/issues/10017 Refs: #10017

Signed-off-by: Loic Dachary <loic@dachary.org>