erasure-code: store and compare encoded contents

Introduce ceph_erasure_code_non_regression to check and compare how an
erasure code plugin encodes and decodes content with a given set of
parameters. For instance:

./ceph_erasure_code_non_regression \
      --plugin jerasure \
      --parameter technique=reed_sol_van \
      --parameter k=2 \
      --parameter m=2 \
      --stripe-width 3181 \
      --create \
      --check

Will create an encoded object (--create) and store it into a directory
along with the chunks, one chunk per file. The directory name is derived
from the parameters. The content of the object is a random pattern of 31
bytes repeated to fill the object size specified with --stripe-width.

The check function (--check) reads the object back from the file,
encodes it and compares the result with the content of the chunks read
from the files. It also attempts recover from one or two erasures.

Chunks encoded by a given version of Ceph are expected to be encoded
exactly in the same way by all Ceph versions going forward.

http://tracker.ceph.com/issues/9420 Refs: #9420

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
(cherry picked from commit f5901303dbf50e9d08f2f1e510a1936a20037909)
erasure-code: workunit to check for encoding regression

Clone the archive of encoded objects and decode all archived objects, up
to and including the current ceph version.

http://tracker.ceph.com/issues/9420 Refs: #9420

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
(cherry picked from commit 7638b15f23976c3265cf766e16cf93af1a7e0091)
erasure-code: s/alignement/alignment/ typos in jerasure

The jerasure-per-chunk-alignment prameter was mispelled and while
useable that would lead to confusion.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
(cherry picked from commit 2c84d0b1db57d918840e669a17bbd8c5ddca9747)
erasure-code: Makefile.am cosmetics

Cluster benchmark related lines together.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
(cherry picked from commit 10c88c8f27080a8e25f128b7065cee5c2f68e91b)
erasure-code: add corpus verification to make check

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
(cherry picked from commit 6fdbdff2ad1b55d4a37dcb95cfbb06c4454cdaf2)
erasure-code: ceph_erasure_code does not need to avoid dlclose

The only reason for not dlclosing plugins at exit is for callgrind but
ceph_erasure_code has no workload that would require callgrind.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
(cherry picked from commit 49613cb2aab6e73e3ea50fa164735b55e80121cd)
erasure_code: implement ceph_erasure_code to assert the existence of a plugin

This is handy when scripting in the context of teuthology and only
conditionally run tests for the isa plugin, for instance.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
(cherry picked from commit efe121d9f2028c312eef2650d32ccf0cbc828edb)
erasure-code: add erasure_code_benchmark --verbose

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
(cherry picked from commit 3ff2816b3eecfb7277295583387549dac5429628)

Conflicts:
	src/test/erasure-code/ceph_erasure_code_benchmark.cc
	src/test/erasure-code/ceph_erasure_code_benchmark.h
erasure-code: erasure_code_benchmark exhaustive erasure exploration

Add the --erasure-generation exhaustive flag to try all combinations of
erasures, not just one at random.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
(cherry picked from commit 2d7adb23bc52e7c0753f4571fecd8eefa209ef02)

Conflicts:
	src/test/erasure-code/ceph_erasure_code_benchmark.h
osd: deep scrub must not abort if hinfo is missing

Instead it should set read_error.

http://tracker.ceph.com/issues/10018 Fixes: #10018

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
(cherry picked from commit 9d84d2e8309d26e39ca849a75166d2d7f2dec9ea)
librbd: protect list_children from invalid child pool IoCtxs

While listing child images, don't ignore error codes returned
from librados when creating an IoCtx. This will prevent seg
faults from occurring when an invalid IoCtx is used.

Fixes: #10123
Backport: giant, firefly, dumpling
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 0d350b6817d7905908a4e432cd359ca1d36bab50)
librbd: don't close an already closed parent image upon failure

If librbd is not able to open a child's parent image, it will
incorrectly close the parent image twice, resulting in a crash.

Fixes: #10030
Backport: firefly, giant
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 61ebfebd59b61ffdc203dfeca01ee1a02315133e)
mon: OSDMonitor: allow adding tiers to FS pools

This was an overly-strict check.  In fact it is perfectly
fine to set an overlay on a pool that is already in use
as a filesystem data or metadata pool.

Fixes: #10135

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 17b5fc9a40440e76dd1fa64f7fc19577ae3b58ce)
Merge pull request #3055 from ceph/wip-10135-giant

mon: OSDMonitor: allow adding tiers to FS pools

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Merge pull request #3062 from ceph/wip-10123-giant

librbd: protect list_children from invalid child pool IoCtxs

Reviewed-by: Sage Weil <sage@redhat.com>
Merge pull request #3077 from ceph/wip-10030-giant

librbd: don't close an already closed parent image upon failure

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
rgw: run radosgw as apache with systemd

Same as sysv.

http://tracker.ceph.com/issues/10125 Fixes: #10125

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 7b621f4abf63456272dec3449aa108c89504a7a5)

Conflicts:
	src/init-radosgw.sysv
common: do not omit shard when ghobject NO_GEN is set

Do not silence the display of shard_id when generation is NO_GEN.
Erasure coded objects JSON representation used by ceph_objectstore_tool
need the shard_id to find the file containing the chunk.

Minimal testing is added to ceph_objectstore_tool.py

http://tracker.ceph.com/issues/10063 Fixes: #10063

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit dcf09aed121f566221f539106d10283a09f15cf5)
Merge pull request #3085 from dachary/wip-10125-radosgw-init-giant

rgw: run radosgw as apache with systemd (giant)
ceph-disk: dmcrypt file permissions

The directory in which key files are stored for dmcrypt must be 700 and
the file 600.

http://tracker.ceph.com/issues/9785 Fixes: #9785

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 58682d1776ab1fd4daddd887d921ca9cc312bf50)