Add annotation to all assembly files to turn off stack-execute bit

See discussion in http://tracker.ceph.com/issues/10114

Building with these changes allows output from readelf like this:

 $ readelf -lW src/.libs/librados.so.2 | grep GNU_STACK
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8

(note the absence of 'X' in 'RW')

Fixes: #10114
Signed-off-by: Dan Mick <dan.mick@redhat.com>
(cherry picked from commit 06a245a9845c0c126fb3106b41b2fd2bc4bc4df3)
(not-yet-present-in-firefly files in isa-l manually removed)
Merge pull request #2961 from ceph/wip-10114-firefly

Add annotation to all assembly files to turn off stack-execute bit

Reviewed-by: Loic Dachary <ldachary@redhat.com>
erasure-code: store and compare encoded contents

Introduce ceph_erasure_code_non_regression to check and compare how an
erasure code plugin encodes and decodes content with a given set of
parameters. For instance:

./ceph_erasure_code_non_regression \
      --plugin jerasure \
      --parameter technique=reed_sol_van \
      --parameter k=2 \
      --parameter m=2 \
      --stripe-width 3181 \
      --create \
      --check

Will create an encoded object (--create) and store it into a directory
along with the chunks, one chunk per file. The directory name is derived
from the parameters. The content of the object is a random pattern of 31
bytes repeated to fill the object size specified with --stripe-width.

The check function (--check) reads the object back from the file,
encodes it and compares the result with the content of the chunks read
from the files. It also attempts recover from one or two erasures.

Chunks encoded by a given version of Ceph are expected to be encoded
exactly in the same way by all Ceph versions going forward.

http://tracker.ceph.com/issues/9420 Refs: #9420

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
erasure-code: workunit to check for encoding regression

Clone the archive of encoded objects and decode all archived objects, up
to and including the current ceph version.

http://tracker.ceph.com/issues/9420 Refs: #9420

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
erasure-code: add corpus verification to make check

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
ceph-disk: encapsulate partprobe / partx calls

Add the update_partition function to reduce code duplication.
The action is made an argument although it always is -a because it will
be -d when deleting a partition.

Use the update_partition function in prepare_journal_dev

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
(cherry picked from commit 922a15ea6865ef915bbdec2597433da6792c1cb2)

Conflicts:
	src/ceph-disk
ceph-disk: run partprobe after zap

Not running partprobe after zapping a device can lead to the following:

* ceph-disk prepare /dev/loop2
* links are created in /dev/disk/by-partuuid
* ceph-disk zap /dev/loop2
* links are not removed from /dev/disk/by-partuuid
* ceph-disk prepare /dev/loop2
* some links are not created in /dev/disk/by-partuuid

This is assuming there is a bug in the way udev events are handled by
the operating system.

http://tracker.ceph.com/issues/9665 Fixes: #9665

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
(cherry picked from commit fed3b06c47a5ef22cb3514c7647544120086d1e7)
ceph-disk: use update_partition in prepare_dev and main_prepare

In the case of prepare_dev the partx alternative was missing and is not
added because update_partition does it.

http://tracker.ceph.com/issues/9721 Fixes: #9721

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
(cherry picked from commit 23e71b1ee816c0ec8bd65891998657c46e364fbe)

Conflicts:
	src/ceph-disk
Merge pull request #3014 from dachary/wip-9665-ceph-disk-partprobe-firefly

ceph disk zap must call partprobe
librbd: protect list_children from invalid child pool IoCtxs

While listing child images, don't ignore error codes returned
from librados when creating an IoCtx. This will prevent seg
faults from occurring when an invalid IoCtx is used.

Fixes: #10123
Backport: giant, firefly, dumpling
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 0d350b6817d7905908a4e432cd359ca1d36bab50)
librbd: don't close an already closed parent image upon failure

If librbd is not able to open a child's parent image, it will
incorrectly close the parent image twice, resulting in a crash.

Fixes: #10030
Backport: firefly, giant
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 61ebfebd59b61ffdc203dfeca01ee1a02315133e)
osd: initialize purged_snap on backfill start; restart backfill if change

If we backfill a PG to a new OSD, we currently neglect to initialize
purged_snaps.  As a result, the first time the snaptrimmer runs it has to
churn through every deleted snap for all time, and to make matters worse
does so in one go with the PG lock held.  This leads to badness on any
cluster with a significant number of removed snaps that experiences
backfill.

Resolve this by initializing purged_snaps when we finish backfill.  The
backfill itself will clear out any stray snaps and ensure the object set
is in sync with purged_snaps.  Note that purged_snaps on the primary
that is driving backfill will not change during this period as the
snaptrimmer is not scheduled unless the PG is clean (which it won't be
during backfill).

If we by chance to interrupt backfill, go clean with other OSDs,
purge snaps, and then let this OSD rejoin, we will either restart
backfill (non-contiguous log) or the log will include the result of
the snap trim (the events that remove the trimmed snap).

Fixes: #9487
Backfill: firefly, dumpling
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 255b430a87201c7d0cf8f10a3c1e62cbe8dd2d93)
ReplicatedPG: don't move on to the next snap immediately

If we have a bunch of trimmed snaps for which we have no
objects, we'll spin for a long time.  Instead, requeue.

Fixes: #9487
Backport: dumpling, firefly, giant
Reviewed-by: Sage Weil <sage@redhat.com>
Signed-off-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit c17ac03a50da523f250eb6394c89cc7e93cb4659)
Merge pull request #3063 from ceph/wip-10123-firefly

librbd: protect list_children from invalid child pool IoCtxs

Reviewed-by: Sage Weil <sage@redhat.com>
Merge pull request #3078 from ceph/wip-10030-firefly

librbd: don't close an already closed parent image upon failure

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
rgw: run radosgw as apache with systemd

Same as sysv.

http://tracker.ceph.com/issues/10125 Fixes: #10125

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 7b621f4abf63456272dec3449aa108c89504a7a5)

Conflicts:
	src/init-radosgw.sysv
common: do not omit shard when ghobject NO_GEN is set

Do not silence the display of shard_id when generation is NO_GEN.
Erasure coded objects JSON representation used by ceph_objectstore_tool
need the shard_id to find the file containing the chunk.

Minimal testing is added to ceph_objectstore_tool.py

http://tracker.ceph.com/issues/10063 Fixes: #10063

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit dcf09aed121f566221f539106d10283a09f15cf5)

Conflicts:
	src/test/ceph_objectstore_tool.py
Merge pull request #3086 from dachary/wip-10125-radosgw-init-firefly

rgw: run radosgw as apache with systemd (firefly)
ceph-disk: dmcrypt file permissions

The directory in which key files are stored for dmcrypt must be 700 and
the file 600.

http://tracker.ceph.com/issues/9785 Fixes: #9785

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 58682d1776ab1fd4daddd887d921ca9cc312bf50)
Merge pull request #3096 from dachary/wip-9785-dmcrypt-keys-permissions-firefly

ceph-disk: dmcrypt file permissions (firefly)