PG::proc_replica_log: adjust oinfo.last_complete based on omissing

Otherwise, search_for_missing may neglect to check the missing
set for some objects assuming that if the need version is
prior to last_complete, the replica must have it.

Fixes: #4994
Signed-off-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit ce7ffc34408bf32c66dc07e6f42d54b7ec489d41)
osd/PG: fix typo, missing -> omissing

From ce7ffc34408bf32c66dc07e6f42d54b7ec489d41.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit dc181224abf6fb8fc583730ae3d90acdf0b80f39)
PG::proc_replica_log: oinfo.last_complete must be *before* first entry in omissing

Fixes: #4189
Signed-off-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 2dae6a68ee85a20220ee940dbe33a2144d43457b)
os/FileStore: check replay guard on src for collection rename

This avoids a problematic sequence like:

     - rename A/ -> B/
     - remove B/1...100
     - destroy B/
     - create A/
     - write A/101...
     <crash>
     - replay A/ -> B/
     - remove B/1...100  (fails but tolerated)
     - destroy B/        (fails with ENOTEMPTY)

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 5fc83c8d9887d2a916af11436ccc94fcbfe59b7a)
client: use 4MB f_bsize and f_frsize for statfs

Old stat(1) reports:

  Block size: 1048576    Fundamental block size: 1048576

and the df(1) arithmetic works out.  New stat(1) reports:

  Block size: 1048576    Fundamental block size: 4096

which is what we are shoving into statvfs, but we have the b_size and
fr_size arithmetic swapped.  However, doing the *correct* reporting would
then break the old stat by making both sizes appear to be 4KB (or
whatever).

Sidestep the issue by making *both* values 4MB.. which is both large enough
to report large FS sizes, and also the default stripe size and thus a
"reasonable" value to report for a block size.

Perhaps in the future, when we no longer care about old userland, we can
report the page size for f_bsize, which is probably the "most correct"
thing to do.

Fixes: #3794.  See also #3793.
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 7c94083643891c9d66a117352f312b268bdb1135)
FileStore: _split_collection should not create the collection

This will simplify adding a replay guard to create_collection.

Backport: bobtail
Signed-off-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit b184ff581a08c9e6ce5b858f06ccbe9d0e2a170b)
FileStore: set replay guard on create_collection

This should prevent sequences like:

rmcoll a
mkcoll a
touch a foo
<crash>

from causing trouble by preventing the rmcoll
and mkcoll from being replayed.

Fixes: 4064
Backport: bobtail
Signed-off-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 411770c45734c9827745ddc4018d86c14f2858a6)
FileStore: fix reversed collection_empty return value

Backport: bobtail
Fixes: #4380
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 7a434d10da0f77e1b329de0b06b6645cd73cc81b)

Conflicts:
	src/os/FileStore.cc
PG,osd_types: improve check_new_interval debugging

Backport: bobtail
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit d611eba9caf45f2d977c627b123462a073f523a4)
PG::GetMissing: need to check need_up_thru in MLogRec handler

Backport: bobtail
Fixes: #4534
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 4fe4deafbe1758a6b3570048aca57485bd562440)
mkcephfs: make remote temp directory name unique

The predictable file name is a security problem.

CVE-2013-1882

Reported-by: Michael Scherer <misc@zarb.org>
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
(cherry picked from commit f463ef78d77b11b5ad78b31e9a3a88d0a6e62bca)
init-ceph: push temp conf file to a unique location on remote host

The predictable file name is a security problem.

CVE-2013-1882

Reported-by: Michael Scherer <misc@zarb.org>
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
(cherry picked from commit 051734522fea92878dd8139f28ec4e6b01371ede)
init-ceph: clean up temp ceph.conf filename on exit

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
(cherry picked from commit 6a7ad2eac1db6abca3d7edb23ca9b80751400a23)
common/MemoryModel: remove logging to /tmp/memlog

This was a hack for dev purposes ages ago; remove it.  The predictable
filename is a security issue.

CVE-2013-1882

Reported-by: Michael Scherer <misc@zarb.org>
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
(cherry picked from commit c524e2e01da41ab5b6362c117939ea1efbd98095)
os/FileJournal: fix aio self-throttling deadlock

This block of code tries to limit the number of aios in flight by waiting
for the amount of data to be written to grow relative to a function of the
number of aios.  Strictly speaking, the condition we are waiting for is a
function of both aio_num and the write queue, but we are only woken by
changes in aio_num, and were (in rare cases) waiting when aio_num == 0 and
there was no possibility of being woken.

Fix this by verifying that aio_num > 0, and restructuring the loop to
recheck that condition on each wakeup.

Fixes: #4079
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit e5940da9a534821d0d8f872c13f9ac26fb05a0f5)
systest: fix race with pool deletion

The second test have pool deletion and object listing wait on the same
semaphore to connect and start. This led to errors sometimes when the
pool was deleted before it could be opened by the listing process. Add
another semaphore so the pool deletion happens only after the listing
has begun.

Fixes: #4147
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
(cherry picked from commit b0271e390564119e998e18189282252d54f75eb6)
systest: restrict list error acceptance

Only ignore errors after the midway point if the midway_sem_post is
defined.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
(cherry picked from commit 5b24a68b6e7d57bac688021b822fb2f73494c3e9)
rgw: bucket index ops on system buckets shouldn't do anything

Fixes: #4508
Backport: bobtail
On certain bucket index operations we didn't check whether
the bucket was a system bucket, which caused the operations
to fail. This triggered an error message on bucket removal
operations.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 70e0ee8ba955322832f0c366537ddf7a0288761e)
v0.56.4
rbd: remove fiemap use from import

On some kernels and filesystems fiemap can be racy and provide
incorrect data even after an fsync. Later we can use SEEK_HOLE and
SEEK_DATA, but for now just detect zero runs like we do with stdin.

Basically this adapts import from stdin to work in the case of a file
or block device, and gets rid of other cruft in the import that used
fiemap.

Fixes: #4388
Backport: bobtail
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
(cherry picked from commit 3091283895e8ffa3e4bda13399318a6e720d498f)