PG: check full ratio again post-reservation

Otherwise, we might queue 30 pgs for backfill at 0.80 fullness
and then never check again filling the osd after pg 11.

Fixes: #9574
Backport: dumpling, firefly, giant
Signed-off-by: Samuel Just <sam.just@inktank.com>
ReplicatedPG: clean out completed trimmed objects as we go

Also, explicitely maintain a max number of concurrently trimming
objects.

Fixes: 9113
Backport: dumpling, firefly, giant
Signed-off-by: Samuel Just <sam.just@inktank.com>
ReplicatedPG: do not queue the snap trimmer constantly

Previously, we continuously requeued the snap trimmer while in
TrimmingObjects.  This is not a good idea now that we try to
limit the number of snap trimming repops in flight and requeue
the snap trimmer directly as those repops complete.

Fixes: #9113
Backport: giant, dumpling, firefly
Signed-off-by: Samuel Just <sam.just@inktank.com>
ReplicatedPG: dump snap_trimq on pg query

Signed-off-by: Samuel Just <sam.just@inktank.com>
rpm: 95-ceph-osd-alt.rules is not needed for centos7 / rhel7

The || instead of && had it always installed. That was fixed in EPEL
already.

http://tracker.ceph.com/issues/9747 Fixes: #9747

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
(cherry picked from commit 5ff4a850a0d809b3f25988c6cceb82c35095ef84)
mon/Paxos: put source mon id in a temp variable

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit bb046ed01ecf58b8c87eeeee2e00a476e6fba467)
mon/Paxos: verify all new peons are still contiguous at end of round

During the collect phase we verify that each peon has overlapping or
contiguous versions as us (and can therefore be caught up with some
series of transactions).  However, we *also* assimilate any new states we
get from those peers, and that may move our own first_committed forward
in time.  This means that an early responder might have originally been
contiguous, but a later one moved us forward, and when the round finished
they were not contiguous any more.  This leads to a crash on the peon
when they get our first begin message.

For example:

 - we have 10..20
 - first peon has 5..15
   - ok!
 - second peon has 18..30
   - we apply this state
 - we are now 18..30
 - we finish the round
   - send commit to first peon (empty.. we aren't contiguous)
   - send no commit to second peon (we match)
 - we send a begin for state 31
   - first peon crashes (it's lc is still 15)

Prevent this by checking at the end of the round if we are still
contiguous.  If not, bootstrap.  This is similar to the check we do above,
but reverse to make sure *we* aren't too far ahead of *them*.

Fixes: #9053
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 3e5ce5f0dcec9bbe9ed4a6b41758ab7802614810)
mon/Paxos: share state and verify contiguity early in collect phase

We verify peons are contiguous and share new paxos states to catch peons
up at the end of the round.  Do this each time we (potentially) get new
states via a collect message.  This will allow peons to be pulled forward
and remain contiguous when they otherwise would not have been able to.
For example, if

  mon.0 (leader)  20..30
  mon.1 (peon)    15..25
  mon.2 (peon)    28..40

If we got mon.1 first and then mon.2 second, we would store the new txns
and then boot mon.1 out at the end because 15..25 is not contiguous with
28..40.  However, with this change, we share 26..30 to mon.1 when we get
the collect, and then 31..40 when we get mon.2's collect, pulling them
both into the final quorum.

It also breaks the 'catch-up' work into smaller pieces, which ought to
smooth out latency a bit.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c54f1e4d66b22bad715ac17e9baa72ab93e48c46)
mon/Paxos: fix off-by-one in last_ vs first_committed check

peon last_committed + 1 == leader first_committed is okay.  Note that the
other check (where I clean up whitespace) gets this correct.

Fixes: #9301 (partly)
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit d81cd7f86695185dce31df76c33c9a02123f0e4a)
mon: re-bootstrap if we get probed by a mon that is way ahead

During bootstrap we verify that our paxos commits overlap with the other
mons we will form a quorum with.  If they do not, we do a sync.

However, it is possible we pass those checks, then fail to join a quorum
before the quorum moves ahead in time such that we no longer overlap.
Currently nothing kicks up back into a probing state to discover we need
to sync... we will just keep trying to call or join an election instead.

Fix this by jumping back to bootstrap if we get a probe that is ahead of
us.  Only do this from non probe or sync states as these will be common;
it is only the active and electing states that matter (and probably just
electing!).

Fixes: #9301
Backport: giant, firefly
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c421b55e8e15ef04ca8aeb47f7d090375eaa8573)
include/util.h: prevent multiple inclusion of header

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
(cherry picked from commit 76eff9503493312cb97e4a2f9236f4dbcbf931df)
common: util: add get_fs_stats() function

simplifies the task of obtaining available/used disk space, as well as
used available percentage.

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>
(cherry picked from commit 3d74230d1c0fbfa15487e2a90ac60b883476e840)
mon: DataHealthService: use get_fs_stats() instead

and relieve the DataStats struct from clutter by using
ceph_data_stats_t instead of multiple fields.

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>
(cherry picked from commit 9996d446988768658db751a7843b13cf3d194213)

Conflicts:
	src/mon/DataHealthService.cc
ceph_mon: check available storage space for mon data dir on start

error out if available storage space is below 'mon data avail crit'

Fixes: #9502

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>
(cherry picked from commit 2da1a2914ac7df18ce842b0aac728fffb5bed2b6)

Conflicts:
	src/ceph_mon.cc
ceph-mon: check fs stats just before preforking

Otherwise statfs may fail if mkfs hasn't been run yet or if the monitor
data directory does not exist.  There are checks to account for the mon
data dir not existing and we should wait for them to clear before we go
ahead and check the fs stats.

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>
(cherry picked from commit 7f71c11666b25e91dd612c58b4eda9ac0d4752f8)

Conflicts:
	src/ceph_mon.cc
ceph_test_rados_api_tier: add EvictSnap2 test case

Verify an evict doesn't create a snapdir object.  Reproduces #8629

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 398c74eacb1ce4e573aef0d24718a5925d90272b)
osd/ReplicatedPG: do not clone or preserve snapdir on cache_evict

If we cache_evict a head in a cache pool, we need to prevent
make_writeable() from cloning the head and finish_ctx() from
preserving the snapdir object.

Fixes: #8629
Backport: firefly
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit ce8eefca13008a9cce3aedd67b11537145e1fd77)
os/FileJournal: Update the journal header when closing journal

When closing journal, it should check must_write_header and update
journal header if must_write_header alreay set.
It can reduce the nosense journal-replay after restarting osd.

Signed-off-by: Ma Jianpeng <jianpeng.ma@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 5bf472aefb7360a1fe17601b42e551df120badfb)
os/FileJournal: signal aio_cond even if seq is 0

This can happen if we write a journal but no events.

Reported-by: Somnath Roy <somnath.roy@sandisk.com>
Reported-by: Ma, Jianpeng <jianpeng.ma@intel.com>
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 57778e2c577c1e1bbf9525232720a2994fa36abc)
os/FileJournal: Tune the judge logic for read_header.

When reading journal-header, it should firstly check the result of
pread and then do decoce operation.

Signed-off-by: Ma Jianpeng <jianpeng.ma@intel.com>
(cherry picked from commit c8e2b89cf6bc36a0ff29887b9e76cbbeceef9f8f)