PG: create macro for simple events

This should make defining no-information events a bit simpler.

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD: add init and shutdown for OSDService

Signed-off-by: Samuel Just <sam.just@inktank.com>
osd/: add backfill reservations

Previously, a new osd would be bombarded by backfills from many osds
simultaneously, resulting in excessively high load.  Instead, we
want to limit the number of backfills coming into and going out
from a single osd.

To that end, each OSDService now has two AsyncReserver instances: one
for backfills going from the osd (local_reserver) and one for backfills
going to the osd (remote_reserver).  For a primary to initiate a
backfill, it must first obtain a reservation from its own
local_reserver.  Then, it must obtain a reservation from the backfill
target's remote_reserver via a MBackfillReserve message. This process is
managed by substates of Active and ReplicaActive (see the changes in
PG.h).  The reservations are dropped either on the Backfilled event,
which is sent on the primary before calling recovery_complete and on the
replica on receipt of the BackfillComplete progress message), or upon
leaving Active or ReplicaActive.

It's important that we always grab the local reservation before the
remote reservation in order to prevent a circular dependency.

Signed-off-by: Samuel Just <sam.just@inktank.com>
osd/: add PG_STATE_BACKFILLING

PG_STATE_BACKFILLING is set when the pg enters the Backfilling state.
That is, +backfilling indicates that the pg has obtained its
reservations and is now actively backfilling.

Signed-off-by: Samuel Just <sam.just@inktank.com>
doc/: added documentation for backfill_reservation

Signed-off-by: Samuel Just <sam.just@inktank.com>
PG: add CEPH_FEATURE for backfill reservation

Also adds backwards compatibility by just post_event-ing
the RemoteBackfillReserved() rather than sending the
message to an older replica.

Signed-off-by: Samuel Just <sam.just@inktank.com>
PG,osd_types,PGMonitor: make backfill state names more descriptive

PG_STATE_BACKFILL->PG_STATE_BACKFILL_WAIT
and
PG_STATE_BACKFILLING->PG_STATE_BACKFILL

backfill -> wait_backfill
backfill+backfilling -> backfill

Signed-off-by: Samuel Just <sam.just@inktank.com>
timer: add unsafe callbacks option

Using unsafe callbacks drops the lock between invocations of event
callbacks. It is useful under some circumstances, but the user must take
caution. See the comment in Timer.h for full details.

Signed-off-by: Mike Ryan <mike.ryan@inktank.com>
PG, OSD: reject backfills when an OSD is nearly full

Reject backfills when an OSD reaches a configurable full ratio. Retry
backfilling periodically in the hopes that the OSD has become less full.

This changeset introduces two configuration options for dealing with
this: osd_refuse_backfill_full_ratio and osd_backfill_retry_interval.

We also introduce two new state transitions in the PG's Active state.

Signed-off-by: Mike Ryan <mike.ryan@inktank.com>
PG: last_complete might be outside of the log

When we encounter a divergent log entry, we put the
object into the missing set at the prior_version
for the divergent event.  Unfortunately, the event
at prior_version might have been trimmed leaving
the missing set with an item with a need prior to
log_tail.  Thus, last_complete also ends up being
prior to log_tail.

Caused #3208.

There is another bug related to this one: #3213.

Signed-off-by: Samuel Just <sam.just@inktank.com>
Merge remote-tracking branch 'gh/wip_backfill_full2'

Conflicts:
	src/include/ceph_features.h
PG: explicitely delay ops on backfill_pos

Previously, we considered backfill_pos degraded in order to delay
ops since a write to backfill_pos could generate a snap before
backfill_pos, and we assume that (0, backfill_pos) is fully
backfilled.  This is a problem since it's possible that
backfill_pos is a valid object, but not one that currently exists.
For example, it might have been deleted since last_backfill was
last changed.  Instead, we will explicitly delay ops on
backfill_pos in waiting_for_backfill_pos.

This error resulted in #2691 since wait_for_degraded_object also
attempts to recover the object. At this point, the primary would
attempt to recover the object, find that it isn't there, and put
it in the missing set with need=0,0.  Eventually, recover_primary
attempts to recover that object, finds that it has been deleted
in the log, and asserts.

Signed-off-by: Samuel Just <sam.just@inktank.com>