fix pg floor tracking
osd/PG: drop unused queue_{notify,info,log}()

Signed-off-by: Sage Weil <sage@inktank.com>
osd/PG: uninline PG::init_primary_up_acting()

Signed-off-by: Sage Weil <sage@inktank.com>
osd/PG: refactor primary_init_up_acting

Get rid of the early return.

Signed-off-by: Sage Weil <sage@redhat.com>
ceph_test_osd_stale_read: add StaleRead test

This reproduces the "read hole".

Signed-off-by: Sage Weil <sage@inktank.com>
osd: do not rebind heartbeat messengers on wrongly marked down

If we are wrongly marked down and need to rejoin, it is important that we
bind to a new cluster address so that our peer to peer sessions are fully
distinct.

However, it we can still use the same heartbeat ports.  If we get a ping
from someone with an old map, it is better to still be there and tell them
that there is a newer map than for them to get crickets.

Signed-off-by: Sage Weil <sage@inktank.com>
doc: stale reads wip

Signed-off-by: Sage Weil <sage@inktank.com>
osd: include up_from, consumed_epoch in MOSDPing

Signed-off-by: Sage Weil <sage@inktank.com>
osd: some minor refactoring/cleanup in handle_osd_ping

Use local con variable.
Use a local utime_t now variable.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: track some heartbeat stats for peer OSDs

We need to keep track of peer epochs and when they last sent or replied
to pings.  Keep this info in a separate ref-counted struct so that we can
point to it directly from other parts of the OSD that will need it.  For
now, this includes just the HeatbeatSession so that we can efficiently
update when we receive ping messages.

Include the ability for a caller to put a PG on a notify list in the
event that a ping or ping reply does arrive.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: note rx_stamp in MNotifyRec

Note the timestamp we received the notify in the MNotifyRec struct.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/osd_types: add first_interval_readable to pg_history_t

Keep track of the oldest interval that may still be readable.

Signed-off-by: Sage Weil <sage@redhat.com>
osd/PG: do not trim past_intervals that may still be readable

We need to maintain information about past intervals long enough to
decide whether to wait for them to become unreadable.  Do not trim them
if they match or follow pg_history_t::first_interval_readable.

Signed-off-by: Sage Weil <sage@redhat.com>
osd/PG: take HeartbeatStamps refs for current interval

For each (peering) interval, take HeartbeatStamps refs for our peer OSDs
so we can efficiently check liveness.

If we are the primary, we care about all replicas.  If we are a replica,
we care only about the primary.

Signed-off-by: Sage Weil <sage@redhat.com>
readable_until wip
wip recalc readable
osd: include readable_delta in pg_notify_t

Whenever we send a pg_notify_t to a peer, include an upper bound on the
amount of time that the PG may remain readable after the message is sent.
For a replica, this value is based on the most recent heartbeat ack that
we have sent to the primary.  For the primary, it is based on all of the
acks we have received from replicas.  In either case, the value we send is
a delta to the current time so that we can avoid problems with clock
skew.  Obviously, the ultimate upper bound the receiver can infer on
receipt is bloated by network transit time.

Signed-off-by: Sage Weil <sage@inktank.com>
messages/MOSDPGNotify: encode new pg_notify_t::readable_delta field

We don't use the pg_notify_t::{encode,decode} methods, so we have to handle
the new fields explicitly.

Signed-off-by: Sage Weil <sage@redhat.com>
osd/PG: track peer readable_until values during peering

Translate the over-the-wire delta value to a timestamp in the process.
Clear on peering event.  Ignore after we are active.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: add PG_STATE_UNREADABLE

This state bit indicates the PG is not currently readable (do to laggy
peers in the current interval, or potentially stale members of previous
intervals).

Signed-off-by: Sage Weil <sage@redhat.com>