osd/osd_types: add first_interval_readable to pg_history_t

Keep track of the oldest interval that may still be readable.

Signed-off-by: Sage Weil <sage@redhat.com>
osd/PG: do not trim past_intervals that may still be readable

We need to maintain information about past intervals long enough to
decide whether to wait for them to become unreadable.  Do not trim them
if they match or follow pg_history_t::first_interval_readable.

Signed-off-by: Sage Weil <sage@redhat.com>
osd: calculate a per-pg readable_until map

For a given PG, calculate a readable_until map that, for each interval
that may still be relevant, includes a current lower bound on the amount
of time we will still be allowed to service reads.  This value is based on
the heartbeats we've responded to (as a non-primary) or gotten replies to
(as a primary).  For a stray, we know nothing.

Signed-off-by: Sage Weil <sage@redhat.com>
osd: include readable_delta in pg_notify_t

Whenever we send a pg_notify_t to a peer, include an upper bound on the
amount of time that the PG may remain readable after the message is sent.
For a replica, this value is based on the most recent heartbeat ack that
we have sent to the primary.  For the primary, it is based on all of the
acks we have received from replicas.  In either case, the value we send is
a delta to the current time so that we can avoid problems with clock
skew.  Obviously, the ultimate upper bound the receiver can infer on
receipt is bloated by network transit time.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/PG: track peer readable_until values during peering

Translate the over-the-wire delta value to a timestamp in the process.
Clear on peering event.  Ignore after we are active.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: recalc readable_until on each new peering interval

This ensure that the last calculated value for prior intervals is accurate,
as after this point we will only update the current interval's value.

Signed-off-by: Sage Weil <sage@inktank.com>
fix recalc_readble_until
asdf
wip
fix unreadable state bit
fix timer
debug
unreadable -> readable after peering is working
nope
wip