osd: fix pg epoch floor tracking

If you call erase() on a multiset it will delete all instances of a value;
we only want to delete one of them.  Fix this by passing an iterator.

Backport: firefly
Signed-off-by: Sage Weil <sage@redhat.com>
Merge pull request #2228 from ceph/wip-pg-epoch

osd: fix pg epoch floor tracking

Reviewed-by: Samuel Just <sam.just@inktank.com>
Merge remote-tracking branch 'gh/next'
msgr: add ms_blackhole_$type options

Add the ability to blackhole messages sent or received from the given
entity type.  This will let us inject a network stall that will appear
similar to a network partition of a single node from its peers.

Signed-off-by: Sage Weil <sage@inktank.com>
librados: add osd_command() to c++ api

This is already present in the C API.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/PG: drop unused queue_{notify,info,log}()

Signed-off-by: Sage Weil <sage@inktank.com>
osd/PG: uninline PG::init_primary_up_acting()

Signed-off-by: Sage Weil <sage@inktank.com>
osd/PG: refactor primary_init_up_acting

Get rid of the early return.

Signed-off-by: Sage Weil <sage@redhat.com>
ceph_test_osd_stale_read: add StaleRead test

This reproduces the "read hole".

Signed-off-by: Sage Weil <sage@inktank.com>
osd: do not rebind heartbeat messengers on wrongly marked down

If we are wrongly marked down and need to rejoin, it is important that we
bind to a new cluster address so that our peer to peer sessions are fully
distinct.

However, it we can still use the same heartbeat ports.  If we get a ping
from someone with an old map, it is better to still be there and tell them
that there is a newer map than for them to get crickets.

Signed-off-by: Sage Weil <sage@inktank.com>
doc: stale reads wip

Signed-off-by: Sage Weil <sage@inktank.com>
osd: include up_from, consumed_epoch in MOSDPing

Signed-off-by: Sage Weil <sage@inktank.com>
osd: some minor refactoring/cleanup in handle_osd_ping

Use local con variable.
Use a local utime_t now variable.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: track some heartbeat stats for peer OSDs

We need to keep track of peer epochs and when they last sent or replied
to pings.  Keep this info in a separate ref-counted struct so that we can
point to it directly from other parts of the OSD that will need it.  For
now, this includes just the HeatbeatSession so that we can efficiently
update when we receive ping messages.

Include the ability for a caller to put a PG on a notify list in the
event that a ping or ping reply does arrive.

For now we do nothing with the affected PGs; that will come later.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: note rx_stamp in MNotifyRec

Note the timestamp we received the notify in the MNotifyRec struct.

Signed-off-by: Sage Weil <sage@inktank.com>
osd/osd_types: add first_interval_readable to pg_history_t

Keep track of the oldest interval that may still be readable.

Signed-off-by: Sage Weil <sage@redhat.com>
osd/PG: do not trim past_intervals that may still be readable

We need to maintain information about past intervals long enough to
decide whether to wait for them to become unreadable.  Do not trim them
if they match or follow pg_history_t::first_interval_readable.

Signed-off-by: Sage Weil <sage@redhat.com>
osd/PG: take HeartbeatStamps refs for current interval

For each (peering) interval, take HeartbeatStamps refs for our peer OSDs
so we can efficiently check liveness.

If we are the primary, we care about all replicas.  If we are a replica,
we care only about the primary.

Signed-off-by: Sage Weil <sage@redhat.com>
osd/PG: add readable_until; prune it

We will use this to track what we know about the readable_until bound
for the current and prior intervals.

We can prune entries from this map if the bound has passed and either
we are active (and only care about the current interval) or if the interval
is before the first_interval_readable bound in pg_history_t.

Signed-off-by: Sage Weil <sage@redhat.com>
osd/PG: PGPool::get_readable_interval

Simple helper to get the readable interval for a pool.  For now this is
a config option, but in the future it may be a pool property.

Signed-off-by: Sage Weil <sage@redhat.com>