osd: include failed_since in MOSDFailure reports

The monitor will need this to dynamically adjust the heartbeat grace.

Closes: #3044
Signed-off-by: Sage Weil <sage@inktank.com>
osd: include boot_epoch in MOSDBoot

This will let the monitor infer whether we were wrongly marked down or
the daemon restarted.

Signed-off-by: Sage Weil <sage@inktank.com>
osdmap: include osd_xinfo_t to track laggy probabilities, timestamps

Track information about laggy probabilities for each OSD.  That is, the
probability that if it is marked down it is because it is laggy, and
the expected interval over which it will take to recovery if it is laggy.

We store this in the OSDMap because it is not convenient to keep it
elsewhere in the monitor.  Yet.  When the new mon infrastructure is in
place, there is a bunch of stuff that can be moved out of the OSDMap
'extended' section into other mon data structures.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: adjust or decay laggy probabilities on osd boot

On each osd boot, determine whether the osd was laggy (wrongly marked down)
or newly booted.  Either update the laggy probability and interval or
decay the values, as appropriate.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: refactor osd failure report tracking

- use structs to track allegedly failed nodes, and reports against them.
- use methods to handle report, and failure threshold logic.

This will make it simpler to extend the logic when we add dynamic
grace periods.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: avoid failure times in the future

Signed-off-by: Sage Weil <sage@inktank.com>
mon: send 'null' reply to requests we won't reply to

This is a no-op if the client was talking to us, but in the forwarded
request case will clean up the request state (and request message) on the
forwarding monitor.  Otherwise, MOSDFailure messages (and probably others)
can accumulate on the non-leader mon indefinitely.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: no_reply() to failure messages we don't reply to

This makes use clean up request state when requests have been forwarded.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: locally apply osd heartbeat grace to failure checks

Aggregate the failure reports into a single mon 'failed_since' value (the
max, currently), and wait until we have exceeded the grace period to
consider the osd failed.

WARNING: This slightly changes the semantics.  Previously, the grace could
be adjusted in the [osd] section.  Now, the [osd] option controls when the
failure messages are sent, and the [mon] option controls when it is marked
down, and sane users should set it once in [global].

Signed-off-by: Sage Weil <sage@inktank.com>
mon: add simple requeue framework

Instead of directly calling _ms_dispatch() from context completions, queue
messages to reprocess and handle them before and after _ms_dispatch() in
ms_dispatch().  This makes the context when these messages are processed
more predictable and consistent.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: reply to all reporters when an osd is failed

Track the latest report message for each reporter.  When the osd is
eventually marked failed, send map updates to them all.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: clean up osd failure logging

Debug log when we get a report, info log when we actual fail the osd.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: check failures in tick

Currently we only trigger a failure on receipt of a failure report.  Move
the checks into a helper and check during tick() too, so that we will
trigger failures even when the thresholds are not met at failure report
time.  This is rarely true now, but will be true once we locally scale the
grace period.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: scale heartbeat grace based on laggy probability, interval

If, based on historical behavior, an observed osd failure is likely to be
due to unresponsiveness and not the daemon stopping, scale the heartbeat
grace period accordingly:

 grace' = grace + laggy_probabiliy * laggy_interval

This will avoid fruitlessly marking OSDs down and generating additional
map update overhead when the cluster is overloaded and potentially
struggling to keep up with map updates.   See #3045.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: factor reporter lagginess into grace adjustment

Use reporters as a proxy for laggy subclusters within the overall cluster.
See #3046.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: decay laggy calculations over time

Add a configurable halflife for the laggy probability and duration and
apply it at the time those values are used to adjust the heartbeat grace
period.  Both are multiplied together, so it doesn't matter which you
think is being decayed (the probability or the interval).

Default to an hour.

Signed-off-by: Sage Weil <sage@inktank.com>