ReplicatedPG::submit_push_complete don't remove the head object

The object would have had to have been removed already.  With
fd caching, this extra remove might check the wrong replay_guard
since the fd caching mechanism assumes that between any operation
on an hobject_t oid and a remove operation, all operations on that
hobject_t must refer to the same inode.

Signed-off-by: Samuel Just <sam.just@inktank.com>
WBThrottle: add perfcounters

Signed-off-by: Samuel Just <sam.just@inktank.com>
doc/dev/osd_internals: add wbthrottle.rst

Signed-off-by: Samuel Just <sam.just@inktank.com>
osd: do not assume head obc object exists when getting snapdir

For a list-snaps operation on the snapdir, do not assume that the obc for the
head means the object exists.  This fixes a race between a head deletion and
a list-snaps that wrongly returns ENOENT, triggered by the DiffItersateStress
test when thrashing OSDs.

Fixes: #5183
Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
mds: stay in SCAN state in file_eval

If we are in the SCAN state, stay there until the recovery finishes.  Do
not jump to another state from file_eval().

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 0071b8e75bd3f5a09cc46e2225a018f6d1ef0680)
osd: simplify is_healthy() check during boot

This has a slight behavior change in that we ask the mon for the latest
osdmap if our internal heartbeat is failing.  That isn't useful yet, but
will be shortly.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: tell peers that ping us if they are dead

Signed-off-by: Sage Weil <sage@inktank.com>
osd: avoid duplicate mon requests for a new osdmap

sub_want() returns true if this is a new sub; only renew then.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: move health checks into a single helper

For now we still only look at the internal heartbeats.

Signed-off-by: Sage Weil <sage@inktank.com>
WBThrottle: rename replica nocache

We may want to influence the caching behavior for other
reasons.

Signed-off-by: Samuel Just <sam.just@inktank.com>
WBThrottle: add some comments and some asserts

Signed-off-by: Samuel Just <sam.just@inktank.com>
Merge branch 'wip_osd_throttle'

Fixes: #4782
Reviewed-by: Sage Weil
osd: initialize new_state field when we use it

If we use operator[] on a new int field its value is undefined; avoid
reading it or using |= et al until we initialize it.

Fixes: #4967
Backport: cuttlefish, bobtail
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: David Zafman <david.zafman@inktank.com>
osd: augment osd heartbeat peers with neighbors and randoms, to up some min

- always include our neighbors to ensure we have a fully-connected
  graph
- include some random neighbors to get at least some min number of peers.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: factor out _remove_heartbeat_peer

Signed-off-by: Sage Weil <sage@inktank.com>
osd: only add pg peers if active

We will soon be in this method for the waiting-for-healthy state.  As
a consequence, we need to remove any down peers.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: remove down hb peers

If a (say, random) peer goes down, filter it out.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: distinguish between definitely healthy and definitely not unhealthy

is_unhealthy() will assume they are healthy for some period after we
send our first ping attempt.  is_healthy() is now a strict check that we
know they are healthy.

Switch the failure report check to use is_unhealthy(); use is_healthy()
everywhere else, including the waiting-for-healthy pre-boot checks.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: wait for healthy pings from peers in waiting-for-healthy state

If we are (wrongly) marked down, we need to go into the waiting-for-healthy
state and verify that our network interfaces are working before trying to
rejoin the cluster.

 - make _is_healthy() check require positive proof of pings working
 - do heartbeat checks and updates in this state
 - reset the random peers every heartbeat_interval, in case we keep picking
   bad ones

Signed-off-by: Sage Weil <sage@inktank.com>
Merge branch 'next'