osd: do not assume head obc object exists when getting snapdir

For a list-snaps operation on the snapdir, do not assume that the obc for the
head means the object exists.  This fixes a race between a head deletion and
a list-snaps that wrongly returns ENOENT, triggered by the DiffItersateStress
test when thrashing OSDs.

Fixes: #5183
Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
mds: stay in SCAN state in file_eval

If we are in the SCAN state, stay there until the recovery finishes.  Do
not jump to another state from file_eval().

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 0071b8e75bd3f5a09cc46e2225a018f6d1ef0680)
osd: simplify is_healthy() check during boot

This has a slight behavior change in that we ask the mon for the latest
osdmap if our internal heartbeat is failing.  That isn't useful yet, but
will be shortly.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: tell peers that ping us if they are dead

Signed-off-by: Sage Weil <sage@inktank.com>
osd: avoid duplicate mon requests for a new osdmap

sub_want() returns true if this is a new sub; only renew then.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: move health checks into a single helper

For now we still only look at the internal heartbeats.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: initialize new_state field when we use it

If we use operator[] on a new int field its value is undefined; avoid
reading it or using |= et al until we initialize it.

Fixes: #4967
Backport: cuttlefish, bobtail
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: David Zafman <david.zafman@inktank.com>
osd: augment osd heartbeat peers with neighbors and randoms, to up some min

- always include our neighbors to ensure we have a fully-connected
  graph
- include some random neighbors to get at least some min number of peers.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: factor out _remove_heartbeat_peer

Signed-off-by: Sage Weil <sage@inktank.com>
osd: only add pg peers if active

We will soon be in this method for the waiting-for-healthy state.  As
a consequence, we need to remove any down peers.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: remove down hb peers

If a (say, random) peer goes down, filter it out.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: distinguish between definitely healthy and definitely not unhealthy

is_unhealthy() will assume they are healthy for some period after we
send our first ping attempt.  is_healthy() is now a strict check that we
know they are healthy.

Switch the failure report check to use is_unhealthy(); use is_healthy()
everywhere else, including the waiting-for-healthy pre-boot checks.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: wait for healthy pings from peers in waiting-for-healthy state

If we are (wrongly) marked down, we need to go into the waiting-for-healthy
state and verify that our network interfaces are working before trying to
rejoin the cluster.

 - make _is_healthy() check require positive proof of pings working
 - do heartbeat checks and updates in this state
 - reset the random peers every heartbeat_interval, in case we keep picking
   bad ones

Signed-off-by: Sage Weil <sage@inktank.com>
Merge branch 'next'
Merge pull request #331 from ceph/wip-osd-interfacecheck

Reviewed-by: Samuel Just <sam.just@inktank.com>
Merge git://github.com/ceph/ceph