monc: let users specify a callback when they reopen their monitor session

Then the callback is triggered when a new session is established, and the
daemon can do whatever it likes. There are no guarantees about how long it
might take to trigger, though. In particular we call the provided callback
while not holding our own lock in order to avoid deadlock. This could lead
to some funny ordering from the user's perspective if they call
reopen_session() again before getting the callback, but there's no way around
that, so they just have to use it appropriately.

Signed-off-by: Greg Farnum <greg@inktank.com>
monc: set "hunting" to true when we reopen the mon session

If we don't have a connecton to a monitor, we want to retry to another
monitor regardless of whether it's the first time or not.

Signed-off-by: Greg Farnum <greg@inktank.com>
monc: backoff the timeout period when reconnecting

If the monitors are systematically slowing down, we don't want to spam
them with reconnect attempts every three seconds. Instead, every time
we issue a reconnect, multiply our timeout period by a configurable; when
we complete the connection, reduce that multipler by 50%. This should let
us respond to monitor load.
Of course, we don't want to do that for initial startup in the case of a
couple down monitors, so don't apply the backoff until we've successfully
connected to a monitor at least once.

Signed-off-by: Greg Farnum <greg@inktank.com>
OSD: disable the PGStatsAck timeout when we are reconnecting to a monitor

Previously, the timeout counter started as soon as we issued the reopen,
but if the reconnect process itself took a while, we might time out and
issue another reopen just as we get to the point where it's possible to
get work done. Since the mon client has its own reconnect timeouts (that is,
the OSD doesn't need to trigger those), we instead disable our timeouts
while the reconnect is happening, and then turn them back on again starting
from when we get the reconnect callback.

Signed-off-by: Greg Farnum <greg@inktank.com>
Merge pull request #1227 from ceph/wip-7394

Improve the OSD and generic monitor timeout mechanisms

Reviewed-by: Sage Weil <sage@inktank.com>