WorkQueue: fix bracing on reset_tp_timeout

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
OSD: suspend tp timeout while taking pg lock in OpWQ

If N op_tp threads are configured, and recovery_max_active
is set to a sufficiently large number, all N op_tp threads
might grab a MOSDPGPush op off of the queue for the same PG.
The last thread to get the lock will have waited
N*time_to_handle_push before completing its item and pinging
the heartbeat timeout.  If that time exceeds the timeout
and there are enough ops waiting, each thread subsequently
will end up exceeding the timeout before completeing an
item preventing the OSD from heartbeating indefinitely.

We prevent this by suspending the timeout while we try to
get the PG lock.  Even if we do block for an excessive
period of time attempting to get the lock, hopefully,
the thread holding the lock will cause the threadpool
to time out.

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
check_new_interval must compare old acting with old osdmap

When trying to establish if the old acting set is either empty or
smaller than the min_size of the osdmap,
pg_interval_t::check_new_interval compares with the min_size of the
new osdmap. Since the goal is to try to determine if the previous
interval may have been writeable, it should not enter the if when
there were not enough osds in the acting set ( i.e. < min_size ). But
it may enter it anyway if min_size was decremented in the new osdmap.

A complete set of unit tests were added to cover the logic of
check_new_interval. The parameters are prepared to describe a
situation where the function returns false (i.e. no new
interval). Each case is described in a separate bloc that introduces
the minimal changes to demonstrate the intended test case.

Because a number of cases have the same output while implementing a
different logic, the debug output is parsed to differentiate between them.

A test case demonstrating the problem ( check_new_interval must
compare old acting with old osdmap ) is added, with a link to the bug
number for future reference. The problem is fixed. The text of two
debug messages are slightly changed to make the maintenance of the
test that match them easier.

http://tracker.ceph.com/issues/5780 refs #5780

Signed-off-by: Loic Dachary <loic@dachary.org>
Reviewed-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>