msgr: atomically queue first message with connect_rank

Atomically queue the first message on the new pipe, without dropping
and retaking pipe_lock.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 7bf0b0854d1f2706a3a2302bcbf92dd5c8c888ef)
msgr: don't queue message on closed pipe

If we have a con that refs a pipe but it is closed, don't use it.  If
the ref is still there, it is only because we are racing with fault()
and it is about to (or just was) be detached.  Either way,

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 6339c5d43974f4b495f15d199e01a141e74235f5)
msgr: fix race on Pipe removal from hash

When a pipe is faulting and shutting down, we have to drop pipe_lock to
take msgr lock and then remove the entry.  The Pipe in this case will
have STATE_CLOSED.  Handle this case in all places we do a lookup on
the rank_pipe hash so that we effectively ignore entries that are
CLOSED.

This fixes a race introduced by the previous commit where we won't use
the CLOSED pipe and try to register a new one, but the old one is still
registered.

See bug #3675.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit e99b4a307b4427945a4eb5ec50e65d6239af4337)
msgr: inject delays at inconvenient times

Exercise some rare races by injecting delays before taking locks
via the 'ms inject internal delays' option.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit a5d692a7b9b4bec2c27993ca37aa3fec4065292b)
msg/Pipe: use state_closed atomic_t for _lookup_pipe

We shouldn't look at Pipe::state in SimpleMessenger::_lookup_pipe() without
holding pipe_lock.  Instead, use an atomic that we set to non-zero only
when transitioning to the terminal STATE_CLOSED state.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 82f8bcddb5fa09913eb477ee26c71d6b4bb8d97c)
ceph: adjust crush tunables via 'ceph osd crush tunables <profile>'

Make it easy to adjust crush tunables.  Create profiles:

 legacy: the legacy values
 argonaut: the argonaut defaults, and what is supported.. legacy! (*(
 bobtail: best that bobtail supports
 optimal: the current optimal values
 default: the current default values

* In actuality, argonaut supports some of the tunables, but it doesn't
  say so via the feature bits.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
(cherry picked from commit 19ee23111585f15a39ee2907fa79e2db2bf523f0)