libceph: block I/O when PAUSE or FULL osd map flags are set

The PAUSEWR and PAUSERD flags are meant to stop the cluster from
processing writes and reads, respectively. The FULL flag is set when
the cluster determines that it is out of space, and will no longer
process writes.  PAUSEWR and PAUSERD are purely client-side settings
already implemented in userspace clients. The osd does nothing special
with these flags.

When the FULL flag is set, however, the osd responds to all writes
with -ENOSPC. For cephfs, this makes sense, but for rbd the block
layer translates this into EIO.  If a cluster goes from full to
non-full quickly, a filesystem on top of rbd will not behave well,
since some writes succeed while others get EIO.

Fix this by blocking any writes when the FULL flag is set in the osd
client. This is the same strategy used by userspace, so apply it by
default.  A follow-on patch makes this configurable.

__map_request() is called to re-target osd requests in case the
available osds changed.  Add a paused field to a ceph_osd_request, and
set it whenever an appropriate osd map flag is set.  Avoid queueing
paused requests in __map_request(), but force them to be resent if
they become unpaused.

Also subscribe to the next osd map from the monitor if any of these
flags are set, so paused requests can be unblocked as soon as
possible.

Fixes: http://tracker.ceph.com/issues/6079

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
libceph: add an option to configure client behavior when osds are full

Default to blocking requests to be consistent with userspace. Some
applications may prefer the previous behavior of returning an error
instead, so make that an option. CephFS implements returning -ENOSPC
at a higher level, so only rbd is really affected by this.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
rbd: document rbd-specific options

osd_full_behavior only affects rbd, so document it along with
read-only and read-write.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>