ceph: tidy ceph_mdsmap_decode() a little

I introduced a new temporary variable "info" instead of
"m->m_info[mds]".  Also I reversed the if condition and pulled
everything in one indent level.

Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com>
Reviewed-by: Alex Elder <elder@inktank.com>
rbd: wait for safe callback for write requests

When a request is sent to the osd, the sender can indicate what
"level" of completion desired before the request is acknowledged.
There will always be at least an acknowledgement sent (ACK), to
indicate the osd had received the message.  But for a write request
the sender may request that the acknowledgement not be sent until
the write operation is durable on the osd.  The osd marks a response
with the ONDISK flag to signal this in its acknowledgement.

    > > > > This needs to be updated--only one request? < < < <

Sometimes this flag is set in the first (only) response, but if not,
a second response will eventually arrive with that flag set.  The
initiator of the request is notified via one callback when the
acknowledgement arrives, and via a different callback when the
ONDISK response arrives.

Currently the rbd client waits only for the acknowledgement response
for all requests, which isn't safe for writes.  Fix that by defining
and using a different callback function that marks write requests
done only when the ONDISK notification arrives.

This resolves:
    http://tracker.ceph.com/issues/5146

Signed-off-by: Alex Elder <elder@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
libceph: add lingering request reference when registered

When an osd request is set to linger, the osd client holds onto the
request so it can be re-submitted following certain osd map changes.
The osd client holds a reference to the request until it is
unregistered.  This is used by rbd for watch requests.

Currently, the reference is taken when the request is marked with
the linger flag.  This means that if an error occurs after that
time but before the the request completes successfully, that
reference is leaked.

There's really no reason to take the reference until the request is
registered in the the osd client's list of lingering requests, and
that only happens when the lingering (watch) request completes
successfully.

So take that reference only when it gets registered following
succesful completion, and drop it (as before) when the request
gets unregistered.  This avoids the reference problem on error
in rbd.

Rearrange ceph_osdc_unregister_linger_request() to avoid using
the request pointer after it may have been freed.

And hold an extra reference in kick_requests() while handling
a linger request that has not yet been registered, to ensure
it doesn't go away.

This resolves:
    http://tracker.ceph.com/issues/3859

Signed-off-by: Alex Elder <elder@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
rbd: flush dcache after zeroing page data

Neither zero_bio_chain() nor zero_pages() contains a call to flush
caches after zeroing a portion of a page.  This can cause problems
on architectures that have caches that allow virtual address
aliasing.

This resolves:
    http://tracker.ceph.com/issues/4777

Signed-off-by: Alex Elder <elder@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>