libceph: drop mutex on error in handle_reply()

The osd client mutex is acquired just before getting a reference to
a request in handle_reply().  However the error paths after that
don't drop the mutex before returning as they should.

Drop the mutex after dropping the request reference.  Also add a
bad_mutex label at that point and use it so the failed request
lookup case can be handled with the rest.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: define ceph_decode_pgid() only once

There are two basically identical definitions of __decode_pgid()
in libceph, one in "net/ceph/osdmap.c" and the other in
"net/ceph/osd_client.c".  Get rid of both, and instead define
a single inline version in "include/linux/ceph/osdmap.h".

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: record length of bio list with bio

When assigning a bio pointer to an osd request, we don't have an
efficient way of knowing the total length bytes in the bio list.
That information is available at the point it's set up by the rbd
code, so record it with the osd data when it's set.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: record message data length

Keep track of the length of the data portion for a message in a
separate field in the ceph_msg structure.  This information has
been maintained in wire byte order in the message header, but
that's going to change soon.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: don't build request in ceph_osdc_new_request()

This patch moves the call to ceph_osdc_build_request() out of
ceph_osdc_new_request() and into its caller.

This is in order to defer formatting osd operation information into
the request message until just before request is started.

The only unusual (ab)user of ceph_osdc_build_request() is
ceph_writepages_start(), where the final length of write request may
change (downward) based on the current inode size or the oldest
snapshot context with dirty data for the inode.

The remaining callers don't change anything in the request after has
been built.

This means the ops array is now supplied by the caller.  It also
means there is no need to pass the mtime to ceph_osdc_new_request()
(it gets provided to ceph_osdc_build_request()).  And rather than
passing a do_sync flag, have the number of ops in the ops array
supplied imply adding a second STARTSYNC operation after the READ or
WRITE requested.

Signed-off-by: Alex Elder <elder@inktank.com>
ceph: define ceph_writepages_osd_request()

Mostly for readability, define ceph_writepages_osd_request() and
use it to allocate the osd request for ceph_writepages_start().

Signed-off-by: Alex Elder <elder@inktank.com>
ceph: kill ceph alloc_page_vec()

There is a helper function alloc_page_vec() that, despite its
generic sounding name depends heavily on an osd request structure
being populated with certain information.

There is only one place this function is used, and it ends up
being a bit simpler to just open code what it does, so get
rid of the helper.

The real motivation for this is deferring building the of the osd
request message, and this is a step in that direction.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: hold off building osd request

Defer building the osd request until just before submitting it in
all callers except ceph_writepages_start().  (That caller will be
handed in the next patch.)

Signed-off-by: Alex Elder <elder@inktank.com>
ceph: build osd request message later for writepages

Hold off building the osd request message in ceph_writepages_start()
until just before it will be submitted to the osd client for
execution.

We'll still create the request and allocate the page pointer array
after we learn we have at least one page to write.  A local variable
will be used to keep track of the allocated array of pages.  Wait
until just before submitting the request for assigning that page
array pointer to the request message.

Note that ceph_osdc_build_request() computes the payload length for
a request based on the length of the ops in the osd request.  This
means that, because we don't build the request message until later,
there is no need to set the payload length any more.  We no longer
need to assign the request header's data_len field either, for the
same reason.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: messenger owns message data length

Whenever a message has a data payload, it is "set" into the message
using one of the ceph_msg_data_set_*() functions.  When that happens
the message's data_length field is updated to record the length of
that data.

Because of this, there is no need for anything other than the
messenger to be concerned with keeping the length of a message's
data correct.  In particular, code outside the messenger has
generally no business setting the data_len field inside a message
header.

It is appropriate to *read* the data_len value when processing an
incoming message however (but that will unnecessary shortly).

Create a new function prepare_out_message_hdr() that does final
formatting of information in the header of a connection's outgoing
message just before we start sending it.  Assign the header length
field in that function (along with a few other things that were
previously set in prepare_write_message()).  Stop doing any setting
of the header data length anywhere but inside that function.

Note that this includes the osd client, which is trusted to
properly set up the message data for any osd op it includes
in an osd request.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: pass message data length to complete_request()

In the osd client, provide the length of the data portion of the
response message for an osd request to complete_requests().  This
will be used in the upcoming patches.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: don't supply message to osd request callback

The only thing the osd request callback functions ever use is the
length of the data portion of the response message.  So stop passing
that, and pass just the data length instead.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: don't pass message header to get_reply()

The only thing out of the ceph message header needed by get_reply()
in the osd client is the transaction id, the front length, and the
data length.  Decode those in the caller, and pass the decoded
values rather than the address of the message header.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: have handle_reply() return a value

Have the osd client handle_reply() return an error code.  Add
warning messages to distinguish various cases, and have the caller
report an generic error message.  Have the caller dump the bad
message in the event of an error.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: don't pass message to handle_reply()

Rather than the entire message, pass only the parts of the response
message that are required to the osd client's handle_response()
function.

With this in place, there are only three functions in the osd client
that drectly manipulate any of the fields in an osd request's
request or response messages:
    - the alloc_msg connection operation, alloc_msg();
    - the dispatch connection operation, dispatch();
    - ceph_osdc_build_request(), whose job it is to format
      osd request information into the data portion of the
      request message; and
    - handle_reply() still decodes data, but it is limited to
      the front portion of the message.

Any data in a message gets transferred to or from the message data
field destination directly by the messenger (based on the way the
ops in the request's op array set it up).

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: move cursor into message

A message will only be processing a single data item at a time, so
there's no need for each data item to have its own cursor.

Move the cursor embedded in the message data structure into the
message itself.  To minimize the impact, keep the data->cursor
field, but make it be a pointer to the cursor in the message.

This allows us to move the definition of ceph_msg_data above that of
ceph_msg_data_cursor (which allow the cursor point to the data
without a forward definition rather than vice-versa).

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: have cursor point to data

Rather than having a ceph message data item point to the cursor it's
associated with, have the cursor point to a data item.  This will
allow a message cursor to be used for more than one data item.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: replace message data pointer with list

In place of the message data pointer, use a list head which links
through message data items.  For now we only support a single entry
on that list.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: trivial cleanups

Blah.

Signed-off-by: Alex Elder <elder@inktank.com>