libceph: set page alignment in start_request()

The page alignment field for a request is currently set in
ceph_osdc_build_request().  It's not needed at that point
nor do either of its callers need that value assigned at
any point before they call ceph_osdc_start_request().

So move that assignment into ceph_osdc_start_request().

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: conditionally set up page info

In ceph_osdc_start_request(), an osd request message has its page
array or bio information along with its trail pointer set
unconditionally.  This is required if these are needed for sending
data to the osd, but otherwise it is not necessary (and a little
misleading).

If the request reads data, the pages (or bio) pointer will be
set up to receive the data in the reply message, in get_reply().
(The trail field is only used for outgoing data.)

Change it so these fields are only set in ceph_osd_start_request()
if they're going to be used in the request message.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: isolate message page field manipulation

Define a function ceph_msg_data_set_pages(), which more clearly
abstracts the assignment page-related fields for data in a ceph
message structure.  Use it in the osd client and mds client.

Rearrange the field order in a ceph_msg structure to group those
that are used to define the possible data payloads.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: rename "total_max_write"

In write_partial_msg_pages() a variable "total_max_write" is used to
determine how many more bytes should be written from the current
piece of the message data (either the beginning of the data, or the
trail).

For some reason that variable name does not convey to me what it
represents, while "resid" (short for residual) would.  It's the
amount of data remaining to transfer out of what was requested.

So rename that variable, and move its definition into the innermost
scope where it is used.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: clean up write_partial_msg_pages()

This change is intended to improve the understandability of
write_partial_msg_pages().

First, add a local variable "page_offset" and use it to represent
the offset within the current page that will be used for the current
send operation.  If it's not a bio request, this is simply a shorter
name for con->out_msg_pos.page_pos.

For a request with bio data, "out_msg_pos" is actually used to
represent the position within the current segment of the current
bio.  This is a bit confusing, so add a comment explaining that.

Get rid of "max_write", and just use the "len" variable to hold the
initial limit for the length of the current sendpage request.  (It
is later limited further to not exceed what remains of the original
request.)

Finally, get rid of "bio_offset", and instead just update the page
offset value to reflect the actual page position from which we'll be
sending data.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: encapsulate assignment of message pagelists

Move the assignment of a message pagelist as well as the trail
portion of a message (which is also a ceph pagelist) into the
messenger code rather than allowing it to be done outside that
module.

Define and use an init function for these fields as well.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: wrap ceph msg trail in a struct

Define a structure to contain the pagelist defining the trail
portion of the data in a ceph message.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: define a trail data type

Add a type field to the structure representing the trail of a
message's data section.  Use it in a few spots to determine and
validate how the trail is used.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: define helpers for trail pages

Define msg_trail_page_get() and msg_trail_page_advance().  These are
helpers for, respectively, getting the first (next) page to process
in the trail of a message and for advancing to the next page in the
trail.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: use list_rotate_left() for tail

This is pulled out as a separate patch just because it warrants a
little extra explanation.

The trail portion of a message is a ceph pagelist.  We process
(i.e., send data from) the trail in out_msg_pos_next().  That
function takes the page supplied by msg_trail_page_get() and uses
it for a sendpage call.

Afterward, out_msg_pos_next() is called to update the connection's
out_msg_pos field to account for the amount of data transferred.
If everything from the current page was transferred, it advances
to the next page, which for data in the trail of the message is
implemented by a call to msg_trail_page_advance(), like so:

	list_move_tail(&page->lru, &msg->trail.pagelist->head);

In other words, the current page is moved to the end of the pagelist.

We know that the current page will be the first page on the list, so
this operation is basically keeping the (circular) list intact but
advancing the head one position.  There's a list operation that does
that--list_rotate_left()--so use it to make it more obvious what's
going on.

This also obviates the need for the "page" parameter in
msg_trail_page_advance(), so get rid of it.

Signed-off-by: Alex Elder <elder@inktank.com>