libceph: encapsulate advancing msg page

In write_partial_msg_pages(), once all the data from a page has been
sent we advance to the next one.  Put the code that takes care of
this into its own function.

While modifying write_partial_msg_pages(), make its local variable
"in_trail" be Boolean, and use the local variable "msg" (which is
just the connection's current out_msg pointer) consistently.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: encapsulate out message data setup

Move the code that prepares to write the data portion of a message
into its own function.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: don't mark footer complete before it is

This is a nit, but prepare_write_message() sets the FOOTER_COMPLETE
flag before the CRC for the data portion (recorded in the footer)
has been completely computed.  Hold off setting the complete flag
until we've decided it's ready to send.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: move init of bio_iter

If a message has a non-null bio pointer, its bio_iter field is
initialized in write_partial_msg_pages() if this has not been done
already.  This is really a one-time setup operation for sending a
message's (bio) data, so move that initialization code into
prepare_write_message_data() which serves that purpose.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: don't use bio_iter as a flag

Recently a bug was fixed in which the bio_iter field in a ceph
message was not being properly re-initialized when a message got
re-transmitted:
    commit 43643528cce60ca184fe8197efa8e8da7c89a037
    Author: Yan, Zheng <zheng.z.yan@intel.com>
    rbd: Clear ceph_msg->bio_iter for retransmitted message

We are now only initializing the bio_iter field when we are about to
start to write message data (in prepare_write_message_data()),
rather than every time we are attempting to write any portion of the
message data (in write_partial_msg_pages()).  This means we no
longer need to use the msg->bio_iter field as a flag.

So just don't do that any more.  Trust prepare_write_message_data()
to ensure msg->bio_iter is properly initialized, every time we are
about to begin writing (or re-writing) a message's bio data.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: refactor write length calculation

In write_partial_msg_pages() there is a bunch of logic that computes
the number of bytes our next send request should contain.  It uses
two local variables--one based on the end of the message (or its
trail portion) and one which is based on the end of the data for
the message that resides the current data page being written.
This patch simplifies this code a bit.

First, if a message's data resides in a bio, the page data for each 
bio_vec entry will never exceed the size of a page.  So there's no
harm in comparing that against the PAGE_SIZE max_write value that is
used for all other data page scenarios handled here.  If we just
set total_max_write to be the size of the next bio_vec we can get
rid of max_write and substitute PAGE_SIZE in the one remaining place
it was used.

Since we have eliminated the need for "max_write" it would be nice
to rename "total_max_write".  But since it's really a temporary
value for the length we'll be writing, just use the existing "len"
variable and get rid of "total_max_write" also.

OSD client messages can sometimes have a "trail" portion.  The size
of this is included in the length of the message's data, but the
pages containing it are always kept in a separate page list.  If
the message data is contained in a bio, it would be an error for
any of the bio data to overlap the trail, so insert a BUG_ON() to
make this invariant obvious.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: move init_bio_*() functions up

Move init_bio_iter() and iter_bio_next() up in their source file so
the'll be defined before they're needed.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: kill bio_offset variable

In write_partial_msg_pages() there is a local variable bio_offset
that is used to hold the (additional) offset into a page for a bio.
This is added to outgoing message's page_pos value to get the offset
into the page from which to send data.

This is a little cumbersome and confusing.  Change it so page_pos
is exactly the offset into the page to start sending data, just like
all the other ways we send pages (page array, page list, etc.).  To
do this, initialize page_pos to be a bio_vec's offset when we first
encounter it, and then update it as data gets sent.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: small changes to messenger.c

This patch gathers a few small changes to write_partial_msg_pages().
  - use a local variable trail_off to represent the offset into
    a message of the trail portion of the data (if present)
  - once we are in the trail portion we will always be there, so we
    don't always need to check against our data position 
  - avoid computing len twice after we've reached the trail
  - get rid of the variable tmpcrc, which is not needed
  - trail_off and trail_len never change so mark them const
  - update some comments

It also includes some other small changes:
  - A small change--basically indentation--to out_msg_pos_next()
  - bio_iovec_idx() will never return an error, so don't bother
    checking for it in read_partial_message_bio()

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: record history of messenger sendpage calls

Create a log buffer to record the ceph_tcp_sendpage() calls that
are performed by write_partial_msg_pages()

Signed-off-by: Alex Elder <elder@inktank.com>