libceph: slightly defer registering osd request

One of the first things ceph_osdc_start_request() does is register
the request.  It then acquires the osd client's map semaphore and
request mutex and proceeds to map and send the request.

There is no reason the request has to be registered before acquiring
the map semaphore.  So hold off doing so until after the map
semaphore is held.

Since register_request() is nothing more than a wrapper around
__register_request(), call the latter function instead, after
acquiring the request mutex.

That leaves register_request() unused, so get rid of it.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: no more kick_requests() race

Since we no longer drop the request mutex between registering and
mapping an osd request in ceph_osdc_start_request(), there is no
chance of a race with kick_requests(), so there's no need to map and
send the new request conditional on (req->r_sent == 0).

So send it unconditionally (but issue a warning in case it occurs).

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: prepend requests in order when kicking

For the osd client, the connection fault method is osd_reset().
That function calls __reset_osd() to close and re-open the
connection, then calls __kick_osd_requests() to cause all
outstanding requests for the affected osd to be re-sent after
the connection has been re-established.

An osd client maintains distinct lists for unsent and in-flight
messages.  (It also has another list for requests for which no osd
target was known when it was mapped.)  When an osd is reset both
in-flight and unsent messages will need to be re-sent.  A ceph_osd
keeps its requests (both sent and un-sent) on a single requests
list.  (Each message is linked into two lists--one for the osd
client and one list for the osd.)

While processing an osd "kick" operation, each message on the
affected osd's request gets moved off whichever osd client list it
was on (unsent or sent) and placed onto the osd client's unsent
list.  (It remains where it is on the osd's request list.)

Finally, __kick_osd_requests() completes, osd_reset() calls
__send_queued() to cause each of the osd client's unsent messages to
be sent.

OK, with that background...

The osd expects incoming requests from a given client to arrive in
order, with the tid for each request being greater than the tid for
requests that have already arrived.

While arranging to re-send requests for an osd, the requests are
processed one-by-one in order, each is then prepended to the osd
client's unsent list in that same order.  The result is that
these moved requests are in *descending* order of tid on the
unsent list.

By traversing the osd request in reverse, their order on the unsent
list after prepending will be ascending by tid as desired.  Note
that this could also have been accomplished by moving these entries
to the end of the unsent list.  However if there were already
entries on that list they would have lower tid than the ones being
added here.  So they're prepended as before to keep the whole list
in order.

Just below that, traverse the linger list in forward order as
before, but add them to the *tail* of the list rather than the head.
They will each be given a new (higher) tid when they are registered
again, so that's where they belong.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: keep request lists in tid order

In __map_request(), when adding a request to an osd client's unsent
list, add it to the tail rather than the head.  That way the newest
entries (with the highest tid value) will be last.

Maintain an osd's request list in order of increasing tid also.

Finally--to be consistent--maintain an osd client's "notarget" list
in that order as well.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: send queued requests when starting new one

An osd expects the transaction ids of arriving request messages from
a given client to increase monotonically.  So the osd client needs
to send its requests in ascending tid order.

The transaction id for a request is set at the time it is
registered, in __register_request().  This is also where the request
gets placed at the end of the osd client's unsent messages list.

At the end of ceph_osdc_start_request(), the request message for a
newly-mapped osd request is supplied to the messenger to be sent
(via __send_request()).  If any other messages were present in the
osd client's unsent list at that point they would be sent *after*
this new request message.  Because those unsent messages have
already been registered, their tids would be lower than the
newly-mapped request message, and sending that message first
violates the tid ordering rule.  (Note, it may well be that
this case never occurs in real life.)

Rather than sending the new request only, send all queued requests
(including the new one) at that point in ceph_osdc_start_request().
This ensures the tid ordering property is preserved.

With this in place, all messages should now be sent in tid order
regardless of whether they're being sent for the first time or
re-sent as a result of a call to osd_reset().

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: verify requests queued in order

Add checking to verify all osd requests for an osd are added to the
queue in order of increasing tid.

Signed-off-by: Alex Elder <elder@inktank.com>