libceph: eliminate connection state "DEAD"

The ceph connection state "DEAD" is never set and is therefore not
needed.  Eliminate it.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: rename socket callbacks

Change the names of the three socket callback functions to make it
more obvious they're specifically associated with a connection's
socket (not the ceph connection that uses it).

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: start separating connection flags from state

A ceph_connection holds a mixture of connection state (as in "state
machine" state) and connection flags in a single "state" field.  To
make the distinction more clear, define a new "flags" field and use
it rather than the "state" field to hold Boolean flag values.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: delete useless SOCK_CLOSED manipulations

In con_close_socket(), SOCK_CLOSED is set in the connection state,
then cleared again after shutting down the socket.  Nothing between
the setting and clearing of that bit will ever be affected by it,
so there's no point in setting/clearing it at all.  So don't.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: start tracking connection socket state

Start explicitly keeping track of the state of a ceph connection's
socket, separate from the state of the connection itself.  Create
placeholder functions to encapsulate the state transitions.  In
time these will become more than just placeholders.

    --------
    | NEW* |  transient initial state
    --------
        | con_sock_state_init()
        v
    ----------
    | CLOSED |  initialized, but no socket (and no
    ----------  TCP connection)
     ^      \
     |       \ con_sock_state_connecting()
     |        ----------------------
     |                              \
     + con_sock_state_closed()       \
     |\                               \
     | \                               \
     |  -----------                     \
     |  | CLOSING |  socket event;       \
     |  -----------  await close          \
     |       ^                            |
     |       |                            |
     |       + con_sock_state_closing()   |
     |      / \                           |
     |     /   ---------------            |
     |    /                   \           v
     |   /                    --------------
     |  /    -----------------| CONNECTING |  socket created, TCP
     |  |   /                 --------------  connect initiated
     |  |   | con_sock_state_connected()
     |  |   v
    -------------
    | CONNECTED |  TCP connection established
    -------------

Make the socket state an atomic variable, reinforcing that it's a
distinct transtion with no possible "intermediate/both" states.
This is almost certainly overkill at this point, though the
transitions into CONNECTED and CLOSING state do get called via
socket callback (the rest of the transitions occur with the
connection mutex held).

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: rename kvec_reset and kvec_add functions

The functions ceph_con_out_kvec_reset() and ceph_con_out_kvec_add()
are entirely private functions, so drop the "ceph_" prefix in their
name to make them slightly more wieldy.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: encapsulate blocks of connection code

There's a funny block of code in try_write() that sets up for a
connection request to a server and puts a ceph connection in a
connected state if it is not already set up.

There's a similar block of code in try_read() that reads in the
response from a server to such a connection request.

Encapsulate both of those into functions that do a better job of
identifying their purpose.

This patch involves little more than code motion, to simplify
review.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: clear CONNECTING state on socket close

If the Linux socket reports it is closed, we mark the connection
SOCK_CLOSED.  If CONNECTING was set at that point, clear it so
we end up with a "clean" state until we can begin a new connection
sequence.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: clear NEGOTIATING when done

A connection state's NEGOTIATING bit gets set while in CONNECTING
state after we have successfully exchanged a ceph banner and IP
addresses with the connection's peer (the server).  But that bit
is not cleared again--at least not until another connection attempt
is initiated.

Instead, clear it as soon as the connection is fully established.
Also, clear it when a socket connection gets prematurely closed
in the midst of establishing a ceph connection (in case we had
reached the point where it was set).

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: define and use an explicit CONNECTED state

There is no state explicitly defined when a ceph connection is fully
operational.  So define one.

It's set when the connection sequence completes successfully, and is
basically cleared at this point whenever CLOSED or SOCK_CLOSED gets
set.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: distinguish two phases of connect sequence

Currently a ceph connection enters a "CONNECTING" state when it
begins the process of (re-)connecting with its peer.  The first
step of this is for both parties to exchange a banner, then to
exchange their IP addresses (client sends its own to server, and
server sends its own and client's back to client).

Once that is complete, the client sends the server a connect
message along with an optional block of authorization data.
During this phase, an additional NEGOTIATING bit is set in
the ceph connection's state--though the CONNECTING bit/state
remains.

Finally, if the server provides a positive response to the client's
connect message, the connection is "alive" and usable.  At this
point both NEGOTIATING and CONNECTING are dropped from the ceph
connection state, and it enteres CONNECTED state.

Rather than have the CONNECTING state continue while the NEGOTIATING
bit is set, interpret these two phases as distinct states.  In other
words, when NEGOTIATING is set, clear CONNECTING.  That way only
one of them will be active at a time.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: clear STANDBY on close notification

A ceph connection socket can receive a TCP_CLOSE notification while
in standby state.  This indicates that the socket was closed by
the "other side" and should not be considered in standby any more.



Signed-off-by: Alex Elder <elder@inktank.com>
libceph: make use of distinct CONNECTING state

Small but significant change that exploits and enforces the fact
that NEGOTIATING is now distinct from CONNECTING.  Done as a
separate commit for verification purposes.

The CONNECTING bit is also now cleared in con_close_socket(), to
reflect that we have ended the connection phase once the socket
has been closed.  (A new one will begin when needed.)

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: separate connection response handling

Currently one function handles processing the responses to both the
banner+address exchange and the connection parameters phases of the
ceph connection connect sequence.  Separate them, and make them be
done based on whether we're in CONNECTING state or NEGOTIATING
state.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: define and use DISCONNECTED state

Rather than just using the SOCK_CLOSED flag, define a new state
DISCONNECTED to represent the condition of having received notice
that a socket has closed (or waiting for close).

We might get multiple close events reported on the socket, so allow
the CLOSING connection socket state to transition to itself.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: disconnects require close of socket

For connections that allow lossy transmission, errors due to lost
data are tolerated and ignored.  The code that implements this is
embedded in ceph_fault().  However ceph_fault() is also used to
handle faults due to notification that the underlying socket is
no longer connected, and in this case we need to close the socket
just like a "normal" connection error.

Change ceph_fault() so LOSSYTX channels only ignore errors that
are due to data loss, not socket connection loss.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: get rid of SOCK_CLOSED flag

Now that the DISCONNECTED state indicates we have been notified a
socket has closed we can get rid of the SOCK_CLOSED flag.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: embed ceph messenger structure in ceph_client

A ceph client has a pointer to a ceph messenger structure in it.
There is always exactly one ceph messenger for a ceph client, so
there is no need to allocate it separate from the ceph client
structure.

Switch the ceph_client structure to embed its ceph_messenger
structure.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: embed ceph connection structure in mon_client

A monitor client has a pointer to a ceph connection structure in it.
This is the only one of the three ceph client types that do it this
way; the OSD and MDS clients embed the connection into their main
structures.  There is always exactly one ceph connection for a
monitor client, so there is no need to allocate it separate from the
monitor client structure.

So switch the ceph_mon_client structure to embed its
ceph_connection structure.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: provide osd number when creating osd

Pass the osd number to the create_osd() routine, and move the
initialization of fields that depend on it therein.

Signed-off-by: Alex Elder <elder@inktank.com>