libceph: eliminate connection state "DEAD"

The ceph connection state "DEAD" is never set and is therefore not
needed.  Eliminate it.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: rename socket callbacks

Change the names of the three socket callback functions to make it
more obvious they're specifically associated with a connection's
socket (not the ceph connection that uses it).

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: start separating connection flags from state

A ceph_connection holds a mixture of connection state (as in "state
machine" state) and connection flags in a single "state" field.  To
make the distinction more clear, define a new "flags" field and use
it rather than the "state" field to hold Boolean flag values.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: start tracking connection socket state

Start explicitly keeping track of the state of a ceph connection's
socket, separate from the state of the connection itself.  Create
placeholder functions to encapsulate the state transitions.  In
time these will become more than just placeholders.

    --------
    | NEW* |  transient initial state
    --------
        | con_sock_state_init()
        v
    ----------
    | CLOSED |  initialized, but no socket (and no
    ----------  TCP connection)
     ^      \
     |       \ con_sock_state_connecting()
     |        ----------------------
     |                              \
     + con_sock_state_closed()       \
     |\                               \
     | \                               \
     |  -----------                     \
     |  | CLOSING |  socket event;       \
     |  -----------  await close          \
     |       ^                            |
     |       |                            |
     |       + con_sock_state_closing()   |
     |      / \                           |
     |     /   ---------------            |
     |    /                   \           v
     |   /                    --------------
     |  /    -----------------| CONNECTING |  socket created, TCP
     |  |   /                 --------------  connect initiated
     |  |   | con_sock_state_connected()
     |  |   v
    -------------
    | CONNECTED |  TCP connection established
    -------------

Make the socket state an atomic variable, reinforcing that it's a
distinct transtion with no possible "intermediate/both" states.
This is almost certainly overkill at this point, though the
transitions into CONNECTED and CLOSING state do get called via
socket callback (the rest of the transitions occur with the
connection mutex held).

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: rename kvec_reset and kvec_add functions

The functions ceph_con_out_kvec_reset() and ceph_con_out_kvec_add()
are entirely private functions, so drop the "ceph_" prefix in their
name to make them slightly more wieldy.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: encapsulate blocks of connection code

There's a funny block of code in try_write() that sets up for a
connection request to a server and puts a ceph connection in a
connected state if it is not already set up.

There's a similar block of code in try_read() that reads in the
response from a server to such a connection request.

Encapsulate both of those into functions that do a better job of
identifying their purpose.

This patch involves little more than code motion, to simplify
review.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: clear CONNECTING state on socket close

If the Linux socket reports it is closed, we mark the connection
SOCK_CLOSED.  If CONNECTING was set at that point, clear it so
we end up with a "clean" state until we can begin a new connection
sequence.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: clear NEGOTIATING when done

A connection state's NEGOTIATING bit gets set while in CONNECTING
state after we have successfully exchanged a ceph banner and IP
addresses with the connection's peer (the server).  But that bit
is not cleared again--at least not until another connection attempt
is initiated.

Instead, clear it as soon as the connection is fully established.
Also, clear it when a socket connection gets prematurely closed
in the midst of establishing a ceph connection (in case we had
reached the point where it was set).

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: define and use an explicit CONNECTED state

There is no state explicitly defined when a ceph connection is fully
operational.  So define one.

It's set when the connection sequence completes successfully, and is
basically cleared at this point whenever CLOSED or SOCK_CLOSED gets
set.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: delete useless SOCK_CLOSED manipulations

In con_close_socket(), SOCK_CLOSED is set in the connection state,
then cleared again after shutting down the socket.  Nothing between
the setting and clearing of that bit will ever be affected by it,
so there's no point in setting/clearing it at all.  So don't.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: distinguish two phases of connect sequence

Currently a ceph connection enters a "CONNECTING" state when it
begins the process of (re-)connecting with its peer.  The first
step of this is for both parties to exchange a banner, then to
exchange their IP addresses (client sends its own to server, and
server sends its own and client's back to client).

Once that is complete, the client sends the server a connect
message along with an optional block of authorization data.
During this phase, an additional NEGOTIATING bit is set in
the ceph connection's state--though the CONNECTING bit/state
remains.

Finally, if the server provides a positive response to the client's
connect message, the connection is "alive" and usable.  At this
point both NEGOTIATING and CONNECTING are dropped from the ceph
connection state, and it enteres CONNECTED state.

Rather than have the CONNECTING state continue while the NEGOTIATING
bit is set, interpret these two phases as distinct states.  In other
words, when NEGOTIATING is set, clear CONNECTING.  That way only
one of them will be active at a time.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: make use of distinct CONNECTING state

Small but significant change that exploits and enforces the fact
that NEGOTIATING is now distinct from CONNECTING.  Done as a
separate commit for verification purposes.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: separate connection response handling

Currently one function handles processing the responses to both the
banner+address exchange and the connection parameters phases of the
ceph connection connect sequence.  Separate them, and make them be
done based on whether we're ion CONNECTING state or NEGOTIATING
state.

Signed-off-by: Alex Elder <elder@inktank.com>