libceph: encapsulate and document connect sequence

Encapsulate the code handles the initial phase of establishing a
ceph connection with a peer, and add a bunch of documentation about
what's involved.  Change process_banner() to return 1 on success
rather than 0, to allow the new ceph_con_connect_response() to
return 0 to indicate the response has not yet been completely read.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: encapsulate and document negotiation phase

Encapsulate the code handles the negotiation phase of establishing a
ceph connection with a peer, and add a bunch of documentation about
what's involved.  Change process_connect() to return 1 on success
rather than 0, to allow the new ceph_con_negotiate_response() to
return 0 to indicate the response has not yet been completely read.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: close the connection's socket on reset

When a ceph connection is reset, all its state is cleared.  However
the underlying socket never actually gets closed.  Do that, to
essentially make the reset process complete.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: don't close socket in OPENING state

The only way a socket enters OPENING state is via ceph_con_open().

The only times ceph_con_open() is called are:
  - In fs/ceph/mds_client.c:register_session(), where it occurs
    soon after a call to ceph_con_init().
  - In fs/ceph/mds_client.c:send_mds_reconnect().  This is
    called in two places.
    - In fs/ceph/mds_client.c:check_new_map(), it is called
      after a call to ceph_con_close()
    - Or in fs/ceph/mds_client.c:peer_reset(), which is also only
      called after reset_connection, which includes a call to
      ceph_con_close().
  - In net/ceph/mon_client.c:__open_session(), where it's called
    right after a call to ceph_con_init().
  - In net/ceph/osd_client.c:__reset_osd(), right after a call
    to ceph_con_close().
  - In net/ceph/osd_client.c:__map_request(), shortly after a call
    to create_osd(), which includes a call to ceph_con_init().

After a call to ceph_con_init(), the state of a ceph connection is
CLOSED, and its socket pointer is null.

Similarly, after a call to ceph_con_close(), the state of the
connection is CLOSED, the underlying socket is closed, and the
connection's socket pointer is null.

Therefore, there is no reason to call con_close_socket() when a
connection is found to be in OPENING state in con_work(), because
the socket will already be closed, and the connection will already
be in CLOSED state.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: change TAG_CLOSE handling

Currently, if a connection is READY in try_read(), and a CLOSE tag
is the what is received next, the connection's state changes from
CONNECTED to CLOSED and try_read() returns.

If this happens, control returns to con_work(), and try_write()
is called.  If there was queued data to send, try_write() appears
to attempt to send it despite the receipt of the CLOSE tag.

Eventually, try_write() will return either:
  - A non-negative value, in which case con_work() will end, and
    will at some point get triggered to run by an event.
  - -EAGAIN, in which case control returns to the top of con_work()
  - Some other error, which will cause con_work() to call
    ceph_fault(), which will close the socket and force a new
    connection sequence to be initiated on the next write.

At the top of con_work(), if the connection is in CLOSED state,
the same fault handling will be done as would happen for any
other error.

Instead of messing with the connection state deep inside try_read(),
just have try_read() return a negative value (an errno), and let
the fault handling code in con_work() take care of resetting the
connection right away.  This will also close the connection before
needlessly sending any queued data to the other end.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: kill fail_protocol()

In the negotiating phase of establishing a connection, the server
can indicate various connection failures using special tag values.
The tags can mean: that the client does not have features needed
by the server; that the protocol advertised by the client is not
what the server expects; or that the authorizer data provided by
the client was not adequate to grant access.

These three cases are handled in process_connect(), which calls
fail_protocal() for all three.  The result of that is that the
connection gets reset, and the connection gets moved to CLOSED
state.

The previous patch description walks through what happens when
a connection gets marked CLOSED within try_read(), and why it's
sufficient (and better) to simply have it return a negative value.

So just do that--don't bother with fail_protocol(), just return a
negative value in these cases and let the caller sort out resetting
things.  Return -EIO in these cases rather than -1 (which can be
confused with -EPERM).

We can get rid of fail_protocol() because it is no longer used.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: close connection on reset tag

When a CEPH_MSGR_TAG_RESETSESSION tag is received, the connection
should be reset, dropping any pending messages and preparing for
a new connection to be negotiated.

Currently, reset_connection() is called to do this, but that only
drops messages.  To really get the connection fully reset, call
ceph_con_close() instead.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: close connection on connect failure

The only time the CLOSED state is set on a ceph connection is in
ceph_con_init() and ceph_con_close().  Both of these will ensure
the connection's socket is closed.  Therefore there is no need
to close the socket in con_work() if the connection is found to
be in CLOSED state.

Rearrange things a bit in ceph_con_close() so we only manipulate
the state and flag bits *after* we've acquired the connection mutex.

Signed-off-by: Alex Elder <elder@inktank.com>
libceph: set CONNECTING state even earlier

Move the setting of the CONNECTING state in a ceph connection
all the way back to where a connection first gets opened.  At
that point the connection's socket pointer is still null, and
the connection sequence is about to begin.

Signed-off-by: Alex Elder <elder@inktank.com>