msgr: move SimpleMessenger function declarations into good order

We now have one declaration each of private and protected, and two
declarations of public -- one for the Messenger-required functions,
and one for the "internal" public functions. They're also in
some semblance of a readable order.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
msgr: convert get_policy() to return a const reference.

The only user who didn't want a const should have been using
direct access anyway (and now is doing so).

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
msgr: convert global_seq_lock to a spinlock.

Because nobody wants to get put to sleep waiting on a unique integer.
(I would just make this an atomic_t but we need to be able to accept
higher values from remote endpoints.)

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
msgr: add some validity checks to ~SimpleMessenger()

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
msgr: add error code returns to send_keepalive.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
msgr: make mark_down_on_empty() behave more consistently.

Previously you could get ms_handle_reset() notifications if you
used mark_down_on_empty(), but only if the Pipe's queue wasn't
empty at the time. Fix that by simply calling stop() instead
of fault(), which is more appropriate for the circumstances anyway.

Also fix the documentation, which was in fact incorrect.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
msgr: group private data members by type

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
msgr: reorder the SimpleMessenger data members a little bit

They make moderately more sense to me in this order.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
msgr: minimally document the SimpleMessenger inner classes

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
msgr: fix some spacing issues in Pipe

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
msgr: start moving functions around in Pipe
msgr: remove the Pipe::disposable flag.

It has no function as it stands. It was intended for new connections
to avoid being marked lossy, but that can't happen anyway.
So... *shrug*

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
msgr: merge submit_message() functions

Our consistency guarantees on address- and Connection-based send_message()
calls have diverged since they have different implementations.
To fix this, have everybody funnel down into a single submit_message()
function.
Convert the old addr-based submit_message() to also take a (possibly-NULL)
Pipe* param, and update documentation (and return codes) appropriately.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
msgr: Create a proper Connection* lazy_send_message()

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
msgr: do not establish a new Pipe for any server

Servers do not establish Connections, and if there's a Connection
without a Pipe it's because the Pipe was deliberately closed.
(Otherwise it would be in STATE_STANDBY.)

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
msgr: merge the send_message and lazy_send_message implementations.

They only varied by the presence of "lazy" (in output) and a flag.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>