stellar/stellard — pull request #98 by graydon (2014-09-09 23:22:49+00:00)
Title: Fix race between queueing redirect and closing connection, close #46.

What's happening here (as far as I can tell) is that the mtENDPOINTS
packet generated during a redirect is being dropped. The sequence of
events is:

- generate the packet
- post a send(packet) to the peer's strand, queueing a send
event for later
- set m_detaching=true
- call async_shutdown() on the socket, nothing sent on the
wire past this point
- peer's strand then dispatches the send()
- sees m_detaching==true and drops packet

The fix is as follows:

1. detach() is split in two: detach() itself just posts a call to
do_detach() to the strand. m_detaching is set in do_detach(),
which is made idempotent.

2. sendPacket() still checks m_detaching and drops a packet if
it's true; but otherwise it always posts a call to the strand.
sendPacketForce renamed do_sendPacket

These two points mean that a synchronous call sequence like

{ send(); detach(); }

in the caller definitely results in two events posted to the strand,
in order. The send will make it through the test of m_detaching and
get posted, because m_detaching won't have been set yet. Then further:

3. do_sendPacket() is modified to queue packets to mSendQ if there's
a write in progress (this formerly raced on the socket and the
queue, as far as I can tell).

4. do_detach() considers queue-not-empty or write-in-progress as
"work pending" and returns early, doesn't fully close the socket.

5. handleWrite re-posts do_sendPacket() or do_detach() based on the
presence of work in mSendQ and the setting of m_detaching,
respectively.

These three points mean that the strand gets drained into the socket
if possible, the queue if necessary, with handleWrite reposting work
that got queued every time a write finishes.