Layout
Unique fields
Avoid I/O inside modifyMVar

Many of the API functions had the structure

    modifyMVar (state of heavyweight connection) $ do
      ...
      send on heavyweight connection
      ...
      return (new heavyweight connection state)

Unfortunately, the thread that handles *incoming* messages also occassionally
needs to make modifications to the state of the heavyweight connection, and
hence also calls modifyMVar in various places. This can lead to "distributed
deadlock": consider endpoints A and B. Both A and B start sending lots of
messages to the other party, so much that in both cases the 'send' in the above
example starts to block (because the OS send buffer fills up). This means that
both parties will be stuck, with the MVar "locked", until the other party
receives some messages. However, it might happen that both parties did a
receive and need to obtain the mvar in order to make some changes -- but that
MVar is still blocked by the sending thread on that side. So we have

                   waiting on
  A send thread ---------------\            /------------ B send thread
       ^                        \          /                    ^
       |                         \        /                     |
       | waiting on       /--------------/                      |
       |                 /         \                            |
  A receive thread <----/           \-------------------> B receive thread

A nice distributed deadlock. To avoid this problem we now avoid doing sends
within a modifyMVar. This is not entirely trivial, however. Simply doing

  thingToSend <- modifyMVar (....)
  send thingToSend

is not good enough, because now if we have to threads doing the above the sends
may be done in the wrong order. We therefore do

  act <- modifyMVar (... ; act <- schedule (...); ...; return act)
  runScheduledAction act

'schedule' adds an IO operation to a queue, and runSCheduledAction runs the
next operation from this queue. This may mean of course that
'runScheduledAction' may not execute the action which was scheduled "textually
before" it, so we have to be careful that the right exceptions are thrown in
the right place. That's why 'schedule' creates an MVar for each operation, and
'runScheduledAction' waits for that MVar to be filled.
Optimize handling of 'Crossed' connection requests

When A tries to connect (set up a heavyweight connection to) B, while B tries
to connect to A at the same time, there will be two threads trying to set up a
heavyweight connection for A (and similarly for B): the thread that deals with
the apiConnect, and the thread that deals with the incoming request from B. In
the case that "B" wins and sends a ConnectionRequestCrossed message to A, the
thread that daels with the apiConenct should simply wait until the
handleIncomingRequest thread completes.

We did this already, but it is important that the handleIncomingRequest thread
*in B* (which loses to B's apiConnect thread) waits until A receives the
ConnectionRequestCrossed message (to avoid certain ordering problems). We had
implemented this by regarding an incoming ConnectionRequestCrossed message as
changing the heavyweight connection from Init to Closed state (and signalling
on resolved), but that's wrong as it will cause A to try and connect again. Now
instead when we receive a ConnectionRequestCrossed message we signal on a (new)
MVar "crossed" (part of the Init state), but leave the EndPoint in Init state
(it will be resolved by the handleIncmoingMessage thread).
Make sure that concurrent sends don't interleave
Switch to asynchronous connect

In the original protocol we had recipient-allocated connection IDs. Hence, when
doing a connect, we would send a connection request to the remote endpoint
(provided that a heavyweight connection is already in place) and then wait for
the recipient to reply with a new connection ID. This however makes connection
much slower than it needs to be. We now use sender-allocated connection IDs, so
that connect turns into a simple message to the remote endpoint that we
allocated a new connection, and we no longer have to wait for a response.

This means that all operations, with the sole exception of creating the
heavyweight connection itself, are now synchronous. Hence, we have completely
removed infrastructure relating to 'remote requests' (doRemoteRequest,
ConnectionRequestResponse, etc.).

What's not obvious is that we also had to change the 'CloseSocket' message.  A
sends a CloseSocket message to B when A thinks that the heavyweight connection
to B is no longer required. A doesn't immediately close the socket, because it
is possible that B has already sent a new connection request (now a
CreatedNewConnection message) to A, even though A has not yet received it.
Instead, it puts the heavyweight connection in 'Closing' state and waits for B
to confirm or deny the request.

When B receives A's CloseSocket message, it checks if it agrees that the
heavyweight connection can be closed. If not, it simply ignores the request and
A will forget it ever sent a CloseSocket request when it receives a
CreatedNewConnection message from B (that is, it moves the endpoint from
Closing to Valid state). If B does agree, then it will reply with a CloseSocket
message, and then actually close the socket. When A receives a CloseSocket
message having sent a CloseSocket message it closes the socket immediately.

However, now that we have asynchronous connection requests we might have the
following potential ordering of messages:

  A sends CloseSocket request, and puts endpoint in closing state

  B sends CreatedNewConnection, CloseConnection, and CloseSocket requests,
  putting its own representation of the endpoint also in closing state.

  B receives the CloseSocket message from A. Since it's endpoint is in Closing
  state, it immediately closes the socket.

  A receives B CreatedNewConnection message, hence *cancelling* the CloseSocket
  message it sent to be and putting the endpoint back in Valid state -- *even
  though* B did closed the socket.

To avoid this problem, we now send the ID of the last created connection along
with the close socket request. Then when B receives a CloseSocket request, it
can compare this ID with the ID of the last connection it opened. If this does
not match, B knows that there is still a CreatedNewConnection message underway
to A, which will cancel the CloseSocket, and hence it will know to ignore the
CloseSocket request from A.