After speaking with Ben Gray and the discussion on the mailing list, this is an attempt to create a sockopt to allow connecting pipes to not immediately be available for traffic. The problem is in a PUSH to many PULL situation, where there is a connect to a PULL which is not there. This connect will immediately create a pipe (unlike bind), and traffic will be load balanced to that pipe. This means if there is a persistently unavailable end point then the traffic will queue until HWM is hit, and older messages will be lost.

This patch adds a sockopt ZMQ_DELAY_ATTACH_ON_CONNECT, which if set to 1 will attempt to preempt this behavior. It does this by extending the use of the session_base to include in the outbound as well as the inbound pipe, and only associates the pipe with the socket once it receives the connected callback via a process_attach message. This works, and a test has been added to show so, but may introduce unexpected complications. The shutdown logic in this class has become marginally more awkward because of this, requiring the session to serve as the sink for both pipes if shutdown occurs with a still-connecting pipe in place. It is also possible there could be issues around flushing the messages, but as I could not directly think how to create such an issue I have not written any code with regards to that.

The documentation has been updated to reflect the change, but please do check over the code and test and review.
Merge branch 'master' of https://github.com/zeromq/libzmq
Remove extra brackets as suggested by Martin H, and fix up a comment which was missing a word
And another typo on the same comment
Remove the extra outpipe handling as the session is quite capable of delaying the creation of the pipe until the connection has happened. Simply don't build the pipe, and let it do that automatically.
Fix a number of whitespace issues in various parts of the code, add validation to most calls on the test and take a first stab at implementing the reconnection pipe blocking.

It didn't seem straightforward to use any of the existing process calls, so I have added a new command to command_t and friends called detach. This instructs the socket_base to remove the pipe from it's pipe list. The session base stores a copy of the outpipe, and will resend the bind command on reconnection. This should allow balancing again.
Fix condition so that PGM and EPGM sockets always create pipes immediately, even if delay_attach_on_connect is set. This allows passing through the icanhasall flag, and is realistic given the fact those protocols should be able to connect immediately
Fix whitespace in test
Rebinding on reconnection to allow for the pipe blocking will set the event sink again, which will cause an assert to be triggered. I have modified that to allow for setting the same sink to be OK. I believe this should be ok - on termination, if a pipe hasn't reconnected, it would just attempt to be erased from the pipes list again, which is reasonable
On the advice of Martin Hurton, removed the new command type and just terminated the pipe in a reconnect situation, and notified the socket of the same. This handles the blocking properly, but at the cost of potentially losing in flight messages. However, this is a reasonable trade off given how much simpler it makes the patch.
Remove unnecessary extra test in pipe assertion - now we are terminating pipes we don't need to allow the case in which a pipe is rebound to the same sink
Fix incorrect whitespace in if statement
Removing unnecessary outpipe values that had been used for reconnecting existing pipes - no longer needed when using a pipe term for the delay_attach situation.
Restoring comment for clarity
Merge branch 'master' of https://github.com/zeromq/libzmq
When detaching a pipe, as well as checking the delay on connect sockopt is set, also ensure that the protocol is not pgm or epgm as we are not implementing the functionality for multicase types
The previous code to block the socket from receiving on that pipe during a disconnect was vulnerable to a race condition. This code calls with the terminate functions of both ends of the pipe - I believe this should be safer. This required storing a pointer to the socket end of the pipe
Merge branch 'master' of github.com:zeromq/libzmq
Merge branch 'master' of github.com:zeromq/libzmq
Reverted to a simpler shutdown. This seems to disconnect and reconnect the pipe properly, but there is a problem in overall shutdown when the pipe has blocked and reconnected - the session seems to get terminated() called on it only in shutdown for the original pipe, by which point it has been replaced. I am not sure at the moment why this only happens then, but this does mean this patch is broken at the moment