localshred/protobuf — pull request #220 by abrandoned (2014-09-16 15:53:37+00:00)
Title: Abrandoned/set timeout on workers avail

ok, learning more about the zmq server/client pairing

this PR moves client timeout (for request/response) to the zmq layer as zmq 3 added timeout on the socket layer directly and works "better" than a poller/polling item and makes the default available to a env var

also sets a rcv timeout and snd timeout on the available workers check, without this timeout there are significant issues when an rpc is abruptly killed and holds open requests from servers (we measured that a single rpc server will swallows 100-200 avail check requests during a `kill -9`); eventually the clients will "restart" the threads that died but if the client thread count is low it will yield an unusable system.

new env vars:
`PB_ZMQ_CLIENT_CHECK_AVAILABLE_RCV_TIMEOUT` - the rcv_timeout on the check avail, can be set very low as it is only sending a single byte message and trying to find an acceptable server, defaulted to 200ms which works under heavy load without issue
`PB_ZMQ_CLIENT_CHECK_AVAILABLE_SND_TIMEOUT` - all zmq timeouts have 2 elements the receive and the send as it knows if a packet has been sent, this is the send timeout for check avail and also set to 200ms by default
`PB_ZMQ_CLIENT_RCV_TIMEOUT` - rcv timeout for the actual request from the client, this sets the default and can be overridden by each call
`PB_ZMQ_CLIENT_SND_TIMEOUT` - snd timeout for the actual request from the client, this sets the default and can be overrideen by each call

Adding these timeouts made a significant change in time to recover after a process failure or process blockage

@quixoten @localshred @liveh2o