dj2/em-jack — pull request #17 by cbascom (2012-06-27 02:44:45+00:00)
Title: Handle various connection failures

The initial problem I set out to fix was that when the beanstalk server was restarted while each_job was being used, the reserve command would end up on the callback queue prior to the use and watch commands issued by the reconnect method. When the reconnect restored the connection to the server, the reserve command would then be the first one sent causing us to wait for jobs on the wrong tube. The proposed fix for this issue is to call the use and watch commands prior to the add_timer call so that they are guaranteed to be on the callback queue prior to the reserve command.

While fixing that issue I noticed that it would work fine if the reconnect succeeded on the first attempt, but the tube state would not be restored if it took 2 or more attempts to reconnect. This is the same issue discussed in #10 except on a reconnect instead of the initial connection. The proposed fix for both of these issues is to store the tube to use and the list of tubes to watch in the @use_on_connect and @use_on_watch instance variables and leave them intact until the connection actually succeeds.

I also found that connection loss in the fiberized version would not result in the fiber being resumed since the deferrable did not have an errback registered. The proposed fix for this is always register an errback on the deferrable that simply resumes the fiber when called. Additionally, when this situation arises in each_job, the supplied block will not be invoked since there is no job to pass back to the caller. Instead, we simply use next_tick just like before to go back to blocking on the reserve call.

Finally, I also added an automatic ignore of the default tube if a specific tube is requested when creating the connection. The default tube will also automatically be ignored again if it is not in the @watched_tubes list when a reconnect occurs.

I added several tests to cover some of these scenarios and tweaked a couple of existing ones (mostly to handle the ignore behavior).

I probably should have done these as separate commits and pull requests, but got carried away getting it all to work. If you want some of this reverted or want it split into multiple commits, let me know and I will get is split up.