ask/kombu — Issue #86: select.select() timeout off by factor of 1000
Author: j00bar
Created: 2011-12-16 17:00:10+00:00
State: open

We're using celery 2.3.3/kombu 1.4.2 with the gevent processor model and a Redis broker. What we would observe is celery would BRPOP off enough elements to fill its prefetch limit, process those tasks, and then sit idle. Redis showed no BRPOP commands being issued. Using strace showed polling activity that yielded nothing and timer checking activity.

Then 300 seconds after the last BRPOP, the event polling would suddenly yield a ready-to-read status, BRPOPs would occur up to the prefetch limit, and the process would repeat.

Adding significant debugging output to the celery consumer, we isolated that kombu was locked up in kombu/utils/eventio.py running select.select(), called from kombu/transport/redis.py running self._poller.poll() in MultiChannelPoller. What interested me was that in redis.py, the timeout value of 1 is multiplied by 1000, transforming from seconds to milliseconds. In the epoll and kqueue polling models, that number is divided by 1000 before waiting for an event. In the select polling model, it remains milliseconds.

However, python stdlib docs suggest that select.select() takes a float value of seconds for its timeout value, not milliseconds. http://docs.python.org/library/select#select.select

We're hypothesizing that kombu is waiting 1000 seconds to timeout but that some socket timeout occurs at 300 seconds, resulting in the behavior we're observing. I can find no relevant reference to 300 seconds elsewhere in celery, kombu, redis-py, or our code. However, dividing the timeout in eventio.py class _select by 1000 before execution did eliminate the 300 second delay.

I'm not sure what the right fix is here. If _select's timeout should be divided by 1000, then all of the eventio classes represent their timeouts in seconds, so they should never be converted to milliseconds to begin with. If python's select() implementation wants seconds but eventlet's wants milliseconds, they shouldn't be the same class.

I'm happy to provide whatever additional information you might need. Thanks!