andymccurdy/redis-py — Issue #225: Socket timeout on PubSub.listen()
Author: za-creature
Created: 2012-02-20 08:46:24+00:00
State: open

The following code will time out and raise a ConnectionError if a message is not published to any listened channel within 5 seconds. This doesn't really make any sense because the connection will be dropped and invalidated when this happens.

client = redis.StrictRedis(socket_timeout=5)
channel = client.pubsub()
channel.subscribe("test")
for message in channel.listen():
print message

Catching the ConnectionError in in the middle loop and calling listen() again will throw the following error (my guess is that _fp is deleted by the internal ConnectionError exception handler)

AttributeError: 'NoneType' object has no attribute 'readline'
Traceback (most recent call last):
File "./sub.py", line 11, in
for message in channel.listen():
File "/usr/lib/python2.7/site-packages/redis-2.4.11-py2.7.egg/redis/client.py", line 1306, in listen
r = self.parse_response()
File "/usr/lib/python2.7/site-packages/redis-2.4.11-py2.7.egg/redis/client.py", line 1250, in parse_response
response = self.connection.read_response()
File "/usr/lib/python2.7/site-packages/redis-2.4.11-py2.7.egg/redis/connection.py", line 263, in read_response
response = self._parser.read_response()
File "/usr/lib/python2.7/site-packages/redis-2.4.11-py2.7.egg/redis/connection.py", line 77, in read_response
response = self.read()
File "/usr/lib/python2.7/site-packages/redis-2.4.11-py2.7.egg/redis/connection.py", line 71, in read
return self._fp.readline()[:-2]

If it were me, I'd implement the listen() command with an optional timeout parameter, and throw a special TimeoutException if no message was received in *THAT* timeout, but leave the connection as is. Imho, the socket_timeout parameter should only be used for detecting network issues. Or at least, send a ping() command and only throw the ConnectionError if *THAT* times out. The only workaround I have found so far involves generating a new pubsub object after each timeout:

client = redis.StrictRedis(socket_timeout=5)
while True:
try:
channel = client.pubsub()
channel.subscribe("test")
for message in channel.listen():
print repr(message)
except KeyboardInterrupt:
raise#want to shutdown on ctrl+c
except:
logging.exception("")

The problem with this is that between the timeout and the generation of the new pubsub object (which I guess isn't all that long but it still counts), time during which new messages will not be delivered. Right now, this issue limits Py-Redis' usefulness in a enterprise environment (I'm designing a distributed system that uses a redis database, and I wanted to use redis for messaging as well, but I guess I'm forced to revert to AMQP)

Also, I believe that the documentation for the PubSub class should be updated. It is not immediately clear to me that PubSub.listen() is a generator from the following description: After subscribing to one or more channels, the listen() method will block until a message arrives on one of the subscribed channels. That message will be returned and it's safe to start listening again.