ask/kombu — Issue #110: SQLAlchemy broker fails with no libpq message, hanging celery worker
Author: gthb
Created: 2012-03-06 11:59:59+00:00
State: open

After I started using the SQLAlchemy broker instead of RabbitMQ, I saw the celery worker repeatedly hang, with the last message in the log looking like this:

@@@
Traceback (most recent call last):
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/celery/execute/trace.py", line 47, in trace
return cls(states.SUCCESS, retval=fun(*args, **kwargs))
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/celery/app/task/__init__.py", line 247, in __call__
return self.run(*args, **kwargs)
File "/opt/pyenv/LIVE/dm/elam/src/datamarket/dataimport/tasks.py", line 272, in wrapper
cls.unsynchronized_run(*args, **kwargs)
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/celery/app/__init__.py", line 175, in run
return fun(*args, **kwargs)
File "/opt/pyenv/LIVE/dm/elam/src/datamarket/dataimport/tasks.py", line 290, in taskfun
invoke_import_function(module, logger)
File "/opt/pyenv/LIVE/dm/elam/src/datamarket/dataimport/tasks.py", line 304, in invoke_import_function
any_updated = module.import_all()
File "/opt/pyenv/LIVE/dm/elam/src/datamarket/provider/federalreservestlouis.py", line 956, in import_all
import_one.delay(s, import_context)
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/celery/app/task/__init__.py", line 338, in delay
return self.apply_async(args, kwargs)
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/celery/app/task/__init__.py", line 456, in apply_async
**options)
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/celery/app/amqp.py", line 236, in delay_task
send(body, exchange=exchange, **extract_msg_options(kwargs))
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/kombu/connection.py", line 296, in _ensured
return fun(*args, **kwargs)
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/kombu/compat.py", line 64, in send
return self.publish(*args, **kwargs)
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/kombu/messaging.py", line 162, in publish
return publish(message, routing_key, mandatory, immediate, exchange)
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/kombu/entity.py", line 214, in publish
immediate=immediate)
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/kombu/transport/virtual/__init__.py", line 432, in basic_publish
exchange, routing_key, **kwargs)
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/kombu/transport/virtual/exchange.py", line 62, in deliver
_put(queue, message, **kwargs)
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/kombu/transport/sqlalchemy/__init__.py", line 65, in _put
self.session.commit()
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/sqlalchemy/orm/session.py", line 655, in commit
self.transaction.commit()
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/sqlalchemy/orm/session.py", line 313, in commit
self._prepare_impl()
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/sqlalchemy/orm/session.py", line 297, in _prepare_impl
self.session.flush()
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/sqlalchemy/orm/session.py", line 1559, in flush
self._flush(objects)
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/sqlalchemy/orm/session.py", line 1648, in _flush
transaction.rollback(_capture_exception=True)
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/sqlalchemy/orm/session.py", line 338, in rollback
transaction._rollback_impl()
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/sqlalchemy/orm/session.py", line 366, in _rollback_impl
t[1].rollback()
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/sqlalchemy/engine/base.py", line 1966, in rollback
self._do_rollback()
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/sqlalchemy/engine/base.py", line 2003, in _do_rollback
self.connection._rollback_impl()
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/sqlalchemy/engine/base.py", line 1221, in _rollback_impl
self._handle_dbapi_exception(e, None, None, None, None)
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/sqlalchemy/engine/base.py", line 1218, in _rollback_impl
self.engine.dialect.do_rollback(self.connection)
File "/opt/pyenv/LIVE/lib/python2.5/site-packages/sqlalchemy/engine/default.py", line 299, in do_rollback
connection.rollback()
DatabaseError: (DatabaseError) error with no message from the libpq
@@@

Switched back to RabbitMQ and did not investigate further. This is on Python 2.5.2 (Debian Lenny) with multiprocessing version 2.6.2.1