ask/celery — Issue #634: Result backend resilience & SSL
Author: hynek
Created: 2012-03-05 10:36:28+00:00
State: open

I understand, that celeryd doesn't cache connections in a pool: https://github.com/ask/celery/issues/603#issuecomment-3848525

However, I have already had a problem twice (apron after three days running) where "something" inside of celeryd broke and I kept getting exceptions when it tried to save the result into a postgresql backend. It never heals until restarted. NB: the postgres server hasn’t been restarted in the time – although I’d prefer celeryd to handle that well nevertheless. :)

It might be coincidence, but in both cases it were SSL errors. The first time:

```
[2012-03-02 03:03:16,307: ERROR/MainProcess] Pool callback raised exception: DatabaseError('(DatabaseError) SSL error: decryption failed or bad record mac\n',)
Traceback (most recent call last):
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/celery/concurrency/processes/pool.py", line 76, in safe_apply_callback
fun(*args)
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/celery/worker/job.py", line 360, in on_failure
self.task.backend.mark_as_failure(self.id, exc_info.exception)
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/celery/backends/base.py", line 96, in mark_as_failure
traceback=traceback)
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/celery/backends/base.py", line 229, in store_result
return self._store_result(task_id, result, status, traceback, **kwargs)
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/celery/backends/database.py", line 55, in _store_result
task = session.query(Task).filter(Task.task_id == task_id).first()
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2024, in first
ret = list(self[0:1])
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 1918, in __getitem__
return list(res)
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2093, in __iter__
return self._execute_and_instances(context)
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2108, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1405, in execute
params)
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1538, in _execute_clauseelement
compiled_sql, distilled_params
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1646, in _execute_context
context)
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1639, in _execute_context
context)
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 330, in do_execute
cursor.execute(statement, parameters)
DatabaseError: (DatabaseError) SSL error: decryption failed or bad record mac
```

and the second time:

```
[2012-03-05 10:28:12,776: CRITICAL/MainProcess] Task copy_tables.tasks.copy_cron[5721505b-2a02-4e21-b91c-e2660a73ce12] INTERNAL ERROR: SQLAlchemyError('(OperationalError) SSL SYSCALL error: EOF detected\n',)
Traceback (most recent call last):
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/celery/execute/trace.py", line 149, in trace_task
"hostname": hostname}, STARTED)
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/celery/backends/base.py", line 229, in store_result
return self._store_result(task_id, result, status, traceback, **kwargs)
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/celery/backends/database.py", line 55, in _store_result
task = session.query(Task).filter(Task.task_id == task_id).first()
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2024, in first
ret = list(self[0:1])
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 1918, in __getitem__
return list(res)
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2093, in __iter__
return self._execute_and_instances(context)
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2108, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1405, in execute
params)
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1538, in _execute_clauseelement
compiled_sql, distilled_params
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1646, in _execute_context
context)
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1639, in _execute_context
context)
File "/vrmd/copytables/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 330, in do_execute
cursor.execute(statement, parameters)
OperationalError: (OperationalError) SSL SYSCALL error: EOF detected
```

In both cases I get various alternations of these exceptions, depending which layer tried to do something with the database (the first one is the initial exception of them, the second one the last).

It's pretty important to me that this works. So, can I do _anything_ to avoid these problems or is it a bug in celery?

Unfortunately it caught me few hours before I have to hand in my notebook and leave on vacation, so I couldn’t investigate inside of celery's code. :(

That said, I’m leaving from March 6th – March 20th on a vacation. Missing replies by me mean no missing courtesy, I won't have much Internet on my travels. This issue is very important to me though.