ask/celery — Issue #621: InterfaceError when postgres connection closed unexpectedly
Author: yaniv-aknin
Created: 2012-02-24 10:45:24+00:00
State: open

It's my understanding that `psycopg2` raises an exception when trying to close an already-closed connection. This is different from most other database drivers (that fail silently) and is the root cause behind django ticket [#15802](https://code.djangoproject.com/ticket/15802), which is fixed in 1.4b1 by commit django/django@5269c7f27487187c6d05da83d8812f29d29c791d. The fix could arguably have deen done at the `psycopg2` level, but so far it hasn't been. Unfortunately, when using django 1.3.1, this affects celery as well (tested on django-celery 2.4.2).

The issue will occur if a celery task has the connection to its database broken unexpectedly, but you can simulate it by running a task with this code:

import os
from django.contrib.auth.models import User
import django.db
User.objects.all()[0] # or any other database operation to initialize the connection
django.db.connection.connection.close()

...then your celery worker will get into a bad state where any future task that uses the database will get a traceback like so:

/home/teolicy/main/.venv/local/lib/python2.7/site-packages/celery/worker/job.py:112: UserWarning: Exception outside body: : connection already closed
Traceback (most recent call last):
File "/home/teolicy/main/.venv/local/lib/python2.7/site-packages/celery/worker/job.py", line 106, in execute_safe return self.execute(*args, **kwargs)
File "/home/teolicy/main/.venv/local/lib/python2.7/site-packages/celery/worker/job.py", line 117, in execute
self.loader.on_task_init(self.task_id, self.task)
File "/home/teolicy/main/.venv/local/lib/python2.7/site-packages/djcelery/loaders.py", line 74, in on_task_init
self.close_database()
File "/home/teolicy/main/.venv/local/lib/python2.7/site-packages/djcelery/loaders.py", line 49, in close_database return django.db.close_connection()
File "/home/teolicy/main/deps/django/django/db/__init__.py", line 85, in close_connection
conn.close() File "/home/teolicy/main/deps/django/django/db/backends/__init__.py", line 244, in close
self.connection.close()
InterfaceError: connection already closed

map(str, (exc.__class__, exc, exc_info.traceback))))

Backporting the fix from django 1.4b1 to django 1.3.1 fixes the problem in django-celery too, but I thought you may want to know about this issue or maybe even add specific protection against it in celery.