ask/celery — Issue #525: celerybeat could exit with error code 0 in case of DatabaseError when database scheduler (from django-celery) is used
Author: kmike
Created: 2011-11-06 10:32:06+00:00
State: open

This is an issue because it prevents restarting the celerybeat with supervisord (using default config) in case of temporary database downtime.

celery log:

```
CRITICAL 2011-11-05 12:42:56,495 celery.beat 24069 celerybeat raised exception : DatabaseError('server closed the connection unexpectedly\n\tThis probably means the server terminated abnormally\n\tbefore or while processing the request.\n',)
Traceback (most recent call last):
File "/home/zina/envs/zina/lib/python2.6/site-packages/celery/apps/beat.py", line 88, in start_scheduler
beat.start()
File "/home/zina/envs/zina/lib/python2.6/site-packages/celery/beat.py", line 400, in start
self.sync()
File "/home/zina/envs/zina/lib/python2.6/site-packages/celery/beat.py", line 403, in sync
self.scheduler.close()
File "/home/zina/envs/zina/lib/python2.6/site-packages/celery/beat.py", line 242, in close
self.sync()
File "/home/zina/envs/zina/src/django/django/db/transaction.py", line 217, in inner
res = func(*args, **kwargs)
File "/home/zina/envs/zina/lib/python2.6/site-packages/djcelery/schedulers.py", line 160, in sync
self.schedule[name].save()
File "/home/zina/envs/zina/lib/python2.6/site-packages/celery/beat.py", line 300, in schedule
return self.get_schedule()
File "/home/zina/envs/zina/lib/python2.6/site-packages/djcelery/schedulers.py", line 181, in get_schedule
if self.schedule_changed():
File "/home/zina/envs/zina/lib/python2.6/site-packages/djcelery/schedulers.py", line 139, in schedule_changed
ts = self.Changes.last_change()
File "/home/zina/envs/zina/lib/python2.6/site-packages/djcelery/models.py", line 165, in last_change
return cls.objects.get(ident=1).last_update
File "/home/zina/envs/zina/src/django/django/db/models/manager.py", line 132, in get
return self.get_query_set().get(*args, **kwargs)
File "/home/zina/envs/zina/src/django/django/db/models/query.py", line 344, in get
num = len(clone)
File "/home/zina/envs/zina/src/django/django/db/models/query.py", line 82, in __len__
self._result_cache = list(self.iterator())
File "/home/zina/envs/zina/src/django/django/db/models/query.py", line 273, in iterator
for row in compiler.results_iter():
File "/home/zina/envs/zina/src/django/django/db/models/sql/compiler.py", line 680, in results_iter
for rows in self.execute_sql(MULTI):
File "/home/zina/envs/zina/src/django/django/db/models/sql/compiler.py", line 735, in execute_sql
cursor.execute(sql, params)
File "/home/zina/envs/zina/src/django/django/db/backends/postgresql_psycopg2/base.py", line 44, in execute
return self.cursor.execute(query, args)
DatabaseError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

```

celerybeat error log (exception occurs again during shutdown):

```
[2011-11-05 12:42:56,658: INFO/MainProcess] process shutting down
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/util.py", line 235, in _run_finalizers
finalizer()
File "/usr/lib/python2.6/multiprocessing/util.py", line 174, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/home/zina/envs/zina/src/django/django/db/transaction.py", line 217, in inner
res = func(*args, **kwargs)
File "/home/zina/envs/zina/lib/python2.6/site-packages/djcelery/schedulers.py", line 160, in sync
self.schedule[name].save()
File "/home/zina/envs/zina/lib/python2.6/site-packages/celery/beat.py", line 300, in schedule
return self.get_schedule()
File "/home/zina/envs/zina/lib/python2.6/site-packages/djcelery/schedulers.py", line 181, in get_schedule
if self.schedule_changed():
File "/home/zina/envs/zina/lib/python2.6/site-packages/djcelery/schedulers.py", line 139, in schedule_changed
ts = self.Changes.last_change()
File "/home/zina/envs/zina/lib/python2.6/site-packages/djcelery/models.py", line 165, in last_change
return cls.objects.get(ident=1).last_update
File "/home/zina/envs/zina/src/django/django/db/models/manager.py", line 132, in get
return self.get_query_set().get(*args, **kwargs)
File "/home/zina/envs/zina/src/django/django/db/models/query.py", line 344, in get
num = len(clone)
File "/home/zina/envs/zina/src/django/django/db/models/query.py", line 82, in __len__
self._result_cache = list(self.iterator())
File "/home/zina/envs/zina/src/django/django/db/models/query.py", line 273, in iterator
for row in compiler.results_iter():
File "/home/zina/envs/zina/src/django/django/db/models/sql/compiler.py", line 680, in results_iter
for rows in self.execute_sql(MULTI):
File "/home/zina/envs/zina/src/django/django/db/models/sql/compiler.py", line 735, in execute_sql
cursor.execute(sql, params)
File "/home/zina/envs/zina/src/django/django/db/backends/postgresql_psycopg2/base.py", line 44, in execute
return self.cursor.execute(query, args)
DatabaseError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

```

Supervisord log record (I think the hour is different because celery logs are using django time and supervisord log uses server time):

```

2011-11-05 04:42:58,307 INFO exited: celerybeat (exit status 0; expected)

```

celery 2.3.3, django-celery 2.3.3.