celery/billiard — pull request #119 by PierreF (2014-08-29 12:24:32+00:00)
Title: Timeout callback could break job timeout

When processing timeout, job could have a timeout callback which is called before sending kill signal. So if that callback raise an Exception, no kill will be sent and job will continue.

This is an issue in some of our processing because the timeout callback (the default of celery, with Django database as backend) raise an exception.
The exception is "MySQL server has gone away". It is due to backend being only rarly used and connection being killed due to too long idle. Django database backend do not recycle the connection and will continue to fail at every database action.
The worse part in our use case, it that further task are no longer processed. From what I've found, it seems to be a semaphor which is not released. I followed the handling of a new events, it got stuck on the _quick_acquire (https://github.com/celery/celery/blob/2fddb876e1b966c3540aadbbc2257dfd493abb85/celery/worker/__init__.py#L226).

The pull request workaround the issue : task continue to be processed (and get killed if took too long), but the timeout callback will no longer write result to database because it will always fail. For the last part, it not a big issue, we do not handle error result for job (and it's probably more a django_celery bug).

I've also created a very basic Django project to exhibit this issue, see https://github.com/PierreF/celery_timeout_testproject