ask/celery — Issue #625: Celery 2.5.0 with CELERYD_FORCE_EXECV = True fails to start processes in virtualenv
Author: millarm
Created: 2012-02-29 17:05:54+00:00
State: open

We have Celery running with django in a linux environment - we've been experiencing hangs in worker processes that seem to be due to autoscaling the process worker. We run celery like this:

python manage.py celeryd -E --loglevel=INFO --autoscale=50,3

If we don't autoscale (do this)

python manage.py celeryd -E --loglevel=INFO --autoscale=50,50

Everything runs fine with celery 2.4.6 and 2.5.0

If however we enable the FORCE_EXECV flag, things go wrong.

Enabling it on a vanilla install we get this problem:

2012-02-29_16:18:27.20842 [2012-02-29 16:18:27,208: INFO/PoolWorker-31] child process calling self.run()
2012-02-29_16:18:27.40646 [2012-02-29 16:18:27,406: INFO/PoolWorker-30] child process calling self.run()
2012-02-29_16:18:27.61684 [2012-02-29 16:18:27,616: INFO/PoolWorker-34] child process calling self.run()
2012-02-29_16:18:29.25847 [2012-02-29 16:18:29,256: ERROR/MainProcess] Thread 'Supervisor' crashed: AttributeError("'Pool' object has no attribute '_putlock'",)
2012-02-29_16:18:29.25853 Traceback (most recent call last):
2012-02-29_16:18:29.25853 File "/sites/tellybug/shared/webserver/lib/python2.7/site-packages/celery/concurrency/processes/pool.py", line 239, in run
2012-02-29_16:18:29.25854 return self.body()
2012-02-29_16:18:29.25855 File "/sites/tellybug/shared/webserver/lib/python2.7/site-packages/celery/concurrency/processes/pool.py", line 261, in body
2012-02-29_16:18:29.25855 self.pool._maintain_pool()
2012-02-29_16:18:29.25856 File "/sites/tellybug/shared/webserver/lib/python2.7/site-packages/celery/concurrency/processes/pool.py", line 715, in _maintain_pool
2012-02-29_16:18:29.25857 self._join_exited_workers()
2012-02-29_16:18:29.25857 File "/sites/tellybug/shared/webserver/lib/python2.7/site-packages/celery/concurrency/processes/pool.py", line 659, in _join_exited_workers
2012-02-29_16:18:29.25858 if self._putlock is not None:
2012-02-29_16:18:29.25858 AttributeError: 'Pool' object has no attribute '_putlock'
2012-02-29_16:18:29.39297 [2012-02-29 16:18:29,390: WARNING/MainProcess] celery@ip-10-58-243-80 has started.
2012-02-29_16:18:29.51107 [2012-02-29 16:18:29,510: INFO/PoolWorker-37] child process calling self.run()
2012-02-29_16:18:31.13774 [2012-02-29 16:18:31,137: INFO/PoolWorker-42] child process calling self.run()
2012-02-29_16:18:31.16120 [2012-02-29 16:18:31,161: INFO/PoolWorker-39] child process calling self.run()

Which seems to be due to the fact that the processes are shutting down before the pool __init__ has completed (so _putlock is not yet there)

Moving the creation of _put_lock to before the creation of the worker pool:

self._putlock = LaxBoundedSemaphore(self._processes)
for i in range(processes):
self._create_worker_process()

Fixes this - but then we discover why the processes were shutting so early.

It's because almost all the imports are failing.

An example

2012-02-29_16:39:38.86627 Traceback (most recent call last):
2012-02-29_16:39:38.92046 File "[path_to_virtual_env]/lib/python2.7/site.py", line 67, in
2012-02-29_16:39:38.92059 import os
2012-02-29_16:39:38.92068 File "[path_to_virtual_env]/lib/python2.7/os.py", line 398, in
2012-02-29_16:39:38.92095 import UserDict
2012-02-29_16:39:38.92103 File "[path_to_virtual_env]/lib/python2.7/UserDict.py", line 83, in
2012-02-29_16:39:38.92113 import _abcoll
2012-02-29_16:39:38.92142 File "[path_to_virtual_env]/lib/python2.7/_abcoll.py", line 11, in
2012-02-29_16:39:38.92169 from abc import ABCMeta, abstractmethod
2012-02-29_16:39:38.92178 File "[path_to_virtual_env]/lib/python2.7/abc.py", line 8, in
2012-02-29_16:39:38.92187 from _weakrefset import WeakSet
2012-02-29_16:39:38.92294 ImportError: No module named _weakrefset

Why? We run celery within a virtualenv - it looks like the execv call isn't picking up the virtualenv, so our processes have no hope of working.