ask/celery — Issue #507: Bad ETA precision with 2.3.3 and default pool
Author: 23doors
Created: 2011-10-14 14:20:21+00:00
State: open

I haven't checked the situation with previous versions, but it doesn't seem like an expected behaviour.
I run a simple test to check celery precision.

Here is the test code. First part in my case is included in a PeriodicTask run method.

logger = logging.getLogger(__name__)

start_date = datetime.now() + timedelta(minutes=1)

for x in range(15):
schedule_time = start_date + timedelta(seconds=x)
logger.info('Scheduling task at %s', schedule_time)
for i in range(10):
SchedulerProcessBlockTask.apply_async(kwargs={'block': (schedule_time,x,i)}, eta=schedule_time)

class SchedulerProcessBlockTask(Task):
def run(self, block, **kwargs):
task_id = kwargs.pop('task_id')
extra = {'task_id': task_id}
now = datetime.now()
precision = now - block[0]
logger.info('Running task at %s with : %s - precision: %s', datetime.now(), block, precision)

And some settings I use:
CELERY_ACKS_LATE = True
CELERY_TASK_PUBLISH_RETRY = True
CELERY_DISABLE_RATE_LIMITS = True
CELERY_IGNORE_RESULT = True
CELERY_SEND_TASK_ERROR_EMAILS = not DEBUG
CELERYD_HIJACK_ROOT_LOGGER = False
CELERYD_PREFETCH_MULTIPLIER = 1
CELERYD_MAX_TASKS_PER_CHILD = 1000
CELERYD_STATE_DB = 'celery_state.db'

CELERYD_ETA_SCHEDULER_PRECISION value doesnt change the results (tried 0.1). Same for CELERYD_PREFETCH_MULTIPLIER. It is not a long running task so CELERYD_MAX_TASKS_PER_CHILD are also probably not relevant.
I am using celery in conjunction with Django (django-celery) but I am not using DatabaseScheduler class.

Increasing concurrency, no effect.

With eventlet pool (default concurrency) I get at most 0.015 precision. Which is good, very good.
With gevent pool I get at most

Now, with default pool, crazy things happen.
Around 30-50% of tasks get > 1 second precision.
There is nothing in between - no tasks running 0.5 seconds or something like that. It's either 0-0.020 precision or 1-1.020.

What the hell? I cant believe it is normal. It seems like it is sleeping 1 second too long for some tasks before dispatching them. Though CELERYD_ETA_SCHEDULER_PRECISION would be the answer. Well, it's not.