Issue 1:
Celery 4.0.0

Python 3.5.2

(rest of report output below)

I recently pushed celery beat to production (I had been using celery without beat with no issues for several months). It had 1 task to run several times daily, but it duplicated the task many times per second, gradually overwhelming the capacity of my small elasticache instance and resulting in OOM error.

I was searching for similar errors online and found the following:

#943 (comment)

While not exactly the same, changing timezone back to default settings seemed to solve the issue for now. To be clear, previously I had:

enable_utc = False

timezone = 'America/New_York'

I changed these to:

enable_utc = True

and it seemed to solve the problem (for now).

** celery -A [proj] report:

software -> celery:4.0.0 (latentcall) kombu:4.0.0 py:3.5.2

billiard:3.5.0.2 redis:2.10.5

platform -> system:Linux arch:64bit, ELF imp:CPython

loader -> celery.loaders.app.AppLoader

settings -> transport:redis results:redis://[elasticache]

result_persistent: False

enable_utc: False

result_serializer: 'json'

include: [tasks -- redacted]

result_backend: 'redis://[elasticache]

task_create_missing_queues: True

task_acks_late: False

timezone: 'America/New_York'

broker_url: '[elasticache]'

task_serializer: 'json'

crontab: <class 'celery.schedules.crontab'>

broker_transport_options: {

'visibility_timeout': 600}

task_time_limit: 200

task_always_eager: False

task_queues:

(<unbound Queue default -> <unbound Exchange default(direct)> -> default>,

<unbound Queue render -> <unbound Exchange media(direct)> -> media.render>,

<unbound Queue emails -> <unbound Exchange media(direct)> -> media.emails>,

<unbound Queue other -> <unbound Exchange media(direct)> -> media.other>)

beat_schedule: {

'lookup-emails-to-send': { 'args': [],

'schedule': <crontab: 0 9,12,15,18,21 * * * (m/h/d/dM/MY)>,

'task': 'send_emails'}}

accept_content: ['json']

task_default_exchange_type: 'direct'

task_routes:

('worker.routes.AppRouter',)

worker_prefetch_multiplier: 1

BROKER_URL: [elasticache]

** Worker error message after OOM:

Traceback (most recent call last):

File "python3.5/site-packages/celery/beat.py", line 299, in apply_async

**entry.options)

File "python3.5/site-packages/celery/app/task.py", line 536, in apply_async

**options

File "python3.5/site-packages/celery/app/base.py", line 717, in send_task

amqp.send_task_message(P, name, message, **options)

File "python3.5/site-packages/celery/app/amqp.py", line 554, in send_task_message

**properties

File "python3.5/site-packages/kombu/messaging.py", line 178, in publish

exchange_name, declare,

File "python3.5/site-packages/kombu/connection.py", line 527, in _ensured

errback and errback(exc, 0)

File "python3.5/contextlib.py", line 77, in exit

self.gen.throw(type, value, traceback)

File "python3.5/site-packages/kombu/connection.py", line 419, in _reraise_as_library_errors

sys.exc_info()[2])

File "python3.5/site-packages/vine/five.py", line 175, in reraise

raise value.with_traceback(tb)

File "python3.5/site-packages/kombu/connection.py", line 414, in _reraise_as_library_errors

yield

File "python3.5/site-packages/kombu/connection.py", line 494, in _ensured

return fun(*args, **kwargs)

File "python3.5/site-packages/kombu/messaging.py", line 200, in _publish

mandatory=mandatory, immediate=immediate,

File "python3.5/site-packages/kombu/transport/virtual/base.py", line 608, in basic_publish

return self._put(routing_key, message, **kwargs)

File "python3.5/site-packages/kombu/transport/redis.py", line 766, in _put

client.lpush(self._q_for_pri(queue, pri), dumps(message))

File "python3.5/site-packages/redis/client.py", line 1227, in lpush

return self.execute_command('LPUSH', name, *values)

File "python3.5/site-packages/redis/client.py", line 573, in execute_command

return self.parse_response(connection, command_name, **options)

File "python3.5/site-packages/redis/client.py", line 585, in parse_response

response = connection.read_response()

File "python3.5/site-packages/redis/connection.py", line 582, in read_response

raise response

kombu.exceptions.OperationalError: OOM command not allowed when used memory > 'maxmemory'.

** Example log showing rapidly duplicated beat tasks:

[2017-05-20 09:00:00,096: INFO/PoolWorker-1] Task send_emails[1d24250f-c254-4fd8-8ffc-cfb17b98a392] succeeded in 0.01704882364720106s: True

[2017-05-20 09:00:00,098: INFO/MainProcess] Received task: send_emails[e968cc65-0924-4d31-b2d7-9a3e3f5aefda]

[2017-05-20 09:00:00,115: INFO/PoolWorker-1] Task send_emails[e968cc65-0823-4d31-b2d7-9a3e3f5aefda] succeeded in 0.015948095358908176s: True

[2017-05-20 09:00:00,117: INFO/MainProcess] Received task: send_emails[2fb9f7ac-5f6a-42e1-813e-25ea4023dc81]

[2017-05-20 09:00:00,136: INFO/PoolWorker-1] Task send_emails[2fb9f7ac-5f6a-42e1-813e-25ea4023dc81] succeeded in 0.018534105271100998s: True

[2017-05-20 09:00:00,140: INFO/MainProcess] Received task: send_emails[59493c0b-1858-497f-a0dc-a4c8e4ba3a63]

[2017-05-20 09:00:00,161: INFO/PoolWorker-1] Task send_emails[59493c0b-1858-497f-a0dc-a4c8e4ba3a63] succeeded in 0.020539570599794388s: True

[2017-05-20 09:00:00,164: INFO/MainProcess] Received task: send_emails[3aac612f-75dd-4530-9b55-1e288bec8db4]

[2017-05-20 09:00:00,205: INFO/PoolWorker-1] Task send_emails[3aac612f-75dd-4530-9b55-1e288bec8db4] succeeded in 0.04063933063298464s: True

[2017-05-20 09:00:00,208: INFO/MainProcess] Received task: send_emails[7edf60cb-d1c4-4ae5-af10-03414da672fa]

Issue 2:
I happened to install celery 4.1.0 with django_celey_beat 1.0.1, the DatabaseScheduler seems not working well.

[2017-08-07 21:12:10,790: DEBUG/MainProcess] DatabaseScheduler: Fetching database schedule

[2017-08-07 21:12:10,797: DEBUG/MainProcess] Current schedule:

<ModelEntry: celery.backend_cleanup celery.backend_cleanup([], **{}) <crontab: 0 4 * * * (m/h/d/dM/MY)>>

<ModelEntry: schedule GeneBank.tasks.test([], **{}) <crontab: * * * * * (m/h/d/dM/MY)>>

[2017-08-07 21:12:10,807: DEBUG/MainProcess] beat: Ticking with max interval->5.00 seconds

[2017-08-07 21:12:10,809: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.

[2017-08-07 21:12:15,813: DEBUG/MainProcess] beat: Synchronizing schedule...

[2017-08-07 21:12:15,813: INFO/MainProcess] Writing entries...

[2017-08-07 21:12:15,816: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.

[2017-08-07 21:12:20,818: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.

[2017-08-07 21:12:25,825: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.

[2017-08-07 21:12:30,831: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.

[2017-08-07 21:12:35,839: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.

[2017-08-07 21:12:40,844: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.

[2017-08-07 21:12:45,851: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.

[2017-08-07 21:12:50,854: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.

[2017-08-07 21:12:55,860: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.

[2017-08-07 21:13:00,862: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.

[2017-08-07 21:13:05,870: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.

^C[2017-08-07 21:13:10,245: INFO/MainProcess] Writing entries...

[2017-08-07 21:13:10,246: INFO/MainProcess] Writing entries...

As u can see, the scheduler was supposed to send beat at every minute, but the beat didn't show up, (i've set crontab to be all *, so couldn't be timezone problem)

But in celery 4.0.2, everything goes fine! I don't know whether it is a bug or not. Maybe the django_celery_beat isn't compatible with 4.1.0.

[2017-08-07 21:18:43,339: DEBUG/MainProcess] Current schedule:

<ModelEntry: celery.backend_cleanup celery.backend_cleanup([], **{}) <crontab: 0 4 * * * (m/h/d/dM/MY)>>

<ModelEntry: schedule GeneBank.tasks.test([], **{}) <crontab: * * * * * (m/h/d/dM/MY)>>

[2017-08-07 21:18:43,351: DEBUG/MainProcess] beat: Ticking with max interval->5.00 seconds

[2017-08-07 21:18:43,364: INFO/MainProcess] Scheduler: Sending due task schedule (GeneBank.tasks.test)

[2017-08-07 21:18:43,376: DEBUG/MainProcess] beat: Synchronizing schedule...

[2017-08-07 21:18:43,376: INFO/MainProcess] Writing entries...

[2017-08-07 21:18:43,380: DEBUG/MainProcess] GeneBank.tasks.test sent. id->9c1bdf10-0a5f-440a-98db-9eb24433a8d4

[2017-08-07 21:18:43,381: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.

[2017-08-07 21:18:48,386: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.

[2017-08-07 21:18:53,392: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.

[2017-08-07 21:18:58,397: DEBUG/MainProcess] beat: Waking up in 1.59 seconds.

[2017-08-07 21:19:00,001: INFO/MainProcess] Scheduler: Sending due task schedule (GeneBank.tasks.test)