Repository: mozilla/telemetry-analysis-service

Issue:
Job kicked off twice yesterday
I got a failure email last night for my txp-install-uninstall-events job but looking at the cluster/job list it looks like it's because the job was queued twice (and both ran at the same time.)


Discussion:
@sunahsuh Yeah, it seems as if the celerybeat isn't behaving correctly, even though it has a distributed lock to prevent duplicated tasks to spawn. I'm investigating this right now as well for stage where the the new scheduling code landed.
I think this is related and possibly a dupe of #353.
I've opened https://github.com/sibson/redbeat/issues/50 to fix this issue, in which the config code of RedBeat incorrectly read the "max_loop_interval" setting due to [a drastic change of setting names in Celery 4.0](http://docs.celeryproject.org/en/latest/whatsnew-4.0.html#v400-upgrade-settings) and essentially returns 0.

The problem is that this value is used for the [timeout of the distributed lock](https://github.com/sibson/redbeat/blob/3f7e97c81411656ed52e2863d4de34983fdd3925/redbeat/schedulers.py#L42) that prevents the beat process to schedule multiple times instead of just once per cluster. Given our way of deployment on AWS we're running the scheduler in multiple containers to not create a bottleneck.
Jezdez++ for mad debugging skills, damn

Resolving patch:
diff --git a/atmo/celery.py b/atmo/celery.py
--- a/atmo/celery.py
+++ b/atmo/celery.py
@@ -8,8 +8,6 @@
 from celery.five import string_t
 from celery.task import current
 from celery.utils.time import maybe_iso8601
-from redbeat.schedulers import add_defaults
-from redis.client import StrictRedis
 
 
 # set the default Django settings module for the 'celery' program.
@@ -96,11 +94,5 @@ def wrapper(*args, **kwargs):
 #   should have a `CELERY_` prefix.
 celery.config_from_object('django.conf:settings', namespace='CELERY')
 
-# Add RedBeat defaults
-add_defaults(celery)
-
-celery.redbeat_redis = StrictRedis.from_url(celery.conf.REDBEAT_REDIS_URL,
-                                            decode_responses=True)
-
 # Load task modules from all registered Django celery configs.
 celery.autodiscover_tasks()
diff --git a/atmo/settings.py b/atmo/settings.py
--- a/atmo/settings.py
+++ b/atmo/settings.py
@@ -55,9 +55,9 @@ class Celery:
     # Maximum time to sleep between re-checking the schedule
     CELERY_BEAT_MAX_LOOP_INTERVAL = 5  # redbeat likes fast loops
     # Unless refreshed the lock will expire after this time
-    REDBEAT_LOCK_TIMEOUT = CELERY_BEAT_MAX_LOOP_INTERVAL * 5
+    CELERY_REDBEAT_LOCK_TIMEOUT = CELERY_BEAT_MAX_LOOP_INTERVAL * 5
     # The default/initial schedule to use.
-    CELERYBEAT_SCHEDULE = CELERY_BEAT_SCHEDULE = {
+    CELERY_BEAT_SCHEDULE = {
         'expire_jobs': {
             'schedule': crontab(minute='*'),
             'task': 'atmo.jobs.tasks.expire_jobs',
@@ -448,7 +448,7 @@ class Base(Core):
         environ_name='REDIS_URL',
     )
     # Use redis as the Celery broker.
-    REDBEAT_REDIS_URL = CELERY_BROKER_URL = os.environ.get('REDIS_URL', REDIS_URL_DEFAULT)
+    CELERY_BROKER_URL = os.environ.get('REDIS_URL', REDIS_URL_DEFAULT)
 
     LOGGING_USE_JSON = values.BooleanValue(False)
 


Test patch: