Issue 1:
Checklist

I have verified that the issue exists against the main branch of Celery.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the main branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the main branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Environment & Settings

Celery version: 5.2.7

celery report Output:

import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')

app = Celery('proj')

app.conf.update({

'broker_url': 'redis://localhost:8000/11',

'result_backend': 'django-db',

'accept_content': ['json'],

'task_serializer': 'json',

'result_serializer': 'json',

'worker_force_execv': True,

'worker_concurrency': 10,

'task_time_limit': 60 * 60,

'worker_max_tasks_per_child': 200,

'beat_tz_aware': False,

'result_extended': True,

})

app.conf.timezone = 'Asia/Shanghai'

app.conf.enable_utc = False

app.autodiscover_tasks()

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.8.6

Minimal Celery Version: 5.2.7

Minimal OS and/or Kernel Version: CentOS Linux release 7.9.2009 (Core)

Python Packages

pip freeze Output:

aioredis==1.3.1
amqp==5.1.1
ansible-runner==2.2.0
asgiref==3.5.2
async-timeout==4.0.2
attrs==21.4.0
autobahn==22.5.1
Automat==20.2.0
backports.zoneinfo==0.2.1
billiard==3.6.4.0
celery==5.2.7
certifi==2022.6.15
cffi==1.15.0
channels==3.0.4
channels-redis==3.4.0
charset-normalizer==2.1.0
click==8.1.3
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.2.0
constantly==15.1.0
cron-descriptor==1.3.0
cryptography==37.0.2
cx-Oracle==8.3.0
daphne==3.0.2
Deprecated==1.2.13
Django==4.0.5
django-celery-beat==2.5.0
django-celery-results==2.5.1
django-cors-headers==3.13.0
django-redis==5.2.0
django-simpleui==2023.3.1
django-timezone-field==5.0
djangorestframework==3.14.0
djangorestframework-simplejwt==5.2.2
docopt==0.6.2
docutils==0.18.1
flower==1.2.0
hiredis==2.0.0
humanize==4.6.0
hyperlink==21.0.0
idna==3.3
incremental==21.3.0
kombu==5.2.4
ldap3==2.9.1
lockfile==0.12.2
msgpack==1.0.4
mysqlclient==2.1.0
packaging==21.3
pexpect==4.8.0
pipreqs==0.4.11
prometheus-client==0.16.0
prompt-toolkit==3.0.29
ptyprocess==0.7.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.21
PyJWT==2.6.0
pyOpenSSL==22.0.0
pyparsing==3.0.9
python-crontab==2.6.0
python-daemon==2.3.0
python-dateutil==2.8.2
pytz==2022.1
PyYAML==6.0
redis==4.3.3
requests==2.28.1
service-identity==21.1.0
six==1.16.0
sqlparse==0.4.2
supervisor==4.2.4
tornado==6.3.1
Twisted==22.4.0
txaio==22.2.1
typing_extensions==4.2.0
tzdata==2022.2
urllib3==1.26.9
vine==5.0.0
wcwidth==0.2.5
wrapt==1.14.1
yapf==0.32.0
yarg==0.1.9
zope.interface==5.4.0

Steps to reproduce

Start Celery with the following configuration:

#proj/celery.py
import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')

app = Celery('proj')
app.conf.update({
'broker_url': 'redis://localhost:8000/11',
'result_backend': 'django-db',
'accept_content': ['json'],
'task_serializer': 'json',
'result_serializer': 'json',
'worker_force_execv': True,
'worker_concurrency': 10,
'task_time_limit': 60 * 60,
'worker_max_tasks_per_child': 200,
'beat_tz_aware': False,
'result_extended': True,
})
app.conf.timezone = 'Asia/Shanghai'
app.conf.enable_utc = False
app.autodiscover_tasks()

Add a task to the app:

# myapp/tasks.py
from proj.celery import app
@app.task
def add():
print('aa')

Attempt to call the task with the following code:

# myapp/views.py
def task_page(request):
from dbaOps.celery import app
print(app.tasks.keys())
return render(request, 'myapp/tasks.html')

Expected behavior

The task myapp.tasks.add should be listed in the output of app.tasks.keys().

Actual Behavior

The task myapp.tasks.add does not appear in the output of app.tasks.keys(). However, when I modify the import statement in views.py to from myapp.tasks import app, the task does appear in the list.

# myapp/views.py
def task_page(request):
from myapp.tasks import app
print(app.tasks.keys())
return render(request, 'myapp/tasks.html')

I would appreciate any help in understanding why this is happening and how to resolve it. Thank you!

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Issue 2:
There's an issue in the implementation of celery.schedules.crontab method that can cause crontab.is_due to claim a schedule is due at a time that is completely unrelated to the given crontab parameters. This appears to happen in cases where the last_run_at value is older than the most recent feasible time the schedule could have run. It looks like that this issue was introduced as part of complex logic added nearly 8 years ago to improve the accuracy of time remaining estimates for crontab periodic tasks: 4ed89ec

Checklist

Issue first observed in a celery deployment running celery v3.1.19

From inspecting the celery.schedules.crontab code it appears likely that this issue is present in all celery versions as far back as v2.1.0 through to v4.1.1 .

Unit tests (please see below) confirm issue is still present in master (b599b96 at time of writing).

Steps to reproduce

Please apply this patch to celery master branch, remove @skip.todo from the first new test, run unit tests, observe the first of these added unit tests fails:

diff --git a/t/unit/app/test_schedules.py b/t/unit/app/test_schedules.py
index a7b3025..0340461 100644
--- a/t/unit/app/test_schedules.py
+++ b/t/unit/app/test_schedules.py
@@ -26,6 +26,18 @@ def patch_crontab_nowfun(cls, retval):
cls.nowfun = prev_nowfun

+def is_time_feasible_wrt_crontab_schedule(t, z):
+ # z : celery.schedules.crontab instance
+ t = z.maybe_make_aware(t)
+ return (
+ t.month in z.month_of_year and
+ (t.isoweekday() % 7) in z.day_of_week and
+ t.day in z.day_of_month and
+ t.hour in z.hour and
+ t.minute in z.minute
+ )
+
+
@skip.unless_module('ephem')
class test_solar:

@@ -803,3 +815,59 @@ class test_crontab_is_due:
due, remaining = self.yearly.is_due(datetime(2009, 3, 12, 7, 30))
assert not due
assert remaining == 4 * 24 * 60 * 60 - 3 * 60 * 60
+
+ @skip.todo('FIXME crontab logic is defective when last_run_at is older than the most recent feasible time wrt schedule')
+ def test_daily_execution_if_last_run_at_was_days_ago_and_current_time_does_not_match_crontab_schedule_then_execution_is_not_due(self):
+ z = self.crontab(hour=7, minute=30)
+ last_run_at = datetime(2018, 6, 1, 7, 30)
+ now = datetime(2018, 6, 9, 23, 48)
+ expected_next_execution_time = datetime(2018, 6, 10, 7, 30)
+ expected_remaining = (expected_next_execution_time - now).total_seconds()
+ # check our assumptions
+ assert is_time_feasible_wrt_crontab_schedule(last_run_at, z)
+ assert not is_time_feasible_wrt_crontab_schedule(now, z)
+ assert is_time_feasible_wrt_crontab_schedule(expected_next_execution_time, z)
+ assert now < expected_next_execution_time
+ assert expected_remaining == (7 * 60 + 30 + 12) * 60
+ # test is_due
+ with patch_crontab_nowfun(z, now):
+ due, remaining = z.is_due(last_run_at=last_run_at)
+ assert remaining == expected_remaining
+ assert not due
+
+ def test_daily_execution_if_last_run_at_was_the_most_recent_feasible_time_wrt_schedule_in_past_and_current_time_does_not_match_crontab_schedule_then_execution_is_not_due(self):
+ z = self.crontab(hour=7, minute=30)
+ last_run_at = datetime(2018, 6, 9, 7, 30)
+ now = datetime(2018, 6, 9, 23, 48)
+ expected_next_execution_time = datetime(2018, 6, 10, 7, 30)
+ expected_remaining = (expected_next_execution_time - now).total_seconds()
+ # check our assumptions
+ assert is_time_feasible_wrt_crontab_schedule(last_run_at, z)
+ assert not is_time_feasible_wrt_crontab_schedule(now, z)
+ assert is_time_feasible_wrt_crontab_schedule(expected_next_execution_time, z)
+ assert now < expected_next_execution_time
+ assert expected_remaining == (7 * 60 + 30 + 12) * 60
+ # test is_due
+ with patch_crontab_nowfun(z, now):
+ due, remaining = z.is_due(last_run_at=last_run_at)
+ assert remaining == expected_remaining
+ assert not due
+
+ def test_daily_execution_if_last_run_at_was_more_recent_than_the_most_recent_feasible_time_wrt_schedule_in_past_and_current_time_does_not_match_crontab_schedule_then_execution_is_not_due(self):
+ z = self.crontab(hour=7, minute=30)
+ last_run_at = datetime(2018, 6, 9, 10, 30) # not feasible wrt to current schedule. case can happen if schedule is modified after a run
+ now = datetime(2018, 6, 9, 23, 48)
+ expected_next_execution_time = datetime(2018, 6, 10, 7, 30)
+ expected_remaining = (expected_next_execution_time - now).total_seconds()
+ # check our assumptions
+ assert not is_time_feasible_wrt_crontab_schedule(last_run_at, z)
+ assert not is_time_feasible_wrt_crontab_schedule(now, z)
+ assert is_time_feasible_wrt_crontab_schedule(expected_next_execution_time, z)
+ assert now < expected_next_execution_time
+ assert expected_remaining == (7 * 60 + 30 + 12) * 60
+ # test is_due
+ with patch_crontab_nowfun(z, now):
+ due, remaining = z.is_due(last_run_at=last_run_at)
+ assert remaining == expected_remaining
+ assert not due
+

Expected behavior

No matter what the value of last_run_at is, crontab.is_due(last_run_at) should never return schedstate(True, rem) when the current time now is not feasible with respect to the given crontab parameters.

Actual behavior

if last_run_at value is older than the most recent time that is feasible with respect to the given crontab parameters then crontab.is_due(last_run_at) returns schedstate(True, rem) when the current time is not feasible with respect to the crontab parameters.

Comments

This behaviour is surprising as it is an undocumented departure from cron-like behaviour. This behaviour is somewhat like an undocumented variant of uncontrollable anacron behaviour. "Uncontrollable" in the sense that unlike anacrontab's START_HOURS_RANGE parameter - there is no control at all over when tasks will be run when a scheduled execution is missed.

We experience this issue in production using the venerable Celery version of 3.1.19 : we have a celery beat process that is backed by a database using custom scheduler code that is derived from the django-celery-beat scheduler. Suppose we stop this celery beat process for some reason such as scheduled maintenance or during a deployment that needs to redeploy code to our celery cluster. When we later restart celery beat again then some or all celery.schedules.crontab scheduled tasks may immediately execute if there was a time during the celery beat downtime period that matches the crontab parameters.

One hack that can mitigate this behaviour is updating the "last_run_at" value for all celery crontab scheduled periodic tasks to the current time when celery beat starts, before celery beat makes any calls to the celery.schedules.crontab.is_due method.