Issue 1:
Checklist

I have verified that the issue exists against the main branch of Celery.

This has already been asked to the discussions forum first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the main branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the main branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

it was first reported in #7255

Possible Duplicates

None

Environment & Settings

Celery version: 5.2.7

software -> celery:5.2.7 (dawn-chorus) kombu:5.2.4 py:3.10.10

billiard:3.6.4.0 py-amqp:5.1.1

platform -> system:Linux arch:64bit

kernel version:4.19.24-7.34.cbp.al7.x86_64 imp:CPython

loader -> celery.loaders.default.Loader

settings -> transport:amqp results:disabled

deprecated_settings: None

celery report Output:

software -> celery:5.2.7 (dawn-chorus) kombu:5.2.4 py:3.10.10
billiard:3.6.4.0 redis:3.5.3
platform -> system:Linux arch:64bit
kernel version:4.19.24-7.34.cbp.al7.x86_64 imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:disabled

deprecated_settings: None
CELERY_BROKER_URL: 'redis://:********@dongtai-redis:6379/0'
CELERY_RESULT_EXPIRES: 600
CELERY_WORKER_TASK_LOG_FORMAT: '%(message)s'
CELERY_WORKER_LOG_FORMAT: '%(message)s'
CELERY_TASK_EAGER_PROPAGATES: True
CELERY_WORKER_REDIRECT_STDOUTS: True
CELERY_WORKER_REDIRECT_STDOUTS_LEVEL: 'ERROR'
CELERY_WORKER_MAX_TASKS_PER_CHILD: 5000
CELERY_TASK_SOFT_TIME_LIMIT: 3600
CELERY_TASK_REJECT_ON_WORKER_LOST: True
task_queues: [<unbound Queue dongtai-method-pool-scan -> <unbound Exchange dongtai-method-pool-scan(direct)> -> dongtai-method-pool-scan>,
<unbound Queue dongtai-replay-vul-scan -> <unbound Exchange dongtai-replay-vul-scan(direct)> -> dongtai-replay-vul-scan>,
<unbound Queue dongtai-sca-task -> <unbound Exchange dongtai-sca-task(direct)> -> dongtai-sca-task>,
<unbound Queue dongtai-function-flush-data -> <unbound Exchange dongtai-function-flush-data(direct)> -> dongtai-function-flush-data>,
<unbound Queue dongtai-es-save-task -> <unbound Exchange dongtai-es-save-task(direct)> -> dongtai-es-save-task>,
<unbound Queue dongtai-method-pool-try-to-bind-method-pool -> <unbound Exchange dongtai-method-pool-try-to-bind-method-pool(direct)> -> dongtai-method-pool-try-to-bind-method-pool>,
<unbound Queue dongtai-method-pool-judge_propagator -> <unbound Exchange dongtai-method-pool-judge_propagator(direct)> -> dongtai-method-pool-judge_propagator>,
<unbound Queue judge-and-append-stack-generic -> <unbound Exchange judge-and-append-stack-generic(direct)> -> judge-and-append-stack-generic>,
<unbound Queue dongtai-method-pool-update-chains -> <unbound Exchange dongtai-method-pool-update-chains(direct)> -> dongtai-method-pool-update-chains>,
<unbound Queue dongtai-export-report-task -> <unbound Exchange dongtai-export-report-task(direct)> -> dongtai-export-report-task>,
<unbound Queue dongtai-vul-notify-task -> <unbound Exchange dongtai-vul-notify-task(direct)> -> dongtai-vul-notify-task>,
<unbound Queue dongtai-vul-webhook-task -> <unbound Exchange dongtai-vul-webhook-task(direct)> -> dongtai-vul-webhook-task>,
<unbound Queue dongtai-asset-vul-notify-task -> <unbound Exchange dongtai-asset-vul-notify-task(direct)> -> dongtai-asset-vul-notify-task>,
<unbound Queue dongtai-method-pool-recheck -> <unbound Exchange dongtai-method-pool-recheck(direct)> -> dongtai-method-pool-recheck>,
<unbound Queue dongtai-periodic-task -> <unbound Exchange dongtai-periodic-task(direct)> -> dongtai-periodic-task>]
task_serializer: 'json'
result_serializer: 'json'
accept_content: ['json']
task_ignore_result: True
task_acks_late: True
task_acks_on_failure_or_timeout: True
broker_connection_retry_on_startup: False
broker_connection_max_retries: 0
broker_pool_limit: 1000
task_routes: {
'dongtai_common.utils.es.handle_batch_save': { 'exchange': 'dongtai-es-save-task',
'routing_key': '********'},
'dongtai_common.utils.es.handle_save': { 'exchange': 'dongtai-es-save-task',
'routing_key': '********'},
'dongtai_conf.plugin.plug_request_chains.try_to_bind_method_pool': { 'exchange': 'dongtai-method-pool-try-to-bind-method-pool',
'routing_key': '********'},
'dongtai_conf.plugin.plug_request_chains.update_chains': { 'exchange': 'dongtai-method-pool-update-chains',
'routing_key': '********'},
'dongtai_engine.chain_propagator_tasks.judge_propagator': { 'exchange': 'dongtai-method-pool-judge_propagator',
'routing_key': '********'},
'dongtai_engine.chain_tasks.judge_and_append_stack_generic': { 'exchange': 'judge-and-append-stack-generic',
'routing_key': '********'},
'dongtai_engine.elatic_search.data_correction': { 'exchange': 'dongtai-es-save-task',
'routing_key': '********'},
'dongtai_engine.plugins.data_clean': { 'exchange': 'dongtai-periodic-task',
'routing_key': '********'},
'dongtai_engine.preheat.function_flush': { 'exchange': 'dongtai-function-flush-data',
'routing_key': '********'},
'dongtai_engine.preheat.function_preheat': { 'exchange': 'dongtai-periodic-task',
'routing_key': '********'},
'dongtai_engine.signals.handlers.notify_controler.consume_notify_list': { 'exchange': 'dongtai-periodic-task',
'routing_key': '********'},
'dongtai_engine.signals.handlers.notify_controler.send_asset_vul_notify': { 'exchange': 'dongtai-asset-vul-notify-task',
'routing_key': '********'},
'dongtai_engine.signals.handlers.notify_controler.send_vul_notify': { 'exchange': 'dongtai-vul-notify-task',
'routing_key': '********'},
'dongtai_engine.signals.handlers.notify_controler.web_hook_app_task': { 'exchange': 'dongtai-vul-webhook-task',
'routing_key': '********'},
'dongtai_engine.tasks.clear_error_log': { 'exchange': 'dongtai-periodic-task',
'routing_key': '********'},
'dongtai_engine.tasks.export_report': { 'exchange': 'dongtai-export-report-task',
'routing_key': '********'},
'dongtai_engine.tasks.heartbeat': { 'exchange': 'dongtai-periodic-task',
'routing_key': '********'},
'dongtai_engine.tasks.search_vul_from_method_pool': { 'exchange': 'dongtai-method-pool-scan',
'routing_key': '********'},
'dongtai_engine.tasks.search_vul_from_replay_method_pool': { 'exchange': 'dongtai-replay-vul-scan',
'routing_key': '********'},
'dongtai_engine.tasks.update_agent_status': { 'exchange': 'dongtai-periodic-task',
'routing_key': '********'},
'dongtai_engine.tasks.vul_recheck': { 'exchange': 'dongtai-periodic-task',
'routing_key': '********'},
'dongtai_web.dongtai_sca.scan.utils.update_one_sca': { 'exchange': 'dongtai-sca-task',
'routing_key': '********'},
'dongtai_web.views.new_recheck.recheck': { 'exchange': 'dongtai-method-pool-recheck',
'routing_key': '********'}}
CELERY_ENABLE_UTC: False
timezone: 'Asia/Shanghai'
DJANGO_CELERY_BEAT_TZ_AWARE: False
CELERY_BEAT_SCHEDULER: 'django_celery_beat.schedulers:DatabaseScheduler'

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.10.10

Minimal Celery Version: 5.2.7

Minimal Kombu Version: 5.2.4

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

aliyun-python-sdk-core==2.13.36
aliyun-python-sdk-core-v3==2.13.33
aliyun-python-sdk-kms==2.16.0
amqp==5.1.1
asgiref==3.6.0
asyncio-gevent==0.2.1
attrs==22.2.0
billiard==3.6.4.0
boto3==1.24.59
boto3-stubs==1.26.90
botocore==1.27.91
botocore-stubs==1.29.90
celery==5.2.7
certifi==2022.12.7
cffi==1.15.1
chardet==4.0.0
click==8.1.3
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.2.0
crcmod==1.7
cryptography==39.0.1
ddt==1.6.0
defusedxml==0.7.1
diff-match-patch==20200713
distlib==0.3.6
Django==3.2.18
django-celery-beat==2.2.0
django-cors-headers==3.14.0
django-cprofile-middleware==1.0.5
django-elasticsearch-dsl==7.2.2
django-filter==22.1
django-import-export==2.5.0
django-mock-queries==2.1.7
django-modeltranslation==0.17.7
django-ranged-response==0.2.0
django-redis==5.2.0
django-rest-framework-proxy==1.6.0
django-simple-captcha==0.5.17
django-stubs==1.15.0
django-stubs-ext==0.7.0
django-timezone-field==4.2.3
django-utils==0.0.2
django-utils-six==2.0
django-xff==1.3.0
djangorestframework==3.12.4
djangorestframework-stubs==1.9.1
docxcompose==1.3.4
docxtpl==0.16.0
drf-spectacular==0.22.1
elasticsearch==7.17.7
elasticsearch-dsl==7.4.1
et-xmlfile==1.1.0
filelock==3.10.0
gevent==21.12.0
greenlet==1.1.3.post0
id-validator==1.0.20
idna==2.10
inflection==0.5.1
Jinja2==3.1.2
jmespath==0.10.0
jq==1.3.0
jsonlog==4.0.0
jsonschema==4.17.0
kombu==5.2.4
lxml==4.9.1
MarkupPy==1.14
MarkupSafe==2.1.2
mock==5.0.1
model-bakery==1.10.1
mypy==1.0.1
mypy-extensions==1.0.0
mysqlclient==2.1.1
numpy==1.24.2
odfpy==1.4.1
openpyxl==3.0.9
oss2==2.13.1
packaging==21.3
pep8==1.7.1
Pillow==9.3.0
pipenv==2023.2.18
platformdirs==3.1.1
prompt-toolkit==3.0.38
pycparser==2.21
pycryptodome==3.17
pycryptodomex==3.14.1
PyMySQL==1.0.2
pyparsing==3.0.9
pyre2==0.3.6
pyrsistent==0.19.1
python-crontab==2.7.1
python-dateutil==2.8.2
python-docx==0.8.11
python-json-logger==2.0.7
pytz==2022.6
PyYAML==6.0
redis==3.5.3
requests==2.25.1
result==0.8.0
s3transfer==0.6.0
simhash==2.1.2
six==1.15.0
sqlparse==0.4.3
tablib==3.3.0
tomli==2.0.1
types-awscrt==0.16.12
types-PyMySQL==1.0.19.5
types-pyOpenSSL==23.0.0.4
types-pytz==2022.7.1.2
types-PyYAML==6.0.12.8
types-redis==4.5.1.5
types-requests==2.28.11.15
types-s3transfer==0.6.0.post5
types-urllib3==1.26.25.8
typing_extensions==4.5.0
uritemplate==4.1.1
urllib3==1.26.5
uWSGI==2.0.21
uwsgitop==0.11
vine==5.0.0
virtualenv==20.21.0
virtualenv-clone==0.5.7
wcwidth==0.2.6
xlrd==2.0.1
xlwt==1.3.0
zope.event==4.6
zope.interface==5.5.2

Other Dependencies

N/A

Minimally Reproducible Test Case

function.apply_async(expires=datetime.now() +timedelta(hours=3),)

then use

self.retry(exc=e, countdown=delay)

Expected Behavior

Perform retries without error.

Actual Behavior

2023-03-16T17:21:20.663973

<class 'str'>

2023-03-16T17:21:20.663973

<class 'str'>

Task dongtai_engine.tasks.search_vul_from_method_pool[8df36afe-850e-4698-ab3c-cd59900b3381] reject requeue=False: '<' not supported between instances of 'str' and 'int'

Traceback (most recent call last):

File "/opt/dongtai/dongtai_engine/tasks.py", line 242, in search_vul_from_method_pool

raise RetryableException(f'漏洞检测方法池 {method_pool_sign} 不存在，重试第 {tries} 次')

dongtai_engine.tasks.RetryableException: 漏洞检测方法池 832a71cc504f47ad9c7ebccff2d838ea 不存在，重试第 1 次

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/usr/local/lib/python3.10/site-packages/celery/app/task.py", line 734, in retry

S.apply_async()

File "/usr/local/lib/python3.10/site-packages/celery/canvas.py", line 219, in apply_async

return _apply(args, kwargs, **options)

File "/usr/local/lib/python3.10/site-packages/celery/app/task.py", line 575, in apply_async

return app.send_task(

File "/usr/local/lib/python3.10/site-packages/celery/app/base.py", line 744, in send_task

if expires_s < 0:

TypeError: '<' not supported between instances of 'str' and 'int'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 451, in trace_task

R = retval = fun(*args, **kwargs)

File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 734, in protected_call

return self.run(*args, **kwargs)

File "/opt/dongtai/dongtai_engine/tasks.py", line 272, in search_vul_from_method_pool

self.retry(exc=e, countdown=delay)

File "/usr/local/lib/python3.10/site-packages/celery/app/task.py", line 736, in retry

raise Reject(exc, requeue=False)

celery.exceptions.Reject: (TypeError("'<' not supported between instances of 'str' and 'int'"), False)

Issue 2:
I'm wondering if we can ditch the prefork pool and use subprocesses that communicate using pipes. Possibly even stdin/stdout.

Would that resolve a lot of the limitations of preforking such as running threads inside the master celery process and running other subprocesses?

What are the limitations of such design?