Issue 1:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version: 5.0.5 (singularity)

celery report Outputsoftware -> celery:5.0.5 (singularity) kombu:5.0.2 py:3.7.7
billiard:3.6.3.0 py-amqp:5.0.2
platform -> system:Darwin arch:64bit
kernel version:20.3.0 imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:amqp results:******

ABSOLUTE_URL_OVERRIDES: {

}

ADMINS: []

ALLOWED_HOSTS: ['']

APPEND_SLASH: False

AUTHENTICATION_BACKENDS: {'auth.auth0backend.Auth0',

'django.contrib.auth.backends.ModelBackend'}

AUTH_PASSWORD_VALIDATORS: '********'

AUTH_USER_MODEL: 'users.CustomUser'

BASE_DIR: ''

BROKER_ADDRESS: None

BROKER_TYPE: None

CACHES: {

'default': {'BACKEND': 'django.core.cache.backends.dummy.DummyCache'}}

CACHE_MIDDLEWARE_ALIAS: 'default'

CACHE_MIDDLEWARE_KEY_PREFIX: ''

CACHE_MIDDLEWARE_SECONDS: 600

CACHE_TIMEOUT: 345600

CELERY_ACCEPT_CONTENT: ['pickle', 'json']

CELERY_BROKER_TRANSPORT_OPTIONS: {

'visibility_timeout': 172800}

CELERY_IMPORTS: ['.orchestrator.tasks',

'.positioning_pipeline.pipeline_tasks',

'.positioning_pipeline.tasks_consumers',

'.services_settings.celery_commands']

CELERY_RESULT_BACKEND: ''

CELERY_RESULT_CHORD_RETRY_INTERVAL: 0.5

CELERY_RESULT_EXTENDED: True

CELERY_RESULT_SERIALIZER: 'json'

CELERY_TASK_QUEUES: [<unbound Queue celery -> <unbound Exchange ''(direct)> -> >,

<unbound Queue renderer_tasks -> <unbound Exchange ''(direct)> -> >,

<unbound Queue small_tasks -> <unbound Exchange ''(direct)> -> >,

<unbound Queue medium_tasks -> <unbound Exchange ''(direct)> -> >,

<unbound Queue big_tasks -> <unbound Exchange ''(direct)> -> >,

<unbound Queue users_tasks -> <unbound Exchange ''(direct)> -> >]

CELERY_TASK_ROUTES: [<bound method Router.route_task of <class '.services_settings.celery_routing.Router'>>]

CELERY_TASK_SERIALIZER: 'pickle'

CELERY_TASK_TRACK_STARTED: True

CELERY_WORKER_CONCURRENCY: 1

CELERY_WORKER_LOG_FORMAT: '%(message)s'

CELERY_WORKER_MAX_MEMORY_PER_CHILD: 3145728

CELERY_WORKER_PREFETCH_MULTIPLIER: 1

CORS_ALLOW_CREDENTIALS: True

CORS_ORIGIN_ALLOW_ALL: True

CORS_URLS_REGEX: '/api/./.-image/.'

CSRF_COOKIE_AGE: 31449600

CSRF_COOKIE_DOMAIN: None

CSRF_COOKIE_HTTPONLY: False

CSRF_COOKIE_NAME: 'csrftoken'

CSRF_COOKIE_PATH: '/'

CSRF_COOKIE_SAMESITE: 'Lax'

CSRF_COOKIE_SECURE: False

CSRF_FAILURE_VIEW: 'django.views.csrf.csrf_failure'

CSRF_HEADER_NAME: 'HTTP_X_CSRFTOKEN'

CSRF_TRUSTED_ORIGINS: []

CSRF_USE_SESSIONS: False

DATABASES: {

'default': { 'ATOMIC_REQUESTS': False,

'AUTOCOMMIT': True,

'CONN_MAX_AGE': 0,

'ENGINE': 'django.db.backends.sqlite3',

'HOST': '',

'NAME': '',

'OPTIONS': {},

'PASSWORD': '',

'PORT': '',

'TEST': { 'CHARSET': None,

'COLLATION': None,

'MIGRATE': True,

'MIRROR': None,

'NAME': None},

'TIME_ZONE': None,

'USER': ''}}

DATABASE_INFO_TO_USE: {

'ATOMIC_REQUESTS': False,

'AUTOCOMMIT': True,

'CONN_MAX_AGE': 0,

'ENGINE': 'django.db.backends.sqlite3',

'HOST': '',

'NAME': '',

'OPTIONS': {},

'PASSWORD': '',

'PORT': '',

'TEST': { 'CHARSET': None,

'COLLATION': None,

'MIGRATE': True,

'MIRROR': None,

'NAME': None},

'TIME_ZONE': None,

'USER': ''}

DATABASE_ROUTERS: ''

DATA_UPLOAD_MAX_MEMORY_SIZE: 2621440

DATA_UPLOAD_MAX_NUMBER_FIELDS: 1000

DATETIME_FORMAT: 'N j, Y, P'

DATETIME_INPUT_FORMATS: ['%Y-%m-%d %H:%M:%S',

'%Y-%m-%d %H:%M:%S.%f',

'%Y-%m-%d %H:%M',

'%m/%d/%Y %H:%M:%S',

'%m/%d/%Y %H:%M:%S.%f',

'%m/%d/%Y %H:%M',

'%m/%d/%y %H:%M:%S',

'%m/%d/%y %H:%M:%S.%f',

'%m/%d/%y %H:%M']

DATE_FORMAT: 'N j, Y'

DATE_INPUT_FORMATS: ['%Y-%m-%d',

'%m/%d/%Y',

'%m/%d/%y',

'%b %d %Y',

'%b %d, %Y',

'%d %b %Y',

'%d %b, %Y',

'%B %d %Y',

'%B %d, %Y',

'%d %B %Y',

'%d %B, %Y']

DEBUG: True

DEBUG_PROPAGATE_EXCEPTIONS: False

DEBUG_TOOLBAR_CONFIG: {

'SHOW_TOOLBAR_CALLBACK': '.settings.show_toolbar'}

DECIMAL_SEPARATOR: '.'

DEFAULT_CHARSET: 'utf-8'

DEFAULT_EXCEPTION_REPORTER: 'django.views.debug.ExceptionReporter'

DEFAULT_EXCEPTION_REPORTER_FILTER: 'django.views.debug.SafeExceptionReporterFilter'

DEFAULT_FILE_STORAGE: 'django.core.files.storage.FileSystemStorage'

DEFAULT_FROM_EMAIL: 'webmaster@localhost'

DEFAULT_HASHING_ALGORITHM: 'sha256'

DEFAULT_INDEX_TABLESPACE: ''

DEFAULT_TABLESPACE: ''

DISALLOWED_USER_AGENTS: []

EMAIL_BACKEND: 'django.core.mail.backends.smtp.EmailBackend'

EMAIL_HOST: 'localhost'

EMAIL_HOST_PASSWORD: ''

EMAIL_HOST_USER: ''

EMAIL_PORT: 25

EMAIL_SSL_CERTFILE: None

EMAIL_SSL_KEYFILE: ''

EMAIL_SUBJECT_PREFIX: '[Django] '

EMAIL_TIMEOUT: None

EMAIL_USE_LOCALTIME: False

EMAIL_USE_SSL: False

EMAIL_USE_TLS: False

FILE_UPLOAD_DIRECTORY_PERMISSIONS: None

FILE_UPLOAD_HANDLERS: ['django.core.files.uploadhandler.MemoryFileUploadHandler',

'django.core.files.uploadhandler.TemporaryFileUploadHandler']

FILE_UPLOAD_MAX_MEMORY_SIZE: 2621440

FILE_UPLOAD_PERMISSIONS: 420

FILE_UPLOAD_TEMP_DIR: None

FIRST_DAY_OF_WEEK: 0

FIXTURE_DIRS: []

FORCE_SCRIPT_NAME: None

FORMAT_MODULE_PATH: None

FORM_RENDERER: 'django.forms.renderers.DjangoTemplates'

IGNORABLE_404_URLS: []

INSTALLED_APPS: ['social_django',

'',

'',

'django.contrib.auth',

'django.contrib.contenttypes',

'django.contrib.sessions',

'django.contrib.messages',

'django.contrib.staticfiles',

'corsheaders',

'rest_framework',

'debug_toolbar',

'.django_celery_results']

INTERNAL_IPS: ['127.0.0.1']

IS_PRODUCTION_OR_DEMO: False

IS_REAL_PRODUCTION: False

JWT_AUTH: {

'JWT_ALGORITHM': 'RS256',

'JWT_AUDIENCE': '',

'JWT_AUTH_HEADER_PREFIX': 'Bearer',

'JWT_ISSUER': '',

'JWT_PAYLOAD_GET_USERNAME_HANDLER': 'auth.auth0user.jwt_get_username_from_payload_handler',

'JWT_PUBLIC_KEY': ''}

LANGUAGES: [('af', 'Afrikaans'),

('ar', 'Arabic'),

('ar-dz', 'Algerian Arabic'),

('ast', 'Asturian'),

('az', 'Azerbaijani'),

('bg', 'Bulgarian'),

('be', 'Belarusian'),

('bn', 'Bengali'),

('br', 'Breton'),

('bs', 'Bosnian'),

('ca', 'Catalan'),

('cs', 'Czech'),

('cy', 'Welsh'),

('da', 'Danish'),

('de', 'German'),

('dsb', 'Lower Sorbian'),

('el', 'Greek'),

('en', 'English'),

('en-au', 'Australian English'),

('en-gb', 'British English'),

('eo', 'Esperanto'),

('es', 'Spanish'),

('es-ar', 'Argentinian Spanish'),

('es-co', 'Colombian Spanish'),

('es-mx', 'Mexican Spanish'),

('es-ni', 'Nicaraguan Spanish'),

('es-ve', 'Venezuelan Spanish'),

('et', 'Estonian'),

('eu', 'Basque'),

('fa', 'Persian'),

('fi', 'Finnish'),

('fr', 'French'),

('fy', 'Frisian'),

('ga', 'Irish'),

('gd', 'Scottish Gaelic'),

('gl', 'Galician'),

('he', 'Hebrew'),

('hi', 'Hindi'),

('hr', 'Croatian'),

('hsb', 'Upper Sorbian'),

('hu', 'Hungarian'),

('hy', 'Armenian'),

('ia', 'Interlingua'),

('id', 'Indonesian'),

('ig', 'Igbo'),

('io', 'Ido'),

('is', 'Icelandic'),

('it', 'Italian'),

('ja', 'Japanese'),

('ka', 'Georgian'),

('kab', 'Kabyle'),

('kk', 'Kazakh'),

('km', 'Khmer'),

('kn', 'Kannada'),

('ko', 'Korean'),

('ky', 'Kyrgyz'),

('lb', 'Luxembourgish'),

('lt', 'Lithuanian'),

('lv', 'Latvian'),

('mk', 'Macedonian'),

('ml', 'Malayalam'),

('mn', 'Mongolian'),

('mr', 'Marathi'),

('my', 'Burmese'),

('nb', 'Norwegian Bokmål'),

('ne', 'Nepali'),

('nl', 'Dutch'),

('nn', 'Norwegian Nynorsk'),

('os', 'Ossetic'),

('pa', 'Punjabi'),

('pl', 'Polish'),

('pt', 'Portuguese'),

('pt-br', 'Brazilian Portuguese'),

('ro', 'Romanian'),

('ru', 'Russian'),

('sk', 'Slovak'),

('sl', 'Slovenian'),

('sq', 'Albanian'),

('sr', 'Serbian'),

('sr-latn', 'Serbian Latin'),

('sv', 'Swedish'),

('sw', 'Swahili'),

('ta', 'Tamil'),

('te', 'Telugu'),

('tg', 'Tajik'),

('th', 'Thai'),

('tk', 'Turkmen'),

('tr', 'Turkish'),

('tt', 'Tatar'),

('udm', 'Udmurt'),

('uk', 'Ukrainian'),

('ur', 'Urdu'),

('uz', 'Uzbek'),

('vi', 'Vietnamese'),

('zh-hans', 'Simplified Chinese'),

('zh-hant', 'Traditional Chinese')]

LANGUAGES_BIDI: ['he', 'ar', 'ar-dz', 'fa', 'ur']

LANGUAGE_CODE: 'en-us'

LANGUAGE_COOKIE_AGE: None

LANGUAGE_COOKIE_DOMAIN: None

LANGUAGE_COOKIE_HTTPONLY: False

LANGUAGE_COOKIE_NAME: 'django_language'

LANGUAGE_COOKIE_PATH: '/'

LANGUAGE_COOKIE_SAMESITE: None

LANGUAGE_COOKIE_SECURE: False

LOCALE_PATHS: []

LOGGING: {

}

LOGGING_CONFIG: 'logging.config.dictConfig'

LOGIN_REDIRECT_URL: '/'

LOGIN_URL: '/login/auth0'

LOGOUT_REDIRECT_URL: ''

MANAGERS: []

MAX_AGE_FOR_DB_CONNECTION_SECONDS: 300

MAX_CACHE_ENTRIES: 1000

MEDIA_ROOT: ''

MEDIA_URL: '/'

MESSAGE_STORAGE: 'django.contrib.messages.storage.fallback.FallbackStorage'

MIDDLEWARE: ['data.middleware.',

'data.routers.RouterMiddleware',

'data.middleware.log_performance_middleware',

'corsheaders.middleware.CorsMiddleware',

'django.contrib.sessions.middleware.SessionMiddleware',

'django.middleware.csrf.CsrfViewMiddleware',

'django.contrib.auth.middleware.AuthenticationMiddleware',

'data.middleware.bind_site_username_middleware',

'django.contrib.messages.middleware.MessageMiddleware',

'django.middleware.clickjacking.XFrameOptionsMiddleware',

'debug_toolbar.middleware.DebugToolbarMiddleware',

'data.middleware.']

MIGRATION_MODULES: {

}

MONTH_DAY_FORMAT: 'F j'

NUMBER_GROUPING: 0

OPTIONAL_CACHES: {

'dummycache': {'BACKEND': 'django.core.cache.backends.dummy.DummyCache'},

'loccache': { 'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',

'OPTIONS': {'MAX_ENTRIES': 1000},

'TIMEOUT': 345600},

'memcached': { 'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',

'LOCATION': None,

'OPTIONS': {'server_max_value_length': 33554432},

'TIMEOUT': 345600}}

OPTIONAL_DATABASES: {

'sqlite': { 'ATOMIC_REQUESTS': False,

'AUTOCOMMIT': True,

'CONN_MAX_AGE': 0,

'ENGINE': 'django.db.backends.sqlite3',

'HOST': '',

'NAME': '',

'OPTIONS': {},

'PASSWORD': '',

'PORT': '',

'TEST': { 'CHARSET': None,

'COLLATION': None,

'MIGRATE': True,

'MIRROR': None,

'NAME': None},

'TIME_ZONE': None,

'USER': ''},

'sqlite-tests': { 'ENGINE': 'django.db.backends.sqlite3',

'NAME': ':memory:'}}

PASSWORD_HASHERS: ''

PASSWORD_RESET_TIMEOUT: ''

PASSWORD_RESET_TIMEOUT_DAYS: ''

PREPEND_WWW: False

PROJECT_DIR: ''

REST_FRAMEWORK: {

'DEFAULT_AUTHENTICATION_CLASSES': ( 'rest_framework_jwt.authentication.JSONWebTokenAuthentication',

'rest_framework.authentication.SessionAuthentication',

'rest_framework.authentication.BasicAuthentication'),

'DEFAULT_PERMISSION_CLASSES': ( 'data..utils.views_utils.ProjectViewerEndpoint',),

'DEFAULT_RENDERER_CLASSES': [ 'rest_framework.renderers.JSONRenderer',

'rest_framework.renderers.BrowsableAPIRenderer'],

'DEFAULT_VERSIONING_CLASS': 'rest_framework.versioning.URLPathVersioning'}

ROOT_URLCONF: '.urls'

SECRET_KEY: ''

SECURE_BROWSER_XSS_FILTER: True

SECURE_CONTENT_TYPE_NOSNIFF: False

SECURE_HSTS_INCLUDE_SUBDOMAINS: False

SECURE_HSTS_PRELOAD: False

SECURE_HSTS_SECONDS: 0

SECURE_PROXY_SSL_HEADER: None

SECURE_REDIRECT_EXEMPT: []

SECURE_REFERRER_POLICY: 'same-origin'

SECURE_SSL_HOST: None

SECURE_SSL_REDIRECT: False

SELF_ADDRESS: 'http://localhost:7000'

SERVER_EMAIL: 'root@localhost'

SESSION_CACHE_ALIAS: 'default'

SESSION_COOKIE_AGE: 1209600

SESSION_COOKIE_DOMAIN: None

SESSION_COOKIE_HTTPONLY: True

SESSION_COOKIE_NAME: ''

SESSION_COOKIE_PATH: '/'

SESSION_COOKIE_SAMESITE: 'Lax'

SESSION_COOKIE_SECURE: False

SESSION_ENGINE: 'django.contrib.sessions.backends.db'

SESSION_EXPIRE_AT_BROWSER_CLOSE: False

SESSION_FILE_PATH: None

SESSION_SAVE_EVERY_REQUEST: False

SESSION_SERIALIZER: 'django.contrib.sessions.serializers.JSONSerializer'

SETTINGS_MODULE: '.settings'

SHORT_DATETIME_FORMAT: 'm/d/Y P'

SHORT_DATE_FORMAT: 'm/d/Y'

SHOULD_PROXY_AUTH_TABLES: False

SIGNING_BACKEND: 'django.core.signing.TimestampSigner'

SILENCED_SYSTEM_CHECKS: []

SOCIAL_AUTH_AUTH0_AUDIENCE: ''

SOCIAL_AUTH_AUTH0_DOMAIN: ''

SOCIAL_AUTH_AUTH0_JWKS: ('{"keys": [{"alg": "RS256", "kty": "RSA", '

'"use": "sig", "n": '

'"", '

'"e": "AQAB", "kid": '

'"", '

'"x5t": '

'"", '

'"x5c": '

'[""]}, '

'{"alg": "RS256", "kty": "RSA", "use": "sig", '

'"n": '

'"", '

'"e": "AQAB", "kid": "kd5L2gFzHj_oIGFj3dAQb", '

'"x5t": "", "x5c": '

'[""]}]}')

SOCIAL_AUTH_AUTH0_KEY: ''

SOCIAL_AUTH_AUTH0_SCOPE: ['openid', 'profile']

SOCIAL_AUTH_AUTH0_SECRET: ''

SOCIAL_AUTH_PIPELINE:

('social_core.pipeline.social_auth.social_details',

'social_core.pipeline.social_auth.social_uid',

'social_core.pipeline.social_auth.social_user',

'social_core.pipeline.user.get_username',

'social_core.pipeline.social_auth.associate_by_email',

'social_core.pipeline.user.create_user',

'social_core.pipeline.social_auth.associate_user',

'social_core.pipeline.social_auth.load_extra_data',

'social_core.pipeline.user.user_details')

SOCIAL_AUTH_REDIRECT_IS_HTTPS: False

: ''

STATICFILES_DIRS: []

STATICFILES_FINDERS: ['django.contrib.staticfiles.finders.FileSystemFinder',

'django.contrib.staticfiles.finders.AppDirectoriesFinder']

STATICFILES_STORAGE: 'django.contrib.staticfiles.storage.StaticFilesStorage'

STATIC_ROOT: ''

STATIC_URL: '/backend_static/'

TEMPLATES: [{'APP_DIRS': True,

'BACKEND': 'django.template.backends.django.DjangoTemplates',

'DIRS': [],

'OPTIONS': {'context_processors': ['django.template.context_processors.debug',

'django.template.context_processors.request',

'django.contrib.auth.context_processors.auth',

'django.contrib.messages.context_processors.messages']}}]

TEST_NON_SERIALIZED_APPS: []

TEST_RUNNER: 'django.test.runner.DiscoverRunner'

THOUSAND_SEPARATOR: ','

TIME_FORMAT: 'P'

TIME_INPUT_FORMATS: ['%H:%M:%S', '%H:%M:%S.%f', '%H:%M']

TIME_ZONE: 'UTC'

USE_I18N: False

USE_L10N: False

USE_THOUSAND_SEPARATOR: False

USE_TZ: True

USE_X_FORWARDED_HOST: True

USE_X_FORWARDED_PORT: True

WSGI_APPLICATION: '.wsgi.application'

X_FRAME_OPTIONS: 'DENY'

YEAR_MONTH_FORMAT: 'F Y'

is_overridden: <bound method Settings.is_overridden of <Settings "******.settings">>

deprecated_settings: None

:

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown

Minimal Celery Version: N/A or Unknown

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:amqp==5.0.6
celery==5.0.5
pickleshare==0.7.5

Other Dependencies

N/A

Minimally Reproducible Test Case

import pickle
from celery.canvas import chord
from io import BytesIO

dict(pickle.load(BytesIO(pickle.dumps(chord([])))))

Expected Behavior

I expect that after pickle load I will get the same object as I dumped

Actual Behavior

dumped:

{'task': 'celery.chord',

'args': (),

'kwargs': {'kwargs': {}, 'header': [], 'body': None},

'options': {},

'subtask_type': 'chord',

'immutable': False,

'chord_size': None}

loaded:

{'task': 'celery.chord',

'args': (),

'kwargs': {'kwargs': {'kwargs': {}}, 'header': [], 'body': None},

'options': {'chord_size': None},

'subtask_type': 'chord',

'immutable': False,

'chord_size': None}

kwargs added multiple times.

Issue 2:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Environment & Settings

Celery version:

celery report Output:

software -> celery:5.1.0 (sun-harmonics) kombu:5.1.0 py:3.9.2
billiard:3.6.4.0 redis:3.5.3
platform -> system:Linux arch:64bit, ELF
kernel version:4.4.0-198-generic imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://localhost/1

broker_url: 'redis://localhost:6379/1'

result_backend: 'redis://localhost/1'

result_serializer: 'pickle'

deprecated_settings: None

task_serializer: 'pickle'

accept_content: ['pickle', 'json']

result_expires: 3600

task_default_queue: 'zabbix'

Steps to Reproduce

Schedule a chord with beat via a task.

I created a task that takes a chain/chord as parameter and executes it, since beat does not allow for scheduling anything but a simple task directly:

@app.task()

def execute_chain(ch):

ch.apply_async(ignore_result=True)

This is added to the beat schedule via

ch = group(*zbxitems) | func.s() | publish_mqtt.s()

sender.add_periodic_task(interval, execute_chain.s(ch), name=name)

Required Dependencies

Python Packages

pip freeze Output:

amqp==5.0.6
billiard==3.6.4.0
celery==5.1.0
certifi==2021.5.30
chardet==4.0.0
click==7.1.2
click-didyoumean==0.0.3
click-plugins==1.1.1
click-repl==0.2.0
colorama==0.4.4
idna==2.10
kombu==5.1.0
paho-mqtt==1.5.1
prompt-toolkit==3.0.18
pytz==2021.1
PyYAML==5.4.1
pyzabbix==0.8.2
redis==3.5.3
requests==2.25.1
six==1.16.0
urllib3==1.26.5
vine==5.0.0
wcwidth==0.2.5
(venv) root@vmt

Other Dependencies

version 3.9.5

Expected Behavior

This should continue to work, executing the chain again and again.

Actual Behavior

I works for several hours till it exits with:

2021-06-11 06:25:54,253:CRITICAL:worker.start:line 207:Unrecoverable error: RecursionError('maximum recursion depth exceeded while pickling an object')

Traceback (most recent call last):

File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/worker/worker.py", line 203, in start

self.blueprint.start(self)

File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/bootsteps.py", line 116, in start

step.start(parent)

File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/bootsteps.py", line 365, in start

return self.obj.start()

File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 326, in start

blueprint.start(self)

File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/bootsteps.py", line 116, in start

step.start(parent)

File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 618, in start

c.loop(*c.loop_args())

File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/worker/loops.py", line 81, in asynloop

next(loop)

File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/kombu/asynchronous/hub.py", line 361, in create_loop

cb(*cbargs)

File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/kombu/transport/redis.py", line 1128, in on_readable

self.cycle.on_readable(fileno)

File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/kombu/transport/redis.py", line 398, in on_readable

chan.handlerstype

File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/kombu/transport/redis.py", line 789, in _brpop_read

self.connection._deliver(loads(bytes_to_str(item)), dest)

File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/kombu/transport/virtual/base.py", line 980, in _deliver

callback(message)

File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/kombu/transport/virtual/base.py", line 630, in _callback

return callback(message)

File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/kombu/messaging.py", line 626, in _receive_callback

return on_m(message) if on_m else self.receive(decoded, message)

File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 586, in on_task_received

strategy(

File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/worker/strategy.py", line 201, in task_message_handler

handle(req)

File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/worker/worker.py", line 221, in _process_task_sem

return self._quick_acquire(self._process_task, req)

File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/kombu/asynchronous/semaphore.py", line 57, in acquire

callback(*partial_args, **partial_kwargs)

File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/worker/worker.py", line 226, in _process_task

req.execute_using_pool(self.pool)

File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/worker/request.py", line 691, in execute_using_pool

result = apply_async(

File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/concurrency/base.py", line 152, in apply_async

return self.on_apply(target, args, kwargs,

File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/billiard/pool.py", line 1530, in apply_async

self._quick_put((TASK, (result._job, None, func, args, kwds)))

File "/usr/share/nd_zabbixexport/venv/lib/python3.9/site-packages/celery/concurrency/asynpool.py", line 859, in send_job

body = dumps(tup, protocol=protocol)

RecursionError: maximum recursion depth exceeded while pickling an object

When json dumping the task, I see this after a few iterations (mind the kwargs):

2021-06-11 16:44:16,370:DEBUG:beat.execute_chain:line 90:{"task": "celery.chord", "args": [], "kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {

"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {"kwargs": {}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}, "header": [{"task": "tasks.zabbix.get

_zabbix_items_value", "args": ["Memory Free", "rd servers"], "kwargs": {}, "options": {}, "subtask_type": null, "immutable": true}, {"task": "tasks.zabbix.get_zabbix_items_value", "args": ["Available memory", "rd servers"], "kwargs": {}, "options": {}, "subtask_type": n

ull, "immutable": true}], "body": {"task": "celery.chain", "args": [], "kwargs": {"tasks": [{"task": "tasks.aggregate.percentage_per_item", "args": [], "kwargs": {}, "options": {}, "subtask_type": null, "immutable": false}, {"task": "tasks.mqtt.publish_mqtt", "args": ["

vmmqtt01", "zabbix/"], "kwargs": {}, "options": {}, "subtask_type": null, "immutable": false}]}, "options": {}, "subtask_type": "chain", "immutable": false}}, "options": {}, "subtask_type": "chord", "immutable": false}

I tried also by cloning the chord first in the scheduled task and executing the clone, but same result.