Issue 1:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Environment & Settings

Celery version: 3.1.19

celery report - **on windows 8.1** Output:

software -> celery:3.1.19 (Cipater) kombu:3.0.37 py:3.6.7rc1
billiard:3.3.0.23 redis:2.10.6
platform -> system:Windows arch:64bit, WindowsPE imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://127.0.0.1:6379/0

ABSOLUTE_URL_OVERRIDES: {
}
ACCEPTED_ANNOTATION_FORMATS: ['xml', 'json']
ACCEPTED_ARCHIVE_FORMATS: ['zip',
'egg',
'jar',
'tar',
'gz',
'tgz',
'tar',
'bz2',
'tz2']
ACCEPTED_IMAGE_FORMATS: ['gif', 'png', 'jpg', 'jpeg', 'tif', 'bmp']
ADMINS: []
ADMIN_GROUP_NAME: 'admins'
ALLOWED_HOSTS: ['0.0.0.0',
'34.90.115.59',
'34.90.38.139',
'$HOST',
'127.0.0.1',
'0.0.0.0']
APPEND_SLASH: True
ARCHIVES_IN_PARALLEL_UPLOAD: 3
ASGI_APPLICATION: 'core.asgi'
AUTHENTICATION_BACKENDS: ['django.contrib.auth.backends.ModelBackend']
AUTH_PASSWORD_VALIDATORS: '********'
AUTH_USER_MODEL: 'accounts.User'
BASE_DIR: 'C:\\Users\\Abdulkader\\Documents\\mirrorr-framework\\mirror'
BROKER_URL: 'redis://127.0.0.1:6379/0'
CACHES: {
'default': {'BACKEND': 'django.core.cache.backends.locmem.LocMemCache'}}
CACHE_MIDDLEWARE_ALIAS: 'default'
CACHE_MIDDLEWARE_KEY_PREFIX: '********'
CACHE_MIDDLEWARE_SECONDS: 600
CELERY_ACCEPT_CONTENT: ['json']
CELERY_RESULT_BACKEND: 'redis://127.0.0.1:6379/0'
CELERY_RESULT_SERIALIZER: 'json'
CELERY_TASK_SERIALIZER: 'json'
CHALLENGE_LENGTH: 6
CHANNEL_LAYERS: {
'default': { 'BACKEND': 'asgi_redis.RedisChannelLayer',
'CONFIG': {'hosts': ['redis://127.0.0.1:6379']},
'ROUTING': 'sources.routing.channel_routing'}}
CLEAN_NON_IMAGE_EXTRACTED: True
CSRF_COOKIE_AGE: 31449600
CSRF_COOKIE_DOMAIN: None
CSRF_COOKIE_HTTPONLY: False
CSRF_COOKIE_NAME: 'csrftoken'
CSRF_COOKIE_PATH: '/'
CSRF_COOKIE_SAMESITE: 'Lax'
CSRF_COOKIE_SECURE: False
CSRF_FAILURE_VIEW: 'django.views.csrf.csrf_failure'
CSRF_HEADER_NAME: 'HTTP_X_CSRFTOKEN'
CSRF_TRUSTED_ORIGINS: []
CSRF_USE_SESSIONS: False
DATABASES: {
'default': { 'ENGINE': 'djongo',
'HOST': '127.0.0.1',
'NAME': 'mirror',
'PASSWORD': '********',
'USER': ''}}
DATABASE_ROUTERS: '********'
DATA_SOURCES:
(('M', 'Image Files Upload'),
('A', 'Archive Upload'),
('GB', 'Google Bucket'),
('AB', 'Amazon S3 Bucket'))
DATA_UPLOAD_MAX_MEMORY_SIZE: 3145728000
DATA_UPLOAD_MAX_NUMBER_FIELDS: 1000
DATETIME_FORMAT: 'N j, Y, P'
DATETIME_INPUT_FORMATS: ['%Y-%m-%d %H:%M:%S',
'%Y-%m-%d %H:%M:%S.%f',
'%Y-%m-%d %H:%M',
'%Y-%m-%d',
'%m/%d/%Y %H:%M:%S',
'%m/%d/%Y %H:%M:%S.%f',
'%m/%d/%Y %H:%M',
'%m/%d/%Y',
'%m/%d/%y %H:%M:%S',
'%m/%d/%y %H:%M:%S.%f',
'%m/%d/%y %H:%M',
'%m/%d/%y']
DATE_FORMAT: 'N j, Y'
DATE_INPUT_FORMATS: ['%Y-%m-%d',
'%m/%d/%Y',
'%m/%d/%y',
'%b %d %Y',
'%b %d, %Y',
'%d %b %Y',
'%d %b, %Y',
'%B %d %Y',
'%B %d, %Y',
'%d %B %Y',
'%d %B, %Y']
DEBUG: True
DEBUG_PROPAGATE_EXCEPTIONS: False
DECIMAL_SEPARATOR: '.'
DEFAULT_ANNOTATIONS_TEMP_DIR: 'C:\\Users\\Abdulkader\\Documents\\mirrorr-framework\\mirror\\public\\media\\annotations-temp'
DEFAULT_ARCHIVE_TEMP_DIR: 'C:\\Users\\Abdulkader\\Documents\\mirrorr-framework\\mirror\\public\\media\\archives-temp'
DEFAULT_BUCKET_TEMP_DIR: 'C:\\Users\\Abdulkader\\Documents\\mirrorr-framework\\mirror\\public\\media\\imported-images'
DEFAULT_CHARSET: 'utf-8'
DEFAULT_CONTENT_TYPE: 'text/html'
DEFAULT_EXCEPTION_REPORTER_FILTER: 'django.views.debug.SafeExceptionReporterFilter'
DEFAULT_FILE_STORAGE: 'storages.backends.gcloud.GoogleCloudStorage'
DEFAULT_FROM_EMAIL: 'webmaster@localhost'
DEFAULT_IMAGE_EXTRACTED_DIR: 'C:\\Users\\Abdulkader\\Documents\\mirrorr-framework\\mirror\\public\\media\\extracted-images'
DEFAULT_IMAGE_HASH_CODE: 'xxxxxxxxxxxxxxxx'
DEFAULT_IMAGE_TEMP_DIR: 'C:\\Users\\Abdulkader\\Documents\\mirrorr-framework\\mirror\\public\\media\\manual-images'
DEFAULT_INDEX_TABLESPACE: ''
DEFAULT_PASSWORD: '********'
DEFAULT_TABLESPACE: ''
DISALLOWED_USER_AGENTS: []
DS_IMAGE_DIRS: {
'A': 'C:\\Users\\Abdulkader\\Documents\\mirrorr-framework\\mirror\\public\\media\\extracted-images',
'GB': 'C:\\Users\\Abdulkader\\Documents\\mirrorr-framework\\mirror\\public\\media\\imported-images',
'M': 'C:\\Users\\Abdulkader\\Documents\\mirrorr-framework\\mirror\\public\\media\\manual-images'}
EMAIL_BACKEND: 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST: 'localhost'
EMAIL_HOST_PASSWORD: '********'
EMAIL_HOST_USER: ''
EMAIL_PORT: 25
EMAIL_SSL_CERTFILE: None
EMAIL_SSL_KEYFILE: '********'
EMAIL_SUBJECT_PREFIX: '[Django] '
EMAIL_TIMEOUT: None
EMAIL_USE_LOCALTIME: False
EMAIL_USE_SSL: False
EMAIL_USE_TLS: False
FILE_CHARSET: 'utf-8'
FILE_UPLOAD_DIRECTORY_PERMISSIONS: None
FILE_UPLOAD_HANDLERS: ['django.core.files.uploadhandler.MemoryFileUploadHandler',
'django.core.files.uploadhandler.TemporaryFileUploadHandler']
FILE_UPLOAD_MAX_MEMORY_SIZE: 2621440
FILE_UPLOAD_PERMISSIONS: None
FILE_UPLOAD_TEMP_DIR: None
FIRST_DAY_OF_WEEK: 0
FIXTURE_DIRS: []
FLATTEN_DIRS_IN_EXTRACT: True
FORCE_SCRIPT_NAME: None
FORMAT_MODULE_PATH: None
FORM_RENDERER: 'django.forms.renderers.DjangoTemplates'
GCS_USE_UNSIGNED_URLS: True
GET_UNIQUE_HASHES: True
GS_BUCKET_NAME: 'data_framework'
GS_CREDENTIALS: <google.oauth2.service_account.Credentials object at 0x000000714B30E5C0>
GS_FILE_OVERWRITE: True
GS_LOCATION: ''
HASHING_METHOD: 'WHash'
HOST: '127.0.0.1'
IGNORABLE_404_URLS: []
IMAGES_IN_PARALLEL_UPLOAD: 30
IMG_PROCESS_TASK_BATCH_SIZE: 50
INSTALLED_APPS: ['django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'django_extensions',
'channels',
'sorl.thumbnail',
'accounts',
'image',
'dashboard',
'sources',
'rest_framework']
INTERNAL_IPS: []
IPYTHON_ARGUMENTS: ['--debug']
LANGUAGES: [('af', 'Afrikaans'),
('ar', 'Arabic'),
('ast', 'Asturian'),
('az', 'Azerbaijani'),
('bg', 'Bulgarian'),
('be', 'Belarusian'),
('bn', 'Bengali'),
('br', 'Breton'),
('bs', 'Bosnian'),
('ca', 'Catalan'),
('cs', 'Czech'),
('cy', 'Welsh'),
('da', 'Danish'),
('de', 'German'),
('dsb', 'Lower Sorbian'),
('el', 'Greek'),
('en', 'English'),
('en-au', 'Australian English'),
('en-gb', 'British English'),
('eo', 'Esperanto'),
('es', 'Spanish'),
('es-ar', 'Argentinian Spanish'),
('es-co', 'Colombian Spanish'),
('es-mx', 'Mexican Spanish'),
('es-ni', 'Nicaraguan Spanish'),
('es-ve', 'Venezuelan Spanish'),
('et', 'Estonian'),
('eu', 'Basque'),
('fa', 'Persian'),
('fi', 'Finnish'),
('fr', 'French'),
('fy', 'Frisian'),
('ga', 'Irish'),
('gd', 'Scottish Gaelic'),
('gl', 'Galician'),
('he', 'Hebrew'),
('hi', 'Hindi'),
('hr', 'Croatian'),
('hsb', 'Upper Sorbian'),
('hu', 'Hungarian'),
('hy', 'Armenian'),
('ia', 'Interlingua'),
('id', 'Indonesian'),
('io', 'Ido'),
('is', 'Icelandic'),
('it', 'Italian'),
('ja', 'Japanese'),
('ka', 'Georgian'),
('kab', 'Kabyle'),
('kk', 'Kazakh'),
('km', 'Khmer'),
('kn', 'Kannada'),
('ko', 'Korean'),
('lb', 'Luxembourgish'),
('lt', 'Lithuanian'),
('lv', 'Latvian'),
('mk', 'Macedonian'),
('ml', 'Malayalam'),
('mn', 'Mongolian'),
('mr', 'Marathi'),
('my', 'Burmese'),
('nb', 'Norwegian Bokmål'),
('ne', 'Nepali'),
('nl', 'Dutch'),
('nn', 'Norwegian Nynorsk'),
('os', 'Ossetic'),
('pa', 'Punjabi'),
('pl', 'Polish'),
('pt', 'Portuguese'),
('pt-br', 'Brazilian Portuguese'),
('ro', 'Romanian'),
('ru', 'Russian'),
('sk', 'Slovak'),
('sl', 'Slovenian'),
('sq', 'Albanian'),
('sr', 'Serbian'),
('sr-latn', 'Serbian Latin'),
('sv', 'Swedish'),
('sw', 'Swahili'),
('ta', 'Tamil'),
('te', 'Telugu'),
('th', 'Thai'),
('tr', 'Turkish'),
('tt', 'Tatar'),
('udm', 'Udmurt'),
('uk', 'Ukrainian'),
('ur', 'Urdu'),
('vi', 'Vietnamese'),
('zh-hans', 'Simplified Chinese'),
('zh-hant', 'Traditional Chinese')]
LANGUAGES_BIDI: ['he', 'ar', 'fa', 'ur']
LANGUAGE_CODE: 'en-us'
LANGUAGE_COOKIE_AGE: None
LANGUAGE_COOKIE_DOMAIN: None
LANGUAGE_COOKIE_NAME: 'django_language'
LANGUAGE_COOKIE_PATH: '/'
LIMITED_GROUP_NAME: 'labelers'
LOCALE_PATHS: []
LOGGING: {
'disable_existing_loggers': False,
'filters': { 'hide_staticfiles': { '()': 'core.settings.project.logger.SkipStaticFilter'},
'require_debug_false': { '()': 'django.utils.log.RequireDebugFalse'}},
'formatters': { 'console': { 'datefmt': '%Y-%m-%d %H:%M:%S',
'format': '%(name)-12s %(levelname)-8s '
'%(message)s'},
'django.channels.server': { '()': 'django.utils.log.ServerFormatter',
'format': '%(asctime)s - '
'%(levelname)s - '
'%(module)s - '
'%(message)s'},
'django.server': { '()': 'django.utils.log.ServerFormatter',
'format': '[{server_time}] '
'{message}',
'style': '{'},
'file': { 'datefmt': '%Y-%m-%d %H:%M:%S',
'format': '%(asctime)s %(levelname)-8s '
'%(message)s'},
'img-logs': { 'datefmt': '%Y-%m-%d %H:%M:%S',
'format': '%(asctime)s %(username)-8s '
'%(levelname)-8s %(message)s'}},
'handlers': { 'console': { 'class': 'logging.StreamHandler',
'filters': ['hide_staticfiles'],
'formatter': 'console'},
'django-error': { 'class': 'logging.FileHandler',
'filename': 'C:\\Users\\Abdulkader\\Documents\\mirrorr-framework\\mirror\\public\\logs\\server-error.log',
'formatter': 'file'},
'django.channels.server': { 'class': 'logging.FileHandler',
'filename': 'C:\\Users\\Abdulkader\\Documents\\mirrorr-framework\\mirror\\public\\logs\\channels.log',
'formatter': 'django.channels.server',
'level': 'INFO'},
'image-operation': { 'class': 'logging.FileHandler',
'filename': 'C:\\Users\\Abdulkader\\Documents\\mirrorr-framework\\mirror\\public\\logs\\image-operation.log',
'formatter': 'img-logs'},
'image-upload': { 'class': 'logging.FileHandler',
'filename': 'C:\\Users\\Abdulkader\\Documents\\mirrorr-framework\\mirror\\public\\logs\\image-upload.log',
'formatter': 'img-logs'}},
'loggers': { 'django': { 'handlers': ['console', 'image-upload'],
'level': 'ERROR'},
'django.channels.server': { 'handlers': [ 'django.channels.server'],
'level': 'DEBUG',
'propagate': True},
'django.request': { 'handlers': ['django-error'],
'level': 'INFO',
'propagate': False},
'image-operation-errors': { 'handlers': [ 'image-operation'],
'level': 'ERROR'},
'image-operation-info': { 'handlers': ['image-operation'],
'level': 'DEBUG'},
'image-operation-warnings': { 'handlers': [ 'image-operation'],
'level': 'WARNING'},
'image-upload-errors': { 'handlers': ['image-upload'],
'level': 'ERROR'},
'image-upload-info': { 'handlers': ['image-upload'],
'level': 'DEBUG'},
'image-upload-warnings': { 'handlers': ['image-upload'],
'level': 'WARNING'}},
'version': 1}
LOGGING_CONFIG: 'logging.config.dictConfig'
LOGIN_REDIRECT_URL: '/home'
LOGIN_URL: '/accounts/login/'
LOGOUT_REDIRECT_URL: '/accounts/login'
LOGS_ROOT: 'C:\\Users\\Abdulkader\\Documents\\mirrorr-framework\\mirror\\public\\logs'
LOGS_URL: '/logs/'
MAIN_STORAGE_URL: 'https://storage.cloud.google.com'
MANAGERS: []
MAX_ARCHIVES_IN_ONE_UPLOAD: 10
MAX_ARCHIVE_SIZE: 300
MAX_IMAGES_IN_ONE_UPLOAD: 200
MAX_IMG_SIZE: 2
MEDIA_ROOT: 'C:\\Users\\Abdulkader\\Documents\\mirrorr-framework\\mirror\\public\\media'
MEDIA_URL: '/media/'
MEMBER_GROUP_NAME: 'members'
MESSAGE_STORAGE: 'django.contrib.messages.storage.fallback.FallbackStorage'
MIDDLEWARE: ['django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware']
MIGRATION_MODULES: {
}
MODE: 'local'
MONTH_DAY_FORMAT: 'F j'
NOTEBOOK_ARGUMENTS: ['--notebook-dir', 'Drafts']
NUMBER_GROUPING: 0
PASSWORD_HASHERS: '********'
PASSWORD_RESET_TIMEOUT_DAYS: '********'
PREPEND_WWW: False
RANDOM_SEED: 5985478547
REST_FRAMEWORK: {
'DEFAULT_AUTHENTICATION_CLASSES': ( 'rest_framework.authentication.BasicAuthentication',
'rest_framework.authentication.SessionAuthentication',
'rest_framework.authentication.TokenAuthentication'),
'DEFAULT_PERMISSION_CLASSES': [ 'rest_framework.permissions.IsAuthenticated']}
ROOT_URLCONF: 'core.urls'
SECRET_KEY: '********'
SECURE_BROWSER_XSS_FILTER: False
SECURE_CONTENT_TYPE_NOSNIFF: False
SECURE_HSTS_INCLUDE_SUBDOMAINS: False
SECURE_HSTS_PRELOAD: False
SECURE_HSTS_SECONDS: 0
SECURE_PROXY_SSL_HEADER: None
SECURE_REDIRECT_EXEMPT: []
SECURE_SSL_HOST: None
SECURE_SSL_REDIRECT: False
SEPERATOR: ','
SERVER_EMAIL: 'root@localhost'
SESSION_CACHE_ALIAS: 'default'
SESSION_COOKIE_AGE: 1209600
SESSION_COOKIE_DOMAIN: None
SESSION_COOKIE_HTTPONLY: True
SESSION_COOKIE_NAME: 'sessionid'
SESSION_COOKIE_PATH: '/'
SESSION_COOKIE_SAMESITE: 'Lax'
SESSION_COOKIE_SECURE: False
SESSION_ENGINE: 'django.contrib.sessions.backends.db'
SESSION_EXPIRE_AT_BROWSER_CLOSE: True
SESSION_FILE_PATH: None
SESSION_SAVE_EVERY_REQUEST: False
SESSION_SERIALIZER: 'django.contrib.sessions.serializers.JSONSerializer'
SETTINGS_MODULE: 'core.settings'
SHORT_DATETIME_FORMAT: 'm/d/Y P'
SHORT_DATE_FORMAT: 'm/d/Y'
SIGNING_BACKEND: 'django.core.signing.TimestampSigner'
SILENCED_SYSTEM_CHECKS: []
STATICFILES_DIRS:
('C:\\Users\\Abdulkader\\Documents\\mirrorr-framework\\mirror\\static',
'C:\\Users\\Abdulkader\\Documents\\mirrorr-framework\\mirror\\static')
STATICFILES_FINDERS: ['django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder']
STATICFILES_STORAGE: 'django.contrib.staticfiles.storage.StaticFilesStorage'
STATIC_ROOT: 'C:\\Users\\Abdulkader\\Documents\\mirrorr-framework\\mirror\\'
STATIC_URL: '/static/'
TASK_STATUS:
(('IN', 'Initiated'),
('ST', 'Started'),
('IP', 'In-Progress'),
('FA', 'Failed'),
('DO', 'Done'))
TEMPLATES: [{'APP_DIRS': True,
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': ['templates',
'C:\\Users\\Abdulkader\\Documents\\mirrorr-framework\\mirror\\templates'],
'OPTIONS': {'context_processors': ['django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages']}}]
TEMP_DIR: 'C:\\Windows\\Temp'
TEST_NON_SERIALIZED_APPS: []
TEST_RUNNER: 'django.test.runner.DiscoverRunner'
THOUSAND_SEPARATOR: ','
THUMBNAIL_DEBUG: True
THUMBNAIL_ENGINE: 'sorl.thumbnail.engines.pil_engine.Engine'
THUMBNAIL_KVSTORE: 'sorl.thumbnail.kvstores.redis_kvstore.KVStore'
THUMBNAIL_OPTIONS: {
'colorspace': 'RGB',
'crop': 'center',
'format': 'JPEG',
'orientation': True,
'padding': False,
'quality': 80,
'upscale': True}
THUMBNAIL_REDIS_DB: 1
THUMBNAIL_REDIS_PORT: 6379
THUMBNAIL_STORAGE: 'storages.backends.gcloud.GoogleCloudStorage'
THUMBNAIL_URL_TIMEOUT: 150000
TIME_FORMAT: 'P'
TIME_INPUT_FORMATS: ['%H:%M:%S', '%H:%M:%S.%f', '%H:%M']
TIME_ZONE: 'UTC'
USE_I18N: True
USE_L10N: True
USE_THOUSAND_SEPARATOR: False
USE_TZ: True
USE_X_FORWARDED_HOST: False
USE_X_FORWARDED_PORT: False
WSGI_APPLICATION: 'core.wsgi.application'
X_FRAME_OPTIONS: 'SAMEORIGIN'
YEAR_MONTH_FORMAT: 'F Y'

celery report - **on server debian 10 ** Output:

Traceback (most recent call last):
File "/usr/local/bin/celery", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/dist-packages/celery/__main__.py", line 30, in main
main()
File "/usr/local/lib/python3.7/dist-packages/celery/bin/celery.py", line 81, in main
cmd.execute_from_commandline(argv)
File "/usr/local/lib/python3.7/dist-packages/celery/bin/celery.py", line 770, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/usr/local/lib/python3.7/dist-packages/celery/bin/base.py", line 311, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "/usr/local/lib/python3.7/dist-packages/celery/bin/celery.py", line 762, in handle_argv
return self.execute(command, argv)
File "/usr/local/lib/python3.7/dist-packages/celery/bin/celery.py", line 694, in execute
).run_from_argv(self.prog_name, argv[1:], command=argv[0])
File "/usr/local/lib/python3.7/dist-packages/celery/bin/base.py", line 315, in run_from_argv
sys.argv if argv is None else argv, command)
File "/usr/local/lib/python3.7/dist-packages/celery/bin/base.py", line 377, in handle_argv
return self(*args, **options)
File "/usr/local/lib/python3.7/dist-packages/celery/bin/base.py", line 274, in __call__
ret = self.run(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/celery/bin/celery.py", line 648, in run
self.out(self.app.bugreport())
File "/usr/local/lib/python3.7/dist-packages/celery/app/base.py", line 438, in bugreport
return bugreport(self)
File "/usr/local/lib/python3.7/dist-packages/celery/app/utils.py", line 220, in bugreport
human_settings=app.conf.humanize(),
File "/usr/local/lib/python3.7/dist-packages/celery/app/utils.py", line 137, in humanize
for key, value in items(self.table(with_defaults, censored)))
File "/usr/local/lib/python3.7/dist-packages/celery/app/utils.py", line 128, in table
self if with_defaults else self.without_defaults())
File "/usr/local/lib/python3.7/dist-packages/celery/app/utils.py", line 192, in filter_hidden_settings
return dict((k, maybe_censor(k, v)) for k, v in items(conf))
File "/usr/local/lib/python3.7/dist-packages/celery/app/utils.py", line 192, in <genexpr>
return dict((k, maybe_censor(k, v)) for k, v in items(conf))
File "/usr/local/lib/python3.7/dist-packages/celery/app/utils.py", line 189, in maybe_censor
return Connection(value).as_uri(mask=mask)
File "/usr/local/lib/python3.7/dist-packages/kombu/connection.py", line 177, in __init__
if not get_transport_cls(transport).can_parse_url:
File "/usr/local/lib/python3.7/dist-packages/kombu/transport/__init__.py", line 109, in get_transport_cls
_transport_cache[transport] = resolve_transport(transport)
File "/usr/local/lib/python3.7/dist-packages/kombu/transport/__init__.py", line 93, in resolve_transport
return symbol_by_name(transport)
File "/usr/local/lib/python3.7/dist-packages/kombu/utils/__init__.py", line 96, in symbol_by_name
module = imp(module_name, package=package, **kwargs)
File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 724, in exec_module
File "<frozen importlib._bootstrap_external>", line 860, in get_code
File "<frozen importlib._bootstrap_external>", line 791, in source_to_code
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/local/lib/python3.7/dist-packages/kombu/transport/redis.py", line 815
def _connparams(self, async=False, _r210_options=(

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.7

Minimal Celery Version: 3.1.19

Minimal Kombu Version: 3.0.37

Minimal Broker Version: redis 3.2.1

Minimal Result Backend Version: Django==2.2.4

Minimal OS and/or Kernel Version: Local: Windows 8.1, Server: Debian 10 Bluster

Minimal Broker Client Version: Redis

Python Packages

pip freeze - on windows 8.1 Output:

absl-py==0.8.1
aioredis==1.3.0
amqp==1.4.9
amqplib==1.0.2
anyjson==0.3.3
appdirs==1.4.3
asgi-redis==1.4.3
asgiref==1.1.2
asn1crypto==0.24.0
astor==0.7.1
astroid==2.0.4
astunparse==1.6.2
async-timeout==2.0.1
attrs==18.2.0
autobahn==19.10.1
Automat==0.7.0
autopep8==1.4.4
Babel==2.3.4
backcall==0.1.0
backports.weakref==1.0rc1
beautifulsoup4==4.6.3
billiard==3.3.0.23
black==18.9b0
bleach==1.5.0
boilerplate==1.2.4
bokeh==1.3.1
boto==2.49.0
bs4==0.0.1
cachetools==3.1.1
carrot==0.10.7
celery==3.1.19
certifi==2018.10.15
cffi==1.12.0
chainer==2.0.0
channels==1.1.8
channels-redis==2.4.0
chardet==3.0.4
Click==7.0
cmake==3.14.4
colorama==0.4.0
coloredlogs==10.0
ColorLogging==0.3.3
colorlover==0.2.1
colour==0.1.5
constantly==15.1.0
cryptography==2.8
cssselect==1.0.3
cufflinks==0.14.6
Currency==0.1.2
cycler==0.10.0
Cython==0.29.13
daphne==1.4.2
dataclasses==0.6
decorator==4.0.10
defusedxml==0.5.0
dicttoxml==1.7.4
dj-database-url==0.5.0
dj-environ==0.0.1
Django==2.2.4
django-appconf==1.0.3
django-background-task==0.1.8
django-background-tasks==1.1.13
django-bootstrap4==0.0.8
django-braces==1.13.0
django-click==2.1.0
django-compat==1.0.15
django-environ==0.4.5
django-extensions==2.2.1
django-faker==0.2
django-mongoengine==0.3.3
django-nose==1.4.4
django-picklefield==2.0
django-redis==4.10.0
django-request-logging==0.7.0
django-sortedm2m==2.0.0
django-storages==1.7.2
django-tagging==0.4.6
django-user-accounts==2.1.0
django-websocket-redis==0.6.0
djangorestframework==3.10.3
djongo==1.2.36
dlib==19.7.0
dnspython==1.16.0
docopt==0.6.2
docutils==0.12
ebaysdk==2.1.5
entrypoints==0.2.3
eventlet==0.25.1
ExifRead==2.1.2
exiv2==0.2
factory-boy==2.12.0
fake-factory==9999.9.9
Faker==2.0.2
feedparser==5.2.1
filelock==3.0.12
Flask==1.0.2
gast==0.2.2
geographiclib==1.49
geojson==2.5.0
geopy==1.20.0
gevent==1.4.0
google-api-core==1.14.3
google-auth==1.6.3
google-cloud==0.34.0
google-cloud-core==1.0.3
google-cloud-storage==1.23.0
google-pasta==0.1.7
google-resumable-media==0.5.0
googleapis-common-protos==1.6.0
greenlet==0.4.15
grpcio==1.17.0
h2==3.1.1
h5py==2.8.0
hachoir==3.0a5
hiredis==1.0.0
hpack==3.0.0
html5lib==0.9999999
humanfriendly==4.18
hyperframe==5.2.0
hyperlink==19.0.0
idna==2.7
imagededup==0.1.0
imutils==0.4.3
incremental==17.5.0
IPy==0.83
ipykernel==5.1.0
ipython==7.2.0
ipython-genutils==0.2.0
ipywidgets==7.4.2
isort==4.3.4
itsdangerous==1.1.0
jedi==0.13.1
Jinja2==2.10
joblib==0.14.0
jsonmodel==3.0
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.2.3
jupyter-console==6.0.0
jupyter-core==4.4.0
jupyter-tensorboard==0.1.8
jupyterthemes==0.20.0
Keras==2.2.4
Keras-Applications==1.0.8
Keras-Preprocessing==1.0.5
kiwisolver==1.0.1
kombu==3.0.37
labpack==0.20
lazy-object-proxy==1.3.1
lesscpy==0.13.0
lxml==4.2.5
Markdown==3.1.1
MarkupSafe==1.1.0
matlab==0.1
matplotlib==3.1.1
mccabe==0.6.1
mistune==0.8.4
mongoengine==0.18.2
monotonic==1.5
mpld3==0.3
msgpack==0.6.2
msgpack-python==0.5.6
nbconvert==5.4.0
nbformat==4.4.0
nltk==3.4
nose==1.3.7
nosedjango==1.0.13
notebook==5.7.2
numpy==1.16.3
opencv-python==4.1.1.26
opt-einsum==3.1.0
packaging==19.1
pandas==0.23.4
pandoc==1.0.2
pandocfilters==1.4.2
parsel==1.5.1
parso==0.3.1
passlib==1.7.1
patsy==0.5.1
pbr==5.1.1
pexpect==4.7.0
pickleshare==0.7.5
Pillow==6.1.0
pipenv==2018.11.26
pipreqs==0.4.9
pixiedust==1.1.17
plotly==3.4.2
ply==3.11
pocketlab==0.10
priority==1.3.0
prometheus-client==0.5.0
prompt-toolkit==2.0.7
protobuf==3.10.0
psutil==5.4.8
psycopg2==2.8.4
psycopg2-binary==2.8.3
ptyprocess==0.6.0
pyaramorph==0.2
pyasn1==0.4.6
pyasn1-modules==0.2.6
pycodestyle==2.5.0
pycparser==2.19
pydata==1.0.0
PyDispatcher==2.0.5
pyexiv2==1.3.0
Pygments==2.3.0
pygorithm==1.0.4
PyHamcrest==1.9.0
PyJWT==1.7.1
pylint==2.1.1
pylint-django==2.0.9
pylint-plugin-utils==0.5
pymongo==3.9.0
pyOpenSSL==19.0.0
pyparsing==2.3.0
PyPDF2==1.26.0
pypiwin32==223
pyreadline==2.1
python-archive==0.2
python-dateutil==2.7.3
pytz==2016.7
pywavelets==1.0.3
pywin32==224
pywinpty==0.5.5
PyYAML==5.1.2
pyzmq==17.1.2
qtconsole==4.4.3
queuelib==1.5.0
rabbitmq==0.2.0
redis==2.10.6
reportlab==3.5.23
requests==2.22.0
retrying==1.3.3
rope==0.12.0
rsa==4.0
ruamel.yaml==0.16.5
ruamel.yaml.clib==0.2.0
scikit-learn==0.21.2
scipy==1.2.1
Scrapy==1.7.3
seaborn==0.9.0
selenium==3.141.0
Send2Trash==1.5.0
service-identity==18.1.0
singledispatch==3.4.0.3
six==1.12.0
sklearn==0.0
sorl-thumbnail==12.5.0
South==1.0
sqlparse==0.2.4
stanfordcorenlp==3.9.1.1
statsmodels==0.10.1
stevedore==1.30.0
tabulate==0.8.5
tensorboard==2.0.0
tensorflow==2.0.0
tensorflow-estimator==2.0.0
tensorflow-tensorboard==0.1.8
termcolor==1.1.0
terminado==0.8.1
testpath==0.4.2
text-unidecode==1.3
toml==0.10.0
tornado==5.1.1
tqdm==4.35.0
traitlets==4.3.2
Twisted==19.7.0
txaio==18.8.1
typed-ast==1.1.0
tzlocal==2.0.0
urllib3==1.23
vine==1.3.0
virtualenv==16.2.0
virtualenv-clone==0.5.0
virtualenvwrapper==4.8.2
w3lib==1.20.0
wcwidth==0.1.7
webencodings==0.5.1
Werkzeug==0.12.2
whitenoise==4.1.2
widgetsnbextension==3.4.2
wrapt==1.11.2
xmltodict==0.12.0
yarg==0.1.9
zope.interface==4.6.0

pip freeze - on server debian 10 Output:

absl-py==0.8.1
aioredis==1.3.0
amqp==1.4.9
anyjson==0.3.3
asgi-redis==1.4.3
asgiref==1.1.2
asn1crypto==0.24.0
astor==0.8.0
astropy==3.1.2
async-timeout==3.0.1
atomicwrites==1.1.5
attrs==18.2.0
autobahn==19.10.1
Automat==0.8.0
billiard==3.3.0.23
bleach==3.1.0
blist==1.3.6
boto==2.49.0
BTrees==4.3.1
cachetools==3.1.1
celery==3.1.19
certifi==2019.9.11
cffi==1.13.1
channels==1.1.8
chardet==3.0.4
colorama==0.4.1
configobj==5.0.6
constantly==15.1.0
cryptography==2.8
cycler==0.10.0
Cython==0.29.2
daphne==1.4.2
dataclasses==0.6
decorator==4.3.0
defusedxml==0.6.0
dicttoxml==1.7.4
distro==1.3.0
distro-info==0.21
dj-database-url==0.5.0
dj-environ==0.0.1
Django==2.2.4
django-extensions==2.2.5
django-mongoengine==0.3.3
django-redis==4.10.0
django-request-logging==0.7.0
django-storages==1.7.2
djangorestframework==3.10.3
djongo==1.2.36
easy-thumbnails==2.6
entrypoints==0.3
ExifRead==2.1.2
gast==0.2.2
gevent==1.3.7
google-api-core==1.14.3
google-auth==1.6.3
google-auth-oauthlib==0.4.1
google-cloud-core==1.0.3
google-cloud-storage==1.23.0
google-compute-engine==20190916.0
google-pasta==0.1.7
google-resumable-media==0.5.0
googleapis-common-protos==1.6.0
greenlet==0.4.15
grpcio==1.16.1
gssapi==1.4.1
gunicorn==19.9.0
h5py==2.10.0
hidapi==0.7.99.post21
hiredis==1.0.0
http-parser==0.8.3
hyperlink==19.0.0
idna==2.8
imagededup==0.1.0
importlib-metadata==0.23
incremental==17.5.0
indexed-gzip==0.8.6
ipykernel==5.1.3
ipython==5.8.0
ipython-genutils==0.2.0
ipywidgets==7.5.1
Jinja2==2.10
joblib==0.14.0
jpy==0.9.0
JPype1==0.6.3
jsonmodel==3.0
jsonschema==3.1.1
jupyter==1.0.0
jupyter-client==5.3.4
jupyter-console==6.0.0
jupyter-core==4.6.1
kdtree==0.7.1
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
keyring==17.1.1
keyrings.alt==3.1.1
kiwisolver==1.1.0
kombu==3.0.37
Kyoto-Cabinet==1.5
labpack==0.20
Markdown==3.1.1
MarkupSafe==1.1.0
matplotlib==3.1.1
mistune==0.8.4
mongoengine==0.18.2
more-itertools==4.2.0
msgpack==0.6.2
msgpack-python==0.5.6
nbconvert==5.6.1
nbformat==4.4.0
notebook==6.0.2
numpy==1.16.3
oauthlib==3.1.0
olefile==0.46
opt-einsum==3.1.0
pandas==0.23.4
pandocfilters==1.4.2
persistent==4.2.2
pexpect==4.6.0
pickleshare==0.7.5
Pillow==6.1.0
pluggy==0.8.0
ply==3.11
pocketlab==0.10
prelude==4.1.0
prometheus-client==0.7.1
prompt-toolkit==2.0.10
protobuf==3.10.0
psutil==5.5.1
psycopg2-binary==2.8.3
ptyprocess==0.6.0
py==1.7.0
py-radix==0.10.0
pyalsa==1.1.6
pyasn1==0.4.2
pyasn1-modules==0.2.1
PyAudio==0.2.11
PyBluez==0.22
pycairo==1.16.2
pycparser==2.19
pycrypto==2.6.1
pycups==1.9.73
pyexiv2==1.3.0
Pygments==2.3.1
PyGObject==3.30.4
PyHamcrest==1.9.0
PyICU==2.2
PyJWT==1.7.1
pymad==0.10
pymongo==3.9.0
pyodbc==4.0.22
pyparsing==2.4.2
pypcap==1.2.2
pyraf==2.1.15
pyrsistent==0.15.5
pysendfile==2.0.1
pySFML==2.2.0
pysmbc==1.0.15.6
pytango==9.2.5
pytest==3.10.1
pytest-arraydiff==0.3
pytest-astropy==0.5.0
pytest-doctestplus==0.2.0
pytest-openfiles==0.3.2
pytest-remotedata==0.3.1
python-apt==1.8.4
python-archive==0.2
python-casacore==3.0.0
python-dateutil==2.8.0
python-ldap==3.1.0
python-sane==2.8.3
pytz==2019.3
PyWavelets==1.0.3
pyxattr==0.6.1
pyxdg==0.25
PyYAML==5.1.2
pyzmq==18.1.1
qrencode==1.2
qtconsole==4.5.5
redis==3.2.1
requests==2.22.0
requests-oauthlib==1.2.0
rpy2==2.9.5
rsa==4.0
ruamel.yaml==0.16.5
ruamel.yaml.clib==0.2.0
scikit-learn==0.21.2
scipy==1.2.1
SecretStorage==2.3.1
Send2Trash==1.5.0
simplegeneric==0.8.1
six==1.12.0
sorl-thumbnail==12.5.0
sqlparse==0.2.4
stsci.tools==3.4.13
subvertpy==0.10.1
supervisor==4.1.0
tabulate==0.8.5
tb-nightly==1.14.0a20190301
tensorboard==2.0.1
tensorflow==2.0.0
tensorflow-estimator==2.0.1
termcolor==1.1.0
terminado==0.8.3
testpath==0.4.4
tf-estimator-nightly==1.14.0.dev2019030115
tornado==6.0.3
tqdm==4.35.0
traitlets==4.3.2
Twisted==19.7.0
txaio==18.8.1
tzlocal==2.0.0
ufw==0.36
unattended-upgrades==0.1
urllib3==1.25.6
urwid==2.0.1
uWSGI==2.0.18
vine==1.3.0
virtualenv==16.7.7
wcwidth==0.1.7
webencodings==0.5.1
Werkzeug==0.16.0
wheezy.template==0.1.167
whitenoise==4.1.2
widgetsnbextension==3.5.1
wrapt==1.11.2
xmltodict==0.12.0
yenc==0.4.0
zipp==0.6.0
zodbpickle==1.0
zope.interface==4.6.0

Other Dependencies

N/A

Minimally Reproducible Test Case

Expected Behavior

in windows it works perfectly, and it handles all requested tasks using channels 1.x via websocket

Actual Behavior

Celery==3.1.19 works on windows-8.1 perfectly along with the following packages:

Django==2.2.4

channels==1.1.8

daphne==1.4.2

redis==3.2.1

asgi-redis==1.4.3

asgiref==1.1.2

the commits of the the local django app have been pushed to a debian-10 Linux, all requirements are installed perfectly.

the following configurations in the folder core that lives next to manage.py:

│ asgi.py
│ celery.py
│ helper.py
│ urls.py
│ wsgi.py
│ __init__.py (first)
│
├───settings
│ │ __init__.py (second)
│ │
│ ├───project
│ │ │ constants.py
│ │ │ django.py
│ │ │ exceptions.py
│ │ │ extra_django.py
│ │ │ local.py
│ │ │ logger.py
│ │ │ production.py
│ │ │ run.py
│ │ │ __init__.py
│ │ │
│ │ └───__pycache__
│ ├───third_parties
│ │ │ celery.py
│ │ │ ws4redis.py
│ │ │ __init__.py

init.py (first):

from .celery import app as celery_app
__all__ = ['celery_app']

init.py (second)

from core.settings.third_parties.celery import *

core\settings\third_parties\celery.py

from . import env
BROKER_URL = env.str('REDIS_URL') +'/0'
CELERY_RESULT_BACKEND = env.str('REDIS_URL') +'/0'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
# CELERY_ALWAYS_EAGER = True

asgi.py

import os
import channels.asgi

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "core.settings")
channel_layer = channels.asgi.get_channel_layer()

some settings:

CHANNEL_LAYERS = {
"default": {
"BACKEND": "asgi_redis.RedisChannelLayer", # use redis backend
"CONFIG": {
"hosts": [env.str('REDIS_URL', default='redis://localhost:6379')], # set redis address
},
# # load routing from our routing.py file ---> deprecated in Channel-2
# Update: for many problems occurred with channels 2, we considered channels 1.1.8
# "ROUTING": "core.settings.third_parties.routing.channel_routing",
"ROUTING": "sources.routing.channel_routing",
},
}
SESSION_EXPIRE_AT_BROWSER_CLOSE = True
# SESSION_COOKIE_SECURE = False

AUTH_USER_MODEL = "accounts.User"

ASGI_APPLICATION = "core.asgi"

Summary:

this is the output of runnging any celery command:

File "/usr/local/lib/python3.7/dist-packages/celery/concurrency/prefork.py", line 20, in <module>
from celery.concurrency.base import BasePool
File "/usr/local/lib/python3.7/dist-packages/celery/concurrency/base.py", line 21, in <module>
from celery.utils import timer2
File "/usr/local/lib/python3.7/dist-packages/celery/utils/timer2.py", line 19
from kombu.async.timer import Entry, Timer as Schedule, to_timestamp, logger
^
SyntaxError: invalid syntax

Note:

I can't use latest version of Celery 4.x becuase its dependencies conflicts with the current needed packahes with channels 1.x, plus when I tried to install celery 4.x on the linux server it requires redis later than 2.10.6 which is required by asgi_redis

I have wasted much time trying to find a solution, thanks for every help I apprecuate your comments

Issue 2:
I suggest to pass the default arguments to the message that gets into the broker.

Having a simple task:

@app.task()
def hello(arg1, arg2=1, arg=2):
pass

hello.delay('test')

And using RabbitMQ as a message broker I see the queued message:

[["test"], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]

Clearly, the default values for arg2 and arg3 are missing in the message.

This causes us trouble, because we interface celery from C++. Missing default values can be restored in the C++ part (python does this natively), but it causes a code duplication. When default value changes, this must be reflected at all places accordingly.

I think that for the portability and robustness of the celery framework, there should be some way of specifying that the default parameters should be included in the message that gets to the message broker.

The produced message should look like this:

[["test", 1, 2], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]

eventually, it can be passed in the kwargs dictionary.