Issue 1:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first. https://groups.google.com/forum/#!topic/celery-users/TBCxDemhSdE

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

#5527, #4454

Possible Duplicates

None

Environment & Settings

Celery version: 4.4.0

celery report Output:

$ celery -A celery_tacks report

software -> celery:4.4.0 (cliffs) kombu:4.6.7 py:3.8.1
billiard:3.6.1.0 py-amqp:2.5.2
platform -> system:Linux arch:64bit, ELF
kernel version:4.15.0-47-generic imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:pyamqp results:mongodb+srv://remains:**@mongo.example.com/database_name

broker_url: 'amqp://remains:********@rabbitmq.example.com:5672//'
result_backend: 'mongodb+srv://remains:********@mongo.example.com/database_name'
include: [
'store_remains', 'providers', 'mongo_utils', 'afinalux', 'afinalux-diff',
'christmass', 'danio', 'danio-diff', 'elgo-msk', 'evan-diff', 'globus',
'lightstar', 'lussole', 'mw-light', 'prorab', 'quadro-stroy', 'sds-group',
'sonex', 'stluce', 'stluce-diff', 'technolight', 'contactplus', 'lex-c', 'lex-c-diff',
'greentrees', 'lifan-moto', 'triumph-nord-rus', 'crawl_site', 'mail2csv', 'mail2diff',
'rotate_mb', 'store_out_of_stock'
]
accept_content: [
'mongo_json', 'application/x-mongo-json', 'json', 'application/json'
]
task_serializer: 'mongo_json'
result_serializer: 'mongo_json'
task_default_queue: 'remains_tasks'
worker_prefetch_multiplier: 1
worker_max_tasks_per_child: 45
worker_hijack_root_logger: False
mongodb_backend_settings: {
'compressors': 'snappy', 'database': '********'}
CELERY_MONGO_SCHEDULER_URI: 'mongodb+srv://remains:********@mongo.example.com/database_name?ssl=false'
CELERY_MONGO_SCHEDULER_DB: 'remains'
CELERY_MONGO_SCHEDULER_ENTRIES_COL: 'schedules_entries'
CELERY_MONGO_SCHEDULER_META_COL: 'schedules_meta'
beat_schedule: {
'afinalux': { 'args': ['afinalux'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 7 0 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'Афина Гарден'},
'afinalux-diff': { 'args': ['afinalux-diff'],
'comment': 'рассылка с изменениями прайса Афина '
'Гарден',
'is_enabled': True,
'schedule': <crontab: 7 0 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'Афина Гарден'},
'allpools': { 'args': ['allpools'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 19 1 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_site',
'title': 'Олпулс'},
'aquapolis': { 'args': ['aquapolis'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 31 0 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_site',
'title': 'Акваполис'},
'aquapolis_bestway': { 'args': ['aquapolis_bestway'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 31 0 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_site',
'title': 'Акваполис_Bestway'},
'audiomania': { 'args': ['audiomania'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 17 0 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_site',
'title': 'Аудиомания'},
'christmass': { 'args': ['christmass'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 13-59/30 * * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'Актив групп'},
'contactplus': { 'args': ['contactplus'],
'comment': 'Остатки Контакт Плюс',
'is_enabled': True,
'schedule': <crontab: 17-59/30 * * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'Контакт Плюс'},
'danio': { 'args': ['danio'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 13 2 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'Данио'},
'danio-diff': { 'args': ['danio-diff'],
'comment': 'Рассылка с изменениями прайса Данио',
'is_enabled': True,
'schedule': <crontab: 17 2 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'Данио'},
'decomo': { 'args': ['decomo'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 23 0 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_site',
'title': 'Декомо'},
'delta': { 'args': ['delta'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 19 0 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_site',
'title': 'Дельта'},
'elgo-msk': { 'args': ['elgo-msk'],
'comment': 'Отключено по просьбе заказчика',
'is_enabled': False,
'schedule': <crontab: 29-59/30 * * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'Эгло'},
'evan-diff': { 'args': ['evan-diff'],
'comment': 'Рассылка с изменениями прайса Эван',
'is_enabled': True,
'schedule': <crontab: 17 3 0 * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'Эван'},
'floraimport': { 'args': ['floraimport'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 11 0 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_site',
'title': 'Флора Импорт'},
'globus': { 'args': ['globus'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 5-59/30 * * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'Глобус'},
'greentrees': { 'args': ['greentrees'],
'comment': 'Остатки Green Trees',
'is_enabled': True,
'schedule': <crontab: 37 1 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'Green Trees'},
'housetech': { 'args': ['housetech'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 13 0 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_site',
'title': 'Домотехника'},
'housetech_nsk': { 'args': ['housetech_nsk'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 23 0 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_site',
'title': 'Домотехника_НСК'},
'housetech_spb': { 'args': ['housetech_spb'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 31 0 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_site',
'title': 'Домотехника_СПБ'},
'huter-gmbh': { 'args': ['huter-gmbh'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 43 0 0 * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_site',
'title': 'Континент ГК'},
'iatec': { 'args': ['iatec'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 41 0 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_site',
'title': 'Три Пятых'},
'interinstrument': { 'args': ['interinstrument'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 47 0 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_site',
'title': 'ИнтерИнструмент'},
'lex-c': { 'args': ['lex-c'],
'comment': 'Остатки Лекс-с',
'is_enabled': True,
'schedule': <crontab: 21 2 0 * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'Лекс-С'},
'lex-c-diff': { 'args': ['lex-c-diff'],
'comment': 'Рассылка с изменениями прайса Лекс-с',
'is_enabled': True,
'schedule': <crontab: 23 2 0 * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'Лекс-С'},
'lifan-moto': { 'args': ['lifan-moto'],
'comment': 'Остатки lifan-moto',
'is_enabled': True,
'schedule': <crontab: 37 1 0 * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'Мото северо-запад'},
'lightstar': { 'args': ['lightstar'],
'comment': 'Отключено по просьбе заказчика',
'is_enabled': False,
'schedule': <crontab: 17-59/30 * * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'Лайтстар'},
'lussole': { 'args': ['lussole'],
'comment': 'Отключено по просьбе заказчика',
'is_enabled': False,
'schedule': <crontab: 29 */2 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'Lussole'},
'mail2csv': { 'is_enabled': True,
'is_system': True,
'schedule': <crontab: 3-59/30 * * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_mail2csv',
'title': 'Остатки из писем'},
'mail2diff': { 'is_enabled': True,
'is_system': True,
'schedule': <crontab: 13 * * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_mail2diff',
'title': 'Diff прайса из писем'},
'mon_site': { 'args': ['poolfactory'],
'is_enabled': False,
'is_system': False,
'schedule': <crontab: 23 3 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_mon_site',
'title': 'Мониторинг продаж с poolfactory'},
'mw-light': { 'args': ['mw-light'],
'comment': 'Отключено по просьбе заказчика',
'is_enabled': False,
'schedule': <crontab: 1-59/30 * * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'MW-Light'},
'novum': { 'args': ['novum'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 7 0 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_site',
'title': 'Novum'},
'pompina': { 'args': ['pompina'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 13 0 0 * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_site',
'title': 'Саблайн сервис'},
'price.allpools.ru': { 'args': ['allpools.ru'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 31 3 0 * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_price_site',
'title': 'Цены с сайта allpools.ru'},
'price.aquapolis.ru': { 'args': ['aquapolis.ru'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 29 3 0 * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_price_site',
'title': 'Цены с сайта aquapolis.ru'},
'price.decomo': { 'args': ['decomo'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 13 1 * 1 * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_price_site',
'title': 'Цены с сайта decomo'},
'price.decomo_rrc': { 'args': ['decomo_rrc'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 13 1 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_price_site',
'title': 'Цены с сайта decomo_rrc'},
'price.ekopromgroup.ru': { 'args': ['ekopromgroup.ru'],
'comment': 'Цены ЭкоПром',
'is_enabled': True,
'schedule': <crontab: 41 3 0 * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_price_site',
'title': 'Цены с сайта ekopromgroup.ru'},
'price.floraimport': { 'args': ['floraimport'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 11 0 0 1-7,15-21,29-31 * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_price_site',
'title': 'Цены с сайта floraimport'},
'price.kamnirus': { 'args': ['kamnirus'],
'comment': 'Отключено по просьбе заказчика',
'is_enabled': False,
'schedule': <crontab: 19 3 0 * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_price_site',
'title': 'Цены с сайта kamnirus'},
'price.novum': { 'args': ['novum'],
'comment': 'Цены Novum',
'is_enabled': False,
'schedule': <crontab: 17 2 0 * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_price_site',
'title': 'Цены с сайта novum'},
'price.polimer-group.com': { 'args': ['polimer-group.com'],
'comment': 'Цены Polimer Group',
'is_enabled': True,
'schedule': <crontab: 37 3 0 * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_price_site',
'title': 'Цены с сайта polimer-group.com'},
'price.prorab': { 'args': ['prorab'],
'comment': '',
'is_enabled': False,
'schedule': <crontab: 37 3 0 * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_price_site',
'title': 'Цены с сайта prorab'},
'price.unisaw': { 'args': ['unisaw'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 23 3 0 * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_price_site',
'title': 'Цены с сайта unisaw'},
'price.Домотехника': { 'args': ['Домотехника'],
'comment': 'Изменения в составе прайсов',
'is_enabled': True,
'schedule': <crontab: 41 3 0 * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_price_site',
'title': 'Цены с сайта Домотехника'},
'prorab': { 'args': ['prorab'],
'comment': '',
'is_enabled': False,
'schedule': <crontab: 7-59/30 * * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'Prorab'},
'quadro-stroy': { 'args': ['quadro-stroy'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 17 1 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'Квадро-Строй'},
'rotate_mb': { 'is_enabled': True,
'is_system': True,
'schedule': <crontab: 23 0 0 * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.rotate_mb',
'title': 'Очистка устаревших файлов'},
'sds-group': { 'args': ['sds-group'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 29 0 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'СДС-Группа'},
'sima-land': { 'args': ['sima-land'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 53 0 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_site',
'title': 'Сима-ленд'},
'sonex': { 'args': ['sonex'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 11-59/30 * * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'Сонекс'},
'stluce': { 'args': ['stluce'],
'comment': 'Остатки ST Luce Velante',
'is_enabled': True,
'schedule': <crontab: 19 1 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'ST Luce'},
'stluce-diff': { 'args': ['stluce-diff'],
'comment': 'рассылка с изменениями прайса ST Luce '
'Velante',
'is_enabled': True,
'schedule': <crontab: 23 1 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'ST Luce'},
'store_outofstock': { 'is_enabled': True,
'is_system': True,
'schedule': <crontab: 49 0-20,23 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.store_outofstock',
'title': 'Автозаливка "нет в наличии"'},
'technolight': { 'args': ['technolight'],
'comment': 'Отключено по просьбе заказчика',
'is_enabled': False,
'schedule': <crontab: 19-59/30 * * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'Тренд Света'},
'triumph-nord-rus': { 'args': ['triumph-nord-rus'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 11 1 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_link',
'title': 'Триумф Норд Рус'},
'unisaw': { 'args': ['unisaw'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 3 0 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_site',
'title': 'Unisaw'},
'vodalux-fontan': { 'args': ['vodalux-fontan'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 37 0 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_site',
'title': 'Водалюкс'},
'vodalux-prud': { 'args': ['vodalux-prud'],
'comment': '',
'is_enabled': True,
'schedule': <crontab: 13 0 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_site',
'title': 'Водалюкс'},
'y-skazki': { 'args': ['y-skazki'],
'comment': 'Отключено по просьбе заказчика',
'is_enabled': False,
'schedule': <crontab: 17 1 * * * (m/h/d/dM/MY)>,
'task': 'celery_tacks.load_site',
'title': 'Bestway Юг'}}

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.7.4

Minimal Celery Version: 4.4.0

Minimal Kombu Version: 4.6.7

Minimal Broker Version: rabbitmq-server 3.6.15-1

Minimal Result Backend Version: mongodb 4.2.0

Minimal OS and/or Kernel Version: ubuntu 18.04

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

amqp==2.5.2
attrs==19.3.0
Automat==0.8.0
billiard==3.6.1.0
Brotli==1.0.7
cchardet==2.1.5
celery==4.4.0
certifi==2019.11.28
cffi==1.13.2
chardet==3.0.4
Click==7.0
clickhouse-cityhash==1.0.2.3
clickhouse-driver==0.1.2
constantly==15.1.0
cryptography==2.8
cssselect==1.1.0
dnspython==1.16.0
et-xmlfile==1.0.1
fake-useragent==0.1.11
fastnumbers==3.0.0
hyperlink==19.0.0
idna==2.8
importlib-metadata==1.3.0
incremental==17.5.0
jdcal==1.4.1
jmespath==0.9.4
jsonlines==1.2.0
kombu==4.6.7
lxml==4.4.2
lz4==3.0.2
marshmallow==2.20.5
more-itertools==8.0.2
mysqlclient==1.4.6
natsort==7.0.0
openpyxl==3.0.3
parsel==1.5.2
pip-tools==4.3.0
Protego==0.1.16
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.19
PyDispatcher==2.0.5
PyHamcrest==1.10.1
PyICU==2.4.2
pymongo==3.10.1
pyOpenSSL==19.1.0
python-dateutil==2.8.1
python-snappy==0.5.4
pytz==2019.3
pyxlsb==1.0.6
PyYAML==5.3
queuelib==1.5.0
requests==2.22.0
Scrapy==1.8.0
sentry-sdk==0.14.0
service-identity==18.1.0
six==1.14.0
Twisted==19.10.0
txmongo==19.2.0
tzlocal==2.0.0
urllib3==1.25.7
vine==1.3.0
w3lib==1.21.0
xlrd==1.2.0
XlsxWriter==1.2.7
zipp==0.6.0
zope.interface==4.7.1
zstandard==0.13.0
zstd==1.4.4.0

Other Dependencies

```
```

Minimally Reproducible Test Case

Code for initialization celery app:

from functools import partial
from bson import json_util

json_options=json_util.JSONOptions(
json_mode=json_util.JSONMode.RELAXED, tz_aware=True, tzinfo=utc)
json_dump = partial(
json_util.dumps, ensure_ascii=False, json_options=json_options)
json_load = partial(json_util.loads, json_options=json_options)

register(
'mongo_json',
encoder=(lambda obj: json_dump(obj).encode('utf-8')),
decoder=json_load,
content_type='application/x-mongo-json')

app = Celery(
'remains_tasks',
broker='pyamqp://...',
backend = 'mongodb://...',
include=[
# many modules
]
)
app.conf.update(
accept_content=[
'mongo_json', 'application/x-mongo-json', 'json', 'application/json'],
task_serializer='mongo_json',
result_serializer='mongo_json',
task_default_queue='remains_tasks',
worker_prefetch_multiplier=1,
worker_max_tasks_per_child=45,
worker_hijack_root_logger=False,

mongodb_backend_settings=dict(database='remains', compressors='snappy'),
)

Connection string look:

mongodb+srv://username:password@mongo.example.com/database?ssl=false

Expected Behavior

No exception

Actual Behavior

Exception after exit tack

KeyError: 'collection'
File "kombu/utils/objects.py", line 42, in __get__
return obj.__dict__[self.__name__]
KeyError: 'database'
File "kombu/utils/objects.py", line 42, in __get__
return obj.__dict__[self.__name__]
OperationFailure: Authentication failed.
File "billiard/pool.py", line 1791, in safe_apply_callback
fun(*args, **kwargs)
File "celery/worker/request.py", line 526, in on_failure
self.task.backend.mark_as_failure(
File "celery/backends/base.py", line 159, in mark_as_failure
self.store_result(task_id, exc, state,
File "celery/backends/base.py", line 406, in store_result
self._store_result(task_id, result, state, traceback,
File "celery/backends/mongodb.py", line 194, in _store_result
self.collection.replace_one({'_id': task_id}, meta, upsert=True)
File "kombu/utils/objects.py", line 44, in __get__
value = obj.__dict__[self.__name__] = self.__get(obj)
File "celery/backends/mongodb.py", line 291, in collection
collection = self.database[self.taskmeta_collection]
File "kombu/utils/objects.py", line 44, in __get__
value = obj.__dict__[self.__name__] = self.__get(obj)
File "celery/backends/mongodb.py", line 286, in database
return self._get_database()
File "celery/backends/mongodb.py", line 275, in _get_database
if not db.authenticate(self.user, self.password, source=source):
File "pymongo/database.py", line 1468, in authenticate
self.client._cache_credentials(
File "pymongo/mongo_client.py", line 754, in _cache_credentials
with server.get_socket(all_credentials) as sock_info:
File "contextlib.py", line 113, in __enter__
return next(self.gen)
File "pymongo/pool.py", line 1138, in get_socket
sock_info.check_auth(all_credentials)
File "pymongo/pool.py", line 712, in check_auth
auth.authenticate(credentials, self)
File "pymongo/auth.py", line 564, in authenticate
auth_func(credentials, sock_info)
File "pymongo/auth.py", line 539, in _authenticate_default
return _authenticate_scram(credentials, sock_info, 'SCRAM-SHA-1')
File "pymongo/auth.py", line 263, in _authenticate_scram
res = sock_info.command(source, cmd)
File "pymongo/pool.py", line 603, in command
return command(self.sock, dbname, spec, slave_ok,
File "pymongo/network.py", line 165, in command
helpers._check_command_response(
File "pymongo/helpers.py", line 159, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)

Issue 2:
PR for proposed fix to this issue: #6020

Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

94dae1b

This PR was made to address the following issue, which has resulted in the issue I am having now. #1930

Related Issues

#1930

Possible Duplicates

None

Environment & Settings

Celery version: celery>=4.0.0 (using it in Airflow)

Steps to Reproduce

(see Minimally Reproducible Test Case for step by step commands. This contains information leading to the issue and a proposed fix)

In Airflow, you can set celery configs. I was setting up cloudsql to use a private IP instead of a proxy. Currently, we use mysql as the results_backend. Changing the host address from local host to the private IP caused some errors, as expected.

OperationalError: (_mysql_exceptions.OperationalError) (1045, "Access denied for user 'airflow'@'10.x.x.xxx' (using password: YES)")

In order to use the private IP, I need to use the SSL cert, key, and ca. I confirmed that by logging into the Airflow worker and scheduler pods that my url and engine arg params worked.

from airflow.models import DagRun
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
e = create_engine({AIRFLOW__CELERY__SQL_ALCHEMY_CONN},connect_args= {'ssl': {'ca': '/path-to-mysql-sslcert/server-ca', 'cert': '/path-to-mysql-sslcert/client-cert', 'key': '/path-to-mysql-sslcert/client-key'}})
s = scoped_session(sessionmaker(autocommit=False, autoflush=False, bind=e))
s.query(DagRun).all()

This worked fine, so I know that the my ssl certs are accessible, the engine can be created, and a session used. Non-celery mysql connections no longer gave an error.

The Celery documentation (https://docs.celeryproject.org/en/stable/userguide/configuration.html#conf-database-result-backend) outlines how to add engine args to via database_engine_options. Therefore, I added

'database_engine_options': {
'connect_args': {'ssl': {'ca': '/path-to-mysql-sslcert/server-ca', 'cert': '/path-to-mysql-sslcert/client-cert', 'key': '/path-to-mysql-sslcert/client-key'}}}

However, I still get the same error.

OperationalError: (_mysql_exceptions.OperationalError) (1045, "Access denied for user 'airflow'@'10.x.x.xxx' (using password: YES)")

Additionally, I get logs in the scheduler like the following:

{{__init__.py:56}} WARNING - Failed operation _get_task_meta_for. Retrying 1 more times.
68918-Traceback (most recent call last):
68919- File "/usr/local/lib/python2.7/dist-packages/celery/backends/database/__init__.py", line 51, in _inner
68920- return fun(*args, **kwargs)
68921- File "/usr/local/lib/python2.7/dist-packages/celery/backends/database/__init__.py", line 154, in _get_task_meta_for
68922: session = self.ResultSession()
68923: File "/usr/local/lib/python2.7/dist-packages/celery/backends/database/__init__.py", line 113, in ResultSession
68924- **self.engine_options)
68925- File "/usr/local/lib/python2.7/dist-packages/celery/backends/database/session.py", line 59, in session_factory
68926- self.prepare_models(engine)
68927- File "/usr/local/lib/python2.7/dist-packages/celery/backends/database/session.py", line 54, in prepare_models
68928- ResultModelBase.metadata.create_all(engine)

After digging through the code with @dangermike, we noticed that get_engine will not use the kwargs passed to it unless it has been forked.(https://github.com/celery/celery/blob/master/celery/backends/database/session.py#L34) Therefore, the SSL params will not be passed in our case. The only place that self.forked = True is after the fork cleanup session. This used to not be the case (94dae1b), but after an issue was made about passing pool_size (#1930), **kwargs were taken out of create_engine() entirely.

Possibly something like the following would allow for kwargs to be passed in, while still addressing the pool params issue.

class SessionManager(object):
# ...
def get_engine(self, dburi, **kwargs):
if self.forked:
try:
return self._engines[dburi]
except KeyError:
engine = self._engines[dburi] = create_engine(dburi, **kwargs)
return engine
else:
kwargs = dict([(k, v) for k, v in kwargs.items() if not k.startswith('pool')])
return create_engine(dburi, poolclass=NullPool, **kwargs)

where kwargs = dict([(k, v) for k, v in kwargs.items() if not k.startswith('pool')]) omits any pool args while keeping the rest.

Required Dependencies

Minimal Python Version: >=2.7

Minimal Celery Version: >=4.0.0

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

Used Airflow

Other Dependencies

N/A

Minimally Reproducible Test Case

In a python shell,

get the url with a private mysql IP to make result_backend, giving something like db+mysql://airflow:***@10.x.xx.xx/airflow

and the celery config

celery_configuration =
{'broker_transport_options': {'visibility_timeout': 21600},
'result_serializer': 'pickle',
'task_acks_late': True,
'database_engine_options': { 'connect_args': {'ssl': {'ca': '/path-to-mysql-sslcert/server-ca', 'cert': '/path-to-mysql-sslcert/client-cert', 'key': '/path-to-mysql-sslcert/client-key'}}},
'task_default_queue': 'default',
'worker_concurrency': 32,
'worker_prefetch_multiplier': 1,
'event_serializer': 'json',
'accept_content': ['json', 'pickle'],
'broker_url': 'redis://{URL}/1',
'result_backend': 'db+mysql://airflow:***@10.x.xx.xx/airflow',
'task_default_exchange': 'default'}

the line most important here is:

'database_engine_options': { 'connect_args': {'ssl': {'ca': '/path-to-mysql-sslcert/server-ca', 'cert': '/path-to-mysql-sslcert/client-cert', 'key': '/path-to-mysql-sslcert/client-key'}}}

then try to connect to result_backend by creating app.

app = Celery(celery_app_name=airflow.executors.celery_executor,
config_source=celery_configuration)

create a database backend

dbbe = database.DatabaseBackend(url={results_backend url without the 'db+' in the beginning}, engine_options=celery_configuration['database_engine_options'], app=app)

and you will get the error again

sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1045, "Access denied for user 'airflow'@'10.xx.xx.xxx' (using password: YES)")
(Background on this error at: http://sqlalche.me/e/e3q8)

Expected Behavior

It seems like the expected behavior here would be for the connection to be successful and use the SSL certs in the **kwargs passed into get_engine.

Actual Behavior

Since self.fork is not True, and will not be True, create_engine is made by:

return create_engine(dburi, poolclass=NullPool)

since the SSL certs are not included, an error is returned and the connection is not successful.

sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1045, "Access denied for user 'airflow'@'10.xx.xx.xxx' (using password: YES)")
(Background on this error at: http://sqlalche.me/e/e3q8)