Issue 1:
Checklist

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version:

celery report Output:

software -> celery:4.3.0 (rhubarb) kombu:4.6.0 py:3.6.8
billiard:3.6.0.0 redis:3.2.0
platform -> system:Linux arch:64bit
kernel version:4.15.0-50-generic imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:django-db

Steps to Reproduce

Run a celery worker with a MySQL backend (possibly through Django).

After the worker is up, shut down the MySQL service.

Try running a task.

Required Dependencies

Minimal Python Version: N/A or Unknown

Minimal Celery Version: 4.2.0

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

celery==4.3.0
Django==2.2.1
mysqlclient==1.4.2

(only relevant packages)

Other Dependencies

N/A

Minimally Reproducible Test Case

Expected Behavior

This is a non-recoverable error. The worker process should terminate, which would be treated as an error by a monitoring tool that will restart it.

Actual Behavior

An exception is printed to stderr. The worker itself keeps running, receiving tasks, but never running them.

An exception is printed after each "Task accepted" log message.

[ERROR/MainProcess] Pool callback raised exception: OperationalError(2002, "Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)")
Traceback (most recent call last):
File "[REDACTED]/lib/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "[REDACTED]/lib/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "[REDACTED]/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 227, in get_new_connection
return Database.connect(**conn_params)
File "[REDACTED]/lib/python3.6/site-packages/MySQLdb/__init__.py", line 84, in Connect
return Connection(*args, **kwargs)
File "[REDACTED]/lib/python3.6/site-packages/MySQLdb/connections.py", line 164, in __init__
super(Connection, self).__init__(*args, **kwargs2)
MySQLdb._exceptions.OperationalError: (2002, "Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "[REDACTED]/lib/python3.6/site-packages/billiard/pool.py", line 1750, in safe_apply_callback
fun(*args, **kwargs)
File "[REDACTED]/lib/python3.6/site-packages/celery/worker/request.py", line 371, in on_failure
store_result=self.store_errors,
File "[REDACTED]/lib/python3.6/site-packages/celery/backends/base.py", line 160, in mark_as_failure
traceback=traceback, request=request)
File "[REDACTED]/lib/python3.6/site-packages/celery/backends/base.py", line 342, in store_result
request=request, **kwargs)
File "[REDACTED]/lib/python3.6/site-packages/django_celery_results/backends/database.py", line 35, in _store_result
task_kwargs=task_kwargs,
File "[REDACTED]/lib/python3.6/site-packages/django_celery_results/managers.py", line 50, in _inner
return fun(*args, **kwargs)
File "[REDACTED]/lib/python3.6/site-packages/django_celery_results/managers.py", line 126, in store_result
obj, created = self.get_or_create(task_id=task_id, defaults=fields)
File "[REDACTED]/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "[REDACTED]/lib/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
return self.get(**kwargs), False
File "[REDACTED]/lib/python3.6/site-packages/django/db/models/query.py", line 402, in get
num = len(clone)
File "[REDACTED]/lib/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
self._fetch_all()
File "[REDACTED]/lib/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "[REDACTED]/lib/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "[REDACTED]/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
cursor = self.connection.cursor()
File "[REDACTED]/lib/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "[REDACTED]/lib/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "[REDACTED]/lib/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "[REDACTED]/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "[REDACTED]/lib/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "[REDACTED]/lib/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "[REDACTED]/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 227, in get_new_connection
return Database.connect(**conn_params)
File "[REDACTED]/lib/python3.6/site-packages/MySQLdb/__init__.py", line 84, in Connect
return Connection(*args, **kwargs)
File "[REDACTED]/lib/python3.6/site-packages/MySQLdb/connections.py", line 164, in __init__
super(Connection, self).__init__(*args, **kwargs2)
django.db.utils.OperationalError: (2002, "Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)")

This is unhelpful. Without looking at the logs you wouldn't know anything was wrong.

In the production setting, the exception is only printed as many times as there are concurrent workers, which makes it even harder to identify. I was not able to reproduce this particular behavior locally, and get an exception each time.

Issue 2:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version:

celery report Output:

software -> celery:5.0.0 (singularity) kombu:5.0.2 py:3.8.5
billiard:3.6.3.0 py-amqp:5.0.1
platform -> system:Darwin arch:64bit
kernel version:18.7.0 imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:amqp results:disabled

deprecated_settings: None

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown

Minimal Celery Version: 5.0.0

Minimal Kombu Version: 5.0.2

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

amqp==5.0.1
appdirs==1.4.4
appnope==0.1.0
asgiref==3.2.10
astor==0.8.1
attrs==20.2.0
backcall==0.2.0
bandit==1.6.2
bcrypt==3.2.0
billiard==3.6.3.0
blinker==1.4
celery==5.0.0
certifi==2020.6.20
cffi==1.14.3
chardet==3.0.4
click==7.1.2
click-default-group==1.2.2
click-didyoumean==0.0.3
click-repl==0.1.6
colorama==0.4.3
coreapi==2.3.3
coreschema==0.0.4
coverage==5.3
darglint==1.5.4
decorator==4.4.2
dennis==0.9
Django==3.0.10
django-admin-tools==0.9.0
django-autocomplete-light==3.5.0
django-coverage-plugin==1.8.0
django-csp==3.7
django-debug-toolbar==2.2
django-extensions==3.0.9
django-feature-policy==3.4.0
django-filter==2.4.0
django-health-check==3.14.1
django-http-referrer-policy==1.0.1
django-lifecycle==0.7.7
django-migration-linter==2.4.0
django-querycount==0.7.0
django-split-settings==1.0.1
django-stubs==1.5.0
django-test-migrations==1.0.0
djangorestframework==3.11.1
docutils==0.16
dotenv-linter==0.1.3
dparse==0.5.1
drf-yasg==1.17.1
eradicate==1.0
Faker==4.1.3
flake8==3.8.3
flake8-bandit==2.1.2
flake8-broken-line==0.2.1
flake8-bugbear==19.8.0
flake8-commas==2.0.0
flake8-comprehensions==3.2.3
flake8-debugger==3.2.1
flake8-django==1.1.1
flake8-docstrings==1.5.0
flake8-eradicate==0.3.0
flake8-isort==3.0.1
flake8-logging-format==0.6.0
flake8-plugin-utils==1.3.1
flake8-polyfill==1.0.2
flake8-pytest-style==1.3.0
flake8-quotes==2.1.2
flake8-rst-docstrings==0.0.12
flake8-string-format==0.2.3
flower==0.9.5
gitdb==4.0.5
GitPython==3.1.8
gunicorn==20.0.4
humanize==2.6.0
hypothesis==5.36.0
idna==2.10
importlib-metadata==1.1.3
inflection==0.5.1
iniconfig==1.0.1
ipython==7.18.1
ipython-genutils==0.2.0
isort==4.3.21
itypes==1.2.0
jedi==0.17.2
Jinja2==2.11.2
kombu==5.0.2
MarkupSafe==1.1.1
mccabe==0.6.1
more-itertools==8.5.0
mypy==0.770
mypy-extensions==0.4.3
nplusone==1.0.0
orjson==3.3.1
packaging==20.4
parso==0.7.1
pbr==5.5.0
pep8-naming==0.9.1
pexpect==4.8.0
pickleshare==0.7.5
pluggy==0.13.1
ply==3.11
polib==1.1.0
prometheus-client==0.8.0
prompt-toolkit==3.0.7
psycopg2==2.8.6
ptyprocess==0.6.0
py==1.9.0
pycodestyle==2.6.0
pycparser==2.20
pydantic==1.6.1
pydocstyle==5.1.1
pyflakes==2.2.0
Pygments==2.7.1
PyJWT==1.7.1
pyparsing==2.4.7
pytest==6.0.2
pytest-cov==2.10.1
pytest-deadfixtures==2.2.1
pytest-django==3.10.0
pytest-randomly==3.4.1
pytest-testmon==1.0.3
pytest-timeout==1.4.2
python-dateutil==2.8.1
python-decouple==3.3
pytz==2020.1
PyYAML==5.3.1
requests==2.24.0
restructuredtext-lint==1.3.1
ruamel.yaml==0.16.12
ruamel.yaml.clib==0.2.2
safety==1.9.0
sentry-sdk==0.17.8
six==1.15.0
smmap==3.0.4
snowballstemmer==2.0.0
sortedcontainers==2.2.2
sqlparse==0.3.1
stevedore==3.2.2
structlog==20.1.0
testfixtures==6.14.2
text-unidecode==1.3
toml==0.10.1
tornado==6.0.4
traitlets==5.0.4
typed-ast==1.4.1
typing-extensions==3.7.4.3
uritemplate==3.0.1
urllib3==1.25.10
urlman==1.3.0
vine==5.0.0
wcwidth==0.2.5
wemake-python-styleguide==0.14.1
wrapt==1.12.1
zipp==3.2.0

Other Dependencies

N/A

Minimally Reproducible Test Case

celery -A proj inspect ping -d celery@$HOSTNAME

Expected Behavior

-> celery@$HOSTNAME: OK
pong

Actual Behavior

Traceback (most recent call last):
File "/Users/aszubarev/Library/Caches/pypoetry/virtualenvs/service-notification-py-xHAuYcng-py3.8/bin/celery", line 8, in <module>
sys.exit(main())
File "/Users/aszubarev/Library/Caches/pypoetry/virtualenvs/service-notification-py-xHAuYcng-py3.8/lib/python3.8/site-packages/celery/__main__.py", line 15, in main
sys.exit(_main())
File "/Users/aszubarev/Library/Caches/pypoetry/virtualenvs/service-notification-py-xHAuYcng-py3.8/lib/python3.8/site-packages/celery/bin/celery.py", line 150, in main
return celery(auto_envvar_prefix="CELERY")
File "/Users/aszubarev/Library/Caches/pypoetry/virtualenvs/service-notification-py-xHAuYcng-py3.8/lib/python3.8/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/Users/aszubarev/Library/Caches/pypoetry/virtualenvs/service-notification-py-xHAuYcng-py3.8/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/Users/aszubarev/Library/Caches/pypoetry/virtualenvs/service-notification-py-xHAuYcng-py3.8/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/aszubarev/Library/Caches/pypoetry/virtualenvs/service-notification-py-xHAuYcng-py3.8/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/aszubarev/Library/Caches/pypoetry/virtualenvs/service-notification-py-xHAuYcng-py3.8/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/Users/aszubarev/Library/Caches/pypoetry/virtualenvs/service-notification-py-xHAuYcng-py3.8/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Users/aszubarev/Library/Caches/pypoetry/virtualenvs/service-notification-py-xHAuYcng-py3.8/lib/python3.8/site-packages/celery/bin/control.py", line 125, in inspect
replies = ctx.obj.app.control.inspect(timeout=timeout,
File "/Users/aszubarev/Library/Caches/pypoetry/virtualenvs/service-notification-py-xHAuYcng-py3.8/lib/python3.8/site-packages/celery/app/control.py", line 97, in _request
return self._prepare(self.app.control.broadcast(
File "/Users/aszubarev/Library/Caches/pypoetry/virtualenvs/service-notification-py-xHAuYcng-py3.8/lib/python3.8/site-packages/celery/app/control.py", line 477, in broadcast
return self.mailbox(conn)._broadcast(
File "/Users/aszubarev/Library/Caches/pypoetry/virtualenvs/service-notification-py-xHAuYcng-py3.8/lib/python3.8/site-packages/kombu/pidbox.py", line 319, in _broadcast
raise ValueError(
ValueError: destination must be a list/tuple not <class 'set'>
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit