Issue 1:
Checklist

I have verified that the issue exists against the master branch of Celery. (v5.0.2)

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

[x ] I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version: 5.0.2

celery report Output:

software -> celery:5.0.2 (singularity) kombu:5.0.2 py:3.6.9
billiard:3.6.3.0 redis:3.5.3
platform -> system:Linux arch:64bit, ELF
kernel version:5.4.0-53-generic imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://localhost/

broker_url: 'redis://localhost:6379//'
result_backend: 'redis://localhost/'
deprecated_settings: None
result_extended: True

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.6.9

Minimal Celery Version: 5.0.2

Minimal Kombu Version: 5.0.2

Minimal Broker Version: Redis server v=6.0.8 sha=00000000:0 malloc=jemalloc-5.1.0 bits=64 build=75cef67090587c6

Minimal Result Backend Version: Redis server v=6.0.8 sha=00000000:0 malloc=jemalloc-5.1.0 bits=64 build=75cef67090587c6

Minimal OS and/or Kernel Version: Linux Machine 5.4.0-53-generic #59~18.04.1-Ubuntu SMP Wed Oct 21 12:14:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

aioredis==1.3.1
alabaster==0.7.8
amqp==5.0.2
appdirs==1.4.4
apturl==0.5.2
argon2-cffi==20.1.0
asgiref==3.3.1
asn1crypto==0.24.0
astroid==1.6.0
async-generator==1.10
async-timeout==3.0.1
attrs==20.2.0
autobahn==20.7.1
Automat==20.2.0
Babel==2.3.4
backcall==0.2.0
beautifulsoup4==4.9.0
billiard==3.6.3.0
black==20.8b1
bleach==3.2.1
blinker==1.4
Brlapi==0.6.6
bs4==0.0.1
celery==5.0.2
certifi==2020.6.20
cffi==1.14.3
channels==2.4.0
channels-redis==3.1.0
chardet==3.0.4
click==7.1.2
click-didyoumean==0.0.3
click-repl==0.1.6
cloudpickle==0.5.2
colorama==0.4.3
colored==1.4.2
command-not-found==0.3
constantly==15.1.0
cryptography==3.1.1
cupshelpers==1.0
cycler==0.10.0
daphne==2.5.0
dataclasses==0.8
decorator==4.4.2
defer==1.0.6
defusedxml==0.6.0
distlib==0.3.1
distro==1.5.0
distro-info===0.18ubuntu0.18.04.1
Django==3.1.3
django-allauth==0.42.0
django-automated-logging==6.0.0
django-cors-headers==3.5.0
django-filter==2.4.0
django-ipware==3.0.2
django-picklefield==3.0.1
django-rest-auth==0.9.5
djangorestframework==3.12.1
docutils==0.12
elasticsearch==7.9.1
entrypoints==0.3
fastapi==0.61.2
feedparser==5.2.1
filelock==3.0.12
Flask==1.1.2
Flask-Cors==3.0.8
future==0.18.2
greenlet==0.4.12
h11==0.11.0
hiredis==1.1.0
html5lib==0.999999999
httpie==0.9.8
httplib2==0.9.2
hyperlink==20.0.1
idna==2.10
imagesize==0.7.1
importlib-metadata==2.0.0
importlib-resources==3.0.0
incremental==17.5.0
ipykernel==5.3.4
ipython==7.16.1
ipython-genutils==0.2.0
isort==4.3.4
itsdangerous==1.1.0
jedi==0.17.2
Jinja2==2.11.2
joblib==0.17.0
json5==0.9.5
jsonschema==3.2.0
jupyter-client==6.1.7
jupyter-core==4.6.3
jupyterlab==2.2.9
jupyterlab-pygments==0.1.2
jupyterlab-server==1.2.0
kafka-python==2.0.2
keyring==10.6.0
keyrings.alt==3.0
kiwisolver==1.3.1
kombu==5.0.2
language-selector==0.1
launchpadlib==1.10.6
lazr.restfulclient==0.13.5
lazr.uri==1.0.3
lazy-object-proxy==1.3.1
logilab-common==1.4.1
louis==3.5.0
lxml==4.5.0
macaroonbakery==1.1.3
Mako==1.0.7
Markdown==3.3
MarkupSafe==1.1.1
marshmallow==3.9.1
matplotlib==3.3.3
mccabe==0.6.1
mistune==0.8.4
msgpack==1.0.0
mypy-extensions==0.4.3
nbclient==0.5.1
nbconvert==6.0.7
nbformat==5.0.8
neovim==0.2.0
nest-asyncio==1.4.2
netifaces==0.10.9
notebook==6.1.4
numpy==1.19.4
numpydoc==0.7.0
oauth==1.0.1
oauthlib==3.1.0
olefile==0.45.1
packaging==20.4
pandas==1.0.3
pandocfilters==1.4.3
parso==0.7.1
pathspec==0.8.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==8.0.1
prometheus-client==0.8.0
prompt-toolkit==3.0.8
protobuf==3.0.0
psutil==5.4.2
ptk==1.3.8
ptyprocess==0.6.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycairo==1.16.2
pycodestyle==2.3.1
pycparser==2.20
pycrypto==2.6.1
pycups==1.9.73
pydantic==1.7.2
pyflakes==1.6.0
Pygments==2.7.2
pygobject==3.26.1
PyHamcrest==2.0.2
PyJWT==1.5.3
pylint==1.8.3
pymacaroons==0.13.0
PyNaCl==1.1.2
pyOpenSSL==19.1.0
pyparsing==2.4.7
pyRFC3339==1.0
pyrsistent==0.17.3
PySocks==1.7.1
python-apt==1.6.5+ubuntu0.3
python-dateutil==2.8.1
python-debian==0.1.32
python-slugify==4.0.1
python3-openid==3.2.0
pytz==2020.4
pyxdg==0.25
PyYAML==3.12
pyzmq==19.0.2
QtAwesome==0.4.4
qtconsole==4.3.1
QtPy==1.3.1
redis==3.5.3
regex==2020.7.14
reportlab==3.4.0
requests==2.24.0
requests-futures==1.0.0
requests-oauthlib==1.3.0
requests-unixsocket==0.1.5
roman==2.0.0
rope==0.10.5
scikit-learn==0.23.2
scipy==1.5.4
screen-resolution-extra==0.0.0
SecretStorage==2.3.1
Send2Trash==1.5.0
service-identity==18.1.0
simplegeneric==0.8.1
simplejson==3.13.2
six==1.15.0
soupsieve==2.0
Sphinx==1.6.7
spyder==3.2.6
sqlparse==0.4.1
starlette==0.13.6
stem==1.8.0
system-service==0.3
systemd-python==234
terminado==0.9.1
testpath==0.4.4
text-unidecode==1.3
threadpoolctl==2.1.0
toml==0.10.1
torch==1.7.0
torchvision==0.8.1
tornado==6.1
torrequest==0.1.0
traitlets==4.3.3
Twisted==20.3.0
txaio==20.4.1
typed-ast==1.4.1
typing-extensions==3.7.4.3
ubuntu-drivers-common==0.0.0
ufw==0.36
unattended-upgrades==0.1
unity-scope-calculator==0.1
unity-scope-chromiumbookmarks==0.1
unity-scope-colourlovers==0.1
unity-scope-devhelp==0.1
unity-scope-firefoxbookmarks==0.1
unity-scope-manpages==0.1
unity-scope-openclipart==0.1
unity-scope-texdoc==0.1
unity-scope-tomboy==0.1
unity-scope-virtualbox==0.1
unity-scope-yelp==0.1
unity-scope-zotero==0.1
unity-tweak-tool==0.0.7
urllib3==1.25.11
usb-creator==0.3.3
uvicorn==0.12.2
vine==5.0.0
virtualenv==20.0.35
wadllib==1.3.2
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==0.57.0
websockets==8.1
Werkzeug==1.0.1
wrapt==1.9.0
xkit==0.0.0
xonsh==0.9.24
zipp==3.4.0
zope.interface==5.1.2

Other Dependencies

N/A

Minimally Reproducible Test Case

## tasks.py
from celery import Celery
import os
from time import sleep
redis_url = os.getenv("REDIS_URL", default="redis://localhost")

app = Celery('tasks', backend=redis_url, broker=redis_url)

app.conf.update(
result_extended=True
)

@app.task(bind=True)
def dyn_sleeper(self,s):
print('Executing task id {0.id}, args: {0.args!r} kwargs: {0.kwargs!r}'.format(
self.request))
sleep(s)
return s

## producer.py
from test import dyn_sleeper
from time import sleep

task_list = []

for i in range(5, 15, 2):
print("example task initiated, SLEEP:",i)
task_list.append(dyn_sleeper.delay(i))

print("num of tasks:", len(task_list))

while True:
completed_tasks = [task for task in task_list if task.ready() ]
if len(completed_tasks) ==len(task_list):
print("All tasks finished.")
for c in task_list:
print(">> ",c.name, c.args," :", c.result)
break
print("There is", len(completed_tasks), "completed tasks out of", len(task_list), "tasks")
print(">> ", len(completed_tasks)/len(task_list)*100)
sleep(1)

sleep(10)

Expected Behavior

Exit without an error

Actual Behavior

I have a tasks like this

## celery -A tasks worker -l INFO --concurrency=6
@app.task(bind=True)
def dyn_sleeper(self,s):
print('Executing task id {0.id}, args: {0.args!r} kwargs: {0.kwargs!r}'.format(
self.request))
sleep(s)
return s

When I start celery and run the task like this:

## producer.py
task_list = []

for i in range(5, 35, 2):
print("example task initiated, SLEEP:",i)
task_list.append(dyn_sleeper.delay(i))

I also check if all task finished:

## producer.py
while True:
completed_tasks = [task for task in task_list if task.ready() ]
if len(completed_tasks) ==len(task_list):
print("All tasks finished.")
for c in task_list:
print(">> ",c.name, c.args," :", c.result)
break
print("There is", len(completed_tasks), "completed tasks out of", len(task_list), "tasks")
sleep(1)

Everything works fine but after the producer.py finishes I get an error. I complete all the tasks but I just get this error while I exit from producer.py.

Traceback (most recent call last):
File "/home/username/.local/lib/python3.6/site-packages/celery/result.py", line 391, in __del__
File "/home/username/.local/lib/python3.6/site-packages/celery/backends/asynchronous.py", line 186, in remove_pending_result
File "/home/username/.local/lib/python3.6/site-packages/celery/backends/asynchronous.py", line 194, in on_result_fulfilled
File "/home/username/.local/lib/python3.6/site-packages/celery/backends/redis.py", line 174, in cancel_for
File "/home/username/.local/lib/python3.6/site-packages/redis/client.py", line 3600, in unsubscribe
File "/home/username/.local/lib/python3.6/site-packages/redis/client.py", line 3468, in execute_command
File "/home/username/.local/lib/python3.6/site-packages/redis/connection.py", line 1192, in get_connection
File "/home/username/.local/lib/python3.6/site-packages/redis/connection.py", line 567, in connect
File "/home/username/.local/lib/python3.6/site-packages/redis/connection.py", line 630, in on_connect
File "/home/username/.local/lib/python3.6/site-packages/redis/connection.py", line 401, in on_connect
ImportError: sys.meta_path is None, Python is likely shutting down

Weird thing is if, I don't use the loop above I don't get an error. Also if I don't use many tasks (tasks less than 20) again I don't get an error.

Issue 2:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussions forum first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

#6271

Possible Duplicates

None

Environment & Settings

Celery version:

celery report Output:

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown

Minimal Celery Version: 5.2

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: SQS

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

Other Dependencies

N/A

Minimally Reproducible Test Case

Expected Behavior

Using pip-tools to compile a requirements.txt should include the locked dependencies for the sqs-backend.

For example using this requirements.in file:

celery[sqs]

Compiled to a requirements.txt by:

pip-compile --generate-hashes --output-file requirements.txt requirements.in

This requirements.txt should include the pinned dependencies for the SQS backend (boto3, pycurl, urllib3) and installable with pip install --require-hashes -r reqs/requirements.txt .

Actual Behavior

Trying to install this requirements.txt by pip install --require-hashes -r reqs/requirements.txt will error since not all dependencies are pinned as required by the option require-hashes. I've tried this with the latest pip (21.3.1). and pip-tools (6.4.0).

$ pip install --require-hashes -r requirements.txt
Collecting amqp==5.0.6
Using cached amqp-5.0.6-py3-none-any.whl (53 kB)
Collecting billiard==3.6.4.0
Using cached billiard-3.6.4.0-py3-none-any.whl (89 kB)
Collecting celery[sqs]==5.2.0
Using cached celery-5.2.0-py3-none-any.whl (404 kB)
Requirement already satisfied: click==8.0.3 in ./venv/lib/python3.10/site-packages (from -r requirements.txt (line 19)) (8.0.3)
Collecting click-didyoumean==0.3.0
Using cached click_didyoumean-0.3.0-py3-none-any.whl (2.7 kB)
Collecting click-plugins==1.1.1
Using cached click_plugins-1.1.1-py2.py3-none-any.whl (7.5 kB)
Collecting click-repl==0.2.0
Using cached click_repl-0.2.0-py3-none-any.whl (5.2 kB)
Collecting kombu==5.2.1
Using cached kombu-5.2.1-py3-none-any.whl (188 kB)
Collecting prompt-toolkit==3.0.22
Using cached prompt_toolkit-3.0.22-py3-none-any.whl (374 kB)
Collecting pytz==2021.3
Using cached pytz-2021.3-py2.py3-none-any.whl (503 kB)
Collecting six==1.16.0
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting vine==5.0.0
Using cached vine-5.0.0-py2.py3-none-any.whl (9.4 kB)
Collecting wcwidth==0.2.5
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: setuptools in ./venv/lib/python3.10/site-packages (from celery[sqs]==5.2.0->-r requirements.txt (line 15)) (58.2.0)
Collecting urllib3>=1.26.7
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
urllib3>=1.26.7 from https://files.pythonhosted.org/packages/af/f4/524415c0744552cce7d8bf3669af78e8a069514405ea4fcbd0cc44733744/urllib3-1.26.7-py2.py3-none-any.whl#sha256=c4fdf4019605b6e5423637e01bc9fe4daef873709a7973e195ceba0a62bbc844 (from kombu==5.2.1->-r requirements.txt (line 39))

I tracked down the cause of this to #6271 where requirements/sqs.txt is modified to install kombu[sqs] instead of the explicit dependencies. See https://github.com/celery/celery/pull/6271/files#diff-d229b45a35f019d718901cd33e806e8d2fbecb8eb66d81fb00f8e481c56ea0b4

I under that it's much nicer to only specify these dependencies in one repository by using this pass through dependency but it appears like pip-tools (or even the resolver in pip?) can not handle it. I haven't really had the time to dig into the details but it might make sense to explicitly define the SQS dependencies in https://github.com/celery/celery/blob/master/requirements/extras/sqs.txt as is done for the other backends until this works upstream since pip-tools is a rather common tool to work with dependencies.