Issue 1:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

#1700

#2725

Possible Duplicates

None

Environment & Settings

Celery version: 4.4.2/master (20b7a80)

celery report Output:

developer@b7825f7f6305:~/celery$ celery -A tasks report

software -> celery:4.4.2 (cliffs) kombu:4.6.8 py:3.8.0
billiard:3.6.3.0 py-amqp:2.5.2
platform -> system:Linux arch:64bit, ELF
kernel version:3.10.0-957.el7.x86_64 imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:amqp results:db+sqlite:///results.sqlite

broker_url: 'amqp://guest:********@rabbit:5672//'
result_backend: 'db+sqlite:///results.sqlite'

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown

Minimal Celery Version: N/A or Unknown

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

alabaster==0.7.12
amqp==2.5.2
appdirs==1.4.3
attrs==19.3.0
aws-xray-sdk==0.95
azure-common==1.1.5
azure-nspkg==3.0.2
azure-storage==0.36.0
azure-storage-common==1.1.0
azure-storage-nspkg==3.1.0
Babel==2.8.0
basho-erlastic==2.1.1
billiard==3.6.3.0
boto==2.49.0
boto3==1.12.37
botocore==1.15.37
bumpversion==0.5.3
case==1.5.3
cassandra-driver==3.23.0
-e git+https://github.com/celery/celery.git@20b7a808398b7b865651a5cf0fd7aa2b74be3a4b#egg=celery
certifi==2020.4.5.1
cffi==1.14.0
cfgv==3.1.0
chardet==3.0.4
click==7.1.1
codecov==2.0.22
couchbase==2.5.11
coverage==5.0.4
cryptography==2.9
DateTime==4.3
distlib==0.3.0
dnspython==1.16.0
docker==4.2.0
docutils==0.16
ecdsa==0.15
elasticsearch==7.6.0
entrypoints==0.3
ephem==3.7.7.1
eventlet==0.25.1
filelock==3.0.12
flake8==3.7.9
flakeplus==1.1.0
future==0.18.2
geomet==0.1.2
gevent==1.4.0
greenlet==0.4.15
identify==1.4.14
idna==2.9
imagesize==1.2.0
Jinja2==2.11.1
jmespath==0.9.5
jsondiff==1.1.1
jsonpickle==1.3
kombu==4.6.8
linecache2==1.0.0
MarkupSafe==1.1.1
mccabe==0.6.1
mock==4.0.2
monotonic==1.5
more-itertools==8.2.0
moto==1.3.7
msgpack==1.0.0
nodeenv==1.3.5
nose==1.3.7
packaging==20.3
pluggy==0.13.1
pre-commit==2.2.0
py==1.8.1
pyaml==20.4.0
pyArango==1.3.4
pycodestyle==2.5.0
pycouchdb==1.14.1
pycparser==2.20
pycryptodome==3.9.7
pycurl==7.43.0.5
pydocstyle==1.1.1
pydocumentdb==2.3.2
pyflakes==2.1.1
Pygments==2.6.1
pylibmc==1.6.1
pymongo==3.10.1
pyparsing==2.4.7
pytest==5.3.4
pytest-cov==2.8.1
pytest-rerunfailures==9.0
pytest-sugar==0.9.2
pytest-travis-fold==1.3.0
python-consul==1.1.0
python-dateutil==2.8.1
python-jose==2.0.2
python-memcached==1.59
python3-protobuf==2.5.0
pytz==2019.3
PyYAML==5.3.1
redis==3.4.1
requests==2.23.0
responses==0.10.12
riak==2.7.0
s3transfer==0.3.3
simplejson==3.17.0
six==1.14.0
snowballstemmer==2.0.0
softlayer-messaging==1.0.3
Sphinx==1.8.5
sphinx-celery==1.4.6
sphinx-testing==0.7.2
sphinx2rst==1.1.0
sphinxcontrib-websupport==1.2.1
SQLAlchemy==1.3.15
tblib==1.6.0
termcolor==1.1.0
toml==0.10.0
tox==3.14.6
traceback2==1.4.0
Unipath==1.1
unittest2==1.1.0
urllib3==1.25.8
vine==1.3.0
virtualenv==20.0.16
wcwidth==0.1.9
websocket-client==0.57.0
Werkzeug==1.0.1
wrapt==1.12.1
xmltodict==0.12.0
zope.interface==5.0.2

Other Dependencies

N/A

Minimally Reproducible Test Case

from celery import Celery, chain, group
app = Celery('tasks',
backend='db+sqlite:///results.sqlite',
broker='amqp://guest:guest@rabbit:5672//')

@app.task()
def foo():
print("this is from foo!")

@app.task()
def fail():
raise ValueError("this is from fail!")

if __name__ == '__main__':
chain(group(chain(foo.si(), fail.si(), foo.si())), foo.si())()

Expected Behavior

I'm not sure what the intended behavior of chord_unlock is here but I feel that its current behavior atleast isn't consistent. The test case I've provided above results in what I've included below in Actual Behavior where it gets stuck in an infinite loop. If I instead remove the call to foo that is directly after fail, then the chord_unlock doesn't get stuck in a loop and instead simply stops after the fail task has executed.

If I extend fail to, for example, the following

@app.task()
def fail():
if not os.path.exists('foo.txt'):
raise ValueError("this is from fail!")
print("file does exist!")

then I can also do the test below to see what happens if I retry the failed task:

chain(group(chain(foo.si(), fail.si().set(task_id="123"), foo.si())), foo.si().set(task_id="456"))()
sleep(10)
open('foo.txt', 'a').close()
fail.apply_async(task_id="123")

Here I would atleast expect the infinite loop to end, but instead I see that the failed task now succeeds successfully (which is great) but the loop continues. I'm not sure if I should also expect the tasks after the retried task to now also run (they currently do not).

Actual Behavior

[2020-04-08 10:06:34,804: INFO/MainProcess] Connected to amqp://guest:**@rabbit:5672//
[2020-04-08 10:06:34,816: INFO/MainProcess] mingle: searching for neighbors
[2020-04-08 10:06:35,861: INFO/MainProcess] mingle: all alone
[2020-04-08 10:06:35,876: INFO/MainProcess] celery@b7825f7f6305 ready.
[2020-04-08 10:06:56,992: INFO/MainProcess] Received task: celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b] ETA:[2020-04-08 10:06:57.952640+00:00]
[2020-04-08 10:06:57,004: INFO/MainProcess] Received task: tasks.foo[a2d3f985-68c1-40d8-9a7e-e3d323824901]
[2020-04-08 10:06:57,005: WARNING/ForkPoolWorker-2] this is from foo!
[2020-04-08 10:06:57,031: INFO/MainProcess] Received task: tasks.fail[5a61ee80-e695-4cf7-b182-68de7d848bc7]
[2020-04-08 10:06:57,057: INFO/ForkPoolWorker-2] Task tasks.foo[a2d3f985-68c1-40d8-9a7e-e3d323824901] succeeded in 0.05272179003804922s: None
[2020-04-08 10:06:57,081: ERROR/ForkPoolWorker-3] Task tasks.fail[5a61ee80-e695-4cf7-b182-68de7d848bc7] raised unexpected: ValueError('this is from fail!')
Traceback (most recent call last):
File "/home/developer/celery/celery/app/trace.py", line 411, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/developer/celery/celery/app/trace.py", line 680, in __protected_call__
return self.run(*args, **kwargs)
File "/home/developer/celery/tasks.py", line 13, in fail
raise ValueError("this is from fail!")
ValueError: this is from fail!
[2020-04-08 10:06:59,908: INFO/MainProcess] Received task: celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b] ETA:[2020-04-08 10:07:00.906229+00:00]
[2020-04-08 10:06:59,908: INFO/ForkPoolWorker-2] Task celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b] retry: Retry in 1s
[2020-04-08 10:07:01,914: INFO/ForkPoolWorker-2] Task celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b] retry: Retry in 1s
[2020-04-08 10:07:01,920: INFO/MainProcess] Received task: celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b] ETA:[2020-04-08 10:07:02.912826+00:00]
[2020-04-08 10:07:03,914: INFO/ForkPoolWorker-2] Task celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b] retry: Retry in 1s
[2020-04-08 10:07:03,915: INFO/MainProcess] Received task: celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b] ETA:[2020-04-08 10:07:04.913285+00:00]
[2020-04-08 10:07:05,876: INFO/ForkPoolWorker-2] Task celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b] retry: Retry in 1s
[2020-04-08 10:07:05,877: INFO/MainProcess] Received task: celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b] ETA:[2020-04-08 10:07:06.875506+00:00]

Issue 2:
Have an issue using celery with SQS: when the connection is being lost and recovered, messages are not being served by celery.

Only after restarting celery (using supervisor), messages are being processed.

Checklist

I have verified that the issue exists against the master branch of Celery.

I have checked the

celery issues 3932

kombu issue 931

for similar or identical bug reports.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Related Issues and Possible Duplicates

Related Issues

#3932

celery/kombu#931

Possible Duplicates

None

Environment & Settings

Celery version:

celery report Output:

software -> celery:4.4.2 (cliffs) kombu:4.6.11 py:3.7.6
billiard:3.6.3.0 py-amqp:2.6.0
platform -> system:Linux arch:64bit
kernel version:4.15.0-1065-aws imp:CPython
loader -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.7.6 imp: CPython

Minimal Celery Version: 4.4.2

Minimal Kombu Version: 4.6.11

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: GNU Linux Ubuntu x86_64 18.04

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

aiohttp==3.6.2
amqp==2.6.0
argh==0.26.2
asgiref==3.2.10
async-timeout==3.0.1
attrs==19.3.0
billiard==3.6.3.0
boto3==1.12.39
botocore==1.15.49
celery==4.4.2
certifi==2020.6.20
chardet==3.0.4
ddtrace==0.37.0
Django==3.0.4
docutils==0.15.2
future==0.18.2
gunicorn==20.0.4
idna==2.10
jmespath==0.10.0
kombu==4.6.11
msgpack==1.0.0
multidict==4.7.6
pathtools==0.1.2
psycopg2-binary==2.8.4
python-dateutil==2.8.1
python-dotenv==0.12.0
pytz==2020.1
PyYAML==5.3.1
requests==2.23.0
s3transfer==0.3.3
setproctitle==1.1.10
six==1.15.0
slackclient==2.5.0
sqlparse==0.3.1
urllib3==1.25.9
vine==1.3.0
yarl==1.4.2

Other Dependencies

N/A

Minimally Reproducible Test Case

It depends. The issue is occurring when we will get a connection lost error.
```python
```

Expected Behavior

Actual Behavior

Got this in logs.

[2020-06-03 01:36:35,499: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/util/connection.py", line 61, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/opt/pyenv/versions/3.7.6/lib/python3.7/socket.py", line 752, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/httpsession.py", line 263, in send
chunked=self._chunked(request.headers),
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 725, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/util/retry.py", line 379, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
raise value
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 976, in _validate_conn
conn.connect()
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connection.py", line 308, in connect
conn = self._new_conn()
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connection.py", line 172, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x7f910d7373d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start
blueprint.start(self)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 599, in start
c.loop(*c.loop_args())
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/celery/worker/loops.py", line 83, in asynloop
next(loop)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 301, in create_loop
item()
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/vine/promises.py", line 170, in __call__
return self.throw()
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/vine/promises.py", line 167, in __call__
retval = fun(*final_args, **final_kwargs)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/transport/SQS.py", line 390, in _schedule_queue
queue, callback=promise(self._loop1, (queue,)),
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/transport/SQS.py", line 406, in _get_bulk_async
return self._get_async(queue, maxcount, callback=callback)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/transport/SQS.py", line 417, in _get_async
callback=transform(self._on_messages_ready, callback, q, queue),
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/transport/SQS.py", line 442, in _get_from_sqs
queue_url = connection.get_queue_url(queue)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/asynchronous/aws/sqs/connection.py", line 45, in get_queue_url
res = self.sqs_connection.get_queue_url(QueueName=queue)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/ddtrace/contrib/botocore/patch.py", line 70, in patched_api_call
result = original_func(*args, **kwargs)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/client.py", line 613, in _make_api_call
operation_model, request_dict, request_context)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/client.py", line 632, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/endpoint.py", line 137, in _send_request
success_response, exception):
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/endpoint.py", line 231, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py", line 183, in __call__
if self._checker(attempts, response, caught_exception):
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py", line 251, in __call__
caught_exception)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py", line 317, in __call__
caught_exception)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py", line 223, in __call__
attempt_number, caught_exception)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/endpoint.py", line 200, in _do_get_response
http_response = self._send(request)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/endpoint.py", line 244, in _send
return self.http_session.send(request)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/httpsession.py", line 283, in send
raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://queue.amazonaws.com/"
[2020-06-03 01:36:47,080: ERROR/MainProcess] consumer: Cannot connect to sqs://localhost//: Could not connect to the endpoint URL: "https://queue.amazonaws.com/".
Trying again in 2.00 seconds... (1/100)

[2020-06-03 01:36:57,674: INFO/MainProcess] Connected to sqs://localhost//

And after the last line logs are empty because celery stops handling any message.