Issue 1:
Checklist

I have verified that the issue exists against the master branch of Celery.

I have checked the

celery issues 3932

kombu issue 931

for similar or identical bug reports.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Related Issues and Possible Duplicates

Related Issues

#3932

celery/kombu#931

Possible Duplicates

None

Environment & Settings

Celery version:

celery report Output:

software -> celery:4.4.2 (cliffs) kombu:4.6.11 py:3.7.6
billiard:3.6.3.0 py-amqp:2.6.0
platform -> system:Linux arch:64bit
kernel version:4.15.0-1065-aws imp:CPython
loader -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.7.6 imp: CPython

Minimal Celery Version: 4.4.2

Minimal Kombu Version: 4.6.11

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: GNU Linux Ubuntu x86_64 18.04

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

aiohttp==3.6.2
amqp==2.6.0
argh==0.26.2
asgiref==3.2.10
async-timeout==3.0.1
attrs==19.3.0
billiard==3.6.3.0
boto3==1.12.39
botocore==1.15.49
celery==4.4.2
certifi==2020.6.20
chardet==3.0.4
ddtrace==0.37.0
Django==3.0.4
docutils==0.15.2
future==0.18.2
gunicorn==20.0.4
idna==2.10
jmespath==0.10.0
kombu==4.6.11
msgpack==1.0.0
multidict==4.7.6
pathtools==0.1.2
psycopg2-binary==2.8.4
python-dateutil==2.8.1
python-dotenv==0.12.0
pytz==2020.1
PyYAML==5.3.1
requests==2.23.0
s3transfer==0.3.3
setproctitle==1.1.10
six==1.15.0
slackclient==2.5.0
sqlparse==0.3.1
urllib3==1.25.9
vine==1.3.0
yarl==1.4.2

Other Dependencies

N/A

Minimally Reproducible Test Case

It depends. The issue is occurring when we will get a connection lost error.
```python
```

Expected Behavior

Actual Behavior

Got this in logs.

[2020-06-03 01:36:35,499: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/util/connection.py", line 61, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/opt/pyenv/versions/3.7.6/lib/python3.7/socket.py", line 752, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/httpsession.py", line 263, in send
chunked=self._chunked(request.headers),
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 725, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/util/retry.py", line 379, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
raise value
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 976, in _validate_conn
conn.connect()
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connection.py", line 308, in connect
conn = self._new_conn()
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connection.py", line 172, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x7f910d7373d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start
blueprint.start(self)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 599, in start
c.loop(*c.loop_args())
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/celery/worker/loops.py", line 83, in asynloop
next(loop)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 301, in create_loop
item()
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/vine/promises.py", line 170, in __call__
return self.throw()
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/vine/promises.py", line 167, in __call__
retval = fun(*final_args, **final_kwargs)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/transport/SQS.py", line 390, in _schedule_queue
queue, callback=promise(self._loop1, (queue,)),
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/transport/SQS.py", line 406, in _get_bulk_async
return self._get_async(queue, maxcount, callback=callback)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/transport/SQS.py", line 417, in _get_async
callback=transform(self._on_messages_ready, callback, q, queue),
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/transport/SQS.py", line 442, in _get_from_sqs
queue_url = connection.get_queue_url(queue)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/asynchronous/aws/sqs/connection.py", line 45, in get_queue_url
res = self.sqs_connection.get_queue_url(QueueName=queue)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/ddtrace/contrib/botocore/patch.py", line 70, in patched_api_call
result = original_func(*args, **kwargs)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/client.py", line 613, in _make_api_call
operation_model, request_dict, request_context)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/client.py", line 632, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/endpoint.py", line 137, in _send_request
success_response, exception):
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/endpoint.py", line 231, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py", line 183, in __call__
if self._checker(attempts, response, caught_exception):
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py", line 251, in __call__
caught_exception)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py", line 317, in __call__
caught_exception)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py", line 223, in __call__
attempt_number, caught_exception)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/endpoint.py", line 200, in _do_get_response
http_response = self._send(request)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/endpoint.py", line 244, in _send
return self.http_session.send(request)
File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/httpsession.py", line 283, in send
raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://queue.amazonaws.com/"
[2020-06-03 01:36:47,080: ERROR/MainProcess] consumer: Cannot connect to sqs://localhost//: Could not connect to the endpoint URL: "https://queue.amazonaws.com/".
Trying again in 2.00 seconds... (1/100)

[2020-06-03 01:36:57,674: INFO/MainProcess] Connected to sqs://localhost//

And after the last line logs are empty because celery stops handling any message.

Issue 2:
Checklist

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version:

celery report Output:software -> celery:4.3.0 (rhubarb) kombu:4.6.3 py:3.6.5
billiard:3.6.0.0 py-amqp:2.5.0
platform -> system:Linux arch:64bit
kernel version:4.15.0-54-generic imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:amqp results:rpc:///

accept_content: ['pickle']

broker_url: 'amqp://amitc005:********@localhost:5672/rest'

include: ['srm_rest_email.email.views']

result_backend: 'rpc:///'

result_expires: 3000

result_extended: True

task_default_queue: 'email'

task_serializer: 'pickle'

task_track_started: True

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown

Minimal Celery Version: N/A or Unknown

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

Problem Description

When I was passing files immutiable dict object by calling a .delay() method, celery is throwing exception,

[2019-07-19 10:58:48,420: CRITICAL/MainProcess] Can't decode message body: DecodeError(RecursionError('maximum recursion depth exceeded while calling a Python object',),) [type:'application/x-python-serialize' encoding:'binary' headers:{'lang': 'py', 'task': 'srm_rest_email.email.views.send_async_email', 'id': 'a6891a6e-e428-460b-bb98-940df5f430b2', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'a6891a6e-e428-460b-bb98-940df5f430b2', 'parent_id': None, 'argsrepr': "({'file': <FileStorage: '2019-05-07.png' ('image/png')>}, {'exp': 1563589252, 'iat': 1563546052, 'sub': 'HTTPie/0.9.9', 'jti': '06df3341-c1cc-49ce-a7a8-d4a4983469ad', 'client_id': 'test_client', 'company_id': 1, 'operator_id': 'SC'})", 'kwargsrepr': '{}', 'origin': 'gen4932@achavhan-ThinkPad-X1-Carbon-6th'}]

body: b'\x80\x02cwerkzeug.datastructures\nImmutableMultiDict\nq\x00]q\x01X\x04\x00\x00\x00fileq\x02cwerkzeug.datastructures\nFileStorage\nq\x03)\x81q\x04c_codecs\nencode\nq\x05XH\x9f\x00\x00\xc2\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x03h\x00\x00\x01\x05\x08\x02\x00\x00\x00\x19\xc2\x81S\xc3\x87\x00\x00\x00\x03sBIT\x08\x08\x08\xc3\x9b\xc3\xa1O\xc3\xa0\x00\x00\x00_zTXtRaw profile type APP1\x00\x00\x08\xc2\x99\xc3\xa3JO\xc3\x8dK-\xc3\x8aLV((\xc3\x8aO\xc3\x8b\xc3\x8cI\xc3\xa5R\x00\x03c\x13.\x13K\x13K\xc2\xa3D\x03\x03\x03\x0b\x03\x0804006\x04\xc2\x92F@\xc2\xb69T(\xc3\x91\x00\x05\xc2\x98\xc2\x98\xc2\x9b\xc2\xa5\x01\xc2\xa1\xc2\xb9Y\xc2\xb2\xc2\x99)\xc2\x88\xc3\x8f\x05\x00O\xc2\xba\x15h\x1b-\xc3\x98\xc2\x8c\x00\x00... (41213b)
Traceback (most recent call last):
File "/home/achavhan/srm_rest_email/env/lib/python3.6/site-packages/kombu/serialization.py", line 50, in _reraise_errors
yield
File "/home/achavhan/srm_rest_email/env/lib/python3.6/site-packages/kombu/serialization.py", line 263, in loads
return decode(data)
File "/home/achavhan/srm_rest_email/env/lib/python3.6/site-packages/kombu/serialization.py", line 337, in unpickle
return pickle_loads(str_to_bytes(s))
File "/home/achavhan/srm_rest_email/env/lib/python3.6/site-packages/kombu/serialization.py", line 59, in pickle_loads
return load(BytesIO(s))
File "/home/achavhan/srm_rest_email/env/lib/python3.6/site-packages/werkzeug/datastructures.py", line 2821, in __getattr__
return getattr(self.stream, name)
File "/home/achavhan/srm_rest_email/env/lib/python3.6/site-packages/werkzeug/datastructures.py", line 2821, in __getattr__
return getattr(self.stream, name)
File "/home/achavhan/srm_rest_email/env/lib/python3.6/site-packages/werkzeug/datastructures.py", line 2821, in __getattr__
return getattr(self.stream, name)
[Previous line repeated 314 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/achavhan/srm_rest_email/env/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 568, in on_task_received
callbacks,
File "/home/achavhan/srm_rest_email/env/lib/python3.6/site-packages/celery/worker/strategy.py", line 137, in task_message_handler
if body is None and 'args' not in message.payload:
File "/home/achavhan/srm_rest_email/env/lib/python3.6/site-packages/kombu/message.py", line 208, in payload
return self._decoded_cache if self._decoded_cache else self.decode()
File "/home/achavhan/srm_rest_email/env/lib/python3.6/site-packages/kombu/message.py", line 193, in decode
self._decoded_cache = self._decode()
File "/home/achavhan/srm_rest_email/env/lib/python3.6/site-packages/kombu/message.py", line 198, in _decode
self.content_encoding, accept=self.accept)
File "/home/achavhan/srm_rest_email/env/lib/python3.6/site-packages/kombu/serialization.py", line 263, in loads
return decode(data)
File "/usr/local/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/home/achavhan/srm_rest_email/env/lib/python3.6/site-packages/kombu/serialization.py", line 54, in _reraise_errors
reraise(wrapper, wrapper(exc), sys.exc_info()[2])
File "/home/achavhan/srm_rest_email/env/lib/python3.6/site-packages/vine/five.py", line 194, in reraise
raise value.with_traceback(tb)
File "/home/achavhan/srm_rest_email/env/lib/python3.6/site-packages/kombu/serialization.py", line 50, in _reraise_errors
yield
File "/home/achavhan/srm_rest_email/env/lib/python3.6/site-packages/kombu/serialization.py", line 263, in loads
return decode(data)
File "/home/achavhan/srm_rest_email/env/lib/python3.6/site-packages/kombu/serialization.py", line 337, in unpickle
return pickle_loads(str_to_bytes(s))
File "/home/achavhan/srm_rest_email/env/lib/python3.6/site-packages/kombu/serialization.py", line 59, in pickle_loads
return load(BytesIO(s))
File "/home/achavhan/srm_rest_email/env/lib/python3.6/site-packages/werkzeug/datastructures.py", line 2821, in __getattr__
return getattr(self.stream, name)
File "/home/achavhan/srm_rest_email/env/lib/python3.6/site-packages/werkzeug/datastructures.py", line 2821, in __getattr__
return getattr(self.stream, name)
File "/home/achavhan/srm_rest_email/env/lib/python3.6/site-packages/werkzeug/datastructures.py", line 2821, in __getattr__
return getattr(self.stream, name)
[Previous line repeated 314 more times]
kombu.exceptions.DecodeError: maximum recursion depth exceeded while calling a Python object