Issue 1:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version:

celery report Output:

Steps to Reproduce

Required Dependencies

Minimal Python Version: python 3.5

Minimal Celery Version: celery 4.3

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: redis 6.0.5

Minimal Result Backend Version: redis 6.0.5

Minimal OS and/or Kernel Version: osx 10.15.5, ubuntu 20.04

Minimal Broker Client Version: redis==3.5.3

Minimal Result Backend Client Version: redis==3.5.3

Python Packages

pip freeze Output:

absl-py==0.8.0
adal==1.2.2
alembic==1.0.1
altair==4.0.1
amqp==2.4.1
aniso8601==7.0.0
antlr4-python3-runtime==4.7.2
apipkg==1.5
applicationinsights==0.11.9
appnope==0.1.0
argcomplete==1.10.0
asn1crypto==0.24.0
astor==0.8.0
astroid==2.3.3
astunparse==1.6.3
atomicwrites==1.1.5
attrs==18.1.0
azure==3.0.0
azure-batch==7.0.0
azure-cli==2.0.71
azure-cli-command-modules-nspkg==2.0.3
azure-cli-core==2.0.71
azure-cli-nspkg==3.0.4
azure-cli-telemetry==1.0.3
azure-common==1.1.11
azure-cosmos==3.1.1
azure-cosmosdb-nspkg==2.0.2
azure-cosmosdb-table==1.0.3
azure-datalake-store==0.0.47
azure-eventgrid==0.1.0
azure-functions-devops-build==0.0.22
azure-graphrbac==0.60.0
azure-keyvault==1.1.0
azure-mgmt==2.0.0
azure-mgmt-advisor==2.0.1
azure-mgmt-appconfiguration==0.1.0
azure-mgmt-applicationinsights==0.1.1
azure-mgmt-authorization==0.52.0
azure-mgmt-batch==6.0.0
azure-mgmt-batchai==2.0.0
azure-mgmt-billing==0.2.0
azure-mgmt-botservice==0.2.0
azure-mgmt-cdn==3.1.0
azure-mgmt-cognitiveservices==5.0.0
azure-mgmt-commerce==1.0.1
azure-mgmt-compute==6.0.0
azure-mgmt-consumption==2.0.0
azure-mgmt-containerinstance==1.5.0
azure-mgmt-containerregistry==3.0.0rc5
azure-mgmt-containerservice==5.3.0
azure-mgmt-cosmosdb==0.7.0
azure-mgmt-datafactory==0.4.0
azure-mgmt-datalake-analytics==0.2.1
azure-mgmt-datalake-nspkg==2.0.0
azure-mgmt-datalake-store==0.5.0
azure-mgmt-datamigration==0.1.0
azure-mgmt-deploymentmanager==0.1.0
azure-mgmt-devtestlabs==2.2.0
azure-mgmt-dns==2.1.0
azure-mgmt-eventgrid==2.2.0
azure-mgmt-eventhub==2.6.0
azure-mgmt-hanaonazure==0.1.1
azure-mgmt-hdinsight==1.1.0
azure-mgmt-imagebuilder==0.2.1
azure-mgmt-iotcentral==1.0.0
azure-mgmt-iothub==0.8.2
azure-mgmt-iothubprovisioningservices==0.2.0
azure-mgmt-keyvault==1.1.0
azure-mgmt-kusto==0.3.0
azure-mgmt-loganalytics==0.2.0
azure-mgmt-logic==2.1.0
azure-mgmt-machinelearningcompute==0.4.1
azure-mgmt-managedservices==1.0.0
azure-mgmt-managementgroups==0.2.0
azure-mgmt-managementpartner==0.1.0
azure-mgmt-maps==0.1.0
azure-mgmt-marketplaceordering==0.1.0
azure-mgmt-media==1.1.1
azure-mgmt-monitor==0.5.2
azure-mgmt-msi==0.2.0
azure-mgmt-netapp==0.5.0
azure-mgmt-network==3.0.0
azure-mgmt-notificationhubs==1.0.0
azure-mgmt-nspkg==2.0.0
azure-mgmt-policyinsights==0.3.1
azure-mgmt-powerbiembedded==1.0.0
azure-mgmt-privatedns==0.1.0
azure-mgmt-rdbms==1.9.0
azure-mgmt-recoveryservices==0.4.0
azure-mgmt-recoveryservicesbackup==0.4.0
azure-mgmt-redis==6.0.0
azure-mgmt-relay==0.1.0
azure-mgmt-reservations==0.3.1
azure-mgmt-resource==2.2.0
azure-mgmt-scheduler==1.1.3
azure-mgmt-search==2.1.0
azure-mgmt-security==0.1.0
azure-mgmt-servermanager==1.2.0
azure-mgmt-servicebus==0.6.0
azure-mgmt-servicefabric==0.2.0
azure-mgmt-signalr==0.1.1
azure-mgmt-sql==0.12.0
azure-mgmt-sqlvirtualmachine==0.4.0
azure-mgmt-storage==4.0.0
azure-mgmt-subscription==0.1.0
azure-mgmt-trafficmanager==0.51.0
azure-mgmt-web==0.42.0
azure-multiapi-storage==0.2.4
azure-nspkg==2.0.0
azure-servicebus==0.21.1
azure-servicefabric==6.1.2.9
azure-servicemanagement-legacy==0.20.6
azure-storage-blob==1.5.0
azure-storage-common==1.4.2
azure-storage-file==1.1.0
azure-storage-nspkg==3.0.0
azure-storage-queue==1.1.0
Babel==2.6.0
backcall==0.1.0
base58==2.0.1
bcrypt==3.1.4
beautifulsoup4==4.6.3
billiard==3.6.0.0
bleach==3.1.0
blinker==1.4
boto3==1.14.7
botocore==1.17.7
bs4==0.0.1
cached-property==1.5.1
cachetools==4.1.0
celery==4.3.0
celery-once==3.0.1
certifi==2018.4.16
cffi==1.11.5
chardet==3.0.4
click==7.1.2
click-completion==0.5.0
cloudpickle==0.5.3
colorama==0.4.1
crayons==0.1.2
cryptography==2.6.1
cssselect==1.0.3
cycler==0.10.0
dask==0.18.1
decorator==4.3.0
defusedxml==0.6.0
dnspython==1.16.0
docker==4.0.2
docker-pycreds==0.3.0
docutils==0.15.2
email-validator==1.1.1
entrypoints==0.2.3
enum-compat==0.0.3
enum34==1.1.6
execnet==1.5.0
exif==0.8.1
fabric==2.4.0
flake8==3.5.0
flake8-polyfill==1.0.2
Flask==1.1.2
Flask-Assets==0.12
Flask-BabelEx==0.9.4
Flask-GraphQL==2.0.0
Flask-Login==0.5.0
Flask-Mail==0.9.1
Flask-Principal==0.4.0
Flask-Security-Too==3.4.3
Flask-Uploads==0.2.1
Flask-Webpack==0.1.0
Flask-WTF==0.14.3
future==0.18.2
futures==3.1.1
gast==0.3.3
gitdb2==2.0.5
GitPython==2.1.11
google-auth==1.17.2
google-auth-oauthlib==0.4.1
google-pasta==0.2.0
graphene==2.1.8
graphene-sqlalchemy==2.2.2
graphql-core==2.2.1
graphql-relay==2.0.0
graphql-server-core==1.1.1
grpcio==1.24.0
h5py==2.10.0
honcho==1.0.1
humanfriendly==4.18
idna==2.6
imageio==2.5.0
imgaug==0.3.0
importlib-metadata==0.23
infinity==1.4
inflection==0.3.1
intervals==0.8.1
invoke==1.3.0
ipykernel==5.3.0
ipython==6.4.0
ipython-genutils==0.2.0
ipywidgets==7.5.1
isodate==0.6.0
isort==4.3.4
itsdangerous==1.1.0
javaproperties==0.5.1
jedi==0.12.0
Jinja2==2.10.1
jmespath==0.9.4
joblib==0.14.0
jsondiff==1.2.0
jsonschema==3.0.1
jupyter==1.0.0
jupyter-client==5.3.1
jupyter-console==6.0.0
jupyter-core==4.5.0
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
keyring==12.2.1
kiwisolver==1.0.1
knack==0.6.3
kombu==4.5.0
lazy-object-proxy==1.4.3
libsass==0.15.1
loggly-python-handler==1.0.0
lxml==4.2.5
Mako==1.0.7
mando==0.6.4
Markdown==3.1.1
MarkupSafe==1.0
matplotlib==2.2.2
mccabe==0.6.1
mistune==0.8.4
mock==2.0.0
more-itertools==4.2.0
msrest==0.6.9
msrestazure==0.6.1
mysql-connector-python==8.0.16
nbconvert==5.5.0
nbformat==4.4.0
networkx==2.1
notebook==6.0.0
numpy==1.17.2
oauthlib==2.1.0
opencv-python==4.2.0.34
opencv-python-headless==4.2.0.34
opt-einsum==3.2.1
packaging==19.2
pandas==0.23.4
pandocfilters==1.4.2
paramiko==2.4.2
parso==0.2.1
passlib==1.7.1
patchwork==1.0.1
pathlib2==2.3.4
pathtools==0.1.2
pbr==5.4.2
pexpect==4.5.0
pickleshare==0.7.4
piexif==1.1.3
Pillow==7.1.2
pluggy==0.13.0
portalocker==1.5.1
prometheus-client==0.7.1
promise==2.2.1
prompt-toolkit==1.0.15
protobuf==3.8.0
psutil==5.6.3
ptyprocess==0.5.2
py==1.5.3
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.5.0
pycparser==2.18
pydeck==0.4.0b2
pydocumentdb==2.3.3
pyflakes==1.6.0
Pygments==2.4.2
PyJWT==1.6.4
pylint==2.4.4
PyNaCl==1.3.0
pyOpenSSL==19.0.0
pyparsing==2.2.0
PyQRCode==1.2.1
pyquery==1.4.0
pyrsistent==0.15.4
pyScss==1.3.5
pytest==5.1.3
pytest-forked==0.2
pytest-xdist==1.23.0
python-dateutil==2.7.3
python-editor==1.0.3
python-json-logger==0.1.11
pytz==2019.2
PyWavelets==0.5.2
PyYAML==5.1.2
pyzmq==18.0.2
qtconsole==4.5.2
radon==4.0.0
redis==3.3.8
requests==2.22.0
requests-futures==1.0.0
requests-oauthlib==1.0.0
rsa==4.6
Rx==1.6.1
s3transfer==0.3.3
scikit-image==0.15.0
scikit-learn==0.22.1
scipy==1.4.1
scp==0.13.2
Send2Trash==1.5.0
Shapely==1.6.4.post2
shellingham==1.2.7
simplegeneric==0.8.1
singledispatch==3.4.0.3
six==1.12.0
sklearn==0.0
slacker==0.9.65
slacker-log-handler==1.7.1
smmap2==2.0.5
speaklater==1.3
SQLAlchemy==1.3.17
sshtunnel==0.1.5
streamlit==0.59.0
tabulate==0.8.3
tensorboard==2.2.2
tensorboard-plugin-wit==1.6.0.post3
tensorflow==2.2.0
tensorflow-estimator==2.2.0
termcolor==1.1.0
terminado==0.8.2
testpath==0.4.2
toml==0.10.1
toolz==0.9.0
tornado==5.1.1
tqdm==4.35.0
traitlets==4.3.2
typed-ast==1.4.0
tzlocal==2.1
urllib3==1.22
uWSGI==2.0.17
validators==0.13.0
vine==1.3.0
visual-logging==1.0
vsts==0.1.25
vsts-cd-manager==1.0.2
watchdog==0.10.2
wcwidth==0.1.7
webassets==0.12.1
webencodings==0.5.1
websocket-client==0.56.0
Werkzeug==1.0.1
widgetsnbextension==3.5.1
wrapt==1.11.2
WTForms==2.2.1
WTForms-Components==0.10.4
xenon==0.7.0
xmltodict==0.12.0
zipp==0.6.0

Other Dependencies

N/A

Minimally Reproducible Test Case

Expected Behavior

Celery worker should be able to re-establish connection to backend instance that was temporarily down when it's live again irrespective of whether BROKER_URL or broker_url config option is used.

Actual Behavior

When celery config uses BROKER_URL connection is reestablished, but with new style config broker_url connection isn't reestablished.

Issue 2:
The issue with redis key getting evicted every time. I read an old issue link. I have confirmed that my Redis instance is not hacked. In fact, we are using Secured Redis.

OperationalError("\nCannot route message for exchange 'reply.celery.pidbox': Table empty or key no longer exists.\nProbably the key ('_kombu.binding.reply.celery.pidbox') has been removed from the Redis database.\n",)

kombu==4.5.0

celery==4.3.0

redis==3.2.1

Is this some issue with redis?