Issue 1:
Checklist

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues in this issue

(If there are none, check this box anyway).

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Description

Tried to make documentation. make docs in / and make html in /docs. Got error:

Exception occurred:
File "/usr/lib/python3.8/site-packages/docutils/nodes.py", line 567, in __getitem__
return self.attributes[key]
KeyError: 'refdoc'
The full traceback has been saved in /tmp/sphinx-err-qa973efh.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
make[1]: *** [Makefile:55: html] Error 2
make[1]: Leaving directory '/home/lash/src/ext/python/celery/docs'
make: *** [Makefile:76: Documentation] Error 2

Tried with master@f2ddd894c32f642a20f03b805b97e460f4fb3b4f and v4.3.0

Using python@3.8.1, sphinx@2.2.1 and sphinx-celery@2.0.0

Suggestions

Issue 2:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version: 5.2.0rc1

celery report Output:

software -> celery:5.2.0rc1 (dawn-chorus) kombu:5.2.0rc1 py:3.8.12
billiard:3.6.4.0 redis:3.5.3
platform -> system:Linux arch:64bit
kernel version:5.4.129+ imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:django-db

ABSOLUTE_URL_OVERRIDES: {
}
ADMINS: []
ALLOWED_HOSTS: ['*']
APPEND_SLASH: True
AUTHENTICATION_BACKENDS: ['django.contrib.auth.backends.ModelBackend']
AUTH_PASSWORD_VALIDATORS: '********'
AUTH_USER_MODEL: 'app.user'
BASE_DIR: '/usr/src/app'
BROKER_URL: 'redis://localhost:6379/0'
CACHES: {
'default': { 'BACKEND': 'django_redis.cache.RedisCache',
'KEY_PREFIX': '********',
'LOCATION': 'redis://localhost:6379/1',
'OPTIONS': { 'CLIENT_CLASS': 'django_redis.client.DefaultClient'},
'TIMEOUT': None}}
CACHE_LOCATION: 'redis://localhost:6379/1'
CACHE_MIDDLEWARE_ALIAS: 'default'
CACHE_MIDDLEWARE_KEY_PREFIX: '********'
CACHE_MIDDLEWARE_SECONDS: 600
CELERY_ACCEPT_CONTENT: ['application/json']
CELERY_ALWAYS_EAGER: False
CELERY_BROKER_URL: 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND: 'django-db'
CELERY_RESULT_SERIALIZER: 'json'
CELERY_TASK_SERIALIZER: 'json'
CELERY_TIMEZONE: 'UTC'
CORE_APPS: ['app.apps.AppConfig']
CORS_ALLOW_CREDENTIALS: True
CORS_ORIGIN_WHITELIST: ['*']
CSRF_COOKIE_AGE: 31449600
CSRF_COOKIE_DOMAIN: None
CSRF_COOKIE_HTTPONLY: False
CSRF_COOKIE_NAME: 'csrftoken'
CSRF_COOKIE_PATH: '/'
CSRF_COOKIE_SAMESITE: 'Lax'
CSRF_COOKIE_SECURE: False
CSRF_FAILURE_VIEW: 'django.views.csrf.csrf_failure'
CSRF_HEADER_NAME: 'HTTP_X_CSRFTOKEN'
CSRF_TRUSTED_ORIGINS: []
CSRF_USE_SESSIONS: False
DATABASE_ROUTERS: '********'
DATA_UPLOAD_MAX_MEMORY_SIZE: 2621440
DATA_UPLOAD_MAX_NUMBER_FIELDS: 1000
DATETIME_FORMAT: 'N j, Y, P'
DATETIME_INPUT_FORMATS: ['%Y-%m-%d %H:%M:%S',
'%Y-%m-%d %H:%M:%S.%f',
'%Y-%m-%d %H:%M',
'%m/%d/%Y %H:%M:%S',
'%m/%d/%Y %H:%M:%S.%f',
'%m/%d/%Y %H:%M',
'%m/%d/%y %H:%M:%S',
'%m/%d/%y %H:%M:%S.%f',
'%m/%d/%y %H:%M']
DATE_FORMAT: 'N j, Y'
DATE_INPUT_FORMATS: ['%Y-%m-%d',
'%m/%d/%Y',
'%m/%d/%y',
'%b %d %Y',
'%b %d, %Y',
'%d %b %Y',
'%d %b, %Y',
'%B %d %Y',
'%B %d, %Y',
'%d %B %Y',
'%d %B, %Y']
DEBUG: 0
DEBUG_PROPAGATE_EXCEPTIONS: False
DECIMAL_SEPARATOR: '.'
DEFAULT_AUTO_FIELD: 'django.db.models.AutoField'
DEFAULT_CHARSET: 'utf-8'
DEFAULT_EXCEPTION_REPORTER: 'django.views.debug.ExceptionReporter'
DEFAULT_EXCEPTION_REPORTER_FILTER: 'django.views.debug.SafeExceptionReporterFilter'
DEFAULT_FILE_STORAGE: 'storages.backends.gcloud.GoogleCloudStorage'
DEFAULT_FROM_EMAIL: 'webmaster@localhost'
DEFAULT_HASHING_ALGORITHM: 'sha256'
DEFAULT_INDEX_TABLESPACE: ''
DEFAULT_TABLESPACE: ''
DISALLOWED_USER_AGENTS: []
FILE_UPLOAD_DIRECTORY_PERMISSIONS: None
FILE_UPLOAD_HANDLERS: ['django.core.files.uploadhandler.MemoryFileUploadHandler',
'django.core.files.uploadhandler.TemporaryFileUploadHandler']
FILE_UPLOAD_MAX_MEMORY_SIZE: 2621440
FILE_UPLOAD_PERMISSIONS: 420
FILE_UPLOAD_TEMP_DIR: None
FIRST_DAY_OF_WEEK: 0
FIXTURE_DIRS: []
FORCE_SCRIPT_NAME: None
FORMAT_MODULE_PATH: None
FORM_RENDERER: 'django.forms.renderers.DjangoTemplates'
GS_CREDENTIALS: <google.oauth2.service_account.Credentials object at 0x7fd7358aba00>
HOST: '******'
IGNORABLE_404_URLS: []
INSTALLED_APPS: ['django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'django.contrib.humanize',
'app.apps.AppConfig',
'rest_framework',
'rest_framework_api_key',
'django_extensions',
'django_celery_beat',
'django_celery_results',
'django_json_widget',
'django_object_actions',
'drf_yasg']
INTERNAL_IPS: ['127.0.0.1']
MONTH_DAY_FORMAT: 'F j'
NUMBER_GROUPING: 0
ROOT_URLCONF: 'app.urls'
SECRET_KEY: '********'
SECURE_BROWSER_XSS_FILTER: False
SECURE_CONTENT_TYPE_NOSNIFF: True
SECURE_HSTS_INCLUDE_SUBDOMAINS: False
SECURE_HSTS_PRELOAD: False
SECURE_HSTS_SECONDS: 0
SECURE_PROXY_SSL_HEADER:
('HTTP_X_FORWARDED_PROTO', 'https')
SECURE_REDIRECT_EXEMPT: []
SECURE_REFERRER_POLICY: 'same-origin'
SECURE_SSL_HOST: None
SECURE_SSL_REDIRECT: False
SERVER_EMAIL: 'root@localhost'
SESSION_CACHE_ALIAS: 'default'
SESSION_COOKIE_AGE: 1209600
SESSION_COOKIE_DOMAIN: None
SESSION_COOKIE_HTTPONLY: True
SESSION_COOKIE_NAME: 'sessionid'
SESSION_COOKIE_PATH: '/'
SESSION_COOKIE_SAMESITE: None
SESSION_COOKIE_SECURE: False
SESSION_ENGINE: 'django.contrib.sessions.backends.db'
SESSION_EXPIRE_AT_BROWSER_CLOSE: False
SESSION_FILE_PATH: None
SESSION_SAVE_EVERY_REQUEST: False
SESSION_SERIALIZER: 'django.contrib.sessions.serializers.JSONSerializer'
SETTINGS_MODULE: 'app.settings'
SHORT_DATETIME_FORMAT: 'm/d/Y P'
SHORT_DATE_FORMAT: 'm/d/Y'
SIGNING_BACKEND: 'django.core.signing.TimestampSigner'
USE_I18N: True
USE_L10N: True
USE_THOUSAND_SEPARATOR: False
USE_TZ: True
USE_X_FORWARDED_HOST: False
USE_X_FORWARDED_PORT: False
WSGI_APPLICATION: 'app.wsgi.application'
X_FRAME_OPTIONS: 'DENY'
YEAR_MONTH_FORMAT: 'F Y'
deprecated_settings: None
broker_transport_options: {
'global_keyprefix': '********'
}

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.8

Minimal Celery Version: 5.2.0rc1

Minimal Kombu Version: 5.2.0rc1

Minimal Broker Version: redis:3.5.3

Minimal Result Backend Version:

Minimal OS and/or Kernel Version: system:Linux arch:64bit kernel version:5.4.129+ imp:CPython

Minimal Broker Client Version: redis:3.5.3

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

$ pip list
Package Version
--------------------------------- -----------
aiohttp 3.7.4.post0
aiohttp-retry 2.4.0
amqp 5.0.6
appdirs 1.4.4
argh 0.26.2
asgiref 3.3.4
async-timeout 3.0.1
attrs 21.2.0
backcall 0.2.0
backports.entry-points-selectable 1.1.0
beautifulsoup4 4.9.3
billiard 3.6.4.0
blis 0.7.4
cached-property 1.5.2
cachetools 4.2.4
catalogue 1.0.0
celery 5.2.0rc1
certifi 2020.12.5
cfgv 3.3.1
chardet 3.0.4
click 8.0.3
click-didyoumean 0.3.0
click-plugins 1.1.1
click-repl 0.2.0
coreapi 2.3.3
coreschema 0.0.4
courlan 0.5.0
cssselect 1.1.0
cycler 0.10.0
cymem 2.0.5
dateparser 1.1.0
decorator 4.4.2
distlib 0.3.3
Django 3.2
django-bulk-update 2.2.0
django-celery-beat 2.2.0
django-celery-results 2.0.0
django-cors-headers 3.4.0
django-debug-toolbar 3.2
django-extensions 3.1.3
django-json-widget 1.0.1
django-object-actions 3.0.2
django-redis 4.12.1
django-sslserver 0.22
django-storages 1.12.1
django-timezone-field 4.1.1
djangorestframework 3.12.4
djangorestframework-api-key 2.0.0
djangorestframework-csv 2.1.1
djangorestframework-simplejwt 4.5.0
drf-writable-nested 0.6.3
drf-yasg 1.20.0
en-core-web-sm 2.3.1
filelock 3.3.0
flake8 3.8.3
flower 1.0.0
fsspec 0.8.7
future 0.18.2
fuzzysearch 0.7.3
fuzzywuzzy 0.18.0
gcsfs 0.7.2
google-api-core 1.23.0
google-api-python-client 1.12.7
google-auth 1.23.0
google-auth-httplib2 0.0.4
google-auth-oauthlib 0.4.2
google-cloud-automl 2.2.0
google-cloud-core 1.5.0
google-cloud-language 2.0.0
google-cloud-storage 1.40.0
google-crc32c 1.3.0
google-resumable-media 1.3.3
googleapis-common-protos 1.52.0
grpcio 1.41.0
gunicorn 20.0.4
h11 0.8.1
h2 3.2.0
hpack 3.0.0
html2text 2020.1.16
htmldate 0.8.1
httmock 1.4.0
http3 0.6.7
httplib2 0.20.1
humanize 3.12.0
hyperframe 5.2.0
identify 2.3.0
idna 2.10
inflection 0.5.1
influxdb-client 1.17.0
ipython 7.19.0
itypes 1.2.0
jedi 0.18.0
Jinja2 3.0.2
jusText 2.2.0
kiwisolver 1.3.2
kombu 5.2.0rc1
langcodes 3.2.1
libcst 0.3.21
lxml 4.6.3
MarkupSafe 2.0.1
matplotlib 3.3.3
mccabe 0.6.1
mock 4.0.2
more-itertools 8.10.0
multidict 5.2.0
murmurhash 1.0.5
mypy-extensions 0.4.3
nodeenv 1.6.0
numpy 1.19.1
oauthlib 3.1.0
ordered-set 3.1.1
ordered-set-stubs 0.1.3
packaging 21.0
pandas 1.3.1
parso 0.8.2
pexpect 4.8.0
pickleshare 0.7.5
Pillow 8.3.2
pip 21.3
plac 1.1.3
platformdirs 2.4.0
poetry-version 0.1.5
pre-commit 2.15.0
preshed 3.0.5
profanity-filter 1.3.3
prometheus-client 0.11.0
prompt-toolkit 3.0.20
proto-plus 1.19.5
protobuf 3.18.1
psycopg2 2.8.6
ptyprocess 0.7.0
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycodestyle 2.6.0
pydantic 1.8.2
pyee 7.0.3
pyflakes 2.2.0
Pygments 2.10.0
PyJWT 1.7.1
pyparsing 2.4.7
pyppeteer 0.0.25
python-crontab 2.5.1
python-dateutil 2.8.2
python-dotenv 0.14.0
pytz 2021.3
PyYAML 5.4.1
readability-lxml 0.8.1
redis 3.5.3
regex 2021.10.8
requests 2.25.0
requests-file 1.5.1
requests-oauthlib 1.3.0
rfc3986 1.5.0
rsa 4.7.2
ruamel.yaml 0.15.100
Rx 3.2.0
scipy 1.7.1
seaborn 0.10.1
searchconsole 0.0.4
sentry-sdk 0.14.4
setuptools 57.5.0
simplejson 3.17.2
six 1.16.0
slacker 0.14.0
soupsieve 2.2.1
spacy 2.3.7
sqlparse 0.4.2
srsly 1.0.5
tabulate 0.8.9
tblib 1.7.0
text-unidecode 1.3
thinc 7.4.5
tld 0.12.6
tldextract 3.0.2
toml 0.10.2
tomlkit 0.5.11
tornado 6.1
tqdm 4.62.3
trafilatura 0.8.1
traitlets 5.1.0
typing 3.7.4.3
typing-extensions 3.10.0.2
typing-inspect 0.7.1
tzlocal 2.1
unicodecsv 0.14.1
Unidecode 1.2.0
uritemplate 3.0.1
urllib3 1.26.7
vine 5.0.0
virtualenv 20.8.1
wasabi 0.8.2
watchdog 2.0.2
wcwidth 0.2.5
websockets 6.0
wheel 0.37.0
xlwt 1.3.0
yarl 1.7.0

Other Dependencies

N/A

Minimally Reproducible Test Case

Celery instantiation is being done as follow:

# app/app/celery.py
import os
import logging
from celery import Celery
logger = logging.getLogger(__name__)

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "app.settings")

app = Celery("app")
app.config_from_object("django.conf:settings", namespace="CELERY")
# Prefix celery messages in broker configured.
app.conf.broker_transport_options = {
'global_keyprefix': '_prefixed_' # <====
}

app.autodiscover_tasks()

@app.task(bind=True)
def debug_task(self):
logger.info('Request: {0!r}'.format(self.request))

Please note that I have used transport setting: 'global_keyprefix': '_prefixed_' (i.e. a recent feature) which works fine as far as tasks processing is concerned but monitoring does not work (celery inspect + flower that uses the same celery inspect under the hood).

Redis state is as expected:

$ localhost:6379> keys *prefixed*
1) "_prefixed__kombu.binding.admin"
2) "_prefixed__kombu.binding.celeryev"
3) "_prefixed__kombu.binding.low_priority"
4) "_prefixed__kombu.binding.celery"
5) "_prefixed__kombu.binding.celery.pidbox"

Tests:

Python Django Shell Version:

2021-10-17 11:18:04,196 DEBUG selector_events Using selector: EpollSelector
Python 3.8.12 (default, Oct 12 2021, 03:05:25)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from app.celery import app as celery_app

In [2]: i = celery_app.control.inspect()

In [3]: i.clock()

In [4]: i.active()

In [5]: i.scheduled()

In [6]: i.reserved()

In [7]: i.stats()

In [8]: i.ping()

In [9]: i.active_queues()

In [10]: i.conf()

In [11]: i.memdump()

In [12]: i.objgraph()

CLI Version:

$ celery -A app.celery inspect registered
TEST MODE: False
Loading env file /dev/null
Error: No nodes replied within time constraint.

Notice inspect calls through django shell does not work AND CLI output ☝️ which clarifies to me that the nodes being listened might not be correct OR not being detected.

Expected Behavior

celery inspect works as expected and is able to see transport option global_keyprefix

Actual Behavior

celery inspect has stopped working and from my limited debugging it looks like we didn't account for transport option global_keyprefix while inspecting and not able to read from the keys that exist in Redis.. Please look at "Minimally Reproducible Test Case" section for more details..

Let me know if anything else's needed from my end in order to move forward. Thanks!!