Repository: celery/celery

Issue:
Celery 5 custom inspect commands doesn't work in the CLI
<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
bug reports which are incomplete.
-->


# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->


* [x] I have verified that the issue exists against the `master` branch of Celery.
* [x] This has already been asked to the [discussion group](https://groups.google.com/forum/#!forum/celery-users) first.
* [x] I have read the relevant section in the
  [contribution guide](http://docs.celeryproject.org/en/latest/contributing.html#other-bugs)
  on reporting bugs.
* [x] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical bug reports.
* [x] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22)
  for existing proposed fixes.
* [x] I have checked the [commit log](https://github.com/celery/celery/commits/master)
  to find out if the bug was already fixed in the master branch.
* [x] I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

## Mandatory Debugging Information
* [ ] I have included the output of ``celery -A proj report`` in the issue.
    (if you are not able to do this, then at least specify the Celery
     version affected).
* [ ] I have verified that the issue exists against the `master` branch of Celery.
* [ ] I have included the contents of ``pip freeze`` in the issue.
* [ ] I have included all the versions of all the external dependencies required
  to reproduce this bug.

## Optional Debugging Information
<!--
Try some of the below if you think they are relevant.
It will help us figure out the scope of the bug and how many users it affects.
-->


* [ ] I have tried reproducing the issue on more than one Python version
  and/or implementation.
* [ ] I have tried reproducing the issue on more than one message broker and/or
  result backend.
* [ ] I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
* [ ] I have tried reproducing the issue on more than one operating system.
* [ ] I have tried reproducing the issue on more than one workers pool.
* [ ] I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
* [ ] I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

## Related Issues and Possible Duplicates
<!--
Please make sure to search and mention any related issues
or possible duplicates to this issue as requested by the checklist above.

This may or may not include issues in other repositories that the Celery project
maintains or other repositories that are dependencies of Celery.

If you don't know how to mention issues, please refer to Github's documentation
on the subject: https://help.github.com/en/articles/autolinked-references-and-urls#issues-and-pull-requests
-->

#### Related Issues
* None

#### Possible Duplicates
* None

## Environment & Settings
<!-- Include the contents of celery --version below -->
**Celery version**:
<!-- Include the output of celery -A proj report below -->
<details>
<summary><b><code>celery report</code> Output:</b></summary>
<p>
software -> celery:5.0.5 (singularity) kombu:5.0.2 py:3.7.5
            billiard:3.6.3.0 py-amqp:5.0.3
platform -> system:Linux arch:64bit, ELF
            kernel version:4.4.0-19041-Microsoft imp:CPython
loader   -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled

deprecated_settings: None


```

```

</p>
</details>

# Steps to Reproduce
## Required Dependencies
<!-- Please fill the required dependencies to reproduce this issue -->


* **Minimal Python Version**: N/A or Unknown
* **Minimal Celery Version**: 5
* **Minimal Kombu Version**: N/A or Unknown
* **Minimal Broker Version**: N/A or Unknown
* **Minimal Result Backend Version**: N/A or Unknown
* **Minimal OS and/or Kernel Version**: N/A or Unknown
* **Minimal Broker Client Version**: N/A or Unknown
* **Minimal Result Backend Client Version**: N/A or Unknown

### Python Packages
<!-- Please fill the contents of pip freeze below -->
<details>
<summary><b><code>pip freeze</code> Output:</b></summary>

absl-py==0.9.0
aiohttp==3.6.2
aiomisc==11.0.0
amqp==5.0.3
appdirs==1.4.4
asgiref==3.2.10
asn1crypto==0.24.0
astor==0.8.1
async-timeout==3.0.1
atomicwrites==1.3.0
attrs==19.3.0
auth0-python==3.9.1
aws2-wrap==1.1.4
awscli==1.14.44
Babel==2.8.0
backcall==0.1.0
billiard==3.6.3.0
boto3==1.14.19
boto3-stubs==1.14.19.0
botocore==1.17.63
Brotli==1.0.9
cachetools==4.0.0
celery==5.0.5
certifi==2019.11.28
cffi==1.14.0
chardet==3.0.4
click==7.1.2
click-didyoumean==0.0.3
click-plugins==1.1.1
click-repl==0.1.6
colorama==0.3.7
colorlog==4.4.0
convertdate==2.2.2
cryptography==2.7
ddtrace==0.44.0
decorator==4.4.2
defusedxml==0.6.0
Django==3.1.2
django-celery-results==1.0.4
django-cors-headers==3.1.0
django-debug-toolbar==3.1.1
django-filter==2.1.0
django-prometheus==1.0.15
django-rest-auth==0.9.5
django-rest-framework-condition==0.1.1
djangorestframework==3.12.1
djangorestframework-jwt==1.11.0
docutils==0.15.2
ecdsa==0.15
ffmpeg-python==0.2.0
fitparse==1.1.0
flower==0.9.3
fusepy==3.0.1
future==0.18.2
gast==0.2.2
gitdb==4.0.2
gitdb2==3.0.0
GitPython==3.1.0
glfw==1.8.2
google-auth==1.11.3
google-auth-oauthlib==0.4.1
google-pasta==0.2.0
gprof2dot==2019.11.30
graphqlclient==0.2.4
grpcio==1.27.2
h5py==2.10.0
holidays==0.10.1
idna==2.6
imageio-ffmpeg==0.3.0
importlib-metadata==1.5.0
imutils==0.5.2
intervaltree==3.1.0
ipython==7.6.1
ipython-genutils==0.2.0
jedi==0.16.0
jmespath==0.9.5
joblib==0.14.1
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
keyring==10.6.0
keyrings.alt==3.0
kombu==5.0.2
kubernetes==12.0.0
logzio-python-handler==2.0.13
lxml==4.4.2
Markdown==3.2.1
memory-profiler==0.57.0
more-itertools==8.2.0
multidict==4.7.5
munkres==1.1.2
mypy-boto3==1.14.19.0
mypy-boto3-cloudformation==1.14.19.0
mypy-boto3-dynamodb==1.14.19.0
mypy-boto3-ec2==1.14.19.0
mypy-boto3-lambda==1.14.19.0
mypy-boto3-rds==1.14.19.0
mypy-boto3-s3==1.14.19.0
mypy-boto3-sqs==1.14.19.0
numpy==1.17.0
oauthlib==3.1.0
olefile==0.45.1
opencv-python==3.4.5.20
opt-einsum==3.2.0
packaging==20.3
pandas==1.1.4
parso==0.6.2
pdfkit==0.6.1
pexpect==4.8.0
pickleshare==0.7.5
Pillow==7.1.2
pipdeptree==1.0.0
pluggy==0.13.1
pprofile==2.0.2
prometheus-client==0.7.1
prompt-toolkit==2.0.10
protobuf==3.11.3
psutil==5.7.0
psycopg2-binary==2.8.3
ptyprocess==0.6.0
py==1.8.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
pycrypto==2.6.1
pyee==7.0.4
Pygments==2.6.1
pygobject==3.26.1
PyJWT==1.7.1
PyMeeus==0.3.7
PyOpenGL==3.1.0
pyparsing==2.4.6
pyperclip==1.7.0
pyppeteer==0.2.2
pyquaternion==0.9.5
PySocks==1.7.1
pytest==4.6.5
pytest-django==3.10.0
pytest-django-ordering==1.2.0
pytest-profiling==1.7.0
python-apt==1.6.5+ubuntu0.2
python-dateutil==2.8.0
python-dotenv==0.10.3
python-http-client==3.2.6
python-jose==3.0.1
python-json-logger==0.1.11
python-memcached==1.59
python3-openid==3.1.0
pytz==2019.3
pyxdg==0.25
PyYAML==5.3.1
redis==3.3.11
requests==2.22.0
requests-oauthlib==1.3.0
retry==0.9.2
roman==2.0.0
rsa==4.0
s3transfer==0.3.3
scikit-fmm==2019.1.30
scikit-learn==0.23.1
scipy==1.3.1
SecretStorage==2.3.1
sendgrid==6.4.1
Shapely==1.6.4.post2
six==1.12.0
smmap==3.0.1
smmap2==3.0.1
social-auth-app-django==3.1.0
social-auth-core==3.2.0
sortedcontainers==2.3.0
sqlparse==0.3.1
starkbank-ecdsa==1.1.0
tenacity==6.3.1
tensorboard==2.0.2
tensorflow==2.0.0
tensorflow-estimator==2.0.1
termcolor==1.1.0
threadpoolctl==2.1.0
tornado==5.1.1
tqdm==4.56.0
traitlets==4.3.3
tripy==1.0.0
twilio==6.29.3
typing-extensions==3.7.4.3
unattended-upgrades==0.1
urllib3==1.25.8
uWSGI==2.0.18
vine==5.0.0
wcwidth==0.1.8
websocket-client==0.57.0
websockets==8.1
Werkzeug==1.0.0
wrapt==1.12.1
xgboost==0.90
xlrd==1.2.0
XlsxWriter==1.2.8
xxhash==1.3.0
yappi==1.0
yarl==1.4.2
zipp==3.1.0
<p>

```

```

</p>
</details>

### Other Dependencies
<!--
Please provide system dependencies, configuration files
and other dependency information if applicable
-->
<details>
<p>
N/A
</p>
</details>

## Minimally Reproducible Test Case
<!--
Please provide a reproducible test case.
Refer to the Reporting Bugs section in our contribution guide.

We prefer submitting test cases in the form of a PR to our integration test suite.
If you can provide one, please mention the PR number below.
If not, please attach the most minimal code example required to reproduce the issue below.
If the test case is too large, please include a link to a gist or a repository below.
-->

<details>
<p>

```python

```

</p>
</details>

# Expected Behavior
<!-- Describe in detail what you expect to happen -->

# Actual Behavior
<!--
Describe in detail what actually happened.
Please include a backtrace and surround it with triple backticks (```).
In addition, include the Celery daemon logs, the broker logs,
the result backend logs and system logs below if they will help us debug
the issue.
-->

Hi, I think there is a bug with version 5+.
When using the guide in the docs for writing custom control commands and trying to run it from the CLI, it fails. When running this command from a python script, it is working (with broadcast). When using celery 4.4.3 on the same code, it also works.
The guide: https://docs.celeryproject.org/en/stable/userguide/workers.html#writing-your-own-remote-control-commands
the error:
$ celery -A route inspect current_prefetch_count
Usage: celery inspect [OPTIONS] [report|conf|query_task|clock|ping|stats|sched
uled|reserved|active|revoked|registered|objgraph|memsamp
le|memdump|active_queues]
Try 'celery inspect --help' for help.

Error: Invalid value for '[report|conf|query_task|clock|ping|stats|scheduled|reserved|active|revoked|registered|objgraph|memsample|memdump|active_queues]': invalid choice: current_prefetch_count. (choose from report, conf, query_task, clock, ping, stats, scheduled, rese
rved, active, revoked, registered, objgraph, memsample, memdump, active_queues)




Discussion:
what you get when you run celery inspect --help
Usage: celery inspect [OPTIONS] [report|conf|query_task|clock|ping|stats|sched
                      uled|reserved|active|revoked|registered|objgraph|memsamp
                      le|memdump|active_queues]

  Inspect the worker at runtime.

  Availability: RabbitMQ (AMQP) and Redis transports.

Remote Control Options:
  -t, --timeout FLOAT             Timeout in seconds waiting for reply.
  -d, --destination COMMA SEPARATED LIST
                                  Comma separated list of destination node
                                  names.

  -j, --json                      Use json as output format.

Options:
  --help  Show this message and exit.
may be @thedrow can share some insight
I have a monitoring script that (among many things) uses inspection API and calls active_queues(). With Celery 5.1.2 the scripts fails to work with key error, as active_queues() returns nothing... 4.4.x works as expected...
The code I used to test is here: https://gitlab.com/dejan/ceex (here I keep all my CElery EXperiments). If you run `celery -A ceex.inspect_node_stats worker -l debug` and try to `celery -A ceex.inspect_node_stats inspect node_stats` with 5.1.2 you will see it does not work. Again, try with 4.4.7 - it does work.
@dejlek The page is not available
Oops, thanks - I did not know it was created as private repo... It is open to public now.
Any update on this?
Looking the code of the control.py 

```
@click.argument("action", type=click.Choice([
    name for name, info in Panel.meta.items()
    if info.type == 'control' and info.visible
]))
```

this above decorator doesn't read the global Meta properly. I'm still trying to figure out why. 
I encountered this bug today. This bug is caused by the fact that `Panel.meta.items()` is evaluated when the `bin/control.py` file is imported. This happens _after_ the command definitions in `worker/control.py`, but _before_ any of my custom commands are defined. All the modules in `bin/` are imported when the `celery` command is run, so I don't see any quickfix or workaround.

I guess you could move away from `click.Choice` and accept any string as the command name instead. We would then need to validate the given string manually – the user's Celery app will be set up by the time we reach the function body, so their custom commands will be registered by then. Maybe a new flag, e.g. `--list`, could print the available commands and exit.

Would this be an acceptable solution? I may try to implement it if desired.
This bug is basically preventing us from moving to new(er) Celery as we have few critical inspect and control commands that we use all the time.
> I encountered this bug today. This bug is caused by the fact that `Panel.meta.items()` is evaluated when the `bin/control.py` file is imported. This happens _after_ the command definitions in `worker/control.py`, but _before_ any of my custom commands are defined. All the modules in `bin/` are imported when the `celery` command is run, so I don't see any quickfix or workaround.
> 
> I guess you could move away from `click.Choice` and accept any string as the command name instead. We would then need to validate the given string manually – the user's Celery app will be set up by the time we reach the function body, so their custom commands will be registered by then. Maybe a new flag, e.g. `--list`, could print the available commands and exit.
> 
> Would this be an acceptable solution? I may try to implement it if desired.

you can come with a draft proof of concept PR with relevant test for review

Resolving patch:
diff --git a/celery/bin/control.py b/celery/bin/control.py
--- a/celery/bin/control.py
+++ b/celery/bin/control.py
@@ -1,5 +1,6 @@
 """The ``celery control``, ``. inspect`` and ``. status`` programs."""
 from functools import partial
+from typing import Literal
 
 import click
 from kombu.utils.json import dumps
@@ -39,18 +40,69 @@ def _consume_arguments(meta, method, args):
         args[:] = args[i:]
 
 
-def _compile_arguments(action, args):
-    meta = Panel.meta[action]
+def _compile_arguments(command, args):
+    meta = Panel.meta[command]
     arguments = {}
     if meta.args:
         arguments.update({
-            k: v for k, v in _consume_arguments(meta, action, args)
+            k: v for k, v in _consume_arguments(meta, command, args)
         })
     if meta.variadic:
         arguments.update({meta.variadic: args})
     return arguments
 
 
+_RemoteControlType = Literal['inspect', 'control']
+
+
+def _verify_command_name(type_: _RemoteControlType, command: str) -> None:
+    choices = _get_commands_of_type(type_)
+
+    if command not in choices:
+        command_listing = ", ".join(choices)
+        raise click.UsageError(
+            message=f'Command {command} not recognized. Available {type_} commands: {command_listing}',
+        )
+
+
+def _list_option(type_: _RemoteControlType):
+    def callback(ctx: click.Context, param, value) -> None:
+        if not value:
+            return
+        choices = _get_commands_of_type(type_)
+
+        formatter = click.HelpFormatter()
+
+        with formatter.section(f'{type_.capitalize()} Commands'):
+            command_list = []
+            for command_name, info in choices.items():
+                if info.signature:
+                    command_preview = f'{command_name} {info.signature}'
+                else:
+                    command_preview = command_name
+                command_list.append((command_preview, info.help))
+            formatter.write_dl(command_list)
+        ctx.obj.echo(formatter.getvalue(), nl=False)
+        ctx.exit()
+
+    return click.option(
+        '--list',
+        is_flag=True,
+        help=f'List available {type_} commands and exit.',
+        expose_value=False,
+        is_eager=True,
+        callback=callback,
+    )
+
+
+def _get_commands_of_type(type_: _RemoteControlType) -> dict:
+    command_name_info_pairs = [
+        (name, info) for name, info in Panel.meta.items()
+        if info.type == type_ and info.visible
+    ]
+    return dict(sorted(command_name_info_pairs))
+
+
 @click.command(cls=CeleryCommand)
 @click.option('-t',
               '--timeout',
@@ -96,10 +148,8 @@ def status(ctx, timeout, destination, json, **kwargs):
 
 @click.command(cls=CeleryCommand,
                context_settings={'allow_extra_args': True})
-@click.argument("action", type=click.Choice([
-    name for name, info in Panel.meta.items()
-    if info.type == 'inspect' and info.visible
-]))
+@click.argument('command')
+@_list_option('inspect')
 @click.option('-t',
               '--timeout',
               cls=CeleryOption,
@@ -121,19 +171,19 @@ def status(ctx, timeout, destination, json, **kwargs):
               help='Use json as output format.')
 @click.pass_context
 @handle_preload_options
-def inspect(ctx, action, timeout, destination, json, **kwargs):
-    """Inspect the worker at runtime.
+def inspect(ctx, command, timeout, destination, json, **kwargs):
+    """Inspect the workers by sending them the COMMAND inspect command.
 
     Availability: RabbitMQ (AMQP) and Redis transports.
     """
+    _verify_command_name('inspect', command)
     callback = None if json else partial(_say_remote_command_reply, ctx,
                                          show_reply=True)
-    arguments = _compile_arguments(action, ctx.args)
+    arguments = _compile_arguments(command, ctx.args)
     inspect = ctx.obj.app.control.inspect(timeout=timeout,
                                           destination=destination,
                                           callback=callback)
-    replies = inspect._request(action,
-                               **arguments)
+    replies = inspect._request(command, **arguments)
 
     if not replies:
         raise CeleryCommandException(
@@ -153,10 +203,8 @@ def inspect(ctx, action, timeout, destination, json, **kwargs):
 
 @click.command(cls=CeleryCommand,
                context_settings={'allow_extra_args': True})
-@click.argument("action", type=click.Choice([
-    name for name, info in Panel.meta.items()
-    if info.type == 'control' and info.visible
-]))
+@click.argument('command')
+@_list_option('control')
 @click.option('-t',
               '--timeout',
               cls=CeleryOption,
@@ -178,16 +226,17 @@ def inspect(ctx, action, timeout, destination, json, **kwargs):
               help='Use json as output format.')
 @click.pass_context
 @handle_preload_options
-def control(ctx, action, timeout, destination, json):
-    """Workers remote control.
+def control(ctx, command, timeout, destination, json):
+    """Send the COMMAND control command to the workers.
 
     Availability: RabbitMQ (AMQP), Redis, and MongoDB transports.
     """
+    _verify_command_name('control', command)
     callback = None if json else partial(_say_remote_command_reply, ctx,
                                          show_reply=True)
     args = ctx.args
-    arguments = _compile_arguments(action, args)
-    replies = ctx.obj.app.control.broadcast(action, timeout=timeout,
+    arguments = _compile_arguments(command, args)
+    replies = ctx.obj.app.control.broadcast(command, timeout=timeout,
                                             destination=destination,
                                             callback=callback,
                                             reply=True,
diff --git a/t/unit/bin/proj/app_with_custom_cmds.py b/t/unit/bin/proj/app_with_custom_cmds.py
new file mode 100644
--- /dev/null
+++ b/t/unit/bin/proj/app_with_custom_cmds.py
@@ -0,0 +1,24 @@
+from celery import Celery
+from celery.worker.control import control_command, inspect_command
+
+
+@control_command(
+    args=[('a', int), ('b', int)],
+    signature='a b',
+)
+def custom_control_cmd(state, a, b):
+    """Ask the workers to reply with a and b."""
+    return {'ok': f'Received {a} and {b}'}
+
+
+@inspect_command(
+    args=[('x', int)],
+    signature='x',
+)
+def custom_inspect_cmd(state, x):
+    """Ask the workers to reply with x."""
+    return {'ok': f'Received {x}'}
+
+
+app = Celery(set_as_current=False)
+app.config_from_object('t.integration.test_worker_config')


Test patch:
diff --git a/t/unit/app/test_preload_cli.py b/t/unit/app/test_preload_cli.py
--- a/t/unit/app/test_preload_cli.py
+++ b/t/unit/app/test_preload_cli.py
@@ -1,34 +1,41 @@
+import contextlib
+from typing import Tuple
+from unittest.mock import patch
+
+import pytest
 from click.testing import CliRunner
 
 from celery.bin.celery import celery
 
 
-def test_preload_options(isolated_cli_runner: CliRunner):
-    # Verify commands like shell and purge can accept preload options.
-    # Projects like Pyramid-Celery's ini option should be valid preload
-    # options.
-
-    # TODO: Find a way to run these separate invoke and assertions
-    # such that order does not matter. Currently, running
-    # the "t.unit.bin.proj.pyramid_celery_app" first seems
-    # to result in cache or memoization of the option.
-    # As a result, the expected exception is not raised when
-    # the invoke on "t.unit.bin.proj.app" is run as a second
-    # call.
+@pytest.fixture(autouse=True)
+def reset_command_params_between_each_test():
+    with contextlib.ExitStack() as stack:
+        for command in celery.commands.values():
+            # We only need shallow copy -- preload options are appended to the list,
+            # existing options are kept as-is
+            params_copy = command.params[:]
+            patch_instance = patch.object(command, "params", params_copy)
+            stack.enter_context(patch_instance)
 
-    res_without_preload = isolated_cli_runner.invoke(
-        celery,
-        ["-A", "t.unit.bin.proj.app", "purge", "-f", "--ini", "some_ini.ini"],
-        catch_exceptions=True,
-    )
+        yield
 
-    assert "No such option: --ini" in res_without_preload.stdout
-    assert res_without_preload.exit_code == 2
 
+@pytest.mark.parametrize(
+    "subcommand_with_params",
+    [
+        ("purge", "-f"),
+        ("shell",),
+    ]
+)
+def test_preload_options(subcommand_with_params: Tuple[str, ...], isolated_cli_runner: CliRunner):
+    # Verify commands like shell and purge can accept preload options.
+    # Projects like Pyramid-Celery's ini option should be valid preload
+    # options.
     res_without_preload = isolated_cli_runner.invoke(
         celery,
-        ["-A", "t.unit.bin.proj.app", "shell", "--ini", "some_ini.ini"],
-        catch_exceptions=True,
+        ["-A", "t.unit.bin.proj.app", *subcommand_with_params, "--ini", "some_ini.ini"],
+        catch_exceptions=False,
     )
 
     assert "No such option: --ini" in res_without_preload.stdout
@@ -39,25 +46,11 @@ def test_preload_options(isolated_cli_runner: CliRunner):
         [
             "-A",
             "t.unit.bin.proj.pyramid_celery_app",
-            "purge",
-            "-f",
+            *subcommand_with_params,
             "--ini",
             "some_ini.ini",
         ],
-        catch_exceptions=True,
+        catch_exceptions=False,
     )
 
-    assert res_with_preload.exit_code == 0
-
-    res_with_preload = isolated_cli_runner.invoke(
-        celery,
-        [
-            "-A",
-            "t.unit.bin.proj.pyramid_celery_app",
-            "shell",
-            "--ini",
-            "some_ini.ini",
-        ],
-        catch_exceptions=True,
-    )
-    assert res_with_preload.exit_code == 0
+    assert res_with_preload.exit_code == 0, res_with_preload.stdout
diff --git a/t/unit/bin/test_control.py b/t/unit/bin/test_control.py
new file mode 100644
--- /dev/null
+++ b/t/unit/bin/test_control.py
@@ -0,0 +1,82 @@
+import os
+import re
+from unittest.mock import patch
+
+import pytest
+from click.testing import CliRunner
+
+from celery.bin.celery import celery
+from celery.platforms import EX_UNAVAILABLE
+
+_GLOBAL_OPTIONS = ['-A', 't.unit.bin.proj.app_with_custom_cmds', '--broker', 'memory://']
+_INSPECT_OPTIONS = ['--timeout', '0']  # Avoid waiting for the zero workers to reply
+
+
+@pytest.fixture(autouse=True)
+def clean_os_environ():
+    # Celery modifies os.environ when given the CLI option --broker memory://
+    # This interferes with other tests, so we need to reset os.environ
+    with patch.dict(os.environ, clear=True):
+        yield
+
+
+@pytest.mark.parametrize(
+    ('celery_cmd', 'custom_cmd'),
+    [
+        ('inspect', ('custom_inspect_cmd', '123')),
+        ('control', ('custom_control_cmd', '123', '456')),
+    ],
+)
+def test_custom_remote_command(celery_cmd, custom_cmd, isolated_cli_runner: CliRunner):
+    res = isolated_cli_runner.invoke(
+        celery,
+        [*_GLOBAL_OPTIONS, celery_cmd, *_INSPECT_OPTIONS, *custom_cmd],
+        catch_exceptions=False,
+    )
+    assert res.exit_code == EX_UNAVAILABLE, (res, res.stdout)
+    assert res.stdout.strip() == 'Error: No nodes replied within time constraint'
+
+
+@pytest.mark.parametrize(
+    ('celery_cmd', 'remote_cmd'),
+    [
+        # Test nonexistent commands
+        ('inspect', 'this_command_does_not_exist'),
+        ('control', 'this_command_does_not_exist'),
+        # Test commands that exist, but are of the wrong type
+        ('inspect', 'custom_control_cmd'),
+        ('control', 'custom_inspect_cmd'),
+    ],
+)
+def test_unrecognized_remote_command(celery_cmd, remote_cmd, isolated_cli_runner: CliRunner):
+    res = isolated_cli_runner.invoke(
+        celery,
+        [*_GLOBAL_OPTIONS, celery_cmd, *_INSPECT_OPTIONS, remote_cmd],
+        catch_exceptions=False,
+    )
+    assert res.exit_code == 2, (res, res.stdout)
+    assert f'Error: Command {remote_cmd} not recognized. Available {celery_cmd} commands: ' in res.stdout
+
+
+_expected_inspect_regex = (
+    '\n  custom_inspect_cmd x\\s+Ask the workers to reply with x\\.\n'
+)
+_expected_control_regex = (
+    '\n  custom_control_cmd a b\\s+Ask the workers to reply with a and b\\.\n'
+)
+
+
+@pytest.mark.parametrize(
+    ('celery_cmd', 'expected_regex'),
+    [
+        ('inspect', re.compile(_expected_inspect_regex, re.MULTILINE)),
+        ('control', re.compile(_expected_control_regex, re.MULTILINE)),
+    ],
+)
+def test_listing_remote_commands(celery_cmd, expected_regex, isolated_cli_runner: CliRunner):
+    res = isolated_cli_runner.invoke(
+        celery,
+        [*_GLOBAL_OPTIONS, celery_cmd, '--list'],
+    )
+    assert res.exit_code == 0, (res, res.stdout)
+    assert expected_regex.search(res.stdout)