Repository: celery/celery

Issue:
Set allow_extra_args to True in context_settings of shell and purge commands
<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
enhancement requests which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->

- [ x ] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Enhancement%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical enhancement to an existing feature.
- [ x ] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22Issue+Type%3A+Enhancement%22+-label%3A%22Category%3A+Documentation%22)
  for existing proposed enhancements.
- [ x ] I have checked the [commit log](https://github.com/celery/celery/commits/main)
  to find out if the if the same enhancement was already implemented in the
  main branch.
- [ x ] I have included all related issues and possible duplicate issues in this issue
      (If there are none, check this box anyway).

## Related Issues and Possible Duplicates
<!--
Please make sure to search and mention any related issues
or possible duplicates to this issue as requested by the checklist above.

This may or may not include issues in other repositories that the Celery project
maintains or other repositories that are dependencies of Celery.

If you don't know how to mention issues, please refer to Github's documentation
on the subject: https://help.github.com/en/articles/autolinked-references-and-urls#issues-and-pull-requests
-->

#### Related Issues

https://github.com/sontek/pyramid_celery/issues/101

#### Possible Duplicates

- None

# Brief Summary
<!--
Please include a brief summary of what the enhancement is
and why it is needed.
-->

The  [pyramid-celery](https://github.com/sontek/pyramid_celery) package adds `ini` and `ini-var` options to the standard commands. These allow the standard Pyramid `ini` configuration files to config Celery. With current Celery however, only the `worker`, `beat`, and `events` are able to be hooked into pyramid-celery's approach because those commands are set with.

```
context_settings={
    'allow_extra_args': True
}
```
See `beat` for example: https://github.com/celery/celery/blob/f3a2cf45a69b443cac6c79a5c85583c8bd91b0a3/celery/bin/beat.py#L11

The `shell` and `purge` commands for some reason do not have context_settings and so do not set `allow_extra_args`. See for example in `shell`, https://github.com/celery/celery/blob/f3a2cf45a69b443cac6c79a5c85583c8bd91b0a3/celery/bin/shell.py#LL82C1-L117C24

The consequence is that the `ini` command options cause errors since these are extra and unknown args to these commands. Excluding the `ini` config casues  `shell` when it comes up to not get the task registration configs and `purge` isn't able to see queues.

# Design

## Architectural Considerations
<!--
If more components other than Celery are involved,
describe them here and the effect it would have on Celery.
-->
[pyramid-celery](https://github.com/sontek/pyramid_celery) is on [pypi](https://pypi.org/project/pyramid-celery). Allows you to use pyramid .ini files to configure celery and have your pyramid configuration inside celery tasks.

## Proposed Behavior
<!--
Please describe in detail how this enhancement is going to change the behavior
of an existing feature.
Describe what happens in case of failures as well if applicable.
-->
This should not affect Celery functionality, but will allow pyramid-celery to function as expected. See related issue tracked in pyramid-celery here https://github.com/sontek/pyramid_celery/issues/101

## Proposed UI/UX
<!--
Please provide your ideas for the API, CLI options,
configuration key names etc. that will be adjusted for this enhancement.
-->

Celery should add `context_settings` with `allow_extra_args`,
```
context_settings={
    'allow_extra_args': True
}
```
to both the `shell` and `purge` commands. This will allow projects like pyramid-celery to hook in their own custom configuration options using the recommended (by Celery) `celery_app.user_options['preload'].add()` and/or `celery_app.user_options[option].add()` approaches. See [here](https://github.com/celery/celery/blob/e7b47a62d789557cf18ed0e56e2dfb99a51a62f7/docs/userguide/extending.rst#adding-new-command-line-options). This will also make `shell` and `purge` commands consistent with other commands like `worker`, `beat`, and `events` that already have these settings.

## Diagrams
<!--
Please include any diagrams that might be relevant
to the implementation of this enhancement such as:
* Class Diagrams
* Sequence Diagrams
* Activity Diagrams
You can drag and drop images into the text box to attach them to this issue.
-->
N/A

## Alternatives
<!--
If you have considered any alternative implementations
describe them in detail below.
-->
None



Discussion:
Hey @dpdoughe :wave:,
Thank you for opening an issue. We will get back to you as soon as we can.
Also, check out our [Open Collective](https://opencollective.com/celery) and consider backing us - every little helps!

We also offer priority support for our sponsors.
If you require immediate assistance please consider sponsoring us.

The inability to configure and run `purge` or `shell` commands with pyramid_celery persists with Celery v5.3.1
I am open to contributions in this regard. my expectation is adding adding missing tests for the newly proposed/missing features

Resolving patch:
diff --git a/celery/bin/purge.py b/celery/bin/purge.py
--- a/celery/bin/purge.py
+++ b/celery/bin/purge.py
@@ -5,7 +5,9 @@
 from celery.utils import text
 
 
-@click.command(cls=CeleryCommand)
+@click.command(cls=CeleryCommand, context_settings={
+    'allow_extra_args': True
+})
 @click.option('-f',
               '--force',
               cls=CeleryOption,
@@ -26,7 +28,7 @@
               help="Comma separated list of queues names not to purge.")
 @click.pass_context
 @handle_preload_options
-def purge(ctx, force, queues, exclude_queues):
+def purge(ctx, force, queues, exclude_queues, **kwargs):
     """Erase all messages from all known task queues.
 
     Warning:
diff --git a/celery/bin/shell.py b/celery/bin/shell.py
--- a/celery/bin/shell.py
+++ b/celery/bin/shell.py
@@ -79,7 +79,9 @@ def _invoke_default_shell(locals):
         _invoke_ipython_shell(locals)
 
 
-@click.command(cls=CeleryCommand)
+@click.command(cls=CeleryCommand, context_settings={
+    'allow_extra_args': True
+})
 @click.option('-I',
               '--ipython',
               is_flag=True,
@@ -117,7 +119,7 @@ def _invoke_default_shell(locals):
 @handle_preload_options
 def shell(ctx, ipython=False, bpython=False,
           python=False, without_tasks=False, eventlet=False,
-          gevent=False):
+          gevent=False, **kwargs):
     """Start shell session with convenient access to celery symbols.
 
     The following symbols will be added to the main globals:
diff --git a/t/unit/bin/proj/pyramid_celery_app.py b/t/unit/bin/proj/pyramid_celery_app.py
new file mode 100644
--- /dev/null
+++ b/t/unit/bin/proj/pyramid_celery_app.py
@@ -0,0 +1,53 @@
+from unittest.mock import MagicMock, Mock
+
+from click import Option
+
+from celery import Celery
+
+# This module defines a mocked Celery application to replicate
+# the behavior of Pyramid-Celery's configuration by preload options.
+# Preload options should propagate to commands like shell and purge etc.
+#
+# The Pyramid-Celery project https://github.com/sontek/pyramid_celery
+# assumes that you want to configure Celery via an ini settings file.
+# The .ini files are the standard configuration file for Pyramid
+# applications.
+# See https://docs.pylonsproject.org/projects/pyramid/en/latest/quick_tutorial/ini.html
+#
+
+app = Celery(set_as_current=False)
+app.config_from_object("t.integration.test_worker_config")
+
+
+class PurgeMock:
+    def queue_purge(self, queue):
+        return 0
+
+
+class ConnMock:
+    default_channel = PurgeMock()
+    channel_errors = KeyError
+
+
+mock = Mock()
+mock.__enter__ = Mock(return_value=ConnMock())
+mock.__exit__ = Mock(return_value=False)
+
+app.connection_for_write = MagicMock(return_value=mock)
+
+# Below are taken from pyramid-celery's __init__.py
+# Ref: https://github.com/sontek/pyramid_celery/blob/cf8aa80980e42f7235ad361874d3c35e19963b60/pyramid_celery/__init__.py#L25-L36 # noqa: E501
+ini_option = Option(
+    (
+        "--ini",
+        "-i",
+    ),
+    help="Paste ini configuration file.",
+)
+
+ini_var_option = Option(
+    ("--ini-var",), help="Comma separated list of key=value to pass to ini."
+)
+
+app.user_options["preload"].add(ini_option)
+app.user_options["preload"].add(ini_var_option)


Test patch:
diff --git a/t/unit/app/test_preload_cli.py b/t/unit/app/test_preload_cli.py
new file mode 100644
--- /dev/null
+++ b/t/unit/app/test_preload_cli.py
@@ -0,0 +1,63 @@
+from click.testing import CliRunner
+
+from celery.bin.celery import celery
+
+
+def test_preload_options(isolated_cli_runner: CliRunner):
+    # Verify commands like shell and purge can accept preload options.
+    # Projects like Pyramid-Celery's ini option should be valid preload
+    # options.
+
+    # TODO: Find a way to run these separate invoke and assertions
+    # such that order does not matter. Currently, running
+    # the "t.unit.bin.proj.pyramid_celery_app" first seems
+    # to result in cache or memoization of the option.
+    # As a result, the expected exception is not raised when
+    # the invoke on "t.unit.bin.proj.app" is run as a second
+    # call.
+
+    res_without_preload = isolated_cli_runner.invoke(
+        celery,
+        ["-A", "t.unit.bin.proj.app", "purge", "-f", "--ini", "some_ini.ini"],
+        catch_exceptions=True,
+    )
+
+    assert "No such option: --ini" in res_without_preload.stdout
+    assert res_without_preload.exit_code == 2
+
+    res_without_preload = isolated_cli_runner.invoke(
+        celery,
+        ["-A", "t.unit.bin.proj.app", "shell", "--ini", "some_ini.ini"],
+        catch_exceptions=True,
+    )
+
+    assert "No such option: --ini" in res_without_preload.stdout
+    assert res_without_preload.exit_code == 2
+
+    res_with_preload = isolated_cli_runner.invoke(
+        celery,
+        [
+            "-A",
+            "t.unit.bin.proj.pyramid_celery_app",
+            "purge",
+            "-f",
+            "--ini",
+            "some_ini.ini",
+        ],
+        catch_exceptions=True,
+    )
+
+    assert res_with_preload.exit_code == 0
+
+    res_with_preload = isolated_cli_runner.invoke(
+        celery,
+        [
+            "-A",
+            "t.unit.bin.proj.pyramid_celery_app",
+            "shell",
+            "--ini",
+            "some_ini.ini",
+        ],
+        catch_exceptions=True,
+    )
+    assert res_with_preload.exit_code == 0