Repository: celery/celery

Issue:
Unlike what doc states, --app does not behave like -A
## Steps to reproduce

* Try to run `celery --app foo *anything*`.
* Look at the "celery usage" output instead of the "*anything*" expected output.

## Expected behavior

"-A foo" and "--app foo" to behave the same way.

## Actual behavior

"-A foo" works, while "--app foo" just outputs the "celery usage" screen

## Environment

```
software -> celery:4.1.0 (latentcall) kombu:4.1.0 py:3.6.3
            billiard:3.5.0.3 py-amqp:2.2.2
platform -> system:Darwin arch:64bit imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:django-db
```


Discussion:
whats the status with celery 4.2.1?
Exact same with v4.2.1 (windowlicker).
There is in fact a behavioral difference that depends on arguments position:

`celery worker -A foo` works
`celery worker --app foo` works
`celery -A foo worker` works
`celery --app foo worker` does not work and show celery usage
do you have any suggested improvement in mind?
Well, that I think it should behave the same whatever the order of arguments is. I struggled and lost a bit of time because I first chosed the way that does not work (you know, Murphy's law). I tried to look in the argparsing code of celery to see if I could patch it, but wasn't clear to me how it worked, hence opening the issue. If you point me to the right direction (like where it should be done, and probably how it is tested today), I may be able to submit a patch.
We're adding both `-A` and `--app` here: https://github.com/celery/celery/blob/eeda18611ceed2560145f95ada4977a1b825d282/celery/bin/base.py#L299
They should be entirely interchangeable.
We haven't done anything else to define that argument.

The relevant test which verifies this passes (and you are welcome to run the tests yourself):
https://github.com/celery/celery/blob/f3ef0df5a6e4008d73fdc6e7b6c02c7edf361d5a/t/unit/bin/test_base.py#L184

I cannot reproduce this behavior locally on my Linux machine.
Could this be a bug in argparse that only reproduces on OSX? I find it unlikely but possible.

Unless you can provide me with new information on how to reproduce this bug (e.g. it only happens on OSX or some certain order of arguments) I think it's is safe to close this one.
This does not mean that there is no problem to be fixed. We're just not sure it's a problem within Celery's codebase.
@thedrow Thanks for the infos.

It does indeed appear with certain order of arguments, using "--app x" (not "--app=x", notice the lack of equal sign) *before the celery subcommand* (worker, inspect, ...). It is 100% reproductible on linux, here is how.

I launch a brand new linux image (docker) running "python:3" official image. The image was run using `docker run -it --rm python:3 bash`, and here is the "bootstrap" script I run by hand after launching the container:

```
$ pip install celery
$ cat > foo.py
from celery import Celery
app = Celery("foo")
^D
```
(edited for copypaste mistake)

Minimalist, right ? Now run in this image:

```
# celery -A foo report

software -> celery:4.2.1 (windowlicker) kombu:4.2.1 py:3.7.0
            billiard:3.5.0.4 py-amqp:2.3.2
platform -> system:Linux arch:64bit imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:disabled
```

Works as expected. Let's switch to `--app foo`:

```
# celery --app foo report
usage: celery <command> [options]

Show help screen and exit.

... etc ...
```

Note that it does work with `--app=foo`, just not with `--app foo` (notice the lack of equal sign). But celery usage says: `Global Options: -A APP, --app APP`.

I don't agree on the fact that it's "plain argparse". I think that the culprit code is somehow related to `CeleryCommand._relocate_args_from_start` which (as far as I understand) change `sys.argv` order before those parameters as passed to `argparse`, so that global arguments are moved after the celery command (like "worker", "inspect", etc).

I understand the goal, but it does not work 100% with arguments as understood by `argparse`, nor as advertised by the celery usage help.

Behaviour is 100% same on OSX and Linux. I tried to write an unit test but my lack of understanding the celery codebase (and especially, what the MockCommand does) made me fail at that.





(note that this bug is probably also affecting all other global options passed before the command name and that have arguments, like `--broker BROKER`, `--result-backend RESULT_BACKEND`, `--loader LOADER`, `--config CONFIG`, `--workdir WORKDIR` ...)

Example:

* `# celery report -A foo --workdir .` works
* `# celery --workdir . report -A foo` shows celery usage
* `# celery --workdir=. report -A foo` works again


  
(see https://github.com/celery/celery/blob/master/celery/bin/celery.py#L438:L473)
Thank you for the analysis. This is indeed a bug and I am able to reproduce it as you said.

I used ```celery --app=foo report``` which is why I thought this was not reproducible.
I did see the "=" sign in answers, hence the small emphasis on not using it.

Note that I'd gladly work on a fix, but the complexity added by the argparse-related hack to reorder arguments and the fact that for now, tests does not check cases with arguments before and after the subcommand makes me feel slightly uncomfortable about submitting a patch, as I don't know how to test it.

However, I do understand why the hack exist, as this is a struggle I often have with argparse (having "global arguments" work both before and after subcommand). I think it's a UX defect of argparse, and I understand that celery want to work around it. But this is something that can really annoy users, especially users like me that only use celery every *insert long period of time here* (and Murphy's law forces me to use the only case that does not work, by default ...).

If you can give a few pointers on how to test cases with arguments before and after the subcommand in the unit tests, I can give it a shot. It looks like the test you pointed me to skips argparsing, as MockCommand overrides `parse_options`:

```
  19   │ class MockCommand(Command):
  20   │     mock_args = ('arg1', 'arg2', 'arg3')
  21   │
  22   │     def parse_options(self, prog_name, arguments, command=None):
  23   │         options = {'foo': 'bar', 'prog_name': prog_name}
  24   │         return options, self.mock_args
``` 

Resolving patch:
diff --git a/celery/bin/base.py b/celery/bin/base.py
--- a/celery/bin/base.py
+++ b/celery/bin/base.py
@@ -474,7 +474,7 @@ def prepare_parser(self, parser):
         return parser
 
     def setup_app_from_commandline(self, argv):
-        preload_options = self.parse_preload_options(argv)
+        preload_options, remaining_options = self.parse_preload_options(argv)
         quiet = preload_options.get('quiet')
         if quiet is not None:
             self.quiet = quiet
@@ -510,18 +510,18 @@ def setup_app_from_commandline(self, argv):
             elif self.app is None:
                 self.app = self.get_app(loader=loader)
             if self.enable_config_from_cmdline:
-                argv = self.process_cmdline_config(argv)
+                remaining_options = self.process_cmdline_config(remaining_options)
         else:
             self.app = Celery(fixups=[])
 
         self._handle_user_preload_options(argv)
 
-        return argv
+        return remaining_options
 
     def _handle_user_preload_options(self, argv):
         user_preload = tuple(self.app.user_options['preload'] or ())
         if user_preload:
-            user_options = self._parse_preload_options(argv, user_preload)
+            user_options, _ = self._parse_preload_options(argv, user_preload)
             signals.user_preload_options.send(
                 sender=self, app=self.app, options=user_options,
             )
@@ -550,8 +550,8 @@ def _parse_preload_options(self, args, options):
         args = [arg for arg in args if arg not in ('-h', '--help')]
         parser = self.Parser()
         self.add_compat_options(parser, options)
-        namespace, _ = parser.parse_known_args(args)
-        return vars(namespace)
+        namespace, unknown_args = parser.parse_known_args(args)
+        return vars(namespace), unknown_args
 
     def add_append_opt(self, acc, opt, value):
         default = opt.default or []
diff --git a/celery/bin/celery.py b/celery/bin/celery.py
--- a/celery/bin/celery.py
+++ b/celery/bin/celery.py
@@ -435,6 +435,13 @@ def on_usage_error(self, exc, command=None):
         )))
 
     def _relocate_args_from_start(self, argv, index=0):
+        """Move options to the end of args.
+
+        This rewrites:
+            -l debug worker -c 3
+        to:
+            worker -c 3 -l debug
+        """
         if argv:
             rest = []
             while index < len(argv):
@@ -466,9 +473,6 @@ def _relocate_args_from_start(self, argv, index=0):
                 # we assume the first argument in argv[i:] is the command
                 # name.
                 return argv[index:] + rest
-            # if there are no more arguments then the last arg in rest'
-            # must be the command.
-            [rest.pop()] + rest
         return []
 
     def prepare_prog_name(self, name):


Test patch:
diff --git a/t/unit/bin/test_base.py b/t/unit/bin/test_base.py
--- a/t/unit/bin/test_base.py
+++ b/t/unit/bin/test_base.py
@@ -353,7 +353,7 @@ class TestCommand(Command):
             def add_preload_arguments(self, parser):
                 parser.add_argument('-s', action='store', dest='silent')
         cmd = TestCommand()
-        acc = cmd.parse_preload_options(['-s', 'yes'])
+        acc, _ = cmd.parse_preload_options(['-s', 'yes'])
         assert acc.get('silent') == 'yes'
 
     def test_parse_preload_options_with_equals_and_append(self):
@@ -363,7 +363,7 @@ class TestCommand(Command):
             def add_preload_arguments(self, parser):
                 parser.add_argument('--zoom', action='append', default=[])
         cmd = Command()
-        acc = cmd.parse_preload_options(['--zoom=1', '--zoom=2'])
+        acc, _ = cmd.parse_preload_options(['--zoom=1', '--zoom=2'])
 
         assert acc, {'zoom': ['1' == '2']}
 
@@ -371,6 +371,6 @@ def test_parse_preload_options_without_equals_and_append(self):
         cmd = Command()
         opt = Option('--zoom', action='append', default=[])
         cmd.preload_options = (opt,)
-        acc = cmd.parse_preload_options(['--zoom', '1', '--zoom', '2'])
+        acc, _ = cmd.parse_preload_options(['--zoom', '1', '--zoom', '2'])
 
         assert acc, {'zoom': ['1' == '2']}
diff --git a/t/unit/bin/test_celery.py b/t/unit/bin/test_celery.py
--- a/t/unit/bin/test_celery.py
+++ b/t/unit/bin/test_celery.py
@@ -16,6 +16,13 @@
 from celery.platforms import EX_FAILURE, EX_OK, EX_USAGE
 
 
+class MyApp(object):
+    user_options = {'preload': None}
+
+
+APP = MyApp()  # <-- Used by test_short_and_long_arguments_be_the_same
+
+
 class test__main__:
 
     def test_main(self):
@@ -204,6 +211,17 @@ def test_handle_argv(self):
         x.handle_argv('celery', ['start', 'foo'])
         x.execute.assert_called_with('start', ['start', 'foo'])
 
+    def test_short_and_long_arguments_be_the_same(self):
+        for arg in "--app", "-A":
+            appstr = '.'.join([__name__, 'APP'])
+            x = CeleryCommand(app=self.app)
+            x.execute = Mock()
+            with pytest.raises(SystemExit):
+                x.execute_from_commandline(['celery', arg, appstr, 'worker'])
+            assert x.execute.called
+            assert x.execute.call_args[0]
+            assert x.execute.call_args[0][0] == "worker"
+
     def test_execute(self):
         x = CeleryCommand(app=self.app)
         Help = x.commands['help'] = Mock()