Repository: celery/celery

Issue:
workdir option no longer works after upgrade to 5.0.1
<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
bug reports which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->
- [x] I have verified that the issue exists against the `master` branch of Celery.
- [x] This has already been asked to the [discussion group](https://groups.google.com/forum/#!forum/celery-users) first.
- [x] I have read the relevant section in the
  [contribution guide](http://docs.celeryproject.org/en/latest/contributing.html#other-bugs)
  on reporting bugs.
- [x] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical bug reports.
- [x] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22)
  for existing proposed fixes.
- [x] I have checked the [commit log](https://github.com/celery/celery/commits/master)
  to find out if the bug was already fixed in the master branch.
- [x] I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

## Mandatory Debugging Information

- [] I have included the output of ``celery -A proj report`` in the issue.
    (if you are not able to do this, then at least specify the Celery
     version affected).
- [x] I have verified that the issue exists against the `master` branch of Celery.
- [x] I have included the contents of ``pip freeze`` in the issue.
- [x] I have included all the versions of all the external dependencies required
  to reproduce this bug.

## Optional Debugging Information
<!--
Try some of the below if you think they are relevant.
It will help us figure out the scope of the bug and how many users it affects.
-->
- [ ] I have tried reproducing the issue on more than one Python version
  and/or implementation.
- [ ] I have tried reproducing the issue on more than one message broker and/or
  result backend.
- [ ] I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
- [ ] I have tried reproducing the issue on more than one operating system.
- [ ] I have tried reproducing the issue on more than one workers pool.
- [ ] I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
- [x] I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

## Related Issues and Possible Duplicates
<!--
Please make sure to search and mention any related issues
or possible duplicates to this issue as requested by the checklist above.

This may or may not include issues in other repositories that the Celery project
maintains or other repositories that are dependencies of Celery.

If you don't know how to mention issues, please refer to Github's documentation
on the subject: https://help.github.com/en/articles/autolinked-references-and-urls#issues-and-pull-requests
-->

#### Related Issues

- None

#### Possible Duplicates

- None

## Environment & Settings
<!-- Include the contents of celery --version below -->
**Celery version**:
<!-- Include the output of celery -A proj report below -->
<details>
<summary><b><code>celery report</code> Output:</b></summary>
<p>

software -> celery:5.0.1 (singularity) kombu:5.0.2 py:3.8.6
            billiard:3.6.3.0 redis:3.5.3
platform -> system:Linux arch:64bit, ELF
            kernel version:4.14.200-155.322.amzn2.x86_64 imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:****************

</p>
</details>

# Steps to Reproduce

## Required Dependencies
<!-- Please fill the required dependencies to reproduce this issue -->
* **Minimal Python Version**: N/A or Unknown
* **Minimal Celery Version**: 5.0.1
* **Minimal Kombu Version**: N/A or Unknown
* **Minimal Broker Version**: N/A or Unknown
* **Minimal Result Backend Version**: N/A or Unknown
* **Minimal OS and/or Kernel Version**: N/A or Unknown
* **Minimal Broker Client Version**: N/A or Unknown
* **Minimal Result Backend Client Version**: N/A or Unknown

### Python Packages
<!-- Please fill the contents of pip freeze below -->
<details>
<summary><b><code>pip freeze</code> Output:</b></summary>
<p>

```
```

</p>
</details>

### Other Dependencies
<!--
Please provide system dependencies, configuration files
and other dependency information if applicable
-->
<details>
<p>
N/A
</p>
</details>

## Minimally Reproducible Test Case
<!--
Please provide a reproducible test case.
Refer to the Reporting Bugs section in our contribution guide.

We prefer submitting test cases in the form of a PR to our integration test suite.
If you can provide one, please mention the PR number below.
If not, please attach the most minimal code example required to reproduce the issue below.
If the test case is too large, please include a link to a gist or a repository below.
-->

<details>
<p>

```python
```

</p>
</details>

# Expected Behavior
<!-- Describe in detail what you expect to happen -->
I think we are supposed to be able to run celery multi start in other directories with --workdir option passed.

# Actual Behavior
<!--
Describe in detail what actually happened.
Please include a backtrace and surround it with triple backticks (```).
In addition, include the Celery daemon logs, the broker logs,
the result backend logs and system logs below if they will help us debug
the issue.
-->
After upgrading to 5.0.1 from 4.x, the celery multi start command stopped working when not running from the workdir, even with workdir option passed in the command. The daemon service could not start as a result.

/opt/Cranberry/venv/bin/celery multi start worker1 --workdir=/opt/Cranberry/ --app=Cranberry

Error from the command above:
Error: Invalid value for '-A' / '--app':
Unable to load celery application.
Module 'cranberry' has no attribute 'celery'
        > worker1@ip-10-0-5-210.ec2.internal: * Child terminated with exit code 2
FAILED

I inserted print('Workdir: ', workdir) at:  https://github.com/celery/celery/blob/7c3da03a07882ca86b801ad78dd509a67cba60af/celery/bin/celery.py#L123 
And it prints:
Workdir:  None

Any help on this is much appreciated!

Updates 14:50:
Tried to debug it a little more and it seems multi ends up calling subprocess with "celery --app=Cranberry worker --workdir=/opt/Cranberry/ ..."
The command goes to celery.bin.celery.APP.convert.
I tried to print ctx.params there, and notice that the --workdir param only shows up when it's before --app in command:

> (venv) 2 ec2-user@ip:/opt$ celery --app=Cranberry --workdir=/opt/Cranberry/ worker
> app convert:  Cranberry {}
> Usage: celery [OPTIONS] COMMAND [ARGS]...
> 
> Error: Invalid value for '-A' / '--app':
> Unable to load celery application.
> The module Cranberry was not found.
> 
> (venv) 2 ec2-user@:/opt$ celery --workdir=/opt/Cranberry/ --app=Cranberry worker
> app convert:  Cranberry {'workdir': '/opt/Cranberry/'}
> Usage: celery [OPTIONS] COMMAND [ARGS]...
> 
> Error: Invalid value for '-A' / '--app':

And this params does not seem to be used in os.chdir or get passed to find_app.


Discussion:
@thedrow - looks like we do the chdir after we load the app since the click refactor. IIUC we need to respect `ctx.params` before we attempt to find the app. I have a draft PR at #6446
> @thedrow - looks like we do the chdir after we load the app since the click refactor.
Why is that the case? It's rather a surprising side effect.
@maybe-sybr @thedrow Thank you for the attention on this! I tried the code change in PR and it seems there's still a problem with args order, that when --app appears before --workdir in the command, the --workdir option would not get parsed to ctx.params.

Unfortunately that's how the celeryd script works. It put --workdir before --app, and then in celery.apps.multi the order gets reversed. https://github.com/celery/celery/blob/0833a270fae4738e128d56a63d0c4446ba0b1927/extra/generic-init.d/celeryd#L257
https://github.com/celery/celery/blob/0833a270fae4738e128d56a63d0c4446ba0b1927/celery/apps/multi.py#L178

I can modify the daemon script to make it work for my use case, but I'm not sure whether there're other impacts with the arg order problem or not.
It is a problem. The arguments order should not have any side effects. 
That makes sense, AFAIK click will instantiate args using the `type` option passed in immediately upon processing so having a partial command line context at that point isn't too surprising - annoying though. I think what we might need is to defer app loading until we actually reach the `celery()` function when we know that the command line options context has been fully parsed,
I think we could potentially make workdir an eager option with a chdir callback per [0] to fix this misbehaviour properly.

[0] https://click.palletsprojects.com/en/7.x/options/?highlight=callback#callbacks-and-eager-options

Resolving patch:
diff --git a/celery/bin/celery.py b/celery/bin/celery.py
--- a/celery/bin/celery.py
+++ b/celery/bin/celery.py
@@ -1,5 +1,6 @@
 """Celery Command Line Interface."""
 import os
+import pathlib
 import traceback
 
 import click
@@ -94,6 +95,9 @@ def convert(self, value, param, ctx):
               help_group="Global Options")
 @click.option('--workdir',
               cls=CeleryOption,
+              type=pathlib.Path,
+              callback=lambda _, __, wd: os.chdir(wd) if wd else None,
+              is_eager=True,
               help_group="Global Options")
 @click.option('-C',
               '--no-color',
@@ -121,8 +125,6 @@ def celery(ctx, app, broker, result_backend, loader, config, workdir,
         click.echo(ctx.get_help())
         ctx.exit()
 
-    if workdir:
-        os.chdir(workdir)
     if loader:
         # Default app takes loader from this env (Issue #1066).
         os.environ['CELERY_LOADER'] = loader


Test patch: