Repository: celery/celery

Issue:
Celery 4.4.3 always trying create /var/run/celery directory, even if it's not needed.
<!--
Please make sure to search and mention any related issues
or possible duplicates to this issue as requested by the checklist above.

This may or may not include issues in other repositories that the Celery project
maintains or other repositories that are dependencies of Celery.

If you don't know how to mention issues, please refer to Github's documentation
on the subject: https://help.github.com/en/articles/autolinked-references-and-urls#issues-and-pull-requests
-->

#### Related Issues

#6017  Celery Multi creates pid and log files in the wrong directory

## Environment & Settings
<!-- Include the contents of celery --version below -->
**Celery version**:
<!-- Include the output of celery -A proj report below -->
<summary><b><code>celery report</code> Output:</b></summary>

```
# celery report

software -> celery:4.4.3 (cliffs) kombu:4.6.9 py:3.7.7
            billiard:3.6.3.0 py-amqp:2.6.0
platform -> system:Linux arch:64bit, ELF
            kernel version:4.19.0-8-amd64 imp:CPython
loader   -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled

```

# Steps to Reproduce

## Required Dependencies
<!-- Please fill the required dependencies to reproduce this issue -->
* **Minimal Celery Version**: 4.4.3

</p>
</details>

## Minimally Reproducible Test Case
<!--
Please provide a reproducible test case.
Refer to the Reporting Bugs section in our contribution guide.

We prefer submitting test cases in the form of a PR to our integration test suite.
If you can provide one, please mention the PR number below.
If not, please attach the most minimal code example required to reproduce the issue below.
If the test case is too large, please include a link to a gist or a repository below.
-->

```
celery multi start ... --pidfile=/var/run/demo/celeryd-%n.pid
```

# Expected Behavior
<!-- Describe in detail what you expect to happen -->
celery runs

# Actual Behavior
<!--
Describe in detail what actually happened.
Please include a backtrace and surround it with triple backticks (```).
In addition, include the Celery daemon logs, the broker logs,
the result backend logs and system logs below if they will help us debug
the issue.
-->
start failed
```
celery multi v4.4.3 (cliffs)
_annotate_with_default_opts: print options
OrderedDict([('--app', 'service.celery:app'),
             ('--pidfile', '/var/run/demo/celeryd-%n.pid'),
             ('--logfile', '/var/log/demo/celeryd-%n%I.log'),
             ('--loglevel', 'INFO'),
             ('--workdir', '/var/lib/demo-celery'),
             ('--events', None),
             ('--heartbeat-interval', '5'),
             ('--without-gossip', None),
             ('--queues', 'high'),
             ('--concurrency', '1'),
             ('-n', 'high@celeryd.worker')])
Traceback (most recent call last):
  File "/var/www/misc/ve-2006011156/bin/celery", line 8, in <module>
    sys.exit(main())
  File "/var/www/misc/ve-2006011156/lib/python3.7/site-packages/celery/__main__.py", line 16, in main
    _main()
  File "/var/www/misc/ve-2006011156/lib/python3.7/site-packages/celery/bin/celery.py", line 322, in main
    cmd.execute_from_commandline(argv)
  File "/var/www/misc/ve-2006011156/lib/python3.7/site-packages/celery/bin/celery.py", line 495, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/var/www/misc/ve-2006011156/lib/python3.7/site-packages/celery/bin/base.py", line 305, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/var/www/misc/ve-2006011156/lib/python3.7/site-packages/celery/bin/celery.py", line 487, in handle_argv
    return self.execute(command, argv)
  File "/var/www/misc/ve-2006011156/lib/python3.7/site-packages/celery/bin/celery.py", line 419, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/var/www/misc/ve-2006011156/lib/python3.7/site-packages/celery/bin/celery.py", line 335, in run_from_argv
    return cmd.execute_from_commandline([command] + argv)
  File "/var/www/misc/ve-2006011156/lib/python3.7/site-packages/celery/bin/multi.py", line 266, in execute_from_commandline
    return self.call_command(argv[0], argv[1:])
  File "/var/www/misc/ve-2006011156/lib/python3.7/site-packages/celery/bin/multi.py", line 273, in call_command
    return self.commands[command](*argv) or EX_OK
  File "/var/www/misc/ve-2006011156/lib/python3.7/site-packages/celery/bin/multi.py", line 143, in _inner
    return fun(self, *args, **kwargs)
  File "/var/www/misc/ve-2006011156/lib/python3.7/site-packages/celery/bin/multi.py", line 151, in _inner
    return fun(self, self.cluster_from_argv(argv), **kwargs)
  File "/var/www/misc/ve-2006011156/lib/python3.7/site-packages/celery/bin/multi.py", line 361, in cluster_from_argv
    _, cluster = self._cluster_from_argv(argv, cmd=cmd)
  File "/var/www/misc/ve-2006011156/lib/python3.7/site-packages/celery/bin/multi.py", line 366, in _cluster_from_argv
    return p, self.Cluster(list(nodes), cmd=cmd)
  File "/var/www/misc/ve-2006011156/lib/python3.7/site-packages/celery/apps/multi.py", line 306, in <genexpr>
    for name in names
  File "/var/www/misc/ve-2006011156/lib/python3.7/site-packages/celery/apps/multi.py", line 314, in _node_from_options
    p.optmerge(namespace, options), p.passthrough)
  File "/var/www/misc/ve-2006011156/lib/python3.7/site-packages/celery/apps/multi.py", line 136, in __init__
    options or OrderedDict())
  File "/var/www/misc/ve-2006011156/lib/python3.7/site-packages/celery/apps/multi.py", line 145, in _annotate_with_default_opts
    self._setdefaultopt(options, ['--pidfile', '-p'], '/var/run/celery/%n.pid')
  File "/var/www/misc/ve-2006011156/lib/python3.7/site-packages/celery/apps/multi.py", line 159, in _setdefaultopt
    os.makedirs(dir_path)
  File "/var/www/misc/ve-2006011156/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/var/run/celery'

systemd[1]: demo@celeryd.service: Control process exited, code=exited, status=1/FAILURE
```

Request on_timeout should ignore soft time limit exception
When Request.on_timeout receive a soft timeout from billiard, it does the same as if it was receiving a hard time limit exception. This is ran by the controller.

But the task may catch this exception and eg. return (this is what soft timeout are for).

This cause:
1. the result to be saved once as an exception by the controller (on_timeout) and another time with the result returned by the task
2. the task status to be passed to failure and to success on the same manner
3. if the task is participating to a chord, the chord result counter (at least with redis) is incremented twice (instead of once), making the chord to return prematurely and eventually loose tasks…

1, 2 and 3 can leads of course to strange race conditions…

## Steps to reproduce (Illustration)

with the program in test_timeout.py:

```python
import time
import celery


app = celery.Celery('test_timeout')
app.conf.update(
    result_backend="redis://localhost/0",
    broker_url="amqp://celery:celery@localhost:5672/host",
)

@app.task(soft_time_limit=1)
def test():
    try:
        time.sleep(2)
    except Exception:
        return 1

@app.task()
def add(args):
    print("### adding", args)
    return sum(args)

@app.task()
def on_error(context, exception, traceback, **kwargs):
    print("### on_error: ", exception)

if __name__ == "__main__":
    result = celery.chord([test.s().set(link_error=on_error.s()), test.s().set(link_error=on_error.s())])(add.s())
    result.get()
```

start a worker and the program:

```
$ celery -A test_timeout worker -l WARNING
$ python3 test_timeout.py
```

## Expected behavior

add method is called with `[1, 1]` as argument and test_timeout.py return normally

## Actual behavior

The test_timeout.py fails, with
```
celery.backends.base.ChordError: Callback error: ChordError("Dependency 15109e05-da43-449f-9081-85d839ac0ef2 raised SoftTimeLimitExceeded('SoftTimeLimitExceeded(True,)',)",
```
On the worker side, the **on_error is called but the add method as well !**

```
[2017-11-29 23:07:25,538: WARNING/MainProcess] Soft time limit (1s) exceeded for test_timeout.test[15109e05-da43-449f-9081-85d839ac0ef2]
[2017-11-29 23:07:25,546: WARNING/MainProcess] ### on_error:
[2017-11-29 23:07:25,546: WARNING/MainProcess] SoftTimeLimitExceeded(True,)
[2017-11-29 23:07:25,547: WARNING/MainProcess] Soft time limit (1s) exceeded for test_timeout.test[38f3f7f2-4a89-4318-8ee9-36a987f73757]
[2017-11-29 23:07:25,553: ERROR/MainProcess] Chord callback for 'ef6d7a38-d1b4-40ad-b937-ffa84e40bb23' raised: ChordError("Dependency 15109e05-da43-449f-9081-85d839ac0ef2 raised SoftTimeLimitExceeded('SoftTimeLimitExceeded(True,)',)",)
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/celery/backends/redis.py", line 290, in on_chord_part_return
    callback.delay([unpack(tup, decode) for tup in resl])
  File "/usr/local/lib/python3.4/dist-packages/celery/backends/redis.py", line 290, in <listcomp>
    callback.delay([unpack(tup, decode) for tup in resl])
  File "/usr/local/lib/python3.4/dist-packages/celery/backends/redis.py", line 243, in _unpack_chord_result
    raise ChordError('Dependency {0} raised {1!r}'.format(tid, retval))
celery.exceptions.ChordError: Dependency 15109e05-da43-449f-9081-85d839ac0ef2 raised SoftTimeLimitExceeded('SoftTimeLimitExceeded(True,)',)
[2017-11-29 23:07:25,565: WARNING/MainProcess] ### on_error:
[2017-11-29 23:07:25,565: WARNING/MainProcess] SoftTimeLimitExceeded(True,)
[2017-11-29 23:07:27,262: WARNING/PoolWorker-2] ### adding
[2017-11-29 23:07:27,264: WARNING/PoolWorker-2] [1, 1]
```

Of course, on purpose did I choose to call the test.s() twice, to show that the count in the chord continues. In fact:
- the chord result is incremented twice by the error of soft time limit
- the chord result is again incremented twice by the correct returning of `test` task

## Conclusion

Request.on_timeout  should not process soft time limit exception. 

here is a quick monkey patch (correction of celery is trivial)

```python
def patch_celery_request_on_timeout():
    from celery.worker import request
    orig = request.Request.on_timeout
    def patched_on_timeout(self, soft, timeout):
        if not soft:
            orig(self, soft, timeout)
    request.Request.on_timeout = patched_on_timeout
patch_celery_request_on_timeout()
```



## version info

software -> celery:4.1.0 (latentcall) kombu:4.0.2 py:3.4.3
            billiard:3.5.0.2 py-amqp:2.1.4
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:redis://10.0.3.253/0



Discussion:
PermissionError: [Errno 13] Permission denied: '/var/run/celery' 

can you try sudo?
@mchataigner can you chime in?
> PermissionError: [Errno 13] Permission denied: '/var/run/celery'
> 
> can you try sudo?

Do you mean
`sudo celery multy start ...`  ?

I use systemd with RuntimeDirectory option

```config
[Unit]
Description = Demo celery workers
# When systemd stops or restarts the app.service, the action is propagated to this unit
PartOf = demo.target
# Start this unit after the demo.service start
After = demo.target

After = redis-server.service
Requires = redis-server.service

[Service]
Type = forking
User = www-data
Group = www-data
PermissionsStartOnly = true
RuntimeDirectory = demo
RuntimeDirectoryMode = 0775
EnvironmentFile = /var/www/misc/conf/celery.conf
ExecStart = /usr/local/sbin/demo-exec celery multi start ${CELERYD_NODES} \
  --app=${CELERY_APP} \
  --hostname=celeryd.worker \
  --pidfile=${CELERYD_PID_FILE} \
  --logfile=${CELERYD_LOG_FILE} \
  --loglevel=${CELERYD_LOG_LEVEL} \
  ${CELERYD_OPTS}
ExecStop = /usr/local/sbin/demo-exec celery multi stopwait ${CELERYD_NODES} \
  --hostname=celeryd.worker \
  --pidfile=${CELERYD_PID_FILE}
ExecReload = /usr/local/sbin/demo-exec celery multi restart ${CELERYD_NODES} \
  --app=${CELERY_APP} \
  --hostname=celeryd.worker \
  --pidfile=${CELERYD_PID_FILE} \
  --logfile=${CELERYD_LOG_FILE} \
  --loglevel=${CELERYD_LOG_LEVEL} \
  ${CELERYD_OPTS}
PrivateTmp = true

[Install]
WantedBy = multi-user.target
```

with celery v4.4.2 everything is working
The problem is in the method [`Node._setdefaultopt`](https://github.com/celery/celery/blob/master/celery/apps/multi.py#L148)

```python
    def _setdefaultopt(self, d, alt, value):
        for opt in alt[1:]:
            try:
                return d[opt]
            except KeyError:
                pass
        value = os.path.normpath(value)
        dir_path = os.path.dirname(value)
        if not os.path.exists(dir_path):
            os.makedirs(dir_path)
        return d.setdefault(alt[0], value)
```

Proof of Concept:
```python
import os

def _setdefaultopt(d, alt, value):
    for opt in alt[1:]:
        try:
            return d[opt]
        except KeyError:
            pass
    value = os.path.normpath(value)
    dir_path = os.path.dirname(value)
    if not os.path.exists(dir_path):
        print("make dir!!!: ", dir_path)
    return d.setdefault(alt[0], value)
```

Run in console:
```python
In [23] _setdefaultopt({'--pidfile': '/var/run/demo/celeryd-%n.pid'}, ['--pidfile', '-p'], '/var/run/celery/%n.pid')
make dir!!!:  /var/run/celery
Out[23]: '/var/run/demo/celeryd-%n.pid'
```
can you come with a fix? with a proper test?
> can you come with a fix? with a proper test?

Yes
@ask I think this is quite a big problem (with a trivial fix).

It requires attention though as it is bringging a new behaviour (but previous behaviour is not well documented, and, in my opinion, the new behaviour was the one expected)
This change in behaviour is what kept my team from upgrading to celery 4. Indeed, the chord callback was often not called at all.
I don't know if it is related, but I modified your code sample and it resulted in some `Exception raised outside body` errors and multiple other errors if you try running `python test_timeout.py` multiple times.

Here is my script:

```python
import time
import celery


app = celery.Celery(
    'test_timeout',
    broker='amqp://localhost',
    backend='redis://localhost')


@app.task(soft_time_limit=1)
def test(nb_seconds):
    try:
        time.sleep(nb_seconds)
        return nb_seconds
    except:
        print("### error handled")
        return nb_seconds


@app.task()
def add(args):
    print("### adding", args)
    return sum(args)


@app.task()
def on_error(context, exception, traceback, **kwargs):
    print("### on_error: ", exception)


if __name__ == "__main__":
    result = celery.chord([
        test.s(i).set(link_error=on_error.s()) for i in range(0, 2)
    ])(add.s())
    result.get()
```

NB: if you update the range for range(2, 4) for instance, the `Exception raised outside body` does not seem to happen. It seems this particular issue happens when the `SoftTimeLimitExceeded` is raised exactly during the `return`.
could you please send a PR with your proposed fix/workaround on master branch?
hi @auvipy, I won't have the time until jannuary. I'll need help on how to write the tests also (that's the reason why I didn't propose a PR).
OK. dont get afraid of sending logical changes just for the reason you don't know how to write test. we will certainly try to help you 
thanks a lot!

Resolving patch:
diff --git a/celery/apps/multi.py b/celery/apps/multi.py
--- a/celery/apps/multi.py
+++ b/celery/apps/multi.py
@@ -151,11 +151,11 @@ def _setdefaultopt(self, d, alt, value):
                 return d[opt]
             except KeyError:
                 pass
-        value = os.path.normpath(value)
+        value = d.setdefault(alt[0], os.path.normpath(value))
         dir_path = os.path.dirname(value)
         if not os.path.exists(dir_path):
             os.makedirs(dir_path)
-        return d.setdefault(alt[0], value)
+        return value
 
     def _prepare_expander(self):
         shortname, hostname = self.name.split('@', 1)


Test patch:
diff --git a/t/unit/apps/test_multi.py b/t/unit/apps/test_multi.py
--- a/t/unit/apps/test_multi.py
+++ b/t/unit/apps/test_multi.py
@@ -64,7 +64,7 @@ def test_parse(self, gethostname):
             '-c:jerry,elaine', '5',
             '--loglevel:kramer=DEBUG',
             '--flag',
-            '--logfile=foo', '-Q', 'bar', 'jerry',
+            '--logfile=/var/log/celery/foo', '-Q', 'bar', 'jerry',
             'elaine', 'kramer',
             '--', '.disable_rate_limits=1',
         ])
@@ -86,19 +86,19 @@ def assert_line_in(name, args):
         assert_line_in(
             '*P*jerry@*S*',
             ['COMMAND', '-n *P*jerry@*S*', '-Q bar',
-             '-c 5', '--flag', '--logfile=foo',
+             '-c 5', '--flag', '--logfile=/var/log/celery/foo',
              '-- .disable_rate_limits=1', '*AP*'],
         )
         assert_line_in(
             '*P*elaine@*S*',
             ['COMMAND', '-n *P*elaine@*S*', '-Q bar',
-             '-c 5', '--flag', '--logfile=foo',
+             '-c 5', '--flag', '--logfile=/var/log/celery/foo',
              '-- .disable_rate_limits=1', '*AP*'],
         )
         assert_line_in(
             '*P*kramer@*S*',
             ['COMMAND', '--loglevel=DEBUG', '-n *P*kramer@*S*',
-             '-Q bar', '--flag', '--logfile=foo',
+             '-Q bar', '--flag', '--logfile=/var/log/celery/foo',
              '-- .disable_rate_limits=1', '*AP*'],
         )
         expand = nodes[0].expander
@@ -278,6 +278,33 @@ def test_logfile(self):
         assert self.node.logfile == self.expander.return_value
         self.expander.assert_called_with(os.path.normpath('/var/log/celery/%n%I.log'))
 
+    @patch('celery.apps.multi.os.path.exists')
+    def test_pidfile_default(self, mock_exists):
+        n = Node.from_kwargs(
+            'foo@bar.com',
+        )
+        assert n.options['--pidfile'] == '/var/run/celery/%n.pid'
+        mock_exists.assert_any_call('/var/run/celery')
+
+    @patch('celery.apps.multi.os.makedirs')
+    @patch('celery.apps.multi.os.path.exists', return_value=False)
+    def test_pidfile_custom(self, mock_exists, mock_dirs):
+        n = Node.from_kwargs(
+            'foo@bar.com',
+            pidfile='/var/run/demo/celery/%n.pid'
+        )
+        assert n.options['--pidfile'] == '/var/run/demo/celery/%n.pid'
+
+        try:
+            mock_exists.assert_any_call('/var/run/celery')
+        except AssertionError:
+            pass
+        else:
+            raise AssertionError("Expected exists('/var/run/celery') to not have been called.")
+
+        mock_exists.assert_any_call('/var/run/demo/celery')
+        mock_dirs.assert_any_call('/var/run/demo/celery')
+
 
 class test_Cluster: