Repository: celery/celery

Issue:
Chain loses priority
<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
bug reports which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->

- [x] I have read the relevant section in the
  [contribution guide](http://docs.celeryproject.org/en/latest/contributing.html#other-bugs)
  on reporting bugs.
- [x] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical bug reports.

## Mandatory Debugging Information

- [x] I have included the output of ``celery -A proj report`` in the issue.
    (if you are not able to do this, then at least specify the Celery
     version affected).
- [x] I have included all the versions of all the external dependencies required
  to reproduce this bug.

## Optional Debugging Information
<!--
Try some of the below if you think they are relevant.
It will help us figure out the scope of the bug and how many users it affects.
-->
- [x] I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

# Steps to Reproduce

## Required Dependencies
<!-- Please fill the required dependencies to reproduce this issue -->
* **Minimal Python Version**: 3.5.3
* **Minimal Celery Version**: 4.3.0
* **Minimal Kombu Version**: 4.6.1
* **Minimal Broker Version**: 3.6.6-1
* **Minimal OS and/or Kernel Version**: Debian 4.9.168-1+deb9u2

## Minimally Reproducible Test Case
<!--
Please provide a reproducible test case.
Refer to the Reporting Bugs section in our contribution guide.

We prefer submitting test cases in the form of a PR to our integration test suite.
If you can provide one, please mention the PR number below.
If not, please attach the most minimal code example required to reproduce the issue below.
If the test case is too large, please include a link to a gist or a repository below.
-->

<details>
<p>

```python
from kombu import Queue, Exchange
from celery import Celery
import time

broker = "amqp://admin:admin@localhost:5672/mqfrontend"


app = Celery(
    "test celery",
    broker=broker
)

app.conf.accept_content = ['json']
app.conf.task_serializer = 'json'
app.conf.result_serializer = 'json'

app.conf.task_ignore_result = True
app.conf.task_routes = {'frontend.*': {'queue': 'first_task', 'routing_key': 'frontend.first_task'}}
app.conf.task_queues = (
    Queue('first_task',    routing_key='frontend.first_task', queue_arguments={'x-max-priority': 10}),                                                                                                                                                        
)

@app.task(name="frontend.first_task", bind=True)
def priority_task(self, arg):
    time.sleep(2)
    print("PRIORITY: i:%s, p:%s"%(arg, self.request.delivery_info['priority']))
    return self.request.delivery_info['priority']

@app.task(name="frontend.second_task", bind=True)
def priorityb_task(self, _, arg):
    time.sleep(2)
    print("PRIORITYB: i:%s, p:%s"%(arg, self.request.delivery_info['priority']))
    return "Test%s"%self.request.delivery_info['priority']
    
if __name__=='__main__':
    import celery
    s = celery.chain(
        app.signature(
            "frontend.first_task",
            args=(5,),
            priority=5
        ),
        app.signature(
            "frontend.second_task",
            args=(5,),
            priority=5
        )
    )
    s.delay()
```

</p>
</details>

# Expected Behavior
<!-- Describe in detail what you expect to happen -->
We expect `first_task` and `second_task` to have a priority of 5 (it works with celery 4.2.1)

Expected output:
```
[tasks]
  . frontend.first_task
  . frontend.second_task

[2019-06-14 09:24:30,628: INFO/MainProcess] Connected to amqp://admin:**@127.0.0.1:5672/mqfrontend
[2019-06-14 09:24:30,640: INFO/MainProcess] mingle: searching for neighbors
[2019-06-14 09:24:31,673: INFO/MainProcess] mingle: sync with 1 nodes
[2019-06-14 09:24:31,674: INFO/MainProcess] mingle: sync complete
[2019-06-14 09:24:31,717: INFO/MainProcess] celery@ocean ready.
[2019-06-14 09:24:33,799: INFO/MainProcess] Received task: frontend.first_task[1c311ee3-436a-44a0-b849-fa938ec9be96]
[2019-06-14 09:24:35,802: WARNING/ForkPoolWorker-1] PRIORITY: i:5, p:5
[2019-06-14 09:24:35,818: INFO/ForkPoolWorker-1] Task frontend.first_task[1c311ee3-436a-44a0-b849-fa938ec9be96] succeeded in 2.01777482801117s: 5
[2019-06-14 09:24:35,819: INFO/MainProcess] Received task: frontend.second_task[fb1f1871-4b24-4f22-8ead-da74f24285d1]
[2019-06-14 09:24:37,822: WARNING/ForkPoolWorker-1] PRIORITYB: i:5, p:5
[2019-06-14 09:24:37,822: INFO/ForkPoolWorker-1] Task frontend.second_task[fb1f1871-4b24-4f22-8ead-da74f24285d1] succeeded in 2.0030388499144465s: 'Test0'
```


# Actual Behavior
<!--
Describe in detail what actually happened.
Please include a backtrace and surround it with triple backticks (```).
In addition, include the Celery daemon logs, the broker logs,
the result backend logs and system logs below if they will help us debug
the issue.
-->
`first_task` has the priority of 5, but `second_task` has no priority.

Actual output:
```
[tasks]
  . frontend.first_task
  . frontend.second_task

[2019-06-14 09:24:30,628: INFO/MainProcess] Connected to amqp://admin:**@127.0.0.1:5672/mqfrontend
[2019-06-14 09:24:30,640: INFO/MainProcess] mingle: searching for neighbors
[2019-06-14 09:24:31,673: INFO/MainProcess] mingle: sync with 1 nodes
[2019-06-14 09:24:31,674: INFO/MainProcess] mingle: sync complete
[2019-06-14 09:24:31,717: INFO/MainProcess] celery@ocean ready.
[2019-06-14 09:24:33,799: INFO/MainProcess] Received task: frontend.first_task[1c311ee3-436a-44a0-b849-fa938ec9be96]
[2019-06-14 09:24:35,802: WARNING/ForkPoolWorker-1] PRIORITY: i:5, p:5
[2019-06-14 09:24:35,818: INFO/ForkPoolWorker-1] Task frontend.first_task[1c311ee3-436a-44a0-b849-fa938ec9be96] succeeded in 2.01777482801117s: 5
[2019-06-14 09:24:35,819: INFO/MainProcess] Received task: frontend.second_task[fb1f1871-4b24-4f22-8ead-da74f24285d1]
[2019-06-14 09:24:37,822: WARNING/ForkPoolWorker-1] PRIORITYB: i:5, p:0
[2019-06-14 09:24:37,822: INFO/ForkPoolWorker-1] Task frontend.second_task[fb1f1871-4b24-4f22-8ead-da74f24285d1] succeeded in 2.0030388499144465s: 'Test0'
```


Discussion:
If this worked in older versions, could you try this in current master? and find the root cause of the regression?
This one looks interesting -- will take a look probably this week.
I not a lot of time at this moment, I can look at it at the end of the month
I think this is perhaps the PR where it broke, granted I haven't tested reverting it, but it did change chain apply to apply kwargs which it was ignoring before, so maybe that has something to do with it?   I found this by diffing celery 4.2.1 with 4.3.0 since you said it worked in 4.2.1 specifically looking for usages of chain.
https://github.com/celery/celery/pull/4952/files
https://github.com/celery/celery/pull/4952 auto reverting this failed
I would say that some piece of code "abused" the fact that kwargs were ignored, intentionally or not, and the change in #4952 revealed that bad behaviour.

In other words, I dont think the changes in #4952 are the root cause of this problem.
Not sure if this is related, but what happens if you define another Queue with `routing_key='frontend.second_task'`? I see that you've enabled priorities for the first_task but not for the other one.

> Queues can be configured to support priorities by setting the `x-max-priority` argument:

I would think that that's missing from your code.
I think I have found a similar issue using the current latest released version 4.3.0.

My problem is that I have 2 queues. The first one is a regular one and the second one is a priority queue. I have a task routing set up. Priority task uses a priority queue and a regular task uses a regular queue.

Imagine this setup:
```py

priority_task_signature = priority_task.signature(args=('foo',), options={'priority': 3}, immutable=True)
regular_task_signature = regular_task.signature(args=('bar',), immutable=True)
result = (regular_task_signature | priority_task_signature)()  # Creates and calls the chain.
```
Now, what happens is that a chain is created and sent to the RMQ. When the first regular task is done, it checks if there is a chained task. It pops the task from the list and creates a signature out of it. It calls the `apply_async()` and passes some parameters into the priority task signature ([source](https://github.com/celery/celery/blob/master/celery/app/trace.py#L443)). The parameter `task_priority` is set to `None`, because the parent task was sent into a regular queue without any priority. It goes next to [this line](https://github.com/celery/celery/blob/master/celery/canvas.py#L224) where it merges the signature options with the passed options.

The problem is that it overrides the options from the priority task signature I created explicitly.  It replaces the value of priority `3` with `None` and sends the task into the RMQ. 

I believe this is a bug. Can you confirm?



@olii why does a regular task pass `task_priority=None`? It should just _not pass it_, right?

It's also strange that args/kwargs are merged with options. I would expect that they are handled separately... now there are "reserved argument names"!
The parental task should not pass the priority if the chained task has set it explicitly. You are right. This is unexpected behavior.  See the source that task priority is initialized from the parental task ([source](https://github.com/celery/celery/blob/master/celery/app/trace.py#L368)). 

`Args` must be handled because in a chain you can pass the return value of the parental task into the chained task. Since the call uses the general method for task named `apply_async()` it handles also the `kwargs` but they are always set to `None`. No merge of `kwargs` is done.
did any of you try this with celery==4.4.0rc3?
Thank you for the suggestion with celery `4.4.0rc3`. I just did the test and the issue is still present (https://github.com/celery/celery/issues/5597#issuecomment-524833030).
I've noticed that `_chain.apply()` is using `**dict(self.options, **options)` instead of `**dict(task.options, **options)`, so it loses any options that are set on the tasks in the chain.  I'm not sure if it should also include `self.options`, but it seems like the omission of `task.options` could explain the loss of priority and other options (e.g. headers).
would you mind sending a PR & mention me to review?
Yep I look at the issue and it's due to
https://github.com/celery/celery/blob/b268171d5d7b0ebf634956c0559883f28296c21c/celery/app/trace.py#L446
which set priority to None and then 
https://github.com/celery/celery/blob/b268171d5d7b0ebf634956c0559883f28296c21c/celery/canvas.py#L240
which erase the `self.option['priority']` priority
Do other primitives suffer from the same issue perhaps? Ie. Group?

Resolving patch:
diff --git a/celery/canvas.py b/celery/canvas.py
--- a/celery/canvas.py
+++ b/celery/canvas.py
@@ -193,6 +193,8 @@ def apply(self, args=None, kwargs=None, **options):
         """
         args = args if args else ()
         kwargs = kwargs if kwargs else {}
+        # Extra options set to None are dismissed
+        options = {k: v for k, v in options.items() if v is not None}
         # For callbacks: extra args are prepended to the stored args.
         args, kwargs, options = self._merge(args, kwargs, options)
         return self.type.apply(args, kwargs, **options)
@@ -214,6 +216,8 @@ def apply_async(self, args=None, kwargs=None, route_name=None, **options):
         """
         args = args if args else ()
         kwargs = kwargs if kwargs else {}
+        # Extra options set to None are dismissed
+        options = {k: v for k, v in options.items() if v is not None}
         try:
             _apply = self._apply_async
         except IndexError:  # pragma: no cover
diff --git a/t/integration/tasks.py b/t/integration/tasks.py
--- a/t/integration/tasks.py
+++ b/t/integration/tasks.py
@@ -203,3 +203,9 @@ def fail(*args):
 @shared_task
 def chord_error(*args):
     return args
+
+
+@shared_task(bind=True)
+def return_priority(self, *_args):
+    return "Priority: %s" % self.request.delivery_info['priority']
+


Test patch:
diff --git a/t/integration/test_canvas.py b/t/integration/test_canvas.py
--- a/t/integration/test_canvas.py
+++ b/t/integration/test_canvas.py
@@ -13,11 +13,10 @@
                     add_to_all_to_chord, build_chain_inside_task, chord_error,
                     collect_ids, delayed_sum, delayed_sum_with_soft_guard,
                     fail, identity, ids, print_unicode, raise_error,
-                    redis_echo, second_order_replace1, tsum)
+                    redis_echo, second_order_replace1, tsum, return_priority)
 
 TIMEOUT = 120
 
-
 class test_chain:
 
     @flaky
@@ -854,3 +853,13 @@ def test_chain_to_a_chord_with_large_header(self, manager):
         c = identity.si(1) | group(identity.s() for _ in range(1000)) | tsum.s()
         res = c.delay()
         assert res.get(timeout=TIMEOUT) == 1000
+
+    @flaky
+    def test_priority(self, manager):
+        c = chain(return_priority.signature(priority=3))()
+        assert c.get(timeout=TIMEOUT) == "Priority: 3"
+
+    @flaky
+    def test_priority_chain(self, manager):
+        c = return_priority.signature(priority=3) | return_priority.signature(priority=5)
+        assert c().get(timeout=TIMEOUT) == "Priority: 5"
diff --git a/t/unit/tasks/test_canvas.py b/t/unit/tasks/test_canvas.py
--- a/t/unit/tasks/test_canvas.py
+++ b/t/unit/tasks/test_canvas.py
@@ -408,6 +408,24 @@ def s(*args, **kwargs):
         for task in c.tasks:
             assert task.options['link_error'] == [s('error')]
 
+    def test_apply_options_none(self):
+        class static(Signature):
+
+            def clone(self, *args, **kwargs):
+                return self
+
+            def _apply_async(self, *args, **kwargs):
+                self.args = args
+                self.kwargs = kwargs
+
+        c = static(self.add, (2, 2), type=self.add, app=self.app, priority=5)
+
+        c.apply_async(priority=4)
+        assert c.kwargs['priority'] == 4
+
+        c.apply_async(priority=None)
+        assert c.kwargs['priority'] == 5
+
     def test_reverse(self):
         x = self.add.s(2, 2) | self.add.s(2)
         assert isinstance(signature(x), _chain)