Repository: celery/celery

Issue:
chord_unlock isn't using the same queue as chord
I've created a tasks_queue and priority_task_queue. in addition to that i've set the tasks_queue to be used as default queue. Now every time i would like a task to be perfrom right away i use priority_queue which is generally empty most of the time. now, the problem araise whenever i'm using chord. i noticed that chords are unable to complete their task because their chord_unlock counterpart always run on the default tasks queue(tasks_queue)

## Steps to reproduce

- Ubnutu 16 server
- celery==4.1.0
- kombu==4.1.0
- boto==2.48.0
- boto3==1.4.7
- botocore==1.7.31
- Using SQS for as broker.



## Expected behavior
I expected the chord_unlock to be performed in the same queue the chord was executed. This causes a major problem for me. Since SQS does not support priority tasks. I'm forced to use two queues to enable the effect of tasks priority. but since chord_unlock will always use the same default queue. priority_tasks using chords, will never be able to finish whenever the default queue is packed.


## Actual behavior
the chord was consumed from priority_task_queue and the chord_unlock got stuck in line on the regular tasks_queue




Discussion:
So after doing some tests I've realized that the problem resides within my code. I've forgotten to set the queue for one of my chords. and it was not running in priority.

My bad. Celery is awesome. Peace!
Okay, so apparently, it wasn't my coding error. a chord_unlock will always be running in the default queue instead of the queue its counterpart chord will run. If you are using celery with SQS as a broker, this is somewhat problematic since priority is a non-existent feature and you are forced to use different queue for priority tasks. but if your tasks canvas is using chord, its unlock could be stuck in a pile of tasks in the default queue.

the way I eventually overcome this problem is by using custom router to route chord_unlock to the queue where its group(chord) was sent.

using the following simple code - 
```
def route_chord_unlock_to_chord_queue(name, args, kwargs, options, task=None, **kw):
    if name == 'celery.chord_unlock':
        callback_sig = args[1]
        callback_sig.get("options").get('queue')
        return {'queue': callback_sig.get("options").get('queue')}
```



it worked. but i still think the default situation is not what the author meant. I might be wrong. but I leave this up to you guys

Resolving patch:
diff --git a/celery/backends/base.py b/celery/backends/base.py
--- a/celery/backends/base.py
+++ b/celery/backends/base.py
@@ -415,8 +415,11 @@ def on_chord_part_return(self, request, state, result, **kwargs):
     def fallback_chord_unlock(self, header_result, body, countdown=1,
                               **kwargs):
         kwargs['result'] = [r.as_tuple() for r in header_result]
+        queue = body.options.get('queue', getattr(body.type, 'queue', None))
         self.app.tasks['celery.chord_unlock'].apply_async(
-            (header_result.id, body,), kwargs, countdown=countdown,
+            (header_result.id, body,), kwargs,
+            countdown=countdown,
+            queue=queue,
         )
 
     def ensure_chords_allowed(self):


Test patch:
diff --git a/t/unit/backends/test_base.py b/t/unit/backends/test_base.py
--- a/t/unit/backends/test_base.py
+++ b/t/unit/backends/test_base.py
@@ -63,6 +63,12 @@ class test_BaseBackend_interface:
     def setup(self):
         self.b = BaseBackend(self.app)
 
+        @self.app.task(shared=False)
+        def callback(result):
+            pass
+
+        self.callback = callback
+
     def test__forget(self):
         with pytest.raises(NotImplementedError):
             self.b._forget('SOMExx-N0Nex1stant-IDxx-')
@@ -80,9 +86,33 @@ def test_apply_chord(self, unlock='celery.chord_unlock'):
             uuid(),
             [self.app.AsyncResult(x) for x in range(3)],
         )
-        self.b.apply_chord(header_result, None)
+        self.b.apply_chord(header_result, self.callback.s())
         assert self.app.tasks[unlock].apply_async.call_count
 
+    def test_chord_unlock_queue(self, unlock='celery.chord_unlock'):
+        self.app.tasks[unlock] = Mock()
+        header_result = self.app.GroupResult(
+            uuid(),
+            [self.app.AsyncResult(x) for x in range(3)],
+        )
+        body = self.callback.s()
+
+        self.b.apply_chord(header_result, body)
+        called_kwargs = self.app.tasks[unlock].apply_async.call_args[1]
+        assert called_kwargs['queue'] is None
+
+        self.b.apply_chord(header_result, body.set(queue='test_queue'))
+        called_kwargs = self.app.tasks[unlock].apply_async.call_args[1]
+        assert called_kwargs['queue'] == 'test_queue'
+
+        @self.app.task(shared=False, queue='test_queue_two')
+        def callback_queue(result):
+            pass
+
+        self.b.apply_chord(header_result, callback_queue.s())
+        called_kwargs = self.app.tasks[unlock].apply_async.call_args[1]
+        assert called_kwargs['queue'] == 'test_queue_two'
+
 
 class test_exception_pickle: