Repository: celery/celery

Issue:
Exception raised when appending to a chain
celery==3.1.20
django-redis==4.0.0
redis==2.10.5
websocket-client==0.35.0

Using the above versions of celery, redis and websocket. On production when I pass nearly 200 tasks to celery, the worker accepts only one task and then freezed and even not generated any error logs. It will not even process the current one. Although it works fine with 150 tasks.


I have also gone through some solutions and suggestions but not be able to fix the issue.

Am I using the correct versions of celery and redis in terms of compatibility?
Is there any fix for this issue? Please let me know If I need to put some other details.


Discussion:
Is there a reason that you prefer the 3.1.20 version? I believe the latest version 4 release has much better handling for Redis brokers.
Because I'm also using django-celery==3.1.16 which is not compatible with the new version of Celery, I mean with Celery 4.0. Is it the only way to fix it?
If you have Django 1.8 or newer you can use Celery 4.0 without the django-celery package http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html#using-celery-with-django
Well, I tried to implement it without django-celery and it gives me error "Tupple index out of range" on line which is folllowed by comment below. I'm using celery signature here and trying to append the result in "part_process". Here is the snapshot of the code.

    from celery import chain
    part_process = chain()
    
    for part_obj in queryset:
        data = {
            'id': part_obj.id, 
            'user_id': user_id, 
            'customer_id': customer_id, 
            'part_id': part_obj.id, 
            'username': username, 
            'part_file': part_obj.file_name, 
            'part_name': part_obj.part_name, 
            'machine_id': part_obj.dataset.id, 
            'ppi_type': part_obj.ppi_type, 
            'revision':part_obj.revision,
            'queue_id': pending_task.id,
            'ip_addr': ip_addr
        }

        part_process |= process_part_library.si(parts=data)  #in this line I get error

    lib_dataset_id = list(queryset.values_list('dataset_id', flat=True))
    Part.update_machine_flat_file(customer_id, lib_dataset_id)

Do I need to pass them in different way? As these things are also in official documentation of celery. I would appreciate any suggestion.
Could you post the versions of Celery you're using and the full traceback?
@r3m0t  I'm using Celery 4.0 right now. Following is the traceback - 

Traceback:

    File "/home/user/.virtualenvs/opti/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
      132.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)
    File "/home/user/.virtualenvs/opti/local/lib/python2.7/site-packages/django/views/decorators/csrf.py" in wrapped_view
      58.         return view_func(*args, **kwargs)
    File "/home/user/.virtualenvs/opti/local/lib/python2.7/site-packages/rest_framework/viewsets.py" in view
      87.             return self.dispatch(request, *args, **kwargs)
    File "/home/user/.virtualenvs/opti/local/lib/python2.7/site-packages/rest_framework/views.py" in dispatch
      466.             response = self.handle_exception(exc)
    File "/home/user/.virtualenvs/opti/local/lib/python2.7/site-packages/rest_framework/views.py" in dispatch
      463.             response = handler(request, *args, **kwargs)
    File "/home/user/worksapce/proj/app/views.py" in create
      1625. 										username=username)
    File "/home/user/worksapce/proj/app/views.py" in process_applib_queue_task
      1377. 					part_process |= process_app_library.si(parts=data)
    File "/home/user/.virtualenvs/opti/local/lib/python2.7/site-packages/celery/canvas.py" in __or__
      426.                 if isinstance(self.tasks[-1], group):

    Exception Type: IndexError at /api/v1/app
    Exception Value: tuple index out of range

Although, It was working fine upto Celery 3.1.25, If I tried with less than 200 tasks. Is there any other alternative for si function in newer version? or I am missing something?

Thanks for your help :)
Can you please provide a test case that we can use to reproduce this problem?
I have the same problem as @gauravt1 with v4.1.0, in v3.1.25 it was still working.
The following testcase shows the problem:
```
from celery import Celery, Task, chain

app = Celery()

@app.task
def add(x, y):
    return x + y

#works
c = chain(add.s(2,2) | add.s(3))
#7
print c.apply().get()

#fails, used to work in 3.1.25
c = chain()
c |= add.s(2,2)
c |= add.s(3)
print c.apply().get()
```
Resulting in:
```
7

Traceback (most recent call last):
  File "/home/jurrian/Development/python/celerytest.py", line 16, in <module>
    c |= add.s(2,2)
  File "/usr/local/lib/python2.7/dist-packages/celery/canvas.py", line 426, in __or__
    if isinstance(self.tasks[-1], group):
IndexError: tuple index out of range
```

Resolving patch:
diff --git a/celery/canvas.py b/celery/canvas.py
--- a/celery/canvas.py
+++ b/celery/canvas.py
@@ -419,13 +419,13 @@ def __or__(self, other):
             return sig
         elif isinstance(other, Signature):
             if isinstance(self, _chain):
-                if isinstance(self.tasks[-1], group):
+                if self.tasks and isinstance(self.tasks[-1], group):
                     # CHAIN [last item is group] | TASK -> chord
                     sig = self.clone()
                     sig.tasks[-1] = chord(
                         sig.tasks[-1], other, app=self._app)
                     return sig
-                elif isinstance(self.tasks[-1], chord):
+                elif self.tasks and isinstance(self.tasks[-1], chord):
                     # CHAIN [last item is chord] -> chain with chord body.
                     sig = self.clone()
                     sig.tasks[-1].body = sig.tasks[-1].body | other


Test patch:
diff --git a/t/unit/tasks/test_canvas.py b/t/unit/tasks/test_canvas.py
--- a/t/unit/tasks/test_canvas.py
+++ b/t/unit/tasks/test_canvas.py
@@ -478,6 +478,16 @@ def test_chord_sets_result_parent(self):
             seen.add(node.id)
             node = node.parent
 
+    def test_append_to_empty_chain(self):
+        x = chain()
+        x |= self.add.s(1, 1)
+        x |= self.add.s(1)
+        x.freeze()
+        tasks, _ = x._frozen
+        assert len(tasks) == 2
+
+        assert x.apply().get() == 3
+
 
 class test_group(CanvasCase):