Repository: celery/celery

Issue:
Complex canvas might raise AttributeError: 'dict' object has no attribute 'clone'
WARNING:  I'm still trying to collect all the necessary details and, if possible, provide a reproducible example.  For the time, the error is happening constantly in one server, but doesn't happen in another one with the same source code.

I'm getting the following:

```
Traceback (most recent call last):
  File "eggs/celery-4.2.1-py2.7.egg/celery/app/trace.py", line 439, in trace_task
    parent_id=uuid, root_id=root_id,
  File "eggs/celery-4.2.1-py2.7.egg/celery/canvas.py", line 1232, in apply_async
    return self.run(tasks, body, args, task_id=task_id, **options)
  File "eggs/celery-4.2.1-py2.7.egg/celery/canvas.py", line 1277, in run
    header_result = header(*partial_args, task_id=group_id, **options)
  File "eggs/celery-4.2.1-py2.7.egg/celery/canvas.py", line 953, in __call__
    return self.apply_async(partial_args, **options)
  File "eggs/celery-4.2.1-py2.7.egg/celery/canvas.py", line 978, in apply_async
    args=args, kwargs=kwargs, **options))
  File "eggs/celery-4.2.1-py2.7.egg/celery/canvas.py", line 1054, in _apply_tasks
    **options)
  File "eggs/celery-4.2.1-py2.7.egg/celery/canvas.py", line 557, in apply_async
    dict(self.options, **options) if options else self.options))
  File "eggs/celery-4.2.1-py2.7.egg/celery/canvas.py", line 573, in run
    task_id, group_id, chord,
  File "eggs/celery-4.2.1-py2.7.egg/celery/canvas.py", line 683, in prepare_steps
    task.link_error(errback)
  File "eggs/celery-4.2.1-py2.7.egg/celery/canvas.py", line 1016, in link_error
    sig = sig.clone().set(immutable=True)
AttributeError: 'dict' object has no attribute 'clone'
```

When trying to execute a complex canvas.  The canvas itself is dynamically created.  We have a single task that dispatches the actual calls.  So all, jobs are instances of this task with different arguments.  The task is: https://github.com/merchise-autrement/odoo/blob/merchise-10.0/odoo/jobs.py#L746

The canvas is constructed in: https://github.com/merchise/xopgi.base/blob/master/xopgi/xopgi_cdr/cdr_agent.py#L181

Basically, we have a layered graphs of events that depend on several evidences, which in turn depend on variables.  We compute each variable in a job.  Each variable job is linked to a group of jobs to update the dependent evidences.  Each evidence job is liked to a group of event jobs.  Each job has an on_error callback to signal the whole cycle had some errors.

I'm trying to detect if the issue comes from a particular workflow of jobs, or other cause.

I'm using celery 4.2.1, but I tested with master.


Discussion:
This went away after updating the redis server.  The server showing the error was using redis 3.  After upgrading to redis 4 the issue went away.
I could reproduce the error in my box using redis 3.

@thedrow  Should we reopen this, or let just this issue as a historical note.  I don't really know how many people with be affected.  Redis 5 was release some weeks ago.  Maybe most people are using Redis 4.
Opps! I have just been hit by the same error with Redis 5.  So, I'm guessing I was just lucky when I upgraded the server in the failing box.  I'm reopening, till I know more.

Update: The server (with redis 4) is now again raising the AttributeError.
The only workaround I have found to make this issue gone in my case, is to change the `link_error` method from:

```python
    def link_error(self, sig):
        sig = sig.clone().set(immutable=True)
        return self.tasks[0].link_error(sig)
```

to:

```python
    def link_error(self, sig):
        if not isinstance(sig, Signature):
            sig = Signature.from_dict(sig)
        sig = sig.clone().set(immutable=True)
        return self.tasks[0].link_error(sig)
```

No explanation or reproducible scenario yet.  But now that I'm "out of the woods" with my server running, I can dedicate some time to try and find the root cause of this issue.
The fix itself makes sense.
Though a more preferable solution would be:
```python
def link_error(self, sig):
        if not isinstance(sig, Signature):
            sig['immutable'] = True
            sig = Signature.from_dict(sig)
        else:
            sig = sig.clone().set(immutable=True)
        return self.tasks[0].link_error(sig)
```

I'd still would like to see a test case so that we can include it in our test suite.
I noticed I had `task_ignore_result` set to True, but changing to False didn't help.

@thedrow I'm trying to create the failing scenario.  But it could take some time.  This is the first time I'm working with canvases; and my case is rather complicated, so producing a simple test case may take some time.

I'm trying to use mypy to set some annotations; but I see that some annotations are already there, but invalid:

```
$ mypy -2 -p celery --ignore-missing-imports
celery/backends/mongodb.py:31: error: Name 'InvalidDocument' already defined (possibly by an import)
celery/beat.py:658: error: Name '_Process' already defined on line 656
celery/contrib/pytest.py:18: error: Type signature has too few arguments
celery/contrib/pytest.py:46: error: Type signature has too few arguments
celery/contrib/pytest.py:66: error: Type signature has too few arguments
celery/contrib/pytest.py:163: error: Type signature has too few arguments
```

What's your stand on making mypy part of the CI pipeline?
It can but we'll need to invest a lot of work in making it pass.
Let's create a different issue about it.

Resolving patch:
diff --git a/celery/canvas.py b/celery/canvas.py
--- a/celery/canvas.py
+++ b/celery/canvas.py
@@ -1045,7 +1045,13 @@ def link(self, sig):
         return self.tasks[0].link(sig)
 
     def link_error(self, sig):
-        sig = sig.clone().set(immutable=True)
+        try:
+            sig = sig.clone().set(immutable=True)
+        except AttributeError:
+            # See issue #5265.  I don't use isinstance because current tests
+            # pass a Mock object as argument.
+            sig['immutable'] = True
+            sig = Signature.from_dict(sig)
         return self.tasks[0].link_error(sig)
 
     def _prepared(self, tasks, partial_args, group_id, root_id, app,


Test patch: