Repository: celery/celery

Issue:
Chaining chords causes TypeError: 'NoneType' object is not iterable, 
```chord_unlock``` task fails after retry with ```TypeError: 'NoneType' object is not iterable``` Exception.

Am I doing something wrong?

## Checklist

- [x] I have included the output of ``celery -A proj report`` in the issue.
      (if you are not able to do this, then at least specify the Celery
       version affected).
- [x] I have verified that the issue exists against the `master` branch of Celery.

```
celery -A tasks report

software -> celery:4.1.0 (latentcall) kombu:4.1.0 py:2.7.12+
            billiard:3.5.0.3 py-amqp:2.2.1
platform -> system:Linux arch:64bit imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:pyamqp results:db+mysql://celery:**@localhost/celery

broker_url: u'amqp://celery:********@rbn-box:5672//'
result_backend: u'db+mysql://celery:********@localhost/celery'
```

## Steps to reproduce
Tasks file
```python
from celery import Celery


app = Celery('tasks', broker='pyamqp://celery:celery@rbn-box//', backend='db+mysql://celery:celery@localhost/celery')


@app.task
def dummy(data):
    print(data)
    import time
    time.sleep(7)
```

Problematic chain
```python
from celery import chain, chord
from tasks import dummy

c1 = chord(
    header=chain([
        dummy.subtask(args=('c1-{}'.format(i),), immutable=True)
        for i in range(0, 3)]),
    body=dummy.subtask(args=('c1 body',), immutable=True))
c2 = chord(
    header=chain([
        dummy.subtask(args=('c2-{}'.format(i),), immutable=True)
        for i in range(0, 3)]),
    body=dummy.subtask(args=('c2 body',), immutable=True))

sig = c1 | c2
sig.apply_async()
```


## Expected behavior
It's possible to chain chords.

## Actual behavior
```
celery worker -A tasks --loglevel INFO -c 10
 
 -------------- celery@rbn-box v4.1.0 (latentcall)
---- **** ----- 
--- * ***  * -- Linux-4.8.0-51-generic-x86_64-with-Ubuntu-16.10-yakkety 2017-08-22 21:47:22
-- * - **** --- 
- ** ---------- [config]
- ** ---------- .> app:         tasks:0x7f2bd97f73d0
- ** ---------- .> transport:   amqp://celery:**@rbn-box:5672//
- ** ---------- .> results:     mysql://celery:**@localhost/celery
- *** --- * --- .> concurrency: 10 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery
                

[tasks]
  . tasks.dummy

[2017-08-22 21:47:23,343: INFO/MainProcess] Connected to amqp://celery:**@rbn-box:5672//
[2017-08-22 21:47:23,352: INFO/MainProcess] mingle: searching for neighbors
[2017-08-22 21:47:24,376: INFO/MainProcess] mingle: all alone
[2017-08-22 21:47:24,407: INFO/MainProcess] celery@rbn-box ready.
[2017-08-22 21:47:36,462: INFO/MainProcess] Received task: tasks.dummy[831ff49c-cd08-4aa8-8ca5-2a3a553a5567]  
[2017-08-22 21:47:36,464: INFO/MainProcess] Received task: tasks.dummy[0e1cf302-aa23-4a10-835f-76496940bd0f]  
[2017-08-22 21:47:36,465: INFO/MainProcess] Received task: tasks.dummy[d02014c5-0b34-4a9c-a0d6-f8d1dc0e4a66]  
[2017-08-22 21:47:36,468: INFO/MainProcess] Received task: celery.chord_unlock[5658c4ce-07dc-4208-acf2-ae00d64247b8]  ETA:[2017-08-22 19:47:37.458625+00:00] 
[2017-08-22 21:47:36,473: WARNING/ForkPoolWorker-10] c1-1
[2017-08-22 21:47:36,479: WARNING/ForkPoolWorker-9] c1-2
[2017-08-22 21:47:36,484: WARNING/ForkPoolWorker-8] c1-0
[2017-08-22 21:47:38,617: INFO/MainProcess] Received task: celery.chord_unlock[5658c4ce-07dc-4208-acf2-ae00d64247b8]  ETA:[2017-08-22 19:47:39.597538+00:00] 
[2017-08-22 21:47:38,623: INFO/ForkPoolWorker-6] Task celery.chord_unlock[5658c4ce-07dc-4208-acf2-ae00d64247b8] retry: Retry in 1s
[2017-08-22 21:47:40,504: ERROR/ForkPoolWorker-1] Task celery.chord_unlock[5658c4ce-07dc-4208-acf2-ae00d64247b8] raised unexpected: TypeError("'NoneType' object is not iterable",)
Traceback (most recent call last):
  File "/home/rbn/.virtualenvs/celery-4.1/local/lib/python2.7/site-packages/celery-4.1.0-py2.7.egg/celery/app/trace.py", line 374, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/rbn/.virtualenvs/celery-4.1/local/lib/python2.7/site-packages/celery-4.1.0-py2.7.egg/celery/app/trace.py", line 629, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/rbn/.virtualenvs/celery-4.1/local/lib/python2.7/site-packages/celery-4.1.0-py2.7.egg/celery/app/builtins.py", line 59, in unlock_chord
    callback = maybe_signature(callback, app)
  File "/home/rbn/.virtualenvs/celery-4.1/local/lib/python2.7/site-packages/celery-4.1.0-py2.7.egg/celery/canvas.py", line 1390, in maybe_signature
    d = signature(d)
  File "/home/rbn/.virtualenvs/celery-4.1/local/lib/python2.7/site-packages/celery-4.1.0-py2.7.egg/celery/canvas.py", line 1365, in signature
    return Signature.from_dict(varies, app=app)
  File "/home/rbn/.virtualenvs/celery-4.1/local/lib/python2.7/site-packages/celery-4.1.0-py2.7.egg/celery/canvas.py", line 147, in from_dict
    return target_cls.from_dict(d, app=app)
  File "/home/rbn/.virtualenvs/celery-4.1/local/lib/python2.7/site-packages/celery-4.1.0-py2.7.egg/celery/canvas.py", line 534, in from_dict
    tasks = [maybe_signature(task, app=app) for task in tasks]
  File "/home/rbn/.virtualenvs/celery-4.1/local/lib/python2.7/site-packages/celery-4.1.0-py2.7.egg/celery/canvas.py", line 1390, in maybe_signature
    d = signature(d)
  File "/home/rbn/.virtualenvs/celery-4.1/local/lib/python2.7/site-packages/celery-4.1.0-py2.7.egg/celery/canvas.py", line 1365, in signature
    return Signature.from_dict(varies, app=app)
  File "/home/rbn/.virtualenvs/celery-4.1/local/lib/python2.7/site-packages/celery-4.1.0-py2.7.egg/celery/canvas.py", line 147, in from_dict
    return target_cls.from_dict(d, app=app)
  File "/home/rbn/.virtualenvs/celery-4.1/local/lib/python2.7/site-packages/celery-4.1.0-py2.7.egg/celery/canvas.py", line 1178, in from_dict
    return _upgrade(d, cls(*args, app=app, **d))
  File "/home/rbn/.virtualenvs/celery-4.1/local/lib/python2.7/site-packages/celery-4.1.0-py2.7.egg/celery/canvas.py", line 1190, in __init__
    dict(kwargs=kwargs, header=_maybe_group(header, app),
  File "/home/rbn/.virtualenvs/celery-4.1/local/lib/python2.7/site-packages/celery-4.1.0-py2.7.egg/celery/canvas.py", line 904, in _maybe_group
    tasks = [signature(t, app=app) for t in tasks]
TypeError: 'NoneType' object is not iterable
[2017-08-22 21:47:43,647: INFO/ForkPoolWorker-9] Task tasks.dummy[d02014c5-0b34-4a9c-a0d6-f8d1dc0e4a66] succeeded in 7.16881482498s: None
[2017-08-22 21:47:43,682: INFO/ForkPoolWorker-10] Task tasks.dummy[0e1cf302-aa23-4a10-835f-76496940bd0f] succeeded in 7.20971017997s: None
[2017-08-22 21:47:43,695: INFO/ForkPoolWorker-8] Task tasks.dummy[831ff49c-cd08-4aa8-8ca5-2a3a553a5567] succeeded in 7.21184302299s: None
```



Discussion:
I've made some investigation and found that it incorrectly reconstructs `celery.chord` task in the `body`.

Initial record

          {
            "task": "celery.chord",
            "args": [],
            "kwargs": {
              "kwargs": {
                "kwargs": {}
              },
              "header": {
                "task": "celery.group",
                "args": [],
                "kwargs": {
                  "tasks": [
                    {
                      "task": "tasks.dummy",
                      "args": [
                        "c2-0"
                      ],
                      "kwargs": {},
                      "options": {
                        "task_id": "37e8d72c-d3b0-47d7-819d-6418309211ca",
                        "reply_to": "0498cd30-a435-3c83-99ef-ff2cf7c1d6ae",
                        "chord": {
                          "task": "tasks.dummy",
                          "args": [
                            "c2 body"
                          ],
                          "kwargs": {},
                          "options": {
                            "task_id": "9d4861c6-0a91-47fe-a1ba-5b5ecaa31f32",
                            "reply_to": "0498cd30-a435-3c83-99ef-ff2cf7c1d6ae"
                          },
                          "subtask_type": null,
                          "chord_size": null,
                          "immutable": true
                        }
                      },
                      "subtask_type": null,
                      "chord_size": null,
                      "immutable": true
                    },
                    {
                      "task": "tasks.dummy",
                      "args": [
                        "c2-1"
                      ],
                      "kwargs": {},
                      "options": {
                        "task_id": "df022257-cbb1-4ad7-ac67-9a2c64972ca2",
                        "reply_to": "0498cd30-a435-3c83-99ef-ff2cf7c1d6ae",
                        "chord": {
                          "task": "tasks.dummy",
                          "args": [
                            "c2 body"
                          ],
                          "kwargs": {},
                          "options": {
                            "task_id": "9d4861c6-0a91-47fe-a1ba-5b5ecaa31f32",
                            "reply_to": "0498cd30-a435-3c83-99ef-ff2cf7c1d6ae"
                          },
                          "subtask_type": null,
                          "chord_size": null,
                          "immutable": true
                        }
                      },
                      "subtask_type": null,
                      "chord_size": null,
                      "immutable": true
                    },
                    {
                      "task": "tasks.dummy",
                      "args": [
                        "c2-2"
                      ],
                      "kwargs": {},
                      "options": {
                        "task_id": "dcce66b7-b932-4f69-bfd4-0f464606031e",
                        "reply_to": "0498cd30-a435-3c83-99ef-ff2cf7c1d6ae",
                        "chord": {
                          "task": "tasks.dummy",
                          "args": [
                            "c2 body"
                          ],
                          "kwargs": {},
                          "options": {
                            "task_id": "9d4861c6-0a91-47fe-a1ba-5b5ecaa31f32",
                            "reply_to": "0498cd30-a435-3c83-99ef-ff2cf7c1d6ae"
                          },
                          "subtask_type": null,
                          "chord_size": null,
                          "immutable": true
                        }
                      },
                      "subtask_type": null,
                      "chord_size": null,
                      "immutable": true
                    }
                  ]
                },
                "options": {
                  "task_id": "4e338e07-4754-45dd-a7ac-439ff955d7f8",
                  "chord": {
                    "task": "tasks.dummy",
                    "args": [
                      "c2 body"
                    ],
                    "kwargs": {},
                    "options": {
                      "task_id": "9d4861c6-0a91-47fe-a1ba-5b5ecaa31f32",
                      "reply_to": "0498cd30-a435-3c83-99ef-ff2cf7c1d6ae"
                    },
                    "subtask_type": null,
                    "chord_size": null,
                    "immutable": true
                  },
                  "root_id": null,
                  "parent_id": null
                },
                "subtask_type": "group",
                "immutable": false,
                "chord_size": null
              },
              "body": {
                "task": "tasks.dummy",
                "args": [
                  "c2 body"
                ],
                "kwargs": {},
                "options": {
                  "task_id": "9d4861c6-0a91-47fe-a1ba-5b5ecaa31f32",
                  "reply_to": "0498cd30-a435-3c83-99ef-ff2cf7c1d6ae"
                },
                "subtask_type": null,
                "chord_size": null,
                "immutable": true
              }
            },
            "options": {
              "chord_size": null,
              "task_id": "4e338e07-4754-45dd-a7ac-439ff955d7f8"
            },
            "subtask_type": "chord",
            "immutable": false,
            "chord_size": null
          }

Retry record

          {
            "task": "celery.chord",
            "args": [],
            "kwargs": {
              "kwargs": {
                "kwargs": {}
              }
            },
            "options": {
              "chord_size": null,
              "task_id": "4e338e07-4754-45dd-a7ac-439ff955d7f8"
            },
            "subtask_type": "chord",
            "immutable": false,
            "chord_size": null
          }
I've found it.

https://github.com/celery/celery/blob/06c6cfefb5948286e5c634cfc5b575dafe9dc98d/celery/canvas.py#L1175-L1178

Here it modifies task's request context (removes `header` and `body` from `d['kwargs']`) which is used by `task.retry` to build new signature for retrying.

Resolving patch:
diff --git a/celery/canvas.py b/celery/canvas.py
--- a/celery/canvas.py
+++ b/celery/canvas.py
@@ -1174,8 +1174,9 @@ class chord(Signature):
 
     @classmethod
     def from_dict(cls, d, app=None):
-        args, d['kwargs'] = cls._unpack_args(**d['kwargs'])
-        return _upgrade(d, cls(*args, app=app, **d))
+        options = d.copy()
+        args, options['kwargs'] = cls._unpack_args(**options['kwargs'])
+        return _upgrade(d, cls(*args, app=app, **options))
 
     @staticmethod
     def _unpack_args(header=None, body=None, **kwargs):


Test patch: