Repository: celery/celery

Issue:
Retrying tasks with an expiry fails with "'str' object has no attribute 'isoformat'"
## Summary
Celery fails to retry tasks with `expires` set. The problem is that celery passes a ISO8601-formatted string to `celery.app.amqp.as_task_v2()` instead of a datetime.

## Steps to reproduce
1. Create a fresh virtualenv and install celery from master.
2. Create tasks.py with the following content:
```
from celery import Celery

app = Celery('tasks', broker='pyamqp://')

@app.task(bind=True, expires=60.0, max_retries=1)
def randint(self):
    if self.request.retries == 0:
        raise self.retry(countdown=0.1)
    return 42
```
3. Trigger the job:
```
$ python -c 'import tasks; tasks.randint.delay()'
$ celery -A tasks worker --loglevel=info
```

## Expected behavior
The task is retried once, then returns 42. Ex. it should log something like the following:
```
... startup message omitted ...

[2017-01-03 12:04:40,705: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
[2017-01-03 12:04:40,715: INFO/MainProcess] mingle: searching for neighbors
[2017-01-03 12:04:41,734: INFO/MainProcess] mingle: all alone
[2017-01-03 12:04:41,746: INFO/MainProcess] celery@mrslim ready.
[2017-01-03 12:04:41,747: INFO/MainProcess] Received task: tasks.randint[2eaa60ce-28d8-435d-8aec-2f0fed8afa28]  
[2017-01-03 12:04:41,872: INFO/PoolWorker-4] Task tasks.randint[2eaa60ce-28d8-435d-8aec-2f0fed8afa28] retry: Retry in 0.1s
[2017-01-03 12:04:41,872: INFO/MainProcess] Received task: tasks.randint[2eaa60ce-28d8-435d-8aec-2f0fed8afa28]  ETA:[2017-01-03 20:04:41.953403+00:00] 
[2017-01-03 12:04:42,717: INFO/PoolWorker-2] Task tasks.randint[2eaa60ce-28d8-435d-8aec-2f0fed8afa28] succeeded in 0.000306855999952s: 42
```

## Actual behavior

The retry call fails and the job is never completed. It logs the following instead:

```
... startup message omitted ...

[2017-01-03 12:05:40,585: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
[2017-01-03 12:05:40,594: INFO/MainProcess] mingle: searching for neighbors
[2017-01-03 12:05:41,615: INFO/MainProcess] mingle: all alone
[2017-01-03 12:05:41,631: INFO/MainProcess] celery@mrslim ready.
[2017-01-03 12:05:41,632: INFO/MainProcess] Received task: tasks.randint[95e4d770-19f7-44d3-a052-738f293036c5]   expires:[2017-01-03 20:06:40.130848+00:00]
[2017-01-03 12:05:41,735: WARNING/PoolWorker-4] Task tasks.randint[95e4d770-19f7-44d3-a052-738f293036c5] reject requeue=False: 'str' object has no attribute 'isoformat'
Traceback (most recent call last):
  File "/home/bremac/src/celery_test/venv/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/bremac/src/celery_test/venv/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/bremac/src/celery_test/tasks.py", line 9, in randint
    raise self.retry(countdown=0.1)
  File "/home/bremac/src/celery_test/venv/lib/python2.7/site-packages/celery/app/task.py", line 687, in retry
    raise Reject(exc, requeue=False)
Reject: (AttributeError("'str' object has no attribute 'isoformat'",), False)
```

Pdb gives me the following backtrace for the original exception. I've omitted everything above the call to `self.retry` for brevity:

```
  /home/bremac/src/celery_test/venv/lib/python2.7/site-packages/celery/app/task.py(685)retry()
-> S.apply_async()
  /home/bremac/src/celery_test/venv/lib/python2.7/site-packages/celery/canvas.py(221)apply_async()
-> return _apply(args, kwargs, **options)
  /home/bremac/src/celery_test/venv/lib/python2.7/site-packages/celery/app/task.py(535)apply_async()
-> **options
  /home/bremac/src/celery_test/venv/lib/python2.7/site-packages/celery/app/base.py(729)send_task()
-> root_id, parent_id, shadow, chain,
  /home/bremac/src/celery_test/venv/lib/python2.7/site-packages/celery/app/amqp.py(334)as_task_v2()
-> expires = expires and expires.isoformat()
```

## Other information

```
$ celery -A tasks report  # installed from a checkout on master

software -> celery:4.0.2 (latentcall) kombu:4.0.2 py:2.7.13
            billiard:3.5.0.2 py-amqp:2.1.4
platform -> system:Linux arch:64bit imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:pyamqp results:disabled

broker_url: u'amqp://guest:********@localhost:5672//'
```


Discussion:

Resolving patch:
diff --git a/celery/app/amqp.py b/celery/app/amqp.py
--- a/celery/app/amqp.py
+++ b/celery/app/amqp.py
@@ -331,7 +331,9 @@ def as_task_v2(self, task_id, name, args=None, kwargs=None,
                 now + timedelta(seconds=expires), tz=timezone,
             )
         eta = eta and eta.isoformat()
-        expires = expires and expires.isoformat()
+        # If we retry a task `expires` will already be ISO8601-formatted.
+        if not isinstance(expires, string_t):
+            expires = expires and expires.isoformat()
 
         if argsrepr is None:
             argsrepr = saferepr(args, self.argsrepr_maxsize)
diff --git a/t/integration/tasks.py b/t/integration/tasks.py
--- a/t/integration/tasks.py
+++ b/t/integration/tasks.py
@@ -55,3 +55,11 @@ def collect_ids(self, res, i):
 
     """
     return res, (self.request.root_id, self.request.parent_id, i)
+
+
+@shared_task(bind=True, expires=60.0, max_retries=1)
+def retry_once(self):
+    """Task that fails and is retried. Returns the number of retries."""
+    if self.request.retries:
+        return self.request.retries
+    raise self.retry(countdown=0.1)


Test patch:
diff --git a/t/integration/test_tasks.py b/t/integration/test_tasks.py
--- a/t/integration/test_tasks.py
+++ b/t/integration/test_tasks.py
@@ -1,7 +1,7 @@
 from __future__ import absolute_import, unicode_literals
 from celery import group
 from .conftest import flaky
-from .tasks import print_unicode, sleeping
+from .tasks import print_unicode, retry_once, sleeping
 
 
 class test_tasks:
@@ -12,6 +12,11 @@ def test_task_accepted(self, manager, sleep=1):
         sleeping.delay(sleep)
         manager.assert_accepted([r1.id])
 
+    @flaky
+    def test_task_retried(self):
+        res = retry_once.delay()
+        assert res.get(timeout=10) == 1  # retried once
+
     @flaky
     def test_unicode_task(self, manager):
         manager.join(