Repository: celery/celery

Issue:
Retry args change BUG
<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
bug reports which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->
- [x] I have verified that the issue exists against the `master` branch of Celery.
- [x] This has already been asked to the [discussion group](https://groups.google.com/forum/#!forum/celery-users) first.
- [x] I have read the relevant section in the
  [contribution guide](http://docs.celeryproject.org/en/latest/contributing.html#other-bugs)
  on reporting bugs.
- [ ] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical bug reports.
- [ ] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22)
  for existing proposed fixes.
- [ ] I have checked the [commit log](https://github.com/celery/celery/commits/master)
  to find out if the bug was already fixed in the master branch.
- [ ] I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

## Mandatory Debugging Information

- [ ] I have included the output of ``celery -A proj report`` in the issue.
    (if you are not able to do this, then at least specify the Celery
     version affected).
- [x] I have verified that the issue exists against the `master` branch of Celery.
- [ ] I have included the contents of ``pip freeze`` in the issue.
- [ ] I have included all the versions of all the external dependencies required
  to reproduce this bug.

## Optional Debugging Information
<!--
Try some of the below if you think they are relevant.
It will help us figure out the scope of the bug and how many users it affects.
-->
- [ ] I have tried reproducing the issue on more than one Python version
  and/or implementation.
- [ ] I have tried reproducing the issue on more than one message broker and/or
  result backend.
- [ ] I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
- [ ] I have tried reproducing the issue on more than one operating system.
- [ ] I have tried reproducing the issue on more than one workers pool.
- [ ] I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
- [ ] I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

## Related Issues and Possible Duplicates
<!--
Please make sure to search and mention any related issues
or possible duplicates to this issue as requested by the checklist above.

This may or may not include issues in other repositories that the Celery project
maintains or other repositories that are dependencies of Celery.

If you don't know how to mention issues, please refer to Github's documentation
on the subject: https://help.github.com/en/articles/autolinked-references-and-urls#issues-and-pull-requests
-->

#### Related Issues

- None

#### Possible Duplicates

- None

## Environment & Settings
<!-- Include the contents of celery --version below -->
**Celery version**: celery==4.4.2
<!-- Include the output of celery -A proj report below -->
<details>
<summary><b><code>celery report</code> Output:</b></summary>
<p>

```
[2020-05-31 23:28:34,434: INFO/MainProcess] Connected to amqp://remote_worker:**@127.0.0.1:5672//
[2020-05-31 23:28:34,453: INFO/MainProcess] mingle: searching for neighbors
[2020-05-31 23:28:35,487: INFO/MainProcess] mingle: all alone
[2020-05-31 23:28:35,528: WARNING/MainProcess] /home/ubuntu/.local/lib/python3.7/site-packages/celery/fixups/django.py:203: UserWarning: Using settings.DEBUG leads to a memory
            leak, never use this setting in production environments!
  leak, never use this setting in production environments!''')
[2020-05-31 23:28:35,529: INFO/MainProcess] celery@testroom ready.
[2020-05-31 23:28:47,351: INFO/MainProcess] Received task: api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906]  
[2020-05-31 23:28:47,689: WARNING/ForkPoolWorker-1] started
[2020-05-31 23:28:47,690: WARNING/ForkPoolWorker-1] retry
[2020-05-31 23:28:47,721: INFO/MainProcess] Received task: api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906]  ETA:[2020-05-31 23:28:57.692348+00:00] 
[2020-05-31 23:28:47,722: INFO/MainProcess] Received task: api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906]  ETA:[2020-05-31 23:28:57.716321+00:00] 
[2020-05-31 23:28:47,777: INFO/ForkPoolWorker-1] Task api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906] retry: Retry in 10s: Retry(Retry(...), Exception('i have filled now'), 10)
[2020-05-31 23:28:57,999: WARNING/ForkPoolWorker-1] started
[2020-05-31 23:28:58,000: WARNING/ForkPoolWorker-1] ended
[2020-05-31 23:28:58,062: INFO/ForkPoolWorker-1] Task api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906] succeeded in 0.34440315900428686s: None
[2020-05-31 23:28:58,301: WARNING/ForkPoolWorker-1] started
[2020-05-31 23:28:58,302: WARNING/ForkPoolWorker-1] retry
[2020-05-31 23:28:58,304: INFO/MainProcess] Received task: api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906]  ETA:[2020-05-31 23:29:08.303091+00:00] 
[2020-05-31 23:28:58,307: INFO/MainProcess] Received task: api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906]  ETA:[2020-05-31 23:29:08.306141+00:00] 
[2020-05-31 23:28:58,368: INFO/ForkPoolWorker-1] Task api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906] retry: Retry in 10s: Retry(Retry(...), Exception('i have filled now'), 10)
[2020-05-31 23:29:08,572: WARNING/ForkPoolWorker-1] started
[2020-05-31 23:29:08,573: WARNING/ForkPoolWorker-1] ended
[2020-05-31 23:29:08,633: INFO/ForkPoolWorker-1] Task api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906] succeeded in 0.3256059319974156s: None
[2020-05-31 23:29:08,872: WARNING/ForkPoolWorker-1] started
[2020-05-31 23:29:08,873: WARNING/ForkPoolWorker-1] retry
[2020-05-31 23:29:08,875: INFO/MainProcess] Received task: api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906]  ETA:[2020-05-31 23:29:18.873799+00:00] 
[2020-05-31 23:29:08,880: INFO/MainProcess] Received task: api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906]  ETA:[2020-05-31 23:29:18.877550+00:00] 
[2020-05-31 23:29:08,940: INFO/ForkPoolWorker-1] Task api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906] retry: Retry in 10s: Retry(Retry(...), Exception('i have filled now'), 10)
[2020-05-31 23:29:19,144: WARNING/ForkPoolWorker-1] started
[2020-05-31 23:29:19,145: WARNING/ForkPoolWorker-1] ended
[2020-05-31 23:29:19,205: INFO/ForkPoolWorker-1] Task api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906] succeeded in 0.326258520995907s: None
[2020-05-31 23:29:19,444: WARNING/ForkPoolWorker-1] started
[2020-05-31 23:29:19,445: WARNING/ForkPoolWorker-1] retry
[2020-05-31 23:29:19,505: ERROR/ForkPoolWorker-1] Task api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906] raised unexpected: Exception('i have filled now')
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.7/site-packages/celery/app/trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/celery/app/trace.py", line 650, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/celery/app/base.py", line 500, in run
    raise task.retry(exc=exc, **retry_kwargs)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/celery/app/task.py", line 704, in retry
    raise_with_context(exc)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/celery/app/base.py", line 487, in run
    return task._orig_run(*args, **kwargs)
  File "/var/www/django_projects/earthalytics-api/api_v3/tests.py", line 26, in execute
    self.retry(exc=Exception("i have filled now"), args=[param_a, param_b], kwargs=kwargs)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/celery/app/task.py", line 704, in retry
    raise_with_context(exc)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/celery/utils/serialization.py", line 288, in raise_with_context
    _raise_with_context(exc, exc_info[1])
  File "<string>", line 1, in _raise_with_context
Exception: i have filled now

```

</p>
</details>

# Steps to Reproduce
Make a celery task with a retry changing one parameters.
Set the max_retries and countdown.

## Required Dependencies
<!-- Please fill the required dependencies to reproduce this issue -->
* **Minimal Python Version**: N/A or Unknown
* **Minimal Celery Version**: N/A or Unknown
* **Minimal Kombu Version**: N/A or Unknown
* **Minimal Broker Version**: N/A or Unknown
* **Minimal Result Backend Version**: N/A or Unknown
* **Minimal OS and/or Kernel Version**: N/A or Unknown
* **Minimal Broker Client Version**: N/A or Unknown
* **Minimal Result Backend Client Version**: N/A or Unknown

### Python Packages
<!-- Please fill the contents of pip freeze below -->
<details>
<summary><b><code>pip freeze</code> Output:</b></summary>
<p>

```
```

</p>
</details>

### Other Dependencies
<!--
Please provide system dependencies, configuration files
and other dependency information if applicable
-->
<details>
<p>
N/A
</p>
</details>

## Minimally Reproducible Test Case
<!--
Please provide a reproducible test case.
Refer to the Reporting Bugs section in our contribution guide.

We prefer submitting test cases in the form of a PR to our integration test suite.
If you can provide one, please mention the PR number below.
If not, please attach the most minimal code example required to reproduce the issue below.
If the test case is too large, please include a link to a gist or a repository below.
-->

<details>
<p>

```python

@app_cluster.task(bind=True, autoretry_for=(Exception,), max_retries=3,
                  default_retry_delay=10)
def execute(self, param_a, param_b=None, **kwargs):
    print("started")
    if param_b is None:
        param_b = "filled"
        print("retry")
        self.retry(exc=Exception("i have filled now"), args=[param_a, param_b], kwargs=kwargs)
    print("ended")

def test_celery(self):
    sig = execute.si("something")
    t = sig.delay()
    t = 0
```

</p>
</details>

# Expected Behavior
<!-- Describe in detail what you expect to happen -->
I expect the task get overrided with updated parameters
# Actual Behavior
<!--
Describe in detail what actually happened.
Please include a backtrace and surround it with triple backticks (```).
In addition, include the Celery daemon logs, the broker logs,
the result backend logs and system logs below if they will help us debug
the issue.
--> The task goes in retry with origianl parameters.
A new task is made with new parameters and get scheduled.
The "old" task get executed.
The "new" task get executed.
The "old" task goes in retry making a "new" "new task" with updated parameters and the "old" goes scheduled again with original parameters.



Discussion:
I tried to fix that by myself but code part is pretty weird...a function get called and never return jumping in another code part.
task.py
```
        if is_eager:
            # if task was executed eagerly using apply(),
            # then the retry must also be executed eagerly.
            S.apply().get()  # This never return
            if throw:
                raise ret
            return ret
```
Anyway, i figured maybe what happens. A signature is called updated and another signature is re-called not updated, making a split in "def retry"(Maybe Retry dosn't get counted as "Safe Exception"?).

EDIT:
Pretty sure that "Retry" launch an exception for a real retry(so a new job get executed updated) BUT Retry is an exception too, so "this job" goes in exception and need to be re-executed.
Here is the guilty:
`autoretry_for=(Exception,)`
OR WORSE:
Retry fail before the real end and can't handle the exception.
Ok, i fixed it.
2 Main error and exception overlap when in eager.
Fix in minutes.
#6137 
I saw in master
```
        if is_eager:
            # if task was executed eagerly using apply(),
            # then the retry must also be executed eagerly.
            S.apply()
            if throw:
                raise ret
            return ret
```
got changed removing "S.apply()". I can't run the master branch but...works? Cause in my "4.4.2" this one run the task in sync locally when eager.


Resolving patch:
diff --git a/celery/app/base.py b/celery/app/base.py
--- a/celery/app/base.py
+++ b/celery/app/base.py
@@ -23,7 +23,7 @@
                            _register_app, _set_current_app, _task_stack,
                            connect_on_app_finalize, get_current_app,
                            get_current_worker_task, set_default_app)
-from celery.exceptions import AlwaysEagerIgnored, ImproperlyConfigured, Ignore
+from celery.exceptions import AlwaysEagerIgnored, ImproperlyConfigured, Ignore, Retry
 from celery.five import (UserDict, bytes_if_py2, python_2_unicode_compatible,
                          values)
 from celery.loaders import get_loader_cls
@@ -492,6 +492,8 @@ def run(*args, **kwargs):
                         # If Ignore signal occures task shouldn't be retried,
                         # even if it suits autoretry_for list
                         raise
+                    except Retry:
+                        raise
                     except autoretry_for as exc:
                         if retry_backoff:
                             retry_kwargs['countdown'] = \


Test patch:
diff --git a/t/unit/tasks/test_tasks.py b/t/unit/tasks/test_tasks.py
--- a/t/unit/tasks/test_tasks.py
+++ b/t/unit/tasks/test_tasks.py
@@ -95,7 +95,7 @@ def retry_task_noargs(self, **kwargs):
         self.retry_task_noargs = retry_task_noargs
 
         @self.app.task(bind=True, max_retries=3, iterations=0, shared=False)
-        def retry_task_without_throw(self, **kwargs):
+        def retry_task_return_without_throw(self, **kwargs):
             self.iterations += 1
             try:
                 if self.request.retries >= 3:
@@ -105,7 +105,60 @@ def retry_task_without_throw(self, **kwargs):
             except Exception as exc:
                 return self.retry(exc=exc, throw=False)
 
-        self.retry_task_without_throw = retry_task_without_throw
+        self.retry_task_return_without_throw = retry_task_return_without_throw
+
+        @self.app.task(bind=True, max_retries=3, iterations=0, shared=False)
+        def retry_task_return_with_throw(self, **kwargs):
+            self.iterations += 1
+            try:
+                if self.request.retries >= 3:
+                    return 42
+                else:
+                    raise Exception("random code exception")
+            except Exception as exc:
+                return self.retry(exc=exc, throw=True)
+
+        self.retry_task_return_with_throw = retry_task_return_with_throw
+
+        @self.app.task(bind=True, max_retries=3, iterations=0, shared=False, autoretry_for=(Exception,))
+        def retry_task_auto_retry_with_single_new_arg(self, ret=None, **kwargs):
+            if ret is None:
+                return self.retry(exc=Exception("I have filled now"), args=["test"], kwargs=kwargs)
+            else:
+                return ret
+
+        self.retry_task_auto_retry_with_single_new_arg = retry_task_auto_retry_with_single_new_arg
+
+        @self.app.task(bind=True, max_retries=3, iterations=0, shared=False)
+        def retry_task_auto_retry_with_new_args(self, ret=None, place_holder=None, **kwargs):
+            if ret is None:
+                return self.retry(args=[place_holder, place_holder], kwargs=kwargs)
+            else:
+                return ret
+
+        self.retry_task_auto_retry_with_new_args = retry_task_auto_retry_with_new_args
+
+        @self.app.task(bind=True, max_retries=3, iterations=0, shared=False, autoretry_for=(Exception,))
+        def retry_task_auto_retry_exception_with_new_args(self, ret=None, place_holder=None, **kwargs):
+            if ret is None:
+                return self.retry(exc=Exception("I have filled"), args=[place_holder, place_holder], kwargs=kwargs)
+            else:
+                return ret
+
+        self.retry_task_auto_retry_exception_with_new_args = retry_task_auto_retry_exception_with_new_args
+
+        @self.app.task(bind=True, max_retries=3, iterations=0, shared=False)
+        def retry_task_raise_without_throw(self, **kwargs):
+            self.iterations += 1
+            try:
+                if self.request.retries >= 3:
+                    return 42
+                else:
+                    raise Exception("random code exception")
+            except Exception as exc:
+                raise self.retry(exc=exc, throw=False)
+
+        self.retry_task_raise_without_throw = retry_task_raise_without_throw
 
         @self.app.task(bind=True, max_retries=3, iterations=0,
                        base=MockApplyTask, shared=False)
@@ -365,7 +418,22 @@ def test_retry_kwargs_can_be_empty(self):
             self.retry_task_mockapply.pop_request()
 
     def test_retry_without_throw_eager(self):
-        assert self.retry_task_without_throw.apply().get() == 42
+        assert self.retry_task_return_without_throw.apply().get() == 42
+
+    def test_raise_without_throw_eager(self):
+        assert self.retry_task_raise_without_throw.apply().get() == 42
+
+    def test_return_with_throw_eager(self):
+        assert self.retry_task_return_with_throw.apply().get() == 42
+
+    def test_eager_retry_with_single_new_params(self):
+        assert self.retry_task_auto_retry_with_single_new_arg.apply().get() == "test"
+
+    def test_eager_retry_with_new_params(self):
+        assert self.retry_task_auto_retry_with_new_args.si(place_holder="test").apply().get() == "test"
+
+    def test_eager_retry_with_autoretry_for_exception(self):
+        assert self.retry_task_auto_retry_exception_with_new_args.si(place_holder="test").apply().get() == "test"
 
     def test_retry_eager_should_return_value(self):
         self.retry_task.max_retries = 3