Repository: ansible/ansible

Issue:
retries are not working properly
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
retries

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
lucas.romano@ubuntudesktop:~/Desktop$ ansible --version
ansible 2.3.2.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/usr/share/ansible']
  python version = 2.7.6 (default, Oct 26 2016, 20:30:19) [GCC 4.8.4]
```
##### OS / ENVIRONMENT
Ubuntu / Amazon Linux / CentOS

##### SUMMARY
"retries" are not retrying the task when is failing

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
---
- hosts: "localhost"
  gather_facts: no
# using a non-existent url to fail
  tasks:
    - name: "check retry"
      uri:
        url: https://yahoo.com/1
      retries: 3
      delay: 1
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
The "retries: 3" needs to retry the task 3 times, but ansible is not doing  that.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
lucas.romano@ubuntudesktop:~/Desktop$ ansible-playbook  -i "localhost," -c local test_retries.yml -vv
Using /etc/ansible/ansible.cfg as config file

PLAYBOOK: test_retries.yml ************************************************************************************************************************************************************************************************************
1 plays in test_retries.yml

PLAY [localhost] **********************************************************************************************************************************************************************************************************************
META: ran handlers

TASK [check retry] ********************************************************************************************************************************************************************************************************************
task path: /home/lucas.romano/Desktop/test_retries.yml:6
 [WARNING]: Module did not set no_log for password

fatal: [localhost]: FAILED! => {"age": "0", "cache_control": "private", "changed": false, "connection": "keep-alive", "content": "<html>\n<meta charset='utf-8'>\n<script>\nvar u='https://www.yahoo.com/?err=404&err_url=https%3a%2f%2fwww.yahoo.com%2f1';\nif(window!=window.top){\n  document.write('<p>Content is currently unavailable.</p><img src=\"//geo.yahoo.com/p?s=1197757039&t='+new Date().getTime()+'&_R='+encodeURIComponent(document.referrer)+'&err=404&err_url='+u+'\" width=\"0px\" height=\"0px\"/>');\n}else{\n  window.location.replace(u);\n}\n</script>\n<noscript><META http-equiv=\"refresh\" content=\"0;URL='https://www.yahoo.com/?err=404&err_url=https%3a%2f%2fwww.yahoo.com%2f1'\"></noscript>\n</html>\n", "content_length": "562", "content_type": "text/html; charset=UTF-8", "date": "Fri, 11 Aug 2017 17:11:44 GMT", "failed": true, "msg": "Status code was not 200", "p3p": "policyref=\"https://policies.yahoo.com/w3c/p3p.xml\", CP=\"CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE LOC GOV\"", "redirected": false, "server": "ATS", "status": 404, "via": "http/1.1 ir14.fp.gq1.yahoo.com (ApacheTrafficServer)"}
	to retry, use: --limit @/home/lucas.romano/Desktop/test_retries.retry

PLAY RECAP ****************************************************************************************************************************************************************************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1 
```



Discussion:
(this is a reformat of the error message for easier reading)
```
fatal: [localhost]: FAILED! =>
{"age": "0",
"cache_control": "private",
"changed": false,
"connection": "keep-alive",
"content": "<html>
            <meta charset='utf-8'>
            <script>
            var u='https://www.yahoo.com/?err=404&err_url=https%3a%2f%2fwww.yahoo.com%2f1';
            if(window!=window.top){
              document.write('<p>Content is currently unavailable.</p><img src="//geo.yahoo.com/p?s=1197757039&t='+new Date().getTime()+'&_R='+encodeURIComponent(document.referrer)+'&err=404&err_url='+u+'" width="0px" height="0px"/>');
            }else{
              window.location.replace(u);
            }
            </script>
            <noscript><META http-equiv="refresh" content="0;URL='https://www.yahoo.com/?err=404&err_url=https%3a%2f%2fwww.yahoo.com%2f1'"></noscript>
            </html>",
"content_length": "562",
"content_type": "text/html; charset=UTF-8",
"date": "Fri, 11 Aug 2017 17:11:44 GMT",
"failed": true,
"msg": "Status code was not 200",
"p3p": "policyref=\"https://policies.yahoo.com/w3c/p3p.xml\", CP=\"CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE LOC GOV\"",
"redirected": false,
"server": "ATS",
"status": 404,
"via": "http/1.1 ir14.fp.gq1.yahoo.com (ApacheTrafficServer)"}

	to retry, use: --limit @/home/lucas.romano/Desktop/test_retries.retry
```


Issue reproduced also in 2.4, it affects all the tasks with "retries" parameter without the "until" parameter defined.

Workaround:
Add an until parameter in the task based on your goals, example:
```
    - name: "check retry"
      uri:
        url: https://yahoo.com/1
      retries: 3
      delay: 1
      register: uri_output
      until: uri_output.status == '404'
      ignore_errors: true
```

Code related to this issue:
https://github.com/ansible/ansible/blob/b35f39479b71f8506889512a4b6c64d564c5d71f/lib/ansible/executor/task_executor.py#L511

If until parameter isn't used for the task, retries is forced to 1.

Documentation doesn't describe the need of until option used when retries is used ( http://docs.ansible.com/ansible/latest/playbooks_loops.html ) but the example show until used with retries. 

I'm checking if the source code can be changed to use retries also without until.

Thanks @giovannisciortino great review. Maybe we can fix it!
I'm using a similar workaround now, generic for any module:
```
register: result
until: result | success
```
Trying to reproduce based on the snippet from https://github.com/ansible/ansible/issues/28078#issuecomment-326000971 with devel f040a37c9fc45333670ac29ee9ddb90929186e54

# added to test/integration/targets/uri/tasks/main.yml
``` yaml
- name: "check retry without an until"
  uri:
    url: http://localhost:{{ http_port }}/this_shouldnt_exist/1
  retries: 3
  delay: 1
  register: uri_output
  # until: uri_output.status == '404'
  ignore_errors: true

- debug:
    var: uri_output

- name: "check retry with until"
  uri:
    url: http://localhost:{{ http_port }}/this_shouldnt_exist/1
  retries: 3
  delay: 1
  register: uri_output
  until: uri_output.status == '404'
  ignore_errors: true

- debug:
    var: uri_output
```

Output:

```
TASK [uri : check retry without an until] **************************************
task path: /root/ansible/test/integration/targets/uri/tasks/main.yml:57
fatal: [testhost]: FAILED! => {"changed": false, "connection": "close", "content": "<head>\n<title>Error response</title>\n</head>\n<body>\n<h1>Error response</h1>\n<p>Error code 404.\n<p>Message: File not found.\n<p>Error code explanation: 404 = Nothing matches the given URI.\n</body>\n", "content_type": "text/html", "date": "Wed, 30 Aug 2017 16:11:07 GMT", "failed": true, "msg": "Status code was not [200]: HTTP Error 404: File not found", "redirected": false, "server": "SimpleHTTP/0.6 Python/2.7.5", "status": 404, "url": "http://localhost:15260/this_shouldnt_exist/1"}
...ignoring

TASK [uri : debug] *************************************************************
task path: /root/ansible/test/integration/targets/uri/tasks/main.yml:66
ok: [testhost] => {
    "uri_output": {
        "changed": false, 
        "connection": "close", 
        "content": "<head>\n<title>Error response</title>\n</head>\n<body>\n<h1>Error response</h1>\n<p>Error code 404.\n<p>Message: File not found.\n<p>Error code explanation: 404 = Nothing matches the given URI.\n</body>\n", 
        "content_type": "text/html", 
        "date": "Wed, 30 Aug 2017 16:11:07 GMT", 
        "failed": true, 
        "msg": "Status code was not [200]: HTTP Error 404: File not found", 
        "redirected": false, 
        "server": "SimpleHTTP/0.6 Python/2.7.5", 
        "status": 404, 
        "url": "http://localhost:15260/this_shouldnt_exist/1"
    }
}

TASK [uri : check retry with until] ********************************************
task path: /root/ansible/test/integration/targets/uri/tasks/main.yml:69
FAILED - RETRYING: check retry with until (3 retries left).
FAILED - RETRYING: check retry with until (2 retries left).
FAILED - RETRYING: check retry with until (1 retries left).
fatal: [testhost]: FAILED! => {"attempts": 3, "changed": false, "connection": "close", "content": "<head>\n<title>Error response</title>\n</head>\n<body>\n<h1>Error response</h1>\n<p>Error code 404.\n<p>Message: File not found.\n<p>Error code explanation: 404 = Nothing matches the given URI.\n</body>\n", "content_type": "text/html", "date": "Wed, 30 Aug 2017 16:11:11 GMT", "failed": true, "msg": "Status code was not [200]: HTTP Error 404: File not found", "redirected": false, "server": "SimpleHTTP/0.6 Python/2.7.5", "status": 404, "url": "http://localhost:15260/this_shouldnt_exist/1"}
...ignoring

TASK [uri : debug] *************************************************************
task path: /root/ansible/test/integration/targets/uri/tasks/main.yml:78
ok: [testhost] => {
    "uri_output": {
        "attempts": 3, 
        "changed": false, 
        "connection": "close", 
        "content": "<head>\n<title>Error response</title>\n</head>\n<body>\n<h1>Error response</h1>\n<p>Error code 404.\n<p>Message: File not found.\n<p>Error code explanation: 404 = Nothing matches the given URI.\n</body>\n", 
        "content_type": "text/html", 
        "date": "Wed, 30 Aug 2017 16:11:11 GMT", 
        "failed": true, 
        "msg": "Status code was not [200]: HTTP Error 404: File not found", 
        "redirected": false, 
        "server": "SimpleHTTP/0.6 Python/2.7.5", 
        "status": 404, 
        "url": "http://localhost:15260/this_shouldnt_exist/1"
    }
}
```

I am seeing the same behavior between stable-2.3 and devel...

Using a simple repro case:

repro_uri.yml:
``` yaml
---
- hosts: localhost
  gather_facts: no
# using a non-existent url to fail
  tasks:
    - name: "check retry"
      uri:
        url: https://yahoo.com/1
      retries: 3
      delay: 1
      ignore_errors: true

    - name: "check retry with an until"
      uri:
        url: https://yahoo.com/1
      retries: 3
      delay: 1
      register: uri_output
      until: uri_output.status == '404'
      ignore_errors: true
```

With stable-2.3:

```
[newswoop:F25:ansible (stable-2.3 % u=)]$ ansible --version
ansible 2.3.2.0 (stable-2.3 8bb12fba3d) last updated 2017/08/30 12:17:35 (GMT -400)
  config file = /home/adrian/src/ansible/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.13 (default, May 10 2017, 20:04:28) [GCC 6.3.1 20161221 (Red Hat 6.3.1-1)]
[newswoop:F25:ansible (stable-2.3 % u=)]$ ansible-playbook -vv repro_uri.yml 
Using /home/adrian/src/ansible/ansible.cfg as config file
 [WARNING]: Host file not found: /etc/ansible/hosts

 [WARNING]: provided hosts list is empty, only localhost is available


PLAYBOOK: repro_uri.yml **************************************************************************
1 plays in repro_uri.yml

PLAY [localhost] *********************************************************************************
META: ran handlers

TASK [check retry] *******************************************************************************
task path: /home/adrian/src/ansible/repro_uri.yml:6
fatal: [localhost]: FAILED! => {"age": "0", "cache_control": "private", "changed": false, "connection": "close", "content": "<html>\n<meta charset='utf-8'>\n<script>\nvar u='https://www.yahoo.com/?err=404&err_url=https%3a%2f%2fwww.yahoo.com%2f1';\nif(window!=window.top){\n  document.write('<p>Content is currently unavailable.</p><img src=\"//geo.yahoo.com/p?s=1197757039&t='+new Date().getTime()+'&_R='+encodeURIComponent(document.referrer)+'&err=404&err_url='+u+'\" width=\"0px\" height=\"0px\"/>');\n}else{\n  window.location.replace(u);\n}\n</script>\n<noscript><META http-equiv=\"refresh\" content=\"0;URL='https://www.yahoo.com/?err=404&err_url=https%3a%2f%2fwww.yahoo.com%2f1'\"></noscript>\n</html>\n", "content_length": "562", "content_type": "text/html; charset=UTF-8", "date": "Wed, 30 Aug 2017 16:19:18 GMT", "failed": true, "msg": "Status code was not [200]: HTTP Error 404: Not Found", "p3p": "policyref=\"https://policies.yahoo.com/w3c/p3p.xml\", CP=\"CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE LOC GOV\"", "redirected": false, "server": "ATS", "status": 404, "url": "https://yahoo.com/1", "via": "http/1.1 ir30.fp.bf1.yahoo.com (ApacheTrafficServer)"}
...ignoring

TASK [check retry with an until] *****************************************************************
task path: /home/adrian/src/ansible/repro_uri.yml:13
FAILED - RETRYING: check retry with an until (3 retries left).
FAILED - RETRYING: check retry with an until (2 retries left).
FAILED - RETRYING: check retry with an until (1 retries left).
fatal: [localhost]: FAILED! => {"age": "0", "attempts": 3, "cache_control": "private", "changed": false, "connection": "close", "content": "<html>\n<meta charset='utf-8'>\n<script>\nvar u='https://www.yahoo.com/?err=404&err_url=https%3a%2f%2fwww.yahoo.com%2f1';\nif(window!=window.top){\n  document.write('<p>Content is currently unavailable.</p><img src=\"//geo.yahoo.com/p?s=1197757039&t='+new Date().getTime()+'&_R='+encodeURIComponent(document.referrer)+'&err=404&err_url='+u+'\" width=\"0px\" height=\"0px\"/>');\n}else{\n  window.location.replace(u);\n}\n</script>\n<noscript><META http-equiv=\"refresh\" content=\"0;URL='https://www.yahoo.com/?err=404&err_url=https%3a%2f%2fwww.yahoo.com%2f1'\"></noscript>\n</html>\n", "content_length": "562", "content_type": "text/html; charset=UTF-8", "date": "Wed, 30 Aug 2017 16:19:29 GMT", "failed": true, "msg": "Status code was not [200]: HTTP Error 404: Not Found", "p3p": "policyref=\"https://policies.yahoo.com/w3c/p3p.xml\", CP=\"CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE LOC GOV\"", "redirected": false, "server": "ATS", "status": 404, "url": "https://yahoo.com/1", "via": "http/1.1 ir51.fp.bf1.yahoo.com (ApacheTrafficServer)"}
...ignoring
META: ran handlers
META: ran handlers

PLAY RECAP ***************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0   
```

With devel 0a2df4cdaf67470410d0791dcd33b179c7cce507
```
[newswoop:F25:ansible (devel % u=)]$ ansible --version
ansible 2.4.0 (devel 0a2df4cdaf) last updated 2017/08/30 12:13:24 (GMT -400)
  config file = /home/adrian/src/ansible/ansible.cfg
  configured module search path = [u'/home/adrian/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/adrian/src/ansible/lib/ansible
  executable location = /home/adrian/src/ansible/bin/ansible
  python version = 2.7.13 (default, May 10 2017, 20:04:28) [GCC 6.3.1 20161221 (Red Hat 6.3.1-1)]
[newswoop:F25:ansible (devel % u=)]$ ansible-playbook -vv repro_uri.yml 
ansible-playbook 2.4.0 (devel 0a2df4cdaf) last updated 2017/08/30 12:13:24 (GMT -400)
  config file = /home/adrian/src/ansible/ansible.cfg
  configured module search path = [u'/home/adrian/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/adrian/src/ansible/lib/ansible
  executable location = /home/adrian/src/ansible/bin/ansible-playbook
  python version = 2.7.13 (default, May 10 2017, 20:04:28) [GCC 6.3.1 20161221 (Red Hat 6.3.1-1)]
Using /home/adrian/src/ansible/ansible.cfg as config file
 [WARNING]: No inventory was parsed, only implicit localhost is available

 [WARNING]: Could not match supplied host pattern, ignoring: all

 [WARNING]: provided hosts list is empty, only localhost is available


PLAYBOOK: repro_uri.yml **************************************************************************
1 plays in repro_uri.yml

PLAY [localhost] *********************************************************************************
META: ran handlers

TASK [check retry] *******************************************************************************
task path: /home/adrian/src/ansible/repro_uri.yml:6
fatal: [localhost]: FAILED! => {"age": "0", "cache_control": "private", "changed": false, "connection": "close", "content": "<html>\n<meta charset='utf-8'>\n<script>\nvar u='https://www.yahoo.com/?err=404&err_url=https%3a%2f%2fwww.yahoo.com%2f1';\nif(window!=window.top){\n  document.write('<p>Content is currently unavailable.</p><img src=\"//geo.yahoo.com/p?s=1197757039&t='+new Date().getTime()+'&_R='+encodeURIComponent(document.referrer)+'&err=404&err_url='+u+'\" width=\"0px\" height=\"0px\"/>');\n}else{\n  window.location.replace(u);\n}\n</script>\n<noscript><META http-equiv=\"refresh\" content=\"0;URL='https://www.yahoo.com/?err=404&err_url=https%3a%2f%2fwww.yahoo.com%2f1'\"></noscript>\n</html>\n", "content_length": "562", "content_type": "text/html; charset=UTF-8", "date": "Wed, 30 Aug 2017 16:20:46 GMT", "failed": true, "msg": "Status code was not [200]: HTTP Error 404: Not Found", "p3p": "policyref=\"https://policies.yahoo.com/w3c/p3p.xml\", CP=\"CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE LOC GOV\"", "redirected": false, "server": "ATS", "status": 404, "url": "https://yahoo.com/1", "via": "http/1.1 ir16.fp.bf1.yahoo.com (ApacheTrafficServer)"}
...ignoring

TASK [check retry with an until] *****************************************************************
task path: /home/adrian/src/ansible/repro_uri.yml:13
FAILED - RETRYING: check retry with an until (3 retries left).
FAILED - RETRYING: check retry with an until (2 retries left).
FAILED - RETRYING: check retry with an until (1 retries left).
fatal: [localhost]: FAILED! => {"age": "0", "attempts": 3, "cache_control": "private", "changed": false, "connection": "close", "content": "<html>\n<meta charset='utf-8'>\n<script>\nvar u='https://www.yahoo.com/?err=404&err_url=https%3a%2f%2fwww.yahoo.com%2f1';\nif(window!=window.top){\n  document.write('<p>Content is currently unavailable.</p><img src=\"//geo.yahoo.com/p?s=1197757039&t='+new Date().getTime()+'&_R='+encodeURIComponent(document.referrer)+'&err=404&err_url='+u+'\" width=\"0px\" height=\"0px\"/>');\n}else{\n  window.location.replace(u);\n}\n</script>\n<noscript><META http-equiv=\"refresh\" content=\"0;URL='https://www.yahoo.com/?err=404&err_url=https%3a%2f%2fwww.yahoo.com%2f1'\"></noscript>\n</html>\n", "content_length": "562", "content_type": "text/html; charset=UTF-8", "date": "Wed, 30 Aug 2017 16:20:56 GMT", "failed": true, "msg": "Status code was not [200]: HTTP Error 404: Not Found", "p3p": "policyref=\"https://policies.yahoo.com/w3c/p3p.xml\", CP=\"CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE LOC GOV\"", "redirected": false, "server": "ATS", "status": 404, "url": "https://yahoo.com/1", "via": "http/1.1 ir36.fp.bf1.yahoo.com (ApacheTrafficServer)"}
...ignoring
META: ran handlers
META: ran handlers

PLAY RECAP ***************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0   

```
(testing with 2.3.2-1 shows the same).

As far as I can tell, this in the intended behavior and hasnt changed. 'retries:' requires a 'until:' to do anything useful.

The docs dont mention that however so definitely need to fix the docs.

Resolving patch:
diff --git a/lib/ansible/executor/task_executor.py b/lib/ansible/executor/task_executor.py
--- a/lib/ansible/executor/task_executor.py
+++ b/lib/ansible/executor/task_executor.py
@@ -509,6 +509,7 @@ def _execute(self, variables=None):
                 retries += 1
         else:
             retries = 1
+            display.warning("The parameter 'retries' hasn't been defined, the value for 'retries' is forced to 1.")
 
         delay = self._task.delay
         if delay < 0:


Test patch: