Repository: ansible/ansible

Issue:
Warning displayed: Failure using method (v2_runner_retry) in callback plugin 

##### ISSUE TYPE
 Bug Report

##### COMPONENT NAME
`retries`

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.4.2.0
  config file = /vagrant/ansible.cfg
  configured module search path = [u'/home/ubuntu/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.12 (default, Nov 20 2017, 18:23:56) [GCC 5.4.0 20160609]
```

##### CONFIGURATION
```
ANSIBLE_SSH_ARGS(/vagrant/ansible.cfg) = -o ControlMaster=auto -o ControlPersist=60s -o UserKnownHostsFile=/dev/null
DEFAULT_MANAGED_STR(/vagrant/ansible.cfg) = !! Ansible managed: This file was created by ansible. Do not modify by hand! Changes w
HOST_KEY_CHECKING(/vagrant/ansible.cfg) = False
RETRY_FILES_ENABLED(/vagrant/ansible.cfg) = False
```

##### OS / ENVIRONMENT
`Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-93-generic x86_64)`

##### SUMMARY
The following warning messages shows when using the retries option

`[WARNING]: Failure using method (v2_runner_retry) in callback plugin (<ansible.plugins.callback.default.CallbackModule object at
0x7fd615555150>): 'retries'`

##### STEPS TO REPRODUCE

<!--- Paste example playbooks or commands between quotes below -->
I'm using the following in my playbook
```yaml
- hosts: rack-clouds
  gather_facts: false
  tasks:
    - name: Wait for rackconnnect automation to complete
      local_action:
        module: rax_facts
        username: "{{ rackspace_username }}"
        api_key: "{{ rackspace_api_key }}"
        id: "{{ rax_id }}"
        region: IAD
      register: rax_facts
      until: rax_facts.ansible_facts['rax_metadata']['rackconnect_automation_status']|default('') == 'DEPLOYED'
      retries: 30
      delay: 10
      no_log: true
```

##### EXPECTED RESULTS
No warning

##### ACTUAL RESULTS
Warning displays. Full verbose output below

<!--- Paste verbatim command output between quotes below -->
```
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/cloud/rackspace/rax_facts.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: ubuntu
<localhost> EXEC /bin/sh -c 'echo ~ && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/ubuntu/.ansible/tmp/ansible-tmp-1512068716.66-101171432868715 `" && echo ansible-tmp-1512068716.66-101171432868715="` echo /home/ubuntu/.ansible/tmp/ansible-tmp-1512068716.66-101171432868715 `" ) && sleep 0'
<localhost> PUT /tmp/tmpSaeJHL TO /home/ubuntu/.ansible/tmp/ansible-tmp-1512068716.66-101171432868715/rax_facts.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/ubuntu/.ansible/tmp/ansible-tmp-1512068716.66-101171432868715/ /home/ubuntu/.ansible/tmp/ansible-tmp-1512068716.66-101171432868715/rax_facts.py && sleep 0'
<localhost> EXEC /bin/sh -c 'sudo -H -S  -p "[sudo via ansible, key=cuuysnrvvcvrdimoiiurkgyxldlbqzfn] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-cuuysnrvvcvrdimoiiurkgyxldlbqzfn; /usr/bin/python /home/ubuntu/.ansible/tmp/ansible-tmp-1512068716.66-101171432868715/rax_facts.py; rm -rf "/home/ubuntu/.ansible/tmp/ansible-tmp-1512068716.66-101171432868715/" > /dev/null 2>&1'"'"' && sleep 0'
 [WARNING]: Failure using method (v2_runner_retry) in callback plugin (<ansible.plugins.callback.default.CallbackModule object at
0x7f9aefb73f10>): 'retries'

Callback Exception: 
  File "/usr/lib/python2.7/dist-packages/ansible/executor/task_queue_manager.py", line 375, in send_callback
    method(*new_args, **kwargs)
   File "/usr/lib/python2.7/dist-packages/ansible/plugins/callback/default.py", line 303, in v2_runner_retry
    msg = "FAILED - RETRYING: %s (%d retries left)." % (task_name, result._result['retries'] - result._result['attempts'])
```





Discussion:
Files identified in the description:
None

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
Got the same error with `command` module:
```
    - name: Check that mysql is running
      command: >
        mysql -uroot -p{{ vault_mysql_root_password }} -h{{ __docker_mysql_ip }} -e "SELECT 'mysql is alive!'"
      register: healthcheck_result
      until: healthcheck_result.rc == 0 and healthcheck_result.stdout.find('mysql is alive!') != -1
      retries: 15
      delay: 1
      changed_when: False
      no_log: True
```
It appears as though the issue occurs when `no_log: true` is placed on the task.  The task result is replaced with:

```
{"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}
```

The `default` callback references both:

* `result._result['retries']`
* `result._result['attempts']`

Those will not exist. Since the `retries` and `attempts` are placed in the result, we'll have to figure out how to get access to these.
Reproducer:

```yaml
---
- hosts: localhost
  gather_facts: false
  tasks:
    - name: Check that mysql is running
      command: fail
      register: r
      until: r.rc == 0
      retries: 10
      delay: 1
      changed_when: False
      no_log: true
```
This was introduced by https://github.com/ansible/ansible/commit/01b6c7c9c6b / https://github.com/ansible/ansible/pull/27175

cc @bcoca 

Resolving patch:
diff --git a/lib/ansible/executor/task_result.py b/lib/ansible/executor/task_result.py
--- a/lib/ansible/executor/task_result.py
+++ b/lib/ansible/executor/task_result.py
@@ -11,6 +11,7 @@
 from ansible.vars.clean import strip_internal_keys
 
 _IGNORE = ('failed', 'skipped')
+_PRESERVE = ('attempts', 'changed', 'retries')
 
 
 class TaskResult:
@@ -90,7 +91,11 @@ def clean_copy(self):
             ignore = _IGNORE
 
         if self._result.get('_ansible_no_log', False):
-            result._result = {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}
+            x = {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}
+            for preserve in _PRESERVE:
+                if preserve in self._result:
+                    x[preserve] = self._result[preserve]
+            result._result = x
         elif self._result:
             result._result = deepcopy(self._result)
 


Test patch: