Repository: ansible/ansible

Issue:
Noisy output using 'debug: var=item.' with 'with_items'
<!---
Verify first that your issue/request is not already reported on GitHub.
THIS FORM WILL BE READ BY A MACHINE, COMPLETE ALL SECTIONS AS DESCRIBED.
Also test if the latest release, and devel branch are affected too.
ALWAYS add information AFTER (OUTSIDE) these html comments.
Otherwise it may end up being automatically closed by our bot. -->

##### SUMMARY
<!--- Explain the problem briefly -->
```
- debug: var=item.X
  with_items: "{{Y}}"
```
unexpectedly shows
```
    "changed": false, 
    "item": {...
```
as well as the var expression and its value for each loop item.

Reversing the fix in issue #12952 did not correct this issue - though it seemed possibly related.
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
<!--- Insert, BELOW THIS COMMENT, the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path-->

##### ANSIBLE VERSION
<!--- Paste, BELOW THIS COMMENT, verbatim output from "ansible --version" between quotes below -->
```
ansible 2.4.3.0
  config file = /home/bruce/projects/biosphere_ops_dev/ansible_with_roles/ansible.cfg
  configured module search path = [u'/home/bruce/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.6 (default, Nov 23 2017, 15:49:48) [GCC 4.8.4]


```

##### CONFIGURATION
<!--- If using Ansible 2.4 or above, paste, BELOW THIS COMMENT, the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).-->
```
ANSIBLE_PIPELINING(/XXX/ansible.cfg) = True
DEFAULT_HOST_LIST(/XXX/ansible.cfg) = [u'/XXX/ansible_with_roles/inventory']
DEFAULT_SCP_IF_SSH(/XXX/ansible.cfg) = True
DEFAULT_VAULT_PASSWORD_FILE(/XXX/ansible.cfg) = /XXX/vault_pass
```
##### OS / ENVIRONMENT
<!--- Mention, BELOW THIS COMMENT, the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.-->
N/A
##### STEPS TO REPRODUCE
<!--- For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used. -->

<!--- Paste example playbooks or commands between quotes below -->
create playbook `debug_var_loop_output_bug.yml`
```yaml
---
- hosts: localhost
  become: false
  gather_facts: no

  tasks:
    - set_fact:
        test_data:
          - a: 1
            b: 2
          - a: 3
            b: 4
    - debug: var=test_data[0].b
    - debug: var=item.b
      with_items: "{{test_data}}"

```
run this playbook as follows
```
ansible-playbook debug_var_loop_output_bug.yml -i localhost,
```
<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
```PLAY [localhost] **********************************************************************************************************************************************************

TASK [set_fact] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [debug] **************************************************************************************************************************************************************
ok: [localhost] => {
    "test_data[0].b": "2"
}

TASK [debug] **************************************************************************************************************************************************************
ok: [localhost] => (item={u'a': 1, u'b': 2}) => {
    "item.b": "2"
}
ok: [localhost] => (item={u'a': 3, u'b': 4}) => {
    "item.b": "4"
}

PLAY RECAP ****************************************************************************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0   
```

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
```
PLAY [localhost] **********************************************************************************************************************************************************

TASK [set_fact] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [debug] **************************************************************************************************************************************************************
ok: [localhost] => {
    "test_data[0].b": "2"
}

TASK [debug] **************************************************************************************************************************************************************
ok: [localhost] => (item={u'a': 1, u'b': 2}) => {
    "changed": false, 
    "item": {
        "a": 1, 
        "b": 2
    }, 
    "item.b": "2"
}
ok: [localhost] => (item={u'a': 3, u'b': 4}) => {
    "changed": false, 
    "item": {
        "a": 3, 
        "b": 4
    }, 
    "item.b": "4"
}

PLAY RECAP ****************************************************************************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0   
```
<!--- Paste verbatim command output between quotes below -->
```

```



Discussion:

Resolving patch:
diff --git a/lib/ansible/executor/task_executor.py b/lib/ansible/executor/task_executor.py
--- a/lib/ansible/executor/task_executor.py
+++ b/lib/ansible/executor/task_executor.py
@@ -278,14 +278,19 @@ def _run_loop(self, items):
         label = None
         loop_pause = 0
         templar = Templar(loader=self._loader, shared_loader_obj=self._shared_loader_obj, variables=self._job_vars)
+
+        # FIXME: move this to the object itself to allow post_validate to take care of templating (loop_control.post_validate)
         if self._task.loop_control:
-            # FIXME: move this to the object itself to allow post_validate to take care of templating
             loop_var = templar.template(self._task.loop_control.loop_var)
             index_var = templar.template(self._task.loop_control.index_var)
             loop_pause = templar.template(self._task.loop_control.pause)
-            # the these may be 'None', so we still need to default to something useful
-            # this is tempalted below after an item is assigned
-            label = (self._task.loop_control.label or ('{{' + loop_var + '}}'))
+
+            # This may be 'None',so it is tempalted below after we ensure a value and an item is assigned
+            label = self._task.loop_control.label
+
+        # ensure we always have a label
+        if label is None:
+            label = '{{' + loop_var + '}}'
 
         if loop_var in task_vars:
             display.warning(u"The loop variable '%s' is already in use. "
@@ -339,8 +344,8 @@ def _run_loop(self, items):
             res['_ansible_item_result'] = True
             res['_ansible_ignore_errors'] = task_fields.get('ignore_errors')
 
-            if label is not None:
-                res['_ansible_item_label'] = templar.template(label, cache=False)
+            # gets templated here unlike rest of loop_control fields, depends on loop_var above
+            res['_ansible_item_label'] = templar.template(label, cache=False)
 
             self._rslt_q.put(
                 TaskResult(
diff --git a/lib/ansible/plugins/callback/__init__.py b/lib/ansible/plugins/callback/__init__.py
--- a/lib/ansible/plugins/callback/__init__.py
+++ b/lib/ansible/plugins/callback/__init__.py
@@ -21,9 +21,12 @@
 
 import difflib
 import json
+import os
 import sys
 import warnings
+
 from copy import deepcopy
+from collections import MutableMapping
 
 from ansible import constants as C
 from ansible.parsing.ajson import AnsibleJSONEncoder
@@ -78,7 +81,7 @@ def __init__(self, display=None, options=None):
         if options is not None:
             self.set_options(options)
 
-        self._hide_in_debug = ('changed', 'failed', 'skipped', 'invocation')
+        self._hide_in_debug = ('changed', 'failed', 'skipped', 'invocation', 'skip_reason')
 
     ''' helper for callbacks, so they don't all have to include deepcopy '''
     _copy_result = deepcopy
@@ -172,7 +175,7 @@ def _get_diff(self, difflist):
                     if 'before' in diff and 'after' in diff:
                         # format complex structures into 'files'
                         for x in ['before', 'after']:
-                            if isinstance(diff[x], dict):
+                            if isinstance(diff[x], MutableMapping):
                                 diff[x] = json.dumps(diff[x], sort_keys=True, indent=4, separators=(',', ': ')) + '\n'
                         if 'before_header' in diff:
                             before_header = "before: %s" % diff['before_header']
@@ -221,27 +224,38 @@ def _get_diff(self, difflist):
                 ret.append(">> the files are different, but the diff library cannot compare unicode strings\n\n")
         return u''.join(ret)
 
-    def _get_item(self, result):
+    def _get_item_label(self, result):
+        ''' retrieves the value to be displayed as a label for an item entry from a result object'''
         if result.get('_ansible_no_log', False):
             item = "(censored due to no_log)"
-        elif result.get('_ansible_item_label', False):
-            item = result.get('_ansible_item_label')
         else:
-            item = result.get('item', None)
-
+            item = result.get('_ansible_item_label', result.get('item'))
         return item
 
+    def _get_item(self, result):
+        ''' here for backwards compat, really should have always been named: _get_item_label'''
+        cback = getattr(self, 'NAME', os.path.basename(__file__))
+        self._display.deprecated("The %s callback plugin should be updated to use the _get_item_label method instead" % cback, version="2.11")
+        return self._get_item_label(result)
+
     def _process_items(self, result):
         # just remove them as now they get handled by individual callbacks
         del result._result['results']
 
     def _clean_results(self, result, task_name):
         ''' removes data from results for display '''
+
+        # mostly controls that debug only outputs what it was meant to
         if task_name in ['debug']:
-            for hideme in self._hide_in_debug:
-                result.pop(hideme, None)
-                if 'msg' in result:
-                    result.pop('item', None)
+            if 'msg' in result:
+                # msg should be alone
+                for key in list(result.keys()):
+                    if key != 'msg' and not key.startswith('_'):
+                        result.pop(key)
+            else:
+                # 'var' value as field, so eliminate others and what is left should be varname
+                for hidme in self._hide_in_debug:
+                    result.pop(hidme, None)
 
     def set_play_context(self, play_context):
         pass
@@ -324,7 +338,7 @@ def v2_runner_on_ok(self, result):
     def v2_runner_on_skipped(self, result):
         if C.DISPLAY_SKIPPED_HOSTS:
             host = result._host.get_name()
-            self.runner_on_skipped(host, self._get_item(getattr(result._result, 'results', {})))
+            self.runner_on_skipped(host, self._get_item_label(getattr(result._result, 'results', {})))
 
     def v2_runner_on_unreachable(self, result):
         host = result._host.get_name()
diff --git a/lib/ansible/plugins/callback/default.py b/lib/ansible/plugins/callback/default.py
--- a/lib/ansible/plugins/callback/default.py
+++ b/lib/ansible/plugins/callback/default.py
@@ -205,7 +205,7 @@ def v2_runner_item_on_ok(self, result):
         else:
             msg += ": [%s]" % result._host.get_name()
 
-        msg += " => (item=%s)" % (self._get_item(result._result),)
+        msg += " => (item=%s)" % (self._get_item_label(result._result),)
 
         if (self._display.verbosity > 0 or '_ansible_verbose_always' in result._result) and '_ansible_verbose_override' not in result._result:
             msg += " => %s" % self._dump_results(result._result)
@@ -224,12 +224,12 @@ def v2_runner_item_on_failed(self, result):
             msg += "[%s]" % (result._host.get_name())
 
         self._handle_warnings(result._result)
-        self._display.display(msg + " (item=%s) => %s" % (self._get_item(result._result), self._dump_results(result._result)), color=C.COLOR_ERROR)
+        self._display.display(msg + " (item=%s) => %s" % (self._get_item_label(result._result), self._dump_results(result._result)), color=C.COLOR_ERROR)
 
     def v2_runner_item_on_skipped(self, result):
         if self._plugin_options.get('show_skipped_hosts', C.DISPLAY_SKIPPED_HOSTS):  # fallback on constants for inherited plugins missing docs
             self._clean_results(result._result, result._task.action)
-            msg = "skipping: [%s] => (item=%s) " % (result._host.get_name(), self._get_item(result._result))
+            msg = "skipping: [%s] => (item=%s) " % (result._host.get_name(), self._get_item_label(result._result))
             if (self._display.verbosity > 0 or '_ansible_verbose_always' in result._result) and '_ansible_verbose_override' not in result._result:
                 msg += " => %s" % self._dump_results(result._result)
             self._display.display(msg, color=C.COLOR_SKIP)


Test patch:
diff --git a/test/units/plugins/callback/test_callback.py b/test/units/plugins/callback/test_callback.py
--- a/test/units/plugins/callback/test_callback.py
+++ b/test/units/plugins/callback/test_callback.py
@@ -51,6 +51,7 @@ def test_display_verbose(self):
 
 
 class TestCallbackResults(unittest.TestCase):
+
     def test_get_item(self):
         cb = CallbackBase()
         results = {'item': 'some_item'}
@@ -67,6 +68,22 @@ def test_get_item_no_log(self):
         res = cb._get_item(results)
         self.assertEquals(res, "some_item")
 
+    def test_get_item_label(self):
+        cb = CallbackBase()
+        results = {'item': 'some_item'}
+        res = cb._get_item_label(results)
+        self.assertEquals(res, 'some_item')
+
+    def test_get_item_label_no_log(self):
+        cb = CallbackBase()
+        results = {'item': 'some_item', '_ansible_no_log': True}
+        res = cb._get_item_label(results)
+        self.assertEquals(res, "(censored due to no_log)")
+
+        results = {'item': 'some_item', '_ansible_no_log': False}
+        res = cb._get_item_label(results)
+        self.assertEquals(res, "some_item")
+
     def test_clean_results_debug_task(self):
         cb = CallbackBase()
         result = {'item': 'some_item',