Repository: ansible/ansible

Issue:
looped include_tasks and include_roles lose list members
<!---
Verify first that your issue/request is not already reported on GitHub.
THIS FORM WILL BE READ BY A MACHINE, COMPLETE ALL SECTIONS AS DESCRIBED.
Also test if the latest release, and devel branch are affected too.
ALWAYS add information AFTER (OUTSIDE) these html comments.
Otherwise it may end up being automatically closed by our bot. -->

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
<!--- Insert, BELOW THIS COMMENT, the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path-->
include_tasks
include_roles
with_items

##### ANSIBLE VERSION
<!--- Paste, BELOW THIS COMMENT, verbatim output from "ansible --version" between quotes below -->
```
ansible 2.5.2
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/Users/pikula/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/Cellar/ansible/2.5.2/libexec/lib/python2.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.14 (default, Mar 22 2018, 14:43:05) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)]
```

##### CONFIGURATION
<!--- If using Ansible 2.4 or above, paste, BELOW THIS COMMENT, the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).-->
DEFAULT_ACTION_PLUGIN_PATH(/etc/ansible/ansible.cfg) = [u'/etc/ansible/plugins/a
DEFAULT_BECOME(/etc/ansible/ansible.cfg) = True
DEFAULT_CALLBACK_PLUGIN_PATH(/etc/ansible/ansible.cfg) = [u'/etc/ansible/plugins
DEFAULT_CONNECTION_PLUGIN_PATH(/etc/ansible/ansible.cfg) = [u'/etc/ansible/plugi
DEFAULT_FILTER_PLUGIN_PATH(/etc/ansible/ansible.cfg) = [u'/etc/ansible/plugins/f
DEFAULT_LOOKUP_PLUGIN_PATH(/etc/ansible/ansible.cfg) = [u'/etc/ansible/plugins/l
DEFAULT_VARS_PLUGIN_PATH(/etc/ansible/ansible.cfg) = [u'/etc/ansible/plugins/var

##### OS / ENVIRONMENT
<!--- Mention, BELOW THIS COMMENT, the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.-->
N/A (should be platform independent, manifests on MacOS, not tested elsewhere)

##### SUMMARY
<!--- Explain the problem briefly -->
any task run in loop using
with_items: [1,2,3,4,4,4,4,4,4,4]
runs 10 times, which is expected behavior. However, if you try to include_task or include_role over the same list, it only runs 4 times.

##### STEPS TO REPRODUCE
<!--- For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used. -->

<!--- Paste example playbooks or commands between quotes below -->
Correct:

```yaml
- hosts: localhost
  tasks:
  - debug:
     msg: '{{ item }}'
    with_items: [ 1,2,3,4,4,4,4,4]
  become: no
```

Incorrect:

```yaml
- hosts: localhost
  tasks:
  - include_tasks: 'inklud.yml'
    with_items: [ 1,2,3,4,4,4,4,4]
  become: no
```

```yaml
- debug:
    msg: '{{ item }}'
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
"msg": 1
"msg": 2
"msg": 3
"msg": 4
"msg": 4
"msg": 4
"msg": 4
"msg": 4
"msg": 4
"msg": 4

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
TASK [include_tasks] ************************************************************************
included: inklud.yml for localhost
included: inklud.yml for localhost
included: inklud.yml for localhost
included: inklud.yml for localhost

TASK [debug] ********************************************************************************
ok: [localhost] => {
    "msg": 1
}

TASK [debug] ********************************************************************************
ok: [localhost] => {
    "msg": 2
}

TASK [debug] ********************************************************************************
ok: [localhost] => {
    "msg": 3
}

TASK [debug] ********************************************************************************
ok: [localhost] => {
    "msg": 4
}
```



Discussion:
Files identified in the description:
* [lib/ansible/modules/utilities/logic/include_tasks.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/utilities/logic/include_tasks.py)
* [lib/ansible/playbook/loop_control.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/playbook/loop_control.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
Context for me later: This is a result of includes using a combo of filename+args+parent to determine uniqueness for merging.

In this case, with duplicated items in the loops, those args, are effectively squashed causing deduplication to happen.
Naive Q before reading the source code: is there any scenario where *dynamic* includes require uniqueness check and deduplication? 
It is slightly complicated, but effectively, we get include results for each combination of:

1. filename
1. args
1. task/parent
1. host

So if we did:

```
- include_tasks: foo.yml
```

...and there were 3 hosts, we get 3 individual include results, that we re-merge into a single `IncludedFile` for later processing.

We use the first 3 "properties", to determine if it is the same include across multiple hosts, and then we merge hosts into that unique include.

In your scenario, I imagine we may need additional logic, to see if the host is already in the `IncludedFile`, and if so, it is treated as a new entry.



Resolving patch:
diff --git a/lib/ansible/playbook/included_file.py b/lib/ansible/playbook/included_file.py
--- a/lib/ansible/playbook/included_file.py
+++ b/lib/ansible/playbook/included_file.py
@@ -44,6 +44,8 @@ def __init__(self, filename, args, task, is_role=False):
     def add_host(self, host):
         if host not in self._hosts:
             self._hosts.append(host)
+            return
+        raise ValueError()
 
     def __eq__(self, other):
         return other._filename == self._filename and other._args == self._args and other._task._parent._uuid == self._task._parent._uuid
@@ -160,12 +162,24 @@ def process_include_results(results, iterator, loader, variable_manager):
 
                         inc_file = IncludedFile(role_name, include_variables, new_task, is_role=True)
 
-                    try:
-                        pos = included_files.index(inc_file)
-                        inc_file = included_files[pos]
-                    except ValueError:
-                        included_files.append(inc_file)
-
-                    inc_file.add_host(original_host)
+                    idx = 0
+                    orig_inc_file = inc_file
+                    while 1:
+                        try:
+                            pos = included_files[idx:].index(orig_inc_file)
+                            # pos is relative to idx since we are slicing
+                            # use idx + pos due to relative indexing
+                            inc_file = included_files[idx + pos]
+                        except ValueError:
+                            included_files.append(orig_inc_file)
+                            inc_file = orig_inc_file
+
+                        try:
+                            inc_file.add_host(original_host)
+                        except ValueError:
+                            # The host already exists for this include, advance forward, this is a new include
+                            idx += pos + 1
+                        else:
+                            break
 
         return included_files


Test patch:
diff --git a/test/integration/targets/include_import/runme.sh b/test/integration/targets/include_import/runme.sh
--- a/test/integration/targets/include_import/runme.sh
+++ b/test/integration/targets/include_import/runme.sh
@@ -73,3 +73,9 @@ if [[ -z "$OUT" ]]; then
     echo "apply on import_tasks did not cause error"
     exit 1
 fi
+
+# Test that duplicate items in loop are not deduped
+ANSIBLE_STRATEGY='linear' ansible-playbook tasks/test_include_dupe_loop.yml -i ../../inventory "$@" | tee test_include_dupe_loop.out
+test "$(grep -c 'item=foo' test_include_dupe_loop.out)" = 3
+ANSIBLE_STRATEGY='free' ansible-playbook tasks/test_include_dupe_loop.yml -i ../../inventory "$@" | tee test_include_dupe_loop.out
+test "$(grep -c 'item=foo' test_include_dupe_loop.out)" = 3
\ No newline at end of file
diff --git a/test/integration/targets/include_import/tasks/debug_item.yml b/test/integration/targets/include_import/tasks/debug_item.yml
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/include_import/tasks/debug_item.yml
@@ -0,0 +1,2 @@
+- debug:
+    msg: "item={{ item }}"
diff --git a/test/integration/targets/include_import/tasks/test_include_dupe_loop.yml b/test/integration/targets/include_import/tasks/test_include_dupe_loop.yml
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/include_import/tasks/test_include_dupe_loop.yml
@@ -0,0 +1,8 @@
+- name: Test Include Duplicate Loop Items
+  hosts: testhost
+  tasks:
+    - include_tasks: debug_item.yml
+      loop:
+        - foo
+        - foo
+        - foo