Repository: ansible/ansible

Issue:
include_role+with_items executes tasks num_hosts*num_items times on *each* host
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
core

##### ANSIBLE VERSION
```
devel
```

##### CONFIGURATION
N/A

##### OS / ENVIRONMENT
N/A

##### SUMMARY
include_role+with_items executes once per item per host **on each host**.

##### STEPS TO REPRODUCE
x.yml::
```
---
- hosts: all
  gather_facts: no
  tasks:
    - include_role: name=p
      with_items: [1,2]
```

roles/p/tasks/main.yml::
```
- command: cat /proc/uptime
```

##### EXPECTED RESULTS
I expected ``cat /proc/uptime`` to be executed twice on a and twice on b.

##### ACTUAL RESULTS
It actually executes ``cat /proc/uptime`` **four times** on both hosts.

```
TASK [p : command] ****************************************************************************************************************************************************************************
changed: [b] => {"changed": true, "cmd": ["cat", "/proc/uptime"], "delta": "0:00:00.002532", "end": "2016-12-05 20:00:08.080587", "rc": 0, "start": "2016-12-05 20:00:08.078055", "stderr": "", "stdout": "987830.64 3614256.86", "stdout_lines": ["987830.64 3614256.86"], "warnings": []}
changed: [a] => {"changed": true, "cmd": ["cat", "/proc/uptime"], "delta": "0:00:00.002532", "end": "2016-12-05 20:00:08.080587", "rc": 0, "start": "2016-12-05 20:00:08.078055", "stderr": "", "stdout": "987830.64 3614256.86", "stdout_lines": ["987830.64 3614256.86"], "warnings": []}

TASK [p : command] ****************************************************************************************************************************************************************************
changed: [b] => {"changed": true, "cmd": ["cat", "/proc/uptime"], "delta": "0:00:00.002069", "end": "2016-12-05 20:00:08.214287", "rc": 0, "start": "2016-12-05 20:00:08.212218", "stderr": "", "stdout": "987830.77 3614257.08", "stdout_lines": ["987830.77 3614257.08"], "warnings": []}
changed: [a] => {"changed": true, "cmd": ["cat", "/proc/uptime"], "delta": "0:00:00.002710", "end": "2016-12-05 20:00:08.214238", "rc": 0, "start": "2016-12-05 20:00:08.211528", "stderr": "", "stdout": "987830.77 3614257.08", "stdout_lines": ["987830.77 3614257.08"], "warnings": []}

TASK [p : command] ****************************************************************************************************************************************************************************
changed: [a] => {"changed": true, "cmd": ["cat", "/proc/uptime"], "delta": "0:00:00.002648", "end": "2016-12-05 20:00:08.340613", "rc": 0, "start": "2016-12-05 20:00:08.337965", "stderr": "", "stdout": "987830.90 3614257.30", "stdout_lines": ["987830.90 3614257.30"], "warnings": []}
changed: [b] => {"changed": true, "cmd": ["cat", "/proc/uptime"], "delta": "0:00:00.002124", "end": "2016-12-05 20:00:08.345231", "rc": 0, "start": "2016-12-05 20:00:08.343107", "stderr": "", "stdout": "987830.91 3614257.30", "stdout_lines": ["987830.91 3614257.30"], "warnings": []}

TASK [p : command] ****************************************************************************************************************************************************************************
changed: [b] => {"changed": true, "cmd": ["cat", "/proc/uptime"], "delta": "0:00:00.002734", "end": "2016-12-05 20:00:08.475102", "rc": 0, "start": "2016-12-05 20:00:08.472368", "stderr": "", "stdout": "987831.04 3614257.51", "stdout_lines": ["987831.04 3614257.51"], "warnings": []}
changed: [a] => {"changed": true, "cmd": ["cat", "/proc/uptime"], "delta": "0:00:00.002468", "end": "2016-12-05 20:00:08.478968", "rc": 0, "start": "2016-12-05 20:00:08.476500", "stderr": "", "stdout": "987831.04 3614257.52", "stdout_lines": ["987831.04 3614257.52"], "warnings": []}

PLAY RECAP ************************************************************************************************************************************************************************************
a                          : ok=4    changed=4    unreachable=0    failed=0   
b                          : ok=4    changed=4    unreachable=0    failed=0   
```



Discussion:
Sorry to say, but it seems to work properly.
I can not see the place, where you would limit or delegate the execution.
You may misunderstood the concept.

Please play with the following:
```

---
- hosts: all
  gather_facts: no
  tasks:
    - debug: msg="{{ item }}"
      with_items: [1,2]

    - include_role: name=p
     with_items: "{{ play_hosts }}"
     #with_items: [1,2]


roles/p/tasks/main.yml::
- debug: msg="{{ item }} {{ inventory_hostname }}"

- command: cat /proc/uptime
  when: item == inventory_hostname

```

The 'when: item == inventory_hostname' would work for you in this example, but please note that this would expect the 'item' variable every time you use this role.
You may rethink your concept, and use a delegate but please remember about KISS (https://en.wikipedia.org/wiki/KISS_principle)
Hi,

I was about to report the same behavior yesterday before noticing this issue. It seems surprising to me, and it certainly behaves very differently than a task include:

```
# roles/example/tasks/example.yml
---
    -   name: "describe {{ included_variable }} with included task"
        debug: msg="{{ included_variable }} {{ ansible_play_batch }}"


# roles/example/tasks/main.yml
---
    -   include_role:
            name: included
            allow_duplicates: no
        with_items:
            -   example role 1
        loop_control:
            loop_var: included_variable

    -   include: example.yml
        vars:
            included_variable: "{{ item }}"
        with_items:
            -   example task 1


# roles/included/tasks/main.yml
---
    -   name: "describe {{ included_variable }} with included role"
        debug: msg="{{ included_variable }} {{ ansible_play_batch }}"


# site.yml
---
    -   hosts: all

        connection: local
        gather_facts: no

        roles:
            -   example


# inventory.ini
local_a ansible_host=127.0.0.1
local_b ansible_host=127.0.0.1
local_c ansible_host=127.0.0.1


# ansible-playbook -i inventory.ini site.yml
PLAY [all] **************************************************************************************************************************************************************************

TASK [example : include_role] *******************************************************************************************************************************************************

TASK [included : describe example role 1 with included role] ************************************************************************************************************************
ok: [local_b] => {
    "msg": "example role 1 [u'local_a', u'local_b', u'local_c']"
}
ok: [local_a] => {
    "msg": "example role 1 [u'local_a', u'local_b', u'local_c']"
}
ok: [local_c] => {
    "msg": "example role 1 [u'local_a', u'local_b', u'local_c']"
}

TASK [included : describe example role 1 with included role] ************************************************************************************************************************
ok: [local_a] => {
    "msg": "example role 1 [u'local_a', u'local_b', u'local_c']"
}
ok: [local_b] => {
    "msg": "example role 1 [u'local_a', u'local_b', u'local_c']"
}
ok: [local_c] => {
    "msg": "example role 1 [u'local_a', u'local_b', u'local_c']"
}

TASK [included : describe example role 1 with included role] ************************************************************************************************************************
ok: [local_a] => {
    "msg": "example role 1 [u'local_a', u'local_b', u'local_c']"
}
ok: [local_b] => {
    "msg": "example role 1 [u'local_a', u'local_b', u'local_c']"
}
ok: [local_c] => {
    "msg": "example role 1 [u'local_a', u'local_b', u'local_c']"
}

TASK [example : include] ************************************************************************************************************************************************************
included: /Users/francesco/Development/ansible_include_role/roles/example/tasks/example.yml for local_b, local_a, local_c

TASK [example : describe example task 1 with included task] *************************************************************************************************************************
ok: [local_a] => {
    "msg": "example task 1 [u'local_a', u'local_b', u'local_c']"
}
ok: [local_b] => {
    "msg": "example task 1 [u'local_a', u'local_b', u'local_c']"
}
ok: [local_c] => {
    "msg": "example task 1 [u'local_a', u'local_b', u'local_c']"
}

PLAY RECAP **************************************************************************************************************************************************************************
local_a                    : ok=5    changed=0    unreachable=0    failed=0   
local_b                    : ok=5    changed=0    unreachable=0    failed=0   
local_c                    : ok=5    changed=0    unreachable=0    failed=0   


```

Am I missing something, perhaps in the documentation? Why would someone include the very same role in each host, one time for every host included in the playbook?
Same problem for me. `allow_duplicates: False `did not work. 
This is also here: http://stackoverflow.com/questions/42171103/ansible-include-role-on-a-loop-running-unexpected-number-of-times

I've set `serial: 1` on the play and it did help (as a workaround). 
Thanks @callipeo's  example code.
There is another workaround that you can just replace `include_role` with `include`.
For example, you can replace

```
  tasks:
    - name: Create users
      include_role:
        name: add_user
      vars:
        add_user:
          name: "{{ item.key }}"
          password:
          key: "{{ item.value.key }}"
          is_admin: no
      with_dict: "{{ users }}"
```

with

```
  tasks:
    - name: Create users
      include: roles/add_user/tasks/main.yml
      vars:
        add_user:
          name: "{{ item.key }}"
          password:
          key: "{{ item.value.key }}"
          is_admin: no
      with_dict: "{{ users }}"
```
it appears that with an inventory like:
```
[all]
host1
host2
host3
host4
```
You can avoid the looping by setting the `include_role` as static:
```
---
- name: Include some other role
  include_role:
    name: other_role
  static: yes
```

I don't really understand _why_ this fixes it, but it stopped us from looping in our setup. It may be related to the fact that setting an include as static doesn't allow you to loop on it with things like `with_items`, etc.
Still exists on devel (2.4)

include_role with loop executes num_hosts*num_items times on *each* host

only thing that helps - serial: 1
Looks like we need to get `import_role` implemented.

Resolving patch:
diff --git a/lib/ansible/playbook/included_file.py b/lib/ansible/playbook/included_file.py
--- a/lib/ansible/playbook/included_file.py
+++ b/lib/ansible/playbook/included_file.py
@@ -34,11 +34,12 @@
 
 class IncludedFile:
 
-    def __init__(self, filename, args, task):
+    def __init__(self, filename, args, task, is_role=False):
         self._filename = filename
         self._args = args
         self._task = task
         self._hosts = []
+        self._is_role = is_role
 
     def add_host(self, host):
         if host not in self._hosts:
@@ -65,7 +66,7 @@ def get_original_host(host):
             original_host = res._host
             original_task = res._task
 
-            if original_task.action in ('include', 'include_tasks'):
+            if original_task.action in ('include', 'include_tasks', 'include_role'):
                 if original_task.loop:
                     if 'results' not in res._result:
                         continue
@@ -75,7 +76,7 @@ def get_original_host(host):
 
                 for include_result in include_results:
                     # if the task result was skipped or failed, continue
-                    if 'skipped' in include_result and include_result['skipped'] or 'failed' in include_result:
+                    if 'skipped' in include_result and include_result['skipped'] or 'failed' in include_result and include_result['failed']:
                         continue
 
                     task_vars = variable_manager.get_vars(play=iterator._play, host=original_host, task=original_task)
@@ -88,49 +89,52 @@ def get_original_host(host):
                     if loop_var in include_result:
                         task_vars[loop_var] = include_variables[loop_var] = include_result[loop_var]
 
-                    include_file = None
-                    if original_task:
-                        if original_task.static:
-                            continue
-
-                        if original_task._parent:
-                            # handle relative includes by walking up the list of parent include
-                            # tasks and checking the relative result to see if it exists
-                            parent_include = original_task._parent
-                            cumulative_path = None
-                            while parent_include is not None:
-                                if not isinstance(parent_include, TaskInclude):
-                                    parent_include = parent_include._parent
-                                    continue
-                                if isinstance(parent_include, IncludeRole):
-                                    parent_include_dir = parent_include._role_path
-                                else:
-                                    parent_include_dir = os.path.dirname(templar.template(parent_include.args.get('_raw_params')))
-                                if cumulative_path is not None and not os.path.isabs(cumulative_path):
-                                    cumulative_path = os.path.join(parent_include_dir, cumulative_path)
-                                else:
-                                    cumulative_path = parent_include_dir
+                    if original_task.action in ('include', 'include_tasks'):
+                        include_file = None
+                        if original_task:
+                            if original_task.static:
+                                continue
+
+                            if original_task._parent:
+                                # handle relative includes by walking up the list of parent include
+                                # tasks and checking the relative result to see if it exists
+                                parent_include = original_task._parent
+                                cumulative_path = None
+                                while parent_include is not None:
+                                    if not isinstance(parent_include, TaskInclude):
+                                        parent_include = parent_include._parent
+                                        continue
+                                    if isinstance(parent_include, IncludeRole):
+                                        parent_include_dir = parent_include._role_path
+                                    else:
+                                        parent_include_dir = os.path.dirname(templar.template(parent_include.args.get('_raw_params')))
+                                    if cumulative_path is not None and not os.path.isabs(cumulative_path):
+                                        cumulative_path = os.path.join(parent_include_dir, cumulative_path)
+                                    else:
+                                        cumulative_path = parent_include_dir
+                                    include_target = templar.template(include_result['include'])
+                                    if original_task._role:
+                                        new_basedir = os.path.join(original_task._role._role_path, 'tasks', cumulative_path)
+                                        include_file = loader.path_dwim_relative(new_basedir, 'tasks', include_target)
+                                    else:
+                                        include_file = loader.path_dwim_relative(loader.get_basedir(), cumulative_path, include_target)
+
+                                    if os.path.exists(include_file):
+                                        break
+                                    else:
+                                        parent_include = parent_include._parent
+
+                        if include_file is None:
+                            if original_task._role:
                                 include_target = templar.template(include_result['include'])
-                                if original_task._role:
-                                    new_basedir = os.path.join(original_task._role._role_path, 'tasks', cumulative_path)
-                                    include_file = loader.path_dwim_relative(new_basedir, 'tasks', include_target)
-                                else:
-                                    include_file = loader.path_dwim_relative(loader.get_basedir(), cumulative_path, include_target)
-
-                                if os.path.exists(include_file):
-                                    break
-                                else:
-                                    parent_include = parent_include._parent
-
-                    if include_file is None:
-                        if original_task._role:
-                            include_target = templar.template(include_result['include'])
-                            include_file = loader.path_dwim_relative(original_task._role._role_path, 'tasks', include_target)
-                        else:
-                            include_file = loader.path_dwim(include_result['include'])
-
-                    include_file = templar.template(include_file)
-                    inc_file = IncludedFile(include_file, include_variables, original_task)
+                                include_file = loader.path_dwim_relative(original_task._role._role_path, 'tasks', include_target)
+                            else:
+                                include_file = loader.path_dwim(include_result['include'])
+
+                        include_file = templar.template(include_file)
+                        inc_file = IncludedFile(include_file, include_variables, original_task)
+                    else:
+                        inc_file = IncludedFile("role", include_variables, original_task, is_role=True)
 
                     try:
                         pos = included_files.index(inc_file)
diff --git a/lib/ansible/plugins/strategy/free.py b/lib/ansible/plugins/strategy/free.py
--- a/lib/ansible/plugins/strategy/free.py
+++ b/lib/ansible/plugins/strategy/free.py
@@ -193,7 +193,18 @@ def run(self, iterator, play_context):
                 for included_file in included_files:
                     display.debug("collecting new blocks for %s" % included_file)
                     try:
-                        new_blocks = self._load_included_file(included_file, iterator=iterator)
+                        if included_file._is_role:
+                            new_ir = included_file._task.copy()
+                            new_ir.vars.update(included_file._args)
+
+                            new_blocks, handler_blocks = new_ir.get_block_list(
+                                play=iterator._play,
+                                variable_manager=self._variable_manager,
+                                loader=self._loader,
+                            )
+                            self._tqm.update_handler_list([handler for handler_block in handler_blocks for handler in handler_block.block])
+                        else:
+                            new_blocks = self._load_included_file(included_file, iterator=iterator)
                     except AnsibleError as e:
                         for host in included_file._hosts:
                             iterator.mark_host_failed(host)
diff --git a/lib/ansible/plugins/strategy/linear.py b/lib/ansible/plugins/strategy/linear.py
--- a/lib/ansible/plugins/strategy/linear.py
+++ b/lib/ansible/plugins/strategy/linear.py
@@ -290,38 +290,8 @@ def run(self, iterator, play_context):
                 display.debug("done queuing things up, now waiting for results queue to drain")
                 if self._pending_results > 0:
                     results += self._wait_on_pending_results(iterator)
-                host_results.extend(results)
 
-                all_role_blocks = []
-                for hr in results:
-                    # handle include_role
-                    if hr._task.action == 'include_role':
-                        loop_var = None
-                        if hr._task.loop:
-                            loop_var = 'item'
-                            if hr._task.loop_control:
-                                loop_var = hr._task.loop_control.loop_var or 'item'
-                            include_results = hr._result.get('results', [])
-                        else:
-                            include_results = [hr._result]
-
-                        for include_result in include_results:
-                            if 'skipped' in include_result and include_result['skipped'] or 'failed' in include_result and include_result['failed']:
-                                continue
-
-                            display.debug("generating all_blocks data for role")
-                            new_ir = hr._task.copy()
-                            new_ir.vars.update(include_result.get('include_variables', dict()))
-                            if loop_var and loop_var in include_result:
-                                new_ir.vars[loop_var] = include_result[loop_var]
-
-                            blocks, handler_blocks = new_ir.get_block_list(play=iterator._play, variable_manager=self._variable_manager, loader=self._loader)
-                            all_role_blocks.extend(blocks)
-                            self._tqm.update_handler_list([handler for handler_block in handler_blocks for handler in handler_block.block])
-
-                if len(all_role_blocks) > 0:
-                    for host in hosts_left:
-                        iterator.add_tasks(host, all_role_blocks)
+                host_results.extend(results)
 
                 try:
                     included_files = IncludedFile.process_include_results(
@@ -339,6 +309,8 @@ def run(self, iterator, play_context):
                 include_failure = False
                 if len(included_files) > 0:
                     display.debug("we have included files to process")
+
+                    # A noop task for use in padding dynamic includes
                     noop_task = Task()
                     noop_task.action = 'meta'
                     noop_task.args['_raw_params'] = 'noop'
@@ -352,7 +324,18 @@ def run(self, iterator, play_context):
                         # included hosts get the task list while those excluded get an equal-length
                         # list of noop tasks, to make sure that they continue running in lock-step
                         try:
-                            new_blocks = self._load_included_file(included_file, iterator=iterator)
+                            if included_file._is_role:
+                                new_ir = included_file._task.copy()
+                                new_ir.vars.update(included_file._args)
+
+                                new_blocks, handler_blocks = new_ir.get_block_list(
+                                    play=iterator._play,
+                                    variable_manager=self._variable_manager,
+                                    loader=self._loader,
+                                )
+                                self._tqm.update_handler_list([handler for handler_block in handler_blocks for handler in handler_block.block])
+                            else:
+                                new_blocks = self._load_included_file(included_file, iterator=iterator)
 
                             display.debug("iterating over new_blocks loaded from include file")
                             for new_block in new_blocks:


Test patch: