Repository: ansible/ansible

Issue:
Ansible v2 (and devel) hangs
##### Issue Type: Bug Report
##### Ansible Version:

ansible 2.1.0 (devel cd3cfa93bf) last updated 2016/01/05 11:51:41 (GMT +200)
  lib/ansible/modules/core: (detached HEAD 002028748f) last updated 2016/01/04 16:51:50 (GMT +200)
  lib/ansible/modules/extras: (detached HEAD f6a7b6dd1f) last updated 2016/01/04 16:51:50 (GMT +200)
  config file = /home/tkontusz/int.rd-config/ansible.cfg
  configured module search path = Default w/o overrides

Also in stable-2.0
##### Ansible Configuration:

```
[defaults]
jinja2_extensions = jinja2.ext.do
ansible_managed = Managed with Ansible
roles_path = ./ext_roles
nocows = 1

[ssh_connection]
pipelining = True
```
##### Environment: Ubuntu Trusyt on host, Debian wheezy/Ubuntu Trusty as target
##### Summary:

Running a specific playbook hangs when the implicit flush_handlers should run. This is the end of ANSIBLE_DEBUG log before killing Ansible (it loops anyway):

```
 30740 1451991428.06899: counting tasks in each state of execution
 30740 1451991428.06903: done counting tasks in each state of execution
 30740 1451991428.06906: advancing hosts in ITERATING_TASKS
 30740 1451991428.06909: starting to advance hosts
 30740 1451991428.06913: done advancing hosts to next task
 30740 1451991428.06935: done queuing things up, now waiting for results queue to drain
 30740 1451991428.06940: waiting for pending results...
 30740 1451991428.06943: no more pending results, returning what we have
 30740 1451991428.06946: results queue empty
 30740 1451991428.06950: getting the remaining hosts for this loop
 30740 1451991428.06957: done getting the remaining hosts for this loop
 30740 1451991428.06975: building list of next tasks for hosts
 30740 1451991428.06981: getting the next task for host psql01.rapidstg01.rdev
 30740 1451991428.06987: done getting next task for host psql01.rapidstg01.rdev
 30740 1451991428.06992:  ^ task is: TASK: meta
 30740 1451991428.06997:  ^ state is: HOST STATE: block=2, task=1, rescue=0, always=0, role=container, run_state=ITERATING_TASKS, fail_state=FAILED_NONE, pending_setup=False, tasks child state? None, rescue child state? None, always child state? None
 30740 1451991428.07003: getting the next task for host gluster01.rapidstg01.rdev
 30740 1451991428.07009: done getting next task for host gluster01.rapidstg01.rdev
 30740 1451991428.07014:  ^ task is: TASK: meta
 30740 1451991428.07021:  ^ state is: HOST STATE: block=1, task=9, rescue=0, always=0, role=container, run_state=ITERATING_RESCUE, fail_state=FAILED_TASKS, pending_setup=False, tasks child state? None, rescue child state? HOST STATE: block=0, task=1, rescue=0, always=0, role=container, run_state=ITERATING_TASKS, fail_state=FAILED_NONE, pending_setup=False, tasks child state? None, rescue child state? None, always child state? None, always child state? None
 30740 1451991428.07027: getting the next task for host proxy01.rapidstg01.rdev
 30740 1451991428.07033: done getting next task for host proxy01.rapidstg01.rdev
 30740 1451991428.07038:  ^ task is: TASK: meta
 30740 1451991428.07044:  ^ state is: HOST STATE: block=2, task=1, rescue=0, always=0, role=container, run_state=ITERATING_TASKS, fail_state=FAILED_NONE, pending_setup=False, tasks child state? None, rescue child state? None, always child state? None
 30740 1451991428.07049: getting the next task for host mail01.rapidstg01.rdev
 30740 1451991428.07055: done getting next task for host mail01.rapidstg01.rdev
 30740 1451991428.07060:  ^ task is: TASK: meta
 30740 1451991428.07068:  ^ state is: HOST STATE: block=1, task=9, rescue=0, always=0, role=container, run_state=ITERATING_RESCUE, fail_state=FAILED_TASKS, pending_setup=False, tasks child state? None, rescue child state? HOST STATE: block=0, task=1, rescue=0, always=0, role=container, run_state=ITERATING_TASKS, fail_state=FAILED_NONE, pending_setup=False, tasks child state? None, rescue child state? None, always child state? None, always child state? None
 30740 1451991428.07073: getting the next task for host monitor01.rapidstg01.rdev
 30740 1451991428.07079: done getting next task for host monitor01.rapidstg01.rdev
 30740 1451991428.07084:  ^ task is: TASK: meta
 30740 1451991428.07092:  ^ state is: HOST STATE: block=1, task=2, rescue=0, always=0, role=container, run_state=ITERATING_RESCUE, fail_state=FAILED_TASKS, pending_setup=False, tasks child state? None, rescue child state? HOST STATE: block=0, task=1, rescue=0, always=0, role=container, run_state=ITERATING_TASKS, fail_state=FAILED_NONE, pending_setup=False, tasks child state? None, rescue child state? None, always child state? None, always child state? None
 30740 1451991428.07097: getting the next task for host extproxy01.rapidstg01.rdev
 30740 1451991428.07104: done getting next task for host extproxy01.rapidstg01.rdev
 30740 1451991428.07109:  ^ task is: TASK: meta
 30740 1451991428.07115:  ^ state is: HOST STATE: block=2, task=1, rescue=0, always=0, role=container, run_state=ITERATING_TASKS, fail_state=FAILED_NONE, pending_setup=False, tasks child state? None, rescue child state? None, always child state? None
 30740 1451991428.07118: done building task lists
 30740 1451991428.07122: counting tasks in each state of execution
 30740 1451991428.07126: done counting tasks in each state of execution
 30740 1451991428.07129: advancing hosts in ITERATING_TASKS
 30740 1451991428.07132: starting to advance hosts
 30740 1451991428.07136: done advancing hosts to next task
 30740 1451991428.07158: done queuing things up, now waiting for results queue to drain
 30740 1451991428.07162: waiting for pending results...
 30740 1451991428.07166: no more pending results, returning what we have
 30740 1451991428.07169: results queue empty
 30740 1451991428.07172: getting the remaining hosts for this loop
 30740 1451991428.07179: done getting the remaining hosts for this loop
 30740 1451991428.07197: building list of next tasks for hosts
 30740 1451991428.07203: getting the next task for host psql01.rapidstg01.rdev
 30740 1451991428.07209: done getting next task for host psql01.rapidstg01.rdev
 30740 1451991428.07214:  ^ task is: TASK: meta
 30740 1451991428.07220:  ^ state is: HOST STATE: block=2, task=1, rescue=0, always=0, role=container, run_state=ITERATING_TASKS, fail_state=FAILED_NONE, pending_setup=False, tasks child state? None, rescue child state? None, always child state? None
 30740 1451991428.07225: getting the next task for host gluster01.rapidstg01.rdev
 30740 1451991428.07231: done getting next task for host gluster01.rapidstg01.rdev
 30740 1451991428.07236:  ^ task is: TASK: meta
 30740 1451991428.07244:  ^ state is: HOST STATE: block=1, task=9, rescue=0, always=0, role=container, run_state=ITERATING_RESCUE, fail_state=FAILED_TASKS, pending_setup=False, tasks child state? None, rescue child state? HOST STATE: block=0, task=1, rescue=0, always=0, role=container, run_state=ITERATING_TASKS, fail_state=FAILED_NONE, pending_setup=False, tasks child state? None, rescue child state? None, always child state? None, always child state? None
 30740 1451991428.07249: getting the next task for host proxy01.rapidstg01.rdev
 30740 1451991428.07255: done getting next task for host proxy01.rapidstg01.rdev
 30740 1451991428.07260:  ^ task is: TASK: meta
 30740 1451991428.07266:  ^ state is: HOST STATE: block=2, task=1, rescue=0, always=0, role=container, run_state=ITERATING_TASKS, fail_state=FAILED_NONE, pending_setup=False, tasks child state? None, rescue child state? None, always child state? None
 30740 1451991428.07271: getting the next task for host mail01.rapidstg01.rdev
 30740 1451991428.07278: done getting next task for host mail01.rapidstg01.rdev
 30740 1451991428.07283:  ^ task is: TASK: meta
 30740 1451991428.07290:  ^ state is: HOST STATE: block=1, task=9, rescue=0, always=0, role=container, run_state=ITERATING_RESCUE, fail_state=FAILED_TASKS, pending_setup=False, tasks child state? None, rescue child state? HOST STATE: block=0, task=1, rescue=0, always=0, role=container, run_state=ITERATING_TASKS, fail_state=FAILED_NONE, pending_setup=False, tasks child state? None, rescue child state? None, always child state? None, always child state? None
 30740 1451991428.07296: getting the next task for host monitor01.rapidstg01.rdev
 30740 1451991428.07302: done getting next task for host monitor01.rapidstg01.rdev
 30740 1451991428.07307:  ^ task is: TASK: meta
 30740 1451991428.07315:  ^ state is: HOST STATE: block=1, task=2, rescue=0, always=0, role=container, run_state=ITERATING_RESCUE, fail_state=FAILED_TASKS, pending_setup=False, tasks child state? None, rescue child state? HOST STATE: block=0, task=1, rescue=0, always=0, role=container, run_state=ITERATING_TASKS, fail_state=FAILED_NONE, pending_setup=False, tasks child state? None, rescue child state? None, always child state? None, always child state? None
 30740 1451991428.07320: getting the next task for host extproxy01.rapidstg01.rdev
 30740 1451991428.07326: done getting next task for host extproxy01.rapidstg01.rdev
 30740 1451991428.07331:  ^ task is: TASK: meta
 30740 1451991428.07336:  ^ state is: HOST STATE: block=2, task=1, rescue=0, always=0, role=container, run_state=ITERATING_TASKS, fail_state=FAILED_NONE, pending_setup=False, tasks child state? None, rescue child state? None, always child state? None
 30740 1451991428.07341: done building task lists
```
##### Steps To Reproduce:

I'm working on this, as reducing the problem seems to be hard - I couldn't build a failing playbook from scratch so far, so I'll have to reduce a proprietary one until it's small enough to paste here.
##### Expected Results:

The playbook running to completion.
##### Actual Results:

The whole thing eating 100% of one CPU.



Discussion:
It looks like delegation and include (next to each other, not delegate inside an include) have to be used to reproduce this.

Ok, I can reproduce it now. Run `ANSIBLE_DEBUG=1 ansible-playbook --diff -i local_inv -c local --check site.yml` with those files:

local_inv

```
d
[containers]
a
b
c

[containers:vars]
delegate_host=d
other_var=blah

[all:vars]
ansible_connection=local
```

site.yml

```
- hosts: containers
  tasks:
    - name: Delegated
      shell: '[ "{{ inventory_hostname }}" != "{{ play_hosts[1] }}" ]'
      always_run: yes
      delegate_to: '{{ delegate_host }}'

    - include: other_file.yml

    - name: Dummy
      shell: 'true'
      when: false
  post_tasks:
    # This is never reached
    - fail: msg=boom
```

other_file.yml

```
- set_fact: test_var={{ other_var }}
```


Resolving patch:
diff --git a/lib/ansible/plugins/strategy/linear.py b/lib/ansible/plugins/strategy/linear.py
--- a/lib/ansible/plugins/strategy/linear.py
+++ b/lib/ansible/plugins/strategy/linear.py
@@ -62,19 +62,32 @@ def _get_next_task_lockstep(self, hosts, iterator):
         num_rescue = 0
         num_always = 0
 
-        lowest_cur_block = len(iterator._blocks)
-
         display.debug("counting tasks in each state of execution")
-        for (k, v) in iteritems(host_tasks):
-            if v is None:
-                continue
-
+        host_tasks_to_run = [(host, state_task)
+                             for host, state_task in iteritems(host_tasks)
+                             if state_task and state_task[1]]
+        # Drop noops
+        host_tasks_to_run = [
+            (host, (state, task))
+            for host, (state, task) in host_tasks_to_run
+            if task.action != 'meta' or task.args.get('_raw_params') != 'noop'
+        ]
+
+        if host_tasks_to_run:
+            lowest_cur_block = min(
+                (s.cur_block for h, (s, t) in host_tasks_to_run
+                if s.run_state != PlayIterator.ITERATING_COMPLETE))
+        else:
+            # empty host_tasks_to_run will just run till the end of the function
+            # without ever touching lowest_cur_block
+            lowest_cur_block = None
+
+        for (k, v) in host_tasks_to_run:
             (s, t) = v
-            if t is None:
-                continue
 
-            if s.cur_block < lowest_cur_block and s.run_state != PlayIterator.ITERATING_COMPLETE:
-                lowest_cur_block = s.cur_block
+            if s.cur_block > lowest_cur_block:
+                # Not the current block, ignore it
+                continue
 
             if s.run_state == PlayIterator.ITERATING_SETUP:
                 num_setups += 1


Test patch: