Repository: ansible/ansible

Issue:
In include/import/nested blocks, `rescue` tasks are mislabeled when a `block` does not fail on all hosts
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest -->
 - Bug Report

##### COMPONENT NAME
<!---
Name of the module, plugin, task or feature
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path
-->

`block`/`rescue`

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.4.2.0
  config file = <redacted>
  configured module search path = [u'/home/timo/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/timo/.virtualenvs/ansible/lib/python2.7/site-packages/ansible
  executable location = /home/timo/.virtualenvs/ansible/bin/ansible
  python version = 2.7.14 (default, Sep 20 2017, 01:25:59) [GCC 7.2.0]

```

##### CONFIGURATION
<!---
If using Ansible 2.4 or above, paste the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->

```
ANSIBLE_PIPELINING = True
DEFAULT_CALLBACK_WHITELIST = ['profile_tasks']
DEFAULT_FORKS = 50
```

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->

Arch, but doubt this matters.

##### SUMMARY
<!--- Explain the problem briefly -->

**When running against multiple hosts**, and if a task inside a `block` **does not fail on all hosts**, the tasks executed in a `rescue` block for that host carry the wrong name.


##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

I've tried to summarize in one sentence, but it might not speak to the imagination. Consider the playbook below.

Edit: updated to contain static imports.

<!--- Paste example playbooks or commands between quotes below -->
```yaml
# block_tasks.yml
---
- block:
    - name: fails_on_host_two
      command: /bin/false
      register: reg_cmd
      failed_when: reg_cmd.rc == 1 and inventory_hostname == 'host_two'

    - name: cannot_change
      command: /bin/true
      changed_when: false

  rescue:
    - name: rescue_one
      command: /bin/true
      changed_when: yes

    - name: rescue_two
      command: /bin/true          command: /bin/true

# playbook.yml
---
- hosts:
    - host_one
    - host_two
  tasks:
    - import_tasks: block_tasks.yml
```

This needs to be run against multiple hosts. Make sure the task in the `block` clause can only fail on one of the hosts, so adapt accordingly.

The playbook above should generate these outputs (adapted from what I'm seeing in my codebase):

```
TASK [fails_on_host_two] ******************************************************
ok: [host_one]
fatal: [host_two]: FAILED! => {"changed": false, "msg": "failure condition"}

TASK [cannot_change] ********************************************************** 
ok: [host_one]
changed: [host_two]

TASK [rescue_two] *************************************************************
ok: [host_one]
ok: [host_two]
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->

I'd expect to see `fails_on_host_two` -> `rescue_one` -> `rescue_two` on host_two.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

Let me preface this with the fact that this is merely a cosmetical bug. The control flow behind the scenes is completely correct, but clearly there is missing output logic to deal with `rescue` conditions. (this is confusing in the first place) The runner also fails to catch on to the fact that the changed result displayed from `cannot_change` is actually the result of `rescue_one`, it's just displayed under the wrong task.

Note that `cannot_change` is explicitly marked with `changed_when: no`, yet it generates a changed result for a task that can never change.



Discussion:
Files identified in the description:
None

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
I tried unsuccessfully to reproduce the bug described in this issue using ansible 2.3.2.0 and on ansible 2.5 on Fedora 26.
In both case the output of your example playbook is:

```
$ ansible-playbook /tmp/tmp.yml

PLAY [rhel6,localhost] ***********************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************
ok: [localhost]
ok: [rhel6]

TASK [fails_on_host_two] *********************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["/bin/false"], "delta": "0:00:00.002052", "end": "2018-02-05 23:41:42.656337", "failed": true, "failed_when_result": true, "rc": 1, "start": "2018-02-05 23:41:42.654285", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [rhel6]

TASK [cannot_change] *************************************************************************************************************************************************************************
ok: [rhel6]

TASK [rescue_one] ****************************************************************************************************************************************************************************
changed: [localhost]

TASK [rescue_two] ****************************************************************************************************************************************************************************
changed: [localhost]

PLAY RECAP ***********************************************************************************************************************************************************************************
localhost                  : ok=3    changed=2    unreachable=0    failed=1   
rhel6                      : ok=3    changed=1    unreachable=0    failed=0   

```

I observe "fails_on_host_two -> rescue_one -> rescue_two on host_two" and I'm not observing the execution of task "cannot_change" on the second node(localhost using my inventory).

Have I understand correctly the bug description?
Hi @giovannisciortino, thanks for trying it out. Sorry for this, the playbook I posted was just an interpretation of my production role and I didn't test it. Fortunately, I've been able to narrow it down, it only seems to happen with *static includes*:

- move all tasks from the playbook to a separate file adjacent to the playbook (eg. `my_tasks.yml`)
- insert a task like `import_tasks: my_tasks.yml`

Real output:
```
PLAY [host_one,host_two] ********************************************************************

TASK [Gathering Facts] **********************************************************************
Thursday 08 February 2018  11:57:54 +0100 (0:00:00.076)       0:00:00.076 ***** 
ok: [host_two]
ok: [host_one]

TASK [fails_on_host_two] ********************************************************************
Thursday 08 February 2018  11:57:59 +0100 (0:00:05.076)       0:00:05.152 ***** 
fatal: [host_two]: FAILED! => {"changed": true, "cmd": ["/bin/false"], "delta": "0:00:00.056866", "end": "2018-02-08 10:57:59.766424", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2018-02-08 10:57:59.709558", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [host_one]

TASK [cannot_change] ************************************************************************
Thursday 08 February 2018  11:57:59 +0100 (0:00:00.629)       0:00:05.782 ***** 
changed: [host_two]
ok: [host_one]

TASK [rescue_two] ***************************************************************************
Thursday 08 February 2018  11:58:00 +0100 (0:00:00.470)       0:00:06.253 ***** 
changed: [host_two]

PLAY RECAP **********************************************************************************
host_one    : ok=3    changed=1    unreachable=0    failed=0   
host_two    : ok=3    changed=2    unreachable=0    failed=1   
```

Notice how `rescue_one` is never shown to be executed, the result is displayed under `cannot_change`. Also, I've noticed a change in behaviour, the execution time with the static import is actually lower, because it doesn't wait for `block` to finish on `host_one` before starting to execute the `rescue`.
@ti-mo I tried your reproducer with dynamic include and I am seeing the issue too, so it doesn't seem to be limited only to static.
It doesn't even need to be import/include but nested block in general:
```
- hosts:
    - host1
    - host2
  gather_facts: no
  tasks:
    - block:
       - block:
         - name: task1
           debug: msg='task1'
           failed_when: inventory_hostname == 'host1'

         - name: task2
           debug: msg='task2'

         rescue:
           - name: rescue1
             debug: msg='rescue1'

           - name: rescue2
             debug: msg='rescue2'
```
Unit test for this issue https://gist.github.com/mkrizek/0db8d155da750385e77d7236d4e5a038

Resolving patch:
diff --git a/lib/ansible/plugins/strategy/linear.py b/lib/ansible/plugins/strategy/linear.py
--- a/lib/ansible/plugins/strategy/linear.py
+++ b/lib/ansible/plugins/strategy/linear.py
@@ -113,7 +113,7 @@ def _get_next_task_lockstep(self, hosts, iterator):
         if host_tasks_to_run:
             try:
                 lowest_cur_block = min(
-                    (s.cur_block for h, (s, t) in host_tasks_to_run
+                    (iterator.get_active_state(s).cur_block for h, (s, t) in host_tasks_to_run
                      if s.run_state != PlayIterator.ITERATING_COMPLETE))
             except ValueError:
                 lowest_cur_block = None
@@ -125,6 +125,7 @@ def _get_next_task_lockstep(self, hosts, iterator):
         for (k, v) in host_tasks_to_run:
             (s, t) = v
 
+            s = iterator.get_active_state(s)
             if s.cur_block > lowest_cur_block:
                 # Not the current block, ignore it
                 continue
@@ -158,6 +159,7 @@ def _advance_selected_hosts(hosts, cur_block, cur_state):
                 if host_state_task is None:
                     continue
                 (s, t) = host_state_task
+                s = iterator.get_active_state(s)
                 if t is None:
                     continue
                 if s.run_state == cur_state and s.cur_block == cur_block:


Test patch:
diff --git a/test/units/plugins/strategy/test_strategy_linear.py b/test/units/plugins/strategy/test_strategy_linear.py
new file mode 100644
--- /dev/null
+++ b/test/units/plugins/strategy/test_strategy_linear.py
@@ -0,0 +1,183 @@
+# Copyright (c) 2018 Ansible Project
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+
+# Make coding more python3-ish
+from __future__ import (absolute_import, division, print_function)
+__metaclass__ = type
+
+
+from ansible.compat.tests import unittest
+from ansible.compat.tests.mock import patch, MagicMock
+
+from ansible.executor.play_iterator import PlayIterator
+from ansible.playbook import Playbook
+from ansible.playbook.play_context import PlayContext
+from ansible.plugins.strategy.linear import StrategyModule
+from ansible.executor.task_queue_manager import TaskQueueManager
+
+from units.mock.loader import DictDataLoader
+from units.mock.path import mock_unfrackpath_noop
+
+
+class TestStrategyLinear(unittest.TestCase):
+
+    def setUp(self):
+        pass
+
+    def tearDown(self):
+        pass
+
+    @patch('ansible.playbook.role.definition.unfrackpath', mock_unfrackpath_noop)
+    def test_noop(self):
+        fake_loader = DictDataLoader({
+            "test_play.yml": """
+            - hosts: all
+              gather_facts: no
+              tasks:
+                - block:
+                   - block:
+                     - name: task1
+                       debug: msg='task1'
+                       failed_when: inventory_hostname == 'host01'
+
+                     - name: task2
+                       debug: msg='task2'
+
+                     rescue:
+                       - name: rescue1
+                         debug: msg='rescue1'
+
+                       - name: rescue2
+                         debug: msg='rescue2'
+            """,
+        })
+
+        mock_var_manager = MagicMock()
+        mock_var_manager._fact_cache = dict()
+        mock_var_manager.get_vars.return_value = dict()
+
+        p = Playbook.load('test_play.yml', loader=fake_loader, variable_manager=mock_var_manager)
+
+        hosts = []
+        for i in range(0, 2):
+            host = MagicMock()
+            host.name = host.get_name.return_value = 'host%02d' % i
+            hosts.append(host)
+
+        mock_var_manager._fact_cache['host00'] = dict()
+
+        inventory = MagicMock()
+        inventory.get_hosts.return_value = hosts
+        inventory.filter_hosts.return_value = hosts
+
+        play_context = PlayContext(play=p._entries[0])
+
+        itr = PlayIterator(
+            inventory=inventory,
+            play=p._entries[0],
+            play_context=play_context,
+            variable_manager=mock_var_manager,
+            all_vars=dict(),
+        )
+
+        mock_options = MagicMock()
+        mock_options.module_path = None
+
+        tqm = TaskQueueManager(
+            inventory=inventory,
+            variable_manager=mock_var_manager,
+            loader=fake_loader,
+            options=mock_options,
+            passwords=None,
+        )
+        tqm._initialize_processes(3)
+        strategy = StrategyModule(tqm)
+
+        # implicit meta: flush_handlers
+        hosts_left = strategy.get_hosts_left(itr)
+        hosts_tasks = strategy._get_next_task_lockstep(hosts_left, itr)
+        host1_task = hosts_tasks[0][1]
+        host2_task = hosts_tasks[1][1]
+        self.assertIsNotNone(host1_task)
+        self.assertIsNotNone(host2_task)
+        self.assertEqual(host1_task.action, 'meta')
+        self.assertEqual(host2_task.action, 'meta')
+
+        # debug: task1, debug: task1
+        hosts_left = strategy.get_hosts_left(itr)
+        hosts_tasks = strategy._get_next_task_lockstep(hosts_left, itr)
+        host1_task = hosts_tasks[0][1]
+        host2_task = hosts_tasks[1][1]
+        self.assertIsNotNone(host1_task)
+        self.assertIsNotNone(host2_task)
+        self.assertEqual(host1_task.action, 'debug')
+        self.assertEqual(host2_task.action, 'debug')
+        self.assertEqual(host1_task.name, 'task1')
+        self.assertEqual(host2_task.name, 'task1')
+
+        # mark the second host failed
+        itr.mark_host_failed(hosts[1])
+
+        # debug: task2, meta: noop
+        hosts_left = strategy.get_hosts_left(itr)
+        hosts_tasks = strategy._get_next_task_lockstep(hosts_left, itr)
+        host1_task = hosts_tasks[0][1]
+        host2_task = hosts_tasks[1][1]
+        self.assertIsNotNone(host1_task)
+        self.assertIsNotNone(host2_task)
+        self.assertEqual(host1_task.action, 'debug')
+        self.assertEqual(host2_task.action, 'meta')
+        self.assertEqual(host1_task.name, 'task2')
+        self.assertEqual(host2_task.name, '')
+
+        # meta: noop, debug: rescue1
+        hosts_left = strategy.get_hosts_left(itr)
+        hosts_tasks = strategy._get_next_task_lockstep(hosts_left, itr)
+        host1_task = hosts_tasks[0][1]
+        host2_task = hosts_tasks[1][1]
+        self.assertIsNotNone(host1_task)
+        self.assertIsNotNone(host2_task)
+        self.assertEqual(host1_task.action, 'meta')
+        self.assertEqual(host2_task.action, 'debug')
+        self.assertEqual(host1_task.name, '')
+        self.assertEqual(host2_task.name, 'rescue1')
+
+        # meta: noop, debug: rescue2
+        hosts_left = strategy.get_hosts_left(itr)
+        hosts_tasks = strategy._get_next_task_lockstep(hosts_left, itr)
+        host1_task = hosts_tasks[0][1]
+        host2_task = hosts_tasks[1][1]
+        self.assertIsNotNone(host1_task)
+        self.assertIsNotNone(host2_task)
+        self.assertEqual(host1_task.action, 'meta')
+        self.assertEqual(host2_task.action, 'debug')
+        self.assertEqual(host1_task.name, '')
+        self.assertEqual(host2_task.name, 'rescue2')
+
+        # implicit meta: flush_handlers
+        hosts_left = strategy.get_hosts_left(itr)
+        hosts_tasks = strategy._get_next_task_lockstep(hosts_left, itr)
+        host1_task = hosts_tasks[0][1]
+        host2_task = hosts_tasks[1][1]
+        self.assertIsNotNone(host1_task)
+        self.assertIsNotNone(host2_task)
+        self.assertEqual(host1_task.action, 'meta')
+        self.assertEqual(host2_task.action, 'meta')
+
+        # implicit meta: flush_handlers
+        hosts_left = strategy.get_hosts_left(itr)
+        hosts_tasks = strategy._get_next_task_lockstep(hosts_left, itr)
+        host1_task = hosts_tasks[0][1]
+        host2_task = hosts_tasks[1][1]
+        self.assertIsNotNone(host1_task)
+        self.assertIsNotNone(host2_task)
+        self.assertEqual(host1_task.action, 'meta')
+        self.assertEqual(host2_task.action, 'meta')
+
+        # end of iteration
+        hosts_left = strategy.get_hosts_left(itr)
+        hosts_tasks = strategy._get_next_task_lockstep(hosts_left, itr)
+        host1_task = hosts_tasks[0][1]
+        host2_task = hosts_tasks[1][1]
+        self.assertIsNone(host1_task)
+        self.assertIsNone(host2_task)