Repository: ansible/ansible

Issue:
max_fail_percentage incorrectly aborts playbook execution
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
max_fail_percentage

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.5.0 (devel 7553c42e09) last updated 2017/10/27 08:35:00 (GMT -400)
  config file = /home/rteague/git/clusters/aws-c1/ansible.cfg
  configured module search path = [u'/home/rteague/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/rteague/git/clusters/aws-c1/ansible/lib/ansible
  executable location = /home/rteague/git/clusters/aws-c1/ansible/bin/ansible
  python version = 2.7.13 (default, Sep  5 2017, 08:53:59) [GCC 7.1.1 20170622 (Red Hat 7.1.1-3)]

```
**NOTE: This issue exists back to v2.1.0.0-1**
I tested against older version and found:
v2.0.2.0-1 Good
v2.1.0.0-1 Bad

##### CONFIGURATION
<!---
If using Ansible 2.4 or above, paste the results of "ansible-config dump --only-changed"

Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).

-->
N/A

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->
N/A

##### SUMMARY
<!--- Explain the problem briefly -->
max_fail_percentage incorrectly aborts playbook execution after single host failure in a batch

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
# test-fail.yml
- hosts: nodes
  gather_facts: no
  serial: "{{ nodes_serial | default(1) }}"
  max_fail_percentage: "{{ nodes_max_fail_percentage | default(0) }}"

  tasks:
  - debug:
      var: inventory_hostname
    failed_when: inventory_hostname == 'host2'

  - debug:
      msg: "Next task"

  - debug:
      msg: "Last task"

```
```
# test-inv
[nodes]
host[1:40] ansible_host=127.0.0.1

```
```
# Command
ansible-playbook -i test-inv test-fail.yml -e nodes_serial=4 -e nodes_max_fail_percentage=30
```
<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
The expectation is that the playbook will continue running all batches unless a single batch has more than one failure.  (One failure = 25%, Two failures = 50%)

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
The playbook exited after the `Next task` after failure and did not process any further batches.

If setting mfp to 24, the playbook will exit after the first failed task in the first batch.  Correct.
If setting mfp to 33, the playbook will exit after the *next* task (only three hosts) in the first batch. Incorrect.
If setting mfp to 34, the playbook will exit after *all* tasks in that batch.  Incorrect.
(Note that there appears to be some impact to task execution because changing mfp between 33 to 34 has some effect, even though four hosts are run in this batch.  It should only be the 24/25 boundary to effect a single host failure on a four host batch.
If setting mfp to 99, the playbook will exit after all tasks in that batch.  Incorrect.
If setting mfp to 100, the playbook will continue through all tasks for all batches.  Correct.


<!--- Paste verbatim command output between quotes below -->
```
$ ansible-playbook -i test-inv test-fail.yml -e nodes_serial=4 -e nodes_max_fail_percentage=30 

PLAY [nodes] *********************************************************************************************************************************

TASK [debug] *********************************************************************************************************************************
ok: [host1] => {
    "failed_when_result": false, 
    "inventory_hostname": "host1"
}
ok: [host3] => {
    "failed_when_result": false, 
    "inventory_hostname": "host3"
}
fatal: [host2]: FAILED! => {
    "changed": false, 
    "failed_when_result": true, 
    "inventory_hostname": "host2"
}
ok: [host4] => {
    "failed_when_result": false, 
    "inventory_hostname": "host4"
}

TASK [debug] *********************************************************************************************************************************
ok: [host1] => {
    "msg": "Next task"
}
ok: [host3] => {
    "msg": "Next task"
}
ok: [host4] => {
    "msg": "Next task"
}

NO MORE HOSTS LEFT ***************************************************************************************************************************

NO MORE HOSTS LEFT ***************************************************************************************************************************
	to retry, use: --limit @/home/rteague/git/clusters/aws-c1/test-fail.retry

PLAY RECAP ***********************************************************************************************************************************
host1                      : ok=2    changed=0    unreachable=0    failed=0   
host2                      : ok=0    changed=0    unreachable=0    failed=1   
host3                      : ok=2    changed=0    unreachable=0    failed=0   
host4                      : ok=2    changed=0    unreachable=0    failed=0   

```




Discussion:
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1504075
Running `git bisect` found 9602e439525e5e14b1e062e18bbdee3b47af8248:
```
9602e439525e5e14b1e062e18bbdee3b47af8248 is the first bad commit
commit 9602e439525e5e14b1e062e18bbdee3b47af8248
Author: Vincent Roy <vincentroy8@gmail.com>
Date:   Tue Apr 12 22:36:08 2016 -0300

    Don't stop executing plays after failure.
    
    https://github.com/ansible/ansible/pull/13750/files

:040000 040000 e788d7b583c0b8c77af417c649ffb2d011560c4a 6d18e95e5da6c429cd8022a7a808be9f4c4561d5 M	lib
bisect run success
```

This may only be part of the overall problem.

Resolving patch:
diff --git a/lib/ansible/executor/play_iterator.py b/lib/ansible/executor/play_iterator.py
--- a/lib/ansible/executor/play_iterator.py
+++ b/lib/ansible/executor/play_iterator.py
@@ -203,7 +203,9 @@ def __init__(self, inventory, play, play_context, variable_manager, all_vars, st
 
         self._host_states = {}
         start_at_matched = False
-        for host in inventory.get_hosts(self._play.hosts):
+        batch = inventory.get_hosts(self._play.hosts)
+        self.batch_size = len(batch)
+        for host in batch:
             self._host_states[host.name] = HostState(blocks=self._blocks)
             # if we're looking to start at a specific task, iterate through
             # the tasks for this host until we find the specified task
diff --git a/lib/ansible/executor/task_queue_manager.py b/lib/ansible/executor/task_queue_manager.py
--- a/lib/ansible/executor/task_queue_manager.py
+++ b/lib/ansible/executor/task_queue_manager.py
@@ -242,22 +242,6 @@ def run(self, play):
             loader=self._loader,
         )
 
-        # Fork # of forks, # of hosts or serial, whichever is lowest
-        num_hosts = len(self._inventory.get_hosts(new_play.hosts, ignore_restrictions=True))
-
-        max_serial = 0
-        if new_play.serial:
-            # the play has not been post_validated here, so we may need
-            # to convert the scalar value to a list at this point
-            serial_items = new_play.serial
-            if not isinstance(serial_items, list):
-                serial_items = [serial_items]
-            max_serial = max([pct_to_int(x, num_hosts) for x in serial_items])
-
-        contenders = [self._options.forks, max_serial, num_hosts]
-        contenders = [v for v in contenders if v is not None and v > 0]
-        self._initialize_processes(min(contenders))
-
         play_context = PlayContext(new_play, self._options, self.passwords, self._connection_lockfile.fileno())
         for callback_plugin in self._callback_plugins:
             if hasattr(callback_plugin, 'set_play_context'):
@@ -268,11 +252,6 @@ def run(self, play):
         # initialize the shared dictionary containing the notified handlers
         self._initialize_notified_handlers(new_play)
 
-        # load the specified strategy (or the default linear one)
-        strategy = strategy_loader.get(new_play.strategy, self)
-        if strategy is None:
-            raise AnsibleError("Invalid play strategy specified: %s" % new_play.strategy, obj=play._ds)
-
         # build the iterator
         iterator = PlayIterator(
             inventory=self._inventory,
@@ -283,6 +262,14 @@ def run(self, play):
             start_at_done=self._start_at_done,
         )
 
+        # adjust to # of workers to configured forks or size of batch, whatever is lower
+        self._initialize_processes(min(self._options.forks, iterator.batch_size))
+
+        # load the specified strategy (or the default linear one)
+        strategy = strategy_loader.get(new_play.strategy, self)
+        if strategy is None:
+            raise AnsibleError("Invalid play strategy specified: %s" % new_play.strategy, obj=play._ds)
+
         # Because the TQM may survive multiple play runs, we start by marking
         # any hosts as failed in the iterator here which may have been marked
         # as failed in previous runs. Then we clear the internal list of failed
diff --git a/lib/ansible/plugins/strategy/linear.py b/lib/ansible/plugins/strategy/linear.py
--- a/lib/ansible/plugins/strategy/linear.py
+++ b/lib/ansible/plugins/strategy/linear.py
@@ -401,7 +401,7 @@ def run(self, iterator, play_context):
                 if iterator._play.max_fail_percentage is not None and len(results) > 0:
                     percentage = iterator._play.max_fail_percentage / 100.0
 
-                    if (len(self._tqm._failed_hosts) / len(results)) > percentage:
+                    if (len(self._tqm._failed_hosts) / iterator.batch_size) > percentage:
                         for host in hosts_left:
                             # don't double-mark hosts, or the iterator will potentially
                             # fail them out of the rescue/always states
@@ -410,6 +410,7 @@ def run(self, iterator, play_context):
                                 iterator.mark_host_failed(host)
                         self._tqm.send_callback('v2_playbook_on_no_hosts_remaining')
                         result |= self._tqm.RUN_FAILED_BREAK_PLAY
+                    display.debug('(%s failed / %s total )> %s max fail' % (len(self._tqm._failed_hosts), iterator.batch_size, percentage))
                 display.debug("done checking for max_fail_percentage")
 
                 display.debug("checking to see if all hosts have failed and the running result is not ok")


Test patch: