Repository: ansible/ansible

Issue:
Failed hosts are re-evaluated in some subsequent play (with serial: 1)
The following playbook consists of two plays. One will have systems randomly fail, the second will serialize over the (remaining) systems. What happens is that the second play is run for _all_ systems, and bails out because one system previously failed.

``` yaml
- name: First play simply fails for a few hosts
  hosts: all
  gather_facts: no

  tasks:
  - action: shell exit $((RANDOM % 2))

- name: Subsequent play runs failed hosts, and bails out
  hosts: all
  gather_facts: no
  serial: 1

  tasks:
  - action: debug msg='As soon as we hit a system that previously fails, bails out.'
```

Here you can see where it fails in two distinct cases:

```
[root@moria ansible]# ansible-playbook -v -c ssh -l localhost,moria test24.yml 

PLAY [First play simply fails for a few hosts] ********************* 

TASK: [shell exit $((RANDOM % 2))] ********************* 
failed: [moria] => {"changed": true, "cmd": "exit $((RANDOM % 2)) ", "delta": "0:00:00.002708", "end": "2012-11-27 17:53:49.378803", "rc": 1, "start": "2012-11-27 17:53:49.376095"}
changed: [localhost] => {"changed": true, "cmd": "exit $((RANDOM % 2)) ", "delta": "0:00:00.002708", "end": "2012-11-27 17:53:49.378803", "rc": 0, "start": "2012-11-27 17:53:49.376095", "stderr": "", "stdout": ""}

PLAY [Subsequent play runs failed hosts, and bails out] ********************* 

TASK: [debug msg='As soon as we hit a system that previously fails, bails out.'] ********************* 
FATAL: no hosts matched or all hosts have already failed -- aborting


PLAY RECAP ********************* 
localhost                      : ok=1    changed=1    unreachable=0    failed=0    
moria                          : ok=0    changed=0    unreachable=0    failed=1  
```

And:

```
[root@moria ansible]# ansible-playbook -v -c ssh -l localhost,moria test24.yml 

PLAY [First play simply fails for a few hosts] ********************* 

TASK: [shell exit $((RANDOM % 2))] ********************* 
changed: [moria] => {"changed": true, "cmd": "exit $((RANDOM % 2)) ", "delta": "0:00:00.002432", "end": "2012-11-27 17:53:46.226772", "rc": 0, "start": "2012-11-27 17:53:46.224340", "stderr": "", "stdout": ""}
failed: [localhost] => {"changed": true, "cmd": "exit $((RANDOM % 2)) ", "delta": "0:00:00.002428", "end": "2012-11-27 17:53:46.226768", "rc": 1, "start": "2012-11-27 17:53:46.224340"}

PLAY [Subsequent play runs failed hosts, and bails out] ********************* 

TASK: [debug msg='As soon as we hit a system that previously fails, bails out.'] ********************* 
ok: [moria] => {"msg": "As soon as we hit a system that previously fails, bails out."}

TASK: [debug msg='As soon as we hit a system that previously fails, bails out.'] ********************* 
FATAL: no hosts matched or all hosts have already failed -- aborting


PLAY RECAP ********************* 
localhost                      : ok=0    changed=0    unreachable=0    failed=1    
moria                          : ok=2    changed=1    unreachable=0    failed=0
```



Discussion:
Speak friend, and enter.

I don't know, it is eventually failing, and it was supposed to fail, it just technically detects the "OH NO, NO MORE HOSTS!" condition, a bit later, right?

The idea is that if a whole batch size of a serial fails, the play is a failure, and as a result the playbook run should stop.

I do agree it would be nice if it reported exactly at that point, but wanted to make sure we are talking about the same thing.  Is that what you are saying, more or less?

Thanks!

So I think I don't understand the purpose of `serial: 1`.

In a normal situation if you have multiple plays (and not serialized), if one host fails, it continues with the others.

In a previous discussion you made it clear that in a serialized play (`serial: 1`) if one host fails, it stops everything.

Now I have a single play that is _not_ serialized, where one host fails. In a normal situation it would mean the next play would no longer process this host (as this one already failed). But instead it stops the next (serialized) play. That is what I do not understand, I don't understand the purpose of this behavior. (I would if everything was serialized using -f 1 given your previous clarification, but now it seems a serialized plays does not care about what happened before when it processes the original host_list ?)

_What we intend to do is make sure one process is not run in parallel when it is delegated, simply because it cannot be run in parallel... I prefer not to have to rely on flock because the host this is delegated to is Solaris, and because this is an important property of an orchestration tool to avoid running things in parallel (without bailing out if one such process fails)._

Here's a three-play example making the problem more obvious:

``` yaml
- name: First play simply fails for a few hosts
  hosts: all
  gather_facts: no
  tasks:
  - action: shell exit $((RANDOM % 2))

- name: Another play to test what happens
  hosts: all
  gather_facts: no
  tasks:
  - action: ping

- name: Subsequent play runs failed hosts, and bails out
  hosts: all
  gather_facts: no
  serial: 1
  tasks:
  - action: debug msg='As soon as we hit a system that previously fails, bails out.'
```

And the output:

```
[root@moria ansible]# ansible-playbook -v -c ssh -l localhost,moria test24.yml 

PLAY [First play simply fails for a few hosts] ********************* 

TASK: [shell exit $((RANDOM % 2))] ********************* 
changed: [moria] => {"changed": true, "cmd": "exit $((RANDOM % 2)) ", "delta": "0:00:00.002079", "end": "2012-11-28 00:47:36.516506", "rc": 0, "start": "2012-11-28 00:47:36.514427", "stderr": "", "stdout": ""}
failed: [localhost] => {"changed": true, "cmd": "exit $((RANDOM % 2)) ", "delta": "0:00:00.002273", "end": "2012-11-28 00:47:36.516661", "rc": 1, "start": "2012-11-28 00:47:36.514388"}

PLAY [Another play to test what happens] ********************* 

TASK: [ping] ********************* 
ok: [moria] => {"changed": false, "ping": "pong"}

PLAY [Subsequent play runs failed hosts, and bails out] ********************* 

TASK: [debug msg='As soon as we hit a system that previously fails, bails out.'] ********************* 
ok: [moria] => {"msg": "As soon as we hit a system that previously fails, bails out."}

TASK: [debug msg='As soon as we hit a system that previously fails, bails out.'] ********************* 
FATAL: no hosts matched or all hosts have already failed -- aborting


PLAY RECAP ********************* 
localhost                      : ok=0    changed=0    unreachable=0    failed=1    
moria                          : ok=3    changed=1    unreachable=0    failed=0
```


Resolving patch:
diff --git a/lib/ansible/playbook/__init__.py b/lib/ansible/playbook/__init__.py
--- a/lib/ansible/playbook/__init__.py
+++ b/lib/ansible/playbook/__init__.py
@@ -239,10 +239,17 @@ def _async_poll(self, poller, async_seconds, async_poll_interval):
 
     # *****************************************************
 
+    def _list_available_hosts(self):
+        ''' returns a list of hosts that haven't failed and aren't dark '''
+
+        return [ h for h in self.inventory.list_hosts() if (h not in self.stats.failures) and (h not in self.stats.dark)]
+
+    # *****************************************************
+
     def _run_task_internal(self, task):
         ''' run a particular module step in a playbook '''
 
-        hosts = [ h for h in self.inventory.list_hosts() if (h not in self.stats.failures) and (h not in self.stats.dark)]
+        hosts = self._list_available_hosts()
         self.inventory.restrict_to(hosts)
 
         runner = ansible.runner.Runner(
@@ -340,8 +347,7 @@ def _flag_handler(self, play, handler_name, host):
     def _do_setup_step(self, play):
         ''' get facts from the remote system '''
 
-        host_list = [ h for h in self.inventory.list_hosts(play.hosts)
-            if not (h in self.stats.failures or h in self.stats.dark) ]
+        host_list = self._list_available_hosts()
 
         if play.gather_facts is False:
             return {}
@@ -390,7 +396,7 @@ def _run_play(self, play):
 
         # now with that data, handle contentional variable file imports!
 
-        all_hosts = self.inventory.list_hosts(play.hosts)
+        all_hosts = self._list_available_hosts()
         play.update_vars_files(all_hosts)
 
         serialized_batch = []
@@ -425,8 +431,7 @@ def _run_play(self, play):
                         # just didn't match anything and that's ok
                         return False
 
-                host_list = [ h for h in self.inventory.list_hosts(play.hosts)
-                    if not (h in self.stats.failures or h in self.stats.dark) ]
+                host_list = self._list_available_hosts()
 
                 # if no hosts remain, drop out
                 if not host_list:


Test patch: