Repository: ansible/ansible

Issue:
handlers are not executed on all hosts when strategy free applied
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report


##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
handlers

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.3.0.0
  config file = /home/jenkins/.ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.5 (default, Sep  5 2016, 02:30:38) [GCC 4.8.5 20150623 (Red Hat 4.8.5-9)]

```

##### CONFIGURATION
<!---
If using Ansible 2.4 or above, paste the results of "ansible-config dump --only-changed"

Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).

-->

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->

N/A

##### SUMMARY
<!--- Explain the problem briefly -->
Tasks in handlers block are not executed on all hosts when strategy **free** applied
The tasks in handlers block are executed OK on all hosts with linear strategy also including when serial batch process applied.

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- name: Test handler
  hosts: admin_host, app_hosts
  strategy: free
  tasks:
    - name: "Step #1"
      pause:
        seconds: "{{ 10 | random }}"

    - name: "Step #2"
      pause:
        seconds: "{{ 10 | random }}"
      notify: process_done

  handlers:
    - name: process_done
      debug:
        msg: "Hostname: {{ inventory_hostname }}"

```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->

```
PLAY [Test handlers - init] *************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************
ok: [localhost]

TASK [add_host] *************************************************************************************************************
changed: [localhost]

TASK [add_host] *************************************************************************************************************
changed: [localhost]

TASK [add_host] *************************************************************************************************************
changed: [localhost]

PLAY [Test handler] *********************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************
ok: [ofmw01]
ok: [ofmw03]
ok: [ofmw02]

TASK [Step #1] **************************************************************************************************************
changed: [ofmw03]
changed: [ofmw02]
changed: [ofmw01]

TASK [Step #2] **************************************************************************************************************
changed: [ofmw01]

RUNNING HANDLER [process_done] **********************************************************************************************

TASK [process_done] *********************************************************************************************************
ok: [ofmw01] => {
    "changed": false,
    "msg": "Hostname: ofmw01"
}

TASK [Step #2] **************************************************************************************************************
changed: [ofmw02]
changed: [ofmw03]

RUNNING HANDLER [process_done] **********************************************************************************************

TASK [process_done] *********************************************************************************************************
ok: [ofmw02] => {
    "changed": false,
    "msg": "Hostname: ofmw02"
}

RUNNING HANDLER [process_done] **********************************************************************************************

TASK [process_done] *********************************************************************************************************
ok: [ofmw03] => {
    "changed": false,
    "msg": "Hostname: ofmw03"
}


PLAY RECAP ******************************************************************************************************************
localhost                  : ok=4    changed=3    unreachable=0    failed=0
ofmw01                     : ok=4    changed=2    unreachable=0    failed=0
ofmw02                     : ok=4    changed=2    unreachable=0    failed=0
ofmw03                     : ok=4    changed=2    unreachable=0    failed=0
```

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
```
PLAY [Test handlers - init] *************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************
ok: [localhost]

TASK [add_host] *************************************************************************************************************
changed: [localhost]

TASK [add_host] *************************************************************************************************************
changed: [localhost]

TASK [add_host] *************************************************************************************************************
changed: [localhost]

PLAY [Test handler] *********************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************
ok: [ofmw01]
ok: [ofmw03]
ok: [ofmw02]

TASK [Step #1] **************************************************************************************************************
changed: [ofmw03]
changed: [ofmw02]
changed: [ofmw01]

TASK [Step #2] **************************************************************************************************************
changed: [ofmw01]

RUNNING HANDLER [process_done] **********************************************************************************************

TASK [process_done] *********************************************************************************************************
ok: [ofmw01] => {
    "changed": false,
    "msg": "Hostname: ofmw01"
}

TASK [Step #2] **************************************************************************************************************
changed: [ofmw02]
changed: [ofmw03]

PLAY RECAP ******************************************************************************************************************
localhost                  : ok=4    changed=3    unreachable=0    failed=0
ofmw01                     : ok=4    changed=2    unreachable=0    failed=0
ofmw02                     : ok=3    changed=2    unreachable=0    failed=0
ofmw03                     : ok=3    changed=2    unreachable=0    failed=0
```
<!--- Paste verbatim command output between quotes below -->
```

```



Discussion:
Are you able to verify your Ansible version again as I've tested this with both the latest devel and 2.3.0.0 and cannot replicate your issue. In fact, the `pause` module cannot be used with the free strategy and it will throw an error when you attempt to do so.

I worked around this by using the below, when running it, the handler is fired for both hosts as expected.

```
---
- hosts: localhost,Centos
  gather_facts: no
  strategy: free

  tasks:
  - name: "Step #1"
    command: hostname

  - name: "Step #2"
    command: hostname
    notify: process_done

  handlers:
    - name: process_done
      debug:
        msg: "Hostname: {{ inventory_hostname }}"
```

needs_info
This also affects me.

As @jborean93 said, pause doesn't work, but I rewrote the playbook to use shell and sleep as follows:

```
---
- name: Test handler
  hosts: all
  strategy: free
  tasks:
    - name: "Step #1"
      shell: "sleep {{ 10 | random }}"

    - name: "Step #2"
      shell: "sleep {{ 10 | random }}"
      notify: process_done

  handlers:
    - name: process_done
      debug:
        msg: "Hostname: {{ inventory_hostname }}"
```

And I can reproduce the bug:
```
$ ansible-playbook -c local -i 127.0.0.1,127.0.0.2 -Dvv /tmp/test.yml
ansible-playbook 2.4.1.0
  config file = None
  configured module search path = [u'/home/pablo/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 2.7.13 (default, Jan 19 2017, 14:48:08) [GCC 6.3.0 20170118]
No config file found; using defaults

PLAYBOOK: test.yml *************************************************************
1 plays in /tmp/test.yml

PLAY [Test handler] ************************************************************

TASK [Gathering Facts] *********************************************************
ok: [127.0.0.2]
META: ran handlers
ok: [127.0.0.1]
META: ran handlers

TASK [Step #1] *****************************************************************
task path: /tmp/test.yml:6
changed: [127.0.0.2] => {"changed": true, "cmd": "sleep 1", "delta": "0:00:01.002640", "end": "2017-11-10 09:40:24.897349", "failed": false, "rc": 0, "start": "2017-11-10 09:40:23.894709", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [127.0.0.1] => {"changed": true, "cmd": "sleep 3", "delta": "0:00:03.002202", "end": "2017-11-10 09:40:26.915542", "failed": false, "rc": 0, "start": "2017-11-10 09:40:23.913340", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
NOTIFIED HANDLER process_done

TASK [Step #2] *****************************************************************
task path: /tmp/test.yml:9
changed: [127.0.0.2] => {"changed": true, "cmd": "sleep 3", "delta": "0:00:03.002339", "end": "2017-11-10 09:40:28.042368", "failed": false, "rc": 0, "start": "2017-11-10 09:40:25.040029", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

RUNNING HANDLER [process_done] *************************************************

TASK [process_done] ************************************************************
task path: /tmp/test.yml:14
ok: [127.0.0.2] => {
    "msg": "Hostname: 127.0.0.2"
}
NOTIFIED HANDLER process_done

TASK [Step #2] *****************************************************************
task path: /tmp/test.yml:9
changed: [127.0.0.1] => {"changed": true, "cmd": "sleep 6", "delta": "0:00:06.002702", "end": "2017-11-10 09:40:33.090821", "failed": false, "rc": 0, "start": "2017-11-10 09:40:27.088119", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
META: ran handlers
META: ran handlers
META: ran handlers
META: ran handlers

PLAY RECAP *********************************************************************
127.0.0.1                  : ok=3    changed=2    unreachable=0    failed=0
127.0.0.2                  : ok=4    changed=2    unreachable=0    failed=0
```
After reviewing debug log, I believe it's a concurrency problem: notification of one host arrives just before the handler execution of the other just finish, wiping all notifications (check trace 'done running handlers').

The debug log is attached.

[handler.free.log.gz](https://github.com/ansible/ansible/files/1466161/handler.free.log.gz)

Check #32852 comments
I provide another example that is closer to my scenario (it involves multiple wait for host).

##### STEPS TO REPRODUCE
Use the following playbook:
```
---
- name: Test handler
  hosts: 127.0.0.*
  connection: local
  strategy: free
  vars:
    timeout:
      '127.0.0.1': 5
      '127.0.0.2': 70
  tasks:
    - name: "Step #1"
      shell: "sleep {{ timeout[inventory_hostname] }}"
      notify: stage1

    - name: "Step #2"
      shell: "sleep {{ 5 | random }}"

  handlers:
    - name: "Stage 1 - Step #1"
      shell: "sleep {{ item }}"
      listen: stage1
      with_items:
        - "{{ 5 | random }}"
        - "{{ 5 | random }}"

    - name: "Stage 1 - Step #2"
      shell: "sleep {{ 5 | random }}"
      listen: stage1
      notify: stage2

    - name: "Stage 2"
      debug:
        msg: "{{ inventory_hostname }} second stage"
      listen: stage2
```

Command line:
```
ansible-playbook -i 127.0.0.1,127.0.0.2 -Dvv -c local /tmp/free.test.yml
```
##### EXPECTED RESULT
```
ansible-playbook 2.4.1.0 (detached HEAD e946c4b3b7) last updated 2017/11/14 08:53:50 (GMT +200)
  config file = None
  configured module search path = [u'/home/pablo/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/pablo/ansible/lib/ansible
  executable location = /home/pablo/ansible/bin/ansible-playbook
  python version = 2.7.13 (default, Jan 19 2017, 14:48:08) [GCC 6.3.0 20170118]
No config file found; using defaults

PLAYBOOK: free.test.yml *********************************************************************************
1 plays in /tmp/free.test.yml

PLAY [Test handler] *************************************************************************************

TASK [Gathering Facts] **********************************************************************************
ok: [127.0.0.1]
META: ran handlers
ok: [127.0.0.2]
META: ran handlers
NOTIFIED HANDLER Stage 1 - Step #1
NOTIFIED HANDLER Stage 1 - Step #2

TASK [Step #1] ******************************************************************************************
task path: /tmp/free.test.yml:6
changed: [127.0.0.2] => {"changed": true, "cmd": "sleep 3", "delta": "0:00:03.002439", "end": "2017-11-14 10:56:11.769406", "failed": false, "rc": 0, "start": "2017-11-14 10:56:08.766967", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

TASK [Step #2] ******************************************************************************************
task path: /tmp/free.test.yml:10
changed: [127.0.0.2] => {"changed": true, "cmd": "sleep 7", "delta": "0:00:07.002994", "end": "2017-11-14 10:56:18.910867", "failed": false, "rc": 0, "start": "2017-11-14 10:56:11.907873", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

RUNNING HANDLER [Stage 1 - Step #1] *********************************************************************
changed: [127.0.0.2] => (item=1) => {"changed": true, "cmd": "sleep 1", "delta": "0:00:01.003190", "end": "2017-11-14 10:56:20.079050", "failed": false, "item": "1", "rc": 0, "start": "2017-11-14 10:56:19.075860", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [127.0.0.2] => (item=1) => {"changed": true, "cmd": "sleep 1", "delta": "0:00:01.002707", "end": "2017-11-14 10:56:21.232590", "failed": false, "item": "1", "rc": 0, "start": "2017-11-14 10:56:20.229883", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

TASK [Stage 1 - Step #1] ********************************************************************************
task path: /tmp/free.test.yml:14

RUNNING HANDLER [Stage 1 - Step #2] *********************************************************************
NOTIFIED HANDLER Stage 2

TASK [Stage 1 - Step #2] ********************************************************************************
task path: /tmp/free.test.yml:21
changed: [127.0.0.2] => {"changed": true, "cmd": "sleep 9", "delta": "0:00:09.002316", "end": "2017-11-14 10:56:30.937223", "failed": false, "rc": 0, "start": "2017-11-14 10:56:21.934907", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

RUNNING HANDLER [Stage 2] *******************************************************************************

TASK [Stage 2] ******************************************************************************************
task path: /tmp/free.test.yml:26
ok: [127.0.0.2] => {
    "msg": "127.0.0.2 second stage"
}

NOTIFIED HANDLER Stage 1 - Step #1
NOTIFIED HANDLER Stage 1 - Step #2

TASK [Step #1] ******************************************************************************************
task path: /tmp/free.test.yml:6
changed: [127.0.0.1] => {"changed": true, "cmd": "sleep 76", "delta": "0:01:16.003352", "end": "2017-11-14 10:57:24.769803", "failed": false, "rc": 0, "start": "2017-11-14 10:56:08.766451", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

TASK [Step #2] ******************************************************************************************
task path: /tmp/free.test.yml:10
changed: [127.0.0.1] => {"changed": true, "cmd": "sleep 7", "delta": "0:00:07.003360", "end": "2017-11-14 10:57:32.132504", "failed": false, "rc": 0, "start": "2017-11-14 10:57:25.129144", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
META: ran handlers
META: ran handlers

RUNNING HANDLER [Stage 1 - Step #1] *********************************************************************
changed: [127.0.0.1] => (item=1) => {"changed": true, "cmd": "sleep 1", "delta": "0:00:01.003190", "end": "2017-11-14 10:57:34.079050", "failed": false, "item": "1", "rc": 0, "start": "2017-11-14 10:57:33.075860", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [127.0.0.1] => (item=1) => {"changed": true, "cmd": "sleep 1", "delta": "0:00:01.002707", "end": "2017-11-14 10:57:35.232590", "failed": false, "item": "1", "rc": 0, "start": "2017-11-14 10:57:34.229883", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

RUNNING HANDLER [Stage 1 - Step #2] *********************************************************************
NOTIFIED HANDLER Stage 2

TASK [Stage 1 - Step #2] ********************************************************************************
task path: /tmp/free.test.yml:21
changed: [127.0.0.1] => {"changed": true, "cmd": "sleep 5", "delta": "0:00:05.002650", "end": "2017-11-14 10:57:40.945462", "failed": false, "rc": 0, "start": "2017-11-14 10:57:35.942812", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

RUNNING HANDLER [Stage 2] *******************************************************************************

TASK [Stage 2] ******************************************************************************************
task path: /tmp/free.test.yml:26
ok: [127.0.0.1] => {
    "msg": "127.0.0.1 second stage"
}
META: ran handlers
META: ran handlers

PLAY RECAP **********************************************************************************************
127.0.0.1                  : ok=6    changed=4    unreachable=0    failed=0
127.0.0.2                  : ok=6    changed=4    unreachable=0    failed=0
```
##### ACTUAL RESULT
```
ansible-playbook 2.4.1.0 (detached HEAD e946c4b3b7) last updated 2017/11/14 08:53:50 (GMT +200)
  config file = None
  configured module search path = [u'/home/pablo/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/pablo/ansible/lib/ansible
  executable location = /home/pablo/ansible/bin/ansible-playbook
  python version = 2.7.13 (default, Jan 19 2017, 14:48:08) [GCC 6.3.0 20170118]
No config file found; using defaults

PLAYBOOK: free.test.yml *********************************************************************************
1 plays in /tmp/free.test.yml

PLAY [Test handler] *************************************************************************************

TASK [Gathering Facts] **********************************************************************************
ok: [127.0.0.1]
META: ran handlers
ok: [127.0.0.2]
META: ran handlers
NOTIFIED HANDLER Stage 1 - Step #1
NOTIFIED HANDLER Stage 1 - Step #2

TASK [Step #1] ******************************************************************************************
task path: /tmp/free.test.yml:6
changed: [127.0.0.2] => {"changed": true, "cmd": "sleep 3", "delta": "0:00:03.002439", "end": "2017-11-14 10:56:11.769406", "failed": false, "rc": 0, "start": "2017-11-14 10:56:08.766967", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

TASK [Step #2] ******************************************************************************************
task path: /tmp/free.test.yml:10
changed: [127.0.0.2] => {"changed": true, "cmd": "sleep 7", "delta": "0:00:07.002994", "end": "2017-11-14 10:56:18.910867", "failed": false, "rc": 0, "start": "2017-11-14 10:56:11.907873", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

RUNNING HANDLER [Stage 1 - Step #1] *********************************************************************
changed: [127.0.0.2] => (item=1) => {"changed": true, "cmd": "sleep 1", "delta": "0:00:01.003190", "end": "2017-11-14 10:56:20.079050", "failed": false, "item": "1", "rc": 0, "start": "2017-11-14 10:56:19.075860", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [127.0.0.2] => (item=1) => {"changed": true, "cmd": "sleep 1", "delta": "0:00:01.002707", "end": "2017-11-14 10:56:21.232590", "failed": false, "item": "1", "rc": 0, "start": "2017-11-14 10:56:20.229883", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

TASK [Stage 1 - Step #1] ********************************************************************************
task path: /tmp/free.test.yml:14


(Here gets frozen)


NOTIFIED HANDLER Stage 1 - Step #1
NOTIFIED HANDLER Stage 1 - Step #2

TASK [Step #1] ******************************************************************************************
task path: /tmp/free.test.yml:6
changed: [127.0.0.1] => {"changed": true, "cmd": "sleep 76", "delta": "0:01:16.003352", "end": "2017-11-14 10:57:24.769803", "failed": false, "rc": 0, "start": "2017-11-14 10:56:08.766451", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

RUNNING HANDLER [Stage 1 - Step #2] *********************************************************************
NOTIFIED HANDLER Stage 2

TASK [Stage 1 - Step #2] ********************************************************************************
task path: /tmp/free.test.yml:21
changed: [127.0.0.1] => {"changed": true, "cmd": "sleep 5", "delta": "0:00:05.002650", "end": "2017-11-14 10:57:29.945462", "failed": false, "rc": 0, "start": "2017-11-14 10:57:24.942812", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
NOTIFIED HANDLER Stage 2
changed: [127.0.0.2] => {"changed": true, "cmd": "sleep 9", "delta": "0:00:09.002316", "end": "2017-11-14 10:57:33.937223", "failed": false, "rc": 0, "start": "2017-11-14 10:57:24.934907", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

RUNNING HANDLER [Stage 2] *******************************************************************************

TASK [Stage 2] ******************************************************************************************
task path: /tmp/free.test.yml:26
ok: [127.0.0.1] => {
    "msg": "127.0.0.1 second stage"
}
ok: [127.0.0.2] => {
    "msg": "127.0.0.2 second stage"
}
META: ran handlers
META: ran handlers

TASK [Step #2] ******************************************************************************************
task path: /tmp/free.test.yml:10
changed: [127.0.0.1] => {"changed": true, "cmd": "sleep 7", "delta": "0:00:07.003360", "end": "2017-11-14 10:57:41.132504", "failed": false, "rc": 0, "start": "2017-11-14 10:57:34.129144", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
META: ran handlers
META: ran handlers

PLAY RECAP **********************************************************************************************
127.0.0.1                  : ok=5    changed=3    unreachable=0    failed=0
127.0.0.2                  : ok=6    changed=4    unreachable=0    failed=0
```
After trying to understand the free strategy, I identified several points contributing to this issue:

1. The [following line](https://github.com/ansible/ansible/blob/v2.4.1.0-1/lib/ansible/plugins/strategy/__init__.py#L778) cause the freeze on free strategy with a handler on a live host and with other ones blocked: 
```
host_results = self._wait_on_pending_results(iterator)
```

2. [This one](https://github.com/ansible/ansible/blob/v2.4.1.0-1/lib/ansible/plugins/strategy/__init__.py#L820) causes a race condition when a notification of another host arrives while it is being executed:
```
self._notified_handlers[handler._uuid] = []
```

3. [Run handler](https://github.com/ansible/ansible/blob/v2.4.1.0-1/lib/ansible/plugins/strategy/__init__.py#L868) without using `target_host` provided causes the execution of another hosts' handlers when they still haven't finished with previous tasks executed after notification:
```
elif meta_action == 'flush_handlers':
    self.run_handlers(iterator, play_context)
```

I've been able to hotfix 2 and 3, but the 1 require to wait only for hosts that are executing the present handler, instead of all of them, and may require a whole rewrite of the functions `_process_pending_results` or `_wait_on_pending_results`, or even rewrite the structure of the metadata stored while queueing tasks in order to easily identify which tasks are handlers and if they are reached the point after they can be executed.

I can provide my modifications, but I think that fix this require a deeper knowledge of the code.

Any suggestions?

so i couldn't reproduce this initially, but it was mostly due to lack of 'differences' in my tests hosts, the following generally reproduces the problem on almost every run.

```yaml
- hosts: all
  gather_facts: False
  strategy: free
  tasks:
    - wait_for: timeout={{5|random}}
      delegate_to: localhost
      changed_when: True
      notify: trip

  handlers:
    - debug: var=inventory_hostname
      name: trip
```

Resolving patch:
diff --git a/lib/ansible/plugins/strategy/__init__.py b/lib/ansible/plugins/strategy/__init__.py
--- a/lib/ansible/plugins/strategy/__init__.py
+++ b/lib/ansible/plugins/strategy/__init__.py
@@ -197,6 +197,10 @@ def __init__(self, tqm):
         # outstanding tasks still in queue
         self._blocked_hosts = dict()
 
+        # this dictionary is used to keep track of hosts that have
+        # flushed handlers
+        self._flushed_hosts = dict()
+
         self._results = deque()
         self._results_lock = threading.Condition(threading.Lock())
 
@@ -657,6 +661,35 @@ def parent_handler_match(target_handler, handler_name):
 
         return ret_results
 
+    def _wait_on_handler_results(self, iterator, handler, notified_hosts):
+        '''
+        Wait for the handler tasks to complete, using a short sleep
+        between checks to ensure we don't spin lock
+        '''
+
+        ret_results = []
+        handler_results = 0
+
+        display.debug("waiting for handler results...")
+        while (self._pending_results > 0 and
+               handler_results < len(notified_hosts) and
+               not self._tqm._terminated):
+
+            if self._tqm.has_dead_workers():
+                raise AnsibleError("A worker was found in a dead state")
+
+            results = self._process_pending_results(iterator)
+            ret_results.extend(results)
+            handler_results += len([
+                r._host for r in results if r._host in notified_hosts and
+                r.task_name == handler.name])
+            if self._pending_results > 0:
+                time.sleep(C.DEFAULT_INTERNAL_POLL_INTERVAL)
+
+        display.debug("no more pending handlers, returning what we have")
+
+        return ret_results
+
     def _wait_on_pending_results(self, iterator):
         '''
         Wait for the shared counter to drop to zero, using a short sleep
@@ -843,14 +876,17 @@ def _do_handler_run(self, handler, handler_name, iterator, play_context, notifie
         #     self._tqm.send_callback('v2_playbook_on_no_hosts_remaining')
         #     result = False
         #     break
-        saved_name = handler.name
-        handler.name = handler_name
-        self._tqm.send_callback('v2_playbook_on_handler_task_start', handler)
-        handler.name = saved_name
-
         if notified_hosts is None:
             notified_hosts = self._notified_handlers[handler._uuid]
 
+        notified_hosts = self._filter_notified_hosts(notified_hosts)
+
+        if len(notified_hosts) > 0:
+            saved_name = handler.name
+            handler.name = handler_name
+            self._tqm.send_callback('v2_playbook_on_handler_task_start', handler)
+            handler.name = saved_name
+
         run_once = False
         try:
             action = action_loader.get(handler.action, class_only=True)
@@ -871,7 +907,7 @@ def _do_handler_run(self, handler, handler_name, iterator, play_context, notifie
                     break
 
         # collect the results from the handler run
-        host_results = self._wait_on_pending_results(iterator)
+        host_results = self._wait_on_handler_results(iterator, handler, notified_hosts)
 
         try:
             included_files = IncludedFile.process_include_results(
@@ -910,11 +946,22 @@ def _do_handler_run(self, handler, handler_name, iterator, play_context, notifie
                     display.warning(str(e))
                     continue
 
-        # wipe the notification list
-        self._notified_handlers[handler._uuid] = []
+        # remove hosts from notification list
+        self._notified_handlers[handler._uuid] = [
+            h for h in self._notified_handlers[handler._uuid]
+            if h not in notified_hosts]
         display.debug("done running handlers, result is: %s" % result)
         return result
 
+    def _filter_notified_hosts(self, notified_hosts):
+        '''
+        Filter notified hosts accordingly to strategy
+        '''
+
+        # As main strategy is linear, we do not filter hosts
+        # We return a copy to avoid race conditions
+        return notified_hosts[:]
+
     def _take_step(self, task, host=None):
 
         ret = False
@@ -962,7 +1009,9 @@ def _evaluate_conditional(h):
         elif meta_action == 'flush_handlers':
             if task.when:
                 self._cond_not_supported_warn(meta_action)
+            self._flushed_hosts[target_host] = True
             self.run_handlers(iterator, play_context)
+            self._flushed_hosts[target_host] = False
             msg = "ran handlers"
         elif meta_action == 'refresh_inventory' or self.flush_cache:
             if task.when:
diff --git a/lib/ansible/plugins/strategy/free.py b/lib/ansible/plugins/strategy/free.py
--- a/lib/ansible/plugins/strategy/free.py
+++ b/lib/ansible/plugins/strategy/free.py
@@ -49,6 +49,15 @@
 
 class StrategyModule(StrategyBase):
 
+    def _filter_notified_hosts(self, notified_hosts):
+        '''
+        Filter notified hosts accordingly to strategy
+        '''
+
+        # We act only on hosts that are ready to flush handlers
+        return [host for host in notified_hosts
+                if host in self._flushed_hosts and self._flushed_hosts[host]]
+
     def run(self, iterator, play_context):
         '''
         The "free" strategy is a bit more complex, in that it allows tasks to


Test patch: