Repository: ansible/ansible

Issue:
Memory Allocated for Included Tasks is Never Recovered
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
include (but probably any task as well)

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.4.0.0
  config file = /root/workspace/ansible-memory-leak/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /root/workspace/ansible-2.4.0.0/lib/ansible
  executable location = /root/workspace/ansible-2.4.0.0/bin/ansible
  python version = 2.7.5 (default, Aug  2 2016, 04:20:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]

```

##### CONFIGURATION
<!---
If using Ansible 2.4 or above, paste the results of "ansible-config dump --only-changed"

Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).

-->
For this test playbook, added the following to ansible.cfg:
```
callback_whitelist = profile_memory
```

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->
Control host: Red Hat Enterprise Linux Server release 7.3 (Maipo)
Targets: NA

##### SUMMARY
<!--- Explain the problem briefly -->
On the control machine, Ansible process memory increases with the number of tasks in the playbook.  The most noticeable jumps are when an include task is processed and several tasks are read in.  The problem is that the memory is never released.  For our production playbook, if the server does not have enough memory, Ansible crashes with this error: 
```
ERROR! Unexpected Exception: [Errno 12] Cannot allocate memory.
```

In our production playbook, each include task increases the memory by about 100MB.


##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
1. Install the memory profiler:
```
pip install -U memory_profiler
```
2. update ansible.cfg with this line:
```
callback_whitelist = profile_memory
```
3. Run the playbook:
```
ansible-playbook memory_leak.yml
```

<!--- Paste example playbooks or commands between quotes below -->
memory_leak.yml
```
---

- name: Demonstrate memory growth as tasks are included
  hosts: localhost
  gather_facts: false


  tasks:

  - include: foo/some_tasks.yml
    with_sequence: start=1 end=10

  - include: foo/some_tasks.yml
    with_sequence: start=1 end=10

  - include: foo/some_tasks.yml
    with_sequence: start=1 end=10

...

```
foo/some_tasks.yml
```
---

- name: Get timestamp
  set_fact:
    timestamp: "{{ lookup('pipe', 'date +%s.%3N') }}"
  delegate_to: localhost
  register: task_result

- debug:
    var: task_result

...
```

callback_plugins/profile_memory.py
```
from ansible.plugins.callback import CallbackBase
from ansible import constants as C
from memory_profiler import profile

class CallbackModule(CallbackBase):
    """
    This callback module tells you how much memory a task takes
    """
    CALLBACK_VERSION = 2.0
    CALLBACK_TYPE = 'aggregate'
    CALLBACK_NAME = 'profile_memory'
    CALLBACK_NEEDS_WHITELIST = True


    def __init__(self):
        super(CallbackModule, self).__init__()

    @profile(precision=3)
    def v2_playbook_on_task_start(self, task, is_conditional):
        self.playbook_on_task_start(task.name, is_conditional)
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
My expectation is that once the life cycle of a task is complete (task has run and results returned), across all applicable hosts, all memory allocated for that task and supporting objects should be released, such that memory use is mostly flat, instead of ever-increasing, for any number of tasks.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```

PLAY [Demonstrate memory growth as tasks are included] ************************************************************************************************************************************************************

TASK [include] ****************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   47.641 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   47.641 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   48.691 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   48.691 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   48.781 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   48.781 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850676.742"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   48.895 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   48.895 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   48.895 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   48.895 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850676.842"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   48.953 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   48.953 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   48.984 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   48.984 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850676.943"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   48.992 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   48.992 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   49.023 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   49.023 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850677.065"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   49.051 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   49.051 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   49.055 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   49.055 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850677.163"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   49.148 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   49.148 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   49.148 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   49.148 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850677.271"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   49.184 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   49.184 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   49.199 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   49.199 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850677.378"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   49.207 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   49.207 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   49.250 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   49.250 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850677.483"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   49.262 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   49.262 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   49.266 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   49.266 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850677.623"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   49.293 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   49.293 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   49.293 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   49.293 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850677.733"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [include] ****************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   49.293 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   49.293 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   50.094 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   50.094 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   50.148 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   50.148 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850678.014"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   50.152 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   50.152 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   50.184 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   50.184 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850678.126"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   50.277 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   50.277 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   50.277 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   50.277 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850678.248"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   50.301 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   50.301 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   50.316 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   50.316 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850678.349"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   50.320 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   50.320 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   50.355 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   50.355 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850678.450"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   50.367 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   50.367 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   50.371 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   50.371 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850678.549"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   50.395 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   50.395 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   50.395 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   50.395 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850678.649"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   50.422 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   50.422 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   50.453 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   50.453 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850678.768"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   50.465 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   50.465 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   50.484 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   50.484 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850678.875"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   50.508 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   50.508 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   50.520 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   50.520 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850678.997"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [include] ****************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   50.531 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   50.531 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost
included: /root/workspace/ansible-memory-leak/foo/some_tasks.yml for localhost

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   52.020 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   52.020 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   52.031 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   52.031 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850679.480"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   52.109 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   52.109 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   52.125 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   52.125 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850679.592"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   52.152 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   52.152 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   52.219 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   52.219 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850679.714"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   52.230 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   52.230 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   52.234 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   52.234 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850679.850"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   52.234 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   52.234 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   52.234 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   52.234 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850679.981"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   52.234 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   52.234 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   52.234 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   52.234 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850680.111"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   52.234 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   52.234 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   52.234 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   52.234 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850680.216"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   52.234 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   52.234 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   52.234 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   52.234 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850680.321"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   52.234 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   52.234 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   52.234 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   52.234 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850680.419"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

TASK [Get timestamp] **********************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   52.262 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   52.262 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost -> localhost]

TASK [debug] ******************************************************************************************************************************************************************************************************
Filename: /root/workspace/ansible-memory-leak/callback_plugins/profile_memory.py

Line #    Mem usage    Increment   Line Contents
================================================
    20   52.262 MiB    0.000 MiB       @profile(precision=3)
    21                                 def v2_playbook_on_task_start(self, task, is_conditional):
    22   52.262 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1507850680.523"
        }, 
        "ansible_facts_cacheable": false, 
        "changed": false, 
        "failed": false
    }
}

PLAY RECAP ********************************************************************************************************************************************************************************************************
localhost                  : ok=90   changed=0    unreachable=0    failed=0   

```



Discussion:
We seem to be having Memory Leak issues with 2.4. I will try and get more details.

From what I have been told, on our clients that we upgraded to 2.4, after the memory leak, we downgraded to 2.3.something, and the problem remains.

Apparently we are using lots of includes.

Perhaps the issues is Python? ( Was a thought that my coworker had )
Besides 2.4, we're seeing the issue in 2.3.1.
I can reproduce this sort of with:
ansible 2.5.0 (devel ff923fb6b0) last updated 2017/12/14 15:54:43 (GMT -400)


I put the repro test case info at https://github.com/alikins/ansible-bug-repro/tree/master/mem_usage_a24_31673

Reproducing with f27/python2.7, the memory profile info grew 55MB to 60MB at the last memory profile output. So u
```
Line #    Mem usage    Increment   Line Contents
================================================
    18   60.086 MiB   60.086 MiB       @profile(precision=3)
    19                                 def v2_playbook_on_task_start(self, task, is_conditional):
    20   60.086 MiB    0.000 MiB           self.playbook_on_task_start(task.name, is_conditional)


ok: [localhost] => {
    "task_result": {
        "ansible_facts": {
            "timestamp": "1513284889.089"
        }, 
        "changed": false, 
        "failed": false
    }
```
I've been hit by this same issue. It rendered our main playbook inoperable. (vs running on Ansible 2.1, in which it ran fine). We run this playbook on an admin host that has 4G of RAM, and normally, we're never even vaguely close to any limits, and now the process fails with OOM.

I think it's because we have a lot of tasks that are included in loops. We need them because we set up hundreds of services on dozens of servers. (So I can't use import_tasks, which I suspect doesn't have such a memory problem.)

On a test system with a simpler run of the same playbook (less services and hosts) I validated that Ansible 2.4 is using many times the RAM of Ansible 2.1, I used /usr/bin/time -v on the whole ansible-playbook invocation, which gives a very clear report of the max RAM (RSS) used.

Is there anything I could do to help here? I don't know anything about developing/debugging Ansible internals, so probably not, but if there are any things that I could do to help move this forward/make it easier for it to be debugged/resolved, I'm motivated.
I did a bunch of test runs against various ansible versions. (Everything that pip would install for me.) It seems like there were two jumps in RSS memory usage, one in 2.2 and one in 2.4:

```
version run1  run2  run3
2.1.3.0 16396 16268 16404
2.1.4.0 16344 16252 16396
2.1.5.0 16380 16336 16380
2.1.6.0 16380 16376 16192
2.2.0.0 43820 43796 43964
2.2.1.0 45556 46332 45936
2.2.2.0 46012 45856 45820
2.2.3.0 46892 46924 47092
2.3.0.0 47508 47520 47536
2.3.1.0 45492 45568 45608
2.3.2.0 45644 45840 45528
2.3.3.0 45752 45544 45688
2.4.0.0 53748 53904 53888
2.4.1.0 53808 53876 54028
2.4.2.0 54020 53732 53732
```

I don't think that anyone should read anything into the magnitude of the jumps. It's just when they happened that I imagine is most relevant.

Next stop is looking at commits, seeing if I can narrow down the commit that caused each jump. I imagine that I will "compile" from source to do that.
Can we test again now that this has been merged? https://github.com/ansible/ansible/pull/34461
Ah, I had hope that this was going to solve the problem, but alas, it seems
it did not.

I used a clean repo, which means I’m operating off of the ‘devel’ branch.

[ec2-user@ip-10-32-2-176 ansible]$ ansible-playbook --version
ansible-playbook 2.5.0 (devel 46263d77f1) last updated 2018/01/08 16:15:23
(GMT +000)
  config file = None
  configured module search path =
[u'/home/ec2-user/.ansible/plugins/modules',
u'/usr/share/ansible/plugins/modules']
  ansible python module location = /bigdisk/ansible/lib/ansible
  executable location = /bigdisk/ansible/bin/ansible-playbook
  python version = 2.7.11 (default, Feb  1 2017, 18:42:03) [GCC 4.1.2
20080704 (Red Hat 4.1.2-54)]

[ec2-user@ip-10-32-2-176 ansible_memory_leak]$ /usr/bin/time -v
ansible-playbook memory_leak.yml  |& grep 'Maximum resident set size'
Maximum resident set size (kbytes): 56992
[ec2-user@ip-10-32-2-176 ansible_memory_leak]$ /usr/bin/time -v
ansible-playbook memory_leak.yml  |& grep 'Maximum resident set size'
Maximum resident set size (kbytes): 56656
[ec2-user@ip-10-32-2-176 ansible_memory_leak]$ /usr/bin/time -v
ansible-playbook memory_leak.yml  |& grep 'Maximum resident set size'
Maximum resident set size (kbytes): 56772

This is similar to other the output of the testing against other 2.4
releases.

Sean

On January 8, 2018 at 8:08:48 AM, Adam Miller (notifications@github.com)
wrote:

Can we test again now that this has been merged? #34461
<https://github.com/ansible/ansible/pull/34461>

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<https://github.com/ansible/ansible/issues/31673#issuecomment-356010372>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AB2E29vbExX4QgEAA2BtSJcrqIOJksygks5tIj2PgaJpZM4P3xfO>
.

@ssuchter thanks for checking
This is against various branches in a checkout:

```
branch     run1  run2  run3  run4  run5  median
stable-2.1 35836 29072 29036 29060 28880 29060
stable-2.2 36452 30004 29928 29968 29824 29968
stable-2.3 45608 45548 45632 45656 45412 45608
stable-2.4 54320 53728 54000 53488 53948 53948
devel      57364 57060 56916 56840 57068 57060
```
Ok, I definitely don't understand *why* but I have found one change that significantly increases the RAM footprint of ansible-playbook. The thing that mystifies me is which change it is:

> commit 6ba6819446d9ff887b5919002123d478d8b7ec10
> Author: James Cammarata <jimi@sngx.net>
> Date:   Thu Dec 8 01:16:08 2016 -0500
> 
>     Re-adding the init py for modules
> 

Here's my evidence that this commit causes significant memory usage. I checked out various revisions and narrowed down a range until I got to the exact commit that caused memory usage increase. I tried many commits immediately before and immediately after this one, and it was exactly this one that caused a significant change, it seems.

In the below table, rev_idx is how many back from the head (currently 46263d77f1) the revision is. I used this to narrow down which revisions to look at.

```
revision   rev_idx median run1  run2  run3  run4  run5  run6  run7  run8  run9  run10 run11 run12
46263d77f1 1       51688  51848 51592 51688 51748 51228 51864 52188 51812 51140 51804 51684 51336
1fd454e5b5 1000    47892  49024 47564 47704 49048 47604 47880 48568 47904 47780
df941df01d 2000    47624  47648 47568 47580 47916 47600 47288 47652 48680 47600
2e85272cd5 4000    47680  47608 47720 47732 47916 47384 47636 47640 47720 47584
0d67a49d5a 5000    50748  50964 50508 50532
6a4324ee99 6000    48992  48928 48956 49028
d1b38cb718 7000    46888  47116 46980 46596 47024 46720 46888
727ca6f436 7200    44904  45056 44752 44684
3fa1ddc9db 7400    44960  44896 44928 44692 45172 44984 44952 45012 44968 45004 45240 44960 44520
345e39e1b1 7401    45036  45108 44752 44964
1da20c9716 7402    44964  45036 44652 44892
ec95cbc96e 7403    44956  45016 44824 44896
71cc868f2b 7404    44308  45208 44800 44928 44308 44096 43796
b9e98406bd 7405    44456  45132 44788 44800 44456 43744 43748
c709b22e5c 7406    44376  44936 44888 44892 44376 43760 43784
1c9a58a885 7407    44416  44912 44896 44680 44416 43840 43788
fa5386c488 7408    44846  44988 44672 45008 45220 44776 44916 43536 43812 43924
6ba6819446 7409    44742  45196 44728 44692 45012 44756 44852 44352 43656 43800
0b47c908eb 7410    26844  27104 26844 26856 26956 26852 26632 27136 26660 26884 26348 25536 25748
250b0dff46 7411    26448  27132 26584 26884 26448 25584 25632
cb1888125d 7412    26348  27052 26660 26584 26348 25752 25644
5628e260f2 7413    25962  26220 25704 25516
573078fc6b 7414    25960  26300 25568 25620
75c281debc 7415    26002  26328 25584 25676
580401c02b 7420    26966  27084 26848 26848
b5a63a556d 7430    26984  27116 26852 26828
d2fc24b771 7440    26874  26864 26884 26856
5bfbeadca3 7450    26648  26936 26648 26636 27148 26860 26648
2f369dff88 7500    26862  26872 26852 26572
7e6fe97f57 7550    26834  26936 26584 26732
1e00adefd5 7600    26688  27108 26660 26584 27144 26872 26688
10ac3ba617 7800    26762  26940 26584 26548
a2f9e0d2fe 8000    26808  26856 26880 26872 26920 26772 26664 26924 26576 26872 26808 26548 26668
```

There seem to be a couple newer jumps too, I'm going to look for them.
  
I found a smaller, but significant jump in memory due to this commit:

> commit dd0257b995a729705fe39b3dc4dc9b8f9ea1ef8f
> Author: James Cammarata <jimi@sngx.net>
> Date:   Tue Dec 20 08:43:39 2016 -0600
> 
>     Template "original_task" fields in _process_pending_results
> 
>     Since we no longer use a post-validated task in _process_pending_results, we
>     need to be sure to template fields used in original_task as they are raw and
>     may contain variables.
> 
>     This patch also moves the handler tracking to be per-uuid, not per-object.
>     Doing it per-object had implications for the above due to the fact that the
>     copy of the original task is now being used, so the only sure way is to track
>     based on the uuid instead.
> 
>     Fixes #18289

That per-uuid change seems like it could cause memory management changes, that might be something to look at.

Evidence:

```
revision   rev_idx median run1  run2  run3  run4  run5  run6
d70d279c4e 7190    47026  47164 46664 46800 47124 46800 46928
446888ef41 7191    46760  47056 46756 46760
0d06b77970 7192    46820  46968 46820 46808
dd0257b995 7193    46808  47108 46808 46804
7faa041636 7194    43808  44472 43808 43756
a999452b68 7195    43804  44396 43796 43804
3691c784d2 7196    43780  44416 43744 43780
91094ec96c 7197    43792  44316 43784 43792
```
There's another jump (although small) in this commit:

> commit 3d65482927e33daa30bb6bfc2153b4c64775ee15
> Author: James Cammarata <jimi@sngx.net>
> Date:   Fri Feb 17 09:14:26 2017 -0600
> 
>     Stash post-validated fields of the task in the TaskResult
> 
>     This allows us to have a snapshot of the fields, which we can restore
>     on the pre-fork side so as to avoid having to re-template fields.

```
revision   rev_idx median run1  run2  run3  run4  run5  run6
c136ff58c0 6002    47276  47792 47276 47332
bedf0279a9 6003    47264  47568 47700 47264
3d65482927 6004    47352  47612 47216 47352 47596 47276 47352
40235d7b99 6005    45840  45840 46696 47132
a1b3664ec4 6006    46546  46236 46600 46376 47452 46492 47048
879feff6b7 6007    46324  46652 46756 46324
01436cf186 6008    46378  45772 45900 46348 46408 46924 46744
```

Another jump (small) in this commit:

> commit a108f0fcd3585ef39f0382a0108bee902d829ba4
> Author: Brian Coca <bcoca@users.noreply.github.com>
> Date:   Mon Apr 24 15:19:52 2017 -0400
> 
>     avoid hosts 'pre templated warnings' (#23878)
> 
>     * avoid hosts 'pre templated warnings'
> 
>     fixed is_template failing on undefined
>     fixes #23057
> 
>     * py3 compat

```
revision   rev_idx median run1  run2  run3  run4  run5  run6
cb13aea88b 5034    48976  49192 48976 49032
d021d8f16b 5035    49016  49228 49016 49088
380ac577c6 5036    49180  49584 48996 48976 49196 49604 49164
a108f0fcd3 5037    49000  49120 49004 49000
74e385e43a 5038    47640  47812 47612 47644 47824 47520 47636
e0a7105c2d 5039    47632  47872 47664 47632
80327d9d47 5040    47718  47836 47644 47792 47844 47624 47612
```
Here's a commit that made a small but meaningful RAM improvement (uses less RAM):

> commit 78478e80ea01c2d752af5abdfbf3f744bfd188ac
> Author: James Cammarata <jimi@sngx.net>
> Date:   Thu May 11 14:57:30 2017 -0500
> 
>     Avoid default serialization of classes in dump_attrs
> 
>     For playbook base objects, when dumping attributes via dump_attrs() an
>     attribute like loop_control is a class. Using the default serialization
>     for these is slow and consumes a lot of memory. Since LoopControl is also
>     based on the Base class, we can use serialize() instead and save a lot of
>     resources.
> 
>     This also adds a from_attrs() complimentary method to nicely turn the
>     dumped attrs back into proper field attributes.
> 
>     Fixes #23579

```
revision   rev_idx median run1  run2  run3  run4  run5  run6
a47fbba8ae 4804    45616  45584 45784 45736 45648 45472 45408
4638b5f2da 4805    45672  45992 45688 45672
78478e80ea 4806    45672  45656 45388 45384 45836 45812 45688
409fe1a3c3 4807    49168  49336 49168 49496
b04df23da2 4808    49518  49192 49544 49492 49620 49616 49232
c50cf22d52 4809    48824  48824 49540 49416
4a5cf0b5c1 4810    49060  49304 49060 49140
```
This commit increases RAM usage somewhat:

> commit db749de5b87f0bb6c7f03e633505c7a4d313bb7f
> Author: Brian Coca <brian.coca+git@gmail.com>
> Date:   Sun Oct 29 00:33:02 2017 -0400
> 
>     namespace facts
> 
>     updated action plugins to use new guranteed facts
>     updated tests to new data clean
>     added cases for ansible_local and some docstrings

```
revision   rev_idx median run1  run2  run3  run4  run5  run6
ee80215fa9 965     55572  56028 55572 55644
9c629f8a1c 966     55580  56004 55580 55740
76202bc7c3 967     55756  56100 55756 55848
fedd1779cc 968     55676  56028 55940 55676
db749de5b8 969     55736  56036 55736 55868
e0cb54a2aa 970     53422  53772 53388 53256 53648 53432 53412
f1fe467c22 971     53180  53684 53568 53180
4443da2d79 972     53148  53772 53148 53964
4464635787 973     53408  53660 53456 53408
356901b72d 974     53328  53700 53328 53504
38444bb76c 975     53216  53540 53492 53216
```
This (very recent) commit improves (lowers) RAM usage somewhat:

> commit 46263d77f145ffd46a0ff05ce61c5b96026c02f8
> Author: Nathaniel Case <this.is@nathanielca.se>
> Date:   Mon Jan 8 10:56:47 2018 -0500
> 
>     Fixes for nxos_config integration tests (#34579)
> 
>     * deletage_to: localhost -> connection: local
> 
>     * Fix test setup

```
revision   rev_idx median run1  run2  run3  run4  run5  run6  run7  run8  run9  run10 run11 run12
46263d77f1 1       51718  51848 51592 51688 51748 51228 51864 52188 51812 51140 51804 51684 51336
42dd48a6d0 2       56924  57356 56924 56984
473b2d58ec 3       56728  57380 56728 57100
ad73bda323 4       56864  57160 56864 56924
943107b1b9 5       56840  57320 56920 56840
```
Ok, I re-looked through all of these commits and ran a more extensive test case (5x number of loops for playbook includes). This was designed to exacerbate the memory "leak" as opposed to things that just increase memory usage by something more fixed.

I'm convinced that we should be looking at this commit:

> commit 6ba6819
> Author: James Cammarata jimi@sngx.net
> Date: Thu Dec 8 01:16:08 2016 -0500
> 
> Re-adding the init py for modules

I have no idea why such a change causes such a huge difference, but it's clearly gigantic. This change increased the memory for my test case (that does 500 includes inside 3 nested loops of a trivial task list (just calls debug once and prints out the loop index numbers) by over 4x. (88Mb)

These memory numbers are not comparable to the earlier ones I was posting. Those were all with 100 includes, this is with 500 includes:

```
revision   rev_idx median run1   run2   run3   run4   run5   run6
46263d77f1 1       126900 127848 128656 126900
42dd48a6d0 2       127824 127844 128568 127824
db749de5b8 969     125660 126380 125660 126696
e0cb54a2aa 970     122744 123072 122872 122744
78478e80ea 4806    114624 114956 114624 114852
409fe1a3c3 4807    135952 136032 136072 135952
a108f0fcd3 5037    135916 136484 135916 135956
74e385e43a 5038    132772 132960 132772 132880
3d65482927 6004    133080 133080 133128 133264
40235d7b99 6005    114296 114588 114636 114296
dd0257b995 7193    115352 115352 115524 115492
7faa041636 7194    113092 114652 113092 114280
1c9a58a885 7407    113364 114768 113904 113364
fa5386c488 7408    114296 114608 114296 114300
6ba6819446 7409    114304 114612 114292 113232 114740 114308 114300
0b47c908eb 7410    25714  26328  25660  25716  26204  25524  25712
250b0dff46 7411    25520  26288  25584  25520
cb1888125d 7412    25584  26352  25584  25680
5628e260f2 7413    25468  26208  25656  25468
```

I also checked that there aren't any weird cross-branch things going on. E.g. there is a direct parent commit chain going on:

1c9a58a885f322d0e4b9c03697f344b2fe9f05d0 -> fa5386c488762c540f393f4a2749d5a1ba2c5633 -> 6ba6819446d9ff887b5919002123d478d8b7ec10 -> 0b47c908ebe3b6816ba34ecd6c1c40ed51fa7e4f -> 250b0dff46c8e7656c8108c0f8132845712d9661 -> cb1888125d6a9edbee278121332a9fdc4337d7af -> 5628e260f23a37cf16419235819ba05d1248ed51 -> 573078fc6b9796601de8817496d278dfeb1f04f0
Attached is the trivial playbook setup that I used to reproduce this issue. memory_leak.yml does 100 includes, memory_leak2.yml does 600 includes (not 500 as I noted above, oops)

[ansible_memory_leak.zip](https://github.com/ansible/ansible/files/1612930/ansible_memory_leak.zip)

@ssuchter - thank you for your time in putting together these tests! Hope that this will make fixing the issue faster (since I can't upgrade Ansible until it's fixed).
Ok, some of those previous results (in particular, the pointing at commit 6ba6819) was wrong. That commit was showing up as a big user because ansible was actually crashing without it, but my test harness didn't catch that and so thought the crashed ansible didn't take much RAM.

I re-ran everything, and found (with 600 includes) some commits that appear to be very costly:

Commit e238ae999b hurts to the tune of 6%. Not huge.
Commit 78478e80ea helps to the tune of 19%. Good!
Commit 390be19e29 hurts to the tune of 17%. Bad!
and last, but certainly not least:
Commit 47acf55fa9 hurts to the tune of 57%. Very bad!!

This is commit 47acf55fa9:

> Cache tasks by uuid in PlayIterator for O(1) lookups
> 
> Rather than repeatedly searching for tasks by uuid via iterating over
> all known blocks, cache the tasks when they are added to the PlayIterator
> so the lookup becomes a simple key check in a dict.

Here's the memory usage before and after that commit:

```
revision   rev_idx median delta run1   run2   run3   run4   run5
47acf55fa9 14760   100184 -0.57 100312 100184 100520 100180 100404
125c53e691 14761   43280  -0.00 43280  43232  43308  43316  43304
```

The obvious thing to look at in that commit is if anything gets cached in self._task_uuid_cache that didn't get added to self._blocks. There are two bits of code I'm suspicious of:

These lines in 192-193 - they seem to cache something from self._play.handlers that isn't in self._blocks:

```
for handler_block in self._play.handlers:
    self.cache_block_tasks(handler_block)
``` 

These lines in add_tasks, lines 573-574. It doesn't seem like these gets added to self._blocks either:

```
for b in task_list:
    self.cache_block_tasks(b)
```


Resolving patch:
diff --git a/lib/ansible/executor/play_iterator.py b/lib/ansible/executor/play_iterator.py
--- a/lib/ansible/executor/play_iterator.py
+++ b/lib/ansible/executor/play_iterator.py
@@ -153,8 +153,6 @@ def __init__(self, inventory, play, play_context, variable_manager, all_vars, st
         self._blocks = []
         self._variable_manager = variable_manager
 
-        self._task_uuid_cache = dict()
-
         # Default options to gather
         gather_subset = play_context.gather_subset
         gather_timeout = play_context.gather_timeout
@@ -242,16 +240,10 @@ def get_host_state(self, host):
         return self._host_states[host.name].copy()
 
     def cache_block_tasks(self, block):
-        def _cache_portion(p):
-            for t in p:
-                if isinstance(t, Block):
-                    self.cache_block_tasks(t)
-                elif t._uuid not in self._task_uuid_cache:
-                    self._task_uuid_cache[t._uuid] = t
-
-        for portion in (block.block, block.rescue, block.always):
-            if portion is not None:
-                _cache_portion(portion)
+        # now a noop, we've changed the way we do caching and finding of
+        # original task entries, but just in case any 3rd party strategies
+        # are using this we're leaving it here for now
+        return
 
     def get_next_task_for_host(self, host, peek=False):
 
@@ -520,19 +512,8 @@ def is_failed(self, host):
         return self._check_failed_state(s)
 
     def get_original_task(self, host, task):
-        '''
-        Finds the task in the task list which matches the UUID of the given task.
-        The executor engine serializes/deserializes objects as they are passed through
-        the different processes, and not all data structures are preserved. This method
-        allows us to find the original task passed into the executor engine.
-        '''
-
-        if isinstance(task, Task):
-            the_uuid = task._uuid
-        else:
-            the_uuid = task
-
-        return self._task_uuid_cache.get(the_uuid, None)
+        # now a noop because we've changed the way we do caching
+        return (None, None)
 
     def _insert_tasks_into_state(self, state, task_list):
         # if we've failed at all, or if the task list is empty, just return the current state
@@ -569,6 +550,4 @@ def _insert_tasks_into_state(self, state, task_list):
         return state
 
     def add_tasks(self, host, task_list):
-        for b in task_list:
-            self.cache_block_tasks(b)
         self._host_states[host.name] = self._insert_tasks_into_state(self.get_host_state(host), task_list)
diff --git a/lib/ansible/plugins/strategy/__init__.py b/lib/ansible/plugins/strategy/__init__.py
--- a/lib/ansible/plugins/strategy/__init__.py
+++ b/lib/ansible/plugins/strategy/__init__.py
@@ -120,9 +120,9 @@ def inner(self, iterator, one_pass=False, max_passes=None):
         for result in results:
             task = result._task
             host = result._host
-            _queue_task_args = self._queue_task_args.pop('%s%s' % (host.name, task._uuid))
-            task_vars = _queue_task_args['task_vars']
-            play_context = _queue_task_args['play_context']
+            _queued_task_args = self._queued_task_cache.pop((host.name, task._uuid), None)
+            task_vars = _queued_task_args['task_vars']
+            play_context = _queued_task_args['play_context']
             # Try to grab the previous host state, if it doesn't exist use get_host_state to generate an empty state
             try:
                 prev_host_state = prev_host_states[host.name]
@@ -179,7 +179,11 @@ def __init__(self, tqm):
         self._final_q = tqm._final_q
         self._step = getattr(tqm._options, 'step', False)
         self._diff = getattr(tqm._options, 'diff', False)
-        self._queue_task_args = {}
+
+        # the task cache is a dictionary of tuples of (host.name, task._uuid)
+        # used to find the original task object of in-flight tasks and to store
+        # the task args/vars and play context info used to queue the task.
+        self._queued_task_cache = {}
 
         # Backwards compat: self._display isn't really needed, just import the global display and use that.
         self._display = display
@@ -270,13 +274,6 @@ def add_tqm_variables(self, vars, play):
     def _queue_task(self, host, task, task_vars, play_context):
         ''' handles queueing the task up to be sent to a worker '''
 
-        self._queue_task_args['%s%s' % (host.name, task._uuid)] = {
-            'host': host,
-            'task': task,
-            'task_vars': task_vars,
-            'play_context': play_context
-        }
-
         display.debug("entering _queue_task() for %s/%s" % (host.name, task.action))
 
         # Add a write lock for tasks.
@@ -306,6 +303,13 @@ def _queue_task(self, host, task, task_vars, play_context):
             while True:
                 (worker_prc, rslt_q) = self._workers[self._cur_worker]
                 if worker_prc is None or not worker_prc.is_alive():
+                    self._queued_task_cache[(host.name, task._uuid)] = {
+                        'host': host,
+                        'task': task,
+                        'task_vars': task_vars,
+                        'play_context': play_context
+                    }
+
                     worker_prc = WorkerProcess(self._final_q, task_vars, host, task, play_context, self._loader, self._variable_manager, shared_loader_obj)
                     self._workers[self._cur_worker][0] = worker_prc
                     worker_prc.start()
@@ -425,7 +429,8 @@ def parent_handler_match(target_handler, handler_name):
 
             # get the original host and task. We then assign them to the TaskResult for use in callbacks/etc.
             original_host = get_original_host(task_result._host)
-            found_task = iterator.get_original_task(original_host, task_result._task)
+            queue_cache_entry = (original_host.name, task_result._task)
+            found_task = self._queued_task_cache.get(queue_cache_entry)['task']
             original_task = found_task.copy(exclude_parent=True, exclude_tasks=True)
             original_task._parent = found_task._parent
             original_task.from_attrs(task_result._task_fields)
@@ -854,8 +859,6 @@ def _do_handler_run(self, handler, handler_name, iterator, play_context, notifie
         host_results = []
         for host in notified_hosts:
             if not handler.has_triggered(host) and (not iterator.is_failed(host) or play_context.force_handlers):
-                if handler._uuid not in iterator._task_uuid_cache:
-                    iterator._task_uuid_cache[handler._uuid] = handler
                 task_vars = self._variable_manager.get_vars(play=iterator._play, host=host, task=handler)
                 self.add_tqm_variables(task_vars, play=iterator._play)
                 self._queue_task(host, handler, task_vars, play_context)


Test patch:
diff --git a/test/units/executor/test_play_iterator.py b/test/units/executor/test_play_iterator.py
--- a/test/units/executor/test_play_iterator.py
+++ b/test/units/executor/test_play_iterator.py
@@ -155,16 +155,6 @@ def test_play_iterator(self):
             all_vars=dict(),
         )
 
-        # lookup up an original task
-        target_task = p._entries[0].tasks[0].block[0]
-        task_copy = target_task.copy(exclude_parent=True)
-        found_task = itr.get_original_task(hosts[0], task_copy)
-        self.assertEqual(target_task, found_task)
-
-        bad_task = Task()
-        found_task = itr.get_original_task(hosts[0], bad_task)
-        self.assertIsNone(found_task)
-
         # pre task
         (host_state, task) = itr.get_next_task_for_host(hosts[0])
         self.assertIsNotNone(task)
diff --git a/test/units/plugins/strategy/test_strategy_base.py b/test/units/plugins/strategy/test_strategy_base.py
--- a/test/units/plugins/strategy/test_strategy_base.py
+++ b/test/units/plugins/strategy/test_strategy_base.py
@@ -20,6 +20,7 @@
 __metaclass__ = type
 
 from units.mock.loader import DictDataLoader
+from copy import deepcopy
 import uuid
 
 from ansible.compat.tests import unittest
@@ -207,15 +208,18 @@ def fake_run(self):
         tqm._initialize_processes(3)
         tqm.hostvars = dict()
 
+        mock_task = MagicMock()
+        mock_task._uuid = 'abcd'
+
         try:
             strategy_base = StrategyBase(tqm=tqm)
-            strategy_base._queue_task(host=mock_host, task=MagicMock(), task_vars=dict(), play_context=MagicMock())
+            strategy_base._queue_task(host=mock_host, task=mock_task, task_vars=dict(), play_context=MagicMock())
             self.assertEqual(strategy_base._cur_worker, 1)
             self.assertEqual(strategy_base._pending_results, 1)
-            strategy_base._queue_task(host=mock_host, task=MagicMock(), task_vars=dict(), play_context=MagicMock())
+            strategy_base._queue_task(host=mock_host, task=mock_task, task_vars=dict(), play_context=MagicMock())
             self.assertEqual(strategy_base._cur_worker, 2)
             self.assertEqual(strategy_base._pending_results, 2)
-            strategy_base._queue_task(host=mock_host, task=MagicMock(), task_vars=dict(), play_context=MagicMock())
+            strategy_base._queue_task(host=mock_host, task=mock_task, task_vars=dict(), play_context=MagicMock())
             self.assertEqual(strategy_base._cur_worker, 0)
             self.assertEqual(strategy_base._pending_results, 3)
         finally:
@@ -282,7 +286,6 @@ def _queue_put(item, *args, **kwargs):
         mock_iterator._play = mock_play
         mock_iterator.mark_host_failed.return_value = None
         mock_iterator.get_next_task_for_host.return_value = (None, None)
-        mock_iterator.get_original_task.return_value = mock_task
 
         mock_handler_block = MagicMock()
         mock_handler_block.block = [mock_handler_task]
@@ -337,12 +340,16 @@ def _has_dead_workers():
         strategy_base._blocked_hosts['test01'] = True
         strategy_base._pending_results = 1
 
-        strategy_base._queue_task_args = MagicMock()
-        strategy_base._queue_task_args.pop.return_value = {
-            'task_vars': {},
-            'play_context': {},
+        mock_queued_task_cache = {
+            (mock_host.name, mock_task._uuid): {
+                'task': mock_task,
+                'host': mock_host,
+                'task_vars': {},
+                'play_context': {},
+            }
         }
 
+        strategy_base._queued_task_cache = deepcopy(mock_queued_task_cache)
         results = strategy_base._wait_on_pending_results(iterator=mock_iterator)
         self.assertEqual(len(results), 1)
         self.assertEqual(results[0], task_result)
@@ -354,6 +361,7 @@ def _has_dead_workers():
         strategy_base._blocked_hosts['test01'] = True
         strategy_base._pending_results = 1
         mock_iterator.is_failed.return_value = True
+        strategy_base._queued_task_cache = deepcopy(mock_queued_task_cache)
         results = strategy_base._wait_on_pending_results(iterator=mock_iterator)
         self.assertEqual(len(results), 1)
         self.assertEqual(results[0], task_result)
@@ -367,6 +375,7 @@ def _has_dead_workers():
         queue_items.append(task_result)
         strategy_base._blocked_hosts['test01'] = True
         strategy_base._pending_results = 1
+        strategy_base._queued_task_cache = deepcopy(mock_queued_task_cache)
         results = strategy_base._wait_on_pending_results(iterator=mock_iterator)
         self.assertEqual(len(results), 1)
         self.assertEqual(results[0], task_result)
@@ -379,6 +388,7 @@ def _has_dead_workers():
         queue_items.append(task_result)
         strategy_base._blocked_hosts['test01'] = True
         strategy_base._pending_results = 1
+        strategy_base._queued_task_cache = deepcopy(mock_queued_task_cache)
         results = strategy_base._wait_on_pending_results(iterator=mock_iterator)
         self.assertEqual(len(results), 1)
         self.assertEqual(results[0], task_result)
@@ -388,6 +398,7 @@ def _has_dead_workers():
         queue_items.append(TaskResult(host=mock_host.name, task=mock_task._uuid, return_data=dict(add_host=dict(host_name='newhost01', new_groups=['foo']))))
         strategy_base._blocked_hosts['test01'] = True
         strategy_base._pending_results = 1
+        strategy_base._queued_task_cache = deepcopy(mock_queued_task_cache)
         results = strategy_base._wait_on_pending_results(iterator=mock_iterator)
         self.assertEqual(len(results), 1)
         self.assertEqual(strategy_base._pending_results, 0)
@@ -396,6 +407,7 @@ def _has_dead_workers():
         queue_items.append(TaskResult(host=mock_host.name, task=mock_task._uuid, return_data=dict(add_group=dict(group_name='foo'))))
         strategy_base._blocked_hosts['test01'] = True
         strategy_base._pending_results = 1
+        strategy_base._queued_task_cache = deepcopy(mock_queued_task_cache)
         results = strategy_base._wait_on_pending_results(iterator=mock_iterator)
         self.assertEqual(len(results), 1)
         self.assertEqual(strategy_base._pending_results, 0)
@@ -404,6 +416,7 @@ def _has_dead_workers():
         queue_items.append(TaskResult(host=mock_host.name, task=mock_task._uuid, return_data=dict(changed=True, _ansible_notify=['test handler'])))
         strategy_base._blocked_hosts['test01'] = True
         strategy_base._pending_results = 1
+        strategy_base._queued_task_cache = deepcopy(mock_queued_task_cache)
         results = strategy_base._wait_on_pending_results(iterator=mock_iterator)
         self.assertEqual(len(results), 1)
         self.assertEqual(strategy_base._pending_results, 0)
@@ -501,6 +514,7 @@ def fake_run(*args):
         mock_handler_task._role = None
         mock_handler_task._parent = None
         mock_handler_task._uuid = 'xxxxxxxxxxxxxxxx'
+        mock_handler_task.copy.return_value = mock_handler_task
 
         mock_handler = MagicMock()
         mock_handler.block = [mock_handler_task]
@@ -516,13 +530,13 @@ def fake_run(*args):
         mock_inventory = MagicMock()
         mock_inventory.get_hosts.return_value = [mock_host]
         mock_inventory.get.return_value = mock_host
+        mock_inventory.get_host.return_value = mock_host
 
         mock_var_mgr = MagicMock()
         mock_var_mgr.get_vars.return_value = dict()
 
         mock_iterator = MagicMock()
         mock_iterator._play = mock_play
-        mock_iterator.get_original_task.return_value = mock_handler_task
 
         fake_loader = DictDataLoader()
         mock_options = MagicMock()
@@ -545,9 +559,11 @@ def fake_run(*args):
             strategy_base._inventory = mock_inventory
             strategy_base._notified_handlers = {mock_handler_task._uuid: [mock_host]}
 
-            task_result = TaskResult(Host('host01'), Handler(), dict(changed=False))
-            strategy_base._queue_task_args = MagicMock()
-            strategy_base._queue_task_args.pop.return_value = {
+            task_result = TaskResult(mock_host.name, mock_handler_task._uuid, dict(changed=False))
+            strategy_base._queued_task_cache = dict()
+            strategy_base._queued_task_cache[(mock_host.name, mock_handler_task._uuid)] = {
+                'task': mock_handler_task,
+                'host': mock_host,
                 'task_vars': {},
                 'play_context': mock_play_context
             }