Repository: ansible/ansible

Issue:
Increased memory usage in ansible-playbook from 2.4.2 to 2.4.3
##### ISSUE TYPE

Bug Report

##### COMPONENT NAME

ansible-playbook

##### ANSIBLE VERSION

```
ansible 2.4.3.0 (detached HEAD 8a7f9beab7) last updated 2018/02/06 09:50:10 (GMT -700)
  config file = None
  configured module search path = [u'/Users/mclay/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/mclay/code/mattclay/ansible/lib/ansible
  executable location = /Users/mclay/code/mattclay/ansible/bin/ansible
  python version = 2.7.13 (default, Jul 18 2017, 09:17:00) [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)]
```

##### CONFIGURATION

None

##### OS / ENVIRONMENT

Ubuntu 16.04

##### SUMMARY

The `git` integration tests consume far more memory under 2.4.3 than they did under 2.4.2.

##### STEPS TO REPRODUCE

1. Apply the [mprof patch](https://gist.github.com/mattclay/c7ee66370ddc15fe649a2e5f289b74f2) to `ansible-test` to track `ansible-playbook` memory usage.
1. Run the `git` integration test: `ansible-test integration git --docker ubuntu1604 -v`
1. Compare the results of `test/results/logs/mprofile*.dat` between a 2.4.2 and 2.4.3 test run.

##### EXPECTED RESULTS

Peak memory usage around 280 MB (2.4.2):

```
MEM 280.367188 1517945476.7207
```

##### ACTUAL RESULTS

Peak memory usage over 1500 MB (2.4.3):

```
MEM 1525.949219 1517944990.4613
```

With a low enough memory limit the test fails with: `[Errno 12] Cannot allocate memory`



Discussion:
After bisecting from 2.4.2 to 2.4.3, it seems the spike in memory utilization comes from https://github.com/ansible/ansible/commit/f1c458303a1ad00675497597e1f7fbc0539e314e

Resolving patch:
diff --git a/lib/ansible/playbook/helpers.py b/lib/ansible/playbook/helpers.py
--- a/lib/ansible/playbook/helpers.py
+++ b/lib/ansible/playbook/helpers.py
@@ -48,29 +48,38 @@ def load_list_of_blocks(ds, play, parent_block=None, role=None, task_include=Non
 
     block_list = []
     if ds:
-        for block_ds in ds:
-            b = Block.load(
-                block_ds,
-                play=play,
-                parent_block=parent_block,
-                role=role,
-                task_include=task_include,
-                use_handlers=use_handlers,
-                variable_manager=variable_manager,
-                loader=loader,
-            )
+        count = iter(range(len(ds)))
+        for i in count:
+            block_ds = ds[i]
             # Implicit blocks are created by bare tasks listed in a play without
             # an explicit block statement. If we have two implicit blocks in a row,
             # squash them down to a single block to save processing time later.
-            if b._implicit and len(block_list) > 0 and block_list[-1]._implicit:
-                for t in b.block:
-                    if isinstance(t._parent, (TaskInclude, IncludeRole)):
-                        t._parent._parent = block_list[-1]
-                    else:
-                        t._parent = block_list[-1]
-                block_list[-1].block.extend(b.block)
-            else:
-                block_list.append(b)
+            implicit_blocks = []
+            while block_ds is not None and not Block.is_block(block_ds):
+                implicit_blocks.append(block_ds)
+                i += 1
+                # Advance the iterator, so we don't repeat
+                next(count, None)
+                try:
+                    block_ds = ds[i]
+                except IndexError:
+                    block_ds = None
+
+            # Loop both implicit blocks and block_ds as block_ds is the next in the list
+            for b in (implicit_blocks, block_ds):
+                if b:
+                    block_list.append(
+                        Block.load(
+                            b,
+                            play=play,
+                            parent_block=parent_block,
+                            role=role,
+                            task_include=task_include,
+                            use_handlers=use_handlers,
+                            variable_manager=variable_manager,
+                            loader=loader,
+                        )
+                    )
 
     return block_list
 
diff --git a/lib/ansible/plugins/strategy/__init__.py b/lib/ansible/plugins/strategy/__init__.py
--- a/lib/ansible/plugins/strategy/__init__.py
+++ b/lib/ansible/plugins/strategy/__init__.py
@@ -763,7 +763,8 @@ def _load_included_file(self, included_file, iterator, is_handler=False):
             elif not isinstance(data, list):
                 raise AnsibleError("included task files must contain a list of tasks")
 
-            ti_copy = included_file._task.copy()
+            ti_copy = included_file._task.copy(exclude_parent=True)
+            ti_copy._parent = included_file._task._parent
             temp_vars = ti_copy.vars.copy()
             temp_vars.update(included_file._args)
             # pop tags out of the include args, if they were specified there, and assign


Test patch: