Repository: ansible/ansible

Issue:
"include" statements are slower as the playbook run, and memory rockets up through the roof
Hello,

**Edit 1:** Sorry i forgot to say, I'm using the devel branch, and have no idea if these speed are reproducible on the 1.9 branch, as i'm using some functionality (Blocks) that isn't available except on the devel branch. So my specs are ansible 2.0 (devel branch), on Yosemite 10.10.4. The version of python i'm using is 2.7.10 as i'm installing it using Homebrew. The VM is running Ubuntu on a vagrant / virtualbox provider.

**Edit 2, Clarification:** I understand why includes are being ran even when nothing will be performed, parsing the tags inside of each sub-task seems relevant for the correct behavior of ansible. What i don't understand however, is the memory consumption / cpu usage going up the roof, and the run times slowing down as the application performs. I noticed that this was accentuated by my usage of dictionary-based variables. Maybe it wasn't clear in the first place.

Consider this role:

```

---
- include:    setup.yml
  tags:       realtime-setup
- include:    database.yml
  tags:       realtime-database
- include:    tokenizer.yml
  tags:       realtime-tokenizer
- include:    tokenizer_ambassador.yml
  tags:       realtime-tokenizer-ambassador
- include:    server.yml
  tags:       realtime-server
- include:    dasherize.yml
  tags:       realtime-dasherize
```

Each of these sub-tasks are themselves including files, that themselves include other files, since most of the functionality is DRY across the role.

I am noticing an insanely slow speed, unrelated to the connection speed (I just have set up a VM to make sure of that).
If i run **all the commands manually** verbatim from what the ansible logs give me, I end up with a 1 minute 20 seconds run total time (Again, _commands equivalent as if i was running the role without tags at all_).
If i run them via ansible, **only using the last tag of the playbook** takes more than 35 minutes!

I also have played around and after simply copy-pasting ansible commands in one single ugly big playbook, the timings are back to something acceptable. Maybe my poking around will solve the issue, but having a fat playbook of 2k lines instead of a nice and tidy-packed DRY approach seems quite unmaintainable to me.

If i start the playbook with the **last tag only** (`realtime-dasherize`), here is what happens:

```
PLAY ***************************************************************************

TASK [setup] *******************************************************************
ok: [vm0]

TASK [docker.*EDIT* : include] ***********************************************
included: setup.yml for vm0

TASK [docker.*EDIT* : include] ***********************************************
included: database.yml for vm0

TASK [docker.*EDIT* : include] ***********************************************
included: roles/host.cargo/tasks/build_container.yml for vm0

TASK [docker.*EDIT* : include] ***********************************************
skipping: [vm0]

TASK [docker.*EDIT* : include] ***********************************************
included: roles/host.cargo/tasks/build_image.yml for vm0

TASK [docker.*EDIT* : include] ***********************************************
included: tokenizer.yml for vm0

TASK [docker.*EDIT* : include] ***********************************************
included: roles/host.cargo/tasks/build_container.yml for vm0

TASK [docker.*EDIT* : include] ***********************************************
included: roles/host.cargo/tasks/build_image.yml for vm0

TASK [docker.*EDIT* : include] ***********************************************
included: roles/host.cargo/tasks/build_image.yml for vm0

TASK [docker.*EDIT* : include] ***********************************************
included: tokenizer_ambassador.yml for vm0

TASK [docker.*EDIT* : include] ***********************************************
included: roles/host.cargo/tasks/build_container.yml for vm0

TASK [docker.*EDIT* : include] ***********************************************
included: roles/host.cargo/tasks/build_image.yml for vm0

TASK [docker.*EDIT* : include] ***********************************************
included: roles/host.cargo/tasks/build_image.yml for vm0

TASK [docker.*EDIT* : include] ***********************************************
included: server.yml for vm0

TASK [docker.*EDIT* : include] ***********************************************
included: roles/host.cargo/tasks/build_container.yml for vm0

TASK [docker.*EDIT* : include] ***********************************************
included: roles/host.cargo/tasks/build_image.yml for vm0

TASK [docker.*EDIT* : include] ***********************************************
included: roles/host.cargo/tasks/build_image.yml for vm0

TASK [docker.*EDIT* : include] ***********************************************
included: *EDIT*.yml for vm0

TASK [docker.*EDIT* : creates *EDIT*-*EDIT* folders] ********************
ok: [vm0] => (item=/home/vagrant/*EDIT*/*EDIT*)
ok: [vm0] => (item=/home/vagrant/*EDIT*/shares/*EDIT*/bundle)
ok: [vm0] => (item=/home/vagrant/*EDIT*/shares/*EDIT*/www)
ok: [vm0] => (item=/home/vagrant/*EDIT*/shares/*EDIT*/bower_components)
ok: [vm0] => (item=/home/vagrant/*EDIT*/shares/*EDIT*/npm)

TASK [docker.*EDIT* : include] ***********************************************
included: roles/host.cargo/tasks/build_container.yml for vm0

TASK [docker.*EDIT* : include] ***********************************************
included: roles/host.cargo/tasks/build_image.yml for vm0

TASK [docker.*EDIT* : attempts to get *EDIT*/*EDIT*/*EDIT* from private registry] ***
changed: [vm0]

TASK [docker.*EDIT* : creates *EDIT*/*EDIT*/*EDIT* temporary dockerfile] ***
changed: [vm0]

TASK [docker.*EDIT* : fills up *EDIT*/*EDIT*/*EDIT* dockerfile] 
skipping: [vm0]

TASK [docker.*EDIT* : copies *EDIT*/*EDIT*/*EDIT* dockerfile] **
changed: [vm0]

TASK [docker.*EDIT* : creates *EDIT*/*EDIT*/*EDIT* image (dynamic dockerfile method)] ***
changed: [vm0]

TASK [docker.*EDIT* : creates *EDIT*/*EDIT*/*EDIT* image (static dockerfile method)] ***
skipping: [vm0]

TASK [docker.*EDIT* : removes the temporary dockerfile] **********************
changed: [vm0]

TASK [docker.*EDIT* : pushes *EDIT*/*EDIT*/*EDIT* to *EDIT*] ***
changed: [vm0]

TASK [docker.*EDIT* : downloads *EDIT*-*EDIT* source code] **************
ok: [vm0]

TASK [docker.*EDIT* : copies *EDIT*-*EDIT* entry point] *****************
ok: [vm0]

TASK [docker.*EDIT* : include] ***********************************************
included: roles/host.cargo/tasks/build_image.yml for vm0

TASK [docker.*EDIT* : attempts to get *EDIT*/*EDIT*/*EDIT* from private registry] ***
skipping: [vm0]

TASK [docker.*EDIT* : creates *EDIT*/*EDIT*/*EDIT* temporary dockerfile] ***
changed: [vm0]

TASK [docker.*EDIT* : fills up *EDIT*/*EDIT*/*EDIT* dockerfile] ***
changed: [vm0]

TASK [docker.*EDIT* : copies *EDIT*/*EDIT*/*EDIT* dockerfile] ***
skipping: [vm0]

TASK [docker.*EDIT* : creates *EDIT*/*EDIT*/*EDIT* image (dynamic dockerfile method)] ***
changed: [vm0]

TASK [docker.*EDIT* : creates *EDIT*/*EDIT*/*EDIT* image (static dockerfile method)] ***
skipping: [vm0]

TASK [docker.*EDIT* : removes the temporary dockerfile] **********************
changed: [vm0]

TASK [docker.*EDIT* : pushes *EDIT*/*EDIT*/*EDIT* to *EDIT*] ***
skipping: [vm0]

TASK [docker.*EDIT* : creates *EDIT*-*EDIT* container] ******************
changed: [vm0]

PLAY RECAP *********************************************************************
vm0                        : ok=35   changed=11   unreachable=0    failed=0
```

Here is a snippet of the included files, because i really think it could profit the community as i had incredible problems using the builtin docker module.

The one that is the most slow is `build_container.yml`, as it takes up to 30 seconds once the playbook reaches the end (So i'm noticing inconsistent timings as the memory goes up... Memleak / stacked variables?).

```
# build_container.yml

---
# Build base image ...
- include:          roles/host.cargo/tasks/build_image.yml
  vars:
    params:         "{{parent}}"
  when:             parent is defined

# Clone repository.
- name:             "downloads {{name}} source code"
  git:
    accept_hostkey: yes
    force:          yes
    key_file:       "{{container['clone_key']}}"
    dest:           "{{container['dir']}}"
    repo:           "{{container['repo']}}"
  when:             container['clone_key'] is defined and container['dir'] is defined and container['repo'] is defined
# Copies entrypoint.
- name:             "copies {{name}} entry point"
  copy:             src={{container['entrypoint']}} dest={{container['dir']}}/entrypoint.sh
  when:             container['entrypoint'] is defined and container['dir'] is defined

# ... then build image ...
- include:          roles/host.cargo/tasks/build_image.yml
  vars:
    params:         "{{container}}"

# ... to finally start container.
- name:             creates {{name}} container
  sudo:             yes
  docker:
    name:           "{{name}}"
    state:          reloaded
    restart_policy: always
    image:          "{{container['image']}}"
    command:        "{{container['command']       | default()}}"
    ports:          "{{container['ports']         | default([])}}"
    expose:         "{{container['expose']        | default([])}}"
    volumes:        "{{container['volumes']       | default([])}}"
    volumes_from:   "{{container['volumes_from']  | default([])}}"
    links:          "{{container['links']         | default([])}}"
    env:            "{{container['env']           | default({})}}"
```

```
# build_image.yml

---
# Tries to get the image from registry first.
- name:             attempts to get {{params['image']}} from private registry
  shell:            "docker pull {{params['image']}}"
  ignore_errors:    yes
  when:             ( params['pull'] | default(False) )

# Create a remote unique temp file.
- name:             creates {{params['image']}} temporary dockerfile
  shell:            mktemp -p {{params['dir'] | default(home)}}
  register:         mktmp_res
  when:             params['from'] is defined or params['dockerfile'] is defined

  # Fill up the dockerfile...
- name:             fills up {{params['image']}} dockerfile
  copy:             content="FROM {{params['from']}}" dest={{mktmp_res.stdout_lines[0]}} force=yes
  when:             params['from'] is defined
# ... or copy it.
- name:             copies {{params['image']}} dockerfile
  template:         src={{params['dockerfile']}} dest={{mktmp_res.stdout_lines[0]}} force=yes
  when:             params['dockerfile'] is defined

# Builds the image.
- name:             creates {{params['image']}} image (dynamic dockerfile method)
  sudo:             yes
  shell:            "docker build --tag {{params['image']}} --file {{mktmp_res.stdout_lines[0]}} {{params['dir'] | default(home)}}"
  when:             params['from'] is defined or params['dockerfile'] is defined
- name:             creates {{params['image']}} image (static dockerfile method)
  sudo:             yes
  shell:            "docker build --tag {{params['image']}} {{params['dir'] | default(home)}}"
  when:             params['from'] is not defined and params['dockerfile'] is not defined

# Get rid of the temp file.
- name:             removes the temporary dockerfile
  file:             path={{mktmp_res.stdout_lines[0]}} state=absent
  when:             params['from'] is defined or params['dockerfile'] is defined

# Push the image if required.
- name:             pushes {{params['image']}} to {{docker_registry['address']}}
  shell:            "docker push {{params['image']}}"
  when:             ( params['push'] | default(False) )
```

After around _35 minutes_ running, here is a snapshot of the process resources usage:

```
41061  Python       100.0 07:45.60 6/1   0    25    324M
```

Thanks for your time reading this, i admit, very long report :)



Discussion:
The include is ALWAYS used, the tags apply to the included tasks and each is evaluated against them.

So even if you just use the last tag, all tasks are evaluated for execution.

I recommend '--start-at-task' to speed things up.

@bcoca Thanks for the --start-at-task tip, it is useful as long as the bug isn't solved. However:

Fun facts:
- Run time of my playbook with ansible 1.9.2: **6 minutes**.
- Run time of the same playbook with 2.0 dev: **41 minutes**.

As i said:
- As of 1.9, an include statement is pretty much displayed "instantly".
- With 2.0 dev, it sometimes takes 2 minutes to tell `skipping: [vm0]`

Also, a very good pointer about the "why" is this memory footprint... Growing and growing, as the application slows down.
Any clue on how i can help to debug this?

It's a bit complicated for me to switch from 1.9.2 to 2.0dev because when you pass a parameter for a file to an included yml from another playbook, the paths are not handled the same. Right now, i have stepped back to 1.9.2 and will stick to it until i see some progress on this report, then i can test again if needed.

Well, 2.0 is currently slower than 1.9, specially in with includes as they are now more dynamic and can be templated. We are working on speeding things up, but there will be some hard limitations here.

That said, I'm sure there are plenty of space to improve on those times, we have already gotten rid of some big bottlenecks by profiling some our tests, if you could do the same for your case it would help us a lot and pinpoint why it is taking so long for you.

@bcoca Understood.

Regarding timing improvements, i really have no idea on how to profile my playbooks (I am learning Ansible for a week now). Do you think you could give me directions on how to do that so i can give the team a report or something similar? A link or two maybe, or command arguments?

sadly no writeups, I use python profiling using cProfile, I'll have to write something up or make it an option.

I think I'm seeing this.

My test playbook, when it has nothing to do, runs in 40s with Ansible 1.9.  It needs 1m34s with the `devel` branch (efa005c228).  Moreover, when I try to limit the tasks using `--tags`, I see it processing includes (very slowly, while eating 100% CPU).  Curiously, this is even slower than running all the tasks: takes 3m45s.  Visually, it looks like a series of includes is getting slower and slower, then a non-include task runs, something gets flushed, and includes become fast again, temporarily, before they start getting slower.

I tried some profiling with [line_profiler](https://pypi.python.org/pypi/line_profiler):

```
python -m kernprof -o /tmp/ansible.prof ~/src/ansible/bin/ansible-playbook test.yml
python -m pstats /tmp/ansible.prof
% sort cumtime
% stats 20
Tue Aug 25 16:00:05 2015    /tmp/ansible.prof

         31626373 function calls (30012526 primitive calls) in 105.404 seconds

   Ordered by: cumulative time
   List reduced from 2301 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000  105.408  105.408 <string>:1(<module>)
        1    0.001    0.001  105.408  105.408 {execfile}
        1    0.002    0.002  105.407  105.407 /home/mg/src/ansible/bin/ansible-playbook:21(<module>)
        1    0.000    0.000  105.124  105.124 /home/mg/src/ansible/lib/ansible/cli/playbook.py:74(run)
        1    0.000    0.000  105.094  105.094 /home/mg/src/ansible/lib/ansible/executor/playbook_executor.py:66(run)
        2    0.000    0.000  104.739   52.369 /home/mg/src/ansible/lib/ansible/executor/task_queue_manager.py:155(run)
        2    0.008    0.004  104.451   52.226 /home/mg/src/ansible/lib/ansible/plugins/strategies/linear.py:127(run)
17194/322    0.074    0.000   86.482    0.269 /home/mg/src/ansible/lib/ansible/playbook/task.py:243(copy)
  2395/74    0.038    0.000   86.436    1.168 /home/mg/src/ansible/lib/ansible/playbook/block.py:147(copy)
    19593    0.671    0.000   86.129    0.004 /home/mg/src/ansible/lib/ansible/playbook/base.py:229(copy)
 7185/354    0.052    0.000   85.966    0.243 /home/mg/src/ansible/lib/ansible/playbook/block.py:148(_dupe_task_list)
8284906/7499724    3.232    0.000   68.325    0.000 {getattr}
964567/596871    3.415    0.000   66.584    0.000 /home/mg/src/ansible/lib/ansible/playbook/base.py:96(_generic_g)
  1005788   63.061    0.000   63.061    0.000 {dir}
       21    0.000    0.000   34.999    1.667 /home/mg/src/ansible/lib/ansible/playbook/block.py:321(filter_tagged_tasks)
100947/89181    0.193    0.000   26.603    0.000 /home/mg/src/ansible/lib/ansible/playbook/task.py:327(_get_parent_attribute)
121835/104199    0.501    0.000   22.397    0.000 /home/mg/src/ansible/lib/ansible/playbook/block.py:262(_get_parent_attribute)
       36    0.025    0.001   16.417    0.456 /home/mg/src/ansible/lib/ansible/plugins/strategies/__init__.py:275(_wait_on_pending_results)
     1486   15.470    0.010   15.470    0.010 {time.sleep}
    41205    2.512    0.000   13.445    0.000 /home/mg/src/ansible/lib/ansible/playbook/base.py:112(_get_base_attributes)
```


Resolving patch:
diff --git a/lib/ansible/playbook/base.py b/lib/ansible/playbook/base.py
--- a/lib/ansible/playbook/base.py
+++ b/lib/ansible/playbook/base.py
@@ -96,7 +96,7 @@ def __init__(self):
     @staticmethod
     def _generic_g(prop_name, self):
         method = "_get_attr_%s" % prop_name
-        if method in dir(self):
+        if hasattr(self, method):
             return getattr(self, method)()
 
         return self._attributes[prop_name]


Test patch: