ansible/ansible — pull request #5519 by j2sol (2014-01-07 05:16:32+00:00)
Title: Improve efficiency of task/play host discovery for large inventory sets

Many calls were being made to inventory.list_hosts which calls some potentially lengthy list comprehensions for each pattern in a hosts: line. This was adding significant delays in executing playbooks, even if the playbook was limited to a single or small set of hosts within an inventory. With 1.4x this is particularly noticeable due to effort put in to always get the same inventory order, but testing has shown speed improvement against 1.3.x as well.

These commits attempt to optimize how host lists are calculated and manipulated throughout a playbook, play, and task execution. The trips through inventory.list_hosts are greatly reduced.

Patches were tested with an inventory below. Playbooks were tested with all success, some failure, all failure, and with a host being added in one play to be used in the second play. Testing was done to make sure fail% is still correctly handled. make tests was also ran with success.

$ cat testinv.py
```
#!/usr/bin/env python

import json

groupa = [str(h) for h in range(5000)]
groupb = [str(h) for h in range(5001, 10000)]

print(json.dumps({'groupa': groupa, 'groupb': groupb, '_meta': {'hostvars': {}}}))
```

$ cat test.yaml
```
---
- name: Test play 1
hosts: all:!groupb
gather_facts: no
connection: local

tasks:
- name: ping
ping:

- name: echo
command: echo

- name: Test play 2
hosts: all:!groupb
gather_facts: no
connection: local

tasks:
- name: ping
ping:

- name: echo
command: echo
```
```
$ time ansible-playbook -i testinv.py -f 3 test.yaml --limit 500:501
```

With my patches:
```
real	0m7.208s
```

with 1.4.4:
```
real	1m13.344s
```

with 1.3.4:
```
real	0m10.844s
```