Issue 1:
ISSUE TYPE

Performance problem

COMPONENT NAME

Inventory

ANSIBLE VERSION

Present in any version of ansible after 2.2.2, including 2.4 rc5 and devel as of Sep 18 2017. The profiles below were captured in v2.4.0.0-0.5.rc5, but are similar to profiles in 2.3 and current devel.

ansible 2.4.0.0
config file = /Users/tpepper/.ansible.cfg
configured module search path = [u'/Users/tpepper/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python2.7/site-packages/ansible
executable location = /usr/local/bin/ansible
python version = 2.7.13 (default, Apr 4 2017, 08:47:57) [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.38)]

CONFIGURATION

Default

OS / ENVIRONMENT

OS X Sierra and CentOS 7

SUMMARY

Ansible's ability to run simple commands (e.g. ansible * --list-hosts) becomes geometrically slower as host counts in inventory increase. This becomes a major concern in large environments. For example, a 5,000 test host inventory takes ~57s to parse. At 10,000 it takes over 6 minutes. Ansible 2.2 and earlier performed the same feat @ 26,000 hosts in less than 4 seconds.

STEPS TO REPRODUCE

First, create a large simple inventory, e.g.:

rm foo; for i in {1..9000}; do echo "host-$i.foo.com" >> foo; done

Then, profile via e.g.:

python -m cProfile -s cumtime /usr/local/bin/ansible -i foo \* --list-hosts

EXPECTED RESULTS

In 2.2 and earlier, we'd regularly process static and dynamic host inventories with 26,000 hosts and ~338 groupings (and groups of groups) in about 5s.

ACTUAL RESULTS

At 1,000 hosts, things are reasonable @ 3s:

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.002 0.002 3.084 3.084 ansible:21(<module>)
1 0.002 0.002 2.080 2.080 adhoc.py:93(run)
1 0.000 0.000 2.034 2.034 __init__.py:767(_play_prereqs)
1 0.000 0.000 2.034 2.034 manager.py:120(__init__)
1 0.000 0.000 2.032 2.032 manager.py:193(parse_sources)
1 0.000 0.000 1.652 1.652 manager.py:217(parse_source)
1 0.000 0.000 1.652 1.652 ini.py:96(parse)
1 0.004 0.004 1.650 1.650 ini.py:136(_parse)
1000 0.002 0.000 1.562 0.002 __init__.py:82(populate_host_vars)
1000 0.108 0.000 1.560 0.002 data.py:176(add_host)
1005/1004 0.006 0.000 0.820 0.001 group.py:140(get_hosts)
1002/1001 0.459 0.000 0.814 0.001 group.py:146(_get_hosts)
1498500 0.638 0.000 0.740 0.000 host.py:39(__eq__)
2000 0.207 0.000 0.720 0.000 group.py:114(add_host)
1 0.001 0.001 0.669 0.669 constants.py:6(<module>)
1 0.000 0.000 0.630 0.630 manager.py:162(__init__)
1 0.000 0.000 0.620 0.620 __init__.py:87(safe_load)
1 0.004 0.004 0.620 0.620 __init__.py:64(load)
1 0.000 0.000 0.616 0.616 constructor.py:35(get_single_data)
1 0.000 0.000 0.589 0.589 composer.py:29(get_single_node)
1 0.000 0.000 0.589 0.589 composer.py:50(compose_document)
3924/1 0.011 0.000 0.589 0.589 composer.py:63(compose_node)
546/1 0.006 0.000 0.589 0.589 composer.py:117(compose_mapping_node)
12449 0.015 0.000 0.546 0.000 parser.py:94(check_event)
29732 0.036 0.000 0.447 0.000 scanner.py:113(check_token)
1 0.004 0.004 0.378 0.378 data.py:105(reconcile_inventory)
1001 0.003 0.000 0.373 0.000 data.py:225(add_child)
6242 0.021 0.000 0.334 0.000 scanner.py:153(fetch_more_tokens)
1009000 0.263 0.000 0.320 0.000 host.py:47(__hash__)
380 0.002 0.000 0.266 0.001 composer.py:99(compose_sequence_node)

At 5,000 hosts, things are slowing down substantially to about a minute:

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.002 0.002 57.737 57.737 ansible:21(<module>)
1 0.009 0.009 56.702 56.702 adhoc.py:93(run)
1 0.000 0.000 56.461 56.461 __init__.py:767(_play_prereqs)
1 0.000 0.000 56.461 56.461 manager.py:120(__init__)
1 0.000 0.000 56.460 56.460 manager.py:193(parse_sources)
1 0.000 0.000 45.775 45.775 manager.py:217(parse_source)
1 0.000 0.000 45.775 45.775 ini.py:96(parse)
1 0.046 0.046 45.772 45.772 ini.py:136(_parse)
5000 0.014 0.000 45.119 0.009 __init__.py:82(populate_host_vars)
5000 2.853 0.001 45.105 0.009 data.py:176(add_host)
5005/5004 0.189 0.000 23.936 0.005 group.py:140(get_hosts)
5002/5001 12.883 0.003 23.747 0.005 group.py:146(_get_hosts)
37492500 20.004 0.000 22.682 0.000 host.py:39(__eq__)
10000 5.722 0.001 21.009 0.002 group.py:114(add_host)
1 0.035 0.035 10.683 10.683 data.py:105(reconcile_inventory)
5001 0.029 0.000 10.643 0.002 data.py:225(add_child)
25045000 8.344 0.000 9.901 0.000 host.py:47(__hash__)
37635991/37635990 2.723 0.000 2.723 0.000 {isinstance}
25046868 1.557 0.000 1.557 0.000 {hash}
12591094 0.988 0.000 0.988 0.000 {method 'append' of 'list' objects}
1 0.001 0.001 0.698 0.698 constants.py:6(<module>)

At 10,000 hosts, we're approaching 4 minutes:

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.002 0.002 220.624 220.624 ansible:21(<module>)
1 0.017 0.017 219.489 219.489 adhoc.py:93(run)
1 0.000 0.000 219.024 219.024 __init__.py:767(_play_prereqs)
1 0.000 0.000 219.024 219.024 manager.py:120(__init__)
1 0.000 0.000 219.023 219.023 manager.py:193(parse_sources)
1 0.000 0.000 178.143 178.143 manager.py:217(parse_source)
1 0.000 0.000 178.142 178.142 ini.py:96(parse)
1 0.098 0.098 178.139 178.139 ini.py:136(_parse)
10000 0.031 0.000 176.771 0.018 __init__.py:82(populate_host_vars)
10000 10.766 0.001 176.740 0.018 data.py:176(add_host)
10005/10004 0.769 0.000 94.028 0.009 group.py:140(get_hosts)
10002/10001 50.390 0.005 93.259 0.009 group.py:146(_get_hosts)
149985000 79.268 0.000 89.636 0.000 host.py:39(__eq__)
20000 21.927 0.001 81.900 0.004 group.py:114(add_host)
1 0.055 0.055 40.878 40.878 data.py:105(reconcile_inventory)
10001 0.046 0.000 40.815 0.004 data.py:225(add_child)
100090000 33.460 0.000 39.429 0.000 host.py:47(__hash__)
150193491/150193490 10.438 0.000 10.438 0.000 {isinstance}
100091868 5.969 0.000 5.969 0.000 {hash}
50128594 3.482 0.000 3.482 0.000 {method 'append' of 'list' objects}
10000 0.036 0.000 1.222 0.000 ini.py:276(_parse_host_definition)
10000 0.052 0.000 0.976 0.000 shlex.py:32(shlex_split)
1 0.001 0.001 0.762 0.762 constants.py:6(<module>)

At 25,000 hosts, we're almost to a half hour:

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.004 0.004 1725.068 1725.068 ansible:21(<module>)
1 0.067 0.067 1724.017 1724.017 adhoc.py:93(run)
1 0.000 0.000 1722.428 1722.428 __init__.py:767(_play_prereqs)
1 0.000 0.000 1722.428 1722.428 manager.py:120(__init__)
1 0.001 0.001 1722.427 1722.427 manager.py:193(parse_sources)
1 0.001 0.001 1215.534 1215.534 manager.py:217(parse_source)
1 0.000 0.000 1215.533 1215.533 ini.py:96(parse)
1 0.296 0.296 1215.527 1215.527 ini.py:136(_parse)
25000 0.089 0.000 1211.709 0.048 __init__.py:82(populate_host_vars)
25000 71.746 0.003 1211.620 0.048 data.py:176(add_host)
937462500 724.016 0.000 799.531 0.000 host.py:39(__eq__)
50000 208.628 0.004 794.945 0.016 group.py:114(add_host)
25005/25004 5.568 0.000 632.880 0.025 group.py:140(get_hosts)
25002/25001 335.514 0.013 627.313 0.025 group.py:146(_get_hosts)
1 0.425 0.425 506.889 506.889 data.py:105(reconcile_inventory)
25001 0.400 0.000 506.415 0.020 data.py:225(add_child)
625225000 229.500 0.000 268.754 0.000 host.py:47(__hash__)
937865991/937865990 75.662 0.000 75.662 0.000 {isinstance}
625226868 39.254 0.000 39.254 0.000 {hash}
312741094 23.178 0.000 23.178 0.000 {method 'append' of 'list' objects}
25000 0.099 0.000 3.366 0.000 ini.py:276(_parse_host_definition)
25000 0.151 0.000 2.709 0.000 shlex.py:32(shlex_split)
75001/50001 2.056 0.000 2.056 0.000 group.py:134(clear_hosts_cache)
25000 0.276 0.000 2.042 0.000 shlex.py:274(split)
50000 0.106 0.000 1.431 0.000 shlex.py:268(next)
25000 0.206 0.000 1.419 0.000 path.py:76(basedir)
50000 0.079 0.000 1.325 0.000 shlex.py:88(get_token)

With ANSIBLE_DEBUG on, we can see how long it's taking to process each host, which is a clue that we've got a recursion concern each time we add another host.

Initially, about 1ms per:

75383 1505780568.42386: Added host host-122.foo.com to inventory
75383 1505780568.42414: Added host host-122.foo.com to group ungrouped
75383 1505780568.42429: set inventory_file for host-123.foo.com
75383 1505780568.42440: set inventory_dir for host-123.foo.com
75383 1505780568.42447: Added host host-123.foo.com to inventory
75383 1505780568.42492: Added host host-123.foo.com to group ungrouped
75383 1505780568.42511: set inventory_file for host-124.foo.com
75383 1505780568.42522: set inventory_dir for host-124.foo.com
75383 1505780568.42530: Added host host-124.foo.com to inventory

Around 20k, about 64ms per:

75383 1505781298.73271: Added host host-20315.foo.com to inventory
75383 1505781298.79559: Added host host-20315.foo.com to group ungrouped
75383 1505781298.79593: set inventory_file for host-20316.foo.com
75383 1505781298.79606: set inventory_dir for host-20316.foo.com
75383 1505781298.79618: Added host host-20316.foo.com to inventory
75383 1505781298.85869: Added host host-20316.foo.com to group ungrouped
75383 1505781298.85900: set inventory_file for host-20317.foo.com
75383 1505781298.85913: set inventory_dir for host-20317.foo.com
75383 1505781298.85919: Added host host-20317.foo.com to inventory

Issue 2:
with ansible 2.0.0-beta2 I get:

# ansible-pull --url myurl --key-file key
Usage: ansible-pull <host-pattern> [options]
....
ERROR! Missing target hosts

Man page still indicates that it should default to local.yml. Seems something is messed up here.