Repository: ansible/ansible

Issue:
High CPU Usage
**Issue Type:** Bug/Annoyance
**Ansible Version:**

```
ansible 2.0.0.2
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides

```

**Ansible Configuration:** 
Default configuration supplied from Ubuntu PPA except uncommented:

```
[privilege_escalation]
become=True
become_method=sudo
```

**Environment:**
Ubuntu 15.10, Python 2.7.10 running the playbook.  The target host in group is ArchLinux with python2 version Python 2.7.11

**Summary:**
When I run a playbook that runs a long task (say backup target machine, or do virus scanning), the machine running the playbook spawns 2 "python /usr/bin/ansible-playbook stress.yml" processes that consume a high amount of CPU (15-20% each on my machine which is not that powerful).  Since the machine running the playbook is pretty much waiting for remote task to complete it shouldn't use up this much CPU.

It doesn't seem to matter whether the host group contains 1 machine or more.  This happens when playbook is executed in serial mode as well.

**Steps To Reproduce:**
For some host group, let's say "stress" (where the machine running ansible command is not part of that group), execute something that takes a while on the remote machine.  Eg:
`ansible stress -m shell -a "dd if\=/dev/random of\=/dev/null bs\=1M count\=2"`
or
`ansible stress -m shell -a "stress --cpu 8 --timeout 60s"`
Monitor cpu usage with top on the machine that ran ansible command.

**Expected Results:**
CPU usage closer to idle.  I did not have this problem with Ansible 1.9.2.

**Actual Results:**
Fairly high CPU consumption.



Discussion:
Maybe related to #12239

I see a very different behaviour as well. I have a small playbook, which will:
- add an APT key and repository
- uninstall 2 packages
- install a new one 
- create 5 files from template. 

When executing on 66 hosts with 20 forks for the second time (nothing changes on the target hosts):
With Ansible 1.9.4 I get:
real    0m41.953s
user    0m33.536s
sys     0m20.690s

With Ansible 2.0.0.2 I get:
real    1m18.602s
user    2m6.828s
sys     1m7.319s

Besides the difference in realtime 40s vs. 78s (+100%), the sum of user and sys time is 53s vs. 193s (almost +300%).

During the run with Ansible 2 I see 2 Python processes which hog the CPUs.

Pinging 126 hosts with 20 forks:
1.9.4:  3.41 real         3.42 user         2.37 sys
2.0.0.2: 5.69 real         7.25 user         6.52 sys

Hello,

I have this kind of problem when I try to migrate to ansible v2.
My inventory contains about 2800 hosts and some groups (the JSON file size is about 1MB) and my test playbook is very simple : 
1. Get fact
2. Fail on some hosts (depends on one fact variable)
3. Ping remaining hosts

I'm running this playbook with the fork option set to 100. With ansible version 1.9.4, the master process was forking 100 process at each step and the playbook was fast.

With the 2.0 version, the master process is now forking a process for each host at each steps. This is more than 20 000 forks in my use case. The master process becomes a bottleneck : forking a process with a high memory usage has a huge cost in the kernel. A second bottleneck is the get_vars() function, taking too much as well. As a result, the number of 100 forks is never reached, since the master is too busy to run more fork.

I have some profiling result. The first one is for memory consumption, benchmarked on the ansible devel branch in yesterday state. In the last step on my playbook, the memory profiling looks like that: 

Partition of a set of 4195831 objects. Total size = 707847728 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0 325780   8 304524640  43 304524640  43 dict (no owner)
     1 2009973  48 177318144  25 481842784  68 unicode
     2 1267753  30 146429608  21 628272392  89 ansible.vars.unsafe_proxy.AnsibleUnsafeText
     3 179677   4 35907816   5 664180208  94 list
     4   6235   0  6534280   1 670714488  95 dict of ansible.playbook.task.Task

That is more than 100 dictionaries and than 700 unicode by host. Most of then are probably generated by the fact polling. By using the redis fact caching, the memory consumption is really lower:

Partition of a set of 661733 objects. Total size = 163487792 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  61474   9 64969648  40  64969648  40 dict (no owner)
     1 137901  21 27826984  17  92796632  57 list
     2 155627  24 17861864  11 110658496  68 unicode

Thanks to this lower memory consumption, the fork is faster (see next benchmark bellow) and the playbook run faster (without be fast like before).

The second benchmark was done with the kernprof module. See three files as attachment (version 1.9.4, devel without redis, devel with redis). I'm not sure to understand why we call 87 millions of host.get_name() (31200 calls by host?), there are probably some improvement to work on.

**Profiling on 1.9.4:**

![playbook-v1 9 4](https://cloud.githubusercontent.com/assets/17849937/13822825/65d35f7e-eba7-11e5-92ca-e1d371404968.png)

**Profiling on devel without redis cache backend:**
![playbook-on-devel](https://cloud.githubusercontent.com/assets/17849937/13822824/65d0fdc4-eba7-11e5-8f2c-381f5824df4b.png)

**Profiling on devel with redis cache backend enabled:**
![playbook-on-devel-with-redis](https://cloud.githubusercontent.com/assets/17849937/13822823/65cf7ff8-eba7-11e5-8db8-97b6852eb8f0.png)

I can reproduce this issue with simple playbook:

```
---
- hosts: localhost
  tasks:
    - pause: minutes=1
```

Should just sleep and do no CPU load, but i see two ansible-playbook process which utilising CPU by 15% each, and one process which actually sleeping. 

Looks like this issue appeared since this commit:
https://github.com/ansible/ansible/commit/5cbeab5a3cbd55a7252c94c74c57ca69aade7846#diff-98f31dea71e7d55980452e925c229e95R146
https://github.com/ansible/ansible/commit/5cbeab5a3cbd55a7252c94c74c57ca69aade7846#diff-b893b8b8319cd4518208dd4fe53f3d7bR328

Sleep for 0.0001 seconds when polling for results is too few, which causes high CPU utilisation.

I tried to change to 0.001 which causes 5% CPU load for process.
And sleep for 0.01 works ok for me.


Resolving patch:
diff --git a/lib/ansible/executor/process/result.py b/lib/ansible/executor/process/result.py
--- a/lib/ansible/executor/process/result.py
+++ b/lib/ansible/executor/process/result.py
@@ -106,7 +106,7 @@ def run(self):
             try:
                 result = self._read_worker_result()
                 if result is None:
-                    time.sleep(0.0001)
+                    time.sleep(0.01)
                     continue
 
                 # send callbacks for 'non final' results
diff --git a/lib/ansible/plugins/strategy/__init__.py b/lib/ansible/plugins/strategy/__init__.py
--- a/lib/ansible/plugins/strategy/__init__.py
+++ b/lib/ansible/plugins/strategy/__init__.py
@@ -196,7 +196,7 @@ def _queue_task(self, host, task, task_vars, play_context):
                 self._cur_worker += 1
                 if self._cur_worker >= len(self._workers):
                     self._cur_worker = 0
-                    time.sleep(0.0001)
+                    time.sleep(0.01)
                 if queued:
                     break
 
@@ -389,7 +389,7 @@ def get_original_host(host):
                     raise AnsibleError("unknown result message received: %s" % result[0])
 
             except Queue.Empty:
-                time.sleep(0.0001)
+                time.sleep(0.01)
 
             if one_pass:
                 break
@@ -408,7 +408,7 @@ def _wait_on_pending_results(self, iterator):
         while self._pending_results > 0 and not self._tqm._terminated:
             results = self._process_pending_results(iterator)
             ret_results.extend(results)
-            time.sleep(0.0001)
+            time.sleep(0.01)
         display.debug("no more pending results, returning what we have")
 
         return ret_results


Test patch: