Repository: ansible/ansible

Issue:
Performance regression in 2.5/devel
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
task executor

##### ANSIBLE VERSION
```
ansible 2.5.0rc3 (stable-2.5 144e5ae65a) last updated 2018/03/17 08:54:48 (GMT -500)
```

##### CONFIGURATION
```
$ ansible-config dump --only-changed
ANSIBLE_PIPELINING(/etc/ansible/ansible.cfg) = True
DEFAULT_SCP_IF_SSH(/etc/ansible/ansible.cfg) = True
RETRY_FILES_ENABLED(/etc/ansible/ansible.cfg) = False
```

##### OS / ENVIRONMENT
N/A

##### SUMMARY
There's a noticeable performance regression in `2.5` (and current `devel`) compared to `2.4.3`.

##### STEPS TO REPRODUCE

Playbook:

```yaml
- hosts: localhost
  gather_facts: no
  tasks:
    - debug: msg="foo"
      with_sequence: count=5000
```

`2.4.3`:

```
$ (ansible --version; time ansible-playbook foo.yml &>/dev/null)
ansible 2.4.3.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/agaffney/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]

real	0m17.287s
user	0m19.416s
sys	0m2.156s
```

`2.5` (current `stable-2.5`):

```
$ (source ~/dev/ansible/hacking/env-setup &>/dev/null; ansible --version; time ansible-playbook foo.yml &>/dev/null)
ansible 2.5.0rc3 (stable-2.5 144e5ae65a) last updated 2018/03/17 08:54:48 (GMT -500)
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/agaffney/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/agaffney/dev/ansible/lib/ansible
  executable location = /home/agaffney/dev/ansible/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]

real	1m1.102s
user	1m4.412s
sys	0m2.556s
```

current `devel`:

```
$ (source ~/dev/ansible/hacking/env-setup &>/dev/null; ansible --version; time ansible-playbook foo.yml &>/dev/null)
ansible 2.6.0 (devel e541bed186) last updated 2018/03/16 14:33:21 (GMT -500)
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/agaffney/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/agaffney/dev/ansible/lib/ansible
  executable location = /home/agaffney/dev/ansible/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]

real	1m1.066s
user	1m4.532s
sys	0m2.592s
```

##### EXPECTED RESULTS
Nice, small runtimes

##### ACTUAL RESULTS
Not so nice and small runtimes


Discussion:
Files identified in the description:
* [lib/ansible/executor/task_executor.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/executor/task_executor.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
This appears to have been introduced in https://github.com/ansible/ansible/commit/bbd6b8bb42cc8fa48157ce5db0efb0a836c94917 (#31677)
@bcoca, the "breaking" commit belongs to you, and it's a bit of a doozy
This is *probably* due to the loading of the docstring and config definitions from the shell plugin. It seems to happen on each task, plus it happens (seemingly) needlessly for the `debug` action (it doesn't use the shell). When you multiply this negligible overhead by 5000, it becomes a noticeable amount of time.
Comparison of debug output:

`2.4.3`

```
 27582 1521379679.12982: Loading ShellModule 'csh' from /home/agaffney/dev/ansible/lib/ansible/plugins/shell/csh.py (found_in_cache=True, class_only=False)
 27582 1521379679.12987: Loading ShellModule 'fish' from /home/agaffney/dev/ansible/lib/ansible/plugins/shell/fish.py (found_in_cache=True, class_only=False)
 27582 1521379679.12991: Loading ShellModule 'powershell' from /home/agaffney/dev/ansible/lib/ansible/plugins/shell/powershell.py (found_in_cache=True, class_only=False)
 27582 1521379679.12994: Loading ShellModule 'sh' from /home/agaffney/dev/ansible/lib/ansible/plugins/shell/sh.py (found_in_cache=True, class_only=False)
 27582 1521379679.12997: Loading ShellModule 'sh' from /home/agaffney/dev/ansible/lib/ansible/plugins/shell/sh.py (found_in_cache=True, class_only=False)
```

`devel`

```
 27096 1521378973.29052: Loading ShellModule 'csh' from /home/agaffney/dev/ansible/lib/ansible/plugins/shell/csh.py (found_in_cache=True, class_only=False)
 27096 1521378973.29092: assigned :doc
 27096 1521378973.29099: Loading ModuleDocFragment 'shell_common' from /home/agaffney/dev/ansible/lib/ansible/utils/module_docs_fragments/shell_common.py (found_in_cache=True, class_only=False)
 27096 1521378973.29212: Loaded config def from plugin (shell/csh)
 27096 1521378973.29217: Loading ShellModule 'fish' from /home/agaffney/dev/ansible/lib/ansible/plugins/shell/fish.py (found_in_cache=True, class_only=False)
 27096 1521378973.29277: assigned :doc
 27096 1521378973.29286: Loading ModuleDocFragment 'shell_common' from /home/agaffney/dev/ansible/lib/ansible/utils/module_docs_fragments/shell_common.py (found_in_cache=True, class_only=False)
 27096 1521378973.29404: Loaded config def from plugin (shell/fish)
 27096 1521378973.29409: Loading ShellModule 'powershell' from /home/agaffney/dev/ansible/lib/ansible/plugins/shell/powershell.py (found_in_cache=True, class_only=False)
 27096 1521378973.29640: assigned :doc
 27096 1521378973.29657: Loaded config def from plugin (shell/powershell)
 27096 1521378973.29661: Loading ShellModule 'sh' from /home/agaffney/dev/ansible/lib/ansible/plugins/shell/sh.py (found_in_cache=True, class_only=False)
 27096 1521378973.29708: assigned :doc
 27096 1521378973.29715: Loading ModuleDocFragment 'shell_common' from /home/agaffney/dev/ansible/lib/ansible/utils/module_docs_fragments/shell_common.py (found_in_cache=True, class_only=False)
 27096 1521378973.29798: Loaded config def from plugin (shell/sh)
 27096 1521378973.29802: Loading ShellModule 'sh' from /home/agaffney/dev/ansible/lib/ansible/plugins/shell/sh.py (found_in_cache=True, class_only=False)
 27096 1521378973.29845: assigned :doc
 27096 1521378973.29852: Loading ModuleDocFragment 'shell_common' from /home/agaffney/dev/ansible/lib/ansible/utils/module_docs_fragments/shell_common.py (found_in_cache=True, class_only=False)
 27096 1521378973.29942: Loaded config def from plugin (shell/sh)
```

What previously took `~0.0002s` now takes `~0.009s`, which is 45x. Since this happens for every task, doing something 5000 times (such as in my test case) adds 45-50s to the runtime.

Resolving patch:
diff --git a/lib/ansible/plugins/loader.py b/lib/ansible/plugins/loader.py
--- a/lib/ansible/plugins/loader.py
+++ b/lib/ansible/plugins/loader.py
@@ -348,9 +348,6 @@ def _load_module_source(self, name, path):
 
     def _update_object(self, obj, name, path):
 
-        # load plugin config data
-        self._load_config_defs(name, path)
-
         # set extra info on the module, in case we want it later
         setattr(obj, '_original_path', path)
         setattr(obj, '_load_name', name)
@@ -394,6 +391,10 @@ def get(self, name, *args, **kwargs):
                     return None
                 raise
 
+        # load plugin config data
+        if not found_in_cache:
+            self._load_config_defs(name, path)
+
         self._update_object(obj, name, path)
         return obj
 
@@ -467,6 +468,10 @@ def all(self, *args, **kwargs):
                 except TypeError as e:
                     display.warning("Skipping plugin (%s) as it seems to be incomplete: %s" % (path, to_text(e)))
 
+            # load plugin config data
+            if not found_in_cache:
+                self._load_config_defs(name, path)
+
             self._update_object(obj, name, path)
             yield obj
 


Test patch: