Repository: ansible/ansible

Issue:
template: significant slowdown after upgrade to 2.3
We have noticed that our production plays run significantly slower on 2.3.2, compared to 2.2.0. It seems, it is caused by `template` tasks (it is worse if they have `with` loops).

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME

template module

##### ANSIBLE VERSION

```
$ ansible --version
ansible 2.3.2.0
  config file = /home/zarnovic/.ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.13 (default, Jun 26 2017, 10:20:05) [GCC 7.1.1 20170622 (Red Hat 7.1.1-3)]
```

##### CONFIGURATION

Default (empty) configuration. In the tests below, I have included also results with `pipelining=True`.

##### OS / ENVIRONMENT

Target host is `m1.small` Ubuntu 16.04.2 on AWS. EC2 hosts generally have bad latency, which contribute to the percieved speed.

The issue is not platform specific, AFAIK.

##### SUMMARY

Template tasks are significantly slower (30-40%) compared to times on Ansible 2.2. These results are from an AWS EC2 instance. In other environments, the slowdown might be less dramatic.

Analyzing the issue, I have found the commit causing the problem: cdd6ab000b2c85d5dd0dc86c2ae8580e6978b819. It is related to https://github.com/ansible/ansible/pull/24477

My understaning of the code change, is that if you are using `template` where destination is a file, then ansible will do one additional remote command to check if the target exists and it is a directory. This will slowdown the task. It is worse, if you have multiple templates files (in a loop) and you are copying them to target host, which is slow by itself.

Time to run Ansible play in various configurations:
```
ansible 2.2.3 (pipelining=False): 17.671, 18.843, 17.411 ~ avg 18.0s
ansible 2.3.2 (pipelining=False): 25.603, 26.426, 25.423 ~ avg 25.8s (+43%)

ansible 2.2.3 (pipelining=True): 9.320, 9.104, 9.279 ~ avg 9.23s
ansible 2.3.2 (pipelining=True): 12.778, 12.127, 12.775 ~ avg 12.56s (+36%)
```

##### STEPS TO REPRODUCE

1. create some empty jinja template files

```bash
touch foo{0..4}.j2
```

2. create an inventory file with your target host

```
echo "foo.example.com" > inventory
```

3. playbook
```yaml
---
- name: 'Foo'
  hosts: all

  tasks:
  - name: 'copy some files'
    template:
      src: "{{ item }}"
      dest: "/tmp/{{ item | basename | splitext | first }}"
      #dest: /tmp/
    with_fileglob: "*.j2"
```

4. run
```bash
time ansible-playbook -i inventory site.yaml
```

##### EXPECTED RESULTS

Times on Ansible 2.2.3:
```bash
$ time ansible-playbook -i inventory site.yaml 
...
real	0m17.295s
user	0m4.479s
sys	0m0.848s
```

##### ACTUAL RESULTS

Times on Ansible 2.3.2:
```
$ time ansible-playbook -i inventory site.yaml 

PLAY [Foo] ***************************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************
ok: [foo.example.com]

TASK [copy some files] ***************************************************************************************************************************************
ok: [foo.example.com] => (item=/home/zarnovic/tmp/ansible_test/foo3.j2)
ok: [foo.example.com] => (item=/home/zarnovic/tmp/ansible_test/foo2.j2)
ok: [foo.example.com] => (item=/home/zarnovic/tmp/ansible_test/foo1.j2)
ok: [foo.example.com] => (item=/home/zarnovic/tmp/ansible_test/foo0.j2)
ok: [foo.example.com] => (item=/home/zarnovic/tmp/ansible_test/foo4.j2)

PLAY RECAP ***************************************************************************************************************************************************
foo.example.com           : ok=2    changed=0    unreachable=0    failed=0   


real	0m25.635s
user	0m4.340s
sys	0m1.226s
```
Notice the time difference: 17.3s vs 25.6s !!


Discussion:
This isn't a bug as ansible's behaviour is more correct now.  I'll see if I can get any of that speed difference back but we have to have correct behaviour even if it is slower.

Resolving patch:
diff --git a/lib/ansible/modules/files/template.py b/lib/ansible/modules/files/template.py
--- a/lib/ansible/modules/files/template.py
+++ b/lib/ansible/modules/files/template.py
@@ -81,6 +81,13 @@
         if the destination does not exist.
     type: bool
     default: 'yes'
+  follow:
+    description:
+      - This flag indicates that filesystem links in the destination, if they exist, should be followed.
+      - Previous to Ansible 2.4, this was hardcoded as C(yes).
+    type: bool
+    default: 'no'
+    version_added: "2.4"
 notes:
   - For Windows you can use M(win_template) which uses '\r\n' as C(newline_sequence).
   - Including a string that uses a date in the template will result in the template being marked 'changed' each time
diff --git a/lib/ansible/plugins/action/copy.py b/lib/ansible/plugins/action/copy.py
--- a/lib/ansible/plugins/action/copy.py
+++ b/lib/ansible/plugins/action/copy.py
@@ -219,6 +219,12 @@ def _copy_file(self, source_full, source_rel, content, content_tempfile,
         else:
             dest_file = self._connection._shell.join_path(dest)
 
+        # Create a tmp path if missing only if this is not recursive.
+        # If this is recursive we already have a tmp path.
+        if delete_remote_tmp:
+            if tmp is None or "-tmp-" not in tmp:
+                tmp = self._make_tmp_path()
+
         # Attempt to get remote file info
         dest_status = self._execute_remote_stat(dest_file, all_vars=task_vars, follow=follow, tmp=tmp, checksum=force)
 
@@ -246,19 +252,13 @@ def _copy_file(self, source_full, source_rel, content, content_tempfile,
         if local_checksum != dest_status['checksum']:
             # The checksums don't match and we will change or error out.
 
-            # Create a tmp path if missing only if this is not recursive.
-            # If this is recursive we already have a tmp path.
-            if delete_remote_tmp:
-                if tmp is None or "-tmp-" not in tmp:
-                    tmp = self._make_tmp_path()
-
             if self._play_context.diff and not raw:
                 result['diff'].append(self._get_diff_data(dest_file, source_full, task_vars))
 
             if self._play_context.check_mode:
                 self._remove_tempfile_if_content_defined(content, content_tempfile)
-                module_return = dict(changed=True)
-                return module_return
+                result['changed'] = True
+                return result
 
             # Define a remote directory that we will copy the file to.
             tmp_src = self._connection._shell.join_path(tmp, 'source')
@@ -321,7 +321,7 @@ def _copy_file(self, source_full, source_rel, content, content_tempfile,
             # If checksums match, and follow = True, find out if 'dest' is a link. If so,
             # change it to point to the source of the link.
             if follow:
-                dest_status_nofollow = self._execute_remote_stat(dest_file, all_vars=task_vars, follow=False)
+                dest_status_nofollow = self._execute_remote_stat(dest_file, all_vars=task_vars, tmp=tmp, follow=False)
                 if dest_status_nofollow['islnk'] and 'lnk_source' in dest_status_nofollow.keys():
                     dest = dest_status_nofollow['lnk_source']
 
@@ -345,7 +345,8 @@ def _copy_file(self, source_full, source_rel, content, content_tempfile,
         if not module_return.get('checksum'):
             module_return['checksum'] = local_checksum
 
-        return module_return
+        result.update(module_return)
+        return result
 
     def _get_file_args(self):
         new_module_args = {'recurse': False}
diff --git a/lib/ansible/plugins/action/template.py b/lib/ansible/plugins/action/template.py
--- a/lib/ansible/plugins/action/template.py
+++ b/lib/ansible/plugins/action/template.py
@@ -18,10 +18,12 @@
 __metaclass__ = type
 
 import os
+import shutil
+import tempfile
 
 from ansible import constants as C
 from ansible.errors import AnsibleError, AnsibleFileNotFound
-from ansible.module_utils._text import to_text
+from ansible.module_utils._text import to_bytes, to_text
 from ansible.module_utils.parsing.convert_bool import boolean
 from ansible.plugins.action import ActionBase
 from ansible.template import generate_ansible_template_vars
@@ -58,6 +60,7 @@ def run(self, tmp=None, task_vars=None):
         source = self._task.args.get('src', None)
         dest = self._task.args.get('dest', None)
         force = boolean(self._task.args.get('force', True), strict=False)
+        follow = boolean(self._task.args.get('follow', False), strict=False)
         state = self._task.args.get('state', None)
         newline_sequence = self._task.args.get('newline_sequence', self.DEFAULT_NEWLINE_SEQUENCE)
         variable_start_string = self._task.args.get('variable_start_string', None)
@@ -92,20 +95,6 @@ def run(self, tmp=None, task_vars=None):
         if 'failed' in result:
             return result
 
-        # Expand any user home dir specification
-        dest = self._remote_expand_user(dest)
-
-        directory_prepended = False
-        if dest.endswith(os.sep):
-            # Optimization.  trailing slash means we know it's a directory
-            directory_prepended = True
-            dest = self._connection._shell.join_path(dest, os.path.basename(source))
-        else:
-            # Find out if it's a directory
-            dest_stat = self._execute_remote_stat(dest, task_vars, True, tmp=tmp)
-            if dest_stat['exists'] and dest_stat['isdir']:
-                dest = self._connection._shell.join_path(dest, os.path.basename(source))
-
         # Get vault decrypted tmp file
         try:
             tmp_source = self._loader.get_real_file(source)
@@ -160,70 +149,35 @@ def run(self, tmp=None, task_vars=None):
         finally:
             self._loader.cleanup_tmp_file(tmp_source)
 
-        if not tmp:
-            tmp = self._make_tmp_path()
-
-        local_checksum = checksum_s(resultant)
-        remote_checksum = self.get_checksum(dest, task_vars, not directory_prepended, source=source, tmp=tmp)
-        if isinstance(remote_checksum, dict):
-            # Error from remote_checksum is a dict.  Valid return is a str
-            result.update(remote_checksum)
-            return result
-
-        diff = {}
-        new_module_args = self._task.args.copy()
-
-        # remove newline_sequence from standard arguments
-        new_module_args.pop('newline_sequence', None)
-        new_module_args.pop('block_start_string', None)
-        new_module_args.pop('block_end_string', None)
-        new_module_args.pop('variable_start_string', None)
-        new_module_args.pop('variable_end_string', None)
-        new_module_args.pop('trim_blocks', None)
-
-        if (remote_checksum == '1') or (force and local_checksum != remote_checksum):
-
-            result['changed'] = True
-            # if showing diffs, we need to get the remote value
-            if self._play_context.diff:
-                diff = self._get_diff_data(dest, resultant, task_vars, source_file=False)
-
-            if not self._play_context.check_mode:  # do actual work through copy
-                xfered = self._transfer_data(self._connection._shell.join_path(tmp, 'source'), resultant)
-
-                # fix file permissions when the copy is done as a different user
-                self._fixup_perms2((tmp, xfered))
-
-                # run the copy module
-                new_module_args.update(
-                    dict(
-                        src=xfered,
-                        dest=dest,
-                        original_basename=os.path.basename(source),
-                        follow=True,
-                    ),
-                )
-                result.update(self._execute_module(module_name='copy', module_args=new_module_args, task_vars=task_vars, tmp=tmp, delete_remote_tmp=False))
-
-            if result.get('changed', False) and self._play_context.diff:
-                result['diff'] = diff
+        new_task = self._task.copy()
+        new_task.args.pop('newline_sequence', None)
+        new_task.args.pop('block_start_string', None)
+        new_task.args.pop('block_end_string', None)
+        new_task.args.pop('variable_start_string', None)
+        new_task.args.pop('variable_end_string', None)
+        new_task.args.pop('trim_blocks', None)
+        try:
+            tempdir = tempfile.mkdtemp()
+            result_file = os.path.join(tempdir, os.path.basename(source))
+            with open(result_file, 'wb') as f:
+                f.write(to_bytes(resultant, errors='surrogate_or_strict'))
 
-        else:
-            # when running the file module based on the template data, we do
-            # not want the source filename (the name of the template) to be used,
-            # since this would mess up links, so we clear the src param and tell
-            # the module to follow links.  When doing that, we have to set
-            # original_basename to the template just in case the dest is
-            # a directory.
-            new_module_args.update(
+            new_task.args.update(
                 dict(
-                    src=None,
-                    original_basename=os.path.basename(source),
-                    follow=True,
+                    src=result_file,
+                    dest=dest,
+                    follow=follow,
                 ),
             )
-            result.update(self._execute_module(module_name='file', module_args=new_module_args, task_vars=task_vars, tmp=tmp, delete_remote_tmp=False))
-
-        self._remove_tmp_path(tmp)
+            copy_action = self._shared_loader_obj.action_loader.get('copy',
+                                                                    task=new_task,
+                                                                    connection=self._connection,
+                                                                    play_context=self._play_context,
+                                                                    loader=self._loader,
+                                                                    templar=self._templar,
+                                                                    shared_loader_obj=self._shared_loader_obj)
+            result.update(copy_action.run(task_vars=task_vars))
+        finally:
+            shutil.rmtree(tempdir)
 
         return result


Test patch: