Repository: ansible/ansible

Issue:
Template module not accepting source that contains UTF8 characters
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
template

##### ANSIBLE VERSION
```
$ ansible --version
ansible 2.3.1.0
  config file = /work/ansible/configuration/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.5 (default, Nov  6 2016, 00:28:07) [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]
```

##### OS / ENVIRONMENT
Ansible runs on CentOS 7.3
Target system on RHEL 6.9

##### SUMMARY
This looks like bug #10110:
Using the template module, when the source name contains an UTF8 character, the module fails with the following error.
Replacing template: with copy:, there is no error (the "copy" module has been used for more than a year and is just now being replaced by template).
If there is UTF8 only in the destination name, there is no error.


##### STEPS TO REPRODUCE

This sample triggers the error:
```
- name: Test UTF8
  template:
    src:    "/tmp/ééé"
    dest:   "/tmp/eee"
```
This sample works:
```
- name: Test UTF8
  template:
    src:    "/tmp/eee"
    dest:   "/tmp/ééé"
```


##### EXPECTED RESULTS
The template being copied to the target.

##### ACTUAL RESULTS

```
TASK [task : Test UTF8 *********************************************************************************************
[WARNING]: Failure using method (v2_runner_item_on_failed) in callback plugin (<ansible.plugins.callback.default.CallbackModule object at 0x35eb190>):
'ascii' codec can't decode byte 0xc3 in position 22: ordinal not in range(128
```



Discussion:
Files identified in the description:
* [lib/ansible/modules/files/template.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/files/template.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @abadger 
I cannot reproduce this error with a checkout of ansible-2.3.1.0 or the head of the current devel branch.
```
  - name: Remove test file
    file:
      state: absent
      path: "/tmp/eee"

  # Basic template with non-ascii names
  - name: Check that non-ascii source works
    template:
      src:    "/tmp/ééé"
      dest: '/tmp/eee'
```
Command run:
```
[pts/46@peru /srv/backup/ansible/ansible/stable-2.3]$ ansible-playbook test.yml            *[ecb38fdf73]  (11:02:05)
 [WARNING]: Host file not found: /etc/ansible/hosts

 [WARNING]: provided hosts list is empty, only localhost is available


PLAY [localhost] ****************************************************************************************************

TASK [Remove test file] *********************************************************************************************
ok: [localhost]

TASK [Check that non-ascii source and dest work] ********************************************************************
changed: [localhost]

PLAY RECAP **********************************************************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0 
```

```
[pts/46@peru /srv/backup/ansible/ansible/stable-2.3]$ ansible --version                    *[ecb38fdf73]  (11:02:07)
ansible 2.3.1.0 (detached HEAD ecb38fdf73) last updated 2018/04/26 10:52:03 (GMT -700)
  config file = 
  configured module search path = Default w/o overrides
  python version = 2.7.14 (default, Mar 14 2018, 13:36:31) [GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]
```

I note that the error message is actually a warning.  The warning seems to indicate that the template module encounters an error that we cannot see and then, when the callback attempts to display that error, the callback throws the decoding error.  So there seems to be two things wrong in your setup.

I tried introducing an error in my playbook (setting dest to an unwritable location) in hopes that I could reproduce your callback error but no luck.  The callback displays the error from the module instead of itself having an error.

I'll need more information from you on how exactly to reproduce the issue to move forward with debugging this.  In addition, 2.3.x is no longer maintained.  If you can try with a newer version of ansible that would be great.

needs_info
Hello,

I've been upgrading Ansible, and this still happens on the latest version. It's odd that you don't get it, as I've got no trouble reproducing it. I'm attaching a complete role with the issue, including one `copy` task that works, and an otherwise identical `template` task that fails.

```
$ ansible --version
ansible 2.5.1
   config file = /data/ansible/configuration/ansible.cfg
   configured module search path =
[u'/home/me/.ansible/plugins/modules',
u'/usr/share/ansible/plugins/modules']
   ansible python module location = /usr/lib/python2.7/site-packages/ansible
   executable location = /usr/bin/ansible
   python version = 2.7.5 (default, Aug  4 2017, 00:39:18) [GCC 4.8.5
20150623 (Red Hat 4.8.5-16)]
```

```
$ ansible-playbook test_template.yml
[DEPRECATION WARNING]: DEFAULT_SUDO_FLAGS option, In favor of Ansible Become,
which is a generic framework. See become_flags. , use become instead. This
feature will be removed in version 2.8. Deprecation warnings can be disabled by
setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: DEFAULT_SUDO_EXE option, In favor of Ansible Become,
which is a generic framework. See become_exe. , use become instead. This
feature will be removed in version 2.8. Deprecation warnings can be disabled by
setting deprecation_warnings=False in ansible.cfg.

PLAY [serveur] *****************************************************************

TASK [Gathering Facts] *********************************************************
ok: [serveur]

TASK [test_template : Copie] ***************************************************
ok: [serveur]

TASK [test_template : Modèle] **************************************************
fatal: [serveur]: FAILED! => {"changed": false, "msg": "UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 67: ordinal not in range(128)"}

PLAY RECAP *********************************************************************
serveur                    : ok=2    changed=0    unreachable=0    failed=1 
```

[role_test_template.tar.gz](https://github.com/ansible/ansible/files/1954590/role_test_template.tar.gz)


I'm still not able to reproduce this.  What are your locale settings (LC_* and LANG)?  I can get this error if I set the locale to an ascii-only locale but if I do that my output is different.  For instance:
```
TASK [test_template : Mod?le] ***************************************************************************************
```

So I don't think that's what's happening in your case.
I have some fixes for running under LC_ALL=C.... I'm still not sure what about your environment is leading to the problem you see but there's a good chance that they'll fix the traceback for you.  I'll post a PR shortly.
I'm using en_US.utf8 all round, both locally and on the remote server. I've tried with LC_ALL=C as well, this produces a related, but different error, note the `encode` instead of `decode`.
`copy` still works as expected.
```

TASK [test_template : Mod?le]  **************************************************
fatal: [serveur]: FAILED! => {"changed": false, "msg":  "UnicodeEncodeError: 'ascii' codec can't encode characters in position  65-67: ordinal not in range(128)"}
```
@L-B do you have any callbacks enabled in your Ansible config?
@L-B can you try applying the changes from https://github.com/ansible/ansible/pull/39424 to see what that does for your issue?  Those were the changes I had to make for it to work in the C locale.  Since I haven't figured out what the difference is in your environment that actually causes this, though, there could be some other problem triggering this.  I give it a 50-50 chance of fixing it.

Resolving patch:
diff --git a/lib/ansible/plugins/action/template.py b/lib/ansible/plugins/action/template.py
--- a/lib/ansible/plugins/action/template.py
+++ b/lib/ansible/plugins/action/template.py
@@ -80,11 +80,12 @@ def run(self, tmp=None, task_vars=None):
                 tmp_source = self._loader.get_real_file(source)
             except AnsibleFileNotFound as e:
                 raise AnsibleActionFail("could not find src=%s, %s" % (source, to_text(e)))
+            b_tmp_source = to_bytes(tmp_source, errors='surrogate_or_strict')
 
             # template the source data locally & get ready to transfer
             try:
-                with open(tmp_source, 'r') as f:
-                    template_data = to_text(f.read())
+                with open(b_tmp_source, 'rb') as f:
+                    template_data = to_text(f.read(), errors='surrogate_or_strict')
 
                 # set jinja2 internal search path for includes
                 searchpath = task_vars.get('ansible_search_path', [])
@@ -124,7 +125,7 @@ def run(self, tmp=None, task_vars=None):
             except Exception as e:
                 raise AnsibleActionFail("%s: %s" % (type(e).__name__, to_text(e)))
             finally:
-                self._loader.cleanup_tmp_file(tmp_source)
+                self._loader.cleanup_tmp_file(b_tmp_source)
 
             new_task = self._task.copy()
             new_task.args.pop('newline_sequence', None)
@@ -138,7 +139,7 @@ def run(self, tmp=None, task_vars=None):
 
             try:
                 result_file = os.path.join(local_tempdir, os.path.basename(source))
-                with open(result_file, 'wb') as f:
+                with open(to_bytes(result_file, errors='surrogate_or_strict'), 'wb') as f:
                     f.write(to_bytes(resultant, errors='surrogate_or_strict'))
 
                 new_task.args.update(
@@ -157,7 +158,7 @@ def run(self, tmp=None, task_vars=None):
                                                                         shared_loader_obj=self._shared_loader_obj)
                 result.update(copy_action.run(task_vars=task_vars))
             finally:
-                shutil.rmtree(local_tempdir)
+                shutil.rmtree(to_bytes(local_tempdir, errors='surrogate_or_strict'))
 
         except AnsibleAction as e:
             result.update(e.result)


Test patch: