Repository: ansible/ansible

Issue:
Template module not accepting source that contains UTF8 characters
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
template

##### ANSIBLE VERSION
```
$ ansible --version
ansible 2.3.1.0
  config file = /work/ansible/configuration/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.5 (default, Nov  6 2016, 00:28:07) [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]
```

##### OS / ENVIRONMENT
Ansible runs on CentOS 7.3
Target system on RHEL 6.9

##### SUMMARY
This looks like bug #10110:
Using the template module, when the source name contains an UTF8 character, the module fails with the following error.
Replacing template: with copy:, there is no error (the "copy" module has been used for more than a year and is just now being replaced by template).
If there is UTF8 only in the destination name, there is no error.


##### STEPS TO REPRODUCE

This sample triggers the error:
```
- name: Test UTF8
  template:
    src:    "/tmp/ééé"
    dest:   "/tmp/eee"
```
This sample works:
```
- name: Test UTF8
  template:
    src:    "/tmp/eee"
    dest:   "/tmp/ééé"
```


##### EXPECTED RESULTS
The template being copied to the target.

##### ACTUAL RESULTS

```
TASK [task : Test UTF8 *********************************************************************************************
[WARNING]: Failure using method (v2_runner_item_on_failed) in callback plugin (<ansible.plugins.callback.default.CallbackModule object at 0x35eb190>):
'ascii' codec can't decode byte 0xc3 in position 22: ordinal not in range(128
```



Discussion:
Files identified in the description:
* [lib/ansible/modules/files/template.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/files/template.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @abadger 
I cannot reproduce this error with a checkout of ansible-2.3.1.0 or the head of the current devel branch.
```
  - name: Remove test file
    file:
      state: absent
      path: "/tmp/eee"

  # Basic template with non-ascii names
  - name: Check that non-ascii source works
    template:
      src:    "/tmp/ééé"
      dest: '/tmp/eee'
```
Command run:
```
[pts/46@peru /srv/backup/ansible/ansible/stable-2.3]$ ansible-playbook test.yml            *[ecb38fdf73]  (11:02:05)
 [WARNING]: Host file not found: /etc/ansible/hosts

 [WARNING]: provided hosts list is empty, only localhost is available


PLAY [localhost] ****************************************************************************************************

TASK [Remove test file] *********************************************************************************************
ok: [localhost]

TASK [Check that non-ascii source and dest work] ********************************************************************
changed: [localhost]

PLAY RECAP **********************************************************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0 
```

```
[pts/46@peru /srv/backup/ansible/ansible/stable-2.3]$ ansible --version                    *[ecb38fdf73]  (11:02:07)
ansible 2.3.1.0 (detached HEAD ecb38fdf73) last updated 2018/04/26 10:52:03 (GMT -700)
  config file = 
  configured module search path = Default w/o overrides
  python version = 2.7.14 (default, Mar 14 2018, 13:36:31) [GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]
```

I note that the error message is actually a warning.  The warning seems to indicate that the template module encounters an error that we cannot see and then, when the callback attempts to display that error, the callback throws the decoding error.  So there seems to be two things wrong in your setup.

I tried introducing an error in my playbook (setting dest to an unwritable location) in hopes that I could reproduce your callback error but no luck.  The callback displays the error from the module instead of itself having an error.

I'll need more information from you on how exactly to reproduce the issue to move forward with debugging this.  In addition, 2.3.x is no longer maintained.  If you can try with a newer version of ansible that would be great.

needs_info
Hello,

I've been upgrading Ansible, and this still happens on the latest version. It's odd that you don't get it, as I've got no trouble reproducing it. I'm attaching a complete role with the issue, including one `copy` task that works, and an otherwise identical `template` task that fails.

```
$ ansible --version
ansible 2.5.1
   config file = /data/ansible/configuration/ansible.cfg
   configured module search path =
[u'/home/me/.ansible/plugins/modules',
u'/usr/share/ansible/plugins/modules']
   ansible python module location = /usr/lib/python2.7/site-packages/ansible
   executable location = /usr/bin/ansible
   python version = 2.7.5 (default, Aug  4 2017, 00:39:18) [GCC 4.8.5
20150623 (Red Hat 4.8.5-16)]
```

```
$ ansible-playbook test_template.yml
[DEPRECATION WARNING]: DEFAULT_SUDO_FLAGS option, In favor of Ansible Become,
which is a generic framework. See become_flags. , use become instead. This
feature will be removed in version 2.8. Deprecation warnings can be disabled by
setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: DEFAULT_SUDO_EXE option, In favor of Ansible Become,
which is a generic framework. See become_exe. , use become instead. This
feature will be removed in version 2.8. Deprecation warnings can be disabled by
setting deprecation_warnings=False in ansible.cfg.

PLAY [serveur] *****************************************************************

TASK [Gathering Facts] *********************************************************
ok: [serveur]

TASK [test_template : Copie] ***************************************************
ok: [serveur]

TASK [test_template : Modèle] **************************************************
fatal: [serveur]: FAILED! => {"changed": false, "msg": "UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 67: ordinal not in range(128)"}

PLAY RECAP *********************************************************************
serveur                    : ok=2    changed=0    unreachable=0    failed=1 
```

[role_test_template.tar.gz](https://github.com/ansible/ansible/files/1954590/role_test_template.tar.gz)


I'm still not able to reproduce this.  What are your locale settings (LC_* and LANG)?  I can get this error if I set the locale to an ascii-only locale but if I do that my output is different.  For instance:
```
TASK [test_template : Mod?le] ***************************************************************************************
```

So I don't think that's what's happening in your case.
I have some fixes for running under LC_ALL=C.... I'm still not sure what about your environment is leading to the problem you see but there's a good chance that they'll fix the traceback for you.  I'll post a PR shortly.
I'm using en_US.utf8 all round, both locally and on the remote server. I've tried with LC_ALL=C as well, this produces a related, but different error, note the `encode` instead of `decode`.
`copy` still works as expected.
```

TASK [test_template : Mod?le]  **************************************************
fatal: [serveur]: FAILED! => {"changed": false, "msg":  "UnicodeEncodeError: 'ascii' codec can't encode characters in position  65-67: ordinal not in range(128)"}
```
@L-B do you have any callbacks enabled in your Ansible config?
@L-B can you try applying the changes from https://github.com/ansible/ansible/pull/39424 to see what that does for your issue?  Those were the changes I had to make for it to work in the C locale.  Since I haven't figured out what the difference is in your environment that actually causes this, though, there could be some other problem triggering this.  I give it a 50-50 chance of fixing it.
 #39424 has been merged to devel and I have a backport PR for 2.5.3: https://github.com/ansible/ansible/pull/39468
@maxamillion 
No callback, my local `ansible.cfg` points to an old default, a non-existent `/usr/share/ansible_plugins/callback_plugins`
'local' as opposed to the one in `/etc/ansible`, which is unmodified, all callback lines are commented there.
@abadger 
I applied the diff straight to my 2.5.1 install, but the issue persists:

```
$ cd /usr/lib/python2.7/site-packages/ansible
$ sudo patch -p3 < /tmp/81f3b4f747a6e45c6c6a56712068c3ac982e0a50.patch
(Stripping trailing CRs from patch; use --binary to disable.)
patching file plugins/action/template.py
Hunk #1 succeeded at 80 (offset -35 lines).
Hunk #2 succeeded at 125 with fuzz 1 (offset -35 lines).
Hunk #3 succeeded at 139 (offset -39 lines).
Hunk #4 succeeded at 158 (offset -39 lines).
```

Still there:
```
TASK [test_template : Modèle] **************************************************
fatal: [serveur]: FAILED! => {"changed": false, "msg": "UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 67: ordinal not in range(128)"}
```

What else can I try to help diagnose?
@L-B Can you give the output when running with -vvvvv (five v's)  Lacking a reproducer, I'll need t oget the traceback which will show where this error is coming from.

If the output doesn't have a traceback, I'll have to figure out where in the code we might be catching the exception and then omitting the traceback information.
I had given a try, but -vvvvvv did not show a traceback or anything looking useful.
However, after starting from scratch on a different box and further prodding, I've found the cause, and I think now you'll be able to reproduce it. It's this line in `/etc/ansible/ansible.cfg`:
```
ansible_managed = Ansible managed: {file} modified on %Y-%m-%d %H:%M:%S by {uid} on {host}
```

Please note that it's the default value, but it's commented out by default. Uncommenting it breaks `template` with a source name using UTF-8. Now I hope you can find out why, because it's rather mysterious to me :)
Reproduced!  Thanks. /me looking for the problem now.

Resolving patch:
diff --git a/lib/ansible/template/__init__.py b/lib/ansible/template/__init__.py
--- a/lib/ansible/template/__init__.py
+++ b/lib/ansible/template/__init__.py
@@ -72,7 +72,6 @@
 
 
 def generate_ansible_template_vars(path):
-
     b_path = to_bytes(path)
     try:
         template_uid = pwd.getpwuid(os.stat(b_path).st_uid).pw_name
@@ -80,10 +79,10 @@ def generate_ansible_template_vars(path):
         template_uid = os.stat(b_path).st_uid
 
     temp_vars = {}
-    temp_vars['template_host'] = os.uname()[1]
-    temp_vars['template_path'] = b_path
+    temp_vars['template_host'] = to_text(os.uname()[1])
+    temp_vars['template_path'] = path
     temp_vars['template_mtime'] = datetime.datetime.fromtimestamp(os.path.getmtime(b_path))
-    temp_vars['template_uid'] = template_uid
+    temp_vars['template_uid'] = to_text(template_uid)
     temp_vars['template_fullpath'] = os.path.abspath(path)
     temp_vars['template_run_date'] = datetime.datetime.now()
 
@@ -93,7 +92,7 @@ def generate_ansible_template_vars(path):
         uid=temp_vars['template_uid'],
         file=temp_vars['template_path'],
     )
-    temp_vars['ansible_managed'] = time.strftime(managed_str, time.localtime(os.path.getmtime(b_path)))
+    temp_vars['ansible_managed'] = to_text(time.strftime(to_native(managed_str), time.localtime(os.path.getmtime(b_path))))
 
     return temp_vars
 


Test patch:
diff --git a/test/integration/targets/template/ansible_managed.cfg b/test/integration/targets/template/ansible_managed.cfg
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/template/ansible_managed.cfg
@@ -0,0 +1,2 @@
+[defaults]
+ansible_managed=ansible_managed = Ansible managed: {file} modified on %Y-%m-%d %H:%M:%S by {uid} on {host}
diff --git a/test/integration/targets/template/ansible_managed.yml b/test/integration/targets/template/ansible_managed.yml
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/template/ansible_managed.yml
@@ -0,0 +1,12 @@
+---
+- hosts: testhost
+  gather_facts: False
+  tasks:
+  - file:
+      path: '{{ output_dir }}/café.txt'
+      state: 'absent'
+  # Smoketest that ansible_managed with non-ascii chars works:
+  # https://github.com/ansible/ansible/issues/27262
+  - template:
+      src: 'templates/café.j2'
+      dest: '{{ output_dir }}/café.txt'
diff --git a/test/integration/targets/template/runme.sh b/test/integration/targets/template/runme.sh
--- a/test/integration/targets/template/runme.sh
+++ b/test/integration/targets/template/runme.sh
@@ -6,3 +6,6 @@ ANSIBLE_ROLES_PATH=../ ansible-playbook template.yml -i ../../inventory -e @../.
 
 # Test for #35571
 ansible testhost -i testhost, -m debug -a 'msg={{ hostvars["localhost"] }}' -e "vars1={{ undef }}" -e "vars2={{ vars1 }}"
+
+# Test for https://github.com/ansible/ansible/issues/27262
+ansible-playbook ansible_managed.yml -c  ansible_managed.cfg -i ../../inventory -e @../../integration_config.yml -v "$@"
diff --git a/test/integration/targets/template/tasks/main.yml b/test/integration/targets/template/tasks/main.yml
--- a/test/integration/targets/template/tasks/main.yml
+++ b/test/integration/targets/template/tasks/main.yml
@@ -51,6 +51,24 @@
     that:
       - "template_result.changed == true"
 
+# Basic template with non-ascii names
+- name: Check that non-ascii source and dest work
+  template:
+    src: 'café.j2'
+    dest: '{{ output_dir }}/café.txt'
+  register: template_results
+
+- name: Check that the resulting file exists
+  stat:
+    path: '{{ output_dir }}/café.txt'
+  register: stat_results
+
+- name: Check that template created the right file
+  assert:
+    that:
+      - 'template_results is changed'
+      - 'stat_results.stat["exists"]'
+
 # test for import with context on jinja-2.9 See https://github.com/ansible/ansible/issues/20494
 - name: fill in a template using import with context ala issue 20494
   template: src=import_with_context.j2 dest={{output_dir}}/import_with_context.templated mode=0644
diff --git "a/test/integration/targets/template/templates/caf\303\251.j2" "b/test/integration/targets/template/templates/caf\303\251.j2"
new file mode 100644
--- /dev/null
+++ "b/test/integration/targets/template/templates/caf\303\251.j2"
@@ -0,0 +1 @@
+{{ ansible_managed }}