Repository: ansible/ansible
Pull request: Properly handle unicode in safe_eval
##### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
Fixes #66943
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->

##### ISSUE TYPE
<!--- Pick one below and delete the rest -->
- Bugfix Pull Request


##### COMPONENT NAME
<!--- Write the short name of the module, plugin, task or feature below -->
`lib/ansible/template/safe_eval.py`

Problem statement:
Unicode breaks dict
<!--- Verify first that your issue is not already reported on GitHub -->
<!--- Also test if the latest release and devel branch are affected too -->
<!--- Complete *all* sections as described, this form is processed automatically -->

##### SUMMARY

Ansible can't parse unicode correctly, when reading template

##### COMPONENT NAME

template

##### ANSIBLE VERSION

```
ansible 2.9.2
  config file = None
  configured module search path = [u'/home/home/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Aug  7 2019, 00:51:29) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]
```

##### OS / ENVIRONMENT
 
CentOS Linux release 7.7.1908 (Core)

##### STEPS TO REPRODUCE

vars.yml

```yaml
some_var: "test"
```

templates/example.json.j2

```json
{
  "something": "test",
  "this-breaks-key": "This breaks the whole file, because of: ü§",
  "another-key": "{{ some_var }}"
}
```

```yaml
- hosts: test
  connection: local
  vars:
    reference: "{{ lookup('template', 'templates/example.json.j2') }}"
    delta: {}
  tasks:
    - name: Test | Get configuration changes
      set_fact:
        delta: "{{ delta | combine({item.key: item.value}, recursive=true) }}"
      loop: "{{ reference | dict2items }}"
```

##### EXPECTED RESULTS

The valid json file should be correctly parsed and unicode should not be a problem in 2020.

Everything works just as expected, when `unicode chars` are removed.

##### ACTUAL RESULTS
<!--- Describe what actually happened. If possible run with extra verbosity (-vvvv) -->

```json
{"msg": "dict2items requires a dictionary, got <type 'unicode'> instead."}
```

The underlying exception that causes this behavior:

```
Traceback (most recent call last):
  File "/Users/matt/projects/ansibledev/ansible/lib/ansible/template/safe_eval.py", line 142, in safe_eval
    compiled = compile(parsed_tree, expr, 'eval')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 87-88: ordinal not in range(128)
```

Hints from issue discussion:
Files identified in the description:
* [`lib/ansible/modules/files/template.py`](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/files/template.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
I was able to reproduce the issue. You can use Python 3 on the controller to work around it.

Note, this also fails with `ANSIBLE_JINJA2_NATIVE` on both Python 2 and Python 3.

Commit message: Properly handle unicode in safe_eval

Fixes #66943

Patch to review:
diff --git a/changelogs/fragments/66943-handle-unicode-in-safe_eval.yml b/changelogs/fragments/66943-handle-unicode-in-safe_eval.yml
new file mode 100644
index 00000000000000..5e8821b0044781
--- /dev/null
+++ b/changelogs/fragments/66943-handle-unicode-in-safe_eval.yml
@@ -0,0 +1,2 @@
+bugfixes:
+  - Properly handle unicode in ``safe_eval``. (https://github.com/ansible/ansible/issues/66943)
diff --git a/lib/ansible/module_utils/common/text/converters.py b/lib/ansible/module_utils/common/text/converters.py
index 014ed10099df79..7237a12f4e60d5 100644
--- a/lib/ansible/module_utils/common/text/converters.py
+++ b/lib/ansible/module_utils/common/text/converters.py
@@ -61,7 +61,7 @@ def container_to_bytes(d, encoding='utf-8', errors='surrogate_or_strict'):
 
 
 def container_to_text(d, encoding='utf-8', errors='surrogate_or_strict'):
-    """Recursively convert dict keys and values to byte str
+    """Recursively convert dict keys and values to text str
 
     Specialized for json return because this only handles, lists, tuples,
     and dict container types (the containers that the json module returns)
diff --git a/lib/ansible/template/safe_eval.py b/lib/ansible/template/safe_eval.py
index 35c70ea03a96cb..ac98ad3ec9a6b6 100644
--- a/lib/ansible/template/safe_eval.py
+++ b/lib/ansible/template/safe_eval.py
@@ -22,7 +22,9 @@
 import sys
 
 from ansible import constants as C
-from ansible.module_utils.six import string_types
+from ansible.module_utils._text import to_native
+from ansible.module_utils.common.text.converters import container_to_text
+from ansible.module_utils.six import string_types, PY2
 from ansible.module_utils.six.moves import builtins
 from ansible.plugins.loader import filter_loader, test_loader
 
@@ -139,11 +141,15 @@ def generic_visit(self, node, inside_call=False):
     try:
         parsed_tree = ast.parse(expr, mode='eval')
         cnv.visit(parsed_tree)
-        compiled = compile(parsed_tree, expr, 'eval')
+        compiled = compile(parsed_tree, to_native(expr), 'eval')
         # Note: passing our own globals and locals here constrains what
         # callables (and other identifiers) are recognized.  this is in
         # addition to the filtering of builtins done in CleansingNodeVisitor
         result = eval(compiled, OUR_GLOBALS, dict(locals))
+        if PY2:
+            # On Python 2 u"{'key': 'value'}" is evaluated to {'key': 'value'},
+            # ensure it is converted to {u'key': u'value'}.
+            result = container_to_text(result)
 
         if include_exceptions:
             return (result, None)
diff --git a/test/integration/targets/templating_lookups/runme.sh b/test/integration/targets/templating_lookups/runme.sh
index 3ddca11cca069f..e681070d770487 100755
--- a/test/integration/targets/templating_lookups/runme.sh
+++ b/test/integration/targets/templating_lookups/runme.sh
@@ -7,3 +7,6 @@ ANSIBLE_ROLES_PATH=../ UNICODE_VAR=café ansible-playbook runme.yml "$@"
 ansible-playbook template_lookup_vaulted/playbook.yml --vault-password-file template_lookup_vaulted/test_vault_pass "$@"
 
 ansible-playbook template_deepcopy/playbook.yml -i template_deepcopy/hosts "$@"
+
+# https://github.com/ansible/ansible/issues/66943
+ansible-playbook template_lookup_safe_eval_unicode/playbook.yml "$@"
diff --git a/test/integration/targets/templating_lookups/template_lookup_safe_eval_unicode/playbook.yml b/test/integration/targets/templating_lookups/template_lookup_safe_eval_unicode/playbook.yml
new file mode 100644
index 00000000000000..29e4b615245dfb
--- /dev/null
+++ b/test/integration/targets/templating_lookups/template_lookup_safe_eval_unicode/playbook.yml
@@ -0,0 +1,8 @@
+- hosts: localhost
+  gather_facts: no
+  vars:
+    original_dict: "{{ lookup('template', 'template.json.j2') }}"
+    copy_dict: {}
+  tasks:
+    - set_fact:
+        copy_dict: "{{ copy_dict | combine(original_dict) }}"
diff --git a/test/integration/targets/templating_lookups/template_lookup_safe_eval_unicode/template.json.j2 b/test/integration/targets/templating_lookups/template_lookup_safe_eval_unicode/template.json.j2
new file mode 100644
index 00000000000000..bc31407ccf4da3
--- /dev/null
+++ b/test/integration/targets/templating_lookups/template_lookup_safe_eval_unicode/template.json.j2
@@ -0,0 +1,4 @@
+{
+  "key1": "ascii_value",
+  "key2": "unicode_value_křížek",
+}


Code review comments:
diff_hunk: @@ -22,7 +22,9 @@
 import sys
 
 from ansible import constants as C
-from ansible.module_utils.six import string_types
+from ansible.module_utils._text import to_native
+from ansible.module_utils.common.text.converters import container_to_text
path: lib/ansible/template/safe_eval.py
text: @user1:
```suggestion
from ansible.module_utils.common.text.converters import container_to_text, to_native
```

@author:
:+1: