Repository: ansible/ansible

Issue:
!unsafe broken in Ansible 2.3.0.0?
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report


##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
jinja2 (?)

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.3.0.0
  config file = /ansible/ansible.cfg
  configured module search path = [u'/ansible/library']
  python version = 2.7.12 (default, Jun 29 2016, 08:57:23) [GCC 5.3.0]

```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
-->
Docker Alpine container running on Ubuntu

##### SUMMARY
<!--- Explain the problem briefly -->
Templating out...
```
!unsafe |
            ALERT InstanceDown
            IF up == 0
            FOR 5m
            LABELS { severity = "page" }
            ANNOTATIONS {
              summary = "Instance {{ $labels.instance }} down",
              description = "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.",
            }
```
Should result in the `{{ $labels.instance }}` and  `{{ $labels.job }}` remaining unchanged (i.e. not interpolated) but infact Ansible is now interpolating them again.

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
```yaml

```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
For variables in unsafe strings to remain un-interpolated.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
The string was interpolated




Discussion:
I can confirm this also affects 2.2.2.0 but is working in 2.2.1.0
Do you have yaml playbook file?

need_info
I didn't reproduce with this playbook with or w/o $ sign in $labels:

```
- hosts: localhost
  connection: local
  gather_facts: no
  vars:
    labels:
      instance: "test"
      job: "test"
    a: !unsafe |
            ALERT InstanceDown
            IF up == 0
            FOR 5m
            LABELS { severity = "page" }
            ANNOTATIONS {
              summary = "Instance {{ $labels.instance }} down",
              description = "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.",
            }

  tasks:
    - debug: msg={{a}}
```
Playbook

```
---

- name: Test unsafe
  hosts: localhost
  become: false
  gather_facts: false
  connection: local
  vars:
    prometheus_alerts:
      - name: test
        rule: !unsafe |
          {{ $test }}
          {{ another_test }}

  tasks:
    - name: create rules
      tags: [prometheus]
      blockinfile:
        dest: /tmp/{{ item.name }}.rules
        create: yes
        block: "{{ item.rule }}"
        marker: "# {{ item.name }} {mark}"
      with_items: "{{ prometheus_alerts }}"

    - command: cat /tmp/test.rules
      register: output
```

Run with `-v` to get debug 

Running in 2.2.1.0

```
PLAY [Test unsafe] *************************************************************

TASK [create rules] ************************************************************
ok: [localhost] => (item={u'name': u'test', u'rule': u'{{ $test }}\n{{ another_test }}\n'}) => {"changed": false, "item": {"name": "test", "rule": "{{ $test }}\n{{ another_test }}\n"}, "msg": ""}

TASK [command] *****************************************************************
changed: [localhost] => {"changed": true, "cmd": ["cat", "/tmp/test.rules"], "delta": "0:00:00.008700", "end": "2017-04-19 12:44:23.552773", "rc": 0, "start": "2017-04-19 12:44:23.544073", "stderr": "", "stdout": "# test BEGIN\n{{ $test }}\n{{ another_test }}\n# test END", "stdout_lines": ["# test BEGIN", "{{ $test }}", "{{ another_test }}", "# test END"], "warnings": []}

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0   
```

You can see the stdout contains the `{{ $test }}` and `{{ another_test }}` lines

Running in 2.2.2.0

```
PLAY [Test unsafe] *************************************************************

TASK [create rules] ************************************************************
changed: [localhost] => (item={u'name': u'test', u'rule': u'{# $test #}\n{# another_test #}\n'}) => {"changed": true, "item": {"name": "test", "rule": "{# $test #}\n{# another_test #}\n"}, "msg": "Block inserted"}

TASK [command] *****************************************************************
changed: [localhost] => {"changed": true, "cmd": ["cat", "/tmp/test.rules"], "delta": "0:00:00.008554", "end": "2017-04-19 12:46:33.281070", "rc": 0, "start": "2017-04-19 12:46:33.272516", "stderr": "", "stdout": "# test BEGIN\n\n# test END", "stdout_lines": ["# test BEGIN", "", "# test END"], "warnings": []}

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=2    unreachable=0    failed=0   
```

You can see stdout just has the block header and footer.
output from `pip list` incase this is something like jinja2 weirdness...

```
ansible (2.2.2.0)
ansible-lint (3.4.12)
appdirs (1.4.3)
asn1crypto (0.22.0)
awscli (1.11.77)
backports.ssl-match-hostname (3.5.0.1)
boto (2.46.1)
botocore (1.5.40)
cffi (1.10.0)
colorama (0.3.7)
cryptography (1.8.1)
docker-py (1.10.6)
docker-pycreds (0.2.1)
docutils (0.13.1)
ecdsa (0.13)
enum34 (1.1.6)
futures (3.1.1)
idna (2.5)
ipaddress (1.0.18)
Jinja2 (2.8.1)
jmespath (0.9.2)
lxml (3.7.3)
MarkupSafe (1.0)
MySQL-python (1.2.5)
packaging (16.8)
paramiko (2.1.2)
pip (8.1.2)
prettytable (0.7.2)
pyasn1 (0.2.3)
pycparser (2.17)
pycrypto (2.6.1)
pyparsing (2.2.0)
pyroute2 (0.4.13)
python-dateutil (2.6.0)
PyYAML (3.12)
requests (2.13.0)
rsa (3.4.2)
s3transfer (0.1.10)
setuptools (35.0.1)
six (1.10.0)
websocket-client (0.40.0)
xmltodict (0.10.2)
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
```
reproduced and _clean_data added {# and #}

Resolving patch:
diff --git a/lib/ansible/playbook/conditional.py b/lib/ansible/playbook/conditional.py
--- a/lib/ansible/playbook/conditional.py
+++ b/lib/ansible/playbook/conditional.py
@@ -139,7 +139,7 @@ def _check_conditional(self, conditional, templar, all_vars):
         if conditional in all_vars and VALID_VAR_REGEX.match(conditional):
             conditional = all_vars[conditional]
 
-        if templar._clean_data(conditional) != conditional:
+        if templar.is_template(conditional):
             display.warning('when statements should not include jinja2 '
                             'templating delimiters such as {{ }} or {%% %%}. '
                             'Found: %s' % conditional)
diff --git a/lib/ansible/playbook/helpers.py b/lib/ansible/playbook/helpers.py
--- a/lib/ansible/playbook/helpers.py
+++ b/lib/ansible/playbook/helpers.py
@@ -289,7 +289,7 @@ def load_list_of_tasks(ds, play, block=None, role=None, task_include=None, use_h
                     needs_templating = False
                     for param in ir.args:
                         if templar._contains_vars(ir.args[param]):
-                            if not templar.templatable(ir.args[param]):
+                            if not templar.is_template(ir.args[param]):
                                 needs_templating = True
                                 break
                     is_static = C.DEFAULT_TASK_INCLUDES_STATIC or \
diff --git a/lib/ansible/template/__init__.py b/lib/ansible/template/__init__.py
--- a/lib/ansible/template/__init__.py
+++ b/lib/ansible/template/__init__.py
@@ -49,6 +49,7 @@
 from ansible.template.safe_eval import safe_eval
 from ansible.template.template import AnsibleJ2Template
 from ansible.template.vars import AnsibleJ2Vars
+from ansible.vars.unsafe_proxy import UnsafeProxy, wrap_var
 
 try:
     from __main__ import display
@@ -311,45 +312,64 @@ def _get_extensions(self):
         return jinja_exts
 
     def _clean_data(self, orig_data):
-        ''' remove jinja2 template tags from a string '''
-
-        if not isinstance(orig_data, string_types) or hasattr(orig_data, '__ENCRYPTED__'):
-            return orig_data
-
-        with contextlib.closing(StringIO(orig_data)) as data:
-            # these variables keep track of opening block locations, as we only
-            # want to replace matched pairs of print/block tags
-            print_openings = []
-            block_openings = []
-            for mo in self._clean_regex.finditer(orig_data):
-                token = mo.group(0)
-                token_start = mo.start(0)
-
-                if token[0] == self.environment.variable_start_string[0]:
-                    if token == self.environment.block_start_string:
-                        block_openings.append(token_start)
-                    elif token == self.environment.variable_start_string:
-                        print_openings.append(token_start)
-
-                elif token[1] == self.environment.variable_end_string[1]:
-                    prev_idx = None
-                    if token == self.environment.block_end_string and block_openings:
-                        prev_idx = block_openings.pop()
-                    elif token == self.environment.variable_end_string and print_openings:
-                        prev_idx = print_openings.pop()
-
-                    if prev_idx is not None:
-                        # replace the opening
-                        data.seek(prev_idx, os.SEEK_SET)
-                        data.write(to_text(self.environment.comment_start_string))
-                        # replace the closing
-                        data.seek(token_start, os.SEEK_SET)
-                        data.write(to_text(self.environment.comment_end_string))
+        ''' remove jinja2 template tags from data '''
+
+        if hasattr(orig_data, '__ENCRYPTED__'):
+            ret = orig_data
+
+        elif isinstance(orig_data, list):
+            clean_list = []
+            for list_item in orig_data:
+                clean_list.append(self._clean_data(list_item))
+            ret = clean_list
+
+        elif isinstance(orig_data, dict):
+            clean_dict = {}
+            for k in orig_data:
+                clean_dict[self._clean_data(k)] =  self._clean_data(orig_data[k])
+            ret = clean_dict
+
+        elif isinstance(orig_data, string_types):
+            # This will error with str data (needs unicode), but all strings should already be converted already.
+            # If you get exception, the problem is at the data origin, do not add to_text here.
+            with contextlib.closing(StringIO(orig_data)) as data:
+                # these variables keep track of opening block locations, as we only
+                # want to replace matched pairs of print/block tags
+                print_openings = []
+                block_openings = []
+                for mo in self._clean_regex.finditer(orig_data):
+                    token = mo.group(0)
+                    token_start = mo.start(0)
+
+                    if token[0] == self.environment.variable_start_string[0]:
+                        if token == self.environment.block_start_string:
+                            block_openings.append(token_start)
+                        elif token == self.environment.variable_start_string:
+                            print_openings.append(token_start)
+
+                    elif token[1] == self.environment.variable_end_string[1]:
+                        prev_idx = None
+                        if token == self.environment.block_end_string and block_openings:
+                            prev_idx = block_openings.pop()
+                        elif token == self.environment.variable_end_string and print_openings:
+                            prev_idx = print_openings.pop()
+
+                        if prev_idx is not None:
+                            # replace the opening
+                            data.seek(prev_idx, os.SEEK_SET)
+                            data.write(to_text(self.environment.comment_start_string))
+                            # replace the closing
+                            data.seek(token_start, os.SEEK_SET)
+                            data.write(to_text(self.environment.comment_end_string))
 
-                else:
-                    raise AnsibleError("Error while cleaning data for safety: unhandled regex match")
+                    else:
+                        raise AnsibleError("Error while cleaning data for safety: unhandled regex match")
+
+                ret = data.getvalue()
+        else:
+            ret = orig_data
 
-            return data.getvalue()
+        return ret
 
     def set_available_variables(self, variables):
         '''
@@ -371,15 +391,13 @@ def template(self, variable, convert_bare=False, preserve_trailing_newlines=True
         before being sent through the template engine.
         '''
 
+        # Don't template unsafe variables, just return them.
+        if hasattr(variable, '__UNSAFE__'):
+            return variable
+
         if fail_on_undefined is None:
             fail_on_undefined = self._fail_on_undefined_errors
 
-        # Don't template unsafe variables, instead drop them back down to their constituent type.
-        if hasattr(variable, '__UNSAFE__'):
-            if isinstance(variable, text_type):
-                rval = self._clean_data(variable)
-                return rval
-
         try:
             if convert_bare:
                 variable = self._convert_bare_variable(variable, bare_deprecated=bare_deprecated)
@@ -435,7 +453,6 @@ def template(self, variable, convert_bare=False, preserve_trailing_newlines=True
                                 if eval_results[1] is None:
                                     result = eval_results[0]
                                     if unsafe:
-                                        from ansible.vars.unsafe_proxy import wrap_var
                                         result = wrap_var(result)
                                 else:
                                     # FIXME: if the safe_eval raised an error, should we do something with it?
@@ -482,6 +499,26 @@ def template(self, variable, convert_bare=False, preserve_trailing_newlines=True
             else:
                 return variable
 
+    def is_template(self, data):
+        ''' lets us know if data has a template'''
+        if isinstance(data, string_types):
+            try:
+                new = self.do_template(data)
+            except UndefinedError:
+                return True
+            except:
+                return False
+            return (new != data)
+        elif isinstance(data, (list, tuple)):
+            for v in data:
+                if self.is_template(v):
+                    return True
+        elif isinstance(data, dict):
+            for k in data:
+                if self.is_template(k) or self.is_template(data[k]):
+                    return True
+        return False
+
     def templatable(self, data):
         '''
         returns True if the data can be templated w/o errors
@@ -552,7 +589,6 @@ def _lookup(self, name, *args, **kwargs):
                 ran = None
 
             if ran:
-                from ansible.vars.unsafe_proxy import UnsafeProxy, wrap_var
                 if wantlist:
                     ran = wrap_var(ran)
                 else:
@@ -593,13 +629,12 @@ def do_template(self, data, preserve_trailing_newlines=True, escape_backslashes=
                     key = key.strip()
                     setattr(myenv, key, ast.literal_eval(val.strip()))
 
-            #FIXME: add tests
+            # Adds Ansible custom filters and tests
             myenv.filters.update(self._get_filters())
             myenv.tests.update(self._get_tests())
 
             if escape_backslashes:
-                # Allow users to specify backslashes in playbooks as "\\"
-                # instead of as "\\\\".
+                # Allow users to specify backslashes in playbooks as "\\" instead of as "\\\\".
                 data = _escape_backslashes(data, myenv)
 
             try:
@@ -627,7 +662,6 @@ def do_template(self, data, preserve_trailing_newlines=True, escape_backslashes=
             try:
                 res = j2_concat(rf)
                 if new_context.unsafe:
-                    from ansible.vars.unsafe_proxy import wrap_var
                     res = wrap_var(res)
             except TypeError as te:
                 if 'StrictUndefined' in to_native(te):


Test patch:
diff --git a/test/integration/targets/copy/tasks/main.yml b/test/integration/targets/copy/tasks/main.yml
--- a/test/integration/targets/copy/tasks/main.yml
+++ b/test/integration/targets/copy/tasks/main.yml
@@ -157,7 +157,7 @@
 - name: assert recursive copied directories mode
   assert:
     that:
-      - "{{item.stat.mode}} == 0700"
+      - "item.stat.mode == '0700'"
   with_items: "{{dir_stats.results}}"
 
 
diff --git a/test/integration/targets/win_path/tasks/main.yml b/test/integration/targets/win_path/tasks/main.yml
--- a/test/integration/targets/win_path/tasks/main.yml
+++ b/test/integration/targets/win_path/tasks/main.yml
@@ -23,8 +23,8 @@
   assert:
     that:
     - item.0 | changed
-    - item.0.path_value == "C:\\{{ item.0.item }}Path"
-    - item.1.stdout_lines[0] == 'C:\\{{ item.0.item }}Path'
+    - item.0.path_value == "C:\\" + item.0.item + "Path"
+    - item.1.stdout_lines[0] == 'C:\\' + item.0.item + 'Path'
   with_together:
   - "{{ pathout.results }}"
   - "{{ varout.results }}"
diff --git a/test/units/template/test_templar.py b/test/units/template/test_templar.py
--- a/test/units/template/test_templar.py
+++ b/test/units/template/test_templar.py
@@ -178,12 +178,6 @@ def test_template_unsafe_non_string_subclass(self):
         res = self.templar.template(unsafe_obj)
         self.assertTrue(self.is_unsafe(res), 'returned value from template.template (%s) is not marked unsafe' % res)
 
-    @patch('ansible.template.Templar._clean_data', side_effect=AnsibleError)
-    def test_template_unsafe_clean_data_exception(self, mock_clean_data):
-        self.assertRaises(AnsibleError,
-                          self.templar.template,
-                          wrap_var('blip bar'))
-
     # TODO: not sure what template is supposed to do it, but it currently throws attributeError
     @patch('ansible.template.Templar._clean_data')
     def test_template_unsafe_non_string_clean_data_exception(self, mock_clean_data):
@@ -234,16 +228,11 @@ def test_clean_data_block(self):
         self.assertEqual(res, u'1 2 {#what#} 3 4 {#foo#} 5 6 7')
 
     def test_clean_data_object(self):
-        obj = {'foo': [1, 2, 3, 'bdasdf', '{what}', '{{foo}}', 5]}
-        res = self.templar._clean_data(obj)
-        self.assertEqual(res, obj)
-
-    def test_clean_data_object_unsafe(self):
-        rval = [1, 2, 3, wrap_var('bdasdf'), '{what}', wrap_var('{{unsafe_foo}}'), 5]
-        obj = {'foo': rval}
+        obj = {u'foo': [1, 2, 3, u'bdasdf', u'{what}', u'{{foo}}', 5]}
+        clean_obj = {u'foo': [1, 2, 3, u'bdasdf', u'{what}', u'{#foo#}', 5]}
         res = self.templar._clean_data(obj)
-        self.assertEqual(res, obj)
-        self.assertTrue(self.is_unsafe(res), 'returned value of _clean_data (%s) is not marked unsafe.' % res)
+        self.assertNotEqual(res, obj)
+        self.assertEqual(res, clean_obj)
 
     def test_clean_data_bad_dict(self):
         res = self.templar._clean_data(u'{{bad_dict}}')