Repository: ansible/ansible

Issue:
Interpolation of undefined variables different between with_items and direct reference from inside templates.
##### Issue Type:

Bug Report
##### Ansible Version:

ansible 1.7.1 (identical behaviour in devel, 267b3fb7b74)
##### Environments:

Ubuntu 14.04 managing Ubuntu 14.04 machines.
##### Summary:

The following `group_vars/all`

``` yaml
things:
- name: "works"
- name: "{{ should_explode }}"
```

... correctly interpolates `should_explode` when that variable is provided, but:
- Results in a verbatim `{{ should_explode }}` when referenced from within a `with_items` loop, and
- Correctly explodes if looped over from within a template.

(`error_on_undefined_vars` is set to True. And I say `{{ should_explode }}`, but when used with the copy or template modules I actually get `{# should_explode #}` instead; I'm considering the {{ vs {# confusion a very secondary issue for the sake of this ticket, though.)
##### Steps To Reproduce:

Have the following in a `group_vars/all` file:

``` yaml
things:
- name: "works"
- name: "{{ should_explode }}"
```

Have two templates: `item_name.j2`:

``` jinja2
{{ item.name }}
```

... and `things_names.j2`:

``` jinja2
{% for thing in things %}
{{ thing.name }}
{% endfor %}
```

Have the following tasks in a playbook:

``` yaml
- name: "things with_items"
  template: >
    src=item_name.j2
    dest=/tmp/{{ item.name }}.txt
  with_items: things

- name: "things direct template reference"
  template: >
    src=things_names.j2
    dest=/tmp/things.txt
```

Add an **ansible.cfg** with the following if you want to be sure about the error-on-undefined:

``` ini
[defaults]
error_on_undefined_vars = True
```

Finally, run the hash integration tests with `cd test/integration; make test_hash`.
##### Expected Results:

```
TASK: [test | things with_items] *************************************
fatal: ... <==== ... of some variety.

TASK: [test | things direct template reference] **********************
fatal: [test] => {'msg': "AnsibleUndefinedVariable: One or more undefined variables: 'should_explode' is undefined", 'failed': True}
fatal: [test] => {'msg': 'One or more items failed.', 'failed': True, 'changed': False, 'results': [{'msg': "AnsibleUndefinedVariable: One or more undefined variables: 'should_explode' is undefined", 'fa}
```
##### Actual Results:

```
TASK: [test | things with_items] *************************************
changed: [test] => (item={'name': 'works'})
changed: [test] => (item={'name': u'{# should_explode #}'})  <==== Instead, this.

TASK: [test | things direct template reference] **********************
fatal: [test] => {'msg': "AnsibleUndefinedVariable: One or more undefined variables: 'should_explode' is undefined", 'failed': True}
fatal: [test] => {'msg': "AnsibleUndefinedVariable: One or more undefined variables: 'should_explode' is undefined", 'failed': True}
```

And on the second run (no modifications):

```
TASK: [test | things with_items] *************************************
ok: [test] => (item={'name': 'works'})
failed: [test] => (item={'name': u'{# should_explode #}'}) => {"failed": true, "item": {"name": "{# should_explode #}"}}
msg: this module requires key=value arguments (['src=item_name.j2', 'dest=/tmp/{#', 'should_explode', '#}.txt'])
```
##### Notes:

I'm not sure if this is known behaviour. The second run is definitely weird at least, even if the inconsistency in variable dereferencing is expected.



Discussion:
I just got bitten by this:

``` yaml
- hosts: examplehost
  tasks:
  - name: "a task"
    shell: "echo this is just an example it applies to any action"
    sudo: "yes"
    sudo_user: "{{ userr }}" # oops, typo'd variable name
  vars:
    user: "myuser"
```

Which produces:

```
TASK: [a task] ****************************************************************
failed: [examplehost] => {"failed": true, "parsed": false}
invalid output was: sudo: unknown user: {{
sudo: unable to initialize policy plugin
```

I can submit another ticket, or I can just leave this as a note on this ticket.

And again with the odd `{# ... #}` case:

```
TASK: [app | config | Create the basic deploy target directories] *************
failed: [app01] => (item={# app_path #}) => {"failed": true, "item": "{# app_path #}", "parsed": false}
```

``` yaml
- name: "config | Create the basic deploy target directories"
  file:
    name: "{{ item }}"
    state: "directory"
    owner: "{{ app_user_username }}"
    group: "{{ app_user_username }}"
  with_items:
  - "{{ app_path }}"
  - "{{ app_path_releases }}"
  # ...
```

This looks like a bug down the lazy evaluation path to me.

The "{{ foo }}" becoming "{# foo #}" is a result of the "anti-new parameter injection" security checking stuff coming in which should not come into play here.  I would expect it to remain "{{ foo }}" and result in an undefined variable error.

@damncabbage is this only happening with variables that are undefined, or are you seeing this with all variables?

Only typo'd / undefined variables.

(In the last example, `app_path` was a typo of `app_path_root`. `app_path_releases` was defined and worked fine.)

Great, just wanted to make sure, I'm looking into this now. Thanks!

@damncabbage I've created the above patch, if you'd like to test it out. Using your first example, the following output is the result:

<pre>
PLAY [localhost] ************************************************************** 
TASK: [things with_items] ***************************************************** 
fatal: [127.0.0.1] => undefined variable in items: 'should_explode' is undefined
FATAL: all hosts have already failed -- aborting
</pre>

Tested the patch; it explodes as expected. Thanks. :smiley: 

My test case (simpler than the initial setup):

``` yaml
- hosts: localhost
  tasks:
  - debug:
      var: "{{ item }}"
    with_items: a_list
  vars:
    a_list:
    - "{{ foo }}"
    - "{{ foob }}"
    foo: "Hello"
```

I do agree with @redbaron; please instead consider:

``` diff
  import subprocess
  import contextlib
+ import jinja2.exceptions

  from vault import VaultLib
```

``` diff
              # if not already a list, get ready to evaluate with Jinja2
              # not sure why the "/" is in above code :)
              try:
-                 new_terms = template.template(basedir, "{{ %s }}" % terms, inject)
+                 new_terms = template.template(basedir, terms, inject, convert_bare=True, fail_on_undefined=C.DEFAULT_UNDEFINED_VAR_BEHAVIOR)
                  if isinstance(new_terms, basestring) and "{{" in new_terms:
                      pass
                  else:
                      terms = new_terms
+             except jinja2.exceptions.UndefinedError, e:
+                 raise errors.AnsibleUndefinedVariable('undefined variable in items: %s' % e)
              except:
                  pass
```

(Is it standard practice with this project to also add a case to the automated test suite?)

Feel free to send a PR for that so you get credit, the way I did it was admittedly not pythonic.


Resolving patch:
diff --git a/lib/ansible/utils/__init__.py b/lib/ansible/utils/__init__.py
--- a/lib/ansible/utils/__init__.py
+++ b/lib/ansible/utils/__init__.py
@@ -48,6 +48,7 @@
 import json
 import subprocess
 import contextlib
+import jinja2.exceptions
 
 from vault import VaultLib
 
@@ -1419,11 +1420,13 @@ def listify_lookup_plugin_terms(terms, basedir, inject):
             # if not already a list, get ready to evaluate with Jinja2
             # not sure why the "/" is in above code :)
             try:
-                new_terms = template.template(basedir, "{{ %s }}" % terms, inject)
+                new_terms = template.template(basedir, terms, inject, convert_bare=True, fail_on_undefined=C.DEFAULT_UNDEFINED_VAR_BEHAVIOR)
                 if isinstance(new_terms, basestring) and "{{" in new_terms:
                     pass
                 else:
                     terms = new_terms
+            except jinja2.exceptions.UndefinedError, e:
+                raise errors.AnsibleUndefinedVariable('undefined variable in items: %s' % e)
             except:
                 pass
 


Test patch:
diff --git a/test/units/TestUtils.py b/test/units/TestUtils.py
--- a/test/units/TestUtils.py
+++ b/test/units/TestUtils.py
@@ -541,11 +541,20 @@ def test_safe_eval(self):
 
     def test_listify_lookup_plugin_terms(self):
         basedir = os.path.dirname(__file__)
+
+        # Straight lookups
         self.assertEqual(ansible.utils.listify_lookup_plugin_terms('things', basedir, dict()),
                          ['things'])
         self.assertEqual(ansible.utils.listify_lookup_plugin_terms('things', basedir, dict(things=['one', 'two'])),
                          ['one', 'two'])
 
+        # Variable interpolation
+        self.assertEqual(ansible.utils.listify_lookup_plugin_terms('things', basedir, dict(things=['{{ foo }}', '{{ bar }}'], foo="hello", bar="world")),
+                         ['hello', 'world'])
+        with self.assertRaises(ansible.errors.AnsibleError) as ex:
+            ansible.utils.listify_lookup_plugin_terms('things', basedir, dict(things=['{{ foo }}', '{{ bar_typo }}'], foo="hello", bar="world"))
+        self.assertTrue("undefined variable in items: 'bar_typo'" in ex.exception.msg)
+
     def test_deprecated(self):
         sys_stderr = sys.stderr
         sys.stderr = StringIO.StringIO()