Repository: ansible/ansible

Issue:
Nested jinja2 include can't use home template variables
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
`template`

##### ANSIBLE VERSION
```
cyril@MBP-Cyril:~/dev/test_ansible_jinja2 $ ansible --version
ansible 2.4.2.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/Users/cyril/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/Cellar/ansible/2.4.2.0_2/libexec/lib/python2.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.10 (default, Jul 15 2017, 17:16:57) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.31)]
```

##### CONFIGURATION
```
cyril@MBP-Cyril:~/dev/test_ansible_jinja2 $ ansible-config dump --only-changed
ANSIBLE_PIPELINING(/etc/ansible/ansible.cfg) = False
ANSIBLE_SSH_CONTROL_PATH(/etc/ansible/ansible.cfg) = %(directory)s/%%h-%%r
HOST_KEY_CHECKING(/etc/ansible/ansible.cfg) = False
```
##### OS / ENVIRONMENT

```macOS High Sierra‎ version 10.13```

##### SUMMARY

Since I updated my version of Ansible from 2.3.2 to 2.4.2, I can't generate a jinja2 template which have 2 nested includes with a variable at the template base. 

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

Command to execute :  `ansible-playbook -c local -i "localhost," test.yml`

`test.yml` : 
```yaml
- hosts: all
  vars:
    boxes:
      - "test1"
      - "test2"
  tasks:
    - template:
        src: templates/boxes.j2
        dest: /tmp/test_boxes.txt
```

`templates/boxes.j2` : 
```yaml
{% for box in boxes %}
{% include 'box.j2' %}

{% endfor %}
```

`templates/box.j2`  : 
```
{% if box is defined %}
{% include 'box_service.j2' %}
{% endif %}
```

`templates/box_service.j2`  : 
```
{{ box }}
```

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->

From 2.3.2 : 
```
test1
test2
```

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
cyril@MBP-Cyril:~/dev/test_ansible_jinja2 $ ansible-playbook -vvv -c local -i "localhost," test.yml
ansible-playbook 2.4.2.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/Users/cyril/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/Cellar/ansible/2.4.2.0_2/libexec/lib/python2.7/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 2.7.10 (default, Jul 15 2017, 17:16:57) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.31)]
Using /etc/ansible/ansible.cfg as config file
Parsed localhost, inventory source with host_list plugin

PLAYBOOK: test.yml ***************************************************************************************************
1 plays in test.yml

PLAY [all] ***********************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************
Using module file /usr/local/Cellar/ansible/2.4.2.0_2/libexec/lib/python2.7/site-packages/ansible/modules/system/setup.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: cyril
<localhost> EXEC /bin/sh -c 'echo ~ && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/cyril/.ansible/tmp/ansible-tmp-1516033326.51-161942920865376 `" && echo ansible-tmp-1516033326.51-161942920865376="` echo /Users/cyril/.ansible/tmp/ansible-tmp-1516033326.51-161942920865376 `" ) && sleep 0'
<localhost> PUT /var/folders/p1/gg7vqthj3dl6zd2g6fjj19dw0000gn/T/tmpMApNIP TO /Users/cyril/.ansible/tmp/ansible-tmp-1516033326.51-161942920865376/setup.py
<localhost> EXEC /bin/sh -c 'chmod u+x /Users/cyril/.ansible/tmp/ansible-tmp-1516033326.51-161942920865376/ /Users/cyril/.ansible/tmp/ansible-tmp-1516033326.51-161942920865376/setup.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /Users/cyril/.ansible/tmp/ansible-tmp-1516033326.51-161942920865376/setup.py; rm -rf "/Users/cyril/.ansible/tmp/ansible-tmp-1516033326.51-161942920865376/" > /dev/null 2>&1 && sleep 0'
ok: [localhost]
META: ran handlers

TASK [template] ******************************************************************************************************
task path: /Users/cyril/dev/test_ansible_jinja2/test.yml:7
fatal: [localhost]: FAILED! => {
    "changed": false,
    "msg": "AnsibleUndefinedVariable: 'box' is undefined"
}
	to retry, use: --limit @/Users/cyril/dev/test_ansible_jinja2/test.retry

PLAY RECAP ***********************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1
```



Discussion:
Files identified in the description:
* [lib/ansible/modules/files/template.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/files/template.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
@JqckB Your test case fails even on `2.3.2` for me. Can you please check if it really worked on `2.3.2` for you? Thanks! It would help with debugging if I could manage it to not fail on some version of Ansible.
Ok, there is the paste of ansible-playbook on 2.3.2 (OS X) : 

```
cyril@MBP-Cyril:~/dev/test_ansible_jinja2 $ ansible-playbook -vvv -c local -i "localhost," test.yml
Using /usr/local/etc/ansible/ansible.cfg as config file

PLAYBOOK: test.yml ************************************************************************************
1 plays in test.yml

PLAY [all] ********************************************************************************************

TASK [Gathering Facts] ********************************************************************************
Using module file /usr/local/Cellar/ansible/2.3.2.0_1/libexec/lib/python2.7/site-packages/ansible/modules/system/setup.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: cyril
<localhost> EXEC /bin/sh -c 'echo ~ && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/cyril/.ansible/tmp/ansible-tmp-1516209027.66-101886160856463 `" && echo ansible-tmp-1516209027.66-101886160856463="` echo /Users/cyril/.ansible/tmp/ansible-tmp-1516209027.66-101886160856463 `" ) && sleep 0'
<localhost> PUT /var/folders/p1/gg7vqthj3dl6zd2g6fjj19dw0000gn/T/tmpiUGWpN TO /Users/cyril/.ansible/tmp/ansible-tmp-1516209027.66-101886160856463/setup.py
<localhost> EXEC /bin/sh -c 'chmod u+x /Users/cyril/.ansible/tmp/ansible-tmp-1516209027.66-101886160856463/ /Users/cyril/.ansible/tmp/ansible-tmp-1516209027.66-101886160856463/setup.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /Users/cyril/.ansible/tmp/ansible-tmp-1516209027.66-101886160856463/setup.py; rm -rf "/Users/cyril/.ansible/tmp/ansible-tmp-1516209027.66-101886160856463/" > /dev/null 2>&1 && sleep 0'
ok: [localhost]
META: ran handlers

TASK [template] ***************************************************************************************
task path: /Users/cyril/dev/test_ansible_jinja2/test.yml:7
Using module file /usr/local/Cellar/ansible/2.3.2.0_1/libexec/lib/python2.7/site-packages/ansible/modules/files/stat.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: cyril
<localhost> EXEC /bin/sh -c 'echo ~ && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/cyril/.ansible/tmp/ansible-tmp-1516209028.35-211178545447115 `" && echo ansible-tmp-1516209028.35-211178545447115="` echo /Users/cyril/.ansible/tmp/ansible-tmp-1516209028.35-211178545447115 `" ) && sleep 0'
<localhost> PUT /var/folders/p1/gg7vqthj3dl6zd2g6fjj19dw0000gn/T/tmpQU1o1u TO /Users/cyril/.ansible/tmp/ansible-tmp-1516209028.35-211178545447115/stat.py
<localhost> EXEC /bin/sh -c 'chmod u+x /Users/cyril/.ansible/tmp/ansible-tmp-1516209028.35-211178545447115/ /Users/cyril/.ansible/tmp/ansible-tmp-1516209028.35-211178545447115/stat.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /Users/cyril/.ansible/tmp/ansible-tmp-1516209028.35-211178545447115/stat.py; rm -rf "/Users/cyril/.ansible/tmp/ansible-tmp-1516209028.35-211178545447115/" > /dev/null 2>&1 && sleep 0'
<localhost> EXEC /bin/sh -c 'echo ~ && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/cyril/.ansible/tmp/ansible-tmp-1516209028.63-43965334885285 `" && echo ansible-tmp-1516209028.63-43965334885285="` echo /Users/cyril/.ansible/tmp/ansible-tmp-1516209028.63-43965334885285 `" ) && sleep 0'
Using module file /usr/local/Cellar/ansible/2.3.2.0_1/libexec/lib/python2.7/site-packages/ansible/modules/files/stat.py
<localhost> PUT /var/folders/p1/gg7vqthj3dl6zd2g6fjj19dw0000gn/T/tmpBP2MzX TO /Users/cyril/.ansible/tmp/ansible-tmp-1516209028.63-43965334885285/stat.py
<localhost> EXEC /bin/sh -c 'chmod u+x /Users/cyril/.ansible/tmp/ansible-tmp-1516209028.63-43965334885285/ /Users/cyril/.ansible/tmp/ansible-tmp-1516209028.63-43965334885285/stat.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /Users/cyril/.ansible/tmp/ansible-tmp-1516209028.63-43965334885285/stat.py && sleep 0'
<localhost> PUT /var/folders/p1/gg7vqthj3dl6zd2g6fjj19dw0000gn/T/tmpHU5BMt TO /Users/cyril/.ansible/tmp/ansible-tmp-1516209028.63-43965334885285/source
<localhost> EXEC /bin/sh -c 'chmod u+x /Users/cyril/.ansible/tmp/ansible-tmp-1516209028.63-43965334885285/ /Users/cyril/.ansible/tmp/ansible-tmp-1516209028.63-43965334885285/source && sleep 0'
Using module file /usr/local/Cellar/ansible/2.3.2.0_1/libexec/lib/python2.7/site-packages/ansible/modules/files/copy.py
<localhost> PUT /var/folders/p1/gg7vqthj3dl6zd2g6fjj19dw0000gn/T/tmpQhv3Vs TO /Users/cyril/.ansible/tmp/ansible-tmp-1516209028.63-43965334885285/copy.py
<localhost> EXEC /bin/sh -c 'chmod u+x /Users/cyril/.ansible/tmp/ansible-tmp-1516209028.63-43965334885285/ /Users/cyril/.ansible/tmp/ansible-tmp-1516209028.63-43965334885285/copy.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /Users/cyril/.ansible/tmp/ansible-tmp-1516209028.63-43965334885285/copy.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /Users/cyril/.ansible/tmp/ansible-tmp-1516209028.63-43965334885285/ > /dev/null 2>&1 && sleep 0'
changed: [localhost] => {
    "changed": true,
    "checksum": "b9a40f46f9b54eecf936d82c89b4b21f6f654e4d",
    "dest": "/tmp/test_boxes.txt",
    "gid": 20,
    "group": "staff",
    "invocation": {
        "module_args": {
            "attributes": null,
            "backup": false,
            "content": null,
            "delimiter": null,
            "dest": "/tmp/test_boxes.txt",
            "directory_mode": null,
            "follow": true,
            "force": true,
            "group": null,
            "mode": null,
            "original_basename": "boxes.j2",
            "owner": null,
            "regexp": null,
            "remote_src": null,
            "selevel": null,
            "serole": null,
            "setype": null,
            "seuser": null,
            "src": "/Users/cyril/.ansible/tmp/ansible-tmp-1516209028.63-43965334885285/source",
            "unsafe_writes": null,
            "validate": null
        }
    },
    "md5sum": "e51dfbea83de9c7e6b49560089d8a170",
    "mode": "0644",
    "owner": "cyril",
    "size": 12,
    "src": "/Users/cyril/.ansible/tmp/ansible-tmp-1516209028.63-43965334885285/source",
    "state": "file",
    "uid": 501
}
META: ran handlers
META: ran handlers

PLAY RECAP ********************************************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0
```

And the result file : 

```
cyril@MBP-Cyril:~/dev/test_ansible_jinja2 $ cat /tmp/test_boxes.txt
test1
test2
```

Don't hesitate to ask me anything else if it can help you to debug.

Resolving patch:
diff --git a/lib/ansible/template/vars.py b/lib/ansible/template/vars.py
--- a/lib/ansible/template/vars.py
+++ b/lib/ansible/template/vars.py
@@ -121,4 +121,11 @@ def add_locals(self, locals):
         '''
         if locals is None:
             return self
-        return AnsibleJ2Vars(self._templar, self._globals, locals=locals, *self._extras)
+
+        # FIXME run this only on jinja2>=2.9?
+        # prior to version 2.9, locals contained all of the vars and not just the current
+        # local vars so this was not necessary for locals to propagate down to nested includes
+        new_locals = self._locals.copy()
+        new_locals.update(locals)
+
+        return AnsibleJ2Vars(self._templar, self._globals, locals=new_locals, *self._extras)


Test patch:
diff --git a/test/integration/targets/template/tasks/main.yml b/test/integration/targets/template/tasks/main.yml
--- a/test/integration/targets/template/tasks/main.yml
+++ b/test/integration/targets/template/tasks/main.yml
@@ -32,23 +32,23 @@
   template: src=foo.j2 dest={{output_dir}}/foo.templated mode=0644
   register: template_result
 
-- assert: 
-    that: 
-        - "'changed' in template_result" 
-        - "'dest' in template_result" 
-        - "'group' in template_result" 
-        - "'gid' in template_result" 
-        - "'md5sum' in template_result" 
+- assert:
+    that:
+        - "'changed' in template_result"
+        - "'dest' in template_result"
+        - "'group' in template_result"
+        - "'gid' in template_result"
+        - "'md5sum' in template_result"
         - "'checksum' in template_result"
-        - "'owner' in template_result" 
-        - "'size' in template_result" 
-        - "'src' in template_result" 
-        - "'state' in template_result" 
-        - "'uid' in template_result" 
+        - "'owner' in template_result"
+        - "'size' in template_result"
+        - "'src' in template_result"
+        - "'state' in template_result"
+        - "'uid' in template_result"
 
 - name: verify that the file was marked as changed
-  assert: 
-    that: 
+  assert:
+    that:
       - "template_result.changed == true"
 
 # test for import with context on jinja-2.9 See https://github.com/ansible/ansible/issues/20494
@@ -69,6 +69,21 @@
         - 'diff_result.stdout == ""'
         - "diff_result.rc == 0"
 
+# test for nested include https://github.com/ansible/ansible/issues/34886
+- name: test if parent variables are defined in nested include
+  template: src=for_loop.j2 dest={{output_dir}}/for_loop.templated mode=0644
+
+- name: save templated output
+  shell: "cat {{output_dir}}/for_loop.templated"
+  register: for_loop_out
+- debug: var=for_loop_out
+- name: verify variables got templated
+  assert:
+    that:
+        - '"foo" in for_loop_out.stdout'
+        - '"bar" in for_loop_out.stdout'
+        - '"bam" in for_loop_out.stdout'
+
 # test for 'import as' on jinja-2.9 See https://github.com/ansible/ansible/issues/20494
 - name: fill in a template using import as ala fails2 case in issue 20494
   template: src=import_as.j2 dest={{output_dir}}/import_as.templated mode=0644
@@ -258,8 +273,8 @@
   register: templated
 
 - name: verify that the file was marked as changed in check mode but was not created
-  assert: 
-    that: 
+  assert:
+    that:
       - "not templated.stat.exists"
       - "template_result is changed"
 
@@ -272,13 +287,13 @@
   check_mode: True
 
 - name: verify that the file was marked as not changes in check mode
-  assert: 
-    that: 
+  assert:
+    that:
       - "template_result is not changed"
       - "'templated_var_loaded' in lookup('file', '{{output_dir | expanduser}}/short.templated')"
 
 - name: change var for the template
-  set_fact: 
+  set_fact:
     templated_var: "changed"
 
 - name: fill in a basic template with changed var in check mode
@@ -287,8 +302,8 @@
   check_mode: True
 
 - name: verify that the file was marked as changed in check mode but the content was not changed
-  assert: 
-    that: 
+  assert:
+    that:
       - "'templated_var_loaded' in lookup('file', '{{output_dir | expanduser }}/short.templated')"
       - "template_result is changed"
 
diff --git a/test/integration/targets/template/templates/for_loop.j2 b/test/integration/targets/template/templates/for_loop.j2
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/template/templates/for_loop.j2
@@ -0,0 +1,4 @@
+{% for par_var in parent_vars %}
+{% include 'for_loop_include.j2' %}
+
+{% endfor %}
diff --git a/test/integration/targets/template/templates/for_loop_include.j2 b/test/integration/targets/template/templates/for_loop_include.j2
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/template/templates/for_loop_include.j2
@@ -0,0 +1,3 @@
+{% if par_var is defined %}
+{% include 'for_loop_include_nested.j2' %}
+{% endif %}
diff --git a/test/integration/targets/template/templates/for_loop_include_nested.j2 b/test/integration/targets/template/templates/for_loop_include_nested.j2
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/template/templates/for_loop_include_nested.j2
@@ -0,0 +1 @@
+{{ par_var }}