Repository: ansible/ansible
Pull request: import_role - support templating _from_files with --extra-vars
##### SUMMARY
Fixes #69097

Makes an imported role's tasks_from/handlers_from/vars_from/defaults_from templatable like `import_playbook` and `import_tasks` and added tests for all three.

##### ISSUE TYPE
<!--- Pick one below and delete the rest -->
- Bugfix Pull Request
- Feature Pull Request



Problem statement:
import_role with extra-vars option
##### SUMMARY

variable from extra-vars is not set while import_role is parsed

##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
import_role

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes -->
```paste below
ansible 2.9.7
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/altiire/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.16 (default, Oct 10 2019, 22:02:15) [GCC 8.3.0]
```

##### CONFIGURATION
<!--- Paste verbatim output from "ansible-config dump --only-changed" between quotes -->
```paste below
ALLOW_WORLD_READABLE_TMPFILES(/etc/ansible/ansible.cfg) = True
```

##### OS / ENVIRONMENT
Debian 10 hosted on GCP compute engine


##### STEPS TO REPRODUCE
<!--- Describe exactly how to reproduce the problem, using a minimal test-case -->

<!--- Paste example playbooks or commands between quotes below -->
```bash
ansible-playbook --connection=local --inventory 127.0.0.1, deploy.yml --tags "docker-compose" --extra-vars '{"deployment_name": test,"del_services": [webserver]}'
```
```yaml
---
- name: Create or update a deployment
  hosts: all
  tasks:
    - name: import deploy role with its overriden variables
      import_role:
        name: deploy
        vars_from: "{{ deployment_name }}"
```

<!--- HINT: You can paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
run tasks specified by the tags


##### ACTUAL RESULTS
<!--- Describe what actually happened. If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes -->
```paste below
ansible-playbook 2.9.7
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/altiire/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 2.7.16 (default, Oct 10 2019, 22:02:15) [GCC 8.3.0]
Using /etc/ansible/ansible.cfg as config file
setting up inventory plugins
Set default localhost to 127.0.0.1
Parsed 127.0.0.1, inventory source with host_list plugin
ERROR! Could not find specified file in role: vars/{{ deployment_name }}
```

Hints from issue discussion:
I'm not positive if there is a resolution to this. It may not be feasible to consult extra vars at this phase of the playbook run.  I'd recommend that you switch to `include_role` instead of `import_role` to resolve this issue.
Tags are not taken into account if I use `include_role`
> Tags are not taken into account if I use `include_role`

Look into using `apply` for `include_role`, which allows you to apply tags to the included tasks.
I can't succeed to make it works

my `deploy.yml` file
```yaml
---
- name: Create or update a deployment
  hosts: all

  vars:
    - deployment_template_path: "{{ deployment_path }}/deployment_template"
    - current_deployment_path: "{{ deployment_path }}/{{ deployment_name }}"

  tasks:
    - name: include deploy role with its overriden variables
      include_role:
        name: deploy
        vars_from: "{{ deployment_name }}"
        apply:
          tags:
            - dockercompose
```

and my `roles/deploy/tasks/main.yml`
```yaml
---
- fail:
    msg: The variable 'locale' is not set
  when: locale is none
- debug:
    msg: The variable 'sftp_password' is not set
  when: sftp_password is none
- include: stop-inotify.yml
- include: files.yml
- include: common_database.yml
- include: network.yml
- include: sftp.yml
- include_tasks:
    file: docker.yml
    apply:
      tags:
        - dockercompose
  tags:
    - dockercompose
- include: stop-inotify.yml
- include: backend.yml
- include: db.yml
- include: module_seed.yml
- include: office.yml
- include: start-inotify.yml
- include: cron.yml
```

My tasks in docker.yml are not executed
`apply` only affects the tasks _within_ the `deploy` role. It does not tag the `include_role` itself.  As such if you are using `-t dockercompose` from the CLI, it will skip the `include_role`because it is not tagged.

You would need to use something like:

```
    - name: include deploy role with its overriden variables
      include_role:
        name: deploy
        vars_from: "{{ deployment_name }}"
        apply:
          tags:
            - dockercompose
      tags:
        - dockercompose
```
But if I set `tags` on the `include_role`, all my tasks in the role get tagged too.

```
PLAY [Create or update a deployment] *******************************************************************************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************************************************************************************
ok: [127.0.0.1]

TASK [include deploy role with its overriden variables] ************************************************************************************************************************************************************

TASK [deploy : fail] ***********************************************************************************************************************************************************************************************
skipping: [127.0.0.1]

TASK [deploy : debug] **********************************************************************************************************************************************************************************************
ok: [127.0.0.1] => {
    "msg": "The variable 'sftp_password' is not set"
}

TASK [deploy : Stop inotify service] *******************************************************************************************************************************************************************************
changed: [127.0.0.1]
```
> But if I set `tags` on the `include_role`, all my tasks in the role get tagged too

No, that is what `apply` does.

I think we've confused something here.  You stated:

> Tags are not taken into account if I use `include_role`

Which I assumed meant you wanted to use `tags:` on that `include_role`, but I think what you are saying, is that if a task is tagged somewhere inside of the included role, that it isn't found.  I think what you want instead is to use the `always` tag:

```
    - name: include deploy role with its overriden variables
      include_role:
        name: deploy
        vars_from: "{{ deployment_name }}"
      tags:
        - always
```

Then the `include_role` always happens, and can then still filter tags that are specified on tasks explicitly within the role.
I was thinking of the `always` tag too but the real problem was the story with apply/tags.

Thank you very much, it is now working and I learned a few things.

I let the issue open for the original issue.

Regards,
+1 user with this problem;

Have changed "include_role" on "import_role" and nothing good happened.

Created my own issue here - https://github.com/ansible/ansible/issues/69916

Commit message: Move the fix earlier to include variables from the import_role task

Add tests for import_playbook and import_tasks

Add a note to the documentation

Patch to review:
diff --git a/changelogs/fragments/75269-import_role-support-from-files-templates.yml b/changelogs/fragments/75269-import_role-support-from-files-templates.yml
new file mode 100644
index 00000000000000..1f89a48dd19b8e
--- /dev/null
+++ b/changelogs/fragments/75269-import_role-support-from-files-templates.yml
@@ -0,0 +1,2 @@
+minor_changes:
+  - import_role - Template tasks_from, vars_from, defaults_from, and handlers_from with --extra-vars (https://github.com/ansible/ansible/issues/69097).
diff --git a/docs/docsite/rst/user_guide/playbooks_reuse.rst b/docs/docsite/rst/user_guide/playbooks_reuse.rst
index bbd7118598f809..24619d959bab8f 100644
--- a/docs/docsite/rst/user_guide/playbooks_reuse.rst
+++ b/docs/docsite/rst/user_guide/playbooks_reuse.rst
@@ -33,6 +33,15 @@ You can incorporate multiple playbooks into a main playbook. However, you can on
 
 Importing incorporates playbooks in other playbooks statically. Ansible runs the plays and tasks in each imported playbook in the order they are listed, just as if they had been defined directly in the main playbook.
 
+Imported playbook filenames can use variables defined with ``--extra-vars`` or the ``vars`` keyword:
+
+.. code-block:: yaml
+
+   - import_playbook: "/path/to/{{ extra_var }}"
+   - import_playbook: "{{ playbook }}"
+     vars:
+       playbook: /path/to/playbook.yml
+
 Re-using files and roles
 ========================
 
@@ -53,6 +62,8 @@ Task include and import statements can be used at arbitrary depth.
 
 You can still use the bare :ref:`roles <roles_keyword>` keyword at the play level to incorporate a role in a playbook statically. However, the bare :ref:`include <include_module>` keyword, once used for both task files and playbook-level includes, is now deprecated.
 
+Included and imported task files and roles support templating filenames. ``import_*`` actions need the variables to be available at playbook parse time (which can be done by defining them with ``--extra-vars``, for example), whereas ``include_*`` actions need the variables at runtime.
+
 Includes: dynamic re-use
 ------------------------
 
diff --git a/lib/ansible/playbook/role_include.py b/lib/ansible/playbook/role_include.py
index 59803f27c153be..a9fd219d2d0f65 100644
--- a/lib/ansible/playbook/role_include.py
+++ b/lib/ansible/playbook/role_include.py
@@ -29,6 +29,7 @@
 from ansible.playbook.role.include import RoleInclude
 from ansible.utils.display import Display
 from ansible.module_utils.six import string_types
+from ansible.template import Templar
 
 __all__ = ['IncludeRole']
 
@@ -79,8 +80,15 @@ def get_block_list(self, play=None, variable_manager=None, loader=None):
         ri = RoleInclude.load(self._role_name, play=myplay, variable_manager=variable_manager, loader=loader, collection_list=self.collections)
         ri.vars.update(self.vars)
 
+        if variable_manager is not None:
+            available_variables = variable_manager.get_vars(play=myplay, task=self)
+        else:
+            available_variables = {}
+        templar = Templar(loader=loader, variables=available_variables)
+        from_files = dict((k, templar.template(v)) for k, v in self._from_files.items())
+
         # build role
-        actual_role = Role.load(ri, myplay, parent_role=self._parent_role, from_files=self._from_files,
+        actual_role = Role.load(ri, myplay, parent_role=self._parent_role, from_files=from_files,
                                 from_include=True, validate=self.rolespec_validate)
         actual_role._metadata.allow_duplicates = self.allow_duplicates
 
diff --git a/test/integration/targets/include_import/playbook/test_templated_filenames.yml b/test/integration/targets/include_import/playbook/test_templated_filenames.yml
new file mode 100644
index 00000000000000..2f78ab098b8bca
--- /dev/null
+++ b/test/integration/targets/include_import/playbook/test_templated_filenames.yml
@@ -0,0 +1,47 @@
+- name: test templating import_playbook with extra vars
+  import_playbook: "{{ pb }}"
+
+- name: test templating import_playbook with vars
+  import_playbook: "{{ test_var }}"
+  vars:
+    test_var: validate_templated_playbook.yml
+
+- name: test templating import_tasks
+  hosts: localhost
+  gather_facts: no
+  vars:
+    play_var: validate_templated_tasks.yml
+  tasks:
+    - name: test templating import_tasks with play vars
+      import_tasks: "{{ play_var }}"
+
+    - name: test templating import_tasks with task vars
+      import_tasks: "{{ task_var }}"
+      vars:
+        task_var: validate_templated_tasks.yml
+
+    - name: test templating import_tasks with extra vars
+      import_tasks: "{{ tasks }}"
+
+- name: test templating import_role from_files
+  hosts: localhost
+  gather_facts: no
+  vars:
+    play_var: templated.yml
+  tasks:
+    - name: test templating import_role tasks_from with play vars
+      import_role:
+        name: role1
+        tasks_from: "{{ play_var }}"
+
+    - name: test templating import_role tasks_from with task vars
+      import_role:
+        name: role1
+        tasks_from: "{{ task_var }}"
+      vars:
+        task_var: templated.yml
+
+    - name: test templating import_role tasks_from with extra vars
+      import_role:
+        name: role1
+        tasks_from: "{{ tasks_from }}"
diff --git a/test/integration/targets/include_import/playbook/validate_templated_playbook.yml b/test/integration/targets/include_import/playbook/validate_templated_playbook.yml
new file mode 100644
index 00000000000000..631ee9b47c1a70
--- /dev/null
+++ b/test/integration/targets/include_import/playbook/validate_templated_playbook.yml
@@ -0,0 +1,5 @@
+---
+- hosts: localhost
+  gather_facts: no
+  tasks:
+  - debug: msg="In imported playbook"
diff --git a/test/integration/targets/include_import/playbook/validate_templated_tasks.yml b/test/integration/targets/include_import/playbook/validate_templated_tasks.yml
new file mode 100644
index 00000000000000..16d682d1fba597
--- /dev/null
+++ b/test/integration/targets/include_import/playbook/validate_templated_tasks.yml
@@ -0,0 +1 @@
+- debug: msg="In imported tasks"
diff --git a/test/integration/targets/include_import/roles/role1/tasks/templated.yml b/test/integration/targets/include_import/roles/role1/tasks/templated.yml
new file mode 100644
index 00000000000000..eb9a997676bdfc
--- /dev/null
+++ b/test/integration/targets/include_import/roles/role1/tasks/templated.yml
@@ -0,0 +1 @@
+- debug: msg="In imported role"
diff --git a/test/integration/targets/include_import/runme.sh b/test/integration/targets/include_import/runme.sh
index 8ddbfa61fab5a7..7029ab6d627048 100755
--- a/test/integration/targets/include_import/runme.sh
+++ b/test/integration/targets/include_import/runme.sh
@@ -128,3 +128,10 @@ ansible-playbook test_include_loop.yml "$@"
 ansible-playbook test_include_loop_fqcn.yml "$@"
 
 ansible-playbook include_role_omit/playbook.yml "$@"
+
+# Test templating import_playbook, import_tasks, and import_role files
+ansible-playbook playbook/test_templated_filenames.yml -e "pb=validate_templated_playbook.yml tasks=validate_templated_tasks.yml tasks_from=templated.yml" "$@" | tee out.txt
+cat out.txt
+test "$(grep out.txt -ce 'In imported playbook')" = 2
+test "$(grep out.txt -ce 'In imported tasks')" = 3
+test "$(grep out.txt -ce 'In imported role')" = 3


Code review comments:
diff_hunk: @@ -33,6 +33,15 @@ You can incorporate multiple playbooks into a main playbook. However, you can on
 
 Importing incorporates playbooks in other playbooks statically. Ansible runs the plays and tasks in each imported playbook in the order they are listed, just as if they had been defined directly in the main playbook.
 
+Imported playbook filenames can use variables defined with ``--extra-vars`` or the ``vars`` keyword:
+
+.. code-block:: yaml
+
+   - import_playbook: "/path/to/{{ extra_var }}"
path: docs/docsite/rst/user_guide/playbooks_reuse.rst
text: @user1:
Hm, this took a few reads for me to parse. How about:
```suggestion
   - import_playbook: "/path/to/{{ import_from_extra_var }}"
```

diff_hunk: @@ -79,8 +80,15 @@ def get_block_list(self, play=None, variable_manager=None, loader=None):
         ri = RoleInclude.load(self._role_name, play=myplay, variable_manager=variable_manager, loader=loader, collection_list=self.collections)
         ri.vars.update(self.vars)
 
+        if variable_manager is not None:
+            available_variables = variable_manager.get_vars(play=myplay, task=self)
+        else:
+            available_variables = {}
+        templar = Templar(loader=loader, variables=available_variables)
+        from_files = dict((k, templar.template(v)) for k, v in self._from_files.items())
path: lib/ansible/playbook/role_include.py
text: @user1:
I think we can just do:
```suggestion
        from_files = templar.template(self._from_files)
```

@author:
thanks, why did I overcomplicate that

diff_hunk: @@ -33,6 +33,15 @@ You can incorporate multiple playbooks into a main playbook. However, you can on
 
 Importing incorporates playbooks in other playbooks statically. Ansible runs the plays and tasks in each imported playbook in the order they are listed, just as if they had been defined directly in the main playbook.
 
+Imported playbook filenames can use variables defined with ``--extra-vars`` or the ``vars`` keyword:
+
+.. code-block:: yaml
+
+   - import_playbook: "/path/to/{{ extra_var }}"
+   - import_playbook: "{{ playbook }}"
+     vars:
+       playbook: /path/to/playbook.yml
path: docs/docsite/rst/user_guide/playbooks_reuse.rst
text: @user1:
If we followed the pattern, and showed the other option for defining the path, maybe:
```suggestion
   - import_playbook: "{{ import_from_vars }}"
     vars:
       import_from_vars: /path/to/one_playbook.yml

If you run this playbook with ``ansible-playbook my_playbook -e import_from_extra_var=other_playbook.yml``, Ansible imports both one_playbook.yml and other_playbook.yml.
```