Repository: ansible/ansible

Issue:
Regression: Relative paths in roles no longer work as before
<!---
Please verify first that your issue/request is not already reported in GitHub
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
roles

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
v2.2

##### SUMMARY
<!--- Explain the problem briefly -->
When using relative paths with e.g. template or copy, they are no longer searched relative to the role-path. One would expect that following search order: for e.g. template:

 - ./roles/role_name
 - ./roles/role_name/templates
 - ./
 - ./templates

So that the file *files/template.ext.j2* is found as *./roles/role_name/files/template.ext.j2*
This used to work with all previous versions, but seems to be a regression in v2.2.

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem.
For new features, show how the feature would be used.
-->

Consider the following setup.

test134.yml
```yaml
- hosts: localhost
  roles:
  - role: test134
```

roles/test134/tasks/main.yml
```yaml
- debug:
    var: ansible_search_path

- copy:
    src: files/test.txt
    dest: /tmp/test.txt

- template:
    src: files/test.txt.j2
    dest: /tmp/test.txt
```

And 2 files test.txt and test.txt.j2 in *roles/test134/files/*

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
```
[dag@moria ansible.testing]$ ansible-playbook test134.yml 

PLAY [localhost] ***************************************************************

TASK [test134 : debug] *********************************************************
ok: [localhost] => {
    "ansible_search_path": "VARIABLE IS NOT DEFINED!"
}

TASK [test134 : copy] **********************************************************
ok: [localhost]

TASK [test134 : template] ******************************************************
ok: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0
```

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
[dag@moria ansible.testing]$ ansible-playbook test134.yml 

PLAY [localhost] ***************************************************************

TASK [setup] *******************************************************************
ok: [localhost]

TASK [test134 : debug] *********************************************************
ok: [localhost] => {
    "ansible_search_path": [
        "/home/dag/home-made/ansible.testing/roles/test134", 
        "/home/dag/home-made/ansible.testing/roles/test134/tasks", 
        "/home/dag/home-made/ansible.testing"
    ]
}

TASK [test134 : copy] **********************************************************
ok: [localhost]

TASK [test134 : template] ******************************************************
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Unable to find 'files/test.txt.j2' in expected paths."}

PLAY RECAP *********************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=1
```



Discussion:
@bcoca I think this one is interesting to you :-)

Related to https://github.com/ansible/ansible/issues/14341#issuecomment-234559431
Here is some more detail about the used search paths:
```
TASK [test134 : template] ******************************************************
task path: /home/dag/home-made/ansible.testing/roles/test134/tasks/main.yml:8
looking for "files/test.txt.j2" at "/home/dag/home-made/ansible.testing/roles/test134/templates/files/test.txt.j2"
looking for "files/test.txt.j2" at "/home/dag/home-made/ansible.testing/roles/test134/tasks/files/test.txt.j2"
looking for "files/test.txt.j2" at "/home/dag/home-made/ansible.testing/roles/test134/tasks/templates/files/test.txt.j2"
looking for "files/test.txt.j2" at "/home/dag/home-made/ansible.testing/roles/test134/tasks/tasks/files/test.txt.j2"
looking for "files/test.txt.j2" at "/home/dag/home-made/ansible.testing/templates/files/test.txt.j2"
looking for "files/test.txt.j2" at "/home/dag/home-made/ansible.testing/files/test.txt.j2"
fatal: [localhost]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_args": {
            "dest": "/tmp/test.txt", 
            "src": "files/test.txt.j2"
        }, 
        "module_name": "template"
    }, 
    "msg": "Unable to find 'files/test.txt.j2' in expected paths."
}
```

Resolving patch:
diff --git a/lib/ansible/parsing/dataloader.py b/lib/ansible/parsing/dataloader.py
--- a/lib/ansible/parsing/dataloader.py
+++ b/lib/ansible/parsing/dataloader.py
@@ -301,6 +301,7 @@ def path_dwim_relative_stack(self, paths, dirname, source):
                 result = test_path
         else:
             search = []
+            display.debug(u'evaluation_path:\n\t%s' % '\n\t'.join(paths))
             for path in paths:
                 upath = unfrackpath(path)
                 b_upath = to_bytes(upath, errors='surrogate_or_strict')
@@ -315,18 +316,20 @@ def path_dwim_relative_stack(self, paths, dirname, source):
                         search.append(os.path.join(b_mydir, b_source))
                     else:
                         # don't add dirname if user already is using it in source
-                        if b_source.split(b'/')[0] == b_dirname:
-                            search.append(os.path.join(b_upath, b_source))
-                        else:
+                        if b_source.split(b'/')[0] != b_dirname:
                             search.append(os.path.join(b_upath, b_dirname, b_source))
-                        search.append(os.path.join(b_upath, b'tasks', b_source))
+                        search.append(os.path.join(b_upath, b_source))
+
                 elif b_dirname not in b_source.split(b'/'):
                     # don't add dirname if user already is using it in source
-                    search.append(os.path.join(b_upath, b_dirname, b_source))
+                    if b_source.split(b'/')[0] != dirname:
+                        search.append(os.path.join(b_upath, b_dirname, b_source))
                     search.append(os.path.join(b_upath, b_source))
 
             # always append basedir as last resort
-            search.append(os.path.join(to_bytes(self.get_basedir()), b_dirname, b_source))
+            # don't add dirname if user already is using it in source
+            if b_source.split(b'/')[0] != dirname:
+                search.append(os.path.join(to_bytes(self.get_basedir()), b_dirname, b_source))
             search.append(os.path.join(to_bytes(self.get_basedir()), b_source))
 
             display.debug(u'search_path:\n\t%s' % to_text(b'\n\t'.join(search)))


Test patch:
diff --git a/test/integration/targets/lookup_paths/play.yml b/test/integration/targets/lookup_paths/play.yml
--- a/test/integration/targets/lookup_paths/play.yml
+++ b/test/integration/targets/lookup_paths/play.yml
@@ -6,53 +6,40 @@
     - file: path={{playbook_dir}}/files state=directory
     - file: path={{playbook_dir}}/roles/showfile/files state=directory
     - copy: dest={{playbook_dir}}/roles/showfile/files/testfile content='in role files'
-    - copy: dest={{playbook_dir}}/roles/showfile/tasks/testfile content='in role tasks'
     - copy: dest={{playbook_dir}}/roles/showfile/testfile content='in role'
+    - copy: dest={{playbook_dir}}/roles/showfile/tasks/testfile content='in role tasks'
     - copy: dest={{playbook_dir}}/files/testfile content='in files'
     - copy: dest={{playbook_dir}}/testfile content='in local'
-    - set_fact: role_out="in role files" play_out="in files" stage='setup'
 
 - include: testplay.yml
-
-- name: remove from role/files
-  hosts: testhost
-  connection: local
-  gather_facts: false
-  tasks:
-    - file: path={{playbook_dir}}/roles/showfile/files/testfile state=absent
-    - set_fact: role_out="in role tasks" play_out="in files" stage='remove 1'
+  vars:
+    remove: nothing
+    role_out: in role files
+    play_out: in files
 
 - include: testplay.yml
-
-- name: remove from role/tasks
-  hosts: testhost
-  connection: local
-  gather_facts: false
-  tasks:
-    - file: path={{playbook_dir}}/roles/showfile/tasks/testfile state=absent
-    - set_fact: role_out="in files" play_out="in files" stage='remote 2'
+  vars:
+    remove: roles/showfile/files/testfile
+    role_out: in role
+    play_out: in files
 
 - include: testplay.yml
-
-- name: remove from role dir
-  hosts: testhost
-  connection: local
-  gather_facts: false
-  tasks:
-    - file: path={{playbook_dir}}/roles/showfile/testfile state=absent
-    - set_fact: role_out="in files" play_out="in files" stage='remove 3'
+  vars:
+    remove: roles/showfile/testfile
+    role_out: in role tasks
+    play_out: in files
 
 - include: testplay.yml
-
-- name: remove from play/files
-  hosts: testhost
-  connection: local
-  gather_facts: false
-  tasks:
-    - file: path={{playbook_dir}}/files/testfile state=absent
-    - set_fact: role_out="in local" play_out="in local" stage='remove 4'
+  vars:
+    remove: roles/showfile/tasks/testfile
+    role_out: in files
+    play_out: in files
 
 - include: testplay.yml
+  vars:
+    remove: files/testfile
+    role_out: in local
+    play_out: in local
 
 - name: cleanup
   hosts: testhost
diff --git a/test/integration/targets/lookup_paths/testplay.yml b/test/integration/targets/lookup_paths/testplay.yml
--- a/test/integration/targets/lookup_paths/testplay.yml
+++ b/test/integration/targets/lookup_paths/testplay.yml
@@ -2,13 +2,16 @@
   hosts: testhost
   connection: local
   gather_facts: false
+  pre_tasks:
+    - name: remove {{ remove }}
+      file: path={{ playbook_dir }}/{{ remove }} state=absent
   roles:
     - showfile
-  tasks:
+  post_tasks:
     - name: from play
       set_fact: play_result="{{lookup('file', 'testfile')}}"
 
-    - name: output output {{stage}}
+    - name: output stage {{ remove }} removed
       debug: msg="play> {{play_out}}, role> {{role_out}}"
 
     - name: verify that result match expected