Repository: ansible/ansible

Issue:
pywinrm doesn't recognize variable expansion from group_vars
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
pywinrm-0.2.2

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
default version:
```
ansible 2.2.1.0
  config file = /home/username/ansible/ansible.cfg
  configured module search path = Default w/o overrides
```

also tried:
```
ansible 2.4.0 (devel 902bc58509) last updated 2017/04/12 11:28:17 (GMT -600)
  config file = /home/username/ansible/ansible.cfg
  configured module search path = [u'/home/username/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /var/tmp/username/ansible/lib/ansible
  executable location = /var/tmp/username/ansible/bin/ansible
  python version = 2.7.5 (default, Aug  2 2016, 04:20:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]
```
##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
n/a

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
-->
RHEL 7.3
python-2.7.5
pywinrm-0.2.2
ansible-2.2.1.0
Windows Server 2016

##### SUMMARY
<!--- Explain the problem briefly -->

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
The Windows 2016 server in this example is named "gravis".

<!--- Paste example playbooks or commands between quotes below -->
hosts:
```
[windows]
gravis
```

group_vars/windows.yml:
```
ansible_connection: winrm
ansible_winrm_scheme: https
ansible_winrm_transport: certificate
ansible_winrm_server_cert_validation: ignore
ansible_winrm_cert_pem: keys/{{ inventory_hostname }}.pem
ansible_winrm_cert_key_pem: keys/{{ inventory_hostname }}.key
```

plays/win_ping.yml
```
---
- hosts: windows
  gather_facts: false
  tasks:
  - debug: msg=ansible_winrm_cert_pem={{ ansible_winrm_cert_pem }}
  - name: win_ping
    win_ping:
```

host_vars/gravis
```
ansible_winrm_cert_pem: keys/gravis.pem
ansible_winrm_cert_key_pem: keys/gravis.key
```
<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
It is expected that the host-specific ansible_winrm_cert_pem and ansible_winrm_cert_key_pem be used. If the host_vars/gravis file that contains the hard-coded hostname is used, then the results are:
```
$ ansible-playbook plays/win_ping.yml -l gravis
PLAY [windows] *****************************************************************

TASK [debug] *******************************************************************
ok: [gravis] => {
    "msg": "ansible_winrm_cert_pem=keys/gravis.pem"
}

TASK [win_ping] ****************************************************************
ok: [gravis]

PLAY RECAP *********************************************************************
gravis                     : ok=2    changed=0    unreachable=0    failed=0   
```

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
If the host_vars/gravis is not used, and only the group_vars/windows.yml is used, the output shows that {{ inventory_hostname }} is being expanded for the debug task, but the win_ping task gives an error showing it does not recognize the variable expansion. The results are:
```
$ ansible-playbook -vvvv plays/win_ping.yml -l gravis
Using /home/username/ansible/ansible.cfg as config file
Loading callback plugin default of type stdout, v2.0 from /usr/lib/python2.7/site-packages/ansible/plugins/callback/__init__.pyc

PLAYBOOK: win_ping.yml *********************************************************
1 plays in plays/win_ping.yml

PLAY [windows] *****************************************************************

TASK [debug] *******************************************************************
task path: /home/username/ansible/plays/win_ping.yml:5
ok: [gravis] => {
    "msg": "ansible_winrm_cert_pem=keys/gravis.pem"
}

TASK [win_ping] ****************************************************************
task path: /home/username/ansible/plays/win_ping.yml:6
Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/windows/win_ping.ps1
<gravis> ESTABLISH WINRM CONNECTION FOR USER: None on PORT 5986 TO gravis
fatal: [gravis]: UNREACHABLE! => {
    "changed": false, 
    "msg": "certificate: cert_pem file not found (keys/{{ inventory_hostname }}.pem)", 
    "unreachable": true
}
	to retry, use: --limit @retry-files/win_ping.retry

PLAY RECAP *********************************************************************
gravis                     : ok=1    changed=0    unreachable=1    failed=0   
```



Discussion:

Resolving patch:
diff --git a/lib/ansible/executor/task_executor.py b/lib/ansible/executor/task_executor.py
--- a/lib/ansible/executor/task_executor.py
+++ b/lib/ansible/executor/task_executor.py
@@ -31,6 +31,7 @@
 from ansible.module_utils._text import to_text
 from ansible.playbook.conditional import Conditional
 from ansible.playbook.task import Task
+from ansible.plugins.connection import ConnectionBase
 from ansible.template import Templar
 from ansible.utils.encrypt import key_for_hostname
 from ansible.utils.listify import listify_lookup_plugin_terms
@@ -476,16 +477,17 @@ def _execute(self, variables=None):
                 self._play_context.remote_addr != self._connection._play_context.remote_addr):
             self._connection = self._get_connection(variables=variables, templar=templar)
             hostvars = variables.get('hostvars', None)
-            if hostvars:
+            # only template the vars if the connection actually implements set_host_overrides
+            # NB: this is expensive, and should be removed once connection-specific vars are being handled by play_context
+            sho_impl = getattr(type(self._connection), 'set_host_overrides', None)
+            if hostvars and sho_impl and sho_impl != ConnectionBase.set_host_overrides:
                 try:
-                    target_hostvars = hostvars.raw_get(self._host.name)
+                    target_hostvars = hostvars.get(self._host.name)
                 except:
                     # FIXME: this should catch the j2undefined error here
                     #        specifically instead of all exceptions
                     target_hostvars = dict()
-            else:
-                target_hostvars = dict()
-            self._connection.set_host_overrides(host=self._host, hostvars=target_hostvars)
+                self._connection.set_host_overrides(host=self._host, hostvars=target_hostvars)
         else:
             # if connection is reused, its _play_context is no longer valid and needs
             # to be replaced with the one templated above, in case other data changed


Test patch: