Repository: ansible/ansible

Issue:
ansible_python_interpreter ignored in Ansible 2.4 for eos* that use network_cli module

##### ISSUE TYPE
 - Bug Report


##### COMPONENT NAME
eos_command

##### ANSIBLE VERSION
```
[user@xcrlxsie02p ansible]$ ansible --version
ansible 2.4.2.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.6/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.6.6 (r266:84292, Nov 21 2013, 10:50:32) [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)]
```

##### CONFIGURATION
<!---
If using Ansible 2.4 or above, paste the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->

##### OS / ENVIRONMENT
This host running Ansible has both python2.6 and 2.7

##### SUMMARY
The eos modules that use the network_cli plugin seem to ignore the ansible_python_interpreter variable

##### STEPS TO REPRODUCE
Launch the  playbook below  on a host where python2.7 isnt the default and set the ansible_python_interpreter variable

<!--- Paste example playbooks or commands between quotes below -->
```yaml

user@xcrlxsie02p ansible]$ cat playbooks/test.yml
---
- name: Search for unused SFPs
  hosts: "{{ device }}"
  connection: local

  tasks:
    - name: Run "show interface" command on the switch
      eos_command:
        commands: "show interfaces status"
        provider: '{{ eos_connection_cli }}'

  vars:
    eos_connection_cli:
      host: "{{ inventory_hostname }}"
      username: <user>
      password: <pass>
      transport: cli

[user@xcrlxsie02p ansible]$ cat hosts
[hosts]
10.125.167.130

[all:vars]
ansible_python_interpreter=/opt/rh/python27/root/usr/bin/python2.7

```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
It would use the python environment pointed to by ansible_python_interpreter=/opt/rh/python27/root/usr/bin/python2.7

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

user@xcrlxsie02p ansible]$ ansible-playbook -i hosts playbooks/test.yml -e device=10.125.167.130

PLAY [Search for unused SFPs] ***********************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************
ok: [10.125.167.130]

TASK [Run "show interface" command on the switch] ***************************************************************************************
fatal: [10.125.167.130]: FAILED! => {"changed": false, "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell"}
        to retry, use: --limit @/home/user/automation/ansible/playbooks/test.retry

PLAY RECAP ******************************************************************************************************************************
10.125.167.130             : ok=1    changed=0    unreachable=0    failed=1

It fails because there is no paramiko installed on python2.6

```

```



Discussion:
Files identified in the description:
* [lib/ansible/modules/network/eos/eos_command.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/network/eos/eos_command.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @privateip @trishnaguha
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Does this work on `devel` or the `stable-2.5` branches?
Also if you change to `connection: network_cli` is it any different?
The issue here seems to be that `ansible-connection` is getting called using `#!/usr/bin/env python` rather than respecting `ansible_python_interpreter`. Fixing this is more complicated than regular modules as we don't have the opportunity to rewrite the shebang in this case.
Maybe we need to document this as a limitation?
Can we detect and set a meaningful error message? 
We set it to 'system.python'  for localhost to match the python that Ansible is using, you might need to do something similar.

Maybe execute Ptyhon and give it ansible-connection as a script vs relying on the shebang.
network_cli does not help, still getting the same error. From the comments above I understand that the issue is acknowledged and will either get fixed or marked as a limitation. 
> Maybe execute Ptyhon and give it ansible-connection as a script vs relying on the shebang.

That was my first thought, but AFAIU, `Popen()` with `shell=False` only searches $PATH for the first parameter, so we'd have to locate and provide the full path to `ansible-connection`. There doesn't seem to be a built-in way to search $PATH, but I might be missing something

Resolving patch:
diff --git a/lib/ansible/executor/task_executor.py b/lib/ansible/executor/task_executor.py
--- a/lib/ansible/executor/task_executor.py
+++ b/lib/ansible/executor/task_executor.py
@@ -9,6 +9,7 @@
 import time
 import json
 import subprocess
+import sys
 import traceback
 
 from ansible import constants as C
@@ -855,7 +856,11 @@ def _start_connection(self):
         Starts the persistent connection
         '''
         master, slave = pty.openpty()
-        p = subprocess.Popen(["ansible-connection", to_text(os.getppid())], stdin=slave, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+
+        python = sys.executable
+        # Assume ansible-connection is in the same dir as sys.argv[0]
+        ansible_connection = os.path.join(os.path.dirname(sys.argv[0]), 'ansible-connection')
+        p = subprocess.Popen([python, ansible_connection, to_text(os.getppid())], stdin=slave, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
         stdin = os.fdopen(master, 'wb', 0)
         os.close(slave)
 
@@ -874,9 +879,12 @@ def _start_connection(self):
         stdin.close()
 
         if p.returncode == 0:
-            result = json.loads(to_text(stdout))
+            result = json.loads(to_text(stdout, errors='surrogate_then_replace'))
         else:
-            result = json.loads(to_text(stderr))
+            try:
+                result = json.loads(to_text(stderr, errors='surrogate_then_replace'))
+            except json.decoder.JSONDecodeError:
+                result = {'error': to_text(stderr, errors='surrogate_then_replace')}
 
         if 'messages' in result:
             for msg in result.get('messages'):
@@ -884,8 +892,9 @@ def _start_connection(self):
 
         if 'error' in result:
             if self._play_context.verbosity > 2:
-                msg = "The full traceback is:\n" + result['exception']
-                display.display(msg, color=C.COLOR_ERROR)
+                if result.get('exception'):
+                    msg = "The full traceback is:\n" + result['exception']
+                    display.display(msg, color=C.COLOR_ERROR)
             raise AnsibleError(result['error'])
 
         return result['socket_path']
diff --git a/lib/ansible/plugins/connection/persistent.py b/lib/ansible/plugins/connection/persistent.py
--- a/lib/ansible/plugins/connection/persistent.py
+++ b/lib/ansible/plugins/connection/persistent.py
@@ -17,6 +17,7 @@
 import pty
 import json
 import subprocess
+import sys
 
 from ansible import constants as C
 from ansible.plugins.connection import ConnectionBase
@@ -75,7 +76,11 @@ def _start_connection(self):
         Starts the persistent connection
         '''
         master, slave = pty.openpty()
-        p = subprocess.Popen(["ansible-connection", to_text(os.getppid())], stdin=slave, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+
+        python = sys.executable
+        # Assume ansible-connection is in the same dir as sys.argv[0]
+        ansible_connection = os.path.join(os.path.dirname(sys.argv[0]), 'ansible-connection')
+        p = subprocess.Popen([python, ansible_connection, to_text(os.getppid())], stdin=slave, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
         stdin = os.fdopen(master, 'wb', 0)
         os.close(slave)
 
@@ -96,7 +101,10 @@ def _start_connection(self):
         if p.returncode == 0:
             result = json.loads(to_text(stdout, errors='surrogate_then_replace'))
         else:
-            result = json.loads(to_text(stderr, errors='surrogate_then_replace'))
+            try:
+                result = json.loads(to_text(stderr, errors='surrogate_then_replace'))
+            except json.decoder.JSONDecodeError:
+                result = {'error': to_text(stderr, errors='surrogate_then_replace')}
 
         if 'messages' in result:
             for msg in result.get('messages'):
@@ -104,8 +112,9 @@ def _start_connection(self):
 
         if 'error' in result:
             if self._play_context.verbosity > 2:
-                msg = "The full traceback is:\n" + result['exception']
-                display.display(msg, color=C.COLOR_ERROR)
+                if result.get('exception'):
+                    msg = "The full traceback is:\n" + result['exception']
+                    display.display(msg, color=C.COLOR_ERROR)
             raise AnsibleError(result['error'])
 
         return result['socket_path']


Test patch: