Repository: ansible/ansible

Issue:
pipelining doesn't work with python3.6 on Mac
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest -->
 - Bug Report

##### COMPONENT NAME
<!---
Name of the module, plugin, task or feature
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path
-->
sudo

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.4.2.0
  config file = ./ansible.cfg
  configured module search path = ['./library', './third-party/kubespray/library']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.6.1 (default, Jun 16 2017, 16:42:41) [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)]
```

##### CONFIGURATION
<!---
If using Ansible 2.4 or above, paste the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
```
ANSIBLE_PIPELINING(./ansible.cfg) = True
ANSIBLE_SSH_ARGS(./ansible.cfg) = -o ControlMaster=auto -o ControlPersist=30m -o ConnectionAttempts=100 -o UserKnownHostsFile=/dev/null
DEFAULT_CALLBACK_WHITELIST(./ansible.cfg) = ['profile_tasks']
DEFAULT_GATHERING(./ansible.cfg) = smart
DEFAULT_MODULE_PATH(./ansible.cfg) = ['./library', './third-party/kubespray/library']
DEFAULT_ROLES_PATH(./ansible.cfg) = ['./roles', './third-party/kubespray/roles', '/Users/chr
DEFAULT_STDOUT_CALLBACK(./ansible.cfg) = skippy
DEPRECATION_WARNINGS(./ansible.cfg) = False
HOST_KEY_CHECKING(./ansible.cfg) = False
```

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->
```
ProductName:    Mac OS X
ProductVersion: 10.13.2
BuildVersion:   17C88
```

##### SUMMARY
<!--- Explain the problem briefly -->
When running sudo through ssh with pipelining enabled, the selector doesn't detect any events. Inside ansible/plugins/connection/ssh.py I can see that it correctly detected the become_prompt and sent the password to the stdin. Then it sits at `events = selector.select(timeout)` until the timeout. The next line is never sent/never detected from stdout. 

This works with pipelining disabled on python3.6. This also works on python 2.7 with or without pipelining. 

Additionally, the error message `waiting for privilege escalation prompt` isn't correct in this instance. We saw the prompt at this time and it's now waiting for the privilege escalation success message (BECOME-SUCCESS-*).

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
I'm just gathering the facts for this test:
```
ANSIBLE_BECOME_PASSWORD=not_my_password ansible -m setup k8s1 -i ~/workspace/kubespray/inventory.cfg -b -v -u k8s --private-key=~/.ssh/id_rsa
```

With the following inventory:
<!--- Paste example playbooks or commands between quotes below -->
```
k8s1     ansible_host=192.168.56.101 ip=192.168.56.101 ansible_become_pass="{{ lookup('env', 'ANSIBLE_BECOME_PASSWORD') }}"
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
I expected the sudo command to succeed.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
Timeout occurred. 

<!--- Paste verbatim command output between quotes below -->
```
$ ANSIBLE_BECOME_PASSWORD=not_my_password ansible -m setup k8s1 -i a.cfg -b -v -u k8s --private-key=~/.ssh/id_rsa -vvvvv
ansible 2.4.2.0
  config file = ./ansible.cfg
  configured module search path = ['./library', './third-party/kubespray/library']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.6.1 (default, Jun 16 2017, 16:42:41) [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)]
Using ./ansible.cfg as config file
setting up inventory plugins
Parsed ./a.cfg inventory source with ini plugin
Loading callback plugin minimal of type stdout, v2.0 from /usr/local/lib/python3.6/site-packages/ansible/plugins/callback/__init__.py
META: ran handlers
<192.168.56.101> ESTABLISH SSH CONNECTION FOR USER: k8s
<192.168.56.101> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=30m)(-o)(ConnectionAttempts=100)(-o)(UserKnownHostsFile=/dev/null)
<192.168.56.101> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<192.168.56.101> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="~/.ssh/id_rsa")
<192.168.56.101> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<192.168.56.101> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=k8s)
<192.168.56.101> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<192.168.56.101> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=~/.ansible/cp/38004cf33b)
<192.168.56.101> SSH: EXEC ssh -vvv -o ControlMaster=auto -o ControlPersist=30m -o ConnectionAttempts=100 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o 'IdentityFile="~/.ssh/id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=k8s -o ConnectTimeout=10 -o ControlPath=~/.ansible/cp/38004cf33b 192.168.56.101 '/bin/sh -c '"'"'sudo -H -S  -p "[sudo via ansible, key=ocoqtxdwvgdhofoilcidzwuxjmwrozum] password: " -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-ocoqtxdwvgdhofoilcidzwuxjmwrozum; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''
k8s1 | FAILED! => {
    "msg": "Timeout (12s) waiting for privilege escalation prompt: "
}
```



Discussion:
Files identified in the description:
None

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
I've been experiencing this issue as well since moving to python3 on my mac. Toggling pipelining gets it working again.
@foozmeat Can you describe what you mean by "toggling"? I don't understand it in this context. That usually means setting it to False and then setting it back to True. I don't understand how that would fix this issue. Thanks.
@eyeofthefrog yes you're right, I made a confusing comment. In the past we've had pipelining set to True but have to set it to False if ansible is running under python3 or else we get the error described above.
@foozmeat I see, you meant "toggle to off". I should have understood that, sorry. Thanks for clarifying. 
I wonder if we should instead, just utilize `selectors2` for py2 and py3 as it's not a 1 to 1 backport of selectors from py3.4+.  From the docs:

> All the additional features that selectors2 provides are real-world problems that have occurred and been reported during the lifetime of its maintenance and use within urllib3.

So I'm guessing that one of those additional features, might have resolved the problem.

Can you of you test a change similar to:

```diff
diff --git a/lib/ansible/compat/selectors/__init__.py b/lib/ansible/compat/selectors/__init__.py
index 6458438d6d..42cee28eb8 100644
--- a/lib/ansible/compat/selectors/__init__.py
+++ b/lib/ansible/compat/selectors/__init__.py
@@ -30,14 +30,10 @@ import os.path
 import sys
 
 try:
-    # Python 3.4+
-    import selectors as _system_selectors
+    # backport package installed in the system
+    import selectors2 as _system_selectors
 except ImportError:
-    try:
-        # backport package installed in the system
-        import selectors2 as _system_selectors
-    except ImportError:
-        _system_selectors = None
+    _system_selectors = None
 
 if _system_selectors:
     selectors = _system_selectors
```

needs_info
@sivel I should have mentioned, I tried that too with no success. Good idea though! :)
I confirm this issue also exist when running ansible on Ubuntu 16.04 in python 3 environment. Disabling pipelining provides a workaround
I've been able to bisect this issue to f921369445900dcc756821e40c9257d5359087af

I've not determined the specific cause within that commit yet.
Ok, that commit wasn't it.  It actually addressed a bug that I was "utilizing" in my testing.  I'll go back to it.
Ok, good news is that I've actually figured it out.  I'll work on a pull request tomorrow and will link here with that resolution.

In my testing this has existed, at minimum, back to 2.3.0 with python3.

Resolving patch:
diff --git a/lib/ansible/plugins/connection/ssh.py b/lib/ansible/plugins/connection/ssh.py
--- a/lib/ansible/plugins/connection/ssh.py
+++ b/lib/ansible/plugins/connection/ssh.py
@@ -780,6 +780,9 @@ def _bare_run(self, cmd, in_data, sudoable=True, checkrc=True):
                     if self._flags['become_prompt']:
                         display.debug('Sending become_pass in response to prompt')
                         stdin.write(to_bytes(self._play_context.become_pass) + b'\n')
+                        # On python3 stdin is a BufferedWriter, and we don't have a guarantee
+                        # that the write will happen without a flush
+                        stdin.flush()
                         self._flags['become_prompt'] = False
                         state += 1
                     elif self._flags['become_success']:


Test patch:
diff --git a/test/units/plugins/connection/test_ssh.py b/test/units/plugins/connection/test_ssh.py
--- a/test/units/plugins/connection/test_ssh.py
+++ b/test/units/plugins/connection/test_ssh.py
@@ -459,6 +459,7 @@ def test_password_with_become(self):
         self.mock_selector.get_map.side_effect = lambda: True
 
         return_code, b_stdout, b_stderr = self.conn._run("ssh", "this is input data")
+        self.mock_popen_res.stdin.flush.assert_called_once_with()
         assert return_code == 0
         assert b_stdout == b'abc'
         assert b_stderr == b'123'