Repository: ansible/ansible

Issue:
'become' causes failure when used on the first task of a play
##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
become

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.4.0 (devel e0a7105c2d) last updated 2017/04/24 11:12:48 (GMT -500)
  config file = /jaguar/ansible/ansible.cfg
  configured module search path = [u'/home/tom/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /tmp/ansible/lib/ansible
  executable location = /tmp/ansible/bin/ansible
  python version = 2.7.13 (default, Apr 21 2017, 19:10:36) [GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)]
```
Behavior is also confirmed on 2.2 and 2.3. Last known working was 2.0.
##### CONFIGURATION
```
pipelining = True
allow_world_readable_tmpfiles = True
```
Hostvars configuration is very basic, to isolate issue.
##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
-->
Controller and target are both running FreeBSD 10.3. Target Python version is 2.7.11

##### SUMMARY
<!--- Explain the problem briefly -->
Under certain circumstances, tasks that specify become/sudo functionality will cause the play to fail. The set of circumstances seems to be very specific:

1. Pipelining for ssh connections MUST be set true in ansible.cfg.
1. A valid ControlPath ssh connection MUST NOT exist (occurs when a new connection is established).
1. The task in question must be the first in the play to run against a host.

If these three conditions are satisfied, the task will fail or timeout, depending on Ansible version. In version 2.2, the task fails with message "Incorrect sudo password". In version 2.3, the task times out, with message "Timeout (12s) waiting for privilege escalation. The error has also been confirmed on the devel branch, as recently as 09b247dd34ff9dd18a7f57ccd0e2319dd922aaa4. Contrary to these errors, the target system log indicates that sudo has been successfully invoked by ansible.

Once the ssh ControlPath connection is established, the task works as expected. This can be demonstrated by immediately re-running the same playbook, or ansible command.

I'm not sure if it's relevant or not, but this testing specifically looks at cases where ansible is invoked as an unprivileged user, and becomes a different unprivileged user. I have not explicitly tested cases where root is used.
##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
The behavior can be reproduced with ansible or ansible-playbook. For a one-off ansible call

<!--- Paste example playbooks or commands between quotes below -->
```
ansible -b -K -vvvvv --become-user=web-admin -a "/usr/bin/true" -i staging wildcat
```

A playbook example follows:

```
---

- hosts: wildcat
  gather_facts: no
  strategy: debug

  tasks:

  #- name: prime the pump?
  #  when: skip_sudo is not defined or not skip_sudo
  #  changed_when: false
  #  command: /usr/bin/true

  - name: check sudo password - if this fails, you need '-K' in your command-line!
    when: skip_sudo is not defined or not skip_sudo
    become: true
    become_user: web-admin
    changed_when: false
    command: /usr/bin/true
```

I have identified a couple of workarounds to this behavior.
1. Disabling pipelining prevents the issue from occurring.
1. Uncommenting the "prime the pump" task in the above playbook example also resolves the issue. Presumably because the "become" task is not the first one to run against the host.

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
I expect both examples to succeed.
##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
The output below shows the playbook being run, and "redo" being specified as soon as the debugger is entered, demonstrating that the bug is not exhibited when an ssh connection is already established.
```
[tom@jaguar-a /jaguar/ansible]$ ansible-playbook -K -vvvvv 999-test.yml
Using /jaguar/ansible/ansible.cfg as config file
SUDO password: 
Loading callback plugin default of type stdout, v2.0 from /tmp/ansible/lib/ansible/plugins/callback/__init__.pyc

PLAYBOOK: 999-test.yml ******************************************************************************************************************************************************************************************************************************************************************************************
1 plays in 999-test.yml

PLAY [wildcat] **************************************************************************************************************************************************************************************************************************************************************************************************
META: ran handlers

TASK [check sudo password - if this fails, you need '-K' in your command-line!] *********************************************************************************************************************************************************************************************************************************
task path: /jaguar/ansible/999-test.yml:14
Using module_utils file /tmp/ansible/lib/ansible/module_utils/basic.py
Using module_utils file /tmp/ansible/lib/ansible/module_utils/six/__init__.py
Using module_utils file /tmp/ansible/lib/ansible/module_utils/_text.py
Using module_utils file /tmp/ansible/lib/ansible/module_utils/pycompat24.py
Using module_utils file /tmp/ansible/lib/ansible/module_utils/six/_six.py
Using module file /tmp/ansible/lib/ansible/modules/commands/command.py
<wildcat> ESTABLISH SSH CONNECTION FOR USER: None
<wildcat> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<wildcat> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<wildcat> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<wildcat> SSH: PlayContext set ssh_common_args: ()
<wildcat> SSH: PlayContext set ssh_extra_args: ()
<wildcat> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/tom/.ansible/cp/7364926df7)
<wildcat> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/tom/.ansible/cp/7364926df7 wildcat '/bin/sh -c '"'"'sudo -H -S  -p "[sudo via ansible, key=zudajyrygjffxolbejewpxigkagmhmqv] password: " -u web-admin /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-zudajyrygjffxolbejewpxigkagmhmqv; /usr/local/bin/python2.7'"'"'"'"'"'"'"'"' && sleep 0'"'"''
fatal: [wildcat]: FAILED! => {
    "failed": true, 
    "msg": "Timeout (12s) waiting for privilege escalation prompt: "
}
Debugger invoked
(debug) r
Using module file /tmp/ansible/lib/ansible/modules/commands/command.py
<wildcat> ESTABLISH SSH CONNECTION FOR USER: None
<wildcat> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<wildcat> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<wildcat> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<wildcat> SSH: PlayContext set ssh_common_args: ()
<wildcat> SSH: PlayContext set ssh_extra_args: ()
<wildcat> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/tom/.ansible/cp/7364926df7)
<wildcat> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/tom/.ansible/cp/7364926df7 wildcat '/bin/sh -c '"'"'sudo -H -S  -p "[sudo via ansible, key=wkxttwkisvqmrilqizwistsqeasmtllc] password: " -u web-admin /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-wkxttwkisvqmrilqizwistsqeasmtllc; /usr/local/bin/python2.7'"'"'"'"'"'"'"'"' && sleep 0'"'"''
<wildcat> (0, '\n{"changed": true, "end": "2017-04-24 11:20:34.944825", "stdout": "", "cmd": ["/usr/bin/true"], "rc": 0, "start": "2017-04-24 11:20:34.918436", "stderr": "", "delta": "0:00:00.026389", "invocation": {"module_args": {"warn": true, "executable": null, "_uses_shell": false, "_raw_params": "/usr/bin/true", "removes": null, "creates": null, "chdir": null}}, "warnings": []}\n', 'OpenSSH_7.2p2, OpenSSL 1.0.1s-freebsd  1 Mar 2016\r\ndebug1: Reading configuration data /home/tom/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 6171\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
ok: [wildcat] => {
    "changed": false, 
    "cmd": [
        "/usr/bin/true"
    ], 
    "delta": "0:00:00.026389", 
    "end": "2017-04-24 11:20:34.944825", 
    "invocation": {
        "module_args": {
            "_raw_params": "/usr/bin/true", 
            "_uses_shell": false, 
            "chdir": null, 
            "creates": null, 
            "executable": null, 
            "removes": null, 
            "warn": true
        }
    }, 
    "rc": 0, 
    "start": "2017-04-24 11:20:34.918436", 
    "stderr": "", 
    "stderr_lines": [], 
    "stdout": "", 
    "stdout_lines": []
}
META: ran handlers
META: ran handlers

PLAY RECAP ******************************************************************************************************************************************************************************************************************************************************************************************************
wildcat                    : ok=1    changed=0    unreachable=0    failed=0 

```



Discussion:
sounds similar to https://github.com/ansible/ansible/issues/13278 
(just the ssh stderr reformatted for easier reading)

```
OpenSSH_7.2p2, OpenSSL 1.0.1s-freebsd  1 Mar 2016
debug1: Reading configuration data /home/tom/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: auto-mux: Trying existing master
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 6171
debug3: mux_client_request_session: session request sent
debug1: mux_client_request_session: master session id: 2
debug3: mux_client_read_packet: read header failed: Broken pipe
debug2: Received exit status from master 0
```
(also sort of kind of similar to https://github.com/ansible/ansible/issues/16731 although that issue tends to happen when the ControlPath has been deleted instead of before it has been created as here...)
Some of the symptoms in the two referenced tickets line-up, but I'm not sure they're exact matches. For an extra data point, I reverted to Ansible 2.0.2.0 and re-tested successfully. Output follows.

```
tom@jaguar-a:/jaguar/ansible-> ansible --version
ansible 2.0.2.0
  config file = /jaguar/ansible/ansible.cfg
  configured module search path = Default w/o overrides
tom@jaguar-a:/jaguar/ansible-> ansible-playbook -vvvvvK 999-test.yml
Using /jaguar/ansible/ansible.cfg as config file
SUDO password: 
Loaded callback default of type stdout, v2.0

PLAYBOOK: 999-test.yml *********************************************************
1 plays in 999-test.yml

PLAY [wildcat] *****************************************************************

TASK [check sudo password - if this fails, you need '-K' in your command-line!] 
task path: /jaguar/ansible/999-test.yml:14
<wildcat> ESTABLISH SSH CONNECTION FOR USER: None
<wildcat> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<wildcat> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<wildcat> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<wildcat> SSH: PlayContext set ssh_common_args: ()
<wildcat> SSH: PlayContext set ssh_extra_args: ()
<wildcat> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/tom/.ansible/cp/ansible-ssh-%h-%p-%r)
<wildcat> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/tom/.ansible/cp/ansible-ssh-%h-%p-%r wildcat '/bin/sh -c '"'"'sudo -H -S  -p "[sudo via ansible, key=hexmtetfrtbvkrplkpnlvncekghhzazx] password: " -u web-admin /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-hexmtetfrtbvkrplkpnlvncekghhzazx; LANG=C LC_ALL=C LC_MESSAGES=C /usr/local/bin/python2.7'"'"'"'"'"'"'"'"''"'"''
ok: [wildcat] => {"changed": false, "cmd": ["/usr/bin/true"], "delta": "0:00:00.018760", "end": "2017-04-24 14:47:03.764275", "invocation": {"module_args": {"_raw_params": "/usr/bin/true", "_uses_shell": false, "chdir": null, "creates": null, "executable": null, "removes": null, "warn": true}, "module_name": "command"}, "rc": 0, "start": "2017-04-24 14:47:03.745515", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}

PLAY RECAP *********************************************************************
wildcat                    : ok=1    changed=0    unreachable=0    failed=0
```
I have tracked this back to https://github.com/ansible/ansible/commit/6f77498700a989c6b42f4f4d050d98df1de74f9e

At least as far is devel is concerned.

In my debugging thus far:

b_output:
```
'[sudo via ansible, key=hlrnksebsptlsehgjmzuirbxesmrzomh] password: debug2: channel 2: written 67 to efd 11\r\n'
```
self._play_context.prompt:
```
'[sudo via ansible, key=hlrnksebsptlsehgjmzuirbxesmrzomh] password: '
```

The code turns those into:

```
[sudo via ansible, key=muulxexlqxbjhrhgnpnoqpsvxdvgzjcy] password:
```

```
[sudo via ansible, key=muulxexlqxbjhrhgnpnoqpsvxdvgzjcy] password: debug2: channel 2: written 67 to efd 11\r\n
```

The logic then:

```
return b_lines[-1].strip().endswith(b_prompt) or b_lines[0].strip().endswith(b_prompt)
```

That fails, since my output ends with:

```
debug2: channel 2: written 67 to efd 11\r\n
```

I'm not sure why we switched to `endswith` instead of `in` or `startswith`
The issue seems to extend a little past just the above, once the above is fixed I continue to get:

```
Incorrect sudo password
```

Although the password is correct, however I think something in `_examine_output` may be to blame, as `check_become_success` never sees the `BECOME-SUCCESS`.
I referenced a different commit above, but it started with fee6e2953b02229a2be658cca144570c8e54bf9f

fee6e2953b02229a2be658cca144570c8e54bf9f caused what would have been a correct match, into a failure.

So fee6e2953b02229a2be658cca144570c8e54bf9f causes the match to fail, resulting in:

```
Incorrect sudo password
```

https://github.com/ansible/ansible/commit/6f77498700a989c6b42f4f4d050d98df1de74f9e causes the match to fail, which ultimately leads to:

```
Timeout (12s) waiting for privilege escalation prompt:
```
Effectively reverting back to pre https://github.com/ansible/ansible/commit/fee6e2953b02229a2be658cca144570c8e54bf9f behavior resolves my issues, but is likely to break the problems in https://github.com/ansible/ansible/issues/23054 and https://github.com/ansible/ansible/issues/20858

https://github.com/ansible/ansible/blob/83fe2170fffa94bfd75c38de22217794335827b9/lib/ansible/plugins/connection/__init__.py#L252-L259

cc @abadger you are likely to have more insight here than I do.

Resolving patch:
diff --git a/lib/ansible/plugins/connection/__init__.py b/lib/ansible/plugins/connection/__init__.py
--- a/lib/ansible/plugins/connection/__init__.py
+++ b/lib/ansible/plugins/connection/__init__.py
@@ -254,10 +254,8 @@ def check_password_prompt(self, b_output):
             return False
         elif isinstance(self._play_context.prompt, string_types):
             b_prompt = to_bytes(self._play_context.prompt).strip()
-            b_lines = b_output.splitlines(True)
-            if not b_lines:
-                return False
-            return b_lines[-1].strip().endswith(b_prompt) or b_lines[0].strip().endswith(b_prompt)
+            b_lines = b_output.splitlines()
+            return any(l.strip().startswith(b_prompt) for l in b_lines)
         else:
             return self._play_context.prompt(b_output)
 


Test patch:
diff --git a/test/units/plugins/connection/test_connection.py b/test/units/plugins/connection/test_connection.py
--- a/test/units/plugins/connection/test_connection.py
+++ b/test/units/plugins/connection/test_connection.py
@@ -45,6 +45,9 @@ class TestConnectionBaseClass(unittest.TestCase):
 
     def setUp(self):
         self.play_context = PlayContext()
+        self.play_context.prompt = (
+            '[sudo via ansible, key=ouzmdnewuhucvuaabtjmweasarviygqq] password: '
+        )
         self.in_stream = StringIO()
 
     def tearDown(self):
@@ -132,3 +135,81 @@ def test_docker_connection_module_wrong_cmd(self, mock_new_docker_version, mock_
     def test_network_cli_connection_module(self):
         self.assertIsInstance(NetworkCliConnection(self.play_context, self.in_stream), NetworkCliConnection)
         self.assertIsInstance(NetworkCliConnection(self.play_context, self.in_stream), ParamikoConnection)
+
+    def test_check_password_prompt(self):
+        local = (
+            b'[sudo via ansible, key=ouzmdnewuhucvuaabtjmweasarviygqq] password: \n'
+            b'BECOME-SUCCESS-ouzmdnewuhucvuaabtjmweasarviygqq\n'
+        )
+
+        ssh_pipelining_vvvv = b'''
+debug3: mux_master_read_cb: channel 1 packet type 0x10000002 len 251
+debug2: process_mux_new_session: channel 1: request tty 0, X 1, agent 1, subsys 0, term "xterm-256color", cmd "/bin/sh -c 'sudo -H -S  -p "[sudo via ansible, key=ouzmdnewuhucvuaabtjmweasarviygqq] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-ouzmdnewuhucvuaabtjmweasarviygqq; /bin/true'"'"' && sleep 0'", env 0
+debug3: process_mux_new_session: got fds stdin 9, stdout 10, stderr 11
+debug2: client_session2_setup: id 2
+debug1: Sending command: /bin/sh -c 'sudo -H -S  -p "[sudo via ansible, key=ouzmdnewuhucvuaabtjmweasarviygqq] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-ouzmdnewuhucvuaabtjmweasarviygqq; /bin/true'"'"' && sleep 0'
+debug2: channel 2: request exec confirm 1
+debug2: channel 2: rcvd ext data 67
+[sudo via ansible, key=ouzmdnewuhucvuaabtjmweasarviygqq] password: debug2: channel 2: written 67 to efd 11
+BECOME-SUCCESS-ouzmdnewuhucvuaabtjmweasarviygqq
+debug3: receive packet: type 98
+'''  # noqa
+
+        ssh_nopipelining_vvvv = b'''
+debug3: mux_master_read_cb: channel 1 packet type 0x10000002 len 251
+debug2: process_mux_new_session: channel 1: request tty 1, X 1, agent 1, subsys 0, term "xterm-256color", cmd "/bin/sh -c 'sudo -H -S  -p "[sudo via ansible, key=ouzmdnewuhucvuaabtjmweasarviygqq] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-ouzmdnewuhucvuaabtjmweasarviygqq; /bin/true'"'"' && sleep 0'", env 0
+debug3: mux_client_request_session: session request sent
+debug3: send packet: type 98
+debug1: Sending command: /bin/sh -c 'sudo -H -S  -p "[sudo via ansible, key=ouzmdnewuhucvuaabtjmweasarviygqq] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-ouzmdnewuhucvuaabtjmweasarviygqq; /bin/true'"'"' && sleep 0'
+debug2: channel 2: request exec confirm 1
+debug2: exec request accepted on channel 2
+[sudo via ansible, key=ouzmdnewuhucvuaabtjmweasarviygqq] password: debug3: receive packet: type 2
+debug3: Received SSH2_MSG_IGNORE
+debug3: Received SSH2_MSG_IGNORE
+
+BECOME-SUCCESS-ouzmdnewuhucvuaabtjmweasarviygqq
+debug3: receive packet: type 98
+'''  # noqa
+
+        ssh_novvvv = (
+            b'[sudo via ansible, key=ouzmdnewuhucvuaabtjmweasarviygqq] password: \n'
+            b'BECOME-SUCCESS-ouzmdnewuhucvuaabtjmweasarviygqq\n'
+        )
+
+        dns_issue = (
+            b'timeout waiting for privilege escalation password prompt:\n'
+            b'sudo: sudo: unable to resolve host tcloud014\n'
+            b'[sudo via ansible, key=ouzmdnewuhucvuaabtjmweasarviygqq] password: \n'
+            b'BECOME-SUCCESS-ouzmdnewuhucvuaabtjmweasarviygqq\n'
+        )
+
+        nothing = b''
+
+        in_front = b'''
+debug1: Sending command: /bin/sh -c 'sudo -H -S  -p "[sudo via ansible, key=ouzmdnewuhucvuaabtjmweasarviygqq] password: " -u root /bin/sh -c '"'"'echo
+'''
+
+        class ConnectionFoo(ConnectionBase):
+            @property
+            def transport(self):
+                pass
+            def _connect(self):
+                pass
+            def exec_command(self):
+                pass
+            def put_file(self):
+                pass
+            def fetch_file(self):
+                pass
+            def close(self):
+                pass
+
+        c = ConnectionFoo(self.play_context, self.in_stream)
+
+        self.assertTrue(c.check_password_prompt(local))
+        self.assertTrue(c.check_password_prompt(ssh_pipelining_vvvv))
+        self.assertTrue(c.check_password_prompt(ssh_nopipelining_vvvv))
+        self.assertTrue(c.check_password_prompt(ssh_novvvv))
+        self.assertTrue(c.check_password_prompt(dns_issue))
+        self.assertFalse(c.check_password_prompt(nothing))
+        self.assertFalse(c.check_password_prompt(in_front))