Repository: ansible/ansible

Issue:
Recent update 2.2.2.0 - shell module fails to become root when delegated to another host
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
[shell module](http://docs.ansible.com/ansible/shell_module.html)

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.2.2.0 from ubuntu ppa
```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
We have `vault_password_file` defined to include a vault with passwords we need in several scripts. Become method is `sudo`.

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
-->
Running instances on Ubuntu 14.04 LTS and Ubuntu 16.04 LTS, same results.

##### SUMMARY
<!--- Explain the problem briefly -->
After upgrading from 2.1.1.0 to the latest 2.2.2.0, some scripts began failing to become root with the following message:

```
TASK [glassfish-deploy-cluster : TESTANDO] *************************************
fatal: [infra00-lab]: FAILED! => {"failed": true, "msg": "Incorrect sudo password"}
```

We managed to isolate the problem to the shell module. All other modules we use seem to be fine with `become: <anyuser>`. A workround we found is to use the [raw module](http://docs.ansible.com/ansible/raw_module.html) to run whatever native shell commands we need on the remote servers.

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
Execute any script using the shell module while delegating the task to another host, other than the remote host ansible is connected to.

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- shell: touch /tmp/asdf.txt
  become: yes
  delegate_to: another_host
```

Oddly enough, the error only appears when the playbook is run for the first time. Ansible will create ssh pipelines to keep connections open for subsequent tasks, and the following playbook executions will not fail. To reproduce the issue, you need to kill any previous SSH connections still open to the remote hosts for a fresh run.

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
Expected privilege escalation to work properly.

##### ACTUAL RESULTS
Script fails with the following error: `Incorrect sudo password`.

<!--- Paste verbatim command output between quotes below -->
```

```



Discussion:
I'm currently unable to reproduce this behaviour.  Here is what I'm testing with:
```
$ cat ansible.cfg                                               (08:10:19)
[defaults]
nocows=1
[ssh_connection]
# get rid of controlpersist for testing
ssh_args = -C
pipelining = False

$ cat 23054.yml                                                  (07:57:51)
---
- hosts: ubuntu14-test
  gather_facts: False
  become_user: nobody
  tasks:
  - raw: whoami
    become: True
    delegate_to: ubuntu16-test
  - shell: touch /tmp/asdf.txt
    become: True
    delegate_to: ubuntu16-test

$ ansible-playbook --version                                    (08:12:34)
ansible-playbook 2.2.2.0 (detached HEAD 31300bd36b) last updated 2017/03/29 08:00:40 (GMT -700)
  lib/ansible/modules/core: (detached HEAD 5546cdb453) last updated 2017/03/29 08:11:27 (GMT -700)
  lib/ansible/modules/extras: (stable-2.2 921bc0d464) last updated 2017/03/28 07:52:17 (GMT -700)
  config file = /srv/ansible/troubleshooting/ansible.cfg
  configured module search path = Default w/o overrides

$ ps ax|grep ssh|grep -i ubuntu|wc -l                           (08:13:29)
0

$ ansible-playbook 23054.yml -i 'ubuntu14-test,ubuntu16-test' -K -v
Using /srv/ansible/troubleshooting/ansible.cfg as config file
SUDO password: 

PLAY [ubuntu14-test] ***********************************************************

TASK [raw] *********************************************************************
changed: [ubuntu14-test -> ubuntu16-test] => {"changed": true, "rc": 0, "stderr": "Connection to ubuntu16-test closed.\r\n", "stdout": "\r\nnobody\r\n", "stdout_lines": ["", "nobody"]}

TASK [command] *****************************************************************
changed: [ubuntu14-test -> ubuntu16-test] => {"changed": true, "cmd": "touch /tmp/asdf.txt", "delta": "0:00:00.002253", "end": "2017-03-29 08:14:41.357779", "rc": 0, "start": "2017-03-29 08:14:41.355526", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": ["Consider using file module with state=touch rather than running touch"]}
 [WARNING]: Consider using file module with state=touch rather than running touch


PLAY RECAP *********************************************************************
ubuntu14-test              : ok=2    changed=2    unreachable=0    failed=0
```

Could you give us more information to repdoruce this or try running my playbook and ansible.cfg to see if it works?

needs_info
They're all pasted into the above comment.  If you'd rather download, here is the playbook and ansible.cfg in a gist: https://gist.github.com/abadger/dfde2c5f6d8e8546b7f30d702f61df93  ( which lets you download as a zipfile: https://gist.github.com/abadger/dfde2c5f6d8e8546b7f30d702f61df93/archive/bfc3d0e84a381a0bf07535b78eef039c62e8a655.zip )
Yes, sorry. I quickly realized how dumb that question was.

I am able to persistently reproduce this, and I found that `pipelining = True|False` makes no difference. The only option that affects the scenario is ssh's connection sharing, `ControlPersist` and `ControlMaster`. In your example, with `ssh_args = -C` it actually works.

Here's what I'm running:

```
$ cat testplaybook.yml
- hosts: "infra00-lab"
  tasks:
    - raw: whoami
      become: True
      delegate_to: infra01-lab
    - shell: touch /tmp/asdf.txt
      become: True
      delegate_to: infra01-lab
```

```
$ ansible-playbook --version
ansible-playbook 2.2.2.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
```

```
$ cat ansible.cfg
[ssh_connection]
ssh_args = -C -o ControlMaster=auto -o ControlPersist=1800s -v
```


With these options, here's the output

```
$ ansible-playbook -K -v /dados/ansible/teste.yml 
Using /etc/ansible/ansible.cfg as config file
SUDO password: 

PLAY [infra00-lab] *************************************************************

TASK [setup] *******************************************************************
ok: [infra00-lab]

TASK [raw] *********************************************************************
fatal: [infra00-lab]: FAILED! => {"failed": true, "msg": "Incorrect sudo password"}

PLAY RECAP *********************************************************************
infra00-lab                : ok=1    changed=0    unreachable=0    failed=1   
```

NOTE: This will only appear in the first run. Subsequent playbook executions will reuse the established ssh connections without errors. To reproduce, run `pkill ssh` do kill all persistent ssh connections.

Now change `ansible.cfg` do disable ControlPersist, and we have your scenario that always works.

```
$ cat ansible.cfg
[ssh_connection]
ssh_args = -C
```

```
$ ansible-playbook -K -v /dados/ansible/teste.yml 
Using /etc/ansible/ansible.cfg as config file
SUDO password: 

PLAY [infra00-lab] *************************************************************

TASK [setup] *******************************************************************
ok: [infra00-lab]

TASK [raw] *********************************************************************
changed: [infra00-lab -> infra01-lab] => {"changed": true, "rc": 0, "stderr": "Connection to infra01-lab closed.\r\n", "stdout": "\r\nroot\r\n", "stdout_lines": ["", "root"]}

TASK [command] *****************************************************************
changed: [infra00-lab -> infra01-lab] => {"changed": true, "cmd": "touch /tmp/asdf.txt", "delta": "0:00:00.002111", "end": "2017-03-29 12:48:39.095635", "rc": 0, "start": "2017-03-29 12:48:39.093524", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": ["Consider using file module with state=touch rather than running touch"]}
 [WARNING]: Consider using file module with state=touch rather than running touch


PLAY RECAP *********************************************************************
infra00-lab                : ok=3    changed=2    unreachable=0    failed=0   
```

Reproduced now.  Thanks!

git bisected to 3551de98c6285a172fff962eb7bcea5c207e6415 as the bad commit
I just realized that even `raw` is not working. I downgraded to the previous version we were using to get our tools up and running again. We can wait for a solution.

Thank you!

Resolving patch:
diff --git a/lib/ansible/plugins/connection/__init__.py b/lib/ansible/plugins/connection/__init__.py
--- a/lib/ansible/plugins/connection/__init__.py
+++ b/lib/ansible/plugins/connection/__init__.py
@@ -253,10 +253,11 @@ def check_password_prompt(self, b_output):
         if self._play_context.prompt is None:
             return False
         elif isinstance(self._play_context.prompt, string_types):
-            b_prompt = to_bytes(self._play_context.prompt)
-            return b_prompt in b_output
-        else:
-            return self._play_context.prompt(b_output)
+            b_prompt = to_bytes(self._play_context.prompt).strip()
+            b_lines = b_output.splitlines(True)
+            if not b_lines:
+                return False
+            return b_lines[-1].strip().endswith(b_prompt) or b_lines[0].strip().endswith(b_prompt)
 
     def check_incorrect_password(self, b_output):
         b_incorrect_password = to_bytes(gettext.dgettext(self._play_context.become_method, C.BECOME_ERROR_STRINGS[self._play_context.become_method]))


Test patch: