Repository: ansible/ansible

Issue:
sshpass broken pipe after ssh retry due to ControlPersist broken pipe
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
plugins/connection/ssh

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.3.2.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.5 (default, Aug  4 2017, 00:39:18) [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]
```
But I believe the bug also exists in v2.4

##### CONFIGURATION
[ssh_connection]
control_path = %(directory)s/%%h-%%p-%%r
<!---
If using Ansible 2.4 or above, paste the results of "ansible-config dump --only-changed"

Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).

-->

##### OS / ENVIRONMENT
CentOS 7 control & target machines
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->

##### SUMMARY
<!--- Explain the problem briefly -->
This is another incarnation of #29095 but in case of ssh retry due to ControlPersist broken pipe (#16731 fix).

##### STEPS TO REPRODUCE
I'm sorry, but it is hard for me to specify reproduction steps.
The issue has occurred in various different tasks of a long playbook.
It was executed on a fairly slow VM control machine and was provisioning over 30 remote machines.
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below 
```yaml

```
-->
<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
sshpass pipe should be recreated in case of retry due to ControlPersist broken pipe

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
(...)
<192.168.0.22> ESTABLISH SSH CONNECTION FOR USER: XXX
(...)
<192.168.0.22> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<192.168.0.22> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<192.168.0.22> SSH: ANSIBLE_REMOTE_PORT/remote_port/ansible_port set: (-o)(Port=22)
<192.168.0.22> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=XXX)
<192.168.0.22> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<192.168.0.22> SSH: PlayContext set ssh_common_args: (-o)(StrictHostKeyChecking=no)
<192.168.0.22> SSH: PlayContext set ssh_extra_args: (-o)(UserKnownHostsFile=/dev/null)
<192.168.0.22> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/XXX/.ansible/cp/%h-%p-%r)
<192.168.0.22> SSH: EXEC sshpass -d14 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o User=XXX -o ConnectTimeout=10 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlPath=/home/XXX/.ansible/cp/%h-%p-%r 192.168.0.22 '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
(...)
RETRYING BECAUSE OF CONTROLPERSIST BROKEN PIPE
<192.168.0.22> SSH: EXEC sshpass -d14 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o User=XXX -o ConnectTimeout=10 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlPath=/home/XXX/.ansible/cp/%h-%p-%r 192.168.0.22 '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
(...)
close failed in file object destructor:
IOError: [Errno 9] Bad file descriptor
(...)
The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 125, in run
    res = self._execute()
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 522, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/package.py", line 69, in run
    result.update(self._execute_module(module_name=module, module_args=new_module_args, task_vars=task_vars, wrap_async=self._task.async))
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 637, in _execute_module
    tmp = self._make_tmp_path()
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 248, in _make_tmp_path
    tmpdir =  self._remote_expand_user(C.DEFAULT_REMOTE_TMP, sudoable=False)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 545, in _remote_expand_user
    data = self._low_level_execute_command(cmd, sudoable=False)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 883, in _low_level_execute_command
    rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 788, in exec_command
    (returncode, stdout, stderr) = self._run(cmd, in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 85, in wrapped
    return_tuple = func(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 461, in _run
    os.write(self.sshpass_pipe[1], to_bytes(self._play_context.password) + b'
')
OSError: [Errno 9] Bad file descriptor
fatal: [192.168.0.22]: FAILED! => {
    "failed": true, 
    "msg": "Unexpected failure during module execution.", 
    "stdout": ""
}
```

Create a new pipe for sshpass on retries. Fixes #29095
##### SUMMARY

Create a new pipe for sshpass on retries. Fixes #29095

##### ISSUE TYPE
 - Bugfix Pull Request

##### COMPONENT NAME
plugins/connection/ssh

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
2.3
2.4
2.5
```


##### ADDITIONAL INFORMATION

This should likely be cherry picked for 2.4, and maybe 2.3 depending on what our plans are with 2.3


Discussion:
Following patch seems to have fixed this for me:
```diff
diff --git lib/ansible/plugins/connection/ssh.py lib/ansible/plugins/connection/ssh.py
index 01b1106..c629c5b 100644
--- lib/ansible/plugins/connection/ssh.py
+++ lib/ansible/plugins/connection/ssh.py
@@ -80,6 +80,11 @@ def _ssh_retry(func):
                     # 1-254 = remote command return code
                     # 255 = failure from the ssh command itself
                 except (AnsibleControlPersistBrokenPipeError) as e:
+                    cmd = args[0]
+                    if self._play_context.password and isinstance(cmd, list):
+                        # If this is a retry, the fd/pipe for sshpass is closed, and we need a new one
+                        self.sshpass_pipe = os.pipe()
+                        cmd[1] = b'-d' + to_bytes(self.sshpass_pipe[0], nonstring='simplerepr', errors='surrogate_or_strict')
                     # Retry one more time because of the ControlPersist broken pipe (see #16731)
                     display.vvv(u"RETRYING BECAUSE OF CONTROLPERSIST BROKEN PIPE")
                     return_tuple = func(self, *args, **kwargs)
```
This was already resolved in https://github.com/ansible/ansible/pull/29099

It has been cherry picked for the following releases:

> Merged to devel and cherry-picked to stable-2.4 and stable-2.3. Will be in 2.4.0rc2 and if we have a 2.3.3 release, there as well
I meant this is similar to #29095 but not a duplicate. I think #29099 fixes only ANSIBLE_SSH_RETRIES != 0 case. In my case ANSIBLE_SSH_RETRIES was unset. Since #16787, ssh is also retried in case of ControlPersist broken pipe.
Ah, my mistake, I didn't understand that initially.  Re-opening.
@sivel 

I am a co-worker of @maciejkowalczyk9ld and also suffered from this issue. Could you reopen this issue? This issue is critical for our usecase to provision over 30 nodes of a distributed computing cluster.
@sivel Thanks!
I really hope this is fixed in next release! I had high hopes for 2.4 due to comments in #28720 ... but it was disappointment to  still face the broken pipes with that version.
@sivel Any chance to get this fix in for next 2.4 release?
I'm also having the issue with a ansible2.4 on CentOS Linux release 7.4.1708 (Core).
I'm hitting this (somewhat unreliably) with ansible 2.5.1 on Ubuntu 14.04

The earlier fix (#29099) resets the fd/pipe ready for a retry using SSH.
This failure occurs because the fd/pipe is not reset before the extra "final" retry after a ControlPersist broken pipe [here](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/connection/ssh.py#L277).
Merged to devel and cherry-picked to stable-2.4 and stable-2.3.  Will be in 2.4.0rc2 and if we have a 2.3.3 release, there as well.

Resolving patch:
diff --git a/lib/ansible/plugins/connection/ssh.py b/lib/ansible/plugins/connection/ssh.py
--- a/lib/ansible/plugins/connection/ssh.py
+++ b/lib/ansible/plugins/connection/ssh.py
@@ -276,6 +276,11 @@ def wrapped(self, *args, **kwargs):
                     # 255 = failure from the ssh command itself
                 except (AnsibleControlPersistBrokenPipeError) as e:
                     # Retry one more time because of the ControlPersist broken pipe (see #16731)
+                    cmd = args[0]
+                    if self._play_context.password and isinstance(cmd, list):
+                        # This is a retry, so the fd/pipe for sshpass is closed, and we need a new one
+                        self.sshpass_pipe = os.pipe()
+                        cmd[1] = b'-d' + to_bytes(self.sshpass_pipe[0], nonstring='simplerepr', errors='surrogate_or_strict')
                     display.vvv(u"RETRYING BECAUSE OF CONTROLPERSIST BROKEN PIPE")
                     return_tuple = func(self, *args, **kwargs)
 


Test patch: