Repository: ansible/ansible

Issue:
SSH Retries broken with sshpass
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
plugins/connection/ssh

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
2.3
```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
N/A

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->
N/A

##### SUMMARY

When connecting to a host with user/pass and using the ssh connection plugin, SSH retries will fail due to bad sshpass descriptor

Caused by https://github.com/ansible/ansible/commit/1fe67f9f436595003f7951dd88159731e6d82498

The problem is that the sshpass file descriptor/pipe is created in `_build_command` and closed on the unsuccessful connection, and future attempts try to use the same file descriptor.

##### STEPS TO REPRODUCE

Try to SSH to a host that does not have ssh running or by connecting to a port ssh is not listening on, with using user/pass auth and the ssh connection plugin.


##### EXPECTED RESULTS

Either get an unreachable error or retry successfully.


##### ACTUAL RESULTS

```
The full traceback is:
Traceback (most recent call last):
  File "/Users/matt/projects/ansibledev/ansible/lib/ansible/executor/task_executor.py", line 125, in run
    res = self._execute()
  File "/Users/matt/projects/ansibledev/ansible/lib/ansible/executor/task_executor.py", line 526, in _execute
    result = self._handler.run(task_vars=variables)
  File "/Users/matt/projects/ansibledev/ansible/lib/ansible/plugins/action/normal.py", line 45, in run
    results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars, wrap_async=wrap_async))
  File "/Users/matt/projects/ansibledev/ansible/lib/ansible/plugins/action/__init__.py", line 639, in _execute_module
    tmp = self._make_tmp_path()
  File "/Users/matt/projects/ansibledev/ansible/lib/ansible/plugins/action/__init__.py", line 245, in _make_tmp_path
    tmpdir = self._remote_expand_user(self._play_context.remote_tmp_dir, sudoable=False)
  File "/Users/matt/projects/ansibledev/ansible/lib/ansible/plugins/action/__init__.py", line 549, in _remote_expand_user
    data = self._low_level_execute_command(cmd, sudoable=False)
  File "/Users/matt/projects/ansibledev/ansible/lib/ansible/plugins/action/__init__.py", line 886, in _low_level_execute_command
    rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
  File "/Users/matt/projects/ansibledev/ansible/lib/ansible/plugins/connection/ssh.py", line 920, in exec_command
    (returncode, stdout, stderr) = self._run(cmd, in_data, sudoable=sudoable)
  File "/Users/matt/projects/ansibledev/ansible/lib/ansible/plugins/connection/ssh.py", line 204, in wrapped
    return_tuple = func(self, *args, **kwargs)
  File "/Users/matt/projects/ansibledev/ansible/lib/ansible/plugins/connection/ssh.py", line 817, in _run
    return self._bare_run(cmd, in_data, sudoable, checkrc)
  File "/Users/matt/projects/ansibledev/ansible/lib/ansible/plugins/connection/ssh.py", line 584, in _bare_run
    os.write(self.sshpass_pipe[1], to_bytes(self._play_context.password) + b'\n')
OSError: [Errno 9] Bad file descriptor
```


Discussion:
I'll need to do some more in depth research. This may not be the full problem.
Proof of concept fix:

```
diff --git a/lib/ansible/plugins/connection/ssh.py b/lib/ansible/plugins/connection/ssh.py
index bbdef53..2fc6ce9 100644
--- a/lib/ansible/plugins/connection/ssh.py
+++ b/lib/ansible/plugins/connection/ssh.py
@@ -199,6 +199,12 @@ def _ssh_retry(func):
         remaining_tries = int(C.ANSIBLE_SSH_RETRIES) + 1
         cmd_summary = "%s..." % args[0]
         for attempt in range(remaining_tries):
+            cmd = args[0]
+            if attempt != 0 and self._play_context.password and isinstance(cmd, list):
+                # If this is a retry, the fd/pipe for sshpass is closed, and we need a new one
+                self.sshpass_pipe = os.pipe()
+                cmd[1] = b'-d' + to_bytes(self.sshpass_pipe[0], nonstring='simplerepr', errors='surrogate_or_strict')
+
             try:
                 try:
                     return_tuple = func(self, *args, **kwargs)
```
cc @abadger would you ming taking a look at this, and my proposed fix?
Sivel, proposed fix looks good to me.

On Sep 7, 2017 9:40 AM, "Matt Martz" <notifications@github.com> wrote:

> cc @abadger <https://github.com/abadger> would you ming taking a look at
> this, and my proposed fix?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/ansible/ansible/issues/29095#issuecomment-327855970>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAMxWmEFa5FXaYo0he2AzaQypUw1fLwAks5sgByLgaJpZM4PQDpj>
> .
>


Resolving patch:
diff --git a/lib/ansible/plugins/connection/ssh.py b/lib/ansible/plugins/connection/ssh.py
--- a/lib/ansible/plugins/connection/ssh.py
+++ b/lib/ansible/plugins/connection/ssh.py
@@ -199,6 +199,12 @@ def wrapped(self, *args, **kwargs):
         remaining_tries = int(C.ANSIBLE_SSH_RETRIES) + 1
         cmd_summary = "%s..." % args[0]
         for attempt in range(remaining_tries):
+            cmd = args[0]
+            if attempt != 0 and self._play_context.password and isinstance(cmd, list):
+                # If this is a retry, the fd/pipe for sshpass is closed, and we need a new one
+                self.sshpass_pipe = os.pipe()
+                cmd[1] = b'-d' + to_bytes(self.sshpass_pipe[0], nonstring='simplerepr', errors='surrogate_or_strict')
+
             try:
                 try:
                     return_tuple = func(self, *args, **kwargs)


Test patch: