Repository: ansible/ansible

Issue:
Keep trying other methods when transfer_method = smart
##### SUMMARY
Commit 1fe67f9 introduced a change to keep retrying file
transfers where previously retry only applied to commands.

This change broke transfer_method = smart in a way that
when sftp transfer is tried, it raises an exception
interrupting the flow to continue trying other methods.

This change catches AnsibleConnectionFailure in
_file_transport_command to continue trying other
methods.

Fixes #23711

##### ISSUE TYPE
 - Bugfix Pull Request

##### COMPONENT NAME
plugins/connection/ssh.py

##### ANSIBLE VERSION
```
$ ansible --version
ansible 2.4.0 (devel 698fa37a44) last updated 2017/04/18 11:06:11 (GMT -500)
  config file = /Users/albertom/Desktop/test/ansible.cfg
  configured module search path = [u'/Users/albertom/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/albertom/github/ansible/lib/ansible
  executable location = /Users/albertom/github/ansible/bin/ansible
  python version = 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 12:39:47) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
```

##### ADDITIONAL INFORMATION
before
```
# ansible-playbook -i hosts test.yml

PLAY [all] ***********************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************
fatal: [wolverine]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed\r\n", "unreachable": true}
	to retry, use: --limit @/Users/albertom/Desktop/test/test.retry

PLAY RECAP ***********************************************************************************************************************************************************************************************
wolverine                  : ok=0    changed=0    unreachable=1    failed=0

```

after
```
$ ansible-playbook -i hosts test.yml

PLAY [all] ***********************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************
 [WARNING]: sftp transfer mechanism failed on [wolverine]. Use ANSIBLE_DEBUG=1 to see detailed information

ok: [wolverine]

TASK [Test] **********************************************************************************************************************************************************************************************
ok: [wolverine] => {
    "msg": "This is a test"
}

PLAY RECAP ***********************************************************************************************************************************************************************************************
wolverine                  : ok=2    changed=0    unreachable=0    failed=0

```



Discussion:
Note to reviews: https://github.com/ansible/ansible/pull/23717/files?w=1
@albertomurillo to follow up on the question @bcoca had, does the following resolve it instead of the approach of the multiple `try/except` blocks?

```diff
diff --git a/lib/ansible/plugins/connection/ssh.py b/lib/ansible/plugins/connection/ssh.py
index 12bdaf5..5701159 100644
--- a/lib/ansible/plugins/connection/ssh.py
+++ b/lib/ansible/plugins/connection/ssh.py
@@ -169,7 +169,10 @@ def _ssh_retry(func):
                     raise AnsibleConnectionFailure("Failed to connect to the host via ssh: %s" % to_native(return_tuple[2]))
             except (AnsibleConnectionFailure, Exception) as e:
                 if attempt == remaining_tries - 1:
-                    raise
+                    if isinstance(e, AnsibleConnectionFailure):
+                        return return_tuple
+                    else
+                        raise
                 else:
                     pause = 2 ** attempt - 1
                     if pause > 30:
```
@sivel yes it works but there are some issue with it.

If i set ANSIBLE_SSH_RETRIES=4, it will try 4 times to run a command with sftp and until then, it will try scp, then the next task, will try 4 times with sftp and then try with scp and so on.
@albertomurillo Ideally, that is expected.  In fact, regardless of which approach you use, it should work the same.  In either solution, either the `return` or the `raise` would not happen until the retries have been completed.

If you could, please implement my new recommendation.

Thanks!
CI failure in unit tests: https://app.shippable.com/github/ansible/ansible/runs/20119/2/tests

```
>                       return return_tuple
E                       UnboundLocalError: local variable 'return_tuple' referenced before assignment

lib/ansible/plugins/connection/ssh.py:173: UnboundLocalError
```
@sivel, so I'm thinking about @albertomurillo's objection and I think his point is valid.  Let's say you're on vacation in some Mediterranean Paradise with really flakey internet access.  You get a call from the big boss that something is really broken, could you please interrupt your vacation to fix it?  You figure out a fix and run an ansible command from your laptop to hit 100 heterogeneous hosts with that fix.  Because of the flakey internet, you have retries set to 10.  You also have transport set to smart because some of these hosts have just scp enabled, others have just sftp, and a few have just ssh.

For the hosts that will only work with piped, do you really want to wait for 20 failures before the first chance at a successful connection?

Further I think that transport = smart could very well mean that you should try each of the transports in turn , if all fail, count that as one retry, and try each of the transports a second time.  The transport type is the smallest atom of code that the retries should be acting upon, not any sub-transports that the transport invokes.

Resolving patch:
diff --git a/lib/ansible/plugins/connection/ssh.py b/lib/ansible/plugins/connection/ssh.py
--- a/lib/ansible/plugins/connection/ssh.py
+++ b/lib/ansible/plugins/connection/ssh.py
@@ -491,8 +491,7 @@ def _examine_output(self, source, state, b_chunk, sudoable):
 
         return b''.join(output), remainder
 
-    @_ssh_retry
-    def _run(self, cmd, in_data, sudoable=True, checkrc=True):
+    def _bare_run(self, cmd, in_data, sudoable=True, checkrc=True):
         '''
         Starts the command and communicates with it until it ends.
         '''
@@ -767,6 +766,13 @@ def _run(self, cmd, in_data, sudoable=True, checkrc=True):
 
         return (p.returncode, b_stdout, b_stderr)
 
+    @_ssh_retry
+    def _run(self, cmd, in_data, sudoable=True, checkrc=True):
+        """Wrapper around _bare_run that retries the connection
+        """
+        return self._bare_run(cmd, in_data, sudoable, checkrc)
+
+    @_ssh_retry
     def _file_transport_command(self, in_path, out_path, sftp_action):
         # scp and sftp require square brackets for IPv6 addresses, but
         # accept them for hostnames and IPv4 addresses too.
@@ -807,14 +813,14 @@ def _file_transport_command(self, in_path, out_path, sftp_action):
                 cmd = self._build_command('sftp', to_bytes(host))
                 in_data = u"{0} {1} {2}\n".format(sftp_action, shlex_quote(in_path), shlex_quote(out_path))
                 in_data = to_bytes(in_data, nonstring='passthru')
-                (returncode, stdout, stderr) = self._run(cmd, in_data, checkrc=False)
+                (returncode, stdout, stderr) = self._bare_run(cmd, in_data, checkrc=False)
             elif method == 'scp':
                 if sftp_action == 'get':
                     cmd = self._build_command('scp', u'{0}:{1}'.format(host, shlex_quote(in_path)), out_path)
                 else:
                     cmd = self._build_command('scp', in_path, u'{0}:{1}'.format(host, shlex_quote(out_path)))
                 in_data = None
-                (returncode, stdout, stderr) = self._run(cmd, in_data, checkrc=False)
+                (returncode, stdout, stderr) = self._bare_run(cmd, in_data, checkrc=False)
             elif method == 'piped':
                 if sftp_action == 'get':
                     # we pass sudoable=False to disable pty allocation, which


Test patch:
diff --git a/test/units/plugins/connection/test_ssh.py b/test/units/plugins/connection/test_ssh.py
--- a/test/units/plugins/connection/test_ssh.py
+++ b/test/units/plugins/connection/test_ssh.py
@@ -198,11 +198,11 @@ def test_plugins_connection_ssh_put_file(self, mock_ospe, mock_sleep):
         new_stdin = StringIO()
         conn = ssh.Connection(pc, new_stdin)
         conn._build_command = MagicMock()
-        conn._run = MagicMock()
+        conn._bare_run = MagicMock()
 
         mock_ospe.return_value = True
         conn._build_command.return_value = 'some command to run'
-        conn._run.return_value = (0, '', '')
+        conn._bare_run.return_value = (0, '', '')
         conn.host = "some_host"
 
         C.ANSIBLE_SSH_RETRIES = 9
@@ -212,41 +212,41 @@ def test_plugins_connection_ssh_put_file(self, mock_ospe, mock_sleep):
         C.DEFAULT_SCP_IF_SSH = 'smart'
         expected_in_data = b' '.join((b'put', to_bytes(shlex_quote('/path/to/in/file')), to_bytes(shlex_quote('/path/to/dest/file')))) + b'\n'
         conn.put_file('/path/to/in/file', '/path/to/dest/file')
-        conn._run.assert_called_with('some command to run', expected_in_data, checkrc=False)
+        conn._bare_run.assert_called_with('some command to run', expected_in_data, checkrc=False)
 
         # Test when SFTP doesn't work but SCP does
-        conn._run.side_effect = [(1, 'stdout', 'some errors'), (0, '', '')]
+        conn._bare_run.side_effect = [(1, 'stdout', 'some errors'), (0, '', '')]
         conn.put_file('/path/to/in/file', '/path/to/dest/file')
-        conn._run.assert_called_with('some command to run', None, checkrc=False)
-        conn._run.side_effect = None
+        conn._bare_run.assert_called_with('some command to run', None, checkrc=False)
+        conn._bare_run.side_effect = None
 
         # test with C.DEFAULT_SCP_IF_SSH enabled
         C.DEFAULT_SCP_IF_SSH = True
         conn.put_file('/path/to/in/file', '/path/to/dest/file')
-        conn._run.assert_called_with('some command to run', None, checkrc=False)
+        conn._bare_run.assert_called_with('some command to run', None, checkrc=False)
 
         conn.put_file(u'/path/to/in/file/with/unicode-fö〩', u'/path/to/dest/file/with/unicode-fö〩')
-        conn._run.assert_called_with('some command to run', None, checkrc=False)
+        conn._bare_run.assert_called_with('some command to run', None, checkrc=False)
 
         # test with C.DEFAULT_SCP_IF_SSH disabled
         C.DEFAULT_SCP_IF_SSH = False
         expected_in_data = b' '.join((b'put', to_bytes(shlex_quote('/path/to/in/file')), to_bytes(shlex_quote('/path/to/dest/file')))) + b'\n'
         conn.put_file('/path/to/in/file', '/path/to/dest/file')
-        conn._run.assert_called_with('some command to run', expected_in_data, checkrc=False)
+        conn._bare_run.assert_called_with('some command to run', expected_in_data, checkrc=False)
 
         expected_in_data = b' '.join((b'put',
                                       to_bytes(shlex_quote('/path/to/in/file/with/unicode-fö〩')),
                                       to_bytes(shlex_quote('/path/to/dest/file/with/unicode-fö〩')))) + b'\n'
         conn.put_file(u'/path/to/in/file/with/unicode-fö〩', u'/path/to/dest/file/with/unicode-fö〩')
-        conn._run.assert_called_with('some command to run', expected_in_data, checkrc=False)
+        conn._bare_run.assert_called_with('some command to run', expected_in_data, checkrc=False)
 
         # test that a non-zero rc raises an error
-        conn._run.return_value = (1, 'stdout', 'some errors')
+        conn._bare_run.return_value = (1, 'stdout', 'some errors')
         self.assertRaises(AnsibleError, conn.put_file, '/path/to/bad/file', '/remote/path/to/file')
 
         # test that a not-found path raises an error
         mock_ospe.return_value = False
-        conn._run.return_value = (0, 'stdout', '')
+        conn._bare_run.return_value = (0, 'stdout', '')
         self.assertRaises(AnsibleFileNotFound, conn.put_file, '/path/to/bad/file', '/remote/path/to/file')
 
     @patch('time.sleep')
@@ -255,10 +255,10 @@ def test_plugins_connection_ssh_fetch_file(self, mock_sleep):
         new_stdin = StringIO()
         conn = ssh.Connection(pc, new_stdin)
         conn._build_command = MagicMock()
-        conn._run = MagicMock()
+        conn._bare_run = MagicMock()
 
         conn._build_command.return_value = 'some command to run'
-        conn._run.return_value = (0, '', '')
+        conn._bare_run.return_value = (0, '', '')
         conn.host = "some_host"
 
         C.ANSIBLE_SSH_RETRIES = 9
@@ -268,36 +268,36 @@ def test_plugins_connection_ssh_fetch_file(self, mock_sleep):
         C.DEFAULT_SCP_IF_SSH = 'smart'
         expected_in_data = b' '.join((b'get', to_bytes(shlex_quote('/path/to/in/file')), to_bytes(shlex_quote('/path/to/dest/file')))) + b'\n'
         conn.fetch_file('/path/to/in/file', '/path/to/dest/file')
-        conn._run.assert_called_with('some command to run', expected_in_data, checkrc=False)
+        conn._bare_run.assert_called_with('some command to run', expected_in_data, checkrc=False)
 
         # Test when SFTP doesn't work but SCP does
-        conn._run.side_effect = [(1, 'stdout', 'some errors'), (0, '', '')]
+        conn._bare_run.side_effect = [(1, 'stdout', 'some errors'), (0, '', '')]
         conn.fetch_file('/path/to/in/file', '/path/to/dest/file')
-        conn._run.assert_called_with('some command to run', None, checkrc=False)
-        conn._run.side_effect = None
+        conn._bare_run.assert_called_with('some command to run', None, checkrc=False)
+        conn._bare_run.side_effect = None
 
         # test with C.DEFAULT_SCP_IF_SSH enabled
         C.DEFAULT_SCP_IF_SSH = True
         conn.fetch_file('/path/to/in/file', '/path/to/dest/file')
-        conn._run.assert_called_with('some command to run', None, checkrc=False)
+        conn._bare_run.assert_called_with('some command to run', None, checkrc=False)
 
         conn.fetch_file(u'/path/to/in/file/with/unicode-fö〩', u'/path/to/dest/file/with/unicode-fö〩')
-        conn._run.assert_called_with('some command to run', None, checkrc=False)
+        conn._bare_run.assert_called_with('some command to run', None, checkrc=False)
 
         # test with C.DEFAULT_SCP_IF_SSH disabled
         C.DEFAULT_SCP_IF_SSH = False
         expected_in_data = b' '.join((b'get', to_bytes(shlex_quote('/path/to/in/file')), to_bytes(shlex_quote('/path/to/dest/file')))) + b'\n'
         conn.fetch_file('/path/to/in/file', '/path/to/dest/file')
-        conn._run.assert_called_with('some command to run', expected_in_data, checkrc=False)
+        conn._bare_run.assert_called_with('some command to run', expected_in_data, checkrc=False)
 
         expected_in_data = b' '.join((b'get',
                                       to_bytes(shlex_quote('/path/to/in/file/with/unicode-fö〩')),
                                       to_bytes(shlex_quote('/path/to/dest/file/with/unicode-fö〩')))) + b'\n'
         conn.fetch_file(u'/path/to/in/file/with/unicode-fö〩', u'/path/to/dest/file/with/unicode-fö〩')
-        conn._run.assert_called_with('some command to run', expected_in_data, checkrc=False)
+        conn._bare_run.assert_called_with('some command to run', expected_in_data, checkrc=False)
 
         # test that a non-zero rc raises an error
-        conn._run.return_value = (1, 'stdout', 'some errors')
+        conn._bare_run.return_value = (1, 'stdout', 'some errors')
         self.assertRaises(AnsibleError, conn.fetch_file, '/path/to/bad/file', '/remote/path/to/file')