Repository: ansible/ansible

Issue:
Regression? - cannot retry ios task in the case of connection error
##### SUMMARY
ios_command tasks cannot be retried in the case of connection errors since Ansbile 2.5

I have the following playbook to make sure that Cisco IOS is available and ready for configuration provisioning:
```
---
- name: check available
  hosts: "{{ target }}"
  gather_facts: no
  connection: local
  tasks:
    - name: check available
      ios_command:
        authorize: yes
        commands:
          - show version
      register: result
      until: result is succeeded
      retries: 2
      delay: 5
```

It works as expected with **Ansible 2.4.5.0** - retries are used

```
PLAY [check available] ********************************************************************************************************************************************************************************

TASK [check available] ********************************************************************************************************************************************************************************
FAILED - RETRYING: check available (2 retries left).
FAILED - RETRYING: check available (1 retries left).
fatal: [d805700e-9aa7-4f5e-acf1-9a418ecfa3a5]: FAILED! => {"attempts": 2, "changed": false, "failed": true, "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell"}

PLAY RECAP ********************************************************************************************************************************************************************************************
d805700e-9aa7-4f5e-acf1-9a418ecfa3a5 : ok=0    changed=0    unreachable=0    failed=1
```

But with **Ansible 2.5.5 and 2.6.0** the connection time-out error is raised and no retries are performed
```
PLAY [check available] ********************************************************************************************************************************************************************************

TASK [check available] ********************************************************************************************************************************************************************************
fatal: [d805700e-9aa7-4f5e-acf1-9a418ecfa3a5]: FAILED! => {"msg": "timed out"}

PLAY RECAP ********************************************************************************************************************************************************************************************
d805700e-9aa7-4f5e-acf1-9a418ecfa3a5 : ok=0    changed=0    unreachable=0    failed=1
```

Possibly is related to the persistent connections because in 2.5.5 I can see
```
The full traceback is:
Traceback (most recent call last):
  File "/workdir/Projects/nfv-cm-runner/.venv/bin/ansible-connection", line 87, in start
    self.connection._connect()
  File "/workdir/Projects/nfv-cm-runner/.venv/lib/python2.7/site-packages/ansible/plugins/connection/network_cli.py", line 292, in _connect
    ssh = self.paramiko_conn._connect()
  File "/workdir/Projects/nfv-cm-runner/.venv/lib/python2.7/site-packages/ansible/plugins/connection/paramiko_ssh.py", line 240, in _connect
    self.ssh = SSH_CONNECTION_CACHE[cache_key] = self._connect_uncached()
  File "/workdir/Projects/nfv-cm-runner/.venv/lib/python2.7/site-packages/ansible/plugins/connection/paramiko_ssh.py", line 348, in _connect_uncached
    raise AnsibleConnectionFailure(msg)
AnsibleConnectionFailure: timed out
```
in debug mode.
But with 2.4.5.0:
```
Retrying task, attempt 1 of 3
FAILED - RETRYING: check available (2 retries left).Result was: {
    "attempts": 1, 
    "changed": false, 
    "failed": true, 
    "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell", 
    "retries": 3
}
```

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
ios_command

##### ANSIBLE VERSION
```
ansible 2.4.5.0
  config file = /workdir/csr/ansible.cfg
  configured module search path = [u'/home/khaberev/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /workdir/csr/ansible_2_4/lib/python2.7/site-packages/ansible
  executable location = /workdir/csr/ansible_2_4/bin/ansible
  python version = 2.7.15 (default, May  1 2018, 20:16:04) [GCC 7.3.1 20180406]
```
```
ansible 2.5.5
  config file = /workdir/csr/ansible.cfg
  configured module search path = [u'/home/khaberev/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /workdir/csr/ansible_2_5/lib/python2.7/site-packages/ansible
  executable location = /workdir/csr/ansible_2_5/bin/ansible
  python version = 2.7.15 (default, May  1 2018, 20:16:04) [GCC 7.3.1 20180406]
```

##### CONFIGURATION
```
[defaults]
internal_poll_interval = 0.5
host_key_checking = False
retry_files_enabled = False

[ssh_connection]
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o ServerAliveInterval=10 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
retries = 3
control_path = %(directory)s/%%C

[persistent_connection]
command_timeout = 30
connect_timeout = 60
```

##### OS / ENVIRONMENT
Archlinux

##### STEPS TO REPRODUCE
1. Use the above playbook with some unreachable host

##### EXPECTED RESULTS
1. Retries are used

##### ACTUAL RESULTS
1. Retries are not used


Discussion:
Files identified in the description:
* [lib/ansible/modules/network/ios/ios_command.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/network/ios/ios_command.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @privateip @rcarrillocruz
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->

Resolving patch:
diff --git a/lib/ansible/module_utils/network/ios/ios.py b/lib/ansible/module_utils/network/ios/ios.py
--- a/lib/ansible/module_utils/network/ios/ios.py
+++ b/lib/ansible/module_utils/network/ios/ios.py
@@ -122,7 +122,11 @@ def to_commands(module, commands):
 
 def run_commands(module, commands, check_rc=True):
     connection = get_connection(module)
-    return connection.run_commands(commands=commands, check_rc=check_rc)
+    try:
+        out = connection.run_commands(commands=commands, check_rc=check_rc)
+        return out
+    except ConnectionError as exc:
+        module.fail_json(msg=to_text(exc))
 
 
 def load_config(module, commands):
diff --git a/lib/ansible/plugins/action/ios.py b/lib/ansible/plugins/action/ios.py
--- a/lib/ansible/plugins/action/ios.py
+++ b/lib/ansible/plugins/action/ios.py
@@ -24,7 +24,7 @@
 
 from ansible import constants as C
 from ansible.module_utils._text import to_text
-from ansible.module_utils.connection import Connection
+from ansible.module_utils.connection import Connection, ConnectionError
 from ansible.plugins.action.normal import ActionModule as _ActionModule
 from ansible.module_utils.network.common.utils import load_provider
 from ansible.module_utils.network.ios.ios import ios_provider_spec
@@ -86,11 +86,14 @@ def run(self, tmp=None, task_vars=None):
             socket_path = self._connection.socket_path
 
         conn = Connection(socket_path)
-        out = conn.get_prompt()
-        while to_text(out, errors='surrogate_then_replace').strip().endswith(')#'):
-            display.vvvv('wrong context, sending exit to device', self._play_context.remote_addr)
-            conn.send_command('exit')
+        try:
             out = conn.get_prompt()
+            while to_text(out, errors='surrogate_then_replace').strip().endswith(')#'):
+                display.vvvv('wrong context, sending exit to device', self._play_context.remote_addr)
+                conn.send_command('exit')
+                out = conn.get_prompt()
+        except ConnectionError as exc:
+            return {'failed': True, 'msg': to_text(exc)}
 
         result = super(ActionModule, self).run(task_vars=task_vars)
         return result


Test patch: