Repository: ansible/ansible

Issue:
AnsibleConnectionFailure("timeout trying to send command: %s" % command.strip())
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest -->
 - Bug Report

##### COMPONENT NAME
<!---
Name of the module, plugin, task or feature
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path
-->
File "/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/connection/network_cli.py", line 302, in send
##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.5.0
  config file = /Users/bklotz/.ansible.cfg
  configured module search path = [u'/Users/bklotz/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible
  executable location = /Users/bklotz/miniconda2/bin/ansible
  python version = 2.7.13 |Continuum Analytics, Inc.| (default, Dec 20 2016, 23:05:08) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
```

##### CONFIGURATION
<!---
If using Ansible 2.4 or above, paste the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
DEFAULT_DEBUG(env: ANSIBLE_DEBUG) = False
DEFAULT_LOG_PATH(env: ANSIBLE_LOG_PATH) = /Users/bklotz/ansible.log
HOST_KEY_CHECKING(/Users/bklotz/.ansible.cfg) = False
##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->
Mac OS X
##### SUMMARY
<!--- Explain the problem briefly -->
Attempting to use simple playbook to change the ip name-servers on Cisco 3850 switches.
##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
Cisco switches, using tacacs for authentication.
<!--- Paste example playbooks or commands between quotes below -->
```yaml
#Playbook
---

- name: Set Anycast name servers
  hosts: first_fl
  gather_facts: no
  tasks:
    - ios_system:
        name_servers:
          - 10.9.8.7
          - 10.9.8.8

# Inventory
[all:vars]
ansible_python_interpreter=/Users/bklotz/miniconda2/bin/python

[basement]
NKNAWHQNWA00DX01
NKNAWHQNWB00DX01
NKNAWHQNWA00AX01
NKNAWHQNWB00AX01

[first_fl]
NKNAWHQNWA01AX01
NKNAWHQNWC01AX01
NKNAWHQNWD01AX01

[second_fl]
NKNAWHQNWA02AX01
NKNAWHQNWA02AX02
NKNAWHQNWC02AX01
NKNAWHQNWD02AX01

[third_fl]
NKNAWHQNWA03AX01
NKNAWHQNWA03AX02
NKNAWHQNWB03AX01
NKNAWHQNWC03AX01
NKNAWHQNWD03AX01

[fourth_fl]
NKNAWHQNWA04AX01
NKNAWHQNWA04AX02
NKNAWHQNWB04AX01
NKNAWHQNWC04AX01
NKNAWHQNWD04AX01

[fifth_fl]
NKNAWHQNWA05AX01
NKNAWHQNWA05AX02
NKNAWHQNWB05AX01
NKNAWHQNWC05AX01
NKNAWHQNWD05AX01
NKNAWHQNWD05AX02

# Variables
---
ansible_connection: network_cli
ansible_network_os: ios
ansible_become: yes
ansible_become_method: enable
ansible_user: s.prime
ansible_ssh_pass: REDACTED
ansible_become_pass: REDACTED

```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
Success in changing the ip name-servers to requested IPs.

##### ACTUAL RESULTS

<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
```
2017-12-18 07:33:18,341 p=2129 u=bklotz |  ansible-playbook 2.5.0
  config file = /Users/bklotz/.ansible.cfg
  configured module search path = [u'/Users/bklotz/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible
  executable location = /Users/bklotz/miniconda2/bin/ansible-playbook
  python version = 2.7.13 |Continuum Analytics, Inc.| (default, Dec 20 2016, 23:05:08) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
2017-12-18 07:33:18,342 p=2129 u=bklotz |  Using /Users/bklotz/.ansible.cfg as config file
2017-12-18 07:33:18,342 p=2129 u=bklotz |  setting up inventory plugins
2017-12-18 07:33:18,392 p=2129 u=bklotz |  Parsed /Users/bklotz/Box Sync/Python/ansible-northwoods/nw-hosts inventory source with ini plugin
2017-12-18 07:33:18,528 p=2129 u=bklotz |  Loading callback plugin default of type stdout, v2.0 from /Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/callback/default.pyc
2017-12-18 07:33:18,769 p=2129 u=bklotz |  PLAYBOOK: testing.yml **************************************************************************************************
2017-12-18 07:33:18,769 p=2129 u=bklotz |  1 plays in testing.yml
2017-12-18 07:33:18,810 p=2129 u=bklotz |  PLAY [Set Anycast name servers] ****************************************************************************************
2017-12-18 07:33:18,858 p=2129 u=bklotz |  META: ran handlers
2017-12-18 07:33:18,865 p=2129 u=bklotz |  TASK [ios_system] ******************************************************************************************************
2017-12-18 07:33:18,865 p=2129 u=bklotz |  task path: /Users/bklotz/Box Sync/Python/ansible-northwoods/testing.yml:7
2017-12-18 07:33:19,967 paramiko.transport starting thread (client mode): 0x41c7910L
2017-12-18 07:33:19,968 paramiko.transport Local version/idstring: SSH-2.0-paramiko_2.4.0
2017-12-18 07:33:19,969 paramiko.transport Remote version/idstring: SSH-2.0-Cisco-1.25
2017-12-18 07:33:19,969 paramiko.transport Connected (version 2.0, client Cisco-1.25)
2017-12-18 07:33:19,972 paramiko.transport kex algos:[u'diffie-hellman-group-exchange-sha1', u'diffie-hellman-group14-sha1', u'diffie-hellman-group1-sha1'] server key:[u'ssh-rsa'] client encrypt:[u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'aes128-cbc', u'3des-cbc', u'aes192-cbc', u'aes256-cbc'] server encrypt:[u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'aes128-cbc', u'3des-cbc', u'aes192-cbc', u'aes256-cbc'] client mac:[u'hmac-sha1', u'hmac-sha1-96'] server mac:[u'hmac-sha1', u'hmac-sha1-96'] client compress:[u'none'] server compress:[u'none'] client lang:[u''] server lang:[u''] kex follows?False
2017-12-18 07:33:19,973 paramiko.transport Kex agreed: diffie-hellman-group-exchange-sha1
2017-12-18 07:33:19,973 paramiko.transport HostKey agreed: ssh-rsa
2017-12-18 07:33:19,973 paramiko.transport Cipher agreed: aes128-ctr
2017-12-18 07:33:19,974 paramiko.transport MAC agreed: hmac-sha1
2017-12-18 07:33:19,974 paramiko.transport Compression agreed: none
2017-12-18 07:33:20,180 paramiko.transport Got server p (2048 bits)
2017-12-18 07:33:20,491 paramiko.transport kex engine KexGex specified hash_algo <built-in function openssl_sha1>
2017-12-18 07:33:20,493 paramiko.transport Switch to new keys ...
2017-12-18 07:33:20,699 paramiko.transport userauth is OK
2017-12-18 07:33:20,700 paramiko.transport Auth banner: 
********************************************************
* WARNING
* This system is owned by Nike, Inc. If you are not
* authorized to access this system, exit immediately.
* Unauthorized access to this system is forbidden by
* company policies, national, and international laws.
* Unauthorized users are subject to criminal and civil
* penalties as well as company initiated disciplinary
* proceedings.
********************************************************

2017-12-18 07:33:20,740 paramiko.transport Authentication (password) successful!
2017-12-18 07:33:20,767 paramiko.transport [chan 0] Max packet in: 32768 bytes
2017-12-18 07:33:20,772 paramiko.transport [chan 0] Max packet out: 4096 bytes
2017-12-18 07:33:20,773 paramiko.transport Secsh channel 0 opened.
2017-12-18 07:33:20,781 paramiko.transport [chan 0] Sesch channel 0 request ok
2017-12-18 07:33:20,786 paramiko.transport [chan 0] Sesch channel 0 request ok
2017-12-18 07:33:41,332 p=2136 u=bklotz |  shutdown complete
2017-12-18 07:33:41,419 paramiko.transport EOF in transport thread
2017-12-18 07:33:42,002 p=2129 u=bklotz |  Traceback (most recent call last):
  File "/Users/bklotz/miniconda2/bin/ansible-connection", line 72, in start
    self.connection._connect()
  File "/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/connection/network_cli.py", line 208, in _connect
    self._terminal.on_become(passwd=auth_pass)
  File "/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/terminal/ios.py", line 72, in on_become
    prompt = self._get_prompt()
  File "/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/terminal/__init__.py", line 74, in _get_prompt
    self._exec_cli_command(b'')
  File "/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/terminal/__init__.py", line 64, in _exec_cli_command
    return self._connection.exec_command(cmd)
  File "/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/connection/network_cli.py", line 130, in exec_command
    return self.send(command=cmd)
  File "/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/connection/network_cli.py", line 302, in send
    raise AnsibleConnectionFailure("timeout trying to send command: %s" % command.strip())
AnsibleConnectionFailure: timeout trying to send command: 

2017-12-18 07:33:42,007 p=2129 u=bklotz |  fatal: [NKNAWHQNWA01AX01]: FAILED! => {
    "msg": "timeout trying to send command: "
}
2017-12-18 07:33:42,012 p=2129 u=bklotz |  	to retry, use: --limit @/Users/bklotz/Box Sync/Python/ansible-northwoods/testing.retry

2017-12-18 07:33:42,012 p=2129 u=bklotz |  PLAY RECAP *************************************************************************************************************
2017-12-18 07:33:42,013 p=2129 u=bklotz |  NKNAWHQNWA01AX01           : ok=0    changed=0    unreachable=0    failed=1   
```

<!--- Paste verbatim command output between quotes below -->
```
[bklotz ansible-northwoods]$ ansible-playbook -i nw-hosts testing.yml --limit=NKNAWHQNWA01AX01 -vvvv
ansible-playbook 2.5.0
  config file = /Users/bklotz/.ansible.cfg
  configured module search path = [u'/Users/bklotz/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible
  executable location = /Users/bklotz/miniconda2/bin/ansible-playbook
  python version = 2.7.13 |Continuum Analytics, Inc.| (default, Dec 20 2016, 23:05:08) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
Using /Users/bklotz/.ansible.cfg as config file
setting up inventory plugins
Parsed /Users/bklotz/Box Sync/Python/ansible-northwoods/nw-hosts inventory source with ini plugin
Loading callback plugin default of type stdout, v2.0 from /Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/callback/default.pyc

PLAYBOOK: testing.yml **************************************************************************************************
1 plays in testing.yml

PLAY [Set Anycast name servers] ****************************************************************************************
META: ran handlers

TASK [ios_system] ******************************************************************************************************
task path: /Users/bklotz/Box Sync/Python/ansible-northwoods/testing.yml:7
<NKNAWHQNWA01AX01> attempting to start connection
<NKNAWHQNWA01AX01> using connection plugin network_cli
<NKNAWHQNWA01AX01> local domain socket does not exist, starting it
<NKNAWHQNWA01AX01> control socket path is /Users/bklotz/.ansible/pc/c00f439313
Traceback (most recent call last):
  File "/Users/bklotz/miniconda2/bin/ansible-connection", line 72, in start
    self.connection._connect()
  File "/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/connection/network_cli.py", line 208, in _connect
    self._terminal.on_become(passwd=auth_pass)
  File "/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/terminal/ios.py", line 72, in on_become
    prompt = self._get_prompt()
  File "/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/terminal/__init__.py", line 74, in _get_prompt
    self._exec_cli_command(b'')
  File "/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/terminal/__init__.py", line 64, in _exec_cli_command
    return self._connection.exec_command(cmd)
  File "/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/connection/network_cli.py", line 130, in exec_command
    return self.send(command=cmd)
  File "/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/connection/network_cli.py", line 302, in send
    raise AnsibleConnectionFailure("timeout trying to send command: %s" % command.strip())
AnsibleConnectionFailure: timeout trying to send command: 
fatal: [NKNAWHQNWA01AX01]: FAILED! => {
    "msg": "timeout trying to send command: "
}
        to retry, use: --limit @/Users/bklotz/Box Sync/Python/ansible-northwoods/testing.retry

PLAY RECAP *************************************************************************************************************
NKNAWHQNWA01AX01           : ok=0    changed=0    unreachable=0    failed=1   

```



Discussion:
Files identified in the description:
* [lib/ansible/plugins/connection/network_cli.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/connection/network_cli.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @Qalthos @ganeshrn @gundalow @kedarX @privateip @rcarrillocruz @trishnaguha
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Alright, I can't reproduce what you're seeing, but I think I know what the issue is. Can you tell me if #34101 fixes this issue for you?
Updated to devel branch, and issue persists.

2017-12-20 09:41:15,063 p=96821 u=bklotz |  ansible-playbook 2.5.0
  config file = /Users/bklotz/.ansible.cfg
  configured module search path =
[u'/Users/bklotz/.ansible/plugins/modules',
u'/usr/share/ansible/plugins/modules']
  ansible python module location =
/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible
  executable location = /Users/bklotz/miniconda2/bin/ansible-playbook
  python version = 2.7.13 |Continuum Analytics, Inc.| (default, Dec 20
2016, 23:05:08) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
2017-12-20 09:41:15,064 p=96821 u=bklotz |  Using
/Users/bklotz/.ansible.cfg as config file
2017-12-20 09:41:15,064 p=96821 u=bklotz |  setting up inventory plugins
2017-12-20 09:41:15,114 p=96821 u=bklotz |  Parsed /Users/bklotz/Box
Sync/Python/ansible-northwoods/nw-hosts inventory source with ini plugin
2017-12-20 09:41:15,485 p=96821 u=bklotz |  Loading callback plugin default
of type stdout, v2.0 from
/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/callback/default.pyc
2017-12-20 09:41:15,751 p=96821 u=bklotz |  PLAYBOOK: testing.yml
**************************************************************************************************
2017-12-20 09:41:15,752 p=96821 u=bklotz |  1 plays in testing.yml
2017-12-20 09:41:15,791 p=96821 u=bklotz |  PLAY [Set Anycast name servers]
****************************************************************************************
2017-12-20 09:41:15,832 p=96821 u=bklotz |  META: ran handlers
2017-12-20 09:41:15,839 p=96821 u=bklotz |  TASK [ios_system]
******************************************************************************************************
2017-12-20 09:41:15,839 p=96821 u=bklotz |  task path: /Users/bklotz/Box
Sync/Python/ansible-northwoods/testing.yml:7
2017-12-20 09:41:16,920 paramiko.transport starting thread (client mode):
0x61249d0L
2017-12-20 09:41:16,920 paramiko.transport Local version/idstring:
SSH-2.0-paramiko_2.4.0
2017-12-20 09:41:16,923 paramiko.transport Remote version/idstring:
SSH-2.0-Cisco-1.25
2017-12-20 09:41:16,924 paramiko.transport Connected (version 2.0, client
Cisco-1.25)
2017-12-20 09:41:16,924 paramiko.transport kex
algos:[u'diffie-hellman-group-exchange-sha1',
u'diffie-hellman-group14-sha1', u'diffie-hellman-group1-sha1'] server
key:[u'ssh-rsa'] client encrypt:[u'aes128-ctr', u'aes192-ctr',
u'aes256-ctr', u'aes128-cbc', u'3des-cbc', u'aes192-cbc', u'aes256-cbc']
server encrypt:[u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'aes128-cbc',
u'3des-cbc', u'aes192-cbc', u'aes256-cbc'] client mac:[u'hmac-sha1',
u'hmac-sha1-96'] server mac:[u'hmac-sha1', u'hmac-sha1-96'] client
compress:[u'none'] server compress:[u'none'] client lang:[u''] server
lang:[u''] kex follows?False
2017-12-20 09:41:16,924 paramiko.transport Kex agreed:
diffie-hellman-group-exchange-sha1
2017-12-20 09:41:16,924 paramiko.transport HostKey agreed: ssh-rsa
2017-12-20 09:41:16,924 paramiko.transport Cipher agreed: aes128-ctr
2017-12-20 09:41:16,925 paramiko.transport MAC agreed: hmac-sha1
2017-12-20 09:41:16,925 paramiko.transport Compression agreed: none
2017-12-20 09:41:17,124 paramiko.transport Got server p (2048 bits)
2017-12-20 09:41:17,432 paramiko.transport kex engine KexGex specified
hash_algo <built-in function openssl_sha1>
2017-12-20 09:41:17,433 paramiko.transport Switch to new keys ...
2017-12-20 09:41:17,637 paramiko.transport userauth is OK
2017-12-20 09:41:17,639 paramiko.transport Auth banner:
********************************************************
* WARNING
* This system is owned by Nike, Inc. If you are not
* authorized to access this system, exit immediately.
* Unauthorized access to this system is forbidden by
* company policies, national, and international laws.
* Unauthorized users are subject to criminal and civil
* penalties as well as company initiated disciplinary
* proceedings.
********************************************************

2017-12-20 09:41:17,667 paramiko.transport Authentication (password)
successful!
2017-12-20 09:41:17,673 paramiko.transport [chan 0] Max packet in: 32768
bytes
2017-12-20 09:41:17,678 paramiko.transport [chan 0] Max packet out: 4096
bytes
2017-12-20 09:41:17,678 paramiko.transport Secsh channel 0 opened.
2017-12-20 09:41:17,686 paramiko.transport [chan 0] Sesch channel 0 request
ok
2017-12-20 09:41:17,692 paramiko.transport [chan 0] Sesch channel 0 request
ok
2017-12-20 09:41:38,231 p=96828 u=bklotz |  shutdown complete
2017-12-20 09:41:38,329 paramiko.transport EOF in transport thread
2017-12-20 09:41:38,885 p=96821 u=bklotz |  Traceback (most recent call
last):
  File "/Users/bklotz/miniconda2/bin/ansible-connection", line 72, in start
    self.connection._connect()
  File
"/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/connection/network_cli.py",
line 210, in _connect
    self._terminal.on_become(passwd=auth_pass)
  File
"/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/terminal/ios.py",
line 72, in on_become
    prompt = self._get_prompt()
  File
"/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/terminal/__init__.py",
line 74, in _get_prompt
    self._exec_cli_command(b'')
  File
"/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/terminal/__init__.py",
line 64, in _exec_cli_command
    return self._connection.exec_command(cmd)
  File
"/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/connection/network_cli.py",
line 132, in exec_command
    return self.send(command=cmd)
  File
"/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/connection/network_cli.py",
line 304, in send
    raise AnsibleConnectionFailure("timeout trying to send command: %s" %
command.strip())
AnsibleConnectionFailure: timeout trying to send command:

2017-12-20 09:41:38,891 p=96821 u=bklotz |  fatal: [NKNAWHQNWA01AX01]:
FAILED! => {
    "msg": "timeout trying to send command: "
}
2017-12-20 09:41:38,894 p=96821 u=bklotz |  to retry, use: --limit
@/Users/bklotz/Box Sync/Python/ansible-northwoods/testing.retry

2017-12-20 09:41:38,894 p=96821 u=bklotz |  PLAY RECAP
*************************************************************************************************************
2017-12-20 09:41:38,894 p=96821 u=bklotz |  NKNAWHQNWA01AX01           :
ok=0    changed=0    unreachable=0    failed=1


On Wed, Dec 20, 2017 at 9:28 AM Nathaniel Case <notifications@github.com>
wrote:

> Alright, I can't reproduce what you're seeing, but can you tell me if
> #34101 <https://github.com/ansible/ansible/pull/34101> fixes this issue
> for you?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/ansible/ansible/issues/33993#issuecomment-353128664>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AbFU9nrHKdBopRANScVFM-HYezM1jvW1ks5tCUPOgaJpZM4RFrMW>
> .
>
-- 
"Have you tried turning it off and on again?"

Sorry for not being clearer, it hasn't been merged yet. Is it possible for you to check out that branch and test that? I'm nearly certain this is the issue, I just want to be sure.
I'd love to, but am not sure how to find that branch. Can you help me with
the syntax to pull it? I have been using:

sudo -H pip install git+https://github.com/ansible/ansible.git@devel
--upgrade

On Wed, Dec 20, 2017 at 2:56 PM Nathaniel Case <notifications@github.com>
wrote:

> Sorry for not being clearer, it hasn't been merged yet. Is it possible for
> you to check out that branch and test that? I'm nearly certain this is the
> issue, I just want to be sure.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/ansible/ansible/issues/33993#issuecomment-353206884>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AbFU9owMo3qBeSmz6tgRGP3c8r10AKY0ks5tCZCHgaJpZM4RFrMW>
> .
>
-- 
"Have you tried turning it off and on again?"

That appears to be `pip install git+https://github.com/Qalthos/ansible.git@restore-prompt --upgrade`
Back to my original issue:

2017-12-22 10:03:27,192 p=14705 u=bklotz |  ansible-playbook 2.5.0
  config file = /Users/bklotz/.ansible.cfg
  configured module search path =
[u'/Users/bklotz/.ansible/plugins/modules',
u'/usr/share/ansible/plugins/modules']
  ansible python module location =
/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible
  executable location = /Users/bklotz/miniconda2/bin/ansible-playbook
  python version = 2.7.13 |Continuum Analytics, Inc.| (default, Dec 20
2016, 23:05:08) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
2017-12-22 10:03:27,193 p=14705 u=bklotz |  Using
/Users/bklotz/.ansible.cfg as config file
2017-12-22 10:03:27,193 p=14705 u=bklotz |  setting up inventory plugins
2017-12-22 10:03:27,242 p=14705 u=bklotz |  Parsed /Users/bklotz/Box
Sync/Python/ansible-northwoods/nw-hosts inventory source with ini plugin
2017-12-22 10:03:27,678 p=14705 u=bklotz |  Loading callback plugin default
of type stdout, v2.0 from
/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/callback/default.pyc
2017-12-22 10:03:27,941 p=14705 u=bklotz |  PLAYBOOK: testing.yml
**************************************************************************************************
2017-12-22 10:03:27,941 p=14705 u=bklotz |  1 plays in testing.yml
2017-12-22 10:03:27,976 p=14705 u=bklotz |  PLAY [Set Anycast name servers]
****************************************************************************************
2017-12-22 10:03:28,020 p=14705 u=bklotz |  META: ran handlers
2017-12-22 10:03:28,025 p=14705 u=bklotz |  TASK [ios_system]
******************************************************************************************************
2017-12-22 10:03:28,025 p=14705 u=bklotz |  task path: /Users/bklotz/Box
Sync/Python/ansible-northwoods/testing.yml:7
2017-12-22 10:03:29,151 paramiko.transport starting thread (client mode):
0x10bd29d0L
2017-12-22 10:03:29,151 paramiko.transport Local version/idstring:
SSH-2.0-paramiko_2.4.0
2017-12-22 10:03:29,173 paramiko.transport Remote version/idstring:
SSH-2.0-Cisco-1.25
2017-12-22 10:03:29,173 paramiko.transport Connected (version 2.0, client
Cisco-1.25)
2017-12-22 10:03:29,178 paramiko.transport kex
algos:[u'diffie-hellman-group-exchange-sha1',
u'diffie-hellman-group14-sha1', u'diffie-hellman-group1-sha1'] server
key:[u'ssh-rsa'] client encrypt:[u'aes128-ctr', u'aes192-ctr',
u'aes256-ctr', u'aes128-cbc', u'3des-cbc', u'aes192-cbc', u'aes256-cbc']
server encrypt:[u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'aes128-cbc',
u'3des-cbc', u'aes192-cbc', u'aes256-cbc'] client mac:[u'hmac-sha1',
u'hmac-sha1-96'] server mac:[u'hmac-sha1', u'hmac-sha1-96'] client
compress:[u'none'] server compress:[u'none'] client lang:[u''] server
lang:[u''] kex follows?False
2017-12-22 10:03:29,178 paramiko.transport Kex agreed:
diffie-hellman-group-exchange-sha1
2017-12-22 10:03:29,178 paramiko.transport HostKey agreed: ssh-rsa
2017-12-22 10:03:29,179 paramiko.transport Cipher agreed: aes128-ctr
2017-12-22 10:03:29,179 paramiko.transport MAC agreed: hmac-sha1
2017-12-22 10:03:29,179 paramiko.transport Compression agreed: none
2017-12-22 10:03:29,426 paramiko.transport Got server p (2048 bits)
2017-12-22 10:03:29,740 paramiko.transport kex engine KexGex specified
hash_algo <built-in function openssl_sha1>
2017-12-22 10:03:29,742 paramiko.transport Switch to new keys ...
2017-12-22 10:03:30,034 paramiko.transport userauth is OK
2017-12-22 10:03:30,035 paramiko.transport Auth banner:
********************************************************
* WARNING
* This system is owned by Nike, Inc. If you are not
* authorized to access this system, exit immediately.
* Unauthorized access to this system is forbidden by
* company policies, national, and international laws.
* Unauthorized users are subject to criminal and civil
* penalties as well as company initiated disciplinary
* proceedings.
********************************************************

2017-12-22 10:03:30,092 paramiko.transport Authentication (password)
successful!
2017-12-22 10:03:30,122 paramiko.transport [chan 0] Max packet in: 32768
bytes
2017-12-22 10:03:30,149 paramiko.transport [chan 0] Max packet out: 4096
bytes
2017-12-22 10:03:30,150 paramiko.transport Secsh channel 0 opened.
2017-12-22 10:03:30,190 paramiko.transport [chan 0] Sesch channel 0 request
ok
2017-12-22 10:03:30,223 paramiko.transport [chan 0] Sesch channel 0 request
ok
2017-12-22 10:03:40,662 p=14712 u=bklotz |  shutdown complete
2017-12-22 10:03:40,754 paramiko.transport EOF in transport thread
2017-12-22 10:03:41,302 p=14705 u=bklotz |  Traceback (most recent call
last):
  File "/Users/bklotz/miniconda2/bin/ansible-connection", line 72, in start
    self.connection._connect()
  File
"/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/connection/network_cli.py",
line 210, in _connect
    self._terminal.on_become(passwd=auth_pass)
  File
"/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/terminal/ios.py",
line 69, in on_become
    raise AnsibleConnectionFailure('unable to elevate privilege to enable
mode')
AnsibleConnectionFailure: unable to elevate privilege to enable mode

2017-12-22 10:03:41,317 p=14705 u=bklotz |  fatal: [NKNAWHQNWA01AX01]:
FAILED! => {
    "msg": "unable to elevate privilege to enable mode"
}
2017-12-22 10:03:41,328 p=14705 u=bklotz |  to retry, use: --limit
@/Users/bklotz/Box Sync/Python/ansible-northwoods/testing.retry

2017-12-22 10:03:41,329 p=14705 u=bklotz |  PLAY RECAP
*************************************************************************************************************
2017-12-22 10:03:41,366 p=14705 u=bklotz |  NKNAWHQNWA01AX01           :
ok=0    changed=0    unreachable=0    failed=1


On Wed, Dec 20, 2017 at 7:26 PM Nathaniel Case <notifications@github.com>
wrote:

> That appears to be pip install git+
> https://github.com/Qalthos/ansible.git@restore-prompt --upgrade
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/ansible/ansible/issues/33993#issuecomment-353248472>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AbFU9t-lo4IzuRsGhFA0x_R58TpsHk4vks5tCc_IgaJpZM4RFrMW>
> .
>
-- 
"Have you tried turning it off and on again?"

I updated using "sudo pip install git+https://github.com/Qalthos/ansible.git@restore-prompt --upgrade" and now have the issue again:

2018-01-02 07:51:24,438 p=1938 u=bklotz |  ansible-playbook 2.5.0
  config file = /Users/bklotz/.ansible.cfg
  configured module search path = [u'/Users/bklotz/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible
  executable location = /Users/bklotz/miniconda2/bin/ansible-playbook
  python version = 2.7.13 |Continuum Analytics, Inc.| (default, Dec 20 2016, 23:05:08) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
2018-01-02 07:51:24,438 p=1938 u=bklotz |  Using /Users/bklotz/.ansible.cfg as config file
2018-01-02 07:51:24,439 p=1938 u=bklotz |  setting up inventory plugins
2018-01-02 07:51:24,481 p=1938 u=bklotz |  Parsed /Users/bklotz/Box Sync/Python/ansible-northwoods/nw-hosts inventory source with ini plugin
2018-01-02 07:51:24,586 p=1938 u=bklotz |  Loading callback plugin default of type stdout, v2.0 from /Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/callback/default.pyc
2018-01-02 07:51:24,765 p=1938 u=bklotz |  PLAYBOOK: testing.yml **************************************************************************************************
2018-01-02 07:51:24,765 p=1938 u=bklotz |  1 plays in testing.yml
2018-01-02 07:51:24,787 p=1938 u=bklotz |  PLAY [Set Anycast name servers] ****************************************************************************************
2018-01-02 07:51:24,819 p=1938 u=bklotz |  META: ran handlers
2018-01-02 07:51:24,828 p=1938 u=bklotz |  TASK [ios_system] ******************************************************************************************************
2018-01-02 07:51:24,828 p=1938 u=bklotz |  task path: /Users/bklotz/Box Sync/Python/ansible-northwoods/testing.yml:7
2018-01-02 07:51:25,704 paramiko.transport starting thread (client mode): 0xb9d8a50L
2018-01-02 07:51:25,704 paramiko.transport Local version/idstring: SSH-2.0-paramiko_2.4.0
2018-01-02 07:51:25,708 paramiko.transport Remote version/idstring: SSH-2.0-Cisco-1.25
2018-01-02 07:51:25,709 paramiko.transport Connected (version 2.0, client Cisco-1.25)
2018-01-02 07:51:25,709 paramiko.transport kex algos:[u'diffie-hellman-group-exchange-sha1', u'diffie-hellman-group14-sha1', u'diffie-hellman-group1-sha1'] server key:[u'ssh-rsa'] client encrypt:[u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'aes128-cbc', u'3des-cbc', u'aes192-cbc', u'aes256-cbc'] server encrypt:[u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'aes128-cbc', u'3des-cbc', u'aes192-cbc', u'aes256-cbc'] client mac:[u'hmac-sha1', u'hmac-sha1-96'] server mac:[u'hmac-sha1', u'hmac-sha1-96'] client compress:[u'none'] server compress:[u'none'] client lang:[u''] server lang:[u''] kex follows?False
2018-01-02 07:51:25,710 paramiko.transport Kex agreed: diffie-hellman-group-exchange-sha1
2018-01-02 07:51:25,710 paramiko.transport HostKey agreed: ssh-rsa
2018-01-02 07:51:25,710 paramiko.transport Cipher agreed: aes128-ctr
2018-01-02 07:51:25,710 paramiko.transport MAC agreed: hmac-sha1
2018-01-02 07:51:25,710 paramiko.transport Compression agreed: none
2018-01-02 07:51:25,918 paramiko.transport Got server p (2048 bits)
2018-01-02 07:51:26,253 paramiko.transport kex engine KexGex specified hash_algo <built-in function openssl_sha1>
2018-01-02 07:51:26,254 paramiko.transport Switch to new keys ...
2018-01-02 07:51:26,459 paramiko.transport userauth is OK
2018-01-02 07:51:26,460 paramiko.transport Auth banner: 
********************************************************
* WARNING
* This system is owned by Nike, Inc. If you are not
* authorized to access this system, exit immediately.
* Unauthorized access to this system is forbidden by
* company policies, national, and international laws.
* Unauthorized users are subject to criminal and civil
* penalties as well as company initiated disciplinary
* proceedings.
********************************************************

2018-01-02 07:51:26,504 paramiko.transport Authentication (password) successful!
2018-01-02 07:51:26,527 paramiko.transport [chan 0] Max packet in: 32768 bytes
2018-01-02 07:51:26,531 paramiko.transport [chan 0] Max packet out: 4096 bytes
2018-01-02 07:51:26,532 paramiko.transport Secsh channel 0 opened.
2018-01-02 07:51:26,545 paramiko.transport [chan 0] Sesch channel 0 request ok
2018-01-02 07:51:26,556 paramiko.transport [chan 0] Sesch channel 0 request ok
2018-01-02 07:51:47,410 p=1945 u=bklotz |  shutdown complete
2018-01-02 07:51:47,440 paramiko.transport EOF in transport thread
2018-01-02 07:51:48,109 p=1938 u=bklotz |  Traceback (most recent call last):
  File "/Users/bklotz/miniconda2/bin/ansible-connection", line 72, in start
    self.connection._connect()
  File "/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/connection/network_cli.py", line 210, in _connect
    self._terminal.on_become(passwd=auth_pass)
  File "/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/terminal/ios.py", line 72, in on_become
    prompt = self._get_prompt()
  File "/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/terminal/__init__.py", line 74, in _get_prompt
    self._exec_cli_command(b'')
  File "/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/terminal/__init__.py", line 64, in _exec_cli_command
    return self._connection.exec_command(cmd)
  File "/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/connection/network_cli.py", line 132, in exec_command
    return self.send(command=cmd)
  File "/Users/bklotz/miniconda2/lib/python2.7/site-packages/ansible/plugins/connection/network_cli.py", line 304, in send
    raise AnsibleConnectionFailure("timeout trying to send command: %s" % command.strip())
AnsibleConnectionFailure: timeout trying to send command: 

2018-01-02 07:51:48,117 p=1938 u=bklotz |  fatal: [NKNAWHQNWA01AX01]: FAILED! => {
    "msg": "timeout trying to send command: "
}
2018-01-02 07:51:48,121 p=1938 u=bklotz |  	to retry, use: --limit @/Users/bklotz/Box Sync/Python/ansible-northwoods/testing.retry

2018-01-02 07:51:48,122 p=1938 u=bklotz |  PLAY RECAP *************************************************************************************************************
2018-01-02 07:51:48,122 p=1938 u=bklotz |  NKNAWHQNWA01AX01           : ok=0    changed=0    unreachable=0    failed=1   

Using a special debug build, we can see that when the enable command is sent, ansible does not send the ansible_become_password regardless of how it is supplied (I tried in a group_vars/all.yml file, in the inventory file and via the command line with -e). When the password is not sent, ansible then sends a \r, which causes the switch to prompt "Enter Old Password:", as it assumes you wish to change your password at this point, which is not what is intended.

```2018-01-11 11:13:51,941 p=40342 u=bklotz |   40351 1515698031.94171: CLI-PROMPT: looking for prompt in | enable\r\n
2018-01-11 11:13:52,159 p=40342 u=bklotz |   40351 1515698032.15927: CLI-PROMPT: looking for prompts in | enable\r\nPassword:
2018-01-11 11:13:52,159 p=40342 u=bklotz |   40351 1515698032.15962: CLI-PROMPT: looking for prompt in | enable\r\nPassword:
2018-01-11 11:14:02,163 p=40342 u=bklotz |   40351 1515698042.16355: CLI-SEND: \r
2018-01-11 11:14:02,380 p=40342 u=bklotz |   40351 1515698042.38048: CLI-PROMPT: looking for prompts in | \r\nEnter Old Password:
2018-01-11 11:14:02,380 p=40342 u=bklotz |   40351 1515698042.38083: CLI-PROMPT: looking for prompt in | \r\nEnter Old Password:
```
@brainclots Can you please test this on latest devel branch? 
There recently has been a PR https://github.com/ansible/ansible/pull/34723 merged which might make this return a meaningful error message. 

To test against devel branch:
```
git clone https://github.com/ansible/ansible.git
source hacking/env-setup
```
Here is what I see on the command line when running the playbook with
'-vvvv':

The full traceback is:
Traceback (most recent call last):
  File "/Users/bklotz/ansible/bin/ansible-connection", line 70, in start
    self.connection._connect()
  File
"/Users/bklotz/ansible/lib/ansible/plugins/connection/network_cli.py", line
323, in _connect
    self._terminal.on_become(passwd=auth_pass)
  File "/Users/bklotz/ansible/lib/ansible/plugins/terminal/ios.py", line
75, in on_become
    prompt = self._get_prompt()
  File "/Users/bklotz/ansible/lib/ansible/plugins/terminal/__init__.py",
line 74, in _get_prompt
    self._exec_cli_command(b'')
  File "/Users/bklotz/ansible/lib/ansible/plugins/terminal/__init__.py",
line 64, in _exec_cli_command
    return self._connection.exec_command(cmd)
  File
"/Users/bklotz/ansible/lib/ansible/plugins/connection/network_cli.py", line
245, in exec_command
    return self.send(command=cmd)
  File
"/Users/bklotz/ansible/lib/ansible/plugins/connection/network_cli.py", line
421, in send
    raise AnsibleConnectionFailure("timeout trying to send command: %s" %
command.strip())
AnsibleConnectionFailure: timeout trying to send command:
fatal: [NKNAWHQNWA01AX01]: FAILED! => {
    "failed": true,
    "msg": "timeout trying to send command: "
}

Here are the entries in ansible.log:
2018-01-25 11:20:10,109 p=47423 u=bklotz |  ansible-playbook 2.5.0 (devel
7abdab6c9e) last updated 2018/01/25 11:07:23 (GMT -700)
  config file = /Users/bklotz/.ansible.cfg
  configured module search path =
[u'/Users/bklotz/.ansible/plugins/modules',
u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/bklotz/ansible/lib/ansible
  executable location = /Users/bklotz/ansible/bin/ansible-playbook
  python version = 2.7.13 |Continuum Analytics, Inc.| (default, Dec 20
2016, 23:05:08) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
2018-01-25 11:20:10,110 p=47423 u=bklotz |  Using
/Users/bklotz/.ansible.cfg as config file
2018-01-25 11:20:10,110 p=47423 u=bklotz |  setting up inventory plugins
2018-01-25 11:20:10,163 p=47423 u=bklotz |  Parsed /Users/bklotz/Box
Sync/Python/ansible-northwoods/nw-hosts inventory source with ini plugin
2018-01-25 11:20:10,251 p=47423 u=bklotz |  Loading callback plugin default
of type stdout, v2.0 from
/Users/bklotz/ansible/lib/ansible/plugins/callback/default.pyc
2018-01-25 11:20:10,555 p=47423 u=bklotz |  PLAYBOOK: testing.yml
**************************************************************************************************
2018-01-25 11:20:10,556 p=47423 u=bklotz |  1 plays in testing.yml
2018-01-25 11:20:10,620 p=47423 u=bklotz |  PLAY [Set Anycast name servers]
****************************************************************************************
2018-01-25 11:20:10,673 p=47423 u=bklotz |  META: ran handlers
2018-01-25 11:20:10,679 p=47423 u=bklotz |  TASK [ios_system]
******************************************************************************************************
2018-01-25 11:20:10,679 p=47423 u=bklotz |  task path: /Users/bklotz/Box
Sync/Python/ansible-northwoods/testing.yml:7
2018-01-25 11:20:12,190 paramiko.transport starting thread (client mode):
0xa1a6650L
2018-01-25 11:20:12,191 paramiko.transport Local version/idstring:
SSH-2.0-paramiko_2.4.0
2018-01-25 11:20:12,194 paramiko.transport Remote version/idstring:
SSH-2.0-Cisco-1.25
2018-01-25 11:20:12,194 paramiko.transport Connected (version 2.0, client
Cisco-1.25)
2018-01-25 11:20:12,195 paramiko.transport kex
algos:[u'diffie-hellman-group-exchange-sha1',
u'diffie-hellman-group14-sha1', u'diffie-hellman-group1-sha1'] server
key:[u'ssh-rsa'] client encrypt:[u'aes128-ctr', u'aes192-ctr',
u'aes256-ctr', u'aes128-cbc', u'3des-cbc', u'aes192-cbc', u'aes256-cbc']
server encrypt:[u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'aes128-cbc',
u'3des-cbc', u'aes192-cbc', u'aes256-cbc'] client mac:[u'hmac-sha1',
u'hmac-sha1-96'] server mac:[u'hmac-sha1', u'hmac-sha1-96'] client
compress:[u'none'] server compress:[u'none'] client lang:[u''] server
lang:[u''] kex follows?False
2018-01-25 11:20:12,195 paramiko.transport Kex agreed:
diffie-hellman-group-exchange-sha1
2018-01-25 11:20:12,195 paramiko.transport HostKey agreed: ssh-rsa
2018-01-25 11:20:12,196 paramiko.transport Cipher agreed: aes128-ctr
2018-01-25 11:20:12,196 paramiko.transport MAC agreed: hmac-sha1
2018-01-25 11:20:12,196 paramiko.transport Compression agreed: none
2018-01-25 11:20:12,406 paramiko.transport Got server p (2048 bits)
2018-01-25 11:20:12,735 paramiko.transport kex engine KexGex specified
hash_algo <built-in function openssl_sha1>
2018-01-25 11:20:12,736 paramiko.transport Switch to new keys ...
2018-01-25 11:20:12,945 paramiko.transport userauth is OK
2018-01-25 11:20:12,945 paramiko.transport Auth banner:
********************************************************
* WARNING
* This system is owned by Nike, Inc. If you are not
* authorized to access this system, exit immediately.
* Unauthorized access to this system is forbidden by
* company policies, national, and international laws.
* Unauthorized users are subject to criminal and civil
* penalties as well as company initiated disciplinary
* proceedings.
********************************************************

2018-01-25 11:20:12,979 paramiko.transport Authentication (password)
successful!
2018-01-25 11:20:13,010 paramiko.transport [chan 0] Max packet in: 32768
bytes
2018-01-25 11:20:13,014 paramiko.transport [chan 0] Max packet out: 4096
bytes
2018-01-25 11:20:13,014 paramiko.transport Secsh channel 0 opened.
2018-01-25 11:20:13,022 paramiko.transport [chan 0] Sesch channel 0 request
ok
2018-01-25 11:20:13,027 paramiko.transport [chan 0] Sesch channel 0 request
ok
2018-01-25 11:20:33,539 p=47434 u=bklotz |  shutdown complete
2018-01-25 11:20:33,621 paramiko.transport EOF in transport thread
2018-01-25 11:20:34,179 p=47423 u=bklotz |  The full traceback is:
Traceback (most recent call last):
  File "/Users/bklotz/ansible/bin/ansible-connection", line 70, in start
    self.connection._connect()
  File
"/Users/bklotz/ansible/lib/ansible/plugins/connection/network_cli.py", line
323, in _connect
    self._terminal.on_become(passwd=auth_pass)
  File "/Users/bklotz/ansible/lib/ansible/plugins/terminal/ios.py", line
75, in on_become
    prompt = self._get_prompt()
  File "/Users/bklotz/ansible/lib/ansible/plugins/terminal/__init__.py",
line 74, in _get_prompt
    self._exec_cli_command(b'')
  File "/Users/bklotz/ansible/lib/ansible/plugins/terminal/__init__.py",
line 64, in _exec_cli_command
    return self._connection.exec_command(cmd)
  File
"/Users/bklotz/ansible/lib/ansible/plugins/connection/network_cli.py", line
245, in exec_command
    return self.send(command=cmd)
  File
"/Users/bklotz/ansible/lib/ansible/plugins/connection/network_cli.py", line
421, in send
    raise AnsibleConnectionFailure("timeout trying to send command: %s" %
command.strip())
AnsibleConnectionFailure: timeout trying to send command:

2018-01-25 11:20:34,185 p=47423 u=bklotz |  fatal: [NKNAWHQNWA01AX01]:
FAILED! => {
    "failed": true,
    "msg": "timeout trying to send command: "
}
2018-01-25 11:20:34,188 p=47423 u=bklotz |  to retry, use: --limit
@/Users/bklotz/Box Sync/Python/ansible-northwoods/testing.retry

2018-01-25 11:20:34,188 p=47423 u=bklotz |  PLAY RECAP
*************************************************************************************************************
2018-01-25 11:20:34,189 p=47423 u=bklotz |  NKNAWHQNWA01AX01           :
ok=0    changed=0    unreachable=0    failed=1


On Wed, Jan 24, 2018 at 9:06 PM Trishna Guha <notifications@github.com>
wrote:

> @brainclots <https://github.com/brainclots> Can you please test this on
> latest devel branch?
> There recently has been a PR #34723
> <https://github.com/ansible/ansible/pull/34723> merged which might makes
> this return a meaningful error message.
>
> To test against devel branch:
>
> git clone https://github.com/ansible/ansible.git
> source hacking/env-setup
>
> —
> You are receiving this because you were mentioned.
>
>
> Reply to this email directly, view it on GitHub
> <https://github.com/ansible/ansible/issues/33993#issuecomment-360362018>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AbFU9mzcpPbX4RBT-qnYmO6VdiiRZPwIks5tOAu4gaJpZM4RFrMW>
> .
>
-- 
"Have you tried turning it off and on again?"

@brainclots In the current implementation, the logic of fetching the current cli prompt is not correct. For fetching the current device prompt terminal plugin executes `\r` on the remote host which is not required because of this the prompt and answer value get's out of sync.

As the current matched prompt is stored in `_matched_prompt` variable of network_cli's `Connection` object the fix is to return this value in the terminal plugin.

I have raised a PR to fix this issue https://github.com/ansible/ansible/pull/35439
Please retest with the patch in PR #35439 and check if you are still seeing this issue.
@brainclots If you are still facing issue after applying patch in PR #35439 
Can you add more debug logs as given here  https://gist.github.com/ganeshrn/eb722f185d14f42128e4a62498c44640 in `network_cli.py` file in your running environment and check the log file if the values of `become_pass` and `prompt` is as expected?
Ganesh,

I would be happy to, but not being a developer myself, I need a little help
with the instructions to add the debug info you linked to above. Could you
send the steps to merge the diff you link to above, please?

On Mon, Jan 29, 2018 at 12:11 AM Ganesh Nalawade <notifications@github.com>
wrote:

> @brainclots <https://github.com/brainclots> If you are still facing issue
> after applying patch in PR #35439
> <https://github.com/ansible/ansible/pull/35439>
> Can you add more debug logs as given here
> https://gist.github.com/ganeshrn/eb722f185d14f42128e4a62498c44640 in
> network_cli.py file in your running environment and check the log file if
> the values of become_pass and prompt is as expected?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/ansible/ansible/issues/33993#issuecomment-361168682>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AbFU9uYH4klcyD41bQgFqsmmKq015imeks5tPX0YgaJpZM4RFrMW>
> .
>
-- 
"Have you tried turning it off and on again?"

To test the patch in PR #35439 please install using the command:
```
pip install git+https://github.com/ganeshrn/ansible.git@terminal_prompt_issue --upgrade
```

I have pushed a branch with debug logs. If you are **still seeing the issue**, please install debug branch using command
```
pip install git+https://github.com/ganeshrn/ansible.git@network_cli_logs --upgrade
```
The debug logs will be available in the file mentioned with `log_path` in ansible configuration file
Thank you. I applied the changes, but see no difference. I have the
ANSIBLE_DEBUG=True set and exported and ran the command with -vvvv. Here is
the relevant output.

2018-01-30 08:50:52,669 paramiko.transport Authentication (password)
successful!
2018-01-30 08:50:52,695 paramiko.transport [chan 0] Max packet in: 32768
bytes
2018-01-30 08:50:52,700 paramiko.transport [chan 0] Max packet out: 4096
bytes
2018-01-30 08:50:52,700 paramiko.transport Secsh channel 0 opened.
2018-01-30 08:50:52,708 paramiko.transport [chan 0] Sesch channel 0 request
ok
2018-01-30 08:50:52,714 paramiko.transport [chan 0] Sesch channel 0 request
ok
2018-01-30 08:50:52,721 p=66007 u=bklotz |   66016 1517331052.72150:
Loading TerminalModule 'ios' from
/Users/bklotz/ansible/lib/ansible/plugins/terminal/ios.py
2018-01-30 08:50:52,728 p=66007 u=bklotz |   66016 1517331052.72792:
Loading Cliconf 'ios' from
/Users/bklotz/ansible/lib/ansible/plugins/cliconf/ios.py
2018-01-30 08:51:13,360 p=66007 u=bklotz |  shutdown complete
2018-01-30 08:51:13,419 paramiko.transport EOF in transport thread
2018-01-30 08:51:14,017 p=65996 u=bklotz |  The full traceback is:
Traceback (most recent call last):
  File "/Users/bklotz/ansible/bin/ansible-connection", line 70, in start
    self.connection._connect()
  File
"/Users/bklotz/ansible/lib/ansible/plugins/connection/network_cli.py", line
323, in _connect
    self._terminal.on_become(passwd=auth_pass)
  File "/Users/bklotz/ansible/lib/ansible/plugins/terminal/ios.py", line
75, in on_become
    prompt = self._get_prompt()
  File "/Users/bklotz/ansible/lib/ansible/plugins/terminal/__init__.py",
line 74, in _get_prompt
    self._exec_cli_command(b'')
  File "/Users/bklotz/ansible/lib/ansible/plugins/terminal/__init__.py",
line 64, in _exec_cli_command
    return self._connection.exec_command(cmd)
  File
"/Users/bklotz/ansible/lib/ansible/plugins/connection/network_cli.py", line
245, in exec_command
    return self.send(command=cmd)
  File
"/Users/bklotz/ansible/lib/ansible/plugins/connection/network_cli.py", line
421, in send
    raise AnsibleConnectionFailure("timeout trying to send command: %s" %
command.strip())
AnsibleConnectionFailure: timeout trying to send command:

2018-01-30 08:51:14,018 p=65996 u=bklotz |   66006 1517331074.01865: done
running TaskExecutor() for NKNAWHQNWA01AX01/TASK: ios_system
[784f438a-ad50-1751-2ef9-000000000022]
2018-01-30 08:51:14,019 p=65996 u=bklotz |   66006 1517331074.01936:
sending task result for task 784f438a-ad50-1751-2ef9-000000000022
2018-01-30 08:51:14,022 p=65996 u=bklotz |   66006 1517331074.02259: done
sending task result for task 784f438a-ad50-1751-2ef9-000000000022
2018-01-30 08:51:14,023 p=65996 u=bklotz |   66006 1517331074.02322: WORKER
PROCESS EXITING
2018-01-30 08:51:14,029 p=65996 u=bklotz |   65996 1517331074.02941:
marking NKNAWHQNWA01AX01 as failed
2018-01-30 08:51:14,030 p=65996 u=bklotz |   65996 1517331074.02992:
marking host NKNAWHQNWA01AX01 failed, current state: HOST STATE: block=2,
task=1, rescue=0, always=0, run_state=ITERATING_TASKS,
fail_state=FAILED_NONE, pending_setup=False, tasks child state? (None),
rescue child state? (None), always child state? (None), did rescue? False,
did start at task? False
2018-01-30 08:51:14,034 p=65996 u=bklotz |   65996 1517331074.03028: ^
failed state is now: HOST STATE: block=2, task=1, rescue=0, always=0,
run_state=ITERATING_COMPLETE, fail_state=FAILED_TASKS, pending_setup=False,
tasks child state? (None), rescue child state? (None), always child state?
(None), did rescue? False, did start at task? False
2018-01-30 08:51:14,034 p=65996 u=bklotz |   65996 1517331074.03457:
getting the next task for host NKNAWHQNWA01AX01
2018-01-30 08:51:14,034 p=65996 u=bklotz |   65996 1517331074.03491: host
NKNAWHQNWA01AX01 is done iterating, returning
2018-01-30 08:51:14,038 p=65996 u=bklotz |  fatal: [NKNAWHQNWA01AX01]:
FAILED! => {
    "failed": true,
    "msg": "timeout trying to send command: "
}
2018-01-30 08:51:14,039 p=65996 u=bklotz |   65996 1517331074.03898: no
more pending results, returning what we have
2018-01-30 08:51:14,039 p=65996 u=bklotz |   65996 1517331074.03935:
results queue empty
2018-01-30 08:51:14,041 p=65996 u=bklotz |   65996 1517331074.03996:
checking for any_errors_fatal
2018-01-30 08:51:14,043 p=65996 u=bklotz |   65996 1517331074.04299: done
checking for any_errors_fatal
2018-01-30 08:51:14,043 p=65996 u=bklotz |   65996 1517331074.04323:
checking for max_fail_percentage
2018-01-30 08:51:14,043 p=65996 u=bklotz |   65996 1517331074.04349: done
checking for max_fail_percentage
2018-01-30 08:51:14,043 p=65996 u=bklotz |   65996 1517331074.04369:
checking to see if all hosts have failed and the running result is not ok
2018-01-30 08:51:14,043 p=65996 u=bklotz |   65996 1517331074.04387: done
checking to see if all hosts have failed
2018-01-30 08:51:14,044 p=65996 u=bklotz |   65996 1517331074.04403:
getting the remaining hosts for this loop
2018-01-30 08:51:14,044 p=65996 u=bklotz |   65996 1517331074.04429: done
getting the remaining hosts for this loop
2018-01-30 08:51:14,044 p=65996 u=bklotz |   65996 1517331074.04457:
building list of next tasks for hosts
2018-01-30 08:51:14,044 p=65996 u=bklotz |   65996 1517331074.04476:
getting the next task for host NKNAWHQNWA01AX01
2018-01-30 08:51:14,045 p=65996 u=bklotz |   65996 1517331074.04496: host
NKNAWHQNWA01AX01 is done iterating, returning
2018-01-30 08:51:14,057 p=65996 u=bklotz |   65996 1517331074.04514: done
building task lists
2018-01-30 08:51:14,058 p=65996 u=bklotz |   65996 1517331074.05799:
counting tasks in each state of execution
2018-01-30 08:51:14,058 p=65996 u=bklotz |   65996 1517331074.05823: done
counting tasks in each state of execution:
num_setups: 0
num_tasks: 0
num_rescue: 0
num_always: 0
2018-01-30 08:51:14,058 p=65996 u=bklotz |   65996 1517331074.05857: all
hosts are done, so returning None's for all hosts
2018-01-30 08:51:14,058 p=65996 u=bklotz |   65996 1517331074.05876: done
queuing things up, now waiting for results queue to drain
2018-01-30 08:51:14,059 p=65996 u=bklotz |   65996 1517331074.05893:
results queue empty
2018-01-30 08:51:14,059 p=65996 u=bklotz |   65996 1517331074.05907:
checking for any_errors_fatal
2018-01-30 08:51:14,059 p=65996 u=bklotz |   65996 1517331074.05921: done
checking for any_errors_fatal
2018-01-30 08:51:14,059 p=65996 u=bklotz |   65996 1517331074.05934:
checking for max_fail_percentage
2018-01-30 08:51:14,059 p=65996 u=bklotz |   65996 1517331074.05948: done
checking for max_fail_percentage
2018-01-30 08:51:14,059 p=65996 u=bklotz |   65996 1517331074.05963:
checking to see if all hosts have failed and the running result is not ok
2018-01-30 08:51:14,059 p=65996 u=bklotz |   65996 1517331074.05979: done
checking to see if all hosts have failed
2018-01-30 08:51:14,060 p=65996 u=bklotz |   65996 1517331074.06004:
getting the next task for host NKNAWHQNWA01AX01
2018-01-30 08:51:14,060 p=65996 u=bklotz |   65996 1517331074.06023: host
NKNAWHQNWA01AX01 is done iterating, returning
2018-01-30 08:51:14,060 p=65996 u=bklotz |   65996 1517331074.06041:
running handlers
2018-01-30 08:51:14,064 p=65996 u=bklotz |  to retry, use: --limit
@/Users/bklotz/Box Sync/Python/ansible-northwoods/testing.retry

2018-01-30 08:51:14,065 p=65996 u=bklotz |  PLAY RECAP
**********************************************************************************************************
2018-01-30 08:51:14,065 p=65996 u=bklotz |  NKNAWHQNWA01AX01           :
ok=0    changed=0    unreachable=0    failed=1
2018-01-30 08:51:14,065 p=65996 u=bklotz |   65996 1517331074.06559:
RUNNING CLEANUP


On Tue, Jan 30, 2018 at 8:52 AM Ganesh Nalawade <notifications@github.com>
wrote:

> I have pushed a branch with debug logs. If you are *still seeing the
> issue*, please install debug branch using command
>
> pip install git+https://github.com/ganeshrn/ansible.git@network_cli_logs --upgrade
>
> The debug logs will be available in the file mentioned in log_path
> configuration file
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/ansible/ansible/issues/33993#issuecomment-361657587>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AbFU9hfVBP3g8C9G4O4apeMdJlbD2pcjks5tP0irgaJpZM4RFrMW>
> .
>
-- 
"Have you tried turning it off and on again?"

Here is the output with the "network_cli_logs" installed.

2018-01-30 08:56:58,591 paramiko.transport Authentication (password)
successful!
2018-01-30 08:56:58,617 paramiko.transport [chan 0] Max packet in: 32768
bytes
2018-01-30 08:56:58,620 paramiko.transport [chan 0] Max packet out: 4096
bytes
2018-01-30 08:56:58,621 paramiko.transport Secsh channel 0 opened.
2018-01-30 08:56:58,629 paramiko.transport [chan 0] Sesch channel 0 request
ok
2018-01-30 08:56:58,640 paramiko.transport [chan 0] Sesch channel 0 request
ok
2018-01-30 08:56:58,643 p=66824 u=bklotz |   66831 1517331418.64353:
Loading TerminalModule 'ios' from
/Users/bklotz/ansible/lib/ansible/plugins/terminal/ios.py
2018-01-30 08:56:58,646 p=66824 u=bklotz |   66831 1517331418.64634:
Loading Cliconf 'ios' from
/Users/bklotz/ansible/lib/ansible/plugins/cliconf/ios.py
2018-01-30 08:57:19,224 p=66824 u=bklotz |  shutdown complete
2018-01-30 08:57:19,225 paramiko.transport EOF in transport thread
2018-01-30 08:57:19,897 p=66813 u=bklotz |  The full traceback is:
Traceback (most recent call last):
  File "/Users/bklotz/ansible/bin/ansible-connection", line 70, in start
    self.connection._connect()
  File
"/Users/bklotz/ansible/lib/ansible/plugins/connection/network_cli.py", line
323, in _connect
    self._terminal.on_become(passwd=auth_pass)
  File "/Users/bklotz/ansible/lib/ansible/plugins/terminal/ios.py", line
75, in on_become
    prompt = self._get_prompt()
  File "/Users/bklotz/ansible/lib/ansible/plugins/terminal/__init__.py",
line 74, in _get_prompt
    self._exec_cli_command(b'')
  File "/Users/bklotz/ansible/lib/ansible/plugins/terminal/__init__.py",
line 64, in _exec_cli_command
    return self._connection.exec_command(cmd)
  File
"/Users/bklotz/ansible/lib/ansible/plugins/connection/network_cli.py", line
245, in exec_command
    return self.send(command=cmd)
  File
"/Users/bklotz/ansible/lib/ansible/plugins/connection/network_cli.py", line
421, in send
    raise AnsibleConnectionFailure("timeout trying to send command: %s" %
command.strip())
AnsibleConnectionFailure: timeout trying to send command:

2018-01-30 08:57:19,899 p=66813 u=bklotz |   66821 1517331439.89910: done
running TaskExecutor() for NKNAWHQNWA01AX01/TASK: ios_system
[784f438a-ad50-b30a-cc5a-000000000022]
2018-01-30 08:57:19,900 p=66813 u=bklotz |   66821 1517331439.89985:
sending task result for task 784f438a-ad50-b30a-cc5a-000000000022
2018-01-30 08:57:19,912 p=66813 u=bklotz |   66821 1517331439.91240: done
sending task result for task 784f438a-ad50-b30a-cc5a-000000000022
2018-01-30 08:57:19,913 p=66813 u=bklotz |   66821 1517331439.91301: WORKER
PROCESS EXITING
2018-01-30 08:57:19,926 p=66813 u=bklotz |   66813 1517331439.92600:
marking NKNAWHQNWA01AX01 as failed
2018-01-30 08:57:19,927 p=66813 u=bklotz |   66813 1517331439.92663:
marking host NKNAWHQNWA01AX01 failed, current state: HOST STATE: block=2,
task=1, rescue=0, always=0, run_state=ITERATING_TASKS,
fail_state=FAILED_NONE, pending_setup=False, tasks child state? (None),
rescue child state? (None), always child state? (None), did rescue? False,
did start at task? False
2018-01-30 08:57:19,928 p=66813 u=bklotz |   66813 1517331439.92787: ^
failed state is now: HOST STATE: block=2, task=1, rescue=0, always=0,
run_state=ITERATING_COMPLETE, fail_state=FAILED_TASKS, pending_setup=False,
tasks child state? (None), rescue child state? (None), always child state?
(None), did rescue? False, did start at task? False
2018-01-30 08:57:19,928 p=66813 u=bklotz |   66813 1517331439.92852:
getting the next task for host NKNAWHQNWA01AX01
2018-01-30 08:57:19,928 p=66813 u=bklotz |   66813 1517331439.92876: host
NKNAWHQNWA01AX01 is done iterating, returning
2018-01-30 08:57:19,934 p=66813 u=bklotz |  fatal: [NKNAWHQNWA01AX01]:
FAILED! => {
    "failed": true,
    "msg": "timeout trying to send command: "
}
2018-01-30 08:57:19,938 p=66813 u=bklotz |   66813 1517331439.93526: no
more pending results, returning what we have
2018-01-30 08:57:19,939 p=66813 u=bklotz |   66813 1517331439.93917:
results queue empty
2018-01-30 08:57:19,939 p=66813 u=bklotz |   66813 1517331439.93973:
checking for any_errors_fatal
2018-01-30 08:57:19,940 p=66813 u=bklotz |   66813 1517331439.93995: done
checking for any_errors_fatal
2018-01-30 08:57:19,940 p=66813 u=bklotz |   66813 1517331439.94012:
checking for max_fail_percentage
2018-01-30 08:57:19,940 p=66813 u=bklotz |   66813 1517331439.94029: done
checking for max_fail_percentage
2018-01-30 08:57:19,940 p=66813 u=bklotz |   66813 1517331439.94053:
checking to see if all hosts have failed and the running result is not ok
2018-01-30 08:57:19,940 p=66813 u=bklotz |   66813 1517331439.94073: done
checking to see if all hosts have failed
2018-01-30 08:57:19,941 p=66813 u=bklotz |   66813 1517331439.94093:
getting the remaining hosts for this loop
2018-01-30 08:57:19,941 p=66813 u=bklotz |   66813 1517331439.94130: done
getting the remaining hosts for this loop
2018-01-30 08:57:19,941 p=66813 u=bklotz |   66813 1517331439.94171:
building list of next tasks for hosts
2018-01-30 08:57:19,942 p=66813 u=bklotz |   66813 1517331439.94192:
getting the next task for host NKNAWHQNWA01AX01
2018-01-30 08:57:19,942 p=66813 u=bklotz |   66813 1517331439.94210: host
NKNAWHQNWA01AX01 is done iterating, returning
2018-01-30 08:57:19,942 p=66813 u=bklotz |   66813 1517331439.94227: done
building task lists
2018-01-30 08:57:19,942 p=66813 u=bklotz |   66813 1517331439.94243:
counting tasks in each state of execution
2018-01-30 08:57:19,942 p=66813 u=bklotz |   66813 1517331439.94269: done
counting tasks in each state of execution:
num_setups: 0
num_tasks: 0
num_rescue: 0
num_always: 0
2018-01-30 08:57:19,943 p=66813 u=bklotz |   66813 1517331439.94290: all
hosts are done, so returning None's for all hosts
2018-01-30 08:57:19,943 p=66813 u=bklotz |   66813 1517331439.94309: done
queuing things up, now waiting for results queue to drain
2018-01-30 08:57:19,943 p=66813 u=bklotz |   66813 1517331439.94328:
results queue empty
2018-01-30 08:57:19,963 p=66813 u=bklotz |   66813 1517331439.94345:
checking for any_errors_fatal
2018-01-30 08:57:19,963 p=66813 u=bklotz |   66813 1517331439.96365: done
checking for any_errors_fatal
2018-01-30 08:57:19,964 p=66813 u=bklotz |   66813 1517331439.96413:
checking for max_fail_percentage
2018-01-30 08:57:19,964 p=66813 u=bklotz |   66813 1517331439.96439: done
checking for max_fail_percentage
2018-01-30 08:57:19,964 p=66813 u=bklotz |   66813 1517331439.96457:
checking to see if all hosts have failed and the running result is not ok
2018-01-30 08:57:19,964 p=66813 u=bklotz |   66813 1517331439.96483: done
checking to see if all hosts have failed
2018-01-30 08:57:19,965 p=66813 u=bklotz |   66813 1517331439.96524:
getting the next task for host NKNAWHQNWA01AX01
2018-01-30 08:57:19,965 p=66813 u=bklotz |   66813 1517331439.96583: host
NKNAWHQNWA01AX01 is done iterating, returning
2018-01-30 08:57:19,966 p=66813 u=bklotz |   66813 1517331439.96607:
running handlers
2018-01-30 08:57:19,968 p=66813 u=bklotz |  to retry, use: --limit
@/Users/bklotz/Box Sync/Python/ansible-northwoods/testing.retry

2018-01-30 08:57:19,968 p=66813 u=bklotz |  PLAY RECAP
**********************************************************************************************************
2018-01-30 08:57:19,969 p=66813 u=bklotz |  NKNAWHQNWA01AX01           :
ok=0    changed=0    unreachable=0    failed=1
2018-01-30 08:57:19,969 p=66813 u=bklotz |   66813 1517331439.96938:
RUNNING CLEANUP


On Tue, Jan 30, 2018 at 8:55 AM Brian Klotz <brainclots@gmail.com> wrote:

> Thank you. I applied the changes, but see no difference. I have the
> ANSIBLE_DEBUG=True set and exported and ran the command with -vvvv. Here is
> the relevant output.
>
> 2018-01-30 08:50:52,669 paramiko.transport Authentication (password)
> successful!
> 2018-01-30 08:50:52,695 paramiko.transport [chan 0] Max packet in: 32768
> bytes
> 2018-01-30 08:50:52,700 paramiko.transport [chan 0] Max packet out: 4096
> bytes
> 2018-01-30 08:50:52,700 paramiko.transport Secsh channel 0 opened.
> 2018-01-30 08:50:52,708 paramiko.transport [chan 0] Sesch channel 0
> request ok
> 2018-01-30 08:50:52,714 paramiko.transport [chan 0] Sesch channel 0
> request ok
> 2018-01-30 08:50:52,721 p=66007 u=bklotz |   66016 1517331052.72150:
> Loading TerminalModule 'ios' from
> /Users/bklotz/ansible/lib/ansible/plugins/terminal/ios.py
> 2018-01-30 08:50:52,728 p=66007 u=bklotz |   66016 1517331052.72792:
> Loading Cliconf 'ios' from
> /Users/bklotz/ansible/lib/ansible/plugins/cliconf/ios.py
> 2018-01-30 08:51:13,360 p=66007 u=bklotz |  shutdown complete
> 2018-01-30 08:51:13,419 paramiko.transport EOF in transport thread
> 2018-01-30 08:51:14,017 p=65996 u=bklotz |  The full traceback is:
> Traceback (most recent call last):
>   File "/Users/bklotz/ansible/bin/ansible-connection", line 70, in start
>     self.connection._connect()
>   File
> "/Users/bklotz/ansible/lib/ansible/plugins/connection/network_cli.py", line
> 323, in _connect
>     self._terminal.on_become(passwd=auth_pass)
>   File "/Users/bklotz/ansible/lib/ansible/plugins/terminal/ios.py", line
> 75, in on_become
>     prompt = self._get_prompt()
>   File "/Users/bklotz/ansible/lib/ansible/plugins/terminal/__init__.py",
> line 74, in _get_prompt
>     self._exec_cli_command(b'')
>   File "/Users/bklotz/ansible/lib/ansible/plugins/terminal/__init__.py",
> line 64, in _exec_cli_command
>     return self._connection.exec_command(cmd)
>   File
> "/Users/bklotz/ansible/lib/ansible/plugins/connection/network_cli.py", line
> 245, in exec_command
>     return self.send(command=cmd)
>   File
> "/Users/bklotz/ansible/lib/ansible/plugins/connection/network_cli.py", line
> 421, in send
>     raise AnsibleConnectionFailure("timeout trying to send command: %s" %
> command.strip())
> AnsibleConnectionFailure: timeout trying to send command:
>
> 2018-01-30 08:51:14,018 p=65996 u=bklotz |   66006 1517331074.01865: done
> running TaskExecutor() for NKNAWHQNWA01AX01/TASK: ios_system
> [784f438a-ad50-1751-2ef9-000000000022]
> 2018-01-30 08:51:14,019 p=65996 u=bklotz |   66006 1517331074.01936:
> sending task result for task 784f438a-ad50-1751-2ef9-000000000022
> 2018-01-30 08:51:14,022 p=65996 u=bklotz |   66006 1517331074.02259: done
> sending task result for task 784f438a-ad50-1751-2ef9-000000000022
> 2018-01-30 08:51:14,023 p=65996 u=bklotz |   66006 1517331074.02322:
> WORKER PROCESS EXITING
> 2018-01-30 08:51:14,029 p=65996 u=bklotz |   65996 1517331074.02941:
> marking NKNAWHQNWA01AX01 as failed
> 2018-01-30 08:51:14,030 p=65996 u=bklotz |   65996 1517331074.02992:
> marking host NKNAWHQNWA01AX01 failed, current state: HOST STATE: block=2,
> task=1, rescue=0, always=0, run_state=ITERATING_TASKS,
> fail_state=FAILED_NONE, pending_setup=False, tasks child state? (None),
> rescue child state? (None), always child state? (None), did rescue? False,
> did start at task? False
> 2018-01-30 08:51:14,034 p=65996 u=bklotz |   65996 1517331074.03028: ^
> failed state is now: HOST STATE: block=2, task=1, rescue=0, always=0,
> run_state=ITERATING_COMPLETE, fail_state=FAILED_TASKS, pending_setup=False,
> tasks child state? (None), rescue child state? (None), always child state?
> (None), did rescue? False, did start at task? False
> 2018-01-30 08:51:14,034 p=65996 u=bklotz |   65996 1517331074.03457:
> getting the next task for host NKNAWHQNWA01AX01
> 2018-01-30 08:51:14,034 p=65996 u=bklotz |   65996 1517331074.03491: host
> NKNAWHQNWA01AX01 is done iterating, returning
> 2018-01-30 08:51:14,038 p=65996 u=bklotz |  fatal: [NKNAWHQNWA01AX01]:
> FAILED! => {
>     "failed": true,
>     "msg": "timeout trying to send command: "
> }
> 2018-01-30 08:51:14,039 p=65996 u=bklotz |   65996 1517331074.03898: no
> more pending results, returning what we have
> 2018-01-30 08:51:14,039 p=65996 u=bklotz |   65996 1517331074.03935:
> results queue empty
> 2018-01-30 08:51:14,041 p=65996 u=bklotz |   65996 1517331074.03996:
> checking for any_errors_fatal
> 2018-01-30 08:51:14,043 p=65996 u=bklotz |   65996 1517331074.04299: done
> checking for any_errors_fatal
> 2018-01-30 08:51:14,043 p=65996 u=bklotz |   65996 1517331074.04323:
> checking for max_fail_percentage
> 2018-01-30 08:51:14,043 p=65996 u=bklotz |   65996 1517331074.04349: done
> checking for max_fail_percentage
> 2018-01-30 08:51:14,043 p=65996 u=bklotz |   65996 1517331074.04369:
> checking to see if all hosts have failed and the running result is not ok
> 2018-01-30 08:51:14,043 p=65996 u=bklotz |   65996 1517331074.04387: done
> checking to see if all hosts have failed
> 2018-01-30 08:51:14,044 p=65996 u=bklotz |   65996 1517331074.04403:
> getting the remaining hosts for this loop
> 2018-01-30 08:51:14,044 p=65996 u=bklotz |   65996 1517331074.04429: done
> getting the remaining hosts for this loop
> 2018-01-30 08:51:14,044 p=65996 u=bklotz |   65996 1517331074.04457:
> building list of next tasks for hosts
> 2018-01-30 08:51:14,044 p=65996 u=bklotz |   65996 1517331074.04476:
> getting the next task for host NKNAWHQNWA01AX01
> 2018-01-30 08:51:14,045 p=65996 u=bklotz |   65996 1517331074.04496: host
> NKNAWHQNWA01AX01 is done iterating, returning
> 2018-01-30 08:51:14,057 p=65996 u=bklotz |   65996 1517331074.04514: done
> building task lists
> 2018-01-30 08:51:14,058 p=65996 u=bklotz |   65996 1517331074.05799:
> counting tasks in each state of execution
> 2018-01-30 08:51:14,058 p=65996 u=bklotz |   65996 1517331074.05823: done
> counting tasks in each state of execution:
> num_setups: 0
> num_tasks: 0
> num_rescue: 0
> num_always: 0
> 2018-01-30 08:51:14,058 p=65996 u=bklotz |   65996 1517331074.05857: all
> hosts are done, so returning None's for all hosts
> 2018-01-30 08:51:14,058 p=65996 u=bklotz |   65996 1517331074.05876: done
> queuing things up, now waiting for results queue to drain
> 2018-01-30 08:51:14,059 p=65996 u=bklotz |   65996 1517331074.05893:
> results queue empty
> 2018-01-30 08:51:14,059 p=65996 u=bklotz |   65996 1517331074.05907:
> checking for any_errors_fatal
> 2018-01-30 08:51:14,059 p=65996 u=bklotz |   65996 1517331074.05921: done
> checking for any_errors_fatal
> 2018-01-30 08:51:14,059 p=65996 u=bklotz |   65996 1517331074.05934:
> checking for max_fail_percentage
> 2018-01-30 08:51:14,059 p=65996 u=bklotz |   65996 1517331074.05948: done
> checking for max_fail_percentage
> 2018-01-30 08:51:14,059 p=65996 u=bklotz |   65996 1517331074.05963:
> checking to see if all hosts have failed and the running result is not ok
> 2018-01-30 08:51:14,059 p=65996 u=bklotz |   65996 1517331074.05979: done
> checking to see if all hosts have failed
> 2018-01-30 08:51:14,060 p=65996 u=bklotz |   65996 1517331074.06004:
> getting the next task for host NKNAWHQNWA01AX01
> 2018-01-30 08:51:14,060 p=65996 u=bklotz |   65996 1517331074.06023: host
> NKNAWHQNWA01AX01 is done iterating, returning
> 2018-01-30 08:51:14,060 p=65996 u=bklotz |   65996 1517331074.06041:
> running handlers
> 2018-01-30 08:51:14,064 p=65996 u=bklotz |  to retry, use: --limit
> @/Users/bklotz/Box Sync/Python/ansible-northwoods/testing.retry
>
> 2018-01-30 08:51:14,065 p=65996 u=bklotz |  PLAY RECAP
> **********************************************************************************************************
> 2018-01-30 08:51:14,065 p=65996 u=bklotz |  NKNAWHQNWA01AX01           :
> ok=0    changed=0    unreachable=0    failed=1
> 2018-01-30 08:51:14,065 p=65996 u=bklotz |   65996 1517331074.06559:
> RUNNING CLEANUP
>
>
> On Tue, Jan 30, 2018 at 8:52 AM Ganesh Nalawade <notifications@github.com>
> wrote:
>
>> I have pushed a branch with debug logs. If you are *still seeing the
>> issue*, please install debug branch using command
>>
>> pip install git+https://github.com/ganeshrn/ansible.git@network_cli_logs --upgrade
>>
>> The debug logs will be available in the file mentioned in log_path
>> configuration file
>>
>> —
>> You are receiving this because you were mentioned.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/ansible/ansible/issues/33993#issuecomment-361657587>,
>> or mute the thread
>> <https://github.com/notifications/unsubscribe-auth/AbFU9hfVBP3g8C9G4O4apeMdJlbD2pcjks5tP0irgaJpZM4RFrMW>
>> .
>>
> --
> "Have you tried turning it off and on again?"
>
-- 
"Have you tried turning it off and on again?"

For some reason, the lines in `terminal_prompt_issue` branch do not match with the stack trace it produces in the logs.

Stacktrace lines
```
  File "/Users/bklotz/ansible/lib/ansible/plugins/terminal/ios.py", line
75, in on_become
    prompt = self._get_prompt()
  File "/Users/bklotz/ansible/lib/ansible/plugins/terminal/__init__.py",
line 74, in _get_prompt
    self._exec_cli_command(b'')
  File "/Users/bklotz/ansible/lib/ansible/plugins/terminal/__init__.py",
line 64, in _exec_cli_command
    return self._connection.exec_command(cmd)
```
different from the one in PR branch
https://github.com/ganeshrn/ansible/blob/terminal_prompt_issue/lib/ansible/plugins/terminal/__init__.py#L74

It seems you are still running Ansible version that does not have PR branch applied.
One more way to test the PR branch is: 
Open new terminal to checkout the source with PR branch and set up your environment to run from the checkout
```
$ git clone -b terminal_prompt_issue git@github.com:ganeshrn/ansible.git
$ cd ansible
$ source hacking/env-setup
$ which ansible
```
The output of the last command should point to `<present-working-directory-path>/bin/ansible`
After that rerun the test case. 




Missing rights.

[bklotz ~]$ git clone -b terminal_prompt_issue git@github.com:
ganeshrn/ansible.git
Cloning into 'ansible'...
Warning: Permanently added 'github.com,192.30.255.113' (RSA) to the list of
known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

On Tue, Jan 30, 2018 at 10:45 AM Ganesh Nalawade <notifications@github.com>
wrote:

> One more way to test the PR branch is:
> Open new terminal and clone the source with PR branch
>
> $ git clone -b terminal_prompt_issue git@github.com:ganeshrn/ansible.git
> $ cd ansible
> $ source hacking/env-setup
> $ which ansible
>
> The output of the last command should point to
> <present-working-directory-path>/bin/ansible
> After that rerun the test case.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/ansible/ansible/issues/33993#issuecomment-361693479>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AbFU9i5ppAK1R1yVkuZtiSIlgT1xD_2Zks5tP2NSgaJpZM4RFrMW>
> .
>
-- 
"Have you tried turning it off and on again?"

I believe the right syntax is:
`git clone -b terminal_prompt_issue https://github.com/ganeshrn/ansible.git`
Thank you, John. New errors below:

2018-01-30 13:19:37,202 paramiko.transport Authentication (password)
successful!
2018-01-30 13:19:37,213 paramiko.transport [chan 0] Max packet in: 32768
bytes
2018-01-30 13:19:37,220 paramiko.transport [chan 0] Max packet out: 4096
bytes
2018-01-30 13:19:37,220 paramiko.transport Secsh channel 0 opened.
2018-01-30 13:19:37,227 paramiko.transport [chan 0] Sesch channel 0 request
ok
2018-01-30 13:19:37,232 paramiko.transport [chan 0] Sesch channel 0 request
ok
2018-01-30 13:19:37,242 p=84828 u=bklotz |   84835 1517347177.24218:
Loading TerminalModule 'ios' from
/Users/bklotz/ansible/lib/ansible/plugins/terminal/ios.py
2018-01-30 13:19:37,258 p=84828 u=bklotz |   84835 1517347177.25826:
Loading Cliconf 'ios' from
/Users/bklotz/ansible/lib/ansible/plugins/cliconf/ios.py
2018-01-30 13:19:47,548 p=84828 u=bklotz |  shutdown complete
2018-01-30 13:19:47,561 paramiko.transport EOF in transport thread
2018-01-30 13:19:48,204 p=84821 u=bklotz |  The full traceback is:
Traceback (most recent call last):
  File "/Users/bklotz/ansible/bin/ansible-connection", line 70, in start
    self.connection._connect()
  File
"/Users/bklotz/ansible/lib/ansible/plugins/connection/network_cli.py", line
323, in _connect
    self._terminal.on_become(passwd=auth_pass)
  File "/Users/bklotz/ansible/lib/ansible/plugins/terminal/ios.py", line
76, in on_become
    raise AnsibleConnectionFailure('unable to elevate privilege to enable
mode, at prompt [%s]' % prompt)
AnsibleConnectionFailure: unable to elevate privilege to enable mode, at
prompt [None]


On Tue, Jan 30, 2018 at 12:15 PM John R Barker <notifications@github.com>
wrote:

> I believe the right syntax is:
> git clone -b terminal_prompt_issue https://github.com/ganeshrn/ansible.git
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/ansible/ansible/issues/33993#issuecomment-361720237>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AbFU9t8jf_9598WZMIyTBSJ0N3oU32gTks5tP3hbgaJpZM4RFrMW>
> .
>
-- 
"Have you tried turning it off and on again?"

Now with the patch, `\r` is not executed on the remote host to get the current prompt which is indicated by the error `unable to elevate privilege to enable mode, at prompt [None]`. 

This error suggests there might be an issue with `become_pass` not provided correctly, to identify the root cause please rerun with debug branch in new terminal
```
$ git clone -b network_cli_logs  https://github.com/ganeshrn/ansible.git
$ cd ansible
$ source hacking/env-setup
$ which ansible
```
After running grep for `_connect become_pass` in the log file and check if become_password is set correctly. If possible please share the logs using gist (https://gist.github.com) after removing environment specific information from the log file.


https://gist.github.com/brainclots/e212e7fe91245c0fb0bd5c4dab06060a

Check out line 74. I note that it is looking for a prompt of "password:",
however the device prompt is actually "Password:". Could that be the issue?

Brian

On Tue, Jan 30, 2018 at 10:49 PM Ganesh Nalawade <notifications@github.com>
wrote:

> Now with the patch, \r is not executed on the remote host to get the
> current prompt which is indicated by the error unable to elevate
> privilege to enable mode, at prompt [None].
>
> This error suggests there might be an issue with become_pass not provided
> correctly, to identify the root cause please rerun with debug branch in new
> terminal
>
> $ git clone -b network_cli_logs  https://github.com/ganeshrn/ansible.git
>
>
> $ cd ansible
> $ source hacking/env-setup
> $ which ansible
>
> After running grep for _connect become_pass in the log file and check if
> become_password is set correctly. If possible please share the log file
> using gist (https://gist.github.com) after removing environment specific
> information from the log file.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/ansible/ansible/issues/33993#issuecomment-361840300>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AbFU9j4PW5ZPZ1lS1cbf2i0ZfGsCPVH5ks5tQA0CgaJpZM4RFrMW>
> .
>
-- 
"Have you tried turning it off and on again?"

@brainclots As the regex match is case insensitive [here](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/connection/network_cli.py#L450)
that shouldn't be an issue.

I am suspecting the space at the end of the line in the regex [here](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/terminal/ios.py#L67) as the culprit.
With current regex
```
$ python
>>> import re
>>> m = re.compile("[\r\n]password: $", re.I)
>>> match = m.search('enable\r\nPassword:')
>>> print(match)
None
```

With updated regex
```
>>> m = re.compile("[\r\n]password: ?$", re.I)
>>> match = m.search('enable\r\nPassword:')
>>> print(match)
<_sre.SRE_Match object at 0x1026cdb28>
>>> match.group()
'\nPassword:'
```

I have rebased devel branch and updated the above password prompt regex along with debugging logs in a new branch `ios_password_regex_debug`. Please retest with this branch.
```
$ git clone -b ios_password_regex_debug  https://github.com/ganeshrn/ansible.git
$ cd ansible
$ source hacking/env-setup
$ which ansible
```

Here is the latest test:
https://gist.github.com/brainclots/af393c8538227b6f2f854ddf20c016f1

On Fri, Feb 2, 2018 at 12:13 AM Ganesh Nalawade <notifications@github.com>
wrote:

> @brainclots <https://github.com/brainclots> As the regex match is case
> insensitive here
> <https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/connection/network_cli.py#L450>
> that shouldn't be an issue.
>
> I am suspecting the space at the end of the line in the regex here
> <https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/terminal/ios.py#L67>
> as the culprit.
> With current regex
>
> $ python
> >>> import re
> >>> m = re.compile("[\r\n]password: $", re.I)
> >>> match = m.search('enable\r\nPassword:')
> >>> print(match)
> None
>
> With updated regex
>
> >>> m = re.compile("[\r\n]password: ?$", re.I)
> >>> match = m.search('enable\r\nPassword:')
> >>> print(match)
> <_sre.SRE_Match object at 0x1026cdb28>
> >>> match.group()
> '\nPassword:'
>
> I have rebased devel branch and updated the above password prompt regex
> along with debugging logs in a new branch ios_password_regex_debug.
> Please retest with this branch.
>
> $ git clone -b ios_password_regex_debug  https://github.com/ganeshrn/ansible.git
>
>
> $ cd ansible
> $ source hacking/env-setup
> $ which ansible
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/ansible/ansible/issues/33993#issuecomment-362515735>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AbFU9lsadmk87A2TtbA2_lNkf8Wq2Crcks5tQsOXgaJpZM4RFrMW>
> .
>
-- 
"Have you tried turning it off and on again?"

Thank you sharing the logs, with the regex change, password prompt is matched correctly and the answer is sent to the remote host, i am now hoping we are close to fixing this issue.
```
2018-02-02 14:01:03,716 p=21277 u=bklotz |   -- _handle_prompt match --
Password:
2018-02-02 14:01:03,717 p=21277 u=bklotz |   -- _handle_prompt regex -- [\r\n]password: ?$
```

Now it is failing because of the wrong format in debug log (my bad)
```
  File "/Users/bklotz/ansible/lib/ansible/plugins/connection/network_cli.py", line 416, in receive
    display.display("-- check for second prompt --" % command, log_only=True)
TypeError: not all arguments converted during string formatting
```

This log should be 
```
display.display("-- check for second prompt %s --" % command, log_only=True)
```

I have pushed this change in the same branch `ios_password_regex_debug`. Please retest it with updated code.
@brainclots waiting on your retest before closing out this issue.

Resolving patch:
diff --git a/lib/ansible/plugins/terminal/ios.py b/lib/ansible/plugins/terminal/ios.py
--- a/lib/ansible/plugins/terminal/ios.py
+++ b/lib/ansible/plugins/terminal/ios.py
@@ -64,7 +64,7 @@ def on_become(self, passwd=None):
         if passwd:
             # Note: python-3.5 cannot combine u"" and r"" together.  Thus make
             # an r string and use to_text to ensure it's text on both py2 and py3.
-            cmd[u'prompt'] = to_text(r"[\r\n]password: $", errors='surrogate_or_strict')
+            cmd[u'prompt'] = to_text(r"[\r\n]password: ?$", errors='surrogate_or_strict')
             cmd[u'answer'] = passwd
             cmd[u'prompt_retry_check'] = True
         try:


Test patch: