Repository: ansible/ansible
Pull request: Fix prompt mismatch issue for ios
##### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
Fixes #40884 #44463 #46082

~~*  Check if the last matched prompt in the response~~
~~is the same as that of the current prompt. This will ensure
   any misinterpreted prompt that is matched in between response is ignored.~~
~~*  To get around command that will result in the prompt change~~
~~example `configure` add a window count check which ensures
   above check is not passed in such scenario's.~~

*  If the command prompt is matched check if data is
   still pending to be read from the buffer.
*  This fix adds a new timer `buffer_read_timeout`
   which will be triggered after the command prompt
   is matched and data is attempted to be read from the channel.
   If no data is present on the channel the timer will expire and 
   the response we be returned to the calling function. If data is
   present it will continue to process the response.

<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->

##### ISSUE TYPE
<!--- Pick one below and delete the rest -->
- Bugfix Pull Request

##### COMPONENT NAME
<!--- Write the short name of the module, plugin, task or feature below -->
network_cli

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes -->
```paste below
2.7
```

##### ADDITIONAL INFORMATION
<!--- Include additional information to help people understand the change here -->
<!--- A step-by-step reproduction of the problem is helpful if there is no related issue -->

<!--- Paste verbatim command output below, e.g. before and after your change -->
```paste below

```


Problem statement:
ios_config traceback on some devices
<!---
Verify first that your issue/request is not already reported on GitHub.
THIS FORM WILL BE READ BY A MACHINE, COMPLETE ALL SECTIONS AS DESCRIBED.
Also test if the latest release, and devel branch are affected too.
ALWAYS add information AFTER (OUTSIDE) these html comments.
Otherwise it may end up being automatically closed by our bot. -->

##### SUMMARY
<!--- Explain the problem briefly -->
On some devices, ios_config will create a traceback when running the ios_config module. Nothing obviously different between devices that work and devices that do not, same model and some firmware. Sometimes running a play multiple times will result in it working, but usually they continue to fail. Attempted to change the tmp file location via remote_tmp and local_tmp, but still looks to be putting the tmp files into /tmp.

All the devices are configured to auto-enable into priv 15, and confirmed that the credentials do in fact work. The ios_facts module works with no problems on this device, only the ios_config module seems to be having issues.
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
<!--- Insert, BELOW THIS COMMENT, the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path-->
ios_config

##### ANSIBLE VERSION
<!--- Paste, BELOW THIS COMMENT, verbatim output from "ansible --version" between quotes below -->
```
ansible 2.5.3
  config file = /home/pffs/ansible-scripts/ansible.cfg
  configured module search path = [u'/home/pffs/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]
```

##### CONFIGURATION
<!--- If using Ansible 2.4 or above, paste, BELOW THIS COMMENT, the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).-->
```
ANSIBLE_PIPELINING(/home/pffs/ansible-scripts/ansible.cfg) = True
DEFAULT_CALLBACK_WHITELIST(/home/pffs/ansible-scripts/ansible.cfg) = ['profile_tasks', 'timer', 'json_result']
DEFAULT_FORKS(/home/pffs/ansible-scripts/ansible.cfg) = 15
DEFAULT_HOST_LIST(/home/pffs/ansible-scripts/ansible.cfg) = [u'/home/pffs/ansible-scripts/inventory']
DEFAULT_LOCAL_TMP(/home/pffs/ansible-scripts/ansible.cfg) = /home/pffs/.ansible/tmp/ansible-local-8904QiZGq4
DEFAULT_LOG_PATH(/home/pffs/ansible-scripts/ansible.cfg) = /home/pffs/ansible-scripts/log
HOST_KEY_CHECKING(/home/pffs/ansible-scripts/ansible.cfg) = False
PERSISTENT_COMMAND_TIMEOUT(/home/pffs/ansible-scripts/ansible.cfg) = 45
PERSISTENT_CONNECT_TIMEOUT(/home/pffs/ansible-scripts/ansible.cfg) = 90
```
##### OS / ENVIRONMENT
<!--- Mention, BELOW THIS COMMENT, the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.-->
Ubuntu 16.04 connecting to Cisco 891FW running 15.4(3)M5
##### STEPS TO REPRODUCE
<!--- For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used. -->

<!--- Paste example playbooks or commands between quotes below -->
main playbook:
```yaml
---
- hosts: all
  gather_facts: no
  vars_files:
    - ../../tacacs.yaml

  tasks:
    - ios_config:
        lines:
          - logging monitor
```
group_vars/all.yaml:
```yaml
---
ansible_connection: network_cli
ansible_network_os: ios
ansible_become: yes
ansible_become_method: enable
ansible_ssh_common_args: '-o ProxyCommand="ssh -W %h:%p -q bastion"'
```
<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
Completes with change
##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
Traces back:
<!--- Paste verbatim command output between quotes below -->
```
ansible-playbook 2.5.3
  config file = /home/pffs/ansible-scripts/ansible.cfg
  configured module search path = [u'/home/pffs/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]
Using /home/pffs/ansible-scripts/ansible.cfg as config file
setting up inventory plugins
Parsed /home/pffs/ansible-scripts/inventory inventory source with ini plugin
Loading callback plugin default of type stdout, v2.0 from /usr/lib/python2.7/dist-packages/ansible/plugins/callback/default.pyc
Loading callback plugin json_result of type aggregate, v1.1 from /home/pffs/.ansible/plugins/callback/json_result.pyc
Loading callback plugin profile_tasks of type aggregate, v2.0 from /usr/lib/python2.7/dist-packages/ansible/plugins/callback/profile_tasks.pyc
Loading callback plugin timer of type aggregate, v2.0 from /usr/lib/python2.7/dist-packages/ansible/plugins/callback/timer.pyc

PLAYBOOK: test.yaml *****************************************************************************************************************************************************************************************************************************************************
1 plays in test.yaml
Read vars_file '../../tacacs.yaml'
Read vars_file '../../tacacs.yaml'

PLAY [all] **************************************************************************************************************************************************************************************************************************************************************
META: ran handlers
Read vars_file '../../tacacs.yaml'

TASK [ios_config] *******************************************************************************************************************************************************************************************************************************************************
task path: /home/pffs/ansible-scripts/test.yaml:8
Wednesday 30 May 2018  10:31:12 -0400 (0:00:00.714)       0:00:00.714 *********
<10.0.0.1> attempting to start connection
<10.0.0.1> using connection plugin network_cli
<10.0.0.1> local domain socket does not exist, starting it
<10.0.0.1> control socket path is /home/pffs/.ansible/pc/1ac44613c3
<10.0.0.1> <10.0.0.1> ESTABLISH CONNECTION FOR USER: scripts on PORT 22 TO 10.0.0.1
<10.0.0.1> <10.0.0.1> CONFIGURE PROXY COMMAND FOR CONNECTION: ssh -W 10.0.0.1:22 -q bastion
<10.0.0.1> <10.0.0.1> ssh connection done, setting terminal
<10.0.0.1> <10.0.0.1> loaded terminal plugin for network_os ios
<10.0.0.1> <10.0.0.1> loaded cliconf plugin for network_os ios
<10.0.0.1> <10.0.0.1> firing event: on_open_shell()
<10.0.0.1> <10.0.0.1> firing event: on_become
<10.0.0.1> <10.0.0.1> ssh connection has completed successfully
<10.0.0.1> connection to remote device started successfully
<10.0.0.1> local domain socket listeners started successfully
<10.0.0.1>
<10.0.0.1> local domain socket path is /home/pffs/.ansible/pc/1ac44613c3
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/network/__init__.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/network/common/parsing.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/six/__init__.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/basic.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/network/common/config.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/network/common/__init__.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/network/ios/ios.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/network/ios/__init__.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/parsing/convert_bool.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/parsing/__init__.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/_text.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/pycompat24.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/network/common/utils.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/connection.py
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/network/ios/ios_config.py
<10.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: pffs
<10.0.0.1> EXEC /bin/sh -c '/usr/bin/python && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_b_9Rr0/ansible_module_ios_config.py", line 583, in <module>
    main()
  File "/tmp/ansible_b_9Rr0/ansible_module_ios_config.py", line 512, in main
    load_config(module, commands)
  File "/tmp/ansible_b_9Rr0/ansible_modlib.zip/ansible/module_utils/network/ios/ios.py", line 168, in load_config
  File "/tmp/ansible_b_9Rr0/ansible_modlib.zip/ansible/module_utils/connection.py", line 149, in __rpc__
ansible.module_utils.connection.ConnectionError: operation requires privilege escalation

fatal: [dmvpn-1]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_b_9Rr0/ansible_module_ios_config.py\", line 583, in <module>\n    main()\n  File \"/tmp/ansible_b_9Rr0/ansible_module_ios_config.py\", line 512, in main\n    load_config(module, command
s)\n  File \"/tmp/ansible_b_9Rr0/ansible_modlib.zip/ansible/module_utils/network/ios/ios.py\", line 168, in load_config\n  File \"/tmp/ansible_b_9Rr0/ansible_modlib.zip/ansible/module_utils/connection.py\", line 149, in __rpc__\nansible.module_utils.connection.Conn
ectionError: operation requires privilege escalation\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE",
    "rc": 1
}
        to retry, use: --limit @/home/pffs/ansible-scripts/test.retry

PLAY RECAP **************************************************************************************************************************************************************************************************************************************************************
dmvpn-1               : ok=0    changed=0    unreachable=0    failed=1
```

Hints from issue discussion:
Files identified in the description:
* [lib/ansible/modules/network/ios/ios_config.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/network/ios/ios_config.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @privateip @rcarrillocruz
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
I'm seeing the same issue on 2.6.  Even running with -b --ask-become-pass doesn't work.
Something I noticed that I forgot to add in here, is that when I take these same hosts and run ios_facts before and pass ansible_net_config to the ios_config module, I don't seem to get the same errors.
Same problem with 2.6.2, appeared suddenly with no reasons.
Playbook works with pack of cisco devices, and one of them begin to show that _operation requires privilege escalation_ even not trying to connect to it. Furthermore, i've tried to connect to that host using another IP, changing it position in hosts file - no luck. But added task with ios_facts works fine (has been placed before iso_config).

> <172.30.40.249> local domain socket does not exist, starting it
<172.30.40.249> control socket path is /home/admin/.ansible/pc/0150193df5
<172.30.40.249> <172.30.40.249> ESTABLISH CONNECTION FOR USER: noc_cisco on PORT 22 TO 172.30.40.249
<172.30.40.249> <172.30.40.249> ssh connection done, setting terminal
<172.30.40.249> <172.30.40.249> loaded terminal plugin for network_os ios
<172.30.40.249> <172.30.40.249> loaded cliconf plugin for network_os ios
<172.30.40.249> <172.30.40.249> firing event: on_open_shell()
<172.30.40.249> <172.30.40.249> ssh connection has completed successfully
<172.30.40.249> connection to remote device started successfully
<172.30.40.249> local domain socket listeners started successfully
<172.30.40.249> 
<172.30.40.249> local domain socket path is /home/admin/.ansible/pc/0150193df5
<172.30.40.249> socket_path: /home/admin/.ansible/pc/0150193df5
<172.30.40.249> ESTABLISH LOCAL CONNECTION FOR USER: root
<172.30.40.249> EXEC /bin/sh -c 'echo ~root && sleep 0'
<172.30.40.249> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1533799671.94-258057751168387 `" && echo ansible-tmp-1533799671.94-258057751168387="` echo /root/.ansible/tmp/ansible-tmp-1533799671.94-258057751168387 `" ) && sleep 0'
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/network/ios/ios_config.py
<172.30.40.249> PUT /home/admin/.ansible/tmp/ansible-local-31894q0Y4O/tmpFQrzxR TO /root/.ansible/tmp/ansible-tmp-1533799671.94-258057751168387/ios_config.py
<172.30.40.249> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1533799671.94-258057751168387/ /root/.ansible/tmp/ansible-tmp-1533799671.94-258057751168387/ios_config.py && sleep 0'
<172.30.40.249> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-1533799671.94-258057751168387/ios_config.py && sleep 0'
<172.30.33.249> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1533799670.54-166425337223836/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_QX49JH/ansible_module_ios_config.py", line 561, in <module>
    main()
  File "/tmp/ansible_QX49JH/ansible_module_ios_config.py", line 490, in main
    load_config(module, commands)
  File "/tmp/ansible_QX49JH/ansible_modlib.zip/ansible/module_utils/network/ios/ios.py", line 168, in load_config
  File "/tmp/ansible_QX49JH/ansible_modlib.zip/ansible/module_utils/connection.py", line 149, in __rpc__
ansible.module_utils.connection.ConnectionError: operation requires privilege escalation
fatal: [172.30.33.249]: FAILED! => {
    "changed": false, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_QX49JH/ansible_module_ios_config.py\", line 561, in <module>\n    main()\n  File \"/tmp/ansible_QX49JH/ansible_module_ios_config.py\", line 490, in main\n    load_config(module, commands)\n  File \"/tmp/ansible_QX49JH/ansible_modlib.zip/ansible/module_utils/network/ios/ios.py\", line 168, in load_config\n  File \"/tmp/ansible_QX49JH/ansible_modlib.zip/ansible/module_utils/connection.py\", line 149, in __rpc__\nansible.module_utils.connection.ConnectionError: operation requires privilege escalation\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "rc": 1
}

Few more things to mention:
- That playbook worked before fine, w\o issues
- All devices manipulated by playbook are same in terms of hw, ios version and priv level for user
- There are four playbooks with ios_config working with that host, all four shows same error
- Tried to run pb with local root privileges and with regular user
Sooo
Small and accurate "debugging" shows very interesting things
I've opened file named `/usr/lib/python2.7/dist-packages/ansible/plugins/cliconf/__init__.py`
On lines 48-49 ansible checks for required privileges by comparing last char of prompt with pound sign and raises an error if fails. I've slightly modified line 49:

> raise AnsibleError('operation requires privilege escalation ' + prompt)

To see what it gets as prompt.
And guess what? It shows me part of cisco config! 

> fatal: [172.30.36.249]: FAILED! => {
    "changed": false, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_R7jpi7/ansible_module_ios_config.py\", line 561, in <module>\n    main()\n  File \"/tmp/ansible_R7jpi7/ansible_module_ios_config.py\", line 490, in main\n    load_config(module, commands)\n  File \"/tmp/ansible_R7jpi7/ansible_modlib.zip/ansible/module_utils/network/ios/ios.py\", line 168, in load_config\n  File \"/tmp/ansible_R7jpi7/ansible_modlib.zip/ansible/module_utils/connection.py\", line 149, in __rpc__\nansible.module_utils.connection.ConnectionError: operation requires privilege escalation OSPF->\n", 

Despite not having ios_facts in that playbook, only two steps using ios_command: modify object-group, write memory. That string, "OSPF->BGP" is a part of prefix list, and cannot be in login prompt/banner by any way.
So, there is a question: why ansible gets full config of device without user-generated commands to do this?
@notonlyvandalzzz If you don't pass ios_config a config via the `config` key, it will get the running config unless you set `match: none`. 

I noticed that I stopped getting the traceback when I added an ios_facts play and used that to provide the config to ios_config instead of letting it get it itself, which with your debugging definitely points towards something odd with the way ios_config is gathering the config for itself.

@Qalthos any chance this is related to #43748 ?
As a beginner, I think I met the issue too because the same err log:
  File "/var/folders/_1/185njz6165jclp7lsllhb1380000gn/T/ansible_lIqvQP/ansible_modlib.zip/ansible/module_utils/network/ios/ios.py", line 117, in get_config
  File "/var/folders/_1/185njz6165jclp7lsllhb1380000gn/T/ansible_lIqvQP/ansible_modlib.zip/ansible/module_utils/connection.py", line 174, in __rpc__
ansible.module_utils.connection.ConnectionError: operation requires privilege escalation

fatal: [192.168.1.100]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/var/folders/_1/185njz6165jclp7lsllhb1380000gn/T/ansible_lIqvQP/ansible_module_ios_config.py\", line 561, in <module>\n    main()\n  File \"/var/folders/_1/185njz6165jclp7lsllhb1380000gn/T/ansible_lIqvQP/ansible_module_ios_config.py\", line 464, in main\n    config, have_banners = get_running_config(module, config, flags=flags)\n  File \"/var/folders/_1/185njz6165jclp7lsllhb1380000gn/T/ansible_lIqvQP/ansible_module_ios_config.py\", line 361, in get_running_config\n    contents = get_config(module, flags=flags)\n  File\"/var/folders/_1/185njz6165jclp7lsllhb1380000gn/T/ansible_lIqvQP/ansible_modlib.zip/ansible/module_utils/network/ios/ios.py\", line 117, in get_config\n  File \"/var/folders/_1/185njz6165jclp7lsllhb1380000gn/T/ansible_lIqvQP/ansible_modlib.zip/ansible/module_utils/connection.py\", line 174, in __rpc__\nansible.module_utils.connection.ConnectionError: operation requires privilege escalation\n",

My env:
ansible 2.6.3
  config file = None
  configured module search path = ['/Users/kz/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/kz/Documents/C/ansibleprj/venv/lib/python3.7/site-packages/ansible
  executable location = /Users/kz/Documents/C/ansibleprj/venv/bin/ansible
  python version = 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24) [Clang 6.0 (clang-600.0.57)]

I did #1/#2 and it works:
1. add become_method in the playbook
```
---

- name: Network Getting Started First Playbook Extended
  connection: network_cli
  become_method: enable
  hosts: all
  tasks:
    - name: Update the hostname
      ios_config:
        lines:
          - hostname ansible-changed
        backup: no
```

2. run the playbook with parameters:
-b
-K
like this:
ansible-playbook -i 192.168.1.100, -u test -k -b -K -e ansible_network_os=ios first_playbook_ext.yml 
> any chance this is related to #43748 ?

I don't think that's very likely. The fact that the config survives being set as a fact indicates to me that it is unrelated to that problem.

The presence of `OSPF->BGP` is almost certainly an issue. It's hard to say precisely how to fix this issue, as `OSPF-` is a possible valid hostname, and as of #39063 we know that we can't always expect a newline before a prompt.

I'm not sure if the other reports are similar, but if `>` appears in your config, it might be the reason. That said, the fact that ios_facts works for @pffs is confounding. It may be we have several different issues here.

Commit message: Update to make buffer timeout float

Patch to review:
diff --git a/lib/ansible/plugins/connection/network_cli.py b/lib/ansible/plugins/connection/network_cli.py
index c27c9ad73cbc48..e7ab2f445572c9 100644
--- a/lib/ansible/plugins/connection/network_cli.py
+++ b/lib/ansible/plugins/connection/network_cli.py
@@ -157,6 +157,21 @@
       - name: ANSIBLE_PERSISTENT_COMMAND_TIMEOUT
     vars:
       - name: ansible_command_timeout
+  persistent_buffer_read_timeout:
+    type: float
+    description:
+      - Configures, in seconds, the amount of time to wait for the data to be read
+        from Paramiko channel after the command prompt is matched. This timeout
+        value ensure that command prompt matched is correct and there is no more data
+        left to be received from remote host.
+    default: 0.1
+    ini:
+      - section: persistent_connection
+        key: buffer_read_timeout
+    env:
+      - name: ANSIBLE_PERSISTENT_BUFFER_READ_TIMEOUT
+    vars:
+      - name: ansible_buffer_read_timeout
 """
 
 import getpass
@@ -164,6 +179,7 @@
 import logging
 import re
 import os
+import signal
 import socket
 import traceback
 from io import BytesIO
@@ -184,6 +200,10 @@
     display = Display()
 
 
+class AnsibleCmdRespRecv(Exception):
+    pass
+
+
 class Connection(NetworkConnectionBase):
     ''' CLI (shell) SSH connections on Paramiko '''
 
@@ -200,6 +220,7 @@ def __init__(self, play_context, new_stdin, *args, **kwargs):
         self._matched_pattern = None
         self._last_response = None
         self._history = list()
+        self._command_response = None
 
         self._terminal = None
         self.cliconf = None
@@ -342,15 +363,34 @@ def receive(self, command=None, prompts=None, answer=None, newline=True, prompt_
         '''
         Handles receiving of output from command
         '''
-        recv = BytesIO()
-        handled = False
-
         self._matched_prompt = None
         self._matched_cmd_prompt = None
+        recv = BytesIO()
+        handled = False
+        command_prompt_matched = False
         matched_prompt_window = window_count = 0
+        command_timeout = self.get_option('persistent_command_timeout')
 
         while True:
-            data = self._ssh_shell.recv(256)
+            if command_prompt_matched:
+                try:
+                    signal.signal(signal.SIGALRM, self._handle_buffer_read_timeout)
+                    signal.setitimer(signal.ITIMER_REAL, self.get_option('persistent_buffer_read_timeout'))
+                    data = self._ssh_shell.recv(256)
+                    signal.alarm(0)
+                    # if data is still received on channel it indicates the prompt string
+                    # is wrongly matched in between response chunks, continue to read
+                    # remaining response.
+                    command_prompt_matched = False
+
+                    # restart command_timeout timer
+                    signal.signal(signal.SIGALRM, self._handle_command_timeout)
+                    signal.alarm(command_timeout)
+
+                except AnsibleCmdRespRecv:
+                    return self._command_response
+            else:
+                data = self._ssh_shell.recv(256)
 
             # when a channel stream is closed, received data will be empty
             if not data:
@@ -376,7 +416,8 @@ def receive(self, command=None, prompts=None, answer=None, newline=True, prompt_
             if self._find_prompt(window):
                 self._last_response = recv.getvalue()
                 resp = self._strip(self._last_response)
-                return self._sanitize(resp, command)
+                self._command_response = self._sanitize(resp, command)
+                command_prompt_matched = True
 
     def send(self, command, prompt=None, answer=None, newline=True, sendonly=False, prompt_retry_check=False, check_all=False):
         '''
@@ -398,6 +439,15 @@ def send(self, command, prompt=None, answer=None, newline=True, sendonly=False,
             display.vvvv(traceback.format_exc(), host=self._play_context.remote_addr)
             raise AnsibleConnectionFailure("timeout trying to send command: %s" % command.strip())
 
+    def _handle_buffer_read_timeout(self, signum, frame):
+        raise AnsibleCmdRespRecv()
+
+    def _handle_command_timeout(self, signum, frame):
+        msg = 'command timeout triggered, timeout value is %s secs.\nSee the timeout setting options in the Network Debug and Troubleshooting Guide.'\
+              % self.get_option('persistent_command_timeout')
+        display.display(msg, log_only=True)
+        raise AnsibleConnectionFailure(msg)
+
     def _strip(self, data):
         '''
         Removes ANSI codes from device response
diff --git a/test/units/plugins/connection/test_network_cli.py b/test/units/plugins/connection/test_network_cli.py
index 4807795601b3dc..cbd64addd2bdb7 100644
--- a/test/units/plugins/connection/test_network_cli.py
+++ b/test/units/plugins/connection/test_network_cli.py
@@ -21,6 +21,7 @@
 __metaclass__ = type
 
 import re
+import time
 import json
 
 from io import StringIO
@@ -28,6 +29,7 @@
 from units.compat import unittest
 from units.compat.mock import patch, MagicMock
 
+from ansible.module_utils._text import to_text
 from ansible.errors import AnsibleConnectionFailure
 from ansible.playbook.play_context import PlayContext
 from ansible.plugins.loader import connection_loader
@@ -131,15 +133,14 @@ def test_network_cli_send(self):
         device#
         """
 
-        mock__shell.recv.return_value = response
-
+        mock__shell.recv.side_effect = [response, None]
         output = conn.send(b'command', None, None, None)
 
         mock__shell.sendall.assert_called_with(b'command\r')
-        self.assertEqual(output, 'command response')
+        self.assertEqual(to_text(conn._command_response), 'command response')
 
         mock__shell.reset_mock()
-        mock__shell.recv.return_value = b"ERROR: error message device#"
+        mock__shell.recv.side_effect = [b"ERROR: error message device#"]
 
         with self.assertRaises(AnsibleConnectionFailure) as exc:
             conn.send(b'command', None, None, None)


Code review comments:
diff_hunk: @@ -342,15 +363,34 @@ def receive(self, command=None, prompts=None, answer=None, newline=True, prompt_
         '''
         Handles receiving of output from command
         '''
-        recv = BytesIO()
-        handled = False
-
         self._matched_prompt = None
         self._matched_cmd_prompt = None
+        recv = BytesIO()
+        handled = False
+        command_prompt_matched = False
         matched_prompt_window = window_count = 0
+        command_timeout = self.get_option('persistent_command_timeout')
 
         while True:
-            data = self._ssh_shell.recv(256)
+            if command_prompt_matched:
+                try:
+                    signal.signal(signal.SIGALRM, self._handle_buffer_read_timeout)
+                    signal.setitimer(signal.ITIMER_REAL, self.get_option('persistent_buffer_read_timeout'))
path: lib/ansible/plugins/connection/network_cli.py
text: @user1:
This will raise `signal.ItimerError` if the timeout is < 0, and fails disastrously (not to mention slowly) when timeout == 0

@author:
Ok i can add a check to throw error is timeout value is <= 0