Repository: ansible/ansible

Issue:
Handling of sub-prompts, particularly for enable mode needs improvement
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest -->
 - Bug Report

##### COMPONENT NAME
<!---
Name of the module, plugin, task or feature
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path
-->

network_cli.py
connection/ios.py

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
devel
```

##### SUMMARY
<!--- Explain the problem briefly -->

Ansible does not fully handle enable mode Password prompting.  It sends a response once, but does not check if the device asked for Password: again.  The changes introduced in #33794 have brought this to the fore.  I worked witth @brainclots to identify that the issue in #33993 was actually that he was not getting to enable mode. 

The crux of the issue is:
- handle_prompts is only ever called once.  On an IOS device, by default it will prompt for the Password: 3 times.  Only after that fails will it return `%Bad Secrets`
- Now that it only sends one CR after a timeout to see if there is a prompt, it fails to see this error message.  (This explains why in #33993 it was failing with an odd message because the prompt was Password:)  Note, without some sort of some other prompt synchronization mechanisms, sending one CR is the proper approach here.  Thus better handling for the device still prompting for a Password should be implemented.
- Please note that #34585 also comes into play here.  Without that fix, you can see below that it ends up sending the password to the normal CLI prompt because it is not seeing the current prompt.

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
ansible-playbook -e ansible_become_password=foobar net-enable-mode-play.yml
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
```
fatal: [bd-csr1000v-00]: FAILED! => {
    "msg": "unable to elevate privilege to enable mode, at prompt [\nbd-csr1000v-00>]"
```

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
*20 second timeout*
```
    raise AnsibleConnectionFailure("timeout trying to send command: %s" % command.strip())
AnsibleConnectionFailure: timeout trying to send command:
```

##### Device Interaction logs
The below logs are from the #33928 (with a few local changes) which really help to identify issues like this.

----
The current devel branch responds like this:
```
18499 1515617839.32807: CLI-PROMPT: looking for prompt in | te
18499 1515617839.34388: CLI-PROMPT: looking for prompt in | term
18499 1515617839.35165: CLI-PROMPT: looking for prompt in | termi
18499 1515617839.35948: CLI-PROMPT: looking for prompt in | termin
18499 1515617839.36730: CLI-PROMPT: looking for prompt in | termina
18499 1515617839.37505: CLI-PROMPT: looking for prompt in | terminal
18499 1515617839.39099: CLI-PROMPT: looking for prompt in | terminal w
18499 1515617839.39869: CLI-PROMPT: looking for prompt in | terminal wi
18499 1515617839.40632: CLI-PROMPT: looking for prompt in | terminal wid
18499 1515617839.41406: CLI-PROMPT: looking for prompt in | terminal widt
18499 1515617839.42186: CLI-PROMPT: looking for prompt in | terminal width
18499 1515617839.43794: CLI-PROMPT: looking for prompt in | terminal width
18499 1515617839.43850: CLI-PROMPT: looking for prompt in | terminal width 5
18499 1515617839.44615: CLI-PROMPT: looking for prompt in | terminal width 51
18499 1515617839.45388: CLI-PROMPT: looking for prompt in | terminal width 512
18499 1515617839.46150: CLI-PROMPT: looking for prompt in | terminal width 512\r\n
18499 1515617839.46150: CLI-PROMPT: looking for prompt in | terminal width 512\r\n
18499 1515617839.46919: CLI-PROMPT: looking for prompt in | terminal width 512\r\nbd-csr1000v-00>
18499 1515617839.46949: CLI-PROMPT: Matched Standard Prompt [\r\n]?[\w+\-\.:\/\[\]]+(?:\([^\)]+\)){,3}(?:>|#) ?$
18499 1515617839.46970: CLI-RECV: terminal width 512\r\nbd-csr1000v-00>
18499 1515617839.46994: CLI-SEND: \r
18499 1515617839.48587: CLI-PROMPT: looking for prompt in | \r\nbd-csr1000v-00>
18499 1515617839.48617: CLI-PROMPT: Matched Standard Prompt [\r\n]?[\w+\-\.:\/\[\]]+(?:\([^\)]+\)){,3}(?:>|#) ?$
18499 1515617839.48638: CLI-RECV: \r\nbd-csr1000v-00>
18499 1515617839.48674: CLI-SEND: enable\r
18499 1515617839.51951: CLI-PROMPT: looking for prompt in | ena
18499 1515617839.51985: CLI-PROMPT: looking for prompt in | enab
18499 1515617839.52744: CLI-PROMPT: looking for prompt in | enabl
18499 1515617839.53518: CLI-PROMPT: looking for prompt in | enable
18499 1515617839.54287: CLI-PROMPT: looking for prompt in | enable\r\n
18499 1515617839.65107: CLI-SUBPROMPT: Matched [\r\n]?password: $, sent '<answer>\r'
18499 1515617839.65140: CLI-PROMPT: looking for prompt in | enable\r\nPassword:
18499 1515617839.65169: CLI-PROMPT: looking for prompt in | enable\r\nPassword: \r\nPassword:
18499 1515617849.65339: CLI-SEND: \r
18499 1515617849.66965: CLI-PROMPT: looking for prompt in | \r\nPassword:
```

----

*As an aside, the logging above shows the buffering issues with paramiko.  I have an inquiry upstream to see if there is a better way to flush the send buffer.  I introduced microsecond delays after sending to see this logging improve, but that is not optimal.  https://github.com/paramiko/paramiko/issues/1145 *

With my patch which checks for the prompt more than once, this changes to:
```
18572 1515618500.48843: CLI-SUBPROMPT: Matched [\r\n]?password: $, sent '<answer>\r'
18572 1515618500.48873: CLI-PROMPT: looking for prompt in | enable\r\nPassword:
18572 1515618500.58946: CLI-SUBPROMPT: Matched [\r\n]?password: $, sent '<answer>\r'
18572 1515618500.58979: CLI-PROMPT: looking for prompt in | enable\r\nPassword: \r\nPassword: \r\nPassword:
18572 1515618500.69058: CLI-SUBPROMPT: Matched [\r\n]?password: $, sent '<answer>\r'
18572 1515618500.69092: CLI-PROMPT: looking for prompt in | enable\r\nPassword: \r\nPassword: \r\nPassword: \r\n% Bad secrets\r\n\r\nbd-csr1000v-00>
18572 1515618500.69122: CLI-PROMPT: Matched Error Prompt [\r\n]?[\w+\-\.:\/\[\]]+(?:\([^\)]+\)){,3}(?:>|#) ?$
18572 1515618500.69155: CLI-SEND: \r
18572 1515618500.70759: CLI-PROMPT: looking for prompt in | \r\nbd-csr1000v-00>
18572 1515618500.70788: CLI-PROMPT: Matched Standard Prompt [\r\n]?[\w+\-\.:\/\[\]]+(?:\([^\)]+\)){,3}(?:>|#) ?$
18572 1515618500.70813: CLI-RECV: \r\nbd-csr1000v-00>

fatal: [bd-csr1000v-00]: FAILED! => {
    "msg": "unable to elevate privilege to enable mode, at prompt [\nbd-csr1000v-00>]"
}
```
-----

Note I also identified an issue with the password prompt regex similar to #34585 in that the newline is optional.  This should also be fixed.  See below, there is a timing issue introduced when checking for the subprompt more than once.  With any sort of network delay or timing, it could "see" the password prompt in the buffer anywhere without regard to starting a newline and end of buffer.  Thus it could send the password more than three times, as is the case below.  It sent it enough times for it to be exposed in the normal cli where cisco performed a host lookup by default.

```
18823 1515619048.83241: CLI-RECV: \r\nbd-csr1000v-00>
18823 1515619048.83276: CLI-SEND: enable\r
18823 1515619048.93427: CLI-SUBPROMPT: Matched [\r\n]?password: $, sent '<answer>\r'
18823 1515619048.93454: CLI-PROMPT: looking for prompt in | enable\r\nPassword:
18823 1515619048.93502: CLI-SUBPROMPT: Matched [\r\n]?password: $, sent '<answer>\r'
18823 1515619048.95096: CLI-PROMPT: looking for prompt in | enable\r\nPassword: \r\nPassword:
18823 1515619048.95168: CLI-SUBPROMPT: Matched [\r\n]?password: $, sent '<answer>\r'
18823 1515619048.95312: CLI-PROMPT: looking for prompt in | enable\r\nPassword: \r\nPassword: \r\nPassword:
18823 1515619048.95375: CLI-SUBPROMPT: Matched [\r\n]?password: $, sent '<answer>\r'
18823 1515619049.01774: CLI-PROMPT: looking for prompt in | enable\r\nPassword: \r\nPassword: \r\nPassword: \r\n% Bad secrets\r\n\r\nbd-csr1000v-00>fo
18823 1515619049.02149: CLI-PROMPT: looking for prompt in | enable\r\nPassword: \r\nPassword: \r\nPassword: \r\n% Bad secrets\r\n\r\nbd-csr1000v-00>foo
18823 1515619049.02932: CLI-PROMPT: looking for prompt in | enable\r\nPassword: \r\nPassword: \r\nPassword: \r\n% Bad secrets\r\n\r\nbd-csr1000v-00>foo\r\n
18823 1515619049.03713: CLI-PROMPT: looking for prompt in | enable\r\nPassword: \r\nPassword: \r\nPassword: \r\n% Bad secrets\r\n\r\nbd-csr1000v-00>foo\r\n% Bad IP address or host name% Unknown command or computer name, or unable to find computer address\r\nbd-csr1000v-00>
18823 1515619049.03758: CLI-PROMPT: Matched Error Prompt [\r\n]?[\w+\-\.:\/\[\]]+(?:\([^\)]+\)){,3}(?:>|#) ?$
18823 1515619049.03815: CLI-SEND: \r
18823 1515619049.13885: CLI-PROMPT: looking for prompt in | \r\nbd-csr1000v-00>
18823 1515619049.13917: CLI-PROMPT: Matched Standard Prompt [\r\n]?[\w+\-\.:\/\[\]]+(?:\([^\)]+\)){,3}(?:>|#) ?$
18823 1515619049.13942: CLI-RECV: \r\nbd-csr1000v-00>
```



Discussion:
Files identified in the description:
* [lib/ansible/plugins/connection/network_cli.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/connection/network_cli.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @Qalthos @ganeshrn @gundalow @kedarX @privateip @rcarrillocruz @trishnaguha
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->

Resolving patch:
diff --git a/lib/ansible/plugins/connection/network_cli.py b/lib/ansible/plugins/connection/network_cli.py
--- a/lib/ansible/plugins/connection/network_cli.py
+++ b/lib/ansible/plugins/connection/network_cli.py
@@ -395,6 +395,10 @@ def receive(self, command=None, prompts=None, answer=None, newline=True):
             window = self._strip(recv.read())
             if prompts and not handled:
                 handled = self._handle_prompt(window, prompts, answer, newline)
+            elif prompts and handled:
+                # check again even when handled, a sub-prompt could be
+                # repeating (like in the case of a wrong enable password, etc)
+                self._handle_prompt(window, prompts, answer, newline)
 
             if self._find_prompt(window):
                 self._last_response = recv.getvalue()
diff --git a/lib/ansible/plugins/terminal/ios.py b/lib/ansible/plugins/terminal/ios.py
--- a/lib/ansible/plugins/terminal/ios.py
+++ b/lib/ansible/plugins/terminal/ios.py
@@ -60,7 +60,7 @@ def on_become(self, passwd=None):
         if passwd:
             # Note: python-3.5 cannot combine u"" and r"" together.  Thus make
             # an r string and use to_text to ensure it's text on both py2 and py3.
-            cmd[u'prompt'] = to_text(r"[\r\n]?password: $", errors='surrogate_or_strict')
+            cmd[u'prompt'] = to_text(r"[\r\n]password: $", errors='surrogate_or_strict')
             cmd[u'answer'] = passwd
 
         try:


Test patch: