Repository: ansible/ansible
Pull request:  Add support to configure network_cli terminal related options
Depends-On: https://github.com/ansible/ansible/pull/60305
##### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
Fixes #59404

*  Add network_cli configurable options to support platform specific login menu
*  Add network_cli configurable options to support configurable options for stdout
       and stderr regex list
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->

##### ISSUE TYPE
<!--- Pick one below and delete the rest -->
- Feature Pull Request

##### COMPONENT NAME
<!--- Write the short name of the module, plugin, task or feature below -->
docs/docsite/rst/network/user_guide/network_debug_troubleshooting.rst
lib/ansible/plugins/connection/network_cli.py


##### ADDITIONAL INFORMATION
<!--- Include additional information to help people understand the change here -->
<!--- A step-by-step reproduction of the problem is helpful if there is no related issue -->

<!--- Paste verbatim command output below, e.g. before and after your change -->
```paste below

```


Problem statement:
IOS Terminal: "ERROR" in an IOS log message causes terminal to incorrectly parse a stderr in response to a command.
<!--- Verify first that your issue is not already reported on GitHub -->
<!--- Also test if the latest release and devel branch are affected too -->
<!--- Complete *all* sections as described, this form is processed automatically -->

##### SUMMARY
<!--- Explain the problem briefly below -->
The word "ERROR" in an IOS log message causes terminal to parse an error. IOS_command modules doesn't register stdout, only the one line of log output with the word "Error" as stderr. Task fails.
##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
<!--- Write the short name of the module, plugin, task or feature below, use your best guess if unsure -->
I'm using ios_command module, however, I believe the ios terminal plugin has the regex to parse stderr.  ansible/lib/ansible/plugins/terminal/ios.py 
##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes -->
```paste below
Using 2.7.7. Difficult to copy and paste due to locked down environment.
```

##### CONFIGURATION
<!--- Paste verbatim output from "ansible-config dump --only-changed" between quotes -->
```paste below
See above.
```

##### OS / ENVIRONMENT
<!--- Provide all relevant information below, e.g. target OS versions, network device firmware, etc. -->
Ubuntu 16.04.6

##### STEPS TO REPRODUCE
<!--- Describe exactly how to reproduce the problem, using a minimal test-case -->
Use ios_command module to show logs in buffer on the device. Register the output as a variable. Debug the contents of the variable. If a line of logs contains the word "ERROR", the task will fail and stdout will be empty.

Actual log lines that have triggered this:
%IPSEC-3-REPLAY_ERROR: IPSec SA recieves anti-replay error, DP Handle 549, src_addr x.x.x.x, dest_addr y.y.y.y, SPI...

ISAKMP-ERROR: (0):Failed to find peer index node to update peer_info_list

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- tasks
  ios_command:
    commands: show logging
  register: logs
```

<!--- HINT: You can paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- Describe what you expected to happen when running the steps above -->
Lines of logs registered in variable.stdout.

##### ACTUAL RESULTS
<!--- Describe what actually happened. If possible run with extra verbosity (-vvvv) -->
Variable has no stdout, only stderr. Stderr is only the one line of logs that contained the word "ERROR".
<!--- Paste verbatim command output between quotes -->
```paste below

```

Hints from issue discussion:
I think it's this line that's the problem in terminal/ios.py:

`         re.compile(br"[%\S] ?Error: ?[\s]+", re.I), `

If we were to just take out the re.I, this regex would no longer match my logs containing "ERROR". Might miss some legitimate error messages though.
@ganeshrn I think you committed this line way back when. What do you think?
cc @justjais @privateip
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Also, yes, I know syslog servers exist. Don't ask...

Commit message: Add support to configure terminal related options

Fixes https://github.com/ansible/ansible/issues/59404

*  Add network_cli configurable options to support platform specific login menu
*  Add network_cli configurable options to support configurable options for stdout
   and stderr regex list

Patch to review:
diff --git a/docs/docsite/rst/network/user_guide/network_debug_troubleshooting.rst b/docs/docsite/rst/network/user_guide/network_debug_troubleshooting.rst
index ba383a058c107f..74908b204ab73e 100644
--- a/docs/docsite/rst/network/user_guide/network_debug_troubleshooting.rst
+++ b/docs/docsite/rst/network/user_guide/network_debug_troubleshooting.rst
@@ -561,6 +561,45 @@ To make this a permanent change, add the following to your ``ansible.cfg`` file:
    connect_retry_timeout = 30
 
 
+Timeout issue due to platform specific login menu with ``network_cli`` connection type
+--------------------------------------------------------------------------------------
+
+In Ansible 2.9 and later, the network_cli connection plugin configuration options are added
+to handle the platform specific login menu. These options can be set as group/host or tasks
+variables.
+
+Example: Handle single login menu prompts with host variables
+
+.. code-block:: console
+
+    $cat host_vars/<hostname>.yaml
+    ---
+    ansible_terminal_initial_prompt:
+      - "Connect to a host"
+    ansible_terminal_initial_answer:
+      - "3"
+
+Example: Handle remote host multiple login menu prompts with host variables
+
+.. code-block:: console
+
+    $cat host_vars/<inventory-hostname>.yaml
+    ---
+    ansible_terminal_initial_prompt:
+      - "Press any key to enter main menu"
+      - "Connect to a host"
+    ansible_terminal_initial_answer:
+      - "\\r"
+      - "3"
+    ansible_terminal_initial_prompt_checkall: True
+
+To handle multiple login menu prompts:
+* The values of ``ansible_terminal_initial_prompt`` and ``ansible_terminal_initial_answer`` should be a list.
+* The prompt sequence should match the answer sequence.
+* The value of ``ansible_terminal_initial_prompt_checkall`` should be set to ``True``.
+
+.. note:: If all the prompts in sequence are not received from remote host at the time connection initialization it will result in a timeout.
+
 
 Playbook issues
 ===============
@@ -757,3 +796,52 @@ To make this a global setting, add the following to your ``ansible.cfg`` file:
    buffer_read_timeout = 2
 
 This timer delay per command executed on remote host can be disabled by setting the value to zero.
+
+
+Task failure due to mismatched error regex within command response using ``network_cli`` connection type
+--------------------------------------------------------------------------------------------------------
+
+In Ansible 2.9 and later, the network_cli connection plugin configuration options are added
+to handle the stdout and stderr regex to identify if the command execution response consist
+of a normal response or an error response. These options can be set group/host variables or as
+tasks variables.
+
+Example: For mismatched error response
+
+.. code-block:: yaml
+
+  - name: fetch logs from remote host
+    ios_command:
+      commands:
+        - show logging
+
+
+Playbook run output:
+
+.. code-block:: console
+
+  TASK [first fetch logs] ********************************************************
+  fatal: [ios01]: FAILED! => {
+      "changed": false,
+      "msg": "RF Name:\r\n\r\n <--nsip-->
+             \"IPSEC-3-REPLAY_ERROR: Test log\"\r\n*Aug  1 08:36:18.483: %SYS-7-USERLOG_DEBUG:
+              Message from tty578(user id: ansible): test\r\nan-ios-02#"}
+
+Suggestions to resolve:
+
+Modify the error regex for individual task.
+
+.. code-block:: yaml
+
+  - name: fetch logs from remote host
+    ios_command:
+      commands:
+        - show logging
+    vars:
+      ansible_terminal_stderr_re:
+        - pattern: 'connection timed out'
+          flags: 're.I'
+
+The terminal plugin regex options ``ansible_terminal_stderr_re`` and ``ansible_terminal_stdout_re`` have
+``pattern`` and ``flags`` as keys. The value of the ``flags`` key should be a value that is accepted by
+the ``re.compile`` python method.
diff --git a/lib/ansible/plugins/connection/network_cli.py b/lib/ansible/plugins/connection/network_cli.py
index 32947664193e40..aa97e5d204e10a 100644
--- a/lib/ansible/plugins/connection/network_cli.py
+++ b/lib/ansible/plugins/connection/network_cli.py
@@ -191,6 +191,65 @@
       - name: ANSIBLE_PERSISTENT_LOG_MESSAGES
     vars:
       - name: ansible_persistent_log_messages
+  terminal_stdout_re:
+    type: list
+    elements: dict
+    description:
+      - A single regex pattern or a sequence of patterns along with optional flags
+        to match the command prompt from the received response chunk. This option
+        accepts C(pattern) and C(flags) keys. The value of C(pattern) is a python
+        regex pattern to match the response and the value of C(flags) is the value
+        accepted by I(flags) argument of I(re.compile) python method to control
+        the way regex is matched with the response, for example I('re.I').
+    vars:
+      - name: ansible_terminal_stdout_re
+  terminal_stderr_re:
+    type: list
+    elements: dict
+    description:
+      - This option provides the regex pattern and optional flags to match the
+        error string from the received response chunk. This option
+        accepts C(pattern) and C(flags) keys. The value of C(pattern) is a python
+        regex pattern to match the response and the value of C(flags) is the value
+        accepted by I(flags) argument of I(re.compile) python method to control
+        the way regex is matched with the response, for example I('re.I').
+    vars:
+      - name: ansible_terminal_stderr_re
+  terminal_initial_prompt:
+    type: list
+    description:
+      - A single regex pattern or a sequence of patterns to evaluate the expected
+        prompt at the time of initial login to the remote host.
+    vars:
+      - name: ansible_terminal_initial_prompt
+  terminal_initial_answer:
+    type: list
+    description:
+      - The answer to reply with if the C(terminal_initial_prompt) is matched. The value can be a single answer
+        or a list of answers for multiple terminal_initial_prompt. In case the login menu has
+        multiple prompts the sequence of the prompt and excepted answer should be in same order and the value
+        of I(terminal_prompt_checkall) should be set to I(True) if all the values in C(terminal_initial_prompt) are
+        expected to be matched and set to I(False) if any one login prompt is to be matched.
+    vars:
+      - name: ansible_terminal_initial_answer
+  terminal_initial_prompt_checkall:
+    type: boolean
+    description:
+      - By default the value is set to I(False) and any one of the prompts mentioned in C(terminal_initial_prompt)
+        option is matched it won't check for other prompts. When set to I(True) it will check for all the prompts
+        mentioned in C(terminal_initial_prompt) option in the given order and all the prompts
+        should be received from remote host if not it will result in timeout.
+    default: False
+    vars:
+      - name: ansible_terminal_initial_prompt_checkall
+  terminal_inital_prompt_newline:
+    type: boolean
+    description:
+      - This boolean flag, that when set to I(True) will send newline in the response if any of values
+        in I(terminal_initial_prompt) is matched.
+    default: True
+    vars:
+      - name: ansible_terminal_initial_prompt_newline
 """
 
 import getpass
@@ -338,11 +397,12 @@ def _connect(self):
 
             self.queue_message('vvvv', 'loaded terminal plugin for network_os %s' % self._network_os)
 
-            self.receive(
-                prompts=self._terminal.terminal_initial_prompt,
-                answer=self._terminal.terminal_initial_answer,
-                newline=self._terminal.terminal_inital_prompt_newline
-            )
+            terminal_initial_prompt = self.get_option('terminal_initial_prompt') or self._terminal.terminal_initial_prompt
+            terminal_initial_answer = self.get_option('terminal_initial_answer') or self._terminal.terminal_initial_answer
+            newline = self.get_option('terminal_inital_prompt_newline') or self._terminal.terminal_inital_prompt_newline
+            check_all = self.get_option('terminal_initial_prompt_checkall') or False
+
+            self.receive(prompts=terminal_initial_prompt, answer=terminal_initial_answer, newline=newline, check_all=check_all)
 
             self.queue_message('vvvv', 'firing event: on_open_shell()')
             self._terminal.on_open_shell()
@@ -386,6 +446,10 @@ def receive(self, command=None, prompts=None, answer=None, newline=True, prompt_
         command_prompt_matched = False
         matched_prompt_window = window_count = 0
 
+        # set terminal regex values for command prompt and errors in response
+        self._terminal_stderr_re = self._get_terminal_std_re('terminal_stderr_re')
+        self._terminal_stdout_re = self._get_terminal_std_re('terminal_stdout_re')
+
         cache_socket_timeout = self._ssh_shell.gettimeout()
         command_timeout = self.get_option('persistent_command_timeout')
         self._validate_timeout_value(command_timeout, "persistent_command_timeout")
@@ -463,8 +527,10 @@ def send(self, command, prompt=None, answer=None, newline=True, sendonly=False,
             if prompt_len != answer_len:
                 raise AnsibleConnectionFailure("Number of prompts (%s) is not same as that of answers (%s)" % (prompt_len, answer_len))
         try:
-            self._history.append(command)
-            self._ssh_shell.sendall(b'%s\r' % command)
+            cmd = b'%s\r' % command
+            self._history.append(cmd)
+            self._ssh_shell.sendall(cmd)
+            self._log_messages('send command: %s' % cmd)
             if sendonly:
                 return
             response = self.receive(command, prompt, answer, newline, prompt_retry_check, check_all)
@@ -513,7 +579,7 @@ def _handle_prompt(self, resp, prompts, answer, newline, prompt_retry_check=Fals
             single_prompt = True
         if not isinstance(answer, list):
             answer = [answer]
-        prompts_regex = [re.compile(r, re.I) for r in prompts]
+        prompts_regex = [re.compile(to_bytes(r), re.I) for r in prompts]
         for index, regex in enumerate(prompts_regex):
             match = regex.search(resp)
             if match:
@@ -557,13 +623,14 @@ def _find_prompt(self, response):
         '''
         errored_response = None
         is_error_message = False
-        for regex in self._terminal.terminal_stderr_re:
+
+        for regex in self._terminal_stderr_re:
             if regex.search(response):
                 is_error_message = True
 
                 # Check if error response ends with command prompt if not
                 # receive it buffered prompt
-                for regex in self._terminal.terminal_stdout_re:
+                for regex in self._terminal_stdout_re:
                     match = regex.search(response)
                     if match:
                         errored_response = response
@@ -573,7 +640,7 @@ def _find_prompt(self, response):
                         break
 
         if not is_error_message:
-            for regex in self._terminal.terminal_stdout_re:
+            for regex in self._terminal_stdout_re:
                 match = regex.search(response)
                 if match:
                     self._matched_pattern = regex.pattern
@@ -604,3 +671,23 @@ def transport_test(self, connect_timeout):
         self.close()
         self._connect()
         self.close()
+
+    def _get_terminal_std_re(self, option):
+        terminal_std_option = self.get_option(option)
+        terminal_std_re = []
+
+        if terminal_std_option:
+            for item in terminal_std_option:
+                if "pattern" not in item:
+                    raise AnsibleConnectionFailure("'pattern' is a required key for option '%s',"
+                                                   " received option value is %s" % (option, item))
+                pattern = br"%s" % to_bytes(item['pattern'])
+                flag = item.get('flags', 0)
+                if flag:
+                    flag = getattr(re, flag.split('.')[1])
+                terminal_std_re.append(re.compile(pattern, flag))
+        else:
+            # To maintain backward compatibility
+            terminal_std_re = getattr(self._terminal, option)
+
+        return terminal_std_re
diff --git a/test/integration/targets/ios_command/tests/cli/error_regex.yaml b/test/integration/targets/ios_command/tests/cli/error_regex.yaml
new file mode 100644
index 00000000000000..c8c8a47c046af7
--- /dev/null
+++ b/test/integration/targets/ios_command/tests/cli/error_regex.yaml
@@ -0,0 +1,59 @@
+---
+- debug: msg="START cli/error_regex.yaml on connection={{ ansible_connection }}"
+
+- block:
+  - name: clear logs 1
+    cli_command: &clear_logs
+      command: clear logging
+      prompt:
+        - Clear logging buffer
+      answer:
+        - "\r"
+    ignore_errors: True
+
+  - name: send log with error regex match 1
+    cli_command: &send_logs
+      command: "send log 'IPSEC-3-REPLAY_ERROR: test log_1'"
+    ignore_errors: True
+
+  - name: fetch logs without command specific error regex
+    ios_command:
+      commands:
+        - show logging
+    register: result
+    ignore_errors: True
+
+  - name: ensure task fails due to mismatched regex
+    assert:
+      that:
+        - "result.failed == true"
+
+  - name: pause to avoid rate limiting
+    pause:
+      seconds: 10
+
+  - name: clear logs 2
+    cli_command: *clear_logs
+    ignore_errors: True
+
+  - name: send log with error regex match 2
+    cli_command: *send_logs
+    ignore_errors: True
+
+  - name: fetch logs with command specific error regex
+    ios_command:
+      commands:
+        - show logging
+    register: result
+    vars:
+      ansible_terminal_stderr_re:
+        - pattern: 'connection timed out'
+          flags: 're.I'
+
+  - name: ensure task with modified error regex is success
+    assert:
+      that:
+        - "result.failed == false"
+  when: ansible_connection == 'network_cli'
+
+- debug: msg="END cli/error_regex.yaml on connection={{ ansible_connection }}"
diff --git a/test/units/plugins/connection/test_network_cli.py b/test/units/plugins/connection/test_network_cli.py
index 6dc8cd292c8aa7..bbb40676f8e61f 100644
--- a/test/units/plugins/connection/test_network_cli.py
+++ b/test/units/plugins/connection/test_network_cli.py
@@ -111,15 +111,16 @@ def test_network_cli_exec_command(self, mocked_super):
         self.assertEqual(out, b'command response')
         mock_send.assert_called_with(command=b'command')
 
+    @patch("ansible.plugins.connection.network_cli.Connection._get_terminal_std_re")
     @patch("ansible.plugins.connection.network_cli.Connection._connect")
-    def test_network_cli_send(self, mocked_connect):
+    def test_network_cli_send(self, mocked_connect, mocked_terminal_re):
+
         pc = PlayContext()
         pc.network_os = 'ios'
         conn = connection_loader.get('network_cli', pc, '/dev/null')
 
         mock__terminal = MagicMock()
-        mock__terminal.terminal_stdout_re = [re.compile(b'device#')]
-        mock__terminal.terminal_stderr_re = [re.compile(b'^ERROR')]
+        mocked_terminal_re.side_effect = [[re.compile(b'^ERROR')], [re.compile(b'device#')]]
         conn._terminal = mock__terminal
 
         mock__shell = MagicMock()
@@ -139,7 +140,7 @@ def test_network_cli_send(self, mocked_connect):
 
         mock__shell.reset_mock()
         mock__shell.recv.side_effect = [b"ERROR: error message device#"]
-
+        mocked_terminal_re.side_effect = [[re.compile(b'^ERROR')], [re.compile(b'device#')]]
         with self.assertRaises(AnsibleConnectionFailure) as exc:
             conn.send(b'command')
         self.assertEqual(str(exc.exception), 'ERROR: error message device#')


Code review comments:
diff_hunk: @@ -561,6 +561,45 @@ To make this a permanent change, add the following to your ``ansible.cfg`` file:
    connect_retry_timeout = 30
 
 
+Timeout issue due to platform specific login menu with ``network_cli`` connection type
+--------------------------------------------------------------------------------------
+
+In Ansible 2.9 and later, the network_cli connection plugin configuration options are added
+to handle the platform specific login menu. These options can be set as group/host or tasks
+variables.
+
+Example: Handle single login menu prompts with host variables
+
+.. code-block:: console
+
+    $cat host_vars/<hostname>.yaml
+    ---
+    ansible_terminal_initial_prompt:
+      - "Connect to a host"
+    ansible_terminal_initial_answer:
+      - "3"
+
+Example: Handle remote host multiple login menu prompts with host variables
+
+.. code-block:: console
+
+    $cat host_vars/<inventory-hostname>.yaml
+    ---
+    ansible_terminal_initial_prompt:
+      - "Press any key to enter main menu"
+      - "Connect to a host"
+    ansible_terminal_initial_answer:
+      - "\\r"
+      - "3"
+    ansible_terminal_initial_prompt_checkall: True
+
+To handle multiple login menu prompts:
+* The values of ``ansible_terminal_initial_prompt`` and ``ansible_terminal_initial_answer`` should be a list.
path: docs/docsite/rst/network/user_guide/network_debug_troubleshooting.rst
text: @user1:
```suggestion

* The values of ``ansible_terminal_initial_prompt`` and ``ansible_terminal_initial_answer`` should be a list.
```