Repository: ansible/ansible
Pull request: Fix junos terminal regex for cluster srx devices


##### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
Fixes #50726

*  Modify junos terminal regex to match for
   string `{primary:node0}` which is also part of the
   prompt.
*  Modify network_cli connection plugin to ignore
   multiple prompt matched lines.
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->

##### ISSUE TYPE
<!--- Pick one below and delete the rest -->
- Bugfix Pull Request

##### COMPONENT NAME
<!--- Write the short name of the module, plugin, task or feature below -->
plugins/terminal/junos.py
plugins/connection/network_cli.py
##### ADDITIONAL INFORMATION
<!--- Include additional information to help people understand the change here -->
<!--- A step-by-step reproduction of the problem is helpful if there is no related issue -->

<!--- Paste verbatim command output below, e.g. before and after your change -->
```paste below

```


Problem statement:
Extraneous data in junos_command output with SRX cluster
<!--- Verify first that your issue is not already reported on GitHub -->
<!--- Also test if the latest release and devel branch are affected too -->
<!--- Complete *all* sections as described, this form is processed automatically -->

##### SUMMARY
<!--- Explain the problem briefly below -->
I am seeing an extraneous string returned while using the junos_command module with the network_cli connection type.  The extraneous string is a portion of the prompt when working with a SRX cluster, `{primary:node0}`.  I have tested with 2.7.5 as well as devel with the same results. 

##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
<!--- Write the short name of the module, plugin, task or feature below, use your best guess if unsure -->
junos_command

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes -->
```paste below
ansible 2.6.11
  config file = /home/david/test/ansible.cfg
  configured module search path = [u'/home/david/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/david/.virtualenvs/testing/local/lib/python2.7/site-packages/ansible
  executable location = /home/david/.virtualenvs/testing/bin/ansible
  python version = 2.7.15p
```

##### CONFIGURATION
<!--- Paste verbatim output from "ansible-config dump --only-changed" between quotes -->
```paste below
DEFAULT_GATHERING(/home/david/test/ansible.cfg) = explicit
DEFAULT_HOST_LIST(/home/david/test/ansible.cfg) = [u'/home/david/test/inventory.yml']
HOST_KEY_CHECKING(/home/david/test/ansible.cfg) = False
PERSISTENT_COMMAND_TIMEOUT(/home/david/test/ansible.cfg) = 60
RETRY_FILES_ENABLED(/home/david/test/ansible.cfg) = False
```

##### OS / ENVIRONMENT
<!--- Provide all relevant information below, e.g. target OS versions, network device firmware, etc. -->
Ansible host:
```
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"
```

Juniper Device:
```
node0:
--------------------------------------------------------------------------
Hostname: vsrx0
Model: vsrx
Junos: 15.1X49-D160.2
JUNOS Software Release [15.1X49-D160.2]

node1:
--------------------------------------------------------------------------
Hostname: vsrx0
Model: vsrx
Junos: 15.1X49-D160.2
JUNOS Software Release [15.1X49-D160.2]
```

##### STEPS TO REPRODUCE
<!--- Describe exactly how to reproduce the problem, using a minimal test-case -->
This issue does not seem to care what command is run, my testing has been with various show commands.  I have also tested different `display` options within `junos_command` and saw the same results.  The simple playbook below

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- name: network_cli
  hosts: srx
  connection: network_cli
  tasks:
    - name: "show version"
      junos_command:
        commands: "show version"
        display: xml
      register: response

    - debug:
        var: response.stdout_lines[0]
```

<!--- HINT: You can paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- Describe what you expected to happen when running the steps above -->
I should get the data back from the device minus the extraneous string containing `{primary:node0}`.  The extraneous text could also state secondary and node 0 or 1. 

##### ACTUAL RESULTS
<!--- Describe what actually happened. If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes -->
```paste below
ansible-playbook 2.6.11
  config file = /home/david/test/ansible.cfg
  configured module search path = [u'/home/david/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/david/.virtualenvs/testing/local/lib/python2.7/site-packages/ansible
  executable location = /home/david/.virtualenvs/testing/bin/ansible-playbook
  python version = 2.7.15rc1 (default, Nov 12 2018, 14:31:15) [GCC 7.3.0]
Using /home/david/test/ansible.cfg as config file
setting up inventory plugins
Parsed /home/david/test/inventory.yml inventory source with yaml plugin
Loading callback plugin default of type stdout, v2.0 from /home/david/.virtualenvs/testing/local/lib/python2.7/site-packages/ansible/plugins/callback/default.pyc

PLAYBOOK: test_playbook.yml ********************************************************************************************************************************
1 plays in test_playbook.yml

PLAY [network_cli] *****************************************************************************************************************************************
META: ran handlers

TASK [show version] ****************************************************************************************************************************************
task path: /home/david/test/test_playbook.yml:35
<192.168.122.10> attempting to start connection
<192.168.122.10> using connection plugin network_cli
<192.168.122.10> local domain socket does not exist, starting it
<192.168.122.10> control socket path is /home/david/.ansible/pc/bbbda4105f
<192.168.122.10> <192.168.122.10> ESTABLISH CONNECTION FOR USER: root on PORT 22 TO 192.168.122.10
<192.168.122.10> <192.168.122.10> ssh connection done, setting terminal
<192.168.122.10> <192.168.122.10> loaded terminal plugin for network_os junos
<192.168.122.10> <192.168.122.10> loaded cliconf plugin for network_os junos
<192.168.122.10> <192.168.122.10> firing event: on_open_shell()
<192.168.122.10> <192.168.122.10> starting cli
<192.168.122.10> <192.168.122.10> ssh connection has completed successfully
<192.168.122.10> connection to remote device started successfully
<192.168.122.10> local domain socket listeners started successfully
<192.168.122.10> 
<192.168.122.10> local domain socket path is /home/david/.ansible/pc/bbbda4105f
<192.168.122.10> ESTABLISH LOCAL CONNECTION FOR USER: david
<192.168.122.10> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/david/.ansible/tmp/ansible-local-12168UkWcRa/ansible-tmp-1547066739.99-193237508171269 `" && echo ansible-tmp-1547066739.99-193237508171269="` echo /home/david/.ansible/tmp/ansible-local-12168UkWcRa/ansible-tmp-1547066739.99-193237508171269 `" ) && sleep 0'
Using module file /home/david/.virtualenvs/testing/local/lib/python2.7/site-packages/ansible/modules/network/junos/junos_command.py
<192.168.122.10> PUT /home/david/.ansible/tmp/ansible-local-12168UkWcRa/tmpCgBNXD TO /home/david/.ansible/tmp/ansible-local-12168UkWcRa/ansible-tmp-1547066739.99-193237508171269/junos_command.py
<192.168.122.10> EXEC /bin/sh -c 'chmod u+x /home/david/.ansible/tmp/ansible-local-12168UkWcRa/ansible-tmp-1547066739.99-193237508171269/ /home/david/.ansible/tmp/ansible-local-12168UkWcRa/ansible-tmp-1547066739.99-193237508171269/junos_command.py && sleep 0'
<192.168.122.10> EXEC /bin/sh -c '/usr/bin/python /home/david/.ansible/tmp/ansible-local-12168UkWcRa/ansible-tmp-1547066739.99-193237508171269/junos_command.py && sleep 0'
<192.168.122.10> EXEC /bin/sh -c 'rm -f -r /home/david/.ansible/tmp/ansible-local-12168UkWcRa/ansible-tmp-1547066739.99-193237508171269/ > /dev/null 2>&1 && sleep 0'
ok: [vsrx0.local] => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "commands": [
                "show version"
            ], 
            "display": "json", 
            "host": null, 
            "interval": 1, 
            "match": "all", 
            "password": null, 
            "port": null, 
            "provider": null, 
            "retries": 10, 
            "rpcs": null, 
            "ssh_keyfile": null, 
            "timeout": null, 
            "transport": null, 
            "username": null, 
            "wait_for": null
        }
    }, 
    "stdout": [
        "{\n    \"multi-routing-engine-results\" : [\n    {\n        \"multi-routing-engine-item\" : [\n        {\n            \"re-name\" : [\n            {\n                \"data\" : \"node0\"\n            }\n            ], \n            \"software-information\" : [\n            {\n                \"host-name\" : [\n                {\n                    \"data\" : \"vsrx0\"\n                }\n                ], \n                \"product-model\" : [\n                {\n                    \"data\" : \"vsrx\"\n                }\n                ], \n                \"product-name\" : [\n                {\n                    \"data\" : \"vsrx\"\n                }\n                ], \n                \"junos-version\" : [\n                {\n                    \"data\" : \"15.1X49-D160.2\"\n                }\n                ], \n                \"package-information\" : [\n                {\n                    \"name\" : [\n                    {\n                        \"data\" : \"junos\"\n                    }\n                    ], \n                    \"comment\" : [\n                    {\n                        \"data\" : \"JUNOS Software Release [15.1X49-D160.2]\"\n                    }\n                    ]\n                }\n                ]\n            }\n            ]\n        }, \n        {\n            \"re-name\" : [\n            {\n                \"data\" : \"node1\"\n            }\n            ], \n            \"software-information\" : [\n            {\n                \"host-name\" : [\n                {\n                    \"data\" : \"vsrx0\"\n                }\n                ], \n                \"product-model\" : [\n                {\n                    \"data\" : \"vsrx\"\n                }\n                ], \n                \"product-name\" : [\n                {\n                    \"data\" : \"vsrx\"\n                }\n                ], \n                \"junos-version\" : [\n                {\n                    \"data\" : \"15.1X49-D160.2\"\n                }\n                ], \n                \"package-information\" : [\n                {\n                    \"name\" : [\n                    {\n                        \"data\" : \"junos\"\n                    }\n                    ], \n                    \"comment\" : [\n                    {\n                        \"data\" : \"JUNOS Software Release [15.1X49-D160.2]\"\n                    }\n                    ]\n                }\n                ]\n            }\n            ]\n        }\n        ]\n    }\n    ]\n}\n\n{primary:node0}"
    ], 
    "stdout_lines": [
        [
            "{", 
            "    \"multi-routing-engine-results\" : [", 
            "    {", 
            "        \"multi-routing-engine-item\" : [", 
            "        {", 
            "            \"re-name\" : [", 
            "            {", 
            "                \"data\" : \"node0\"", 
            "            }", 
            "            ], ", 
            "            \"software-information\" : [", 
            "            {", 
            "                \"host-name\" : [", 
            "                {", 
            "                    \"data\" : \"vsrx0\"", 
            "                }", 
            "                ], ", 
            "                \"product-model\" : [", 
            "                {", 
            "                    \"data\" : \"vsrx\"", 
            "                }", 
            "                ], ", 
            "                \"product-name\" : [", 
            "                {", 
            "                    \"data\" : \"vsrx\"", 
            "                }", 
            "                ], ", 
            "                \"junos-version\" : [", 
            "                {", 
            "                    \"data\" : \"15.1X49-D160.2\"", 
            "                }", 
            "                ], ", 
            "                \"package-information\" : [", 
            "                {", 
            "                    \"name\" : [", 
            "                    {", 
            "                        \"data\" : \"junos\"", 
            "                    }", 
            "                    ], ", 
            "                    \"comment\" : [", 
            "                    {", 
            "                        \"data\" : \"JUNOS Software Release [15.1X49-D160.2]\"", 
            "                    }", 
            "                    ]", 
            "                }", 
            "                ]", 
            "            }", 
            "            ]", 
            "        }, ", 
            "        {", 
            "            \"re-name\" : [", 
            "            {", 
            "                \"data\" : \"node1\"", 
            "            }", 
            "            ], ", 
            "            \"software-information\" : [", 
            "            {", 
            "                \"host-name\" : [", 
            "                {", 
            "                    \"data\" : \"vsrx0\"", 
            "                }", 
            "                ], ", 
            "                \"product-model\" : [", 
            "                {", 
            "                    \"data\" : \"vsrx\"", 
            "                }", 
            "                ], ", 
            "                \"product-name\" : [", 
            "                {", 
            "                    \"data\" : \"vsrx\"", 
            "                }", 
            "                ], ", 
            "                \"junos-version\" : [", 
            "                {", 
            "                    \"data\" : \"15.1X49-D160.2\"", 
            "                }", 
            "                ], ", 
            "                \"package-information\" : [", 
            "                {", 
            "                    \"name\" : [", 
            "                    {", 
            "                        \"data\" : \"junos\"", 
            "                    }", 
            "                    ], ", 
            "                    \"comment\" : [", 
            "                    {", 
            "                        \"data\" : \"JUNOS Software Release [15.1X49-D160.2]\"", 
            "                    }", 
            "                    ]", 
            "                }", 
            "                ]", 
            "            }", 
            "            ]", 
            "        }", 
            "        ]", 
            "    }", 
            "    ]", 
            "}", 
            "", 
            "{primary:node0}"
        ]
    ]
}

TASK [debug] ***********************************************************************************************************************************************
task path: /home/david/test/test_playbook.yml:44
<192.168.122.10> attempting to start connection
<192.168.122.10> using connection plugin network_cli
<192.168.122.10> found existing local domain socket, using it!
<192.168.122.10> updating play_context for connection
<192.168.122.10> 
<192.168.122.10> local domain socket path is /home/david/.ansible/pc/bbbda4105f
ok: [vsrx0.local] => {
    "response.stdout_lines[0]": [
        "{", 
        "    \"multi-routing-engine-results\" : [", 
        "    {", 
        "        \"multi-routing-engine-item\" : [", 
        "        {", 
        "            \"re-name\" : [", 
        "            {", 
        "                \"data\" : \"node0\"", 
        "            }", 
        "            ], ", 
        "            \"software-information\" : [", 
        "            {", 
        "                \"host-name\" : [", 
        "                {", 
        "                    \"data\" : \"vsrx0\"", 
        "                }", 
        "                ], ", 
        "                \"product-model\" : [", 
        "                {", 
        "                    \"data\" : \"vsrx\"", 
        "                }", 
        "                ], ", 
        "                \"product-name\" : [", 
        "                {", 
        "                    \"data\" : \"vsrx\"", 
        "                }", 
        "                ], ", 
        "                \"junos-version\" : [", 
        "                {", 
        "                    \"data\" : \"15.1X49-D160.2\"", 
        "                }", 
        "                ], ", 
        "                \"package-information\" : [", 
        "                {", 
        "                    \"name\" : [", 
        "                    {", 
        "                        \"data\" : \"junos\"", 
        "                    }", 
        "                    ], ", 
        "                    \"comment\" : [", 
        "                    {", 
        "                        \"data\" : \"JUNOS Software Release [15.1X49-D160.2]\"", 
        "                    }", 
        "                    ]", 
        "                }", 
        "                ]", 
        "            }", 
        "            ]", 
        "        }, ", 
        "        {", 
        "            \"re-name\" : [", 
        "            {", 
        "                \"data\" : \"node1\"", 
        "            }", 
        "            ], ", 
        "            \"software-information\" : [", 
        "            {", 
        "                \"host-name\" : [", 
        "                {", 
        "                    \"data\" : \"vsrx0\"", 
        "                }", 
        "                ], ", 
        "                \"product-model\" : [", 
        "                {", 
        "                    \"data\" : \"vsrx\"", 
        "                }", 
        "                ], ", 
        "                \"product-name\" : [", 
        "                {", 
        "                    \"data\" : \"vsrx\"", 
        "                }", 
        "                ], ", 
        "                \"junos-version\" : [", 
        "                {", 
        "                    \"data\" : \"15.1X49-D160.2\"", 
        "                }", 
        "                ], ", 
        "                \"package-information\" : [", 
        "                {", 
        "                    \"name\" : [", 
        "                    {", 
        "                        \"data\" : \"junos\"", 
        "                    }", 
        "                    ], ", 
        "                    \"comment\" : [", 
        "                    {", 
        "                        \"data\" : \"JUNOS Software Release [15.1X49-D160.2]\"", 
        "                    }", 
        "                    ]", 
        "                }", 
        "                ]", 
        "            }", 
        "            ]", 
        "        }", 
        "        ]", 
        "    }", 
        "    ]", 
        "}", 
        "", 
        "{primary:node0}"
    ]
}
META: ran handlers
META: ran handlers

PLAY RECAP *************************************************************************************************************************************************
vsrx0.local                : ok=2    changed=0    unreachable=0    failed=0   
```

Hints from issue discussion:
Files identified in the description:
* [lib/ansible/modules/network/junos/junos_command.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/network/junos/junos_command.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @Qalthos @ganeshrn @privateip
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
@dselders, just so you are aware we have a dedicated Working Group for network.
You can find other people interested in this in `#ansible-network` on Freenode IRC
For more information about communities, meetings and agendas see https://github.com/ansible/community

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: community_workgroups --->

Commit message: Fix junos terminal regex for cluster srx devices

Fixes #50726

*  Modify junos terminal regex to match for
   string `{primary:node0}` which is also part of the
   prompt.
*  Modify network_cli connection plugin to ignore
   multiple prompt matched lines.

Patch to review:
diff --git a/lib/ansible/plugins/connection/network_cli.py b/lib/ansible/plugins/connection/network_cli.py
index 93e6f749394e6e..099ebe80daf0f6 100644
--- a/lib/ansible/plugins/connection/network_cli.py
+++ b/lib/ansible/plugins/connection/network_cli.py
@@ -539,9 +539,15 @@ def _sanitize(self, resp, command=None):
         '''
         cleaned = []
         for line in resp.splitlines():
-            if (command and line.strip() == command.strip()) or self._matched_prompt.strip() in line:
+            if command and line.strip() == command.strip():
                 continue
-            cleaned.append(line)
+            ignore_line = False
+            for prompt in self._matched_prompt.strip().splitlines():
+                if prompt.strip() in line:
+                    ignore_line = True
+                    break
+            if not ignore_line:
+                cleaned.append(line)
         return b'\n'.join(cleaned).strip()
 
     def _find_prompt(self, response):
diff --git a/lib/ansible/plugins/terminal/junos.py b/lib/ansible/plugins/terminal/junos.py
index 735e8bdf37d9e4..8562581156bccd 100644
--- a/lib/ansible/plugins/terminal/junos.py
+++ b/lib/ansible/plugins/terminal/junos.py
@@ -32,7 +32,7 @@
 class TerminalModule(TerminalBase):
 
     terminal_stdout_re = [
-        re.compile(br"[\r\n]?[\w@+\-\.:\/\[\]]+[>#%] ?$"),
+        re.compile(br"({primary:node\d+})?[\r\n]?[\w@+\-\.:\/\[\]]+[>#%] ?$"),
     ]
 
     terminal_stderr_re = [


Code review comments:
diff_hunk: @@ -539,9 +539,15 @@ def _sanitize(self, resp, command=None):
         '''
         cleaned = []
         for line in resp.splitlines():
-            if (command and line.strip() == command.strip()) or self._matched_prompt.strip() in line:
+            if command and line.strip() == command.strip():
                 continue
-            cleaned.append(line)
+            ignore_line = False
+            for prompt in self._matched_prompt.strip().splitlines():
+                if prompt.strip() in line:
+                    ignore_line = True
+                    break
+            if not ignore_line:
path: lib/ansible/plugins/connection/network_cli.py
text: @user1:
```suggestion
            else:
```

I'm trying to make sure I understand what's happening here, will this change have the same intended effect?

(I don't know if anyone else likes for-elses as much as I do, but it would allow you to get rid of the `ignore_line` sentinel variable)

@author:
Yes `else` will have the same intended effect. Fixed your review comment. Thanks!