Repository: ansible/ansible

Issue:
junos_command: cannot get output of a command
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
junos_command

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.3.1.0
  config file = /home/app/saint/ansible/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.5 (default, Nov  6 2016, 00:28:07) [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]
```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
```
host_key_checking = False
```

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->

##### SUMMARY
<!--- Explain the problem briefly -->
Cannot get command output from a Juniper device.

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

`ansible-playbook dev-conf.yml --extra-vars "dev=athens os=juniper"`

<!--- Paste example playbooks or commands between quotes below -->
```yaml
---

  - name: dev-conf
    hosts: LOCAL
    roles:
      - common


    tasks:

      - name: "Retrieve router's configuration: junos"
        connection: local
        junos_command:
          commands: "show configuration | display set | no-more"
          provider:
            host: "{{ dev }}"
            port: 22
            username: "{{ lookup('env','USR_NETOPS') }}"
            password: "{{ lookup('env','PASS_NETOPS') }}"
        when: dev is defined and os is defined and os == "juniper"
        register: _

      - debug: msg="{{ _ }}"
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
Get router's configuration in `display set` format

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
$ ansible-playbook dev-conf.yml --extra-vars "dev=athens os=juniper" -vvvv
Using /home/app/saint/ansible/ansible.cfg as config file
Loading callback plugin default of type stdout, v2.0 from /usr/lib/python2.7/site-packages/ansible/plugins/callback/__init__.pyc

PLAYBOOK: dev-conf.yml *******************************************************************************************************************************
1 plays in dev-conf.yml

PLAY [dev-conf] **************************************************************************************************************************************

TASK [Gathering Facts] *******************************************************************************************************************************
Using module file /usr/lib/python2.7/site-packages/ansible/modules/system/setup.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: app
<localhost> EXEC /bin/sh -c 'echo ~ && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/app/.ansible/tmp/ansible-tmp-1499090338.08-16740590805618 `" && echo ansible-tmp-1499090338.08-16740590805618="` echo /home/app/.ansible/tmp/ansible-tmp-1499090338.08-16740590805618 `" ) && sleep 0'
<localhost> PUT /tmp/tmpMTpD5T TO /home/app/.ansible/tmp/ansible-tmp-1499090338.08-16740590805618/setup.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/app/.ansible/tmp/ansible-tmp-1499090338.08-16740590805618/ /home/app/.ansible/tmp/ansible-tmp-1499090338.08-16740590805618/setup.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /home/app/.ansible/tmp/ansible-tmp-1499090338.08-16740590805618/setup.py; rm -rf "/home/app/.ansible/tmp/ansible-tmp-1499090338.08-16740590805618/" > /dev/null 2>&1 && sleep 0'
ok: [localhost]
META: ran handlers

TASK [Retrieve router's configuration: junos] ********************************************************************************************************
task path: /home/app/saint/ansible/dev-conf.yml:12
<athens> using connection plugin netconf
<athens> socket_path: /home/app/.ansible/pc/d090e15c31
open_session() returned 0 ok 
Using module file /usr/lib/python2.7/site-packages/ansible/modules/network/junos/junos_command.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: app
<localhost> EXEC /bin/sh -c 'echo ~ && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/app/.ansible/tmp/ansible-tmp-1499090340.43-6328802200596 `" && echo ansible-tmp-1499090340.43-6328802200596="` echo /home/app/.ansible/tmp/ansible-tmp-1499090340.43-6328802200596 `" ) && sleep 0'
<localhost> PUT /tmp/tmpGOR9Z2 TO /home/app/.ansible/tmp/ansible-tmp-1499090340.43-6328802200596/junos_command.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/app/.ansible/tmp/ansible-tmp-1499090340.43-6328802200596/ /home/app/.ansible/tmp/ansible-tmp-1499090340.43-6328802200596/junos_command.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /home/app/.ansible/tmp/ansible-tmp-1499090340.43-6328802200596/junos_command.py; rm -rf "/home/app/.ansible/tmp/ansible-tmp-1499090340.43-6328802200596/" > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_t7Cbbz/ansible_module_junos_command.py", line 388, in <module>
    main()
  File "/tmp/ansible_t7Cbbz/ansible_module_junos_command.py", line 338, in main
    responses = rpc(module, items)
  File "/tmp/ansible_t7Cbbz/ansible_module_junos_command.py", line 229, in rpc
    responses.append(data.text.strip())
AttributeError: 'NoneType' object has no attribute 'text'

fatal: [localhost]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_t7Cbbz/ansible_module_junos_command.py\", line 388, in <module>\n    main()\n  File \"/tmp/ansible_t7Cbbz/ansible_module_junos_command.py\", line 338, in main\n    responses = rpc(module, items)\n  File \"/tmp/ansible_t7Cbbz/ansible_module_junos_command.py\", line 229, in rpc\n    responses.append(data.text.strip())\nAttributeError: 'NoneType' object has no attribute 'text'\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "rc": 0
}
	to retry, use: --limit @/home/app/saint/ansible/dev-conf.retry

PLAY RECAP *******************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1  
```


Discussion:
cc @Qalthos @ganeshrn
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->

Resolving patch:
diff --git a/lib/ansible/module_utils/netconf.py b/lib/ansible/module_utils/netconf.py
--- a/lib/ansible/module_utils/netconf.py
+++ b/lib/ansible/module_utils/netconf.py
@@ -37,7 +37,7 @@
 NS_MAP = {'nc': "urn:ietf:params:xml:ns:netconf:base:1.0"}
 
 
-def send_request(module, obj, check_rc=True):
+def send_request(module, obj, check_rc=True, ignore_warning=True):
     request = tostring(obj)
     rc, out, err = exec_command(module, request)
     if rc != 0 and check_rc:
@@ -54,7 +54,7 @@ def send_request(module, obj, check_rc=True):
             message = rpc_error.find('./nc:error-message', NS_MAP).text
             severity = rpc_error.find('./nc:error-severity', NS_MAP).text
 
-            if severity == 'warning':
+            if severity == 'warning' and ignore_warning:
                 warnings.append(message)
             else:
                 module.fail_json(msg=str(err))
diff --git a/lib/ansible/modules/network/junos/junos_command.py b/lib/ansible/modules/network/junos/junos_command.py
--- a/lib/ansible/modules/network/junos/junos_command.py
+++ b/lib/ansible/modules/network/junos/junos_command.py
@@ -96,11 +96,12 @@
         This handles how to properly understand the output and apply the
         conditionals path to the result set. For I(rpcs) argument default
         display is C(xml) and for I(commands) argument default display
-        is C(text).
+        is C(text). Value C(set) is applicable only for fetching configuration
+        from device.
     required: false
     default: depends on input argument I(rpcs) or I(commands)
     aliases: ['format', 'output']
-    choices: ['text', 'json', 'xml']
+    choices: ['text', 'json', 'xml', 'set']
     version_added: "2.3"
 requirements:
   - jxmlease
@@ -142,8 +143,12 @@
 
 - name: run rpc on the remote device
   junos_command:
-    rpcs: get-software-information
+    commands: show configuration
+    display: set
 
+- name: run rpc on the remote device
+  junos_command:
+    rpcs: get-software-information
 """
 
 RETURN = """
@@ -173,7 +178,7 @@
 import shlex
 
 from ansible.module_utils.basic import AnsibleModule
-from ansible.module_utils.junos import junos_argument_spec, check_args
+from ansible.module_utils.junos import junos_argument_spec, check_args, get_configuration
 from ansible.module_utils.netcli import Conditional, FailedConditionalError
 from ansible.module_utils.netconf import send_request
 from ansible.module_utils.six import string_types, iteritems
@@ -208,6 +213,7 @@ def rpc(module, items):
     for item in items:
         name = item['name']
         xattrs = item['xattrs']
+        fetch_config = False
 
         args = item.get('args')
         text = item.get('text')
@@ -217,6 +223,9 @@ def rpc(module, items):
         if all((module.check_mode, not name.startswith('get'))):
             module.fail_json(msg='invalid rpc for running in check_mode')
 
+        if name == 'command' and text.startswith('show configuration') or name == 'get-configuration':
+            fetch_config = True
+
         element = Element(name, xattrs)
 
         if text:
@@ -235,15 +244,31 @@ def rpc(module, items):
                     if value is not True:
                         child.text = value
 
-        reply = send_request(module, element)
+        if fetch_config:
+            reply = get_configuration(module, format=xattrs['format'])
+        else:
+            reply = send_request(module, element, ignore_warning=False)
 
         if xattrs['format'] == 'text':
-            data = reply.find('.//output')
+            if fetch_config:
+                data = reply.find('.//configuration-text')
+            else:
+                data = reply.find('.//output')
+
+            if data is None:
+                module.fail_json(msg=tostring(reply))
+
             responses.append(data.text.strip())
 
         elif xattrs['format'] == 'json':
             responses.append(module.from_json(reply.text.strip()))
 
+        elif xattrs['format'] == 'set':
+            data = reply.find('.//configuration-set')
+            if data is None:
+                module.fail_json(msg="Display format 'set' is not supported by remote device.")
+            responses.append(data.text.strip())
+
         else:
             responses.append(tostring(reply))
 
@@ -277,8 +302,11 @@ def parse_rpcs(module):
                 args[key] = str(value)
 
         display = module.params['display'] or 'xml'
-        xattrs = {'format': display}
 
+        if display == 'set' and rpc != 'get-configuration':
+            module.fail_json(msg="Invalid display option '%s' given for rpc '%s'" % ('set', name))
+
+        xattrs = {'format': display}
         items.append({'name': name, 'args': args, 'xattrs': xattrs})
 
     return items
@@ -298,14 +326,20 @@ def parse_commands(module, warnings):
         text = parts[0]
 
         display = module.params['display'] or 'text'
-        xattrs = {'format': display}
 
         if '| display json' in command:
-            xattrs['format'] = 'json'
+            display = 'json'
 
         elif '| display xml' in command:
-            xattrs['format'] = 'xml'
+            display = 'xml'
 
+        if display == 'set' or '| display set' in command:
+            if command.startswith('show configuration'):
+                display = 'set'
+            else:
+                module.fail_json(msg="Invalid display option '%s' given for command '%s'" % ('set', command))
+
+        xattrs = {'format': display}
         items.append({'name': 'command', 'xattrs': xattrs, 'text': text})
 
     return items
@@ -318,7 +352,7 @@ def main():
         commands=dict(type='list'),
         rpcs=dict(type='list'),
 
-        display=dict(choices=['text', 'json', 'xml'], aliases=['format', 'output']),
+        display=dict(choices=['text', 'json', 'xml', 'set'], aliases=['format', 'output']),
 
         wait_for=dict(type='list', aliases=['waitfor']),
         match=dict(default='all', choices=['all', 'any']),
@@ -351,7 +385,6 @@ def main():
 
     while retries > 0:
         responses = rpc(module, items)
-
         transformed = list()
         output = list()
         for item, resp in zip(items, responses):


Test patch: