Repository: ansible/ansible

Issue:
Unable to use the junos_facts module with netconf connection
<!---
Verify first that your issue/request is not already reported on GitHub.
THIS FORM WILL BE READ BY A MACHINE, COMPLETE ALL SECTIONS AS DESCRIBED.
Also test if the latest release, and devel branch are affected too.
ALWAYS add information AFTER (OUTSIDE) these html comments.
Otherwise it may end up being automatically closed by our bot. -->

##### SUMMARY
Unable to use the junos_facts module with netconf connection.

Works fine when used with the old local connection and provider variable.

other junos_* modules seems to be working fine with netconf.

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
<!--- Insert, BELOW THIS COMMENT, the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path-->
junos_facts

##### ANSIBLE VERSION
<!--- Paste, BELOW THIS COMMENT, verbatim output from "ansible --version" between quotes below -->
```
ansible 2.6.0
  config file = /home/sveinov/ansible/ansible.cfg
  configured module search path = [u'/home/sveinov/ansible/library']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.15 (default, May  1 2018, 05:55:50) [GCC 7.3.0]

```

##### CONFIGURATION
<!--- If using Ansible 2.4 or above, paste, BELOW THIS COMMENT, the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).-->
DEFAULT_HOST_LIST(/home/sveinov/ansible/ansible.cfg) = [u'/home/sveinov/ansible/inventory']
DEFAULT_MODULE_PATH(/home/sveinov/ansible/ansible.cfg) = [u'/home/sveinov/ansible/library']
HOST_KEY_CHECKING(/home/sveinov/ansible/ansible.cfg) = False

##### OS / ENVIRONMENT
<!--- Mention, BELOW THIS COMMENT, the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.-->

Debian testing (buster), Juniper MX204 Junos 17.4

##### STEPS TO REPRODUCE
<!--- For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used. -->

<!--- Paste example playbooks or commands between quotes below -->

```
group_vars/ansible_vars.yml
```

```yaml
---
ansible_user: "ansible"
ansible_dir: "{{ lookup('env','HOME') }}/ansible"
ansible_network_os: "{{ os }}"
ansible_host: "{{ ip }}"
```
```
main.yml
```
```yaml
---
- name: Main site playbook
  hosts: all
  connection: netconf
  gather_facts: false

  tasks:
    - name: Gathering facts
      junos_facts:
        gather_subset: all
        config_format: json
      register: facts

```


<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS

Getting those sweet Junos facts.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
sveinov@causality:~/ansible$ ansible-playbook main.yml -vvvv
ansible-playbook 2.6.0
  config file = /home/sveinov/ansible/ansible.cfg
  configured module search path = [u'/home/sveinov/ansible/library']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 2.7.15 (default, May  1 2018, 05:55:50) [GCC 7.3.0]
Using /home/sveinov/ansible/ansible.cfg as config file
setting up inventory plugins
Parsed /home/sveinov/ansible/inventory/labb.yml inventory source with yaml plugin
Loading callback plugin default of type stdout, v2.0 from /usr/local/lib/python2.7/dist-packages/ansible/plugins/callback/default.pyc
 ____________________
< PLAYBOOK: main.yml >
 --------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

1 plays in main.yml
 ___________________________
< PLAY [Main site playbook] >
 ---------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

META: ran handlers
 ________________________
< TASK [Gathering facts] >
 ------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

task path: /home/sveinov/ansible/main.yml:9
<10.0.0.10> attempting to start connection
<10.0.0.10> using connection plugin netconf
<10.0.0.10> local domain socket does not exist, starting it
<10.0.0.10> control socket path is /home/sveinov/.ansible/pc/86ca832b46
<10.0.0.10> connection to remote device started successfully
<10.0.0.10> local domain socket listeners started successfully
<10.0.0.10> 
<10.0.0.10> local domain socket path is /home/sveinov/.ansible/pc/86ca832b46
<10.0.0.10> ESTABLISH LOCAL CONNECTION FOR USER: sveinov
<10.0.0.10> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/sveinov/.ansible/tmp/ansible-local-3634_KSsU6/ansible-tmp-1530696521.94-262612295735323 `" && echo ansible-tmp-1530696521.94-262612295735323="` echo /home/sveinov/.ansible/tmp/ansible-local-3634_KSsU6/ansible-tmp-1530696521.94-262612295735323 `" ) && sleep 0'
Using module file /usr/local/lib/python2.7/dist-packages/ansible/modules/network/junos/junos_facts.py
<10.0.0.10> PUT /home/sveinov/.ansible/tmp/ansible-local-3634_KSsU6/tmphv8FzM TO /home/sveinov/.ansible/tmp/ansible-local-3634_KSsU6/ansible-tmp-1530696521.94-262612295735323/junos_facts.py
<10.0.0.10> EXEC /bin/sh -c 'chmod u+x /home/sveinov/.ansible/tmp/ansible-local-3634_KSsU6/ansible-tmp-1530696521.94-262612295735323/ /home/sveinov/.ansible/tmp/ansible-local-3634_KSsU6/ansible-tmp-1530696521.94-262612295735323/junos_facts.py && sleep 0'
<10.0.0.10> EXEC /bin/sh -c '/usr/bin/python /home/sveinov/.ansible/tmp/ansible-local-3634_KSsU6/ansible-tmp-1530696521.94-262612295735323/junos_facts.py && sleep 0'
<10.0.0.10> EXEC /bin/sh -c 'rm -f -r /home/sveinov/.ansible/tmp/ansible-local-3634_KSsU6/ansible-tmp-1530696521.94-262612295735323/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_WHmIeb/ansible_module_junos_facts.py", line 379, in <module>
    main()
  File "/tmp/ansible_WHmIeb/ansible_module_junos_facts.py", line 371, in main
    ansible_facts.update(Facts(module).populate())
  File "/tmp/ansible_WHmIeb/ansible_module_junos_facts.py", line 275, in populate
    device = self._connect(self.module)
  File "/tmp/ansible_WHmIeb/ansible_module_junos_facts.py", line 265, in _connect
    device = Device(host, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/jnpr/junos/device.py", line 1140, in __init__
    raise ValueError("You must provide the 'host' value")
ValueError: You must provide the 'host' value

fatal: [mx204]: FAILED! => {
    "changed": false, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_WHmIeb/ansible_module_junos_facts.py\", line 379, in <module>\n    main()\n  File \"/tmp/ansible_WHmIeb/ansible_module_junos_facts.py\", line 371, in main\n    ansible_facts.update(Facts(module).populate())\n  File \"/tmp/ansible_WHmIeb/ansible_module_junos_facts.py\", line 275, in populate\n    device = self._connect(self.module)\n  File \"/tmp/ansible_WHmIeb/ansible_module_junos_facts.py\", line 265, in _connect\n    device = Device(host, **kwargs)\n  File \"/usr/local/lib/python2.7/dist-packages/jnpr/junos/device.py\", line 1140, in __init__\n    raise ValueError(\"You must provide the 'host' value\")\nValueError: You must provide the 'host' value\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "rc": 1
}
	to retry, use: --limit @/home/sveinov/ansible/main.retry
 ____________
< PLAY RECAP >
 ------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

mx204                      : ok=0    changed=0    unreachable=0    failed=1   

```



Discussion:
Files identified in the description:
* [lib/ansible/modules/network/junos/junos_facts.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/network/junos/junos_facts.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @Qalthos @ganeshrn @qalthos
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
`gather_subset: all` with gather old style facts that will not have `ansible_` prefix to fact keys.
If you don't want old style fact you set gather_subset to 
```
gather_subset:
  - default
  - hardware
  - config
  - interfaces
```

In case you want to fetch old style facts also which uses [junos_eznc library](https://github.com/Juniper/py-junos-eznc) to fetch facts internally you need to set `provider` option within task and `connection=local`
```
  tasks:
    - name: Gathering facts
      junos_facts:
        gather_subset: all
        config_format: json
        provider:
          host: <hostname>
          port: <port>
          username: <username>
          password: <password>
      register: facts
```

This behavior needs to be documented as part of the fix for this issue and make `ofacts` as one of the options for `gather_subsets`

Resolving patch:
diff --git a/lib/ansible/modules/network/junos/junos_facts.py b/lib/ansible/modules/network/junos/junos_facts.py
--- a/lib/ansible/modules/network/junos/junos_facts.py
+++ b/lib/ansible/modules/network/junos/junos_facts.py
@@ -36,9 +36,13 @@
         with an initial C(M(!)) to specify that a specific subset should
         not be collected. To maintain backward compatbility old style facts
         can be retrieved using all value, this reqires junos-eznc to be installed
-        as a prerequisite.
+        as a prerequisite. Valid value of gather_subset are default, hardware,
+        config, interfaces, ofacts. If C(ofacts) is present in the list it fetches
+        the old style facts (fact keys without 'ansible_' prefix) and it requires
+        junos-eznc library to be installed on control node and the device login credentials
+        must be given in C(provider) option.
     required: false
-    default: "!config"
+    default: ['!config', '!ofacts']
     version_added: "2.3"
   config_format:
     description:
@@ -46,10 +50,11 @@
          when serializing output from the device. This argument is applicable
          only when C(config) value is present in I(gather_subset).
          The I(config_format) should be supported by the junos version running on
-         device.
+         device. This value is not applicable while fetching old style facts that is
+         when value of I(gather_subset) C(all) or C(ofacts) is present in the value.
     required: false
-    default: text
-    choices: ['xml', 'set', 'text', 'json']
+    default: 'text'
+    choices: ['xml', 'text', 'set', 'json']
     version_added: "2.3"
 requirements:
   - ncclient (>=v0.5.2)
@@ -62,6 +67,8 @@
   - Tested against vSRX JUNOS version 15.1X49-D15.4, vqfx-10000 JUNOS Version 15.1X53-D60.4.
   - Recommended connection is C(netconf). See L(the Junos OS Platform Options,../network/user_guide/platform_junos.html).
   - This module also works with C(local) connections for legacy playbooks.
+  - Fetching old style facts requires junos-eznc library to be installed on control node and the device login credentials
+    must be given in provider option.
 """
 
 EXAMPLES = """
@@ -244,7 +251,7 @@ def populate(self):
         self.facts['interfaces'] = interfaces
 
 
-class Facts(FactsBase):
+class OFacts(FactsBase):
     def _connect(self, module):
         host = get_param(module, 'host')
 
@@ -260,7 +267,6 @@ def _connect(self, module):
             kwargs['ssh_private_key_file'] = get_param(module, 'ssh_keyfile')
 
         kwargs['gather_facts'] = False
-
         try:
             device = Device(host, **kwargs)
             device.open()
@@ -292,7 +298,8 @@ def populate(self):
     default=Default,
     hardware=Hardware,
     config=Config,
-    interfaces=Interfaces
+    interfaces=Interfaces,
+    ofacts=OFacts
 )
 
 VALID_SUBSETS = frozenset(FACT_SUBSETS.keys())
@@ -302,7 +309,7 @@ def main():
     """ Main entry point for AnsibleModule
     """
     argument_spec = dict(
-        gather_subset=dict(default=['!config'], type='list'),
+        gather_subset=dict(default=['!config', '!ofacts'], type='list'),
         config_format=dict(default='text', choices=['xml', 'text', 'set', 'json']),
     )
 
@@ -314,7 +321,6 @@ def main():
     get_connection(module)
     warnings = list()
     gather_subset = module.params['gather_subset']
-    ofacts = False
 
     runable_subsets = set()
     exclude_subsets = set()
@@ -322,14 +328,12 @@ def main():
     for subset in gather_subset:
         if subset == 'all':
             runable_subsets.update(VALID_SUBSETS)
-            ofacts = True
             continue
 
         if subset.startswith('!'):
             subset = subset[1:]
             if subset == 'all':
                 exclude_subsets.update(VALID_SUBSETS)
-                ofacts = False
                 continue
             exclude = True
         else:
@@ -354,6 +358,16 @@ def main():
     facts['gather_subset'] = list(runable_subsets)
 
     instances = list()
+    ansible_facts = dict()
+
+    if 'ofacts' in runable_subsets:
+        if HAS_PYEZ:
+            ansible_facts.update(OFacts(module).populate())
+        else:
+            warnings += ['junos-eznc is required to gather old style facts but does not appear to be installed. '
+                         'It can be installed using `pip  install junos-eznc`']
+        runable_subsets.remove('ofacts')
+
     for key in runable_subsets:
         instances.append(FACT_SUBSETS[key](module))
 
@@ -361,17 +375,10 @@ def main():
         inst.populate()
         facts.update(inst.facts)
 
-    ansible_facts = dict()
     for key, value in iteritems(facts):
         key = 'ansible_net_%s' % key
         ansible_facts[key] = value
 
-    if ofacts:
-        if HAS_PYEZ:
-            ansible_facts.update(Facts(module).populate())
-        else:
-            warnings += ['junos-eznc is required to gather old style facts but does not appear to be installed. '
-                         'It can be installed using `pip  install junos-eznc`']
     module.exit_json(ansible_facts=ansible_facts, warnings=warnings)
 
 
diff --git a/lib/ansible/plugins/action/junos.py b/lib/ansible/plugins/action/junos.py
--- a/lib/ansible/plugins/action/junos.py
+++ b/lib/ansible/plugins/action/junos.py
@@ -91,8 +91,11 @@ def run(self, tmp=None, task_vars=None):
         elif self._play_context.connection in ('netconf', 'network_cli'):
             provider = self._task.args.get('provider', {})
             if any(provider.values()):
-                display.warning('provider is unnecessary when using %s and will be ignored' % self._play_context.connection)
-                del self._task.args['provider']
+                # for legacy reasons provider value is required for junos_facts(optional) and junos_package
+                # modules as it uses junos_eznc library to connect to remote host
+                if not (self._task.action == 'junos_facts' or self._task.action == 'junos_package'):
+                    display.warning('provider is unnecessary when using %s and will be ignored' % self._play_context.connection)
+                    del self._task.args['provider']
 
             if (self._play_context.connection == 'network_cli' and self._task.action not in CLI_SUPPORTED_MODULES) or \
                     (self._play_context.connection == 'netconf' and self._task.action == 'junos_netconf'):


Test patch: