Repository: ansible/ansible

Issue:
junos_facts not backward compatible?  missing "personality" fact
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
junos_facts

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
$ ansible --version
ansible 2.3.0.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.13 (default, May 10 2017, 20:04:28) [GCC 6.3.1 20161221 (Red Hat 6.3.1-1)]
```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
-->
Fedora release 25 (Twenty Five)

##### SUMMARY
<!--- Explain the problem briefly -->
The junos_facts provided by Ansible 2.3 are not backward compatible with 2.2.
For example, we use the "personality" fact as decision check.  With 2.3, "personality" is not available.


##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
Upgrade from 2.2 to 2.3 and try running the same playbook

<!--- Paste example playbooks or commands between quotes below -->
```yaml
#WORKS in 2.2
- name: template MX jflow configuration
  always_run: yes
  when:
    - inventory_hostname != "localhost"
    - personality == "MX"
  template:
    src: mx_jflow.conf.j2
    dest: "{{ juniper_build_path }}/mx_jflow.conf"

```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
Expected the playbook to work as it did under 2.2.  junos_get_facts listed below:

```


ok: [eugn-p2-gw.nero.net] => {
    "changed": false, 
    "facts": {
        "HOME": "/var/home/SCRUBBED", 
        "RE0": {
            "last_reboot_reason": "0x2000:hypervisor reboot", 
            "mastership_state": "master", 
            "model": "RE-QFX10002-36Q", 
            "status": "OK", 
            "up_time": "282 days, 23 hours, 29 minutes, 21 seconds"
        }, 
        "RE1": null, 
        "RE_hw_mi": false, 
        "current_re": [
            "master", 
            "node", 
            "fwdd", 
            "member", 
            "pfem", 
            "re0", 
            "feb0", 
            "fpc16"
        ], 
        "domain": "nero.net", 
        "fqdn": "eugn-p2-gw.nero.net", 
        "has_2RE": false, 
        "hostname": "eugn-p2-gw", 
        "hostname_info": {
            "re0": "eugn-p2-gw"
        }, 
        "ifd_style": "SWITCH", 
        "junos_info": {
            "re0": {
                "object": {
                    "build": 3, 
                    "major": [
                        15, 
                        1
                    ], 
                    "minor": [
                        53, 
                        "D", 
                        33
                    ], 
                    "type": "X"
                }, 
                "text": "15.1X53-D33.3"
            }
        }, 
        "master": "RE0", 
        "model": "QFX10002-36Q", 
        "model_info": {
            "re0": "QFX10002-36Q"
        }, 
        "personality": "SWITCH", 
        "re_info": {
            "default": {
                "0": {
                    "last_reboot_reason": "0x2000:hypervisor reboot", 
                    "mastership_state": "master", 
                    "model": "RE-QFX10002-36Q", 
                    "status": "OK"
                }, 
                "default": {
                    "last_reboot_reason": "0x2000:hypervisor reboot", 
                    "mastership_state": "master", 
                    "model": "RE-QFX10002-36Q", 
                    "status": "OK"
                }
            }
        }, 
        "re_master": {
            "default": "0"
        }, 
        "serialnumber": "SCRUBBED", 
        "srx_cluster": null, 
        "srx_cluster_id": null, 
        "switch_style": "VLAN_L2NG", 
        "vc_capable": true, 
        "vc_fabric": false, 
        "vc_master": "0", 
        "vc_mode": "Enabled", 
        "version": "15.1X53-D33.3", 
        "version_RE0": "15.1X53-D33.3", 
        "version_RE1": null, 
        "version_info": {
            "build": 3, 
            "major": [
                15, 
                1
            ], 
            "minor": [
                53, 
                "D", 
                33
            ], 
            "type": "X"
        }, 
        "virtual": false
    }, 
    "invocation": {
        "module_args": {
            "console": null, 
            "host": "eugn-p2-gw.nero.net", 
            "logfile": null, 
            "mode": null, 
            "passwd": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "port": "830", 
            "savedir": "/tmp/junos", 
            "ssh_private_key_file": null, 
            "user": "SCRUBBED"
        }
    }
}
```

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
ERROR RECEIVED
```
fatal: [eugn-p2-gw]: FAILED! => {
    "failed": true, 
    "msg": "The conditional check 'personality == \"MX\"' failed. The error was: error while evaluating conditional (personality == \"MX\"): 'personality' is undefined\n\nThe error appears to have been in '/home/user/Repo/playbooks/roles/juniper/tasks/configuration.yml': line 44, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: template MX jflow configuration\n  ^ here\n"
}

```
Facts provided by Ansible junos_facts
```
{
    "ansible_facts": {
        "ansible_net_filesystems": [
            "/dev/ad0s1a", 
            "devfs", 
            "/dev/md0", 
            "/dev/md1", 
            "/dev/md2", 
            "/dev/md3", 
            "/dev/md4", 
            "/dev/md5", 
            "/dev/md6", 
            "/dev/md7", 
            "/dev/md8", 
            "/dev/md9", 
            "/dev/ad2s1e", 
            "procfs", 
            "/dev/ad1s1f", 
            "/dev/ad1s1d", 
            "/dev/ad1s1a", 
            "/dev/ad1s1e", 
            "/dev/ad3s2", 
            "/dev/ad3s1", 
            "/var/jails/rest-api", 
            "/dev/md10"
        ], 
        "ansible_net_gather_subset": [
            "hardware", 
            "default", 
            "interfaces"
        ], 
        "ansible_net_hostname": "eugn-p2-gw", 
        "ansible_net_interfaces": {
            ".local.": {
                "admin-status": "up", 
                "macaddress": "Unspecified", 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "Unlimited", 
                "type": "Loopback"
            }, 
            "bme0": {
                "admin-status": "up", 
                "macaddress": "02:00:00:00:00:0a", 
                "mtu": "9512", 
                "oper-status": "up", 
                "speed": "Unspecified", 
                "type": "Ethernet"
            }, 
            "dsc": {
                "admin-status": "up", 
                "macaddress": "Unspecified", 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "Unspecified", 
                "type": "Software-Pseudo"
            }, 
            "em0": {
                "admin-status": "down", 
                "macaddress": "80:ac:ac:90:7f:f8", 
                "mtu": "1514", 
                "oper-status": "down", 
                "speed": "1000mbps", 
                "type": "Ethernet"
            }, 
            "em1": {
                "admin-status": "up", 
                "macaddress": "80:ac:ac:90:7f:f9", 
                "mtu": "1514", 
                "oper-status": "down", 
                "speed": "Unspecified", 
                "type": "Ethernet"
            }, 
            "em2": {
                "admin-status": "up", 
                "macaddress": "80:ac:ac:90:7f:f0", 
                "mtu": "1514", 
                "oper-status": "up", 
                "speed": "1000mbps", 
                "type": "Ethernet"
            }, 
            "em3": {
                "admin-status": "up", 
                "macaddress": "80:ac:ac:90:7f:f1", 
                "mtu": "9512", 
                "oper-status": "up", 
                "speed": "1000mbps", 
                "type": "Ethernet"
            }, 
            "esi": {
                "admin-status": "up", 
                "macaddress": "Unspecified", 
                "mtu": "1600", 
                "oper-status": "up", 
                "speed": "Unlimited", 
                "type": "Software-Pseudo"
            }, 
            "et-0/0/1": {
                "admin-status": "up", 
                "macaddress": "80:ac:ac:90:70:07", 
                "mtu": "9216", 
                "oper-status": "up", 
                "speed": "100Gbps", 
                "type": null
            }, 
            "gr-0/0/0": {
                "admin-status": "up", 
                "macaddress": null, 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "800mbps", 
                "type": "GRE"
            }, 
            "gre": {
                "admin-status": "up", 
                "macaddress": null, 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "Unlimited", 
                "type": "GRE"
            }, 
            "ipip": {
                "admin-status": "up", 
                "macaddress": null, 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "Unlimited", 
                "type": "IPIP"
            }, 
            "irb": {
                "admin-status": "up", 
                "macaddress": "80:ac:ac:90:70:00", 
                "mtu": "1514", 
                "oper-status": "up", 
                "speed": "Unspecified", 
                "type": "Ethernet"
            }, 
            "lo0": {
                "admin-status": "up", 
                "macaddress": "Unspecified", 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "Unspecified", 
                "type": "Loopback"
            }, 
            "lsi": {
                "admin-status": "up", 
                "macaddress": "Unspecified", 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "Unlimited", 
                "type": "Software-Pseudo"
            }, 
            "mtun": {
                "admin-status": "up", 
                "macaddress": null, 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "Unlimited", 
                "type": "Multicast-GRE"
            }, 
            "pfe-0/0/0": {
                "admin-status": "up", 
                "macaddress": "Unspecified", 
                "mtu": "0", 
                "oper-status": "up", 
                "speed": "Unspecified", 
                "type": "Unspecified"
            }, 
            "pfh-0/0/0": {
                "admin-status": "up", 
                "macaddress": "Unspecified", 
                "mtu": "0", 
                "oper-status": "up", 
                "speed": "Unspecified", 
                "type": "Unspecified"
            }, 
            "pimd": {
                "admin-status": "up", 
                "macaddress": null, 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "Unlimited", 
                "type": "PIMD"
            }, 
            "pime": {
                "admin-status": "up", 
                "macaddress": null, 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "Unlimited", 
                "type": "PIME"
            }, 
            "tap": {
                "admin-status": "up", 
                "macaddress": "Unspecified", 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "Unlimited", 
                "type": "Software-Pseudo"
            }, 
            "vtep": {
                "admin-status": "up", 
                "macaddress": "Unspecified", 
                "mtu": "1600", 
                "oper-status": "up", 
                "speed": "Unlimited", 
                "type": "Software-Pseudo"
            }, 
            "xe-0/0/34:0": {
                "admin-status": "up", 
                "macaddress": "80:ac:ac:90:70:8b", 
                "mtu": "9216", 
                "oper-status": "up", 
                "speed": "10Gbps", 
                "type": null
            }, 
            "xe-0/0/34:1": {
                "admin-status": "up", 
                "macaddress": "80:ac:ac:90:70:8c", 
                "mtu": "9216", 
                "oper-status": "up", 
                "speed": "10Gbps", 
                "type": null
            }, 
            "xe-0/0/34:2": {
                "admin-status": "up", 
                "macaddress": "80:ac:ac:90:70:8d", 
                "mtu": "9212", 
                "oper-status": "up", 
                "speed": "10Gbps", 
                "type": null
            }, 
            "xe-0/0/34:3": {
                "admin-status": "up", 
                "macaddress": "80:ac:ac:90:70:8e", 
                "mtu": "9216", 
                "oper-status": "up", 
                "speed": "10Gbps", 
                "type": null
            }, 
            "xe-0/0/35:0": {
                "admin-status": "up", 
                "macaddress": "80:ac:ac:90:70:8f", 
                "mtu": "9216", 
                "oper-status": "up", 
                "speed": "10Gbps", 
                "type": null
            }, 
            "xe-0/0/35:1": {
                "admin-status": "down", 
                "macaddress": "80:ac:ac:90:70:90", 
                "mtu": "1514", 
                "oper-status": "down", 
                "speed": "10Gbps", 
                "type": null
            }, 
            "xe-0/0/35:2": {
                "admin-status": "up", 
                "macaddress": "80:ac:ac:90:70:91", 
                "mtu": "9216", 
                "oper-status": "up", 
                "speed": "10Gbps", 
                "type": null
            }, 
            "xe-0/0/35:3": {
                "admin-status": "up", 
                "macaddress": "80:ac:ac:90:70:92", 
                "mtu": "9216", 
                "oper-status": "down", 
                "speed": "10Gbps", 
                "type": null
            }
        }, 
        "ansible_net_memfree_mb": 1560464, 
        "ansible_net_memtotal_mb": 3407320, 
        "ansible_net_model": "qfx10002-36q", 
        "ansible_net_serialnum": "SCRUBBED", 
        "ansible_net_version": "15.1X53-D33.3"
    }, 
    "changed": false, 
    "invocation": {
        "module_args": {
            "config_format": "json", 
            "gather_subset": [
                "!config"
            ], 
            "host": "eugn-p2-gw.nero.net", 
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "port": null, 
            "provider": null, 
            "ssh_keyfile": null, 
            "timeout": 10, 
            "transport": null, 
            "username": "SCRUBBEDf"
        }
    }
}

```



Discussion:
cc @Qalthos @ganeshrn
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
@rhicks There are some changes in junos_facts that are not backward compatible.
Can you please change your playbook to use `ansible_net_model` instead of `personality `
@ganeshrn 
Is there a discussion on why junos_facts changed and broke backward compatibility?

`personality` isn't the only fact we use that is no longer available.  `has_2RE` is very important to us, as are others. 

Without notice they are gone, and all our playbooks are broken under Ansible 2.3.
This is very concerning.
@rhicks In 2.2 junos_fact was dependent on [junos-eznc](https://github.com/Juniper/py-junos-eznc) to get facts from device. 
However, this dependency was removed in 2.3 to support [persistent connection](https://www.ansible.com/blog/networking-features-in-ansible-2-3) framework that was added in 2.3. Hence the change in behavior and missing of some facts value.

Can you please let us know which all facts you are using and important to you so that we can work on adding them back.
@ganeshrn 
Currently we use the following, but I can see us making use of additional old facts as we add playbooks for other tasks (verifying REs versions are in sync, verifying capability of device, etc...).
`switch_style`
`master`
`model`
`personality`
`has_2RE`
`version`

Resolving patch:
diff --git a/lib/ansible/module_utils/junos.py b/lib/ansible/module_utils/junos.py
--- a/lib/ansible/module_utils/junos.py
+++ b/lib/ansible/module_utils/junos.py
@@ -191,3 +191,6 @@ def load_config(module, candidate, warnings, action='merge', commit=False, forma
                 discard_changes(module)
 
         return diff
+
+def get_param(module, key):
+    return module.params[key] or module.params['provider'].get(key)
diff --git a/lib/ansible/modules/network/junos/junos_facts.py b/lib/ansible/modules/network/junos/junos_facts.py
--- a/lib/ansible/modules/network/junos/junos_facts.py
+++ b/lib/ansible/modules/network/junos/junos_facts.py
@@ -42,7 +42,9 @@
         all, hardware, config, and interfaces.  Can specify a list of
         values to include a larger subset.  Values can also be used
         with an initial C(M(!)) to specify that a specific subset should
-        not be collected.
+        not be collected. To maintain backward compatbility old style facts
+        can be retrieved using all value, this reqires junos-eznc to be installed
+        as a prerequisite.
     required: false
     default: "!config"
     version_added: "2.3"
@@ -78,18 +80,25 @@
   type: dict
 """
 
-import re
 from xml.etree.ElementTree import Element, SubElement, tostring
 
 from ansible.module_utils.basic import AnsibleModule
+from ansible.module_utils.pycompat24 import get_exception
 from ansible.module_utils.six import iteritems
-from ansible.module_utils.junos import junos_argument_spec, check_args
+from ansible.module_utils.junos import junos_argument_spec, check_args, get_param
 from ansible.module_utils.junos import command, get_configuration
 from ansible.module_utils.netconf import send_request
 
+try:
+    from jnpr.junos import Device
+    from jnpr.junos.exception import ConnectError
+    HAS_PYEZ = True
+except ImportError:
+    HAS_PYEZ = False
 
 USE_PERSISTENT_CONNECTION = True
 
+
 class FactsBase(object):
 
     def __init__(self, module):
@@ -140,7 +149,7 @@ def populate(self):
         config_format = self.module.params['config_format']
         reply = get_configuration(self.module, format=config_format)
 
-        if config_format =='xml':
+        if config_format == 'xml':
             config = tostring(reply.find('configuration')).strip()
 
         elif config_format == 'text':
@@ -155,7 +164,6 @@ def populate(self):
         self.facts['config'] = config
 
 
-
 class Hardware(FactsBase):
 
     def populate(self):
@@ -202,11 +210,55 @@ def populate(self):
         self.facts['interfaces'] = interfaces
 
 
+class Facts(FactsBase):
+    def _connect(self, module):
+        host = get_param(module, 'host')
+
+        kwargs = {
+            'port': get_param(module, 'port') or 830,
+            'user': get_param(module, 'username')
+        }
+
+        if get_param(module, 'password'):
+            kwargs['passwd'] = get_param(module, 'password')
+
+        if get_param(module, 'ssh_keyfile'):
+            kwargs['ssh_private_key_file'] = get_param(module, 'ssh_keyfile')
+
+        kwargs['gather_facts'] = False
+
+        try:
+            device = Device(host, **kwargs)
+            device.open()
+            device.timeout = get_param(module, 'timeout') or 10
+        except ConnectError:
+            exc = get_exception()
+            module.fail_json('unable to connect to %s: %s' % (host, str(exc)))
+
+        return device
+
+    def populate(self):
+
+        device = self._connect(self.module)
+        facts = dict(device.facts)
+
+        if '2RE' in facts:
+            facts['has_2RE'] = facts['2RE']
+            del facts['2RE']
+
+        facts['version_info'] = dict(facts['version_info'])
+        if 'junos_info' in facts:
+            for key, value in facts['junos_info'].items():
+                if 'object' in value:
+                    value['object'] = dict(value['object'])
+
+        return facts
+
 FACT_SUBSETS = dict(
     default=Default,
     hardware=Hardware,
     config=Config,
-    interfaces=Interfaces,
+    interfaces=Interfaces
 )
 
 VALID_SUBSETS = frozenset(FACT_SUBSETS.keys())
@@ -229,6 +281,7 @@ def main():
     check_args(module, warnings)
 
     gather_subset = module.params['gather_subset']
+    ofacts = False
 
     runable_subsets = set()
     exclude_subsets = set()
@@ -236,12 +289,14 @@ def main():
     for subset in gather_subset:
         if subset == 'all':
             runable_subsets.update(VALID_SUBSETS)
+            ofacts = True
             continue
 
         if subset.startswith('!'):
             subset = subset[1:]
             if subset == 'all':
                 exclude_subsets.update(VALID_SUBSETS)
+                ofacts = False
                 continue
             exclude = True
         else:
@@ -278,6 +333,13 @@ def main():
         key = 'ansible_net_%s' % key
         ansible_facts[key] = value
 
+    if ofacts:
+        if HAS_PYEZ:
+            ansible_facts.update(Facts(module).populate())
+        else:
+            warnings += ['junos-eznc is required to gather old style facts but does not appear to be installed. '
+                         'It can be installed using `pip  install junos-eznc`']
+
     module.exit_json(ansible_facts=ansible_facts, warnings=warnings)
 
 
diff --git a/lib/ansible/modules/network/junos/junos_package.py b/lib/ansible/modules/network/junos/junos_package.py
--- a/lib/ansible/modules/network/junos/junos_package.py
+++ b/lib/ansible/modules/network/junos/junos_package.py
@@ -99,7 +99,7 @@
 """
 from ansible.module_utils.basic import AnsibleModule
 from ansible.module_utils.pycompat24 import get_exception
-from ansible.module_utils.junos import junos_argument_spec
+from ansible.module_utils.junos import junos_argument_spec, get_param
 
 try:
     from jnpr.junos import Device
@@ -110,8 +110,6 @@
     HAS_PYEZ = False
 
 
-get_param = lambda x, y: x.params[y] or x.params['provider'].get(y)
-
 def connect(module):
     host = get_param(module, 'host')
 
@@ -138,6 +136,7 @@ def connect(module):
 
     return device
 
+
 def install_package(module, device):
     junos = SW(device)
     package = module.params['src']


Test patch:
diff --git a/test/sanity/pep8/legacy-files.txt b/test/sanity/pep8/legacy-files.txt
--- a/test/sanity/pep8/legacy-files.txt
+++ b/test/sanity/pep8/legacy-files.txt
@@ -413,7 +413,6 @@ lib/ansible/modules/net_tools/ipinfoio_facts.py
 lib/ansible/modules/network/junos/_junos_template.py
 lib/ansible/modules/network/junos/junos_command.py
 lib/ansible/modules/network/junos/junos_config.py
-lib/ansible/modules/network/junos/junos_facts.py
 lib/ansible/modules/network/junos/junos_netconf.py
 lib/ansible/modules/network/junos/junos_package.py
 lib/ansible/modules/network/lenovo/cnos_conditional_template.py