Repository: ansible/ansible
Pull request: Move missing library abort to use rather than import for netconf
##### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->

<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
Fixes #55381

`ncclient` imports on implementing plugins have also been moved, but the check in the superclass should suffice for all of them.

##### ISSUE TYPE
<!--- Pick one below and delete the rest -->
- Bugfix Pull Request

##### COMPONENT NAME
<!--- Write the short name of the module, plugin, task or feature below -->
netconf

Problem statement:
Many NXOS Modules fail with 'msg": "ncclient is not installed' for local connection plugin
##### SUMMARY
Many of our NX-OS integration tests started to fail on 4/12/2019 with the following message:

`"msg": "ncclient is not installed"`

We have also seen the problem in `stable-2.7`

NOTE: If we checkout a version of `devel` on 4/7/2019 we don't see the problem.

##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
nxos

##### ANSIBLE VERSION

```
ansible 2.8.0a1.post0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/Users/mwiebe/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/mwiebe/Projects/nxos_ansible/fix_ansible/lib/ansible
  executable location = /Users/mwiebe/Projects/nxos_ansible/fix_ansible/bin/ansible
  python version = 2.7.13 (default, Apr  4 2017, 08:47:57) [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.38)]
```

Hints from issue discussion:
Files identified in the description:
* [lib/ansible/module_utils/network/nxos/__init__.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/network/nxos/__init__.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
@mikewiebe: Greetings! Thanks for taking the time to open this issue. In order for the community to handle your issue effectively, we need a bit more information.

Here are the items we could not find in your description:
- ansible version

Please set the description of this issue with this template:
https://raw.githubusercontent.com/ansible/ansible/devel/.github/ISSUE_TEMPLATE.md

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: issue_missing_data --->
cc @chrisvanheuveln @rahushen @rcarrillocruz @trishnaguha
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
This is a bug for the `local` connection plugin which was introduced into when the PR https://github.com/ansible/ansible/pull/54860 got merged. Commit sha https://github.com/ansible/ansible/commit/6579dfda1717cda64bdb2a6de31f0b8ec7841471

This seems to be the case for all other networking platforms as well.
`ncclient` should be only required when `netconf` connection plugin is used.

cc @bcoca @Qalthos 
@trishnaguha not `local` related, but `persistent`

What tests are failing specifically?

needs_info

Commit message: Move missing library abort to use rather than import

Patch to review:
diff --git a/lib/ansible/plugins/netconf/__init__.py b/lib/ansible/plugins/netconf/__init__.py
index 4920b4fbbb0689..a1c718c6b2eadb 100644
--- a/lib/ansible/plugins/netconf/__init__.py
+++ b/lib/ansible/plugins/netconf/__init__.py
@@ -29,8 +29,11 @@
 try:
     from ncclient.operations import RPCError
     from ncclient.xml_ import to_xml, to_ele
-except ImportError:
-    raise AnsibleError("ncclient is not installed")
+    HAS_NCCLIENT = True
+    NCCLIENT_IMP_ERR = None
+except (ImportError, AttributeError) as err:  # paramiko and gssapi are incompatible and raise AttributeError not ImportError
+    HAS_NCCLIENT = False
+    NCCLIENT_IMP_ERR = err
 
 try:
     from lxml.etree import Element, SubElement, tostring, fromstring
@@ -101,6 +104,11 @@ class NetconfBase(AnsiblePlugin):
     __rpc__ = ['get_config', 'edit_config', 'get_capabilities', 'get']
 
     def __init__(self, connection):
+        if not HAS_NCCLIENT:
+            raise AnsibleError(
+                'The required "ncclient" python library is required to use the netconf connection type: %s.\n'
+                'Please run pip install ncclient' % to_native(NCCLIENT_IMP_ERR)
+            )
         self._connection = connection
 
     @property
diff --git a/lib/ansible/plugins/netconf/ce.py b/lib/ansible/plugins/netconf/ce.py
index ef41b54994c4aa..c74af3476090f3 100644
--- a/lib/ansible/plugins/netconf/ce.py
+++ b/lib/ansible/plugins/netconf/ce.py
@@ -33,8 +33,9 @@
     from ncclient.operations import RPCError
     from ncclient.transport.errors import SSHUnknownHostError
     from ncclient.xml_ import to_ele, to_xml, new_ele
-except ImportError:
-    raise AnsibleError("ncclient is not installed")
+    HAS_NCCLIENT = True
+except (ImportError, AttributeError):  # paramiko and gssapi are incompatible and raise AttributeError not ImportError
+    HAS_NCCLIENT = False
 
 
 class Netconf(NetconfBase):
diff --git a/lib/ansible/plugins/netconf/iosxr.py b/lib/ansible/plugins/netconf/iosxr.py
index d80f0dbeeafb3e..3c6d2cc3682654 100644
--- a/lib/ansible/plugins/netconf/iosxr.py
+++ b/lib/ansible/plugins/netconf/iosxr.py
@@ -37,16 +37,22 @@
     from ncclient.operations import RPCError
     from ncclient.transport.errors import SSHUnknownHostError
     from ncclient.xml_ import to_ele, to_xml, new_ele
-except ImportError:
-    raise AnsibleError("ncclient is not installed")
+    HAS_NCCLIENT = True
+except (ImportError, AttributeError):  # paramiko and gssapi are incompatible and raise AttributeError not ImportError
+    HAS_NCCLIENT = False
 
 try:
     from lxml import etree
+    HAS_LXML = True
 except ImportError:
-    raise AnsibleError("lxml is not installed")
+    HAS_LXML = False
 
 
 class Netconf(NetconfBase):
+    def __init__(self, *args, **kwargs):
+        if not HAS_LXML:
+            raise AnsibleError("lxml is not installed")
+        super(self, Netconf).__init__(*args, **kwargs)
 
     @ensure_connected
     def get_device_info(self):
diff --git a/lib/ansible/plugins/netconf/junos.py b/lib/ansible/plugins/netconf/junos.py
index 55ca60fb9957f4..8966fb30e63871 100644
--- a/lib/ansible/plugins/netconf/junos.py
+++ b/lib/ansible/plugins/netconf/junos.py
@@ -33,8 +33,9 @@
     from ncclient.operations import RPCError
     from ncclient.transport.errors import SSHUnknownHostError
     from ncclient.xml_ import to_ele, to_xml, new_ele, sub_ele
-except ImportError:
-    raise AnsibleError("ncclient is not installed")
+    HAS_NCCLIENT = True
+except (ImportError, AttributeError):  # paramiko and gssapi are incompatible and raise AttributeError not ImportError
+    HAS_NCCLIENT = False
 
 
 class Netconf(NetconfBase):
diff --git a/lib/ansible/plugins/netconf/sros.py b/lib/ansible/plugins/netconf/sros.py
index c4504ca398c2c3..8a6c53d53fdf26 100644
--- a/lib/ansible/plugins/netconf/sros.py
+++ b/lib/ansible/plugins/netconf/sros.py
@@ -33,16 +33,23 @@
     from ncclient.operations import RPCError
     from ncclient.transport.errors import SSHUnknownHostError
     from ncclient.xml_ import to_ele, to_xml, new_ele
-except ImportError:
-    raise AnsibleError("ncclient is not installed")
+    HAS_NCCLIENT = True
+except (ImportError, AttributeError):  # paramiko and gssapi are incompatible and raise AttributeError not ImportError
+    HAS_NCCLIENT = False
 
 try:
     from lxml import etree
+    HAS_LXML = True
 except ImportError:
-    raise AnsibleError("lxml is not installed")
+    HAS_LXML = False
 
 
 class Netconf(NetconfBase):
+    def __init__(self, *args, **kwargs):
+        if not HAS_LXML:
+            raise AnsibleError("lxml is not installed")
+        super(self, Netconf).__init__(*args, **kwargs)
+
     def get_text(self, ele, tag):
         try:
             return to_text(ele.find(tag).text, errors='surrogate_then_replace').strip()


Code review comments:
diff_hunk: @@ -101,6 +104,11 @@ class NetconfBase(AnsiblePlugin):
     __rpc__ = ['get_config', 'edit_config', 'get_capabilities', 'get']
 
     def __init__(self, connection):
+        if not HAS_NCCLIENT:
+            raise AnsibleError(
+                'The required "ncclient" python library is required to use the netconf connection type: %s.\n'
+                'Please run pip install ncclient' % to_native(NCCLIENT_IMP_ERR)
+            )
path: lib/ansible/plugins/netconf/__init__.py
text: @user1:
i recommend you use `missing_required_lib` from module_utils.basic as it adds 'hostname' and 'which python' to avoid confusion

@user1:
isn't `__init__` too early still? normally we return the error when we attempt to use the plugin, since they can be instantiated for other reasons (i.e doc reading/config/etc)