Repository: ansible/ansible

Issue:
netconf_config module needs to support ncclient device handlers
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
netconf_config

##### ANSIBLE VERSION
```
ansible 2.4.3.0
  config file = None
  configured module search path = [u'/home/pod1user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/pod1user/.virtualenvs/ansible/lib/python2.7/site-packages/ansible
  executable location = /home/pod1user/.virtualenvs/ansible/bin/ansible
  python version = 2.7.5 (default, Aug  4 2017, 00:39:18) [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]
```

##### CONFIGURATION
$ ansible-config dump --only-changed
$ 

##### OS / ENVIRONMENT
$ cat /etc/centos-release
CentOS Linux release 7.3.1611 (Core) 
$

##### SUMMARY
When using the netconf_config module in a task where the host is a Cisco Nexus device, this module would fail and timeout with the below error msg and traceback:
```
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_GtNH9A/ansible_module_netconf_config.py", line 303, in main
    datastore=datastore,
  File "/tmp/ansible_GtNH9A/ansible_module_netconf_config.py", line 164, in netconf_edit_config
    m.lock(target=datastore)
  File "/usr/lib/python2.7/site-packages/ncclient/manager.py", line 162, in wrapper
    return self.execute(op_cls, *args, **kwds)
  File "/usr/lib/python2.7/site-packages/ncclient/manager.py", line 232, in execute
    raise_mode=self._raise_mode).request(*args, **kwds)
  File "/usr/lib/python2.7/site-packages/ncclient/operations/lock.py", line 35, in request
    return self._request(node)
  File "/usr/lib/python2.7/site-packages/ncclient/operations/rpc.py", line 343, in _request
    raise TimeoutExpiredError('ncclient timed out while waiting for an rpc reply.')
TimeoutExpiredError: ncclient timed out while waiting for an rpc reply.

fatal: [10.0.249.251]: FAILED! => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "allow_agent": false, 
            "datastore": "running", 
            "host": "10.0.249.251", 
            "hostkey_verify": false, 
            "look_for_keys": false, 
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "port": 830, 
            "save": false, 
            "src": "<config>\n    <System xmlns=\"http://cisco.com/ns/yang/cisco-nx-os-device\">\n        <fm-items>\n            <bgp-items>\n                <********St>enabled</********St>\n            </bgp-items>\n        </fm-items>\n    </System>\n</config>", 
            "username": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "xml": null
        }
    }, 
    "msg": "error editing configuration: ncclient timed out while waiting for an rpc reply."
}
```

##### STEPS TO REPRODUCE
Playbook execution:

ansible-playbook -i hosts site.yml -vvv
<!--- Paste example playbooks or commands between quotes below -->
Ansible playbook directory layout:

```
$ tree
.
|-- group_vars
|     |-- all.yml
|-- hosts
|-- roles
|    |-- netconf
|          |-- defaults
|          |     |-- main.yml
|          |-- files
|          |     |-- enable_bgp.xml
|          |-- handlers
|          |     |-- main.yml
|          |-- meta
|          |     |-- main.yml
|          |-- README.md
|          |-- tasks
|          |     |-- main.yml
|          |-- templates
|          |-- tests
|          |    |-- inventory
|          |    |-- test.yml
|          |-- vars
|                |-- main.yml
|-- site.yml

11 directories, 12 files
```

group_vars/all:

```yaml
---
ansible_connection: local
```

roles/netconf/files/enable_bgp.xml:

```xml
<config>
    <System xmlns="http://cisco.com/ns/yang/cisco-nx-os-device">
        <fm-items>
            <bgp-items>
                <adminSt>enabled</adminSt>
            </bgp-items>
        </fm-items>
    </System>
</config>
```

roles/netconf/tasks/main.yml:

```yaml
---
# tasks file for netconf
- name: NETCONF ENABLE BGP
  netconf_config:
    host: 10.0.249.251
    port: 830
    username: admin
    password: cisco.123
    look_for_keys: False
    hostkey_verify: False
    allow_agent: False
    datastore: running
    src: /home/pod1user/Devel/ansible/playbooks/netconf/roles/netconf/files/enable_bgp.xml
```

main playbook:

```yaml
---
- hosts: n9kv
  roles:
    - role: netconf
```

##### EXPECTED RESULTS
The expected result is a successful rpc to the specified device, not just a Cisco Nexus device. To solve this, some simple modifications can be made to the module params that aligns with ncclient's device handlers. Here's is a recommended solution:
```
module = AnsibleModule(
        argument_spec=dict(
            host=dict(type='str', required=True),
            port=dict(type='int', default=830),
            hostkey_verify=dict(type='bool', default=True),
            allow_agent=dict(type='bool', default=True),
            look_for_keys=dict(type='bool', default=True),
            datastore=dict(choices=['auto', 'candidate', 'running'], default='auto'),
            save=dict(type='bool', default=False),
            username=dict(type='str', required=True, no_log=True),
            password=dict(type='str', required=True, no_log=True),
            xml=dict(type='str', required=False),
            src=dict(type='path', required=False),
            device_handler=dict(choices=['junos', 'csr', 'nexus', 'iosxr', 'iosxe', 'huawei', 'alu', 'h3c', 'hpcomware', 'default'], default='default'),
        ),
        mutually_exclusive=[('xml', 'src')]
    )
<snip>
nckwargs = dict(
        host=module.params['host'],
        port=module.params['port'],
        hostkey_verify=module.params['hostkey_verify'],
        allow_agent=module.params['allow_agent'],
        look_for_keys=module.params['look_for_keys'],
        username=module.params['username'],
        password=module.params['password'],
        device_params={'name': module.params['device_handler']}
    )
```
```yaml
---
# tasks file for netconf
- name: NETCONF ENABLE BGP
  netconf_config:
    host: 10.0.249.251
    port: 830
    username: admin
    password: cisco.123
    look_for_keys: False
    hostkey_verify: False
    allow_agent: False
    datastore: running
    device_handler: nexus
    src: /home/pod1user/Devel/ansible/playbooks/netconf/roles/netconf/files/enable_bgp.xml
```

##### ACTUAL RESULTS
Actual results:
```
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_GtNH9A/ansible_module_netconf_config.py", line 303, in main
    datastore=datastore,
  File "/tmp/ansible_GtNH9A/ansible_module_netconf_config.py", line 164, in netconf_edit_config
    m.lock(target=datastore)
  File "/usr/lib/python2.7/site-packages/ncclient/manager.py", line 162, in wrapper
    return self.execute(op_cls, *args, **kwds)
  File "/usr/lib/python2.7/site-packages/ncclient/manager.py", line 232, in execute
    raise_mode=self._raise_mode).request(*args, **kwds)
  File "/usr/lib/python2.7/site-packages/ncclient/operations/lock.py", line 35, in request
    return self._request(node)
  File "/usr/lib/python2.7/site-packages/ncclient/operations/rpc.py", line 343, in _request
    raise TimeoutExpiredError('ncclient timed out while waiting for an rpc reply.')
TimeoutExpiredError: ncclient timed out while waiting for an rpc reply.

fatal: [10.0.249.251]: FAILED! => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "allow_agent": false, 
            "datastore": "running", 
            "host": "10.0.249.251", 
            "hostkey_verify": false, 
            "look_for_keys": false, 
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "port": 830, 
            "save": false, 
            "src": "<config>\n    <System xmlns=\"http://cisco.com/ns/yang/cisco-nx-os-device\">\n        <fm-items>\n            <bgp-items>\n                <********St>enabled</********St>\n            </bgp-items>\n        </fm-items>\n    </System>\n</config>", 
            "username": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "xml": null
        }
    }, 
    "msg": "error editing configuration: ncclient timed out while waiting for an rpc reply."
}
```


Discussion:
Files identified in the description:
* [lib/ansible/modules/network/netconf/netconf_config.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/network/netconf/netconf_config.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @ganeshrn @lpenz @userlerueda
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
@privateip I wonder how this fits with connection=netconf. Does it make sense?
I can confirm that this doesn't break the behavior I'm testing.
@lpenz Would you mind sharing your playbook structure and tasks? I've installed 2.5rc1 and in my hosts file set the ansible_connection=netconf. I initially tried ```ansible_network_os=nxos```, but that failed with this error : ```{"msg": "connection=netconf is not supported on nxos"}``` (understood to most likely not be addressed in this issue ticket), so then I retried with ```ansible_network_os=default```, but this time the session closes repeatedly with this error msg: ```"msg": "error editing configuration: Unexpected session close"```.
I was able to get netconf_config working on another Cisco device iosxr. netconf_config is platform agnostic module with raw xml data so it should work with Cisco nxos also. Can you please confirm if netconf agent is running on nxos by running following command ?
ssh -p 830 <username>@<host-ip>  -s netconf

it should return the netconf capabilities if netconf agent is running..
@gdpak Yep, I wouldn't open an issue and not have the fundamentals configured. It'd be good to test apples to apples. Given this Ansible module actually relies on ncclient, I've replicated this and various other rpcs for Nexus devices directly using ncclient and the nexus connector, but with this module it fails. Are you saying this is device agnostic because it uses the default connector?

Here's the verification you asked for:
```
# ssh -s admin@10.0.249.251 -p 830 netconf
User Access Verification
Password: 
<?xml version="1.0" encoding="UTF-8"?>
<hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
    <capabilities>
        <capability>urn:ietf:params:netconf:base:1.0</capability>
        <capability>urn:ietf:params:netconf:base:1.1</capability>
        <capability>urn:ietf:params:netconf:capability:writable-running:1.0</capability>
        <capability>urn:ietf:params:netconf:capability:rollback-on-error:1.0</capability>
        <capability>urn:ietf:params:netconf:capability:candidate:1.0</capability>
        <capability>urn:ietf:params:netconf:capability:validate:1.1</capability>
        <capability>urn:ietf:params:netconf:capability:confirmed-commit:1.1</capability>
        <capability>http://cisco.com/ns/yang/cisco-nx-os-device?revision=2018-02-12&amp;module=Cisco-NX-OS-device&amp;deviations=Cisco-NX-OS-device-deviations</capability>
        <capability>http://openconfig.net/yang/bgp?revision=2016-06-06&amp;module=openconfig-bgp&amp;deviations=openconfig-bgp-deviations</capability>
        <capability>http://openconfig.net/yang/bgp-multiprotocol?revision=2016-06-06&amp;module=openconfig-bgp-multiprotocol&amp;deviations=openconfig-bgp-multiprotocol-deviations</capability>
        <capability>http://openconfig.net/yang/interfaces?revision=2016-05-26&amp;module=openconfig-interfaces&amp;deviations=openconfig-interfaces-deviations</capability>
        <capability>http://openconfig.net/yang/interfaces/ip?revision=2016-05-26&amp;module=openconfig-if-ip&amp;deviations=openconfig-if-ip-deviations</capability>
        <capability>http://openconfig.net/yang/local-routing?revision=2016-05-11&amp;module=openconfig-local-routing&amp;deviations=openconfig-local-routing-deviations</capability>
        <capability>http://openconfig.net/yang/routing-policy?revision=2016-05-12&amp;module=openconfig-routing-policy&amp;deviations=openconfig-routing-policy-deviations</capability>
        <capability>http://openconfig.net/yang/vlan?revision=2016-05-26&amp;module=openconfig-vlan&amp;deviations=openconfig-vlan-deviations</capability>
    </capabilities>
    <session-id>1703378757</session-id>
</hello>
]]>]]>
```


Resolving patch:
diff --git a/lib/ansible/plugins/connection/netconf.py b/lib/ansible/plugins/connection/netconf.py
--- a/lib/ansible/plugins/connection/netconf.py
+++ b/lib/ansible/plugins/connection/netconf.py
@@ -157,6 +157,10 @@
 
 logging.getLogger('ncclient').setLevel(logging.INFO)
 
+network_os_device_param_map = {
+    "nxos": "nexus"
+}
+
 
 class Connection(ConnectionBase):
     """NetConf connections"""
@@ -238,7 +242,7 @@ def _connect(self):
                 if network_os:
                     display.display('discovered network_os %s' % network_os, log_only=True)
 
-        device_params = {'name': (network_os or 'default')}
+        device_params = {'name': (network_os_device_param_map.get(network_os) or network_os or 'default')}
 
         ssh_config = os.getenv('ANSIBLE_NETCONF_SSH_CONFIG', False)
         if ssh_config in BOOLEANS_TRUE:
@@ -276,7 +280,8 @@ def _connect(self):
         if self._netconf:
             display.display('loaded netconf plugin for network_os %s' % network_os, log_only=True)
         else:
-            display.display('unable to load netconf for network_os %s' % network_os)
+            self._netconf = netconf_loader.get("default", self)
+            display.display('unable to load netconf plugin for network_os %s, falling back to default plugin' % network_os)
 
         return 0, to_bytes(self._manager.session_id, errors='surrogate_or_strict'), b''
 


Test patch: