Repository: ansible/ansible

Issue:
netconf_config module needs to support ncclient device handlers
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
netconf_config

##### ANSIBLE VERSION
```
ansible 2.4.3.0
  config file = None
  configured module search path = [u'/home/pod1user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/pod1user/.virtualenvs/ansible/lib/python2.7/site-packages/ansible
  executable location = /home/pod1user/.virtualenvs/ansible/bin/ansible
  python version = 2.7.5 (default, Aug  4 2017, 00:39:18) [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]
```

##### CONFIGURATION
$ ansible-config dump --only-changed
$ 

##### OS / ENVIRONMENT
$ cat /etc/centos-release
CentOS Linux release 7.3.1611 (Core) 
$

##### SUMMARY
When using the netconf_config module in a task where the host is a Cisco Nexus device, this module would fail and timeout with the below error msg and traceback:
```
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_GtNH9A/ansible_module_netconf_config.py", line 303, in main
    datastore=datastore,
  File "/tmp/ansible_GtNH9A/ansible_module_netconf_config.py", line 164, in netconf_edit_config
    m.lock(target=datastore)
  File "/usr/lib/python2.7/site-packages/ncclient/manager.py", line 162, in wrapper
    return self.execute(op_cls, *args, **kwds)
  File "/usr/lib/python2.7/site-packages/ncclient/manager.py", line 232, in execute
    raise_mode=self._raise_mode).request(*args, **kwds)
  File "/usr/lib/python2.7/site-packages/ncclient/operations/lock.py", line 35, in request
    return self._request(node)
  File "/usr/lib/python2.7/site-packages/ncclient/operations/rpc.py", line 343, in _request
    raise TimeoutExpiredError('ncclient timed out while waiting for an rpc reply.')
TimeoutExpiredError: ncclient timed out while waiting for an rpc reply.

fatal: [10.0.249.251]: FAILED! => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "allow_agent": false, 
            "datastore": "running", 
            "host": "10.0.249.251", 
            "hostkey_verify": false, 
            "look_for_keys": false, 
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "port": 830, 
            "save": false, 
            "src": "<config>\n    <System xmlns=\"http://cisco.com/ns/yang/cisco-nx-os-device\">\n        <fm-items>\n            <bgp-items>\n                <********St>enabled</********St>\n            </bgp-items>\n        </fm-items>\n    </System>\n</config>", 
            "username": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "xml": null
        }
    }, 
    "msg": "error editing configuration: ncclient timed out while waiting for an rpc reply."
}
```

##### STEPS TO REPRODUCE
Playbook execution:

ansible-playbook -i hosts site.yml -vvv
<!--- Paste example playbooks or commands between quotes below -->
Ansible playbook directory layout:

```
$ tree
.
|-- group_vars
|     |-- all.yml
|-- hosts
|-- roles
|    |-- netconf
|          |-- defaults
|          |     |-- main.yml
|          |-- files
|          |     |-- enable_bgp.xml
|          |-- handlers
|          |     |-- main.yml
|          |-- meta
|          |     |-- main.yml
|          |-- README.md
|          |-- tasks
|          |     |-- main.yml
|          |-- templates
|          |-- tests
|          |    |-- inventory
|          |    |-- test.yml
|          |-- vars
|                |-- main.yml
|-- site.yml

11 directories, 12 files
```

group_vars/all:

```yaml
---
ansible_connection: local
```

roles/netconf/files/enable_bgp.xml:

```xml
<config>
    <System xmlns="http://cisco.com/ns/yang/cisco-nx-os-device">
        <fm-items>
            <bgp-items>
                <adminSt>enabled</adminSt>
            </bgp-items>
        </fm-items>
    </System>
</config>
```

roles/netconf/tasks/main.yml:

```yaml
---
# tasks file for netconf
- name: NETCONF ENABLE BGP
  netconf_config:
    host: 10.0.249.251
    port: 830
    username: admin
    password: cisco.123
    look_for_keys: False
    hostkey_verify: False
    allow_agent: False
    datastore: running
    src: /home/pod1user/Devel/ansible/playbooks/netconf/roles/netconf/files/enable_bgp.xml
```

main playbook:

```yaml
---
- hosts: n9kv
  roles:
    - role: netconf
```

##### EXPECTED RESULTS
The expected result is a successful rpc to the specified device, not just a Cisco Nexus device. To solve this, some simple modifications can be made to the module params that aligns with ncclient's device handlers. Here's is a recommended solution:
```
module = AnsibleModule(
        argument_spec=dict(
            host=dict(type='str', required=True),
            port=dict(type='int', default=830),
            hostkey_verify=dict(type='bool', default=True),
            allow_agent=dict(type='bool', default=True),
            look_for_keys=dict(type='bool', default=True),
            datastore=dict(choices=['auto', 'candidate', 'running'], default='auto'),
            save=dict(type='bool', default=False),
            username=dict(type='str', required=True, no_log=True),
            password=dict(type='str', required=True, no_log=True),
            xml=dict(type='str', required=False),
            src=dict(type='path', required=False),
            device_handler=dict(choices=['junos', 'csr', 'nexus', 'iosxr', 'iosxe', 'huawei', 'alu', 'h3c', 'hpcomware', 'default'], default='default'),
        ),
        mutually_exclusive=[('xml', 'src')]
    )
<snip>
nckwargs = dict(
        host=module.params['host'],
        port=module.params['port'],
        hostkey_verify=module.params['hostkey_verify'],
        allow_agent=module.params['allow_agent'],
        look_for_keys=module.params['look_for_keys'],
        username=module.params['username'],
        password=module.params['password'],
        device_params={'name': module.params['device_handler']}
    )
```
```yaml
---
# tasks file for netconf
- name: NETCONF ENABLE BGP
  netconf_config:
    host: 10.0.249.251
    port: 830
    username: admin
    password: cisco.123
    look_for_keys: False
    hostkey_verify: False
    allow_agent: False
    datastore: running
    device_handler: nexus
    src: /home/pod1user/Devel/ansible/playbooks/netconf/roles/netconf/files/enable_bgp.xml
```

##### ACTUAL RESULTS
Actual results:
```
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_GtNH9A/ansible_module_netconf_config.py", line 303, in main
    datastore=datastore,
  File "/tmp/ansible_GtNH9A/ansible_module_netconf_config.py", line 164, in netconf_edit_config
    m.lock(target=datastore)
  File "/usr/lib/python2.7/site-packages/ncclient/manager.py", line 162, in wrapper
    return self.execute(op_cls, *args, **kwds)
  File "/usr/lib/python2.7/site-packages/ncclient/manager.py", line 232, in execute
    raise_mode=self._raise_mode).request(*args, **kwds)
  File "/usr/lib/python2.7/site-packages/ncclient/operations/lock.py", line 35, in request
    return self._request(node)
  File "/usr/lib/python2.7/site-packages/ncclient/operations/rpc.py", line 343, in _request
    raise TimeoutExpiredError('ncclient timed out while waiting for an rpc reply.')
TimeoutExpiredError: ncclient timed out while waiting for an rpc reply.

fatal: [10.0.249.251]: FAILED! => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "allow_agent": false, 
            "datastore": "running", 
            "host": "10.0.249.251", 
            "hostkey_verify": false, 
            "look_for_keys": false, 
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "port": 830, 
            "save": false, 
            "src": "<config>\n    <System xmlns=\"http://cisco.com/ns/yang/cisco-nx-os-device\">\n        <fm-items>\n            <bgp-items>\n                <********St>enabled</********St>\n            </bgp-items>\n        </fm-items>\n    </System>\n</config>", 
            "username": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "xml": null
        }
    }, 
    "msg": "error editing configuration: ncclient timed out while waiting for an rpc reply."
}
```


Discussion:
Files identified in the description:
* [lib/ansible/modules/network/netconf/netconf_config.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/network/netconf/netconf_config.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @ganeshrn @lpenz @userlerueda
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
@privateip I wonder how this fits with connection=netconf. Does it make sense?
I can confirm that this doesn't break the behavior I'm testing.
@lpenz Would you mind sharing your playbook structure and tasks? I've installed 2.5rc1 and in my hosts file set the ansible_connection=netconf. I initially tried ```ansible_network_os=nxos```, but that failed with this error : ```{"msg": "connection=netconf is not supported on nxos"}``` (understood to most likely not be addressed in this issue ticket), so then I retried with ```ansible_network_os=default```, but this time the session closes repeatedly with this error msg: ```"msg": "error editing configuration: Unexpected session close"```.
I was able to get netconf_config working on another Cisco device iosxr. netconf_config is platform agnostic module with raw xml data so it should work with Cisco nxos also. Can you please confirm if netconf agent is running on nxos by running following command ?
ssh -p 830 <username>@<host-ip>  -s netconf

it should return the netconf capabilities if netconf agent is running..
@gdpak Yep, I wouldn't open an issue and not have the fundamentals configured. It'd be good to test apples to apples. Given this Ansible module actually relies on ncclient, I've replicated this and various other rpcs for Nexus devices directly using ncclient and the nexus connector, but with this module it fails. Are you saying this is device agnostic because it uses the default connector?

Here's the verification you asked for:
```
# ssh -s admin@10.0.249.251 -p 830 netconf
User Access Verification
Password: 
<?xml version="1.0" encoding="UTF-8"?>
<hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
    <capabilities>
        <capability>urn:ietf:params:netconf:base:1.0</capability>
        <capability>urn:ietf:params:netconf:base:1.1</capability>
        <capability>urn:ietf:params:netconf:capability:writable-running:1.0</capability>
        <capability>urn:ietf:params:netconf:capability:rollback-on-error:1.0</capability>
        <capability>urn:ietf:params:netconf:capability:candidate:1.0</capability>
        <capability>urn:ietf:params:netconf:capability:validate:1.1</capability>
        <capability>urn:ietf:params:netconf:capability:confirmed-commit:1.1</capability>
        <capability>http://cisco.com/ns/yang/cisco-nx-os-device?revision=2018-02-12&amp;module=Cisco-NX-OS-device&amp;deviations=Cisco-NX-OS-device-deviations</capability>
        <capability>http://openconfig.net/yang/bgp?revision=2016-06-06&amp;module=openconfig-bgp&amp;deviations=openconfig-bgp-deviations</capability>
        <capability>http://openconfig.net/yang/bgp-multiprotocol?revision=2016-06-06&amp;module=openconfig-bgp-multiprotocol&amp;deviations=openconfig-bgp-multiprotocol-deviations</capability>
        <capability>http://openconfig.net/yang/interfaces?revision=2016-05-26&amp;module=openconfig-interfaces&amp;deviations=openconfig-interfaces-deviations</capability>
        <capability>http://openconfig.net/yang/interfaces/ip?revision=2016-05-26&amp;module=openconfig-if-ip&amp;deviations=openconfig-if-ip-deviations</capability>
        <capability>http://openconfig.net/yang/local-routing?revision=2016-05-11&amp;module=openconfig-local-routing&amp;deviations=openconfig-local-routing-deviations</capability>
        <capability>http://openconfig.net/yang/routing-policy?revision=2016-05-12&amp;module=openconfig-routing-policy&amp;deviations=openconfig-routing-policy-deviations</capability>
        <capability>http://openconfig.net/yang/vlan?revision=2016-05-26&amp;module=openconfig-vlan&amp;deviations=openconfig-vlan-deviations</capability>
    </capabilities>
    <session-id>1703378757</session-id>
</hello>
]]>]]>
```

@mtarking Can you please apply the referenced PR #36819 in your environment and rerun with `ansible_connection: netconf` and `ansible_network_os=nxos` on stable-2.5 base branch
@ganeshrn I pip installed 2.5.0rc1 which showed green/shippable. I then copied your netconf.py file and replaced the default file. Using my same playbook, I changed from local to ```ansible_connection=netconf``` and ```ansible_network_os=nxos``` in my hosts file. I'm executing the playbook as follows: ```ansible-playbook -i hosts site.yml -u admin -k -vvv```
I thought your fix would have done it as that's essentially what I did local to the module, but I'm getting an exception now:
```
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_sfXWer/ansible_module_netconf_config.py", line 343, in <module>
    main()
  File "/tmp/ansible_sfXWer/ansible_module_netconf_config.py", line 245, in main
    capabilities = module.from_json(m.get_capabilities())
  File "/tmp/ansible_sfXWer/ansible_modlib.zip/ansible/module_utils/connection.py", line 146, in __rpc__
ansible.module_utils.connection.ConnectionError: Method not found
````
I looked in connection.py file and do not see a method called get_capabilities() for the Connection class. Just quickly skimming the code from netconf_config.py, it looks like you get into this section of the code when we no longer use local_connection.
If you have a suggestions or comments that I'm doing something incorrect, please let me know.
The error message you are seeing now is because nxos netconf plugin does not exist yet. I have updated the same PR to fallback on default netconf plugin in case the platform specific plugin does not exist.

An easier way to test PR branch is to open a new terminal and run below commands
```
$ git clone -b netconf_config_fix  https://github.com/ganeshrn/ansible.git
$ cd ansible
$ source hacking/env-setup
$ which ansible
```
The output of the last command should point to `<present-working-directory-path>/bin/ansible`
After that rerun the playbook in the same terminal window.
@mtarking The repository https://github.com/lpenz/ansible-testing-netconf has the tests I use. It's essentially a smoke test against openyuma, nothing sophisticated.
The theory is that it doesn't do a whole lot more than `ssh -s admin@10.0.249.251 -p 830 netconf < config.xml` with some checks and activating the config afterwards - but it really depends on the portability of ncclient as well.
@ganeshrn I pulled your repo and sourced it. I pip installed requirements and ncclient. I reran my playbook same as before with ```ansible-playbook -i hosts site.yml -u admin -k -vvvv```, but now getting this traceback:
```
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_DHqx0I/ansible_module_netconf_config.py", line 329, in main
    local_connection=local_connection
  File "/tmp/ansible_DHqx0I/ansible_module_netconf_config.py", line 198, in netconf_edit_config
    m.unlock(target=datastore)
  File "/tmp/ansible_DHqx0I/ansible_modlib.zip/ansible/module_utils/connection.py", line 140, in __rpc__
    response = self._exec_jsonrpc(name, *args, **kwargs)
  File "/tmp/ansible_DHqx0I/ansible_modlib.zip/ansible/module_utils/connection.py", line 115, in _exec_jsonrpc
    raise ConnectionError('socket_path does not exist or cannot be found')
ConnectionError: socket_path does not exist or cannot be found
```
Task:
```
---
# tasks file for netconf
- name: NETCONF ENABLE BGP
  netconf_config:
    look_for_keys: False
    hostkey_verify: False
    allow_agent: False
    datastore: running
    src: /home/pod1user/Devel/ansible/playbooks/netconf/roles/netconf/files/enable_bgp.xml
```
Hosts file:
```
[n9kv]
10.0.249.251 ansible_connection=netconf ansible_network_os=nxos
```
@mtarking The traceback indicates it is failing while running `unlock()`, most likely seems a command_timeout related issue. Can you please try the troubleshooting steps mentioned here http://docs.ansible.com/ansible/devel/network/user_guide/network_debug_troubleshooting.html#category-socket-path-issue



Please see the attached snippet from the ansible.log where at line 107 appears to indeed be a timeout occurring with ncclient.
[snippet_ansible.log](https://github.com/ansible/ansible/files/1768327/snippet_ansible.log)

That's right, command timeout is triggered based on logs.
`2018-02-28 13:10:19,754 p=3510 u=pod1user |  command timeout triggered, timeout value is 10 secs`

Please retry after increasing the `command_timeout` and `connect_timeout` value in the active ansible.cfg file
```
[persistent_connection]
connect_timeout = 100
command_timeout = 100
```
@ganeshrn Setting the timeouts worked. 
```
10.0.249.251               : ok=2    changed=1    unreachable=0    failed=0  
```
I will admit, I haven't used those variables before, so learned something for troubleshooting. What is the proper way to handle this though, in the module or plugin?

To recap thus far, there has been no code modifications to the netconf_config module, so ```ansible_connection: local``` would still be busted for this device connector type using ncclient. You made code modifications to the netconf plugin using a dict and the nxos value from ```ansible_network_os: nxos``` as the key to get the value ```nexus``` for passing as a device param to ncclient. Please correct if I've misunderstood.
@mtarking The support for `ansible_connetcion=netconf` which uses underlying Ansible persistent connection framework was added in 2.5 as  part of PR https://github.com/ansible/ansible/pull/33400

`ansible_connection=local` initiates a netconf session with remote host for every task run hence it is recommended to use `ansible_connection=netconf`

`ansible_connetcion=local` is supported for backward compatibility and will be deprecated in future releases.


@ganeshrn Right, understood. Just was clarifying that for the netconf_config module to work toward a Cisco Nexus device, the ```ansible_connection=netconf``` would have to used starting in 2.5 as there was no code update to the netconf_config module itself, thus ```ansible_connection=local``` would continue to fail (but I guess doesn't matter given it will be deprecated in the future) for any release (2.5 or prior).

Again, what is the proper way to handle the timeout issue; module, plugin, etc??
The right way to handle timeout issue is to increase the timer value either in `ansible.cfg` file or by setting timer related environment as mentioned in the troubleshooting guide.

Due to the current limitation in persistent connection framework timeout related messages are not displayed on the console.
In coming releases, we will be working on sending timeout related error messages to console so that user don't have to go through the troubleshooting steps.

Resolving patch:
diff --git a/lib/ansible/plugins/connection/netconf.py b/lib/ansible/plugins/connection/netconf.py
--- a/lib/ansible/plugins/connection/netconf.py
+++ b/lib/ansible/plugins/connection/netconf.py
@@ -157,6 +157,10 @@
 
 logging.getLogger('ncclient').setLevel(logging.INFO)
 
+network_os_device_param_map = {
+    "nxos": "nexus"
+}
+
 
 class Connection(ConnectionBase):
     """NetConf connections"""
@@ -238,7 +242,7 @@ def _connect(self):
                 if network_os:
                     display.display('discovered network_os %s' % network_os, log_only=True)
 
-        device_params = {'name': (network_os or 'default')}
+        device_params = {'name': (network_os_device_param_map.get(network_os) or network_os or 'default')}
 
         ssh_config = os.getenv('ANSIBLE_NETCONF_SSH_CONFIG', False)
         if ssh_config in BOOLEANS_TRUE:
@@ -276,7 +280,8 @@ def _connect(self):
         if self._netconf:
             display.display('loaded netconf plugin for network_os %s' % network_os, log_only=True)
         else:
-            display.display('unable to load netconf for network_os %s' % network_os)
+            self._netconf = netconf_loader.get("default", self)
+            display.display('unable to load netconf plugin for network_os %s, falling back to default plugin' % network_os)
 
         return 0, to_bytes(self._manager.session_id, errors='surrogate_or_strict'), b''
 


Test patch: