Repository: ansible/ansible

Issue:
junos_config using netconf no longer honours PERSISTENT_COMMAND_TIMEOUT from configuration file
<!---
Verify first that your issue/request is not already reported on GitHub.
THIS FORM WILL BE READ BY A MACHINE, COMPLETE ALL SECTIONS AS DESCRIBED.
Also test if the latest release, and devel branch are affected too.
ALWAYS add information AFTER (OUTSIDE) these html comments.
Otherwise it may end up being automatically closed by our bot. -->

##### SUMMARY
<!--- Explain the problem briefly -->
I have the following in my configuration file
```
[persistent_connection]
command_timeout = 60
```
as committing a configuration on some of our slower Juniper devices take longer than the default 10 seconds. In Ansible 2.5.3 this worked fine, but in 2.6.1 I once again hit the timeout. If I pass the option as an environmental variable instead
```
ANSIBLE_PERSISTENT_COMMAND_TIMEOUT=60 ansible-playbook test.yml
```
it works as expected. The option in the configuration file no longer appears to be honoured.

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
<!--- Insert, BELOW THIS COMMENT, the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path-->
junos_config

##### ANSIBLE VERSION
<!--- Paste, BELOW THIS COMMENT, verbatim output from "ansible --version" between quotes below -->
```
ansible 2.6.1
  config file = /home/myuser/ansible-test/ansible.cfg
  configured module search path = ['/home/myuser/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/myuser/ansible/lib/python3.6/site-packages/ansible
  executable location = /home/myuser/bin/ansible
  python version = 3.6.4 (default, Dec 19 2017, 14:47:53) [GCC 4.4.7 20120313 (Red Hat 4.4.7-18)]
```

##### CONFIGURATION
<!--- If using Ansible 2.4 or above, paste, BELOW THIS COMMENT, the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).-->
DEFAULT_HOST_LIST(/home/myuser/ansible-test/ansible.cfg) = ['/home/myuser/ansible-test/inventory']
HOST_KEY_CHECKING(/home/myuser/ansible-test/ansible.cfg) = False
PARAMIKO_LOOK_FOR_KEYS(/home/myuser/ansible-test/ansible.cfg) = False
PERSISTENT_COMMAND_TIMEOUT(/home/myuser/ansible-test/ansible.cfg) = 60

##### OS / ENVIRONMENT
<!--- Mention, BELOW THIS COMMENT, the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.-->
OS from: CentOS 6
OS to: JUNOS 12.3X48-D15.4

##### STEPS TO REPRODUCE
<!--- For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used. -->
Use basic configuration file, inventory and playbook as provided below.
Make new device config (test.cfg) a copy of current device config, with a minor change (applied with "load override" to make it more time consuming).
Run "ansible-playbook test.yml" under Ansible 2.5.3 and observe success.
Run "ansible-playbook test.yml" under Ansible 2.6.1 and observe failure.
Run "ANSIBLE_PERSISTENT_COMMAND_TIMEOUT=60 ansible-playbook test.yml" and observe success.

<!--- Paste example playbooks or commands between quotes below -->
Playbook (test.yml)
```yaml
---
- hosts: junos
  gather_facts: no
  tasks:
    - name: check config
      check_mode: yes
      diff: yes
      junos_config:
        src: test.cfg
        update: override
```

Inventory (inventory)
```
[junos]
myhost.mydomain

[junos:vars]
ansible_connection=netconf
ansible_network_os=junos
ansible_user=ansible
ansible_ssh_private_key_file=/home/myuser/ansible-test/mykey
```

Configuration file (ansible.cfg)
```
[defaults]
host_key_checking = False
inventory = /home/myuser/ansible-test/inventory

[paramiko_connection]
look_for_keys = False

[persistent_connection]
command_timeout = 60
```

Packages in old Ansible virtualenv:
```
ansible (2.5.3)
asn1crypto (0.24.0)
bcrypt (3.1.4)
cffi (1.11.5)
cryptography (2.2.2)
idna (2.6)
Jinja2 (2.10)
lxml (4.1.1)
MarkupSafe (1.0)
ncclient (0.5.3)
paramiko (2.4.1)
pip (9.0.1)
pyasn1 (0.4.3)
pycparser (2.18)
PyNaCl (1.2.1)
PyYAML (3.12)
setuptools (39.2.0)
six (1.11.0)
wheel (0.30.0)
```

Packages in new virtualenv:
```
ansible (2.6.1)
asn1crypto (0.24.0)
bcrypt (3.1.4)
cffi (1.11.5)
cryptography (2.3)
idna (2.7)
Jinja2 (2.10)
lxml (4.1.1)
MarkupSafe (1.0)
ncclient (0.5.3)
paramiko (2.4.1)
pip (9.0.1)
pyasn1 (0.4.3)
pycparser (2.18)
PyNaCl (1.2.1)
PyYAML (3.13)
setuptools (40.0.0)
six (1.11.0)
wheel (0.30.0)
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
Under Ansible 2.5.3 the following works as expected:
```
# ansible-playbook test.yml

PLAY [junos] *******************************************************************

TASK [check config] ************************************************************
[edit groups]
+  TEST;
changed: [myhost.mydomain]

PLAY RECAP *********************************************************************
myhost.mydomain       : ok=1    changed=1    unreachable=0    failed=0
```

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
But under 2.6.1 it fails:
```
# ansible-playbook test.yml

PLAY [junos] *******************************************************************

TASK [check config] ************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible.module_utils.connection.ConnectionError: socket_path does not exist or cannot be found. Please check https://docs.ansible.com/ansible/latest/network/user_guide/network_debug_troubleshooting.html#category-socket-path-issue
fatal: [myhost.mydomain]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_nfn0mG/ansible_module_junos_config.py\", line 390, in <module>\n    main()\n  File \"/tmp/ansible_nfn0mG/ansible_module_junos_config.py\", line 377, in main\n    result['diff'] = {'prepared': diff}\n  File \"/usr/lib64/python2.6/contextlib.py\", line 34, in __exit__\n    self.gen.throw(type, value, traceback)\n  File \"/tmp/ansible_nfn0mG/ansible_modlib.zip/ansible/module_utils/network/junos/junos.py\", line 180, in locked_config\n  File \"/tmp/ansible_nfn0mG/ansible_modlib.zip/ansible/module_utils/network/junos/junos.py\", line 171, in unlock_configuration\n  File \"/tmp/ansible_nfn0mG/ansible_modlib.zip/ansible/module_utils/network/common/netconf.py\", line 73, in __rpc__\n  File \"/tmp/ansible_nfn0mG/ansible_modlib.zip/ansible/module_utils/connection.py\", line 117, in _exec_jsonrpc\nansible.module_utils.connection.ConnectionError: socket_path does not exist or cannot be found. Please check https://docs.ansible.com/ansible/latest/network/user_guide/network_debug_troubleshooting.html#category-socket-path-issue\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1}
        to retry, use: --limit @/home/nsg/ansible-test/test.retry

PLAY RECAP *********************************************************************
myhost.mydomain       : ok=0    changed=0    unreachable=0    failed=1
```

<!--- Paste verbatim command output between quotes below -->
If I set PERSISTENT_COMMAND_TIMEOUT via the environment instead, it works:
```
# ANSIBLE_PERSISTENT_COMMAND_TIMEOUT=60 ansible-playbook test.yml

PLAY [junos] *******************************************************************

TASK [check config] ************************************************************
[edit groups]
+  TEST;
changed: [myhost.mydomain]

PLAY RECAP *********************************************************************
myhost.mydomain       : ok=1    changed=1    unreachable=0    failed=0
```

If I set "ANSIBLE_LOG_PATH=ansible.log ANSIBLE_DEBUG=True" and run with "-vvvvv", I see this in the log file
```
2018-07-22 11:31:46,147 ncclient.operations.rpc Requesting 'Validate'
2018-07-22 11:31:56,147 p=65252 u=myuser |  command timeout triggered, timeout value is 10 secs
2018-07-22 11:31:56,147 ncclient.operations.rpc Requesting 'CloseSession'
```
despite having set the command timeout to 60 secs via the configuration.


Discussion:
Files identified in the description:
* [lib/ansible/modules/network/junos/junos_config.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/network/junos/junos_config.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @Qalthos @ganeshrn @privateip
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
I was stuck on this for quite a while today. Glad to have stumbled on this bug report. Good find, @davidstockdale ... i now have a work-around 👍 

Resolving patch:
diff --git a/lib/ansible/plugins/connection/netconf.py b/lib/ansible/plugins/connection/netconf.py
--- a/lib/ansible/plugins/connection/netconf.py
+++ b/lib/ansible/plugins/connection/netconf.py
@@ -73,8 +73,8 @@
       - The private SSH key or certificate file used to to authenticate to the
         remote device when first establishing the SSH connection.
     ini:
-     section: defaults
-     key: private_key_file
+      - section: defaults
+        key: private_key_file
     env:
       - name: ANSIBLE_PRIVATE_KEY_FILE
     vars:
@@ -97,8 +97,8 @@
         option on production systems as it could create a security vulnerability.
     default: 'no'
     ini:
-      section: paramiko_connection
-      key: host_key_auto_add
+      - section: paramiko_connection
+        key: host_key_auto_add
     env:
       - name: ANSIBLE_HOST_KEY_AUTO_ADD
   persistent_connect_timeout:
@@ -110,8 +110,8 @@
         will fail
     default: 30
     ini:
-      section: persistent_connection
-      key: persistent_connect_timeout
+      - section: persistent_connection
+        key: connect_timeout
     env:
       - name: ANSIBLE_PERSISTENT_CONNECT_TIMEOUT
   persistent_command_timeout:
@@ -123,8 +123,8 @@
         close
     default: 10
     ini:
-      section: persistent_connection
-      key: persistent_command_timeout
+      - section: persistent_connection
+        key: command_timeout
     env:
       - name: ANSIBLE_PERSISTENT_COMMAND_TIMEOUT
 """
diff --git a/lib/ansible/plugins/connection/network_cli.py b/lib/ansible/plugins/connection/network_cli.py
--- a/lib/ansible/plugins/connection/network_cli.py
+++ b/lib/ansible/plugins/connection/network_cli.py
@@ -68,8 +68,8 @@
       - The private SSH key or certificate file used to to authenticate to the
         remote device when first establishing the SSH connection.
     ini:
-     section: defaults
-     key: private_key_file
+      - section: defaults
+        key: private_key_file
     env:
       - name: ANSIBLE_PRIVATE_KEY_FILE
     vars:
@@ -94,8 +94,8 @@
       - Can be configured form the CLI via the C(--become) or C(-b) options
     default: False
     ini:
-      section: privilege_escalation
-      key: become
+      - section: privilege_escalation
+        key: become
     env:
       - name: ANSIBLE_BECOME
     vars:
@@ -107,8 +107,8 @@
         C(enable) but could be defined as other values.
     default: sudo
     ini:
-      section: privilege_escalation
-      key: become_method
+      - section: privilege_escalation
+        key: become_method
     env:
       - name: ANSIBLE_BECOME_METHOD
     vars:
@@ -125,8 +125,8 @@
         option on production systems as it could create a security vulnerability.
     default: False
     ini:
-      section: paramiko_connection
-      key: host_key_auto_add
+      - section: paramiko_connection
+        key: host_key_auto_add
     env:
       - name: ANSIBLE_HOST_KEY_AUTO_ADD
   persistent_connect_timeout:


Test patch: