Repository: ansible/ansible

Issue:
Ansible/Juniper QFX10K stuck in `re-using existing socket`
##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
junos_config

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.5.0 (detached HEAD 5b1c91bf19) last updated 2017/09/14 17:32:18 (GMT +100)
  config file = /root/DCNetBuild/ansible.cfg
  configured module search path = [u'/usr/share/ansible', u'/usr/local/lib/python2.7/dist-packages/napalm_ansible']
  ansible python module location = /root/ansible/lib/ansible
  executable location = /root/ansible/bin/ansible
  python version = 2.7.9 (default, Jun 29 2016, 13:08:31) [GCC 4.9.2]
```

##### CONFIGURATION
source /root/ansible/hacking/env-setup silent

ansible.cfg
-------------
[defaults]
retry_files_enabled = False
library = /usr/share/ansible:/usr/local/lib/python2.7/dist-packages/napalm_ansible
host_key_checking = False
#log_path = /var/log/ansible.log
[privilege_escalation]
[paramiko_connection]
[ssh_connection]
[persistent_connection]
connect_timeout = 30
connect_retries = 30
connect_interval = 1
[accelerate]
[selinux]
[colors]
[diff]

##### OS / ENVIRONMENT
cat /etc/issue
-------------
Debian GNU/Linux 8 \n \l

Juniper QFX10K
Model: qfx10002-72q
Junos: 15.1X53-D61.7

##### SUMMARY
<!--- Explain the problem briefly -->
We saw 2 out of 4 of our Juniper QFX10K switches/Ansible get in a stuck state where they kept trying `re-using existing socket`. Our QFX5K switches and other devices were unaffected. We have been running Ansible against them daily for about a year and I have never seen this before. `show system connections | grep 830` on the Juniper side showed a stuck `ESTABLISHED` NETCONF connection, even when the control server was shutdown. In order to clear the issue, we needed to delete `~/.ansible/pc` and disable/reenable NETCONF on the Juniper.

##### STEPS TO REPRODUCE
<!---
Only occurred once - haven't successfully reproduced.
-->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- name: Overwriting configuration on live device for devices that are already live
  when: not awaitingfirstprovision and not temporaryskiprun
  junos_config:
     src: spine.j2
     update: override
     backup: no
     src_format: text
     provider:
        host: "{{ inventory_hostname }}.{{ sitedomainname }}"
        username: "XXXXX"
        timeout: 100
```
##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
Normal successful Ansible run with no config changes applied.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
vi /var/log/ansible.log 
2017-09-14 10:55:49,704 p=5626 u=root |  PLAY [] *****************************************************************************************************************************************************
2017-09-14 10:55:49,794 p=5626 u=root |  TASK [ : Generating configuration in XXXXX for any devices that are awaiting first build] ****************************************************
2017-09-14 10:55:49,802 p=5626 u=root |  skipping: []
2017-09-14 10:55:49,814 p=5626 u=root |  TASK [dcspine : Overwriting configuration on live device for devices that are already live] *******************************************************************************
2017-09-14 10:55:50,466 p=5638 u=root |  re-using existing socket for XXXX@XXXXX:830
2017-09-14 10:56:05,474 p=5638 u=root |  connect retry timeout expired, unable to connect to control socket
2017-09-14 10:56:05,474 p=5638 u=root |  persistent_connect_retry_timeout is 15 secs
2017-09-14 10:56:05,489 p=5626 u=root |  fatal: [XXXXXX]: FAILED! => {"changed": false, "failed": true, "msg": "unable to open shell. Please see: https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell"}
2017-09-14 10:56:05,491 p=5626 u=root |  PLAY RECAP ****************************************************************************************************************************************************************
2017-09-14 10:56:05,491 p=5626 u=root |  XXXXXX            : ok=0    changed=0    unreachable=0    failed=1

And on the Juniper side, for a significant time (several hours) we saw this, even when the control server was rebooted.

{master:0}
XXXX@XXXXX> show system connections | grep 830    
tcp4       0      0  X.X.X.X.830                               X.X.X.X.37758                               ESTABLISHED
tcp4       0      0  *.830                                         *.*                                           LISTEN

<!--- Paste verbatim command output between quotes below -->
```

```



Discussion:
cc @Qalthos @ganeshrn @gundalow @privateip @rcarrillocruz @trishnaguha
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
@christ2016 This seems to be a socket connect timeout issue.
```
2017-09-14 10:56:05,474 p=5638 u=root | persistent_connect_retry_timeout is 15 secs
```
From 2.4 onwards `connect_retries` and `connect_interval` setting is replaced by single timer called `connect_retry_timeout `, more details can be found [here](https://github.com/ansible/ansible/blob/devel/examples/ansible.cfg#L421)

Please remove `connect_retries` and `connect_interval` config setting and add
`connect_retry_timeout=30` under `persistent_connection` section in `ansible.cfg` file and retry. 


needs_info
Many thanks, I have made the proposed changes. Currently everything seems to be working fine - however this problem was intermittent so we won't know for sure for a while.

Resolving patch:
diff --git a/lib/ansible/modules/monitoring/monit.py b/lib/ansible/modules/monitoring/monit.py
--- a/lib/ansible/modules/monitoring/monit.py
+++ b/lib/ansible/modules/monitoring/monit.py
@@ -100,7 +100,7 @@ def parse_current(parts):
         else:
             return ''
 
-    def status():
+    def get_status():
         """Return the status of the process in monit, or the empty string if not present."""
         rc, out, err = module.run_command('%s %s' % (MONIT, SUMMARY_COMMAND), check_rc=True)
         for line in out.split('\n'):
@@ -116,14 +116,14 @@ def status():
     def run_command(command):
         """Runs a monit command, and returns the new status."""
         module.run_command('%s %s %s' % (MONIT, command, name), check_rc=True)
-        return status()
+        return get_status()
 
     def wait_for_monit_to_stop_pending():
         """Fails this run if there is no status or it's pending/initializing for timeout"""
         timeout_time = time.time() + timeout
         sleep_time = 5
 
-        running_status = status()
+        running_status = get_status()
         while running_status == '' or 'pending' in running_status or 'initializing' in running_status:
             if time.time() >= timeout_time:
                 module.fail_json(
@@ -134,7 +134,7 @@ def wait_for_monit_to_stop_pending():
                 )
 
             time.sleep(sleep_time)
-            running_status = status()
+            running_status = get_status()
 
     MONIT_MAJOR_VERSION, MONIT_MINOR_VERSION = monit_version()
 
@@ -149,7 +149,7 @@ def wait_for_monit_to_stop_pending():
         wait_for_monit_to_stop_pending()
         module.exit_json(changed=True, name=name, state=state)
 
-    present = status() != ''
+    present = get_status() != ''
 
     if not present and not state == 'present':
         module.fail_json(msg='%s process not presently configured with monit' % name, name=name, state=state)
@@ -165,7 +165,7 @@ def wait_for_monit_to_stop_pending():
         module.exit_json(changed=False, name=name, state=state)
 
     wait_for_monit_to_stop_pending()
-    running = 'running' in status()
+    running = 'running' in get_status()
 
     if running and state in ['started', 'monitored']:
         module.exit_json(changed=False, name=name, state=state)


Test patch: