Repository: DataDog/dd-agent

Issue:
[core] network_check active threads count causing check aborts
There's a [check on network_checks.py:120](https://github.com/DataDog/dd-agent/blob/6df4554465338b22441f5bba04913538fe7ef1a2/checks/network_checks.py#L120-L121) that looks to see if there are too many active threads.

```Python
        if threading.activeCount() > 5 * self.pool_size + 5: # On Windows the agent runs on multiple threads so we need to have an offset of 5 in case the pool_size is 1
            raise Exception("Thread number (%s) is exploding. Skipping this check" % threading.activeCount())
```

This works fine as long as you only have one network_check running on a box or otherwise have network checks configured with similar values for `pool_size`. This breaks when you have one or more large pools alongside smaller ones. This is easy situation to get into if you have a large number of instances in one network_check as it sets the pool size to the number of instances.

All of the big pools instance check just fine, but the smaller network_checks never run. We have a custom snmpwalk based check with a VERY large number of instances that runs alongside a small set. As soon as we turned on the large set we lost the small one.

**Additional environment details (Operating System, Cloud provider, etc):**


**Steps to reproduce the issue:**
1. Configure a `NetworkCheck` derived check with a large number of instances each of which will be active for a bit of time
2. Configure another `NetworkCheck` derived check with a small number of instances, e.g. 1

**Describe the results you received:**

```
2016-12-13 12:33:25 PST | ERROR | dd.collector | checks.snmpwalk(__init__.py:788) | Check 'snmpwalk' instance #0 failed
Traceback (most recent call last):
  File "/opt/datadog-agent/agent/checks/__init__.py", line 771, in run
    self.check(copy.deepcopy(instance))
  File "/opt/datadog-agent/agent/checks/network_checks.py", line 122, in check
    raise Exception("Thread number (%s) is exploding. Skipping this check" % threading.activeCount())
Exception: Thread number (32) is exploding. Skipping this check
```

**Describe the results you expected:**

The active thread count should be looked at globally before kicking off the whole check process and take into account the sum of all the pool_sizes rather than compare the global active_threads to a per-instance pool_size.


Discussion:
So I've dug around and found a workaround for the immediate issue. If i set `threads_count` to the same value for all of the network_check based checks it avoids throwing the exception, but it does result in [pre-allocating a lot of threads](https://github.com/DataDog/dd-agent/blob/6df4554465338b22441f5bba04913538fe7ef1a2/checks/libs/thread_pool.py#L83-L84) that go unused.

```yaml
init_config:
    threads_count: <N>
```

Resolving patch:
diff --git a/checks/network_checks.py b/checks/network_checks.py
--- a/checks/network_checks.py
+++ b/checks/network_checks.py
@@ -18,7 +18,6 @@
 MAX_LOOP_ITERATIONS = 1000
 FAILURE = "FAILURE"
 
-
 class Status:
     DOWN = "DOWN"
     WARNING = "WARNING"
@@ -34,6 +33,7 @@ class EventType:
 class NetworkCheck(AgentCheck):
     SOURCE_TYPE_NAME = 'servicecheck'
     SERVICE_CHECK_PREFIX = 'network_check'
+    _global_current_pool_size = 0
 
     STATUS_TO_SERVICE_CHECK = {
         Status.UP : AgentCheck.OK,
@@ -68,6 +68,7 @@ def __init__(self, name, init_config, agentConfig, instances):
         self.statuses = {}
         self.notified = {}
         self.nb_failures = 0
+        self.pool_size = 0
         self.pool_started = False
 
         # Make sure every instance has a name that we use as a unique key
@@ -95,6 +96,9 @@ def start_pool(self):
         default_size = min(self.instance_count(), DEFAULT_SIZE_POOL)
         self.pool_size = int(self.init_config.get('threads_count', default_size))
 
+        # To keep track on the total number of threads we should have running
+        NetworkCheck._global_current_pool_size += self.pool_size
+
         self.pool = Pool(self.pool_size)
 
         self.resultsq = Queue()
@@ -104,6 +108,10 @@ def start_pool(self):
 
     def stop_pool(self):
         self.log.info("Stopping Thread Pool")
+
+        # To keep track on the total number of threads we should have running
+        NetworkCheck._global_current_pool_size -= self.pool_size
+
         if self.pool_started:
             self.pool.terminate()
             self.pool.join()
@@ -117,7 +125,8 @@ def restart_pool(self):
     def check(self, instance):
         if not self.pool_started:
             self.start_pool()
-        if threading.activeCount() > 5 * self.pool_size + 5: # On Windows the agent runs on multiple threads so we need to have an offset of 5 in case the pool_size is 1
+        if threading.activeCount() > 5 * NetworkCheck._global_current_pool_size + 6:
+            # On Windows the agent runs on multiple threads because of WMI so we need an offset of 6
             raise Exception("Thread number (%s) is exploding. Skipping this check" % threading.activeCount())
         self._process_results()
         self._clean()


Test patch: