internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: DataDog/dd-agent

Issue:
[mcache] report `listen_disable_num`
I propose to report `listen_disable_num` value from Memcached statistics. It's of much value when diagnosing connection errors, etc. [memcached Wiki](https://code.google.com/p/memcached/wiki/Timeouts) recommends consulting this value at the first when troubleshooting timeouts. So it would make sense to monitor the increase of such events.



Discussion:
Hi @alaz, thanks for your input! That seems to be something useful we could improve indeed, I add this to our backlog.
Stay tuned.


Resolving patch:
diff --git a/checks.d/mcache.py b/checks.d/mcache.py
--- a/checks.d/mcache.py
+++ b/checks.d/mcache.py
@@ -46,6 +46,8 @@
 #                            use for storage.
 # threads           32u      Number of worker threads requested.
 #                            (see doc/threads.txt)
+# listen_disabled_num 32u    How many times the server has reached maxconns
+#                            (see https://code.google.com/p/memcached/wiki/Timeouts)
 #     >>> mc.get_stats()
 # [('127.0.0.1:11211 (1)', {'pid': '2301', 'total_items': '2',
 # 'uptime': '80', 'listen_disabled_num': '0', 'version': '1.2.8',
@@ -96,7 +98,8 @@ class Memcache(AgentCheck):
         "cas_misses",
         "cas_hits",
         "cas_badval",
-        "total_connections"
+        "total_connections",
+        "listen_disabled_num"
     ]
 
     SERVICE_CHECK = 'memcache.can_connect'


Test patch:
diff --git a/tests/checks/integration/test_mcache.py b/tests/checks/integration/test_mcache.py
--- a/tests/checks/integration/test_mcache.py
+++ b/tests/checks/integration/test_mcache.py
@@ -44,7 +44,8 @@
     "cas_misses",
     "cas_hits",
     "cas_badval",
-    "total_connections"
+    "total_connections",
+    "listen_disabled_num"
 ]
 
 SERVICE_CHECK = 'memcache.can_connect'
Repository: DataDog/dd-agent

Issue:
Cannot find Docker cgroup directory. Be sure your system is supported.
Running docker 1.8.3 on Ubuntu 14.04. Running dd-agent under Kubernetes.  Kept getting the error message in the subject until I made a patch to 5.6.2 https://gist.github.com/bakins/00cc12f26e2947af6679

I am not running the packaged version of docker (ie, the deb) - just the binary (issues with Kubernetes for me using packaged version).



Discussion:
Thanks for the feedback and the patch @bakins
Going to add this new directory to the Agent. 


Resolving patch:
diff --git a/utils/dockerutil.py b/utils/dockerutil.py
--- a/utils/dockerutil.py
+++ b/utils/dockerutil.py
@@ -89,6 +89,7 @@ def find_cgroup_filename_pattern(mountpoints, container_id):
         stat_file_path_coreos = os.path.join(mountpoint, "system.slice")
         stat_file_path_kubernetes = os.path.join(mountpoint, container_id)
         stat_file_path_kubernetes_docker = os.path.join(mountpoint, "system", "docker", container_id)
+        stat_file_path_docker_daemon = os.path.join(mountpoint, "docker-daemon", "docker", container_id)
 
         if os.path.exists(stat_file_path_lxc):
             return os.path.join('%(mountpoint)s/lxc/%(id)s/%(file)s')
@@ -100,6 +101,9 @@ def find_cgroup_filename_pattern(mountpoints, container_id):
             return os.path.join('%(mountpoint)s/%(id)s/%(file)s')
         elif os.path.exists(stat_file_path_kubernetes_docker):
             return os.path.join('%(mountpoint)s/system/docker/%(id)s/%(file)s')
+        elif os.path.exists(stat_file_path_docker_daemon):
+            return os.path.join('%(mountpoint)s/docker-daemon/docker/%(id)s/%(file)s')
+
 
     raise MountException("Cannot find Docker cgroup directory. Be sure your system is supported.")
 


Test patch:
Repository: DataDog/dd-agent

Issue:
Check for FreeBSD for syslog
https://github.com/DataDog/dd-agent/blob/master/config.py#L1027
- I check you want to use the Platform.is_darwin check here?
- For FreeBSD the file is /var/run/log



Discussion:
Thanks for the feedback @zecke.

We'll work on a fix for the next release.


Resolving patch:
diff --git a/config.py b/config.py
--- a/config.py
+++ b/config.py
@@ -1035,9 +1035,11 @@ def initialize_logging(logger_name):
                     sys_log_addr = (logging_config['syslog_host'], logging_config['syslog_port'])
                 else:
                     sys_log_addr = "/dev/log"
-                    # Special-case macs
-                    if sys.platform == 'darwin':
+                    # Special-case BSDs
+                    if Platform.is_darwin():
                         sys_log_addr = "/var/run/syslog"
+                    elif Platform.is_freebsd():
+                        sys_log_addr = "/var/run/log"
 
                 handler = SysLogHandler(address=sys_log_addr, facility=SysLogHandler.LOG_DAEMON)
                 handler.setFormatter(logging.Formatter(get_syslog_format(logger_name), get_log_date_format()))


Test patch:
Repository: DataDog/dd-agent

Issue:
docker_daemon reports warnings when containers are excluded
Sample warnings are below. This occurs because _crawl_container_pids() tries to access container_dict[container_id] but since the container is excluded, _get_and_count_containers() does not create an entry for it in the dict. Presumably the fix is to skip excluded containers in _crawl_container_pids()

Datadog's docker_daemon integration is reporting:
Instance #0[WARNING]:Cannot parse /proc/4619/cgroup content: '14185c665d9131004725d9f7b8dc92fc797f5b58e8b7250608941751e2092f25'



Discussion:

Resolving patch:
diff --git a/checks.d/docker_daemon.py b/checks.d/docker_daemon.py
--- a/checks.d/docker_daemon.py
+++ b/checks.d/docker_daemon.py
@@ -718,6 +718,9 @@ def _crawl_container_pids(self, container_dict):
                 match = CONTAINER_ID_RE.search(cpuacct)
                 if match:
                     container_id = match.group(0)
+                    if container_id not in container_dict:
+                        self.log.debug("Container %s not in container_dict, it's likely excluded", container_id)
+                        continue
                     container_dict[container_id]['_pid'] = folder
                     container_dict[container_id]['_proc_root'] = os.path.join(proc_path, folder)
             except Exception, e:


Test patch:
Repository: DataDog/dd-agent

Issue:
[docker] support tagging for events
Despite the documentation, the docker agent module does not include tags when creating events:

```
https://github.com/DataDog/dd-agent/blob/master/checks.d/docker.py#L321-L328
```



Discussion:
Hey @robparrott, thanks for opening this issue.
Can you:
- indicate what you're referring to by _the documentation_?
- tell us what kind of tags you'd like to see in docker events?

:+1: for this.

Events pertaining to a container should be tagged with the same tags that metrics include. This would probably require keeping an internal mapping of `container id -> tags` for every container and pruning the list when containers die.

Having this feature would be great since it's currently not possible to filter docker events by `image_name` or `container_name` (or any docker labels if you're setting `collect_labels_as_tags`).


Resolving patch:
diff --git a/checks.d/docker_daemon.py b/checks.d/docker_daemon.py
--- a/checks.d/docker_daemon.py
+++ b/checks.d/docker_daemon.py
@@ -599,15 +599,17 @@ def _format_events(self, aggregated_events, containers_by_id):
             max_timestamp = 0
             status = defaultdict(int)
             status_change = []
-            container_names = set()
+            container_tags = set()
             for event in event_group:
                 max_timestamp = max(max_timestamp, int(event['time']))
                 status[event['status']] += 1
                 container_name = event['id'][:11]
                 if event['id'] in containers_by_id:
-                    container_name = container_name_extractor(containers_by_id[event['id']])[0]
+                    cont = containers_by_id[event['id']]
+                    container_name = container_name_extractor(cont)[0]
+                    container_tags.update(self._get_tags(cont, PERFORMANCE))
+                    container_tags.add('container_name:%s' % container_name)
 
-                container_names.add(container_name)
                 status_change.append([container_name, event['status']])
 
             status_text = ", ".join(["%d %s" % (count, st) for st, count in status.iteritems()])
@@ -632,7 +634,7 @@ def _format_events(self, aggregated_events, containers_by_id):
                 'msg_text': msg_body,
                 'source_type_name': EVENT_TYPE,
                 'event_object': 'docker:%s' % image_name,
-                'tags': ['container_name:%s' % c_name for c_name in container_names]
+                'tags': list(container_tags)
             })
 
         return events


Test patch:
Repository: DataDog/dd-agent

Issue:
Make directory check works on Windows
http://docs.datadoghq.com/integrations/directory/ does not work on Windows now.



Discussion:
I have just run shell.exe with:

```
>>> from checks import run_check
>>> run_check('directory')
```

and got proper metrics.

Why `directory.py` marked as not Windows-supported?

@yannmh thoughts on this one?


Resolving patch:
diff --git a/win32/gui.py b/win32/gui.py
--- a/win32/gui.py
+++ b/win32/gui.py
@@ -115,7 +115,6 @@
 EXCLUDED_WINDOWS_CHECKS = [
     'btrfs',
     'cacti',
-    'directory',
     'docker',
     'gearmand',
     'gunicorn',


Test patch:
Repository: DataDog/dd-agent

Issue:
Mesos master port in mesos_slave.py is hardcoded
https://github.com/DataDog/dd-agent/blob/master/checks.d/mesos_slave.py#L132

Effecting mesos_slave configuration doesn't work if the master port is other than 5050.
Probably should be added in configuration.



Discussion:

Resolving patch:
diff --git a/checks.d/mesos_slave.py b/checks.d/mesos_slave.py
--- a/checks.d/mesos_slave.py
+++ b/checks.d/mesos_slave.py
@@ -8,6 +8,7 @@
 # project
 from checks import AgentCheck, CheckException
 
+DEFAULT_MASTER_PORT = 5050
 
 class MesosSlave(AgentCheck):
     GAUGE = AgentCheck.gauge
@@ -123,13 +124,16 @@ def _get_stats(self, url, timeout):
             endpoint = '/stats.json'
         return self._get_json(url + endpoint, timeout)
 
-    def _get_constant_attributes(self, url, timeout):
+    def _get_constant_attributes(self, url, timeout, master_port):
         state_metrics = None
         if self.cluster_name is None:
             state_metrics = self._get_state(url, timeout)
             if state_metrics is not None:
                 self.version = map(int, state_metrics['version'].split('.'))
-                master_state = self._get_state('http://' + state_metrics['master_hostname'] + ':5050', timeout)
+                master_state = self._get_state(
+                    'http://{0}:{1}'.format(state_metrics['master_hostname'], master_port),
+                    timeout
+                )
                 if master_state is not None:
                     self.cluster_name = master_state.get('cluster')
 
@@ -144,8 +148,9 @@ def check(self, instance):
         tasks = instance.get('tasks', [])
         default_timeout = self.init_config.get('default_timeout', 5)
         timeout = float(instance.get('timeout', default_timeout))
+        master_port = instance.get("master_port", DEFAULT_MASTER_PORT)
 
-        state_metrics = self._get_constant_attributes(url, timeout)
+        state_metrics = self._get_constant_attributes(url, timeout, master_port)
         tags = None
 
         if state_metrics is None:


Test patch:
Repository: DataDog/dd-agent

Issue:
checks.http_check(http_check.py:239) | Incorrect HTTP return code. Expected (1|2|3)\d\d, got 403
It seems that list INFO message is incomplete which makes it totally useless. We have more than 20 tests inside the http_check and if this like appears without telling which instance from the http_check caused this, it makes the entire message useless.

```
checks.http_check(http_check.py:239) | Incorrect HTTP return code. Expected (1|2|3)\d\d, got 403
```



Discussion:
Thanks for reporting @ssbarnea . Fair, that log message could certainly be more helpful. Queueing it up, should be a simple fix.


Resolving patch:
diff --git a/checks.d/http_check.py b/checks.d/http_check.py
--- a/checks.d/http_check.py
+++ b/checks.d/http_check.py
@@ -28,6 +28,7 @@
 from util import headers as agent_headers
 from utils.proxy import get_proxy
 
+DEFAULT_EXPECTED_CODE = "(1|2|3)\d\d"
 
 class WeakCiphersHTTPSConnection(urllib3.connection.VerifiedHTTPSConnection):
 
@@ -168,7 +169,7 @@ def _load_conf(self, instance):
         tags = instance.get('tags', [])
         username = instance.get('username')
         password = instance.get('password')
-        http_response_status_code = str(instance.get('http_response_status_code', "(1|2|3)\d\d"))
+        http_response_status_code = str(instance.get('http_response_status_code', DEFAULT_EXPECTED_CODE))
         timeout = int(instance.get('timeout', 10))
         config_headers = instance.get('headers', {})
         headers = agent_headers(self.agentConfig)
@@ -254,14 +255,20 @@ def _check(self, instance):
 
         # Check HTTP response status code
         if not (service_checks or re.match(http_response_status_code, str(r.status_code))):
-            self.log.info("Incorrect HTTP return code. Expected %s, got %s"
-                          % (http_response_status_code, str(r.status_code)))
+            if http_response_status_code == DEFAULT_EXPECTED_CODE:
+                expected_code = "1xx or 2xx or 3xx"
+            else:
+                expected_code = http_response_status_code
+
+            message = "Incorrect HTTP return code for url %s. Expected %s, got %s" % (
+                addr, expected_code, str(r.status_code))
+
+            self.log.info(message)
 
             service_checks.append((
                 self.SC_STATUS,
                 Status.DOWN,
-                "Incorrect HTTP return code. Expected %s, got %s"
-                % (http_response_status_code, str(r.status_code))
+                message
             ))
 
         if not service_checks:


Test patch:
Repository: DataDog/dd-agent

Issue:
[zookeeper] metric is misleading
zookeeper.bytes_receieved_per_second
zookeeper.bytes_received
zookeeper.bytes_sent_per_second
zookeeper.bytes_sent

All derive from the zookeeper internals that are counting packets, not bytes.

See https://github.com/apache/zookeeper/blob/bd9a1448f9b29859092e6bdca93da121ec166b7a/src/java/main/org/apache/zookeeper/server/ServerStats.java#L83-L89



Discussion:
Thanks for reporting this issue @miketheman,

I am adding the ticket to our 5.8.0 milestone.

@yannmh Thanks! Considering the `bytes_per_second` metrics are new and unreleased, could thse be fixed before 5.7.0 is released?

That's a good point.
I'll work on a quick fix before the 5.7.0 gets released.

Thanks again


Resolving patch:
diff --git a/checks.d/zk.py b/checks.d/zk.py
--- a/checks.d/zk.py
+++ b/checks.d/zk.py
@@ -295,12 +295,16 @@ def parse_stat(self, buf):
         # Received: 101032173
         _, value = buf.readline().split(':')
         metrics.append(ZKMetric('zookeeper.bytes_received', long(value.strip())))
-        metrics.append(ZKMetric('zookeeper.bytes_received_per_second', long(value.strip()), "rate"))
+        metrics.append(
+            ZKMetric('zookeeper.packets_received', long(value.strip()), "rate")
+        )
 
         # Sent: 1324
         _, value = buf.readline().split(':')
         metrics.append(ZKMetric('zookeeper.bytes_sent', long(value.strip())))
-        metrics.append(ZKMetric('zookeeper.bytes_sent_per_second', long(value.strip()), "rate"))
+        metrics.append(
+            ZKMetric('zookeeper.packets_sent', long(value.strip()), "rate")
+        )
 
         if has_connections_val:
             # Connections: 1
@@ -343,7 +347,6 @@ def parse_stat(self, buf):
 
         return metrics, tags, mode, version
 
-
     def parse_mntr(self, buf):
         ''' `buf` is a readable file-like object
             Returns: a tuple (metrics, mode)


Test patch:
diff --git a/tests/checks/integration/test_zk.py b/tests/checks/integration/test_zk.py
--- a/tests/checks/integration/test_zk.py
+++ b/tests/checks/integration/test_zk.py
@@ -39,8 +39,8 @@ class ZooKeeperTestCase(AgentCheckTest):
         'zookeeper.latency.max',
         'zookeeper.bytes_received',
         'zookeeper.bytes_sent',
-        'zookeeper.bytes_received_per_second',
-        'zookeeper.bytes_sent_per_second',
+        'zookeeper.packets_received',
+        'zookeeper.packets_sent',
         'zookeeper.connections',
         'zookeeper.connections',
         'zookeeper.bytes_outstanding',
Repository: DataDog/dd-agent

Issue:
5.7.1 consul check fails when /v1/status/peers returns null
Testing datadog consul check on 1 node consul server, and it's crashing on the agent check:

```
2016-03-10 09:16:21,708 | INFO | dd.collector | checks.collector(collector.py:533) | Running check consul
2016-03-10 09:16:21,859 | WARNING | dd.collector | checks.consul(consul.py:109) | Consul Leader information is not available!
2016-03-10 09:16:21,863 | ERROR | dd.collector | checks.consul(__init__.py:763) | Check 'consul' instance #0 failed
Traceback (most recent call last):
  File "/opt/datadog-agent/agent/checks/__init__.py", line 746, in run
    self.check(copy.deepcopy(instance))
  File "/opt/datadog-agent/agent/checks.d/consul.py", line 179, in check
    self.gauge("consul.peers", len(peers), tags=main_tags + ["mode:follower"])
TypeError: object of type 'NoneType' has no len()
Metrics: 
[]
Events: 
[]
Service Checks: 
[]
Service Metadata: 
[{}]
    consul
    ------
      - instance #0 [ERROR]: "object of type 'NoneType' has no len()"
        Traceback (most recent call last):
          File "/opt/datadog-agent/agent/checks/__init__.py", line 746, in run
            self.check(copy.deepcopy(instance))
          File "/opt/datadog-agent/agent/checks.d/consul.py", line 179, in check
            self.gauge("consul.peers", len(peers), tags=main_tags + ["mode:follower"])
        TypeError: object of type 'NoneType' has no len()

      - Collected 0 metrics, 0 events & 0 service checks
```

Querying the endpoint shows it returns 'null':

```
ubuntu@ip-10-10-129-217:~$ curl http://localhost:8500/v1/status/peers
nullubuntu@ip-10-10-129-217:~$ 
```

I suspect that's somehow causing `peers` to equal `None` and thus `len(peers)` results in error?



Discussion:

Resolving patch:
diff --git a/checks.d/consul.py b/checks.d/consul.py
--- a/checks.d/consul.py
+++ b/checks.d/consul.py
@@ -137,7 +137,7 @@ def _check_for_leader_change(self, instance):
 
     ### Consul Catalog Accessors
     def get_peers_in_cluster(self, instance):
-        return self.consul_request(instance, '/v1/status/peers')
+        return self.consul_request(instance, '/v1/status/peers') or []
 
     def get_services_in_cluster(self, instance):
         return self.consul_request(instance, '/v1/catalog/services')


Test patch:
Repository: DataDog/dd-agent

Issue:
[marathon] Marathon plugin slows down agent when marathon has many apps running
We are monitoring a marathon framework using datadog which has over 150 apps, and the marathon check seems to be slowing down the entire datadog process.

After investigating what the plugin actually does, the problem seems to be this loop: https://github.com/DataDog/dd-agent/blob/5.4.4/checks.d/marathon.py#L46.  It appears that the agent is sequentially hitting the API 150 times, which is enough to stop the agent from reporting metrics long enough to trigger some of our other alerts.



Discussion:
Hi @Zarkantho,

Sorry to hear you have trouble with our Marathon integration. Could you reach out to support AT datadoghq.com with those details and a ['flare'](https://github.com/DataDog/dd-agent/wiki/Send-logs-to-support) archive please ? 

That's a valuable feedback to help us understanding your needs and improving the check.  Thank you.

Hi @yannmh,

Sorry about this, but we actually already replaced the marathon check with a custom check to work around this problem, which means the logs would probably not be relevant at this point.

All we did was comment out this line: https://github.com/DataDog/dd-agent/blob/5.4.4/checks.d/marathon.py#L53, since we don't really care about the "versions" of each application in marathon.  This made the problem go away.

Sorry again that we can't use the standard tool here, but I hope this description was at least somewhat helpful for diagnosing the problem.

Happy to know you find a solution to this issue.

Still, let's keep the ticket opened so we can assess for the 5.6.0 agent release based on your feedback. Thanks !


Resolving patch:
diff --git a/checks.d/marathon.py b/checks.d/marathon.py
--- a/checks.d/marathon.py
+++ b/checks.d/marathon.py
@@ -56,12 +56,6 @@ def check(self, instance):
                     if attr in app:
                         self.gauge('marathon.' + attr, app[attr], tags=tags)
 
-                query_url = urljoin(url, "/v2/apps/{0}/versions".format(app['id']))
-                versions_reply = self.get_json(query_url, timeout, auth)
-
-                if versions_reply is not None:
-                    self.gauge('marathon.versions', len(versions_reply['versions']), tags=tags)
-
     def get_json(self, url, timeout, auth):
         try:
             r = requests.get(url, timeout=timeout, auth=auth)


Test patch:
Repository: DataDog/dd-agent

Issue:
[wmi] Filter documentation
The comment in https://github.com/DataDog/dd-agent/blob/master/checks/libs/wmi/sampler.py#L314-L317 says that `*` is a valid wilcard to use in WMI filters, but it's not actually processed in the code that follows said comment. 
Only `%` is processed, which is fine, but I think the `*` should be removed from that comment.
A filter with `*` is also used in https://help.datadoghq.com/hc/en-us/articles/205016075-How-to-retrieve-WMI-metrics , it should be changed to `%`

Also there are no examples of wildcards in https://github.com/DataDog/dd-agent/blob/master/conf.d/wmi_check.yaml.example



Discussion:
Sorry for the late answer @mausch.

You are completely right, we'll fix that shortly ! Thanks again.


Resolving patch:
diff --git a/checks/libs/wmi/sampler.py b/checks/libs/wmi/sampler.py
--- a/checks/libs/wmi/sampler.py
+++ b/checks/libs/wmi/sampler.py
@@ -295,8 +295,8 @@ def _format_filter(filters, and_props=[]):
 
                 NOTE: If we just provide a value we defailt to '=' comparison operator.
                 Otherwise, specify the operator in a tuple as above: (comp_op, value)
-                If we detect a wildcard character such as '*' or '%' we will override
-                the operator to use LIKE
+                If we detect a wildcard character ('%') we will override the operator
+                to use LIKE
         """
         def build_where_clause(fltr):
             f = fltr.pop()


Test patch: