internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: DataDog/dd-agent

Issue:
[zookeeper] bytes_outstanding metric returns number of requests served by zk node
User checking through the ZooKeeper check noticed that the bytes_outsanding metric the DD agent is scrapping is not bytes, but rather the number of inflight requests currently being serviced by the ZK node.

https://github.com/apache/zookeeper/blob/ca480231e1bdcbe6beda9142766fc3caa2031cd8/src/java/main/org/apache/zookeeper/server/ServerStats.java#L104

https://github.com/apache/zookeeper/blob/ca480231e1bdcbe6beda9142766fc3caa2031cd8/src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java#L740



Discussion:
Thanks a lot @handigarde looks like you're right.


Resolving patch:
diff --git a/checks.d/zk.py b/checks.d/zk.py
--- a/checks.d/zk.py
+++ b/checks.d/zk.py
@@ -175,7 +175,10 @@ def parse_stat(self, buf):
 
         # Outstanding: 0
         _, value = buf.readline().split(':')
+        # Fixme: This metric name is wrong. It should be removed in a major version of the agent
+        # See https://github.com/DataDog/dd-agent/issues/1383
         metrics.append(('zookeeper.bytes_outstanding', long(value.strip())))
+        metrics.append(('zookeeper.outstanding_requests', long(value.strip())))
 
         # Zxid: 0x1034799c7
         _, value = buf.readline().split(':')


Test patch:
diff --git a/tests/test_zookeeper.py b/tests/test_zookeeper.py
--- a/tests/test_zookeeper.py
+++ b/tests/test_zookeeper.py
@@ -86,6 +86,7 @@ def test_zk_stat_parsing_lt_v344(self):
             ('zookeeper.bytes_sent',                0L),
             ('zookeeper.connections',                6),
             ('zookeeper.bytes_outstanding',         0L),
+            ('zookeeper.outstanding_requests',      0L),
             ('zookeeper.zxid.epoch',                 1),
             ('zookeeper.zxid.count',          55024071),
             ('zookeeper.nodes',                    487L),
@@ -115,6 +116,7 @@ def test_zk_stat_parsing_gte_v344(self):
             ('zookeeper.bytes_sent',                0L),
             ('zookeeper.connections',                1),
             ('zookeeper.bytes_outstanding',         0L),
+            ('zookeeper.outstanding_requests',      0L),
             ('zookeeper.zxid.epoch',                 1),
             ('zookeeper.zxid.count',          55024071),
             ('zookeeper.nodes',                    487L),
Repository: DataDog/dd-agent

Issue:
SQL Server hang
SQL Server hanging when trying to run RECONFIGURE - caused by SQLServer check in Agent 

See: https://datadog.desk.com/agent/case/19284



Discussion:
Fixed by https://github.com/DataDog/dd-agent/pull/1440 thanks @degemer !


Resolving patch:
diff --git a/checks.d/sqlserver.py b/checks.d/sqlserver.py
--- a/checks.d/sqlserver.py
+++ b/checks.d/sqlserver.py
@@ -48,6 +48,8 @@ class SQLServer(AgentCheck):
 
     SOURCE_TYPE_NAME = 'sql server'
     SERVICE_CHECK_NAME = 'sqlserver.can_connect'
+    # FIXME: 6.x, set default to 5s (like every check)
+    DEFAULT_COMMAND_TIMEOUT = 30
 
     METRICS = [
         ('sqlserver.buffer.cache_hit_ratio', 'Buffer cache hit ratio', ''),  # RAW_LARGE_FRACTION
@@ -181,8 +183,11 @@ def get_cursor(self, instance):
 
         if conn_key not in self.connections:
             try:
-                conn_str = self._conn_string(instance)
-                conn = adodbapi.connect(conn_str)
+                conn = adodbapi.connect(
+                    self._conn_string(instance),
+                    timeout=int(instance.get('command_timeout',
+                                             self.DEFAULT_COMMAND_TIMEOUT))
+                )
                 self.connections[conn_key] = conn
                 self.service_check(self.SERVICE_CHECK_NAME, AgentCheck.OK, tags=service_check_tags)
             except Exception:


Test patch:
Repository: DataDog/dd-agent

Issue:
Agent should collect system swap metrics
When doing system checks with the agent, we don't look at /proc/vmstat, so we don't get metrics like "swapped in" and "swapped out".  We should add this to our unix system checks.



Discussion:
We would love to see this implemented. We are investigating switching from a very large graphite + dashboard implmentation to datadog for monitoring and alerting. This is one of the pain points in switching.

+1

Thanks for the feedback @dtorgy .
Changing the milestone to 5.3.0 to figure out what we can do.


Resolving patch:
diff --git a/checks.d/system_swap.py b/checks.d/system_swap.py
new file mode 100644
--- /dev/null
+++ b/checks.d/system_swap.py
@@ -0,0 +1,9 @@
+from checks import AgentCheck
+import psutil
+
+class SystemSwap(AgentCheck):
+
+    def check(self, instance):
+        swap_mem = psutil.swap_memory()
+        self.rate('system.swap.swapped_in', swap_mem.sin)
+        self.rate('system.swap.swapped_out', swap_mem.sout)


Test patch:
diff --git a/tests/test_system_swap.py b/tests/test_system_swap.py
new file mode 100644
--- /dev/null
+++ b/tests/test_system_swap.py
@@ -0,0 +1,29 @@
+import mock
+
+from tests.common import AgentCheckTest
+
+class _PSUtilSwapStatsMock(object):
+    def __init__(self, sin, sout):
+        self.sin = sin
+        self.sout = sout
+
+ORIG_SWAP_IN = 115332743168
+ORIG_SWAP_OUT = 22920884224
+
+SWAP_IN_INCR = 2
+SWAP_OUT_INCR = 4
+
+MOCK_PSUTIL_SWAP_STATS = [
+    _PSUtilSwapStatsMock(ORIG_SWAP_IN, ORIG_SWAP_OUT),
+    _PSUtilSwapStatsMock(ORIG_SWAP_IN + SWAP_IN_INCR, ORIG_SWAP_OUT + SWAP_OUT_INCR),
+]
+
+class SystemSwapTestCase(AgentCheckTest):
+
+    CHECK_NAME = 'system_swap'
+
+    @mock.patch('psutil.swap_memory', side_effect=MOCK_PSUTIL_SWAP_STATS)
+    def test_system_swap(self, mock_swap_stats):
+        self.run_check_twice({"instances": [{}]}) # Run check twice, sleeping for 1 sec in between
+        self.assertMetric('system.swap.swapped_in', value=SWAP_IN_INCR, count=1)
+        self.assertMetric('system.swap.swapped_out', value=SWAP_OUT_INCR, count=1)
Repository: DataDog/dd-agent

Issue:
[kafka_consumer] Kafka client takes 2 minutes to time out
If the KafkaClient can not connect to kafka, it takes 2 minutes to time out. The code at https://github.com/DataDog/dd-agent/blob/5.2.x/checks.d/kafka_consumer.py#L59-L72 is using the default timeout for kafka-client, which is 120 seconds: https://github.com/mumrah/kafka-python/blob/92aa7e94288cbfc4aed0dfbd52021d21694bced4/kafka/conn.py#L14

During the time that it's trying to connect to kafka, no data is reported by the agent.



Discussion:

Resolving patch:
diff --git a/checks.d/kafka_consumer.py b/checks.d/kafka_consumer.py
--- a/checks.d/kafka_consumer.py
+++ b/checks.d/kafka_consumer.py
@@ -1,20 +1,30 @@
 # stdlib
 from collections import defaultdict
-import random
 
-# project
-from checks import AgentCheck
-
-# 3rd party
+# 3p
 from kafka.client import KafkaClient
 from kafka.common import OffsetRequest
 from kazoo.client import KazooClient
 from kazoo.exceptions import NoNodeError
 
+# project
+from checks import AgentCheck
+
+DEFAULT_KAFKA_TIMEOUT = 5
+DEFAULT_ZK_TIMEOUT = 5
+
+
 class KafkaCheck(AgentCheck):
 
     SOURCE_TYPE_NAME = 'kafka'
 
+    def __init__(self, name, init_config, agentConfig, instances=None):
+        AgentCheck.__init__(self, name, init_config, agentConfig, instances=instances)
+        self.zk_timeout = int(
+            init_config.get('zk_timeout', DEFAULT_ZK_TIMEOUT))
+        self.kafka_timeout = int(
+            init_config.get('kafka_timeout', DEFAULT_KAFKA_TIMEOUT))
+
     def check(self, instance):
         consumer_groups = self.read_config(instance, 'consumer_groups',
                                            cast=self._validate_consumer_groups)
@@ -26,7 +36,7 @@ def check(self, instance):
         zk_path_tmpl = zk_prefix + '/consumers/%s/offsets/%s/%s'
 
         # Connect to Zookeeper
-        zk_conn = KazooClient(zk_connect_str)
+        zk_conn = KazooClient(zk_connect_str, timeout=self.zk_timeout)
         zk_conn.start()
 
         try:
@@ -56,7 +66,7 @@ def check(self, instance):
                 self.log.exception('Error cleaning up Zookeeper connection')
 
         # Connect to Kafka
-        kafka_conn = KafkaClient(kafka_host_ports)
+        kafka_conn = KafkaClient(kafka_host_ports, timeout=self.kafka_timeout)
 
         try:
             # Query Kafka for the broker offsets


Test patch:
Repository: DataDog/dd-agent

Issue:
[GCE] Non unique hostname
The agent currently uses the instance name as the hostname of GCE instances.
If one has several projects with instances having the same name, this hostname isn't unique, thus the tags and metrics not matching the expected values.
A unique hostname would be either what's called `internal-hostname` or `name-project`



Discussion:

Resolving patch:
diff --git a/checks/collector.py b/checks/collector.py
--- a/checks/collector.py
+++ b/checks/collector.py
@@ -761,6 +761,11 @@ def _get_hostname_metadata(self):
 
         metadata["hostname"] = get_hostname()
 
+        # Add cloud provider aliases
+        host_aliases = GCE.get_host_aliases(self.agentConfig)
+        if host_aliases:
+            metadata['host_aliases'] = host_aliases
+
         return metadata
 
     def _should_send_additional_data(self, data_name):
diff --git a/config.py b/config.py
--- a/config.py
+++ b/config.py
@@ -528,6 +528,10 @@ def get_config(parse_args=True, cfg_path=None, options=None):
         if config.has_option("Main", "utf8_decoding"):
             agentConfig["utf8_decoding"] = _is_affirmative(config.get("Main", "utf8_decoding"))
 
+        agentConfig["gce_updated_hostname"] = False
+        if config.has_option("Main", "gce_updated_hostname"):
+            agentConfig["gce_updated_hostname"] = _is_affirmative(config.get("Main", "gce_updated_hostname"))
+
     except ConfigParser.NoSectionError, e:
         sys.stderr.write('Config file not found or incorrectly formatted.\n')
         sys.exit(2)
diff --git a/util.py b/util.py
--- a/util.py
+++ b/util.py
@@ -234,7 +234,7 @@ class GCE(object):
     TIMEOUT = 0.1 # second
     SOURCE_TYPE_NAME = 'google cloud platform'
     metadata = None
-    EXCLUDED_ATTRIBUTES = ["sshKeys"]
+    EXCLUDED_ATTRIBUTES = ["sshKeys", "user-data", "cli-cert", "ipsec-cert", "ssl-cert"]
 
 
     @staticmethod
@@ -304,7 +304,21 @@ def get_tags(agentConfig):
     def get_hostname(agentConfig):
         try:
             host_metadata = GCE._get_metadata(agentConfig)
-            return host_metadata['instance']['hostname'].split('.')[0]
+            hostname = host_metadata['instance']['hostname']
+            if agentConfig.get('gce_updated_hostname'):
+                return hostname
+            else:
+                return hostname.split('.')[0]
+        except Exception:
+            return None
+
+    @staticmethod
+    def get_host_aliases(agentConfig):
+        try:
+            host_metadata = GCE._get_metadata(agentConfig)
+            project_id = host_metadata['project']['projectId']
+            instance_name = host_metadata['instance']['hostname'].split('.')[0]
+            return ['%s.%s' % (instance_name, project_id)]
         except Exception:
             return None
 


Test patch:
Repository: DataDog/dd-agent

Issue:
[process] Correctly handle psutil.NoProcessFound when getting process state
In situations where a process disappears after we discover its PID, we should be skipping over the subsequent interrogation of the process [here](https://github.com/DataDog/dd-agent/blob/cc5a67bb47110ef145e58561ab4471b946354742/checks.d/process.py#L173-L181). Currently the check is throwing errors like:

```
  File "/opt/datadog-agent/agent/checks.d/process.py", line 181, in get_process_state
    meminfo = self.psutil_wrapper(p, 'memory_info', ['rss', 'vms'])
UnboundLocalError: local variable 'p' referenced before assignment
```



Discussion:
We're seeing this too, thanks for attending to it.

+1 noticed this as well


Resolving patch:
diff --git a/checks.d/process.py b/checks.d/process.py
--- a/checks.d/process.py
+++ b/checks.d/process.py
@@ -175,8 +175,9 @@ def get_process_state(self, name, pids, cpu_check_interval):
             # Skip processes dead in the meantime
             except psutil.NoSuchProcess:
                 self.warning('Process %s disappeared while scanning' % pid)
-                # reset the PID cache now, something chaned
+                # reset the PID cache now, something changed
                 self.last_pid_cache_ts[name] = 0
+                continue
 
             meminfo = self.psutil_wrapper(p, 'memory_info', ['rss', 'vms'])
             st['rss'].append(meminfo.get('rss'))
@@ -259,10 +260,10 @@ def check(self, instance):
 
     def _process_service_check(self, name, nb_procs, bounds):
         '''
-        Repport a service check, for each processes in search_string.
-        Repport as OK if the process is in the warning thresolds
-                   CRITICAL             out of the critical thresolds
-                   WARNING              out of the warning thresolds
+        Report a service check, for each process in search_string.
+        Report as OK if the process is in the warning thresholds
+                   CRITICAL             out of the critical thresholds
+                   WARNING              out of the warning thresholds
         '''
         tag = ["process:%s" % name]
         status = AgentCheck.OK


Test patch:
Repository: DataDog/dd-agent

Issue:
Couchbase service check do not send tags on OK status
Missing a `tags=service_check_tags` on line:
https://github.com/DataDog/dd-agent/blob/master/checks.d/couchbase.py#L104

Pretty small fix.



Discussion:

Resolving patch:
diff --git a/checks.d/couchbase.py b/checks.d/couchbase.py
--- a/checks.d/couchbase.py
+++ b/checks.d/couchbase.py
@@ -101,7 +101,8 @@ def get_data(self, server, instance):
                 tags=service_check_tags, message=str(e))
             raise
         else:
-            self.service_check(self.SERVICE_CHECK_NAME, AgentCheck.OK)
+            self.service_check(self.SERVICE_CHECK_NAME, AgentCheck.OK,
+                tags=service_check_tags)
 
         couchbase['stats'] = overall_stats
 


Test patch:
Repository: DataDog/dd-agent

Issue:
WARNING | dd.forwarder | forwarder(ddagent.py:175) | You are a Datadog user so we will send data to https://app.datadoghq.com
`WARNING | dd.forwarder | forwarder(ddagent.py:175) | You are a Datadog user so we will send data to https://app.datadoghq.com`

This line is clearly wrong:
https://github.com/DataDog/dd-agent/blame/master/ddagent.py#L107

I do not see any reason why this would be a warning, when that's the expected behaviour. This should be at INFO level OR even DEBUG one, but clearly not an warning.



Discussion:
I agree with you. We'll change this message LOG level in the 5.5.0. 

Thanks again @ssbarnea !

I'll write that here for posterity:

The reason was that older versions of the Agent that were bundling Pup (https://github.com/DataDog/dd-agent/tree/4.0.0/pup)  could be used without a Datadog account so we wanted to warn users when data was sent to Datadog.


Resolving patch:
diff --git a/config.py b/config.py
--- a/config.py
+++ b/config.py
@@ -360,12 +360,6 @@ def get_config(parse_args=True, cfg_path=None, options=None):
         #
 
         # FIXME unnecessarily complex
-
-        if config.has_option('Main', 'use_dd'):
-            agentConfig['use_dd'] = config.get('Main', 'use_dd').lower() in ("yes", "true")
-        else:
-            agentConfig['use_dd'] = True
-
         agentConfig['use_forwarder'] = False
         if options is not None and options.use_forwarder:
             listen_port = 17123
@@ -400,10 +394,6 @@ def get_config(parse_args=True, cfg_path=None, options=None):
         else:
             agentConfig['use_web_info_page'] = True
 
-        if not agentConfig['use_dd']:
-            sys.stderr.write("Please specify at least one endpoint to send metrics to. This can be done in datadog.conf.")
-            exit(2)
-
         # Which API key to use
         agentConfig['api_key'] = config.get('Main', 'api_key')
 
diff --git a/ddagent.py b/ddagent.py
--- a/ddagent.py
+++ b/ddagent.py
@@ -163,19 +163,14 @@ def get_tr_manager(cls):
 
     @classmethod
     def set_endpoints(cls):
-
-        # Only send data to Datadog if an API KEY exists
-        # i.e. user is also Datadog user
-        try:
-            is_dd_user = 'api_key' in cls._application._agentConfig\
-                and 'use_dd' in cls._application._agentConfig\
-                and cls._application._agentConfig['use_dd']\
-                and cls._application._agentConfig.get('api_key')
-            if is_dd_user:
-                log.warn("You are a Datadog user so we will send data to https://app.datadoghq.com")
-                cls._endpoints.append(DD_ENDPOINT)
-        except Exception:
-            log.info("Not a Datadog user")
+        """
+        Set Datadog endpoint if an API key exists.
+        """
+        if not cls._application._agentConfig.get('api_key'):
+            log.warning(u"No API key was found. Aborting endpoint setting.")
+            return
+
+        cls._endpoints.append(DD_ENDPOINT)
 
     def __init__(self, data, headers, msg_type=""):
         self._data = data


Test patch:
Repository: DataDog/dd-agent

Issue:
Wrong metric type in redis check
the `keyspace_hits` metrics in the redis check are sent as GAUGEs, but should be COUNTERS as their value is always increasing.
<img width="778" alt="screen shot 2015-09-03 at 10 41 01 am" src="https://cloud.githubusercontent.com/assets/1532259/9661350/55a2849a-5228-11e5-98af-3cbcf2729ff0.png">
cc @yannmh 



Discussion:

Resolving patch:
diff --git a/checks.d/redisdb.py b/checks.d/redisdb.py
--- a/checks.d/redisdb.py
+++ b/checks.d/redisdb.py
@@ -48,8 +48,6 @@ class Redis(AgentCheck):
         'expired_keys':                 'redis.keys.expired',
 
         # stats
-        'keyspace_hits':                'redis.stats.keyspace_hits',
-        'keyspace_misses':              'redis.stats.keyspace_misses',
         'latest_fork_usec':             'redis.perf.latest_fork_usec',
 
         # pubsub
@@ -83,6 +81,10 @@ class Redis(AgentCheck):
         'used_cpu_sys_children':        'redis.cpu.sys_children',
         'used_cpu_user':                'redis.cpu.user',
         'used_cpu_user_children':       'redis.cpu.user_children',
+
+        # stats
+        'keyspace_hits':                'redis.stats.keyspace_hits',
+        'keyspace_misses':              'redis.stats.keyspace_misses',
     }
 
     def __init__(self, name, init_config, agentConfig, instances=None):
@@ -205,10 +207,11 @@ def _check_db(self, instance, custom_tags=None):
                     self.gauge(metric, val, tags=db_tags)
 
         # Save a subset of db-wide statistics
-        for k in set(info).intersection(self.GAUGE_KEYS):
-            self.gauge(self.GAUGE_KEYS[k], info[k], tags=tags)
-        for k in set(info).intersection(self.RATE_KEYS):
-            self.rate(self.RATE_KEYS[k], info[k], tags=tags)
+        for info_name, value in info.iteritems():
+            if info_name in self.GAUGE_KEYS:
+                self.gauge(self.GAUGE_KEYS[info_name], info[info_name], tags=tags)
+            elif info_name in self.RATE_KEYS:
+                self.rate(self.RATE_KEYS[info_name], info[info_name], tags=tags)
 
         # Save the number of commands.
         self.rate('redis.net.commands', info['total_commands_processed'],


Test patch:
diff --git a/tests/checks/integration/test_redisdb.py b/tests/checks/integration/test_redisdb.py
--- a/tests/checks/integration/test_redisdb.py
+++ b/tests/checks/integration/test_redisdb.py
@@ -20,7 +20,7 @@
 SLAVE_HEALTHY_PORT = 36379
 SLAVE_UNHEALTHY_PORT = 46379
 DEFAULT_PORT = 6379
-MISSING_KEY_TOLERANCE = 0.5
+MISSING_KEY_TOLERANCE = 0.6
 
 
 @attr(requires='redis')
Repository: DataDog/dd-agent

Issue:
Incorrect conversion of system.io.bytes_per_s metric
Incorrect conversion between megabytes and bytes [HERE](https://github.com/DataDog/dd-agent/blob/09249df7a895c534e6150ca066f544ba72606354/checks/system/unix.py#L75).

Also in the test file: tests/checks/mock/test_sysstat.py

```
'system.io.bytes_per_s': mb_s * 10**6
```

Should be

```
'system.io.bytes_per_s': mb_s * 2**20
```

I'm assuming here that we always mean Mebibytes when we say Megabytes. When metrics metadata is fully implemented the plan is to completely shift to the unambiguous kibi, mebi, gibi byte prefixes and no longer refer to the ambiguous kilo, mega, giga scientific prefixes for byte units.



Discussion:

Resolving patch:
diff --git a/checks/system/unix.py b/checks/system/unix.py
--- a/checks/system/unix.py
+++ b/checks/system/unix.py
@@ -72,7 +72,7 @@ def _parse_darwin(self, output):
         for idx, disk in enumerate(disks):
             kb_t, tps, mb_s = map(float, lastline[(3 * idx):(3 * idx) + 3])  # 3 cols at a time
             io[disk] = {
-                'system.io.bytes_per_s': mb_s * 10**6,
+                'system.io.bytes_per_s': mb_s * 2**20,
             }
         return io
 


Test patch:
Repository: DataDog/dd-agent

Issue:
[nginx] No debug logging when doing Pokémon exception handling
The [nginx code](https://github.com/DataDog/dd-agent/blob/49952b4429b125619bc8d9f51bb6564e7c0d2e12/checks.d/nginx.py#L45-L50) follows a [pattern of catching all exceptions](http://blog.codinghorror.com/new-programming-jargon/) when attempting to submit a metric, which makes sense due to wanting to continue without error. Unfortunately, it additionally eats the real error message and displays a super generic message in the logs. This makes it very difficult to actually debug issues.

I ended up manually modifying the code on my local install to figure out that my `nginx.yaml` had a syntax error in it. It would have been more obvious if the true exception (`unhashable type: 'dict'`) had been printed out. (Additionally, there should probably be error checking when tags are loaded instead of reported.)

I'm willing to fix either or both of these issues, but would like advice on how you would like to see them done (e.g. log at a different level? Log to stderr?). Thanks!



Discussion:

Resolving patch:
diff --git a/checks.d/nginx.py b/checks.d/nginx.py
--- a/checks.d/nginx.py
+++ b/checks.d/nginx.py
@@ -49,8 +49,8 @@ def check(self, instance):
                 name, value, tags, metric_type = row
                 func = funcs[metric_type]
                 func(name, value, tags)
-            except Exception:
-                self.log.error(u'Could not submit metric: %s' % repr(row))
+            except Exception, e:
+                self.log.error(u'Could not submit metric: %s: %s' % (repr(row), str(e)))
 
     def _get_data(self, instance):
         url = instance.get('nginx_status_url')


Test patch: