internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: DataDog/dd-agent

Issue:
Agent 'connection refused' errors on stop/restart
Sometimes when the Agent is stopped or restarted, the forwarder is stopping prior to Dogstatsd. Then when dogstatsd tries to flush metrics to the forwarder it fails.

Everything continues to work, it's just the logging to syslog with errors is disturbing.

```
flushed 0 metrics and 0 events
2014-09-25 16:27:12 EDT | INFO | dd.dogstatsd | dogstatsd(dogstatsd.py:145) | Flush #2: flushed 1 metric and 0 events
2014-09-25 16:27:20 EDT | INFO | dd.dogstatsd | dogstatsd(dogstatsd.py:100) | Stopping reporter
2014-09-25 16:27:20 EDT | ERROR | dd.dogstatsd | dogstatsd(dogstatsd.py:160) | Error flushing metrics
Traceback (most recent call last):
File "/opt/datadog-agent/agent/dogstatsd.py", line 134, in flush
self.submit(metrics)
File "/opt/datadog-agent/agent/dogstatsd.py", line 177, in submit
conn.request(method, url, body, headers)
File "/opt/datadog-agent/embedded/lib/python2.7/httplib.py", line 995, in request
self._send_request(method, url, body, headers)
File "/opt/datadog-agent/embedded/lib/python2.7/httplib.py", line 1029, in _send_request
self.endheaders(body)
File "/opt/datadog-agent/embedded/lib/python2.7/httplib.py", line 991, in endheaders
self._send_output(message_body)
File "/opt/datadog-agent/embedded/lib/python2.7/httplib.py", line 844, in _send_output
self.send(msg)
File "/opt/datadog-agent/embedded/lib/python2.7/httplib.py", line 806, in send
self.connect()
File "/opt/datadog-agent/embedded/lib/python2.7/httplib.py", line 787, in connect
self.timeout, self.source_address)
File "/opt/datadog-agent/embedded/lib/python2.7/socket.py", line 571, in create_connection
raise err
error: [Errno 111] Connection refused
2014-09-25 16:27:20 EDT | INFO | dd.dogstatsd | dogstatsd(dogstatsd.py:343) | Dogstatsd is stopped
2014-09-25 16:29:16 EDT | INFO | dd.dogstatsd | dogstatsd(dogstatsd.py:105) | Reporting to localhost:17123 every 10s
2014-09-25 16:29:16 EDT | INFO | dd.dogstatsd | dogstatsd(dogstatsd.py:271) | Listening on host & port: ('localhost', 8125)
2014-09-25 16:29:26 EDT | INFO | dd.dogstatsd | dogstatsd(dogstatsd.py:145) | Flush #1: flushed 0 metrics and 0 events
```



Discussion:
any workarounds? I'm getting syslogs notifications like crazy.

@kcampos Sorry about that.

We are going to fix that for the next version of the agent that will go out in the next couple of weeks.

In the meantime you could disable the logging to syslog https://github.com/DataDog/dd-agent/blob/master/datadog.conf.example#L204

Thx for the workaround


Resolving patch:
diff --git a/dogstatsd.py b/dogstatsd.py
--- a/dogstatsd.py
+++ b/dogstatsd.py
@@ -157,7 +157,10 @@ def flush(self):
             ).persist()
 
         except Exception:
-            log.exception("Error flushing metrics")
+            if self.finished.isSet():
+                log.debug("Couldn't flush metrics, but that's expected as we're stopping")
+            else:
+                log.exception("Error flushing metrics")
 
     def submit(self, metrics):
         # Copy and pasted from dogapi, because it's a bit of a pain to distribute python


Test patch:
Repository: DataDog/dd-agent

Issue:
[postgres] use better query to determine if master/replica
Currently the method to determine if the replication metrics (delay) should be calculated and reported are using an evaluation on whether the `hot_standby` flag is turned on.
https://github.com/DataDog/dd-agent/blob/0b4af226054d10e43981794c3bc371119b4327e6/checks.d/postgres.py#L168-L171

Master:

``` sql
SELECT count(*) from pg_settings where name = 'hot_standby' AND setting = 'on';
 count
-------
     0
```

Replica:

``` sql
SELECT count(*) from pg_settings where name = 'hot_standby' AND setting = 'on';
 count
-------
     1
```

This flag can only be changed at server startup, and when using master/replica failover techniques, it is uncommon to change a config directive and restart the postgresql postmaster service (which controls all the others), so this flag can remain active and when no longer replicating from another node, the `pg_last_xact_replay_timestamp` will never increase, so dd-agent will then misreport the replication state for a newly-promoted master node.

A better method to detect whether a node is in active replication state is to evaluate whether the node is in "recovery mode" - a state in which streaming replicas are persistently in until they are promoted to accept writes from anyone other than their replication master.

Master:

``` sql
SELECT pg_is_in_recovery();
 pg_is_in_recovery
-------------------
 f
```

Replica:

``` sql
SELECT pg_is_in_recovery();
 pg_is_in_recovery
-------------------
 t
```

Once a replica is promoted to master, the call to `pg_is_in_recovery` will return false without any config changes or server restarts.



Discussion:

Resolving patch:
diff --git a/checks.d/postgres.py b/checks.d/postgres.py
--- a/checks.d/postgres.py
+++ b/checks.d/postgres.py
@@ -166,9 +166,7 @@ class PostgreSql(AgentCheck):
         'relation': False,
         'query': """
 SELECT %s
-  FROM pg_settings
- WHERE name = 'hot_standby'
-   AND setting = 'on'"""
+ WHERE (SELECT pg_is_in_recovery())"""
     }
 
     CONNECTION_METRICS = {


Test patch:
Repository: DataDog/dd-agent

Issue:
Marathon >= 0.7 failing
The Marathon plugin checks the TaskRateLimit attribute from the Marathon 0.6 API version, which has been removed https://github.com/mesosphere/marathon/blob/master/docs/docs/upgrade/06xto070.md#task-rate-limit

you can see the dependency here:
https://github.com/DataDog/dd-agent/blob/73dd75035f1982d31818bb8a4a2a6cc856cf8bb7/checks.d/marathon.py#L30

using marathon v0.7.6, the `dd-agent info` looks like this:

```
    marathon
    --------
      - instance #0 [ERROR]: KeyError('taskRateLimit',)
      - Collected 1 metric, 0 events & 1 service check
```



Discussion:

Resolving patch:
diff --git a/checks.d/marathon.py b/checks.d/marathon.py
--- a/checks.d/marathon.py
+++ b/checks.d/marathon.py
@@ -1,11 +1,9 @@
 # stdlib
 import time
 from hashlib import md5
-import urllib2
 
 # project
 from checks import AgentCheck
-from util import headers
 
 # 3rd party
 import simplejson as json
@@ -27,8 +25,9 @@ def check(self, instance):
             self.gauge('marathon.apps', len(response['apps']), tags=instance_tags)
             for app in response['apps']:
                 tags = ['app_id:' + app['id'], 'version:' + app['version']] + instance_tags
-                for attr in ['taskRateLimit','instances','cpus','mem','tasksStaged','tasksRunning']:
-                    self.gauge('marathon.' + attr, app[attr], tags=tags)
+                for attr in ['taskRateLimit', 'instances', 'cpus', 'mem', 'tasksStaged', 'tasksRunning', 'backoffSeconds', 'backoffFactor']:
+                    if attr in app:
+                        self.gauge('marathon.' + attr, app[attr], tags=tags)
                 versions_reply = self.get_v2_app_versions(url, app['id'], timeout)
                 if versions_reply is not None:
                     self.gauge('marathon.versions', len(versions_reply['versions']), tags=tags)
@@ -38,7 +37,7 @@ def get_v2_apps(self, url, timeout):
         aggregation_key = md5(url).hexdigest()
         try:
             r = requests.get(url + "/v2/apps", timeout=timeout)
-        except requests.exceptions.Timeout as e:
+        except requests.exceptions.Timeout:
             # If there's a timeout
             self.timeout_event(url, timeout, aggregation_key)
             raise Exception("Timeout when hitting %s" % url)
@@ -56,10 +55,10 @@ def get_v2_apps(self, url, timeout):
     def get_v2_app_versions(self, url, app_id, timeout):
         # Use a hash of the URL as an aggregation key
         aggregation_key = md5(url).hexdigest()
-        
+
         try:
             r = requests.get(url + "/v2/apps/" + app_id + "/versions", timeout=timeout)
-        except requests.exceptions.Timeout as e:
+        except requests.exceptions.Timeout:
             # If there's a timeout
             self.timeout_event(url, timeout, aggregation_key)
             self.warning("Timeout when hitting %s" % url)


Test patch:
Repository: DataDog/dd-agent

Issue:
`dd-agent check` command does not work with network checks
To make it work it should:
1. Call a run so that it launches the thread pool
2. Join all the thread from the pool to make sure we got the result
3. Display the result and terminate the threads gracefully



Discussion:

Resolving patch:
diff --git a/agent.py b/agent.py
--- a/agent.py
+++ b/agent.py
@@ -283,6 +283,7 @@ def parent_func(): agent.start_event = False
                         check.run()
                         print check.get_metrics()
                         print check.get_events()
+                    check.stop()
 
     elif 'configcheck' == command or 'configtest' == command:
         osname = get_os()


Test patch:
Repository: DataDog/dd-agent

Issue:
system.mem.pct_usable is missing on Windows
We don't compute this very useful metric on Windows. We should to have more consistency with Unix.
Need to hack around there: 
https://github.com/DataDog/dd-agent/blob/master/checks/system/win32.py#L166



Discussion:
@whatarthurcodes Can you take a pass at it while you're working on windows ? Should be pretty straight forward.


Resolving patch:
diff --git a/checks/system/win32.py b/checks/system/win32.py
--- a/checks/system/win32.py
+++ b/checks/system/win32.py
@@ -57,10 +57,23 @@ def __init__(self, logger):
         self.gauge('system.mem.free')
         self.gauge('system.mem.used')
         self.gauge('system.mem.total')
+        # area of physical memory that stores recently used pages of data
+        # for applications
         self.gauge('system.mem.cached')
+        # Committed memory is physical memory for which space has been
+        # reserved on the disk paging file in case it must be written
+        # back to disk
         self.gauge('system.mem.committed')
+        # physical memory used by the operating system, for objects
+        # that can be written to disk when they are not being used
         self.gauge('system.mem.paged')
+        # physical memory used by the operating system for objects that
+        # cannot be written to disk, but must remain in physical memory
+        # as long as they are allocated.
         self.gauge('system.mem.nonpaged')
+        # usable = free + cached
+        self.gauge('system.mem.usable')
+        self.gauge('system.mem.pct_usable')
 
     def check(self, agentConfig):
         try:
@@ -69,6 +82,10 @@ def check(self, agentConfig):
             self.logger.info('Missing Win32_OperatingSystem. No memory metrics will be returned.')
             return
 
+        total = 0
+        free = 0
+        cached = 0
+
         if os.TotalVisibleMemorySize is not None and os.FreePhysicalMemory is not None:
             total = int(os.TotalVisibleMemorySize) / KB2MB
             free = int(os.FreePhysicalMemory) / KB2MB
@@ -76,9 +93,11 @@ def check(self, agentConfig):
             self.save_sample('system.mem.free', free)
             self.save_sample('system.mem.used', total - free)
 
+
         mem = w.Win32_PerfFormattedData_PerfOS_Memory()[0]
         if mem.CacheBytes is not None:
-            self.save_sample('system.mem.cached', int(mem.CacheBytes) / B2MB)
+            cached = int(mem.CacheBytes) / B2MB
+            self.save_sample('system.mem.cached', cached)
         if mem.CommittedBytes is not None:
             self.save_sample('system.mem.committed', int(mem.CommittedBytes) / B2MB)
         if mem.PoolPagedBytes is not None:
@@ -86,6 +105,12 @@ def check(self, agentConfig):
         if mem.PoolNonpagedBytes is not None:
             self.save_sample('system.mem.nonpaged', int(mem.PoolNonpagedBytes) / B2MB)
 
+        usable = free + cached
+        self.save_sample('system.mem.usable', usable)
+        if total > 0:
+            pct_usable = float(usable) / total
+            self.save_sample('system.mem.pct_usable', pct_usable)
+
         return self.get_metrics()
 
 class Cpu(Check):


Test patch:
Repository: DataDog/dd-agent

Issue:
[postgres] buffers_backend_fsync is not available on < 9.1
Add another version test to handle this case.



Discussion:

Resolving patch:
diff --git a/checks.d/postgres.py b/checks.d/postgres.py
--- a/checks.d/postgres.py
+++ b/checks.d/postgres.py
@@ -70,10 +70,13 @@ class PostgreSql(AgentCheck):
         'buffers_clean'        : ('postgresql.bgwriter.buffers_clean', MONOTONIC),
         'maxwritten_clean'     : ('postgresql.bgwriter.maxwritten_clean', MONOTONIC),
         'buffers_backend'      : ('postgresql.bgwriter.buffers_backend', MONOTONIC),
-        'buffers_backend_fsync': ('postgresql.bgwriter.buffers_backend_fsync', MONOTONIC),
         'buffers_alloc'        : ('postgresql.bgwriter.buffers_alloc', MONOTONIC),
     }
 
+    NEWER_91_BGW_METRICS = {
+        'buffers_backend_fsync': ('postgresql.bgwriter.buffers_backend_fsync', MONOTONIC),
+    }
+
     NEWER_92_BGW_METRICS = {
         'checkpoint_write_time': ('postgresql.bgwriter.write_time', MONOTONIC),
         'checkpoint_sync_time' : ('postgresql.bgwriter.sync_time', MONOTONIC),
@@ -217,6 +220,9 @@ def _is_above(self, key, db, version_to_compare):
 
         return False
 
+    def _is_9_1_or_above(self, key, db):
+        return self._is_above(key, db, [9,1,0])
+
     def _is_9_2_or_above(self, key, db):
         return self._is_above(key, db, [9,2,0])
 
@@ -243,10 +249,11 @@ def _get_bgw_metrics(self, key, db):
         # Extended 9.2+ metrics if needed
         metrics = self.bgw_metrics.get(key)
         if metrics is None:
+            self.bgw_metrics[key] = dict(self.COMMON_BGW_METRICS)
+            if self._is_9_1_or_above(key, db):
+                self.bgw_metrics[key].update(self.NEWER_91_BGW_METRICS)
             if self._is_9_2_or_above(key, db):
-                self.bgw_metrics[key] = dict(self.COMMON_BGW_METRICS, **self.NEWER_92_BGW_METRICS)
-            else:
-                self.bgw_metrics[key] = dict(self.COMMON_BGW_METRICS)
+                self.bgw_metrics[key].update(self.NEWER_92_BGW_METRICS)
             metrics = self.bgw_metrics.get(key)
         return metrics
 


Test patch:
Repository: DataDog/dd-agent

Issue:
[snmp] config option for non-increasing OID support
If the snmp library we use encounters non-increasing oids it fails.  This is to prevent from infinite loops during snmp walks.

http://pysnmp.sourceforge.net/faq.html#10

We could add a flag to the snmp config that would allow increasing oids for a given instance:

```
cls.cmd_generator = cmdgen.CommandGenerator()
cls.cmd_generator.ignoreNonIncreasingOid = True
```



Discussion:

Resolving patch:
diff --git a/checks.d/snmp.py b/checks.d/snmp.py
--- a/checks.d/snmp.py
+++ b/checks.d/snmp.py
@@ -3,6 +3,7 @@
 
 # project
 from checks import AgentCheck
+from config import _is_affirmative
 
 # 3rd party
 from pysnmp.entity.rfc3413.oneliner import cmdgen
@@ -41,18 +42,21 @@ def __init__(self, name, init_config, agentConfig, instances=None):
 
         # Load Custom MIB directory
         mibs_path = None
+        ignore_nonincreasing_oid = False
         if init_config is not None:
             mibs_path = init_config.get("mibs_folder")
-        SnmpCheck.create_command_generator(mibs_path)
+            ignore_nonincreasing_oid = _is_affirmative(init_config.get("ignore_nonincreasing_oid", False))
+        SnmpCheck.create_command_generator(mibs_path, ignore_nonincreasing_oid)
 
     @classmethod
-    def create_command_generator(cls, mibs_path=None):
+    def create_command_generator(cls, mibs_path, ignore_nonincreasing_oid):
         '''
         Create a command generator to perform all the snmp query.
         If mibs_path is not None, load the mibs present in the custom mibs
         folder. (Need to be in pysnmp format)
         '''
         cls.cmd_generator = cmdgen.CommandGenerator()
+        cls.cmd_generator.ignoreNonIncreasingOid = ignore_nonincreasing_oid
         if mibs_path is not None:
             mib_builder = cls.cmd_generator.snmpEngine.msgAndPduDsp.\
                           mibInstrumController.mibBuilder


Test patch:
diff --git a/tests/test_snmp.py b/tests/test_snmp.py
--- a/tests/test_snmp.py
+++ b/tests/test_snmp.py
@@ -34,6 +34,15 @@ def testInit(self):
                 custom_folder_represented = True
                 break
         self.assertTrue(custom_folder_represented)
+        self.assertFalse(self.check.cmd_generator.ignoreNonIncreasingOid)
+
+        self.config = {
+            "init_config": {
+                "ignore_nonincreasing_oid": True
+            }
+        }
+        self.check = load_check('snmp', self.config, self.agentConfig)
+        self.assertTrue(self.check.cmd_generator.ignoreNonIncreasingOid)
 
     def test_scalar_SNMPCheck(self):
         self.config = {
Repository: DataDog/dd-agent

Issue:
Invalid metric type GAUGE for postgresql.deadlocks
Looking at graphs from my postgresql servers, I can see that the graphs for postgresql.deadlocks shows the total number of deadlocks since startup. Wouldn't it make more sense to use RATE instead?

https://github.com/DataDog/dd-agent/blob/2bd69dc0ee50eb1d246ad0b958efb415d6ec5c7e/checks.d/postgres.py#L54



Discussion:
Agreed. This should be RATE instead.


Resolving patch:
diff --git a/checks.d/postgres.py b/checks.d/postgres.py
--- a/checks.d/postgres.py
+++ b/checks.d/postgres.py
@@ -51,7 +51,7 @@ class PostgreSql(AgentCheck):
     }
 
     NEWER_92_METRICS = {
-        'deadlocks'         : ('postgresql.deadlocks', GAUGE),
+        'deadlocks'         : ('postgresql.deadlocks', RATE),
         'temp_bytes'        : ('postgresql.temp_bytes', RATE),
         'temp_files'        : ('postgresql.temp_files', RATE),
     }


Test patch:
Repository: DataDog/dd-agent

Issue:
[config] autorestart flag expected not present on windows
```
Python could not construct the class instance 
Traceback (most recent call last):
  File "win32\agent.pyc", line 47, in __init__
  File "config.pyc", line 431, in get_config
AttributeError: Values instance has no attribute 'autorestart' 
%2: %3
```

This the current master pre-5.2 release 7a4294fe7b09df939b8c03a6ff47dedaf7a60ce4.
Introduced by f62d32900ebd2a33e3c7280345e71c22dcee13fe.



Discussion:

Resolving patch:
diff --git a/win32/agent.py b/win32/agent.py
--- a/win32/agent.py
+++ b/win32/agent.py
@@ -39,6 +39,7 @@ def __init__(self, args):
 
         # Setup the correct options so the agent will use the forwarder
         opts, args = Values({
+            'autorestart': False,
             'dd_url': None,
             'clean': False,
             'use_forwarder': True,


Test patch:
Repository: DataDog/dd-agent

Issue:
[marathon] 404 when the configured URL has a trailing slash
The web server powering Marathon apparently doesn't like double slashes, so dd-agent is getting a 404 back when trying to hit the /v2/apps endpoint.

e.g., with this configuration:

```
instances:
- url: "http://10.0.10.1:8080/"
```

The agent reports this error:

```
instance #0 [ERROR]: Exception('Got 404 when hitting http://10.0.10.1:8080/',)
```

This works fine if you leave off the trailing slash. 



Discussion:
Thanks for reporting this. Queuing it next, we should add least add something in the YAML comments or ensure that no double slashes are present in the check.


Resolving patch:
diff --git a/checks.d/marathon.py b/checks.d/marathon.py
--- a/checks.d/marathon.py
+++ b/checks.d/marathon.py
@@ -1,3 +1,6 @@
+# stdlib
+from urlparse import urljoin
+
 # project
 from checks import AgentCheck
 
@@ -31,7 +34,7 @@ def check(self, instance):
         default_timeout = self.init_config.get('default_timeout', self.DEFAULT_TIMEOUT)
         timeout = float(instance.get('timeout', default_timeout))
 
-        response = self.get_json(url + "/v2/apps", timeout)
+        response = self.get_json(urljoin(url, "/v2/apps"), timeout)
         if response is not None:
             self.gauge('marathon.apps', len(response['apps']), tags=instance_tags)
             for app in response['apps']:
@@ -39,7 +42,10 @@ def check(self, instance):
                 for attr in self.APP_METRICS:
                     if attr in app:
                         self.gauge('marathon.' + attr, app[attr], tags=tags)
-                versions_reply = self.get_json(url + "/v2/apps/" + app['id'] + "/versions", timeout)
+
+                query_url = urljoin(url, "/v2/apps/{0}/versions".format(app['id']))
+                versions_reply = self.get_json(query_url, timeout)
+
                 if versions_reply is not None:
                     self.gauge('marathon.versions', len(versions_reply['versions']), tags=tags)
 


Test patch:
Repository: DataDog/dd-agent

Issue:
No way to force SNMPv1
I have a device that only speaks SNMPv1, if you attempt SNMPv2 it simply ignores it.   pysnmp supports this by setting mpModel=0 with the community, but the agent does not have a way to set this.



Discussion:
Thanks a lot for the feedback @slushpupie ! 
We'll try to add this by 5.3.0 but would you like to open a pull request to implement this feature ?


Resolving patch:
diff --git a/checks.d/snmp.py b/checks.d/snmp.py
--- a/checks.d/snmp.py
+++ b/checks.d/snmp.py
@@ -83,7 +83,13 @@ def get_auth_data(cls, instance):
         '''
         if "community_string" in instance:
             # SNMP v1 - SNMP v2
-            return cmdgen.CommunityData(instance['community_string'])
+
+            # See http://pysnmp.sourceforge.net/docs/current/security-configuration.html
+            if int(instance.get("snmp_version", 2)) == 1:
+                return cmdgen.CommunityData(instance['community_string'],
+                    mpModel=0)
+            return cmdgen.CommunityData(instance['community_string'], mpModel=1)
+
         elif "user" in instance:
             # SNMP v3
             user = instance["user"]


Test patch:
Repository: DataDog/dd-agent

Issue:
System Proxy Settings return a string instead of an integer for port
When we pull system proxy settings from urllib we end up with this error:

```
Traceback (most recent call last):
  File "transaction.pyc", line 190, in flush_next
  File "ddagent.pyc", line 272, in flush
  File "tornado\httpclient.pyc", line 206, in fetch
  File "tornado\curl_httpclient.pyc", line 95, in fetch_impl
  File "tornado\curl_httpclient.pyc", line 233, in _process_queue
  File "tornado\curl_httpclient.pyc", line 358, in _curl_setup_request
TypeError: strings are not supported for this option
```

this is because the proxy port is being set to a string, which tornado doesn't accept:

https://github.com/DataDog/dd-agent/blob/master/config.py#L606-L608



Discussion:

Resolving patch:
diff --git a/config.py b/config.py
--- a/config.py
+++ b/config.py
@@ -605,7 +605,7 @@ def get_proxy(agentConfig, use_system_settings=False):
                 pass
             px = proxy.split(':')
             proxy_settings['host'] = px[0]
-            proxy_settings['port'] = px[1]
+            proxy_settings['port'] = int(px[1])
             proxy_settings['user'] = None
             proxy_settings['password'] = None
             proxy_settings['system_settings'] = True


Test patch: