internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: DataDog/dd-agent

Issue:
[dogstatsd] Service checks: message field parsing confused by `m:`
See https://github.com/DataDog/dd-agent/commit/1ca4a67163a4e563e558e4d7b2953453ee3ce152

The test fails because `m:` is regarded as the start of a message field when it shouldn't be (it's just part of the tag).

See also https://github.com/DataDog/dd-agent/blob/b09211d5410bc4ff874e09381908286914cf5271/aggregator.py#L557 for the line of code that's probably responsible



Discussion:

Resolving patch:
diff --git a/aggregator.py b/aggregator.py
--- a/aggregator.py
+++ b/aggregator.py
@@ -558,7 +558,7 @@ def parse_sc_packet(self, packet):
                 'status': int(status)
             }
 
-            message_delimiter = '|m:' if '|m:' in metadata else 'm:'
+            message_delimiter = 'm:' if metadata.startswith('m:') else '|m:'
             if message_delimiter in metadata:
                 meta, message = metadata.rsplit(message_delimiter, 1)
                 service_check['message'] = self._unescape_sc_content(message)


Test patch:
diff --git a/tests/core/test_dogstatsd.py b/tests/core/test_dogstatsd.py
--- a/tests/core/test_dogstatsd.py
+++ b/tests/core/test_dogstatsd.py
@@ -754,6 +754,27 @@ def test_service_check_tags(self):
         nt.assert_equal(fourth['check'], 'check.4')
         nt.assert_equal(fourth['tags'], sorted(['t1', 't2:v2', 't3', 't4']))
 
+    def test_service_check_tag_key_ends_with_m(self):
+        stats = MetricsAggregator('myhost')
+        stats.submit_packets('_sc|check.1|0|#keym:value')
+        stats.submit_packets('_sc|check.2|0|#key2m:value|m:fakeout')
+        stats.submit_packets('_sc|check.3|0|#key:valuem:value2,key2:value2|m:fakeoutm\:|h:#5')
+
+        service_checks = self.sort_service_checks(stats.flush_service_checks())
+
+        assert len(service_checks) == 3
+        first, second, third = service_checks
+
+        nt.assert_equal(first['check'], 'check.1')
+        nt.assert_equal(first['tags'], ['keym:value'])
+        nt.assert_not_in('message', first)
+        nt.assert_equal(second['check'], 'check.2')
+        nt.assert_equal(second['tags'], ['key2m:value'])
+        nt.assert_equal(second['message'], 'fakeout')
+        nt.assert_equal(third['check'], 'check.3')
+        nt.assert_equal(third['tags'], sorted(['key:valuem:value2', 'key2:value2']))
+        nt.assert_equal(third['message'], 'fakeoutm:|h:#5')
+
     def test_recent_point_threshold(self):
         threshold = 100
         # The min is not enabled by default
Repository: DataDog/dd-agent

Issue:
Include container id in docker_daemon metrics
Currently, the docker_daemon check [does not include the container id](https://github.com/DataDog/dd-agent/blob/master/checks.d/docker_daemon.py#L71-L82) as a tag in the metrics that are generated. Including this would be a great addition. Adding it would be pretty straightforward, but I'm not sure if there was any reasoning behind not including it in the first place.



Discussion:
Hi @ejholmes and thanks for opening this issue!

We recommend using a combination of the `image_name` and the `image_tag` tags to identify the containers.

Basically we want to avoid tagging the metrics by container ID as it would create a lot of different (and possibly short-lived) sets of tags for each metric, which would put some stress on our backend while (as far as we know) not adding much value in terms of usability.

Do you have a use-case where using the `image_name` and the `image_tag` tags does not work well?

@olivielpeau understandable. Basically, our use case is that `image_name` and `image_tag` aren't suitable for uniquely identifying a single container. We have other systems that report the container id, so this would mostly help us correlate with those systems.

We're using ECS, so in theory we're already creating a lot of short lived `container_name`'s (sorry! :)).

Thanks @ejholmes, we'll discuss this internally and see if/how we could add this tag. Will keep you updated!

@ejholmes I think you can just edit this line https://github.com/DataDog/dd-agent/blob/master/conf.d/docker_daemon.yaml.example#L105 and add "container_id" to it to collect the container_id as a tag. Let us know how it goes and feel free to reopen if it doesn't work!

@remh thanks for the heads up, but it [looks like there's no tag extractor for container_id](https://github.com/DataDog/dd-agent/blob/master/checks.d/docker_daemon.py#L90-L96). Would you guys accept a PR if `container_id` is disabled by default?


Resolving patch:
diff --git a/checks.d/docker_daemon.py b/checks.d/docker_daemon.py
--- a/checks.d/docker_daemon.py
+++ b/checks.d/docker_daemon.py
@@ -104,6 +104,7 @@
     "image_tag": lambda c: DockerUtil.image_tag_extractor(c, 1),
     "container_command": lambda c: [c["Command"]],
     "container_name": DockerUtil.container_name_extractor,
+    "container_id": lambda c: [c["Id"]],
 }
 
 CONTAINER = "container"


Test patch:
Repository: DataDog/dd-agent

Issue:
MongoDB agent fails on arbier
The mongodb agent fails to complete monitoring when monitored instance is an arbiter:
This is due to the fact that arbiter does not have an optimeDate field (https://github.com/DataDog/dd-agent/blob/master/checks.d/mongo.py:703):

> ```
>             if current is not None and primary is not None:
>                 lag = primary['optimeDate'] - current['optimeDate']
>                 data['replicationLag'] = total_seconds(lag)
> ```

Maybe need to cahgne to 

> ```
>             if current is not None and primary is not None and optimeData in current:
>                 lag = primary['optimeDate'] - current['optimeDate']
>                 data['replicationLag'] = total_seconds(lag)
>             else:
>                 data['replicationLag'] = 0
> ```



Discussion:
See error:

> ```
> mongo
> -----
>   - instance #0 [ERROR]: "global name 'optimeData' is not defined"
>   - Collected 0 metrics, 0 events & 2 service checks
> ```


Resolving patch:
diff --git a/checks.d/mongo.py b/checks.d/mongo.py
--- a/checks.d/mongo.py
+++ b/checks.d/mongo.py
@@ -701,8 +701,9 @@ def total_seconds(td):
 
                 # Compute a lag time
                 if current is not None and primary is not None:
-                    lag = primary['optimeDate'] - current['optimeDate']
-                    data['replicationLag'] = total_seconds(lag)
+                    if 'optimeDate' in primary and 'optimeDate' in current:
+                        lag = primary['optimeDate'] - current['optimeDate']
+                        data['replicationLag'] = total_seconds(lag)
 
                 if current is not None:
                     data['health'] = current['health']


Test patch:
Repository: DataDog/dd-agent

Issue:
Kubernetes doesn't perform kubelet checks correctly and alerts is always in NO DATA state
I am not sure if it depends on the Kubernetes version, but kubelet checks don't work on my setup because of a missing 'verbose' query parameter in the request, as a result the corresponding alert is always in the "NO DATA" state.

The datadog agent kubernetes plugin expects an output like this:

```
[+]ping ok
[+]syncloop ok
[+]pleg ok
healthz check passed
```

wheras by default the output is just this:

```
ok
```

After having a look at the kubernetes source code, it seems you have to add the verbose parameter to have this output:
https://github.com/ArtfulCoder/kubernetes/blob/c54097d96c2fa49caf2dd2bd82cd01966a70af21/pkg/healthz/healthz.go#L113



Discussion:

Resolving patch:
diff --git a/checks.d/kubernetes.py b/checks.d/kubernetes.py
--- a/checks.d/kubernetes.py
+++ b/checks.d/kubernetes.py
@@ -71,7 +71,7 @@ def _perform_kubelet_checks(self, url):
         service_check_base = NAMESPACE + '.kubelet.check'
         is_ok = True
         try:
-            r = requests.get(url)
+            r = requests.get(url, params={'verbose': True})
             for line in r.iter_lines():
 
                 # avoid noise; this check is expected to fail since we override the container hostname


Test patch:
Repository: DataDog/dd-agent

Issue:
Consul Integration issue warning when whitelist not specified and < 50 services
The Consul integration will indicate that there is an integration issue when a whitelist has not been specified.

But in our current scenario we have less than 50 services registering with consul, and would not like to have to maintain or engineer a solution to update that whitelist every time a dynamic service comes online or goes offline from consul.

We would like the consul integration to only report a warning when there are more than 50 services discovered and a whitelist has not been specified, because at that point the integration starts to truncate the service discovery list. 

Thanks,

Jackson
Alianza Inc.



Discussion:

Resolving patch:
diff --git a/checks.d/consul.py b/checks.d/consul.py
--- a/checks.d/consul.py
+++ b/checks.d/consul.py
@@ -204,7 +204,7 @@ def _cull_services_list(self, services, service_whitelist):
             services = [s for s in services if s in service_whitelist][:self.MAX_SERVICES]
         else:
             if len(services) <= self.MAX_SERVICES:
-                self.warning('Consul service whitelist not defined. Agent will poll for all %d services found' % len(services))
+                self.log.debug('Consul service whitelist not defined. Agent will poll for all %d services found', len(services))
             else:
                 self.warning('Consul service whitelist not defined. Agent will poll for at most %d services' % self.MAX_SERVICES)
                 services = list(islice(services.iterkeys(), 0, self.MAX_SERVICES))


Test patch:
Repository: DataDog/dd-agent

Issue:
Bug: info command prints OK for process checks that don’t find a process
This is unintuitive and unhelpful.


Discussion:

Resolving patch:
diff --git a/checks.d/process.py b/checks.d/process.py
--- a/checks.d/process.py
+++ b/checks.d/process.py
@@ -356,6 +356,9 @@ def check(self, instance):
         self.log.debug('ProcessCheck: process %s analysed', name)
         self.gauge('system.processes.number', len(pids), tags=tags)
 
+        if len(pids) == 0:
+            self.warning("No matching process was found")
+
         for attr, mname in ATTR_TO_METRIC.iteritems():
             vals = [x for x in proc_state[attr] if x is not None]
             # skip []


Test patch:
Repository: DataDog/dd-agent

Issue:
[kubernetes] Support custom label prefixes
Currently, the `extract_kube_labels` function in `kubeutil.py` hard codes a `kube_` prefix on all pod labels. It would be great if this prefix were configurable (including allowing it to be blank) to give more control over how these labels appear in the Datadog UI.

The use case is that we're also exporting stats from pods (using their labels as tags), and we want the tags to match up in our Datadog dashboards; currently, because of the `kube_` prefix, the stats exported for the Docker Daemon and other integrations don't match up with the ones exported from the app itself.


Discussion:
Hi @yolken 
Thanks for reaching out!
Agreed, it could be useful to make this prefix configurable. I'll add it to our backlog, but if you (or someone else looking at the issue) wants to give it a shot we'll happily review a PR :)
A tip for interested readers: the label prefix could be declared in kubernetes.yaml (and default to `kube_` for backward compatibility).

Resolving patch:
diff --git a/utils/kubernetes/kubeutil.py b/utils/kubernetes/kubeutil.py
--- a/utils/kubernetes/kubeutil.py
+++ b/utils/kubernetes/kubeutil.py
@@ -38,6 +38,7 @@ class KubeUtil:
     DEFAULT_HTTPS_KUBELET_PORT = 10250
     DEFAULT_MASTER_PORT = 8080
     DEFAULT_MASTER_NAME = 'kubernetes'  # DNS name to reach the master from a pod.
+    DEFAULT_LABEL_PREFIX = 'kube_'
     CA_CRT_PATH = '/var/run/secrets/kubernetes.io/serviceaccount/ca.crt'
     AUTH_TOKEN_PATH = '/var/run/secrets/kubernetes.io/serviceaccount/token'
 
@@ -80,6 +81,7 @@ def __init__(self, instance=None):
         self.kubelet_host = self.kubelet_api_url.split(':')[1].lstrip('/')
         self.pods_list_url = urljoin(self.kubelet_api_url, KubeUtil.PODS_LIST_PATH)
         self.kube_health_url = urljoin(self.kubelet_api_url, KubeUtil.KUBELET_HEALTH_PATH)
+        self.kube_label_prefix = instance.get('label_to_tag_prefix', KubeUtil.DEFAULT_LABEL_PREFIX)
 
         # cadvisor
         self.cadvisor_port = instance.get('port', KubeUtil.DEFAULT_CADVISOR_PORT)
@@ -194,6 +196,7 @@ def extract_kube_labels(self, pods_list, excluded_keys=None):
         excluded_keys = excluded_keys or []
         kube_labels = defaultdict(list)
         pod_items = pods_list.get("items") or []
+        prefix = self.kube_label_prefix
         for pod in pod_items:
             metadata = pod.get("metadata", {})
             name = metadata.get("name")
@@ -206,7 +209,7 @@ def extract_kube_labels(self, pods_list, excluded_keys=None):
                     if k in excluded_keys:
                         continue
 
-                    kube_labels[key].append(u"kube_%s:%s" % (k, v))
+                    kube_labels[key].append(u"%s%s:%s" % (prefix, k, v))
 
         return kube_labels
 


Test patch:
Repository: DataDog/dd-agent

Issue:
Support TLS client certificates for prometheus check
Hi,

I've created https://github.com/latency-at/datadog-agent-prometheus to scape Prometheus metrics with datadog by using the PrometheusCheck: https://github.com/latency-at/datadog-agent-prometheus/blob/master/check.d/prometheus.py#L32

Now I want to scrape endpoint that requires TLS client authentication (a secure etcd-member) but that doesn't seem to be possible with the PrometheusCheck.


Discussion:
Hi @discordianfish thanks for reaching out. We heard good things from your project here!
@mfpierre will TLS client cert be supported in the new prom check?

Resolving patch:
diff --git a/checks/prometheus_check.py b/checks/prometheus_check.py
--- a/checks/prometheus_check.py
+++ b/checks/prometheus_check.py
@@ -116,6 +116,19 @@ def __init__(self, name, init_config, agentConfig, instances=None):
         # a label can hold this information, this transfer it to the hostname
         self.label_to_hostname = None
 
+        # Can either be only the path to the certificate and thus you should specify the private key
+        # or it can be the path to a file containing both the certificate & the private key
+        self.ssl_cert = None
+
+        # Needed if the certificate does not include the private key
+        #
+        # /!\ The private key to your local certificate must be unencrypted.
+        # Currently, Requests does not support using encrypted keys.
+        self.ssl_private_key = None
+
+        # The path to the trusted CA used for generating custom certificates
+        self.ssl_ca_cert = None
+
     def check(self, instance):
         """
         check should take care of getting the url and other params
@@ -400,8 +413,19 @@ def poll(self, endpoint, pFormat=PrometheusFormat.PROTOBUF, headers=None):
             headers['accept'] = 'application/vnd.google.protobuf; ' \
                                 'proto=io.prometheus.client.MetricFamily; ' \
                                 'encoding=delimited'
-
-        response = requests.get(endpoint, headers=headers, stream=True)
+        cert = None
+        if isinstance(self.ssl_cert, basestring):
+            cert = self.ssl_cert
+            if isinstance(self.ssl_private_key, basestring):
+                cert = (self.ssl_cert, self.ssl_private_key)
+        verify = True
+        if isinstance(self.ssl_ca_cert, basestring):
+            verify = self.ssl_ca_cert
+        try:
+            response = requests.get(endpoint, headers=headers, stream=True, cert=cert, verify=verify)
+        except (IOError, requests.exceptions.SSLError):
+            self.log.error("Invalid SSL settings for requesting {} endpoint".format(endpoint))
+            raise
         try:
             response.raise_for_status()
             return response


Test patch:
Repository: MycroftAI/mycroft-core

Issue:
custom skills
Currently there is no architecture to create a 3rd party skill
- User dir `~/.mycroft/third_party_skills`
- `mycroft/third_party_skills`



Discussion:

Resolving patch:
diff --git a/mycroft/configuration/__init__.py b/mycroft/configuration/__init__.py
--- a/mycroft/configuration/__init__.py
+++ b/mycroft/configuration/__init__.py
@@ -31,7 +31,9 @@
 
 DEFAULT_CONFIG = join(dirname(__file__), 'mycroft.ini')
 SYSTEM_CONFIG = '/etc/mycroft/mycroft.ini'
-USER_CONFIG = join(expanduser('~'), '.mycroft/mycroft.ini')
+USER_DIR = join(expanduser('~'), '.mycroft')
+USER_CONFIG = join(USER_DIR, 'mycroft.ini')
+USER_THIRD_PARTY_SKILLS_DIR = join(USER_DIR, "third_party_skills")
 
 
 class ConfigurationLoader(object):
diff --git a/mycroft/skills/core.py b/mycroft/skills/core.py
--- a/mycroft/skills/core.py
+++ b/mycroft/skills/core.py
@@ -23,7 +23,7 @@
 import os.path
 import re
 from adapt.intent import Intent
-from os.path import join, dirname, splitext, isdir
+from os.path import join, dirname, splitext, isdir, realpath
 
 from mycroft.client.enclosure.api import EnclosureAPI
 from mycroft.configuration import ConfigurationManager
@@ -37,6 +37,8 @@
 PRIMARY_SKILLS = ['intent', 'wake']
 BLACKLISTED_SKILLS = ["send_sms"]
 SKILLS_BASEDIR = dirname(__file__)
+THIRD_PARTY_SKILLS_DIR = realpath(
+    join(SKILLS_BASEDIR, "../third_party_skills"))
 
 MainModule = '__init__'
 
diff --git a/mycroft/skills/main.py b/mycroft/skills/main.py
--- a/mycroft/skills/main.py
+++ b/mycroft/skills/main.py
@@ -17,9 +17,11 @@
 
 
 import json
+from os.path import exists
 
+from mycroft.configuration import USER_THIRD_PARTY_SKILLS_DIR
 from mycroft.messagebus.client.ws import WebsocketClient
-from mycroft.skills.core import load_skills
+from mycroft.skills.core import load_skills, THIRD_PARTY_SKILLS_DIR
 from mycroft.util.log import getLogger
 logger = getLogger("Skills")
 
@@ -31,6 +33,11 @@
 def load_skills_callback():
     global client
     load_skills(client)
+    if exists(THIRD_PARTY_SKILLS_DIR):
+        load_skills(client, skills_root=THIRD_PARTY_SKILLS_DIR)
+
+    if exists(USER_THIRD_PARTY_SKILLS_DIR):
+        load_skills(client, skills_root=USER_THIRD_PARTY_SKILLS_DIR)
 
 
 def connect():


Test patch:
Repository: MycroftAI/mycroft-core

Issue:
custom skills
Currently there is no architecture to create a 3rd party skill
- User dir `~/.mycroft/third_party_skills`
- `mycroft/third_party_skills`



Discussion:
Would this replace the [skill-sdk](https://docs.mycroft.ai/development/skills-sdk) approach or be in addition to it?

No, it's intended to be used by 3rd party developers of skills.  It's for people like me who want to create custom skills and release them, but know that there's just no way it's going to be adapted to mycroft-core.


Resolving patch:
diff --git a/mycroft/skills/core.py b/mycroft/skills/core.py
--- a/mycroft/skills/core.py
+++ b/mycroft/skills/core.py
@@ -37,6 +37,7 @@
 PRIMARY_SKILLS = ['intent', 'wake']
 BLACKLISTED_SKILLS = ["send_sms"]
 SKILLS_BASEDIR = dirname(__file__)
+THIRD_PARTY_SKILLS_DIR = "/opt/mycroft/third_party"
 
 MainModule = '__init__'
 
diff --git a/mycroft/skills/main.py b/mycroft/skills/main.py
--- a/mycroft/skills/main.py
+++ b/mycroft/skills/main.py
@@ -17,9 +17,11 @@
 
 
 import json
+from os.path import join, expanduser, exists
 
+from mycroft.configuration import ConfigurationManager
 from mycroft.messagebus.client.ws import WebsocketClient
-from mycroft.skills.core import load_skills
+from mycroft.skills.core import load_skills, THIRD_PARTY_SKILLS_DIR
 from mycroft.util.log import getLogger
 logger = getLogger("Skills")
 
@@ -31,6 +33,20 @@
 def load_skills_callback():
     global client
     load_skills(client)
+    config = ConfigurationManager.get()
+    config_core = config.get("core")
+
+    try:
+        ini_third_party_skills_dir = expanduser(
+            config_core.get("third_party_skills_dir"))
+    except AttributeError as e:
+        logger.warning(e.message)
+
+    if exists(THIRD_PARTY_SKILLS_DIR):
+        load_skills(client, THIRD_PARTY_SKILLS_DIR)
+
+    if ini_third_party_skills_dir and exists(ini_third_party_skills_dir):
+        load_skills(client, ini_third_party_skills_dir)
 
 
 def connect():


Test patch:
Repository: MycroftAI/mycroft-core

Issue:
Too many false positives when detecting wake word



Discussion:

Resolving patch:
diff --git a/mycroft/client/speech/local_recognizer.py b/mycroft/client/speech/local_recognizer.py
--- a/mycroft/client/speech/local_recognizer.py
+++ b/mycroft/client/speech/local_recognizer.py
@@ -27,7 +27,8 @@
 
 
 class LocalRecognizer(object):
-    def __init__(self, sample_rate=16000, lang="en-us", key_phrase="mycroft"):
+    def __init__(self, sample_rate=16000, lang="en-us",
+                 key_phrase="hey mycroft"):
         self.lang = lang
         self.key_phrase = key_phrase
         self.sample_rate = sample_rate
@@ -40,7 +41,7 @@ def configure(self):
         config.set_string('-dict', os.path.join(BASEDIR, 'model', self.lang,
                                                 'mycroft-en-us.dict'))
         config.set_string('-keyphrase', self.key_phrase)
-        config.set_float('-kws_threshold', float('1e-45'))
+        config.set_float('-kws_threshold', float('1e-90'))
         config.set_float('-samprate', self.sample_rate)
         config.set_int('-nfft', 2048)
         config.set_string('-logfn', '/dev/null')


Test patch:
Repository: MycroftAI/mycroft-core

Issue:
Mycroft should only repeat the query in wolfram when short to speak
When the estimated time to speak "I am searching for blah blah blah blah" will take longer than the estimated time to get the result from wolfram, only the result should be said rather than the redundant "I am searching for ...".



Discussion:

Resolving patch:
diff --git a/mycroft/skills/wolfram_alpha/__init__.py b/mycroft/skills/wolfram_alpha/__init__.py
--- a/mycroft/skills/wolfram_alpha/__init__.py
+++ b/mycroft/skills/wolfram_alpha/__init__.py
@@ -138,21 +138,22 @@ def get_result(self, res):
                 return result
 
     def handle_fallback(self, message):
+        self.enclosure.mouth_think()
         logger.debug(
             "Could not determine intent, falling back to WolframAlpha Skill!")
         utterance = message.metadata.get('utterance')
         parsed_question = self.question_parser.parse(utterance)
 
-        # biding some time
-        if parsed_question:
-            self.speak("I am searching for " + parsed_question.get('Query'))
-        else:
-            self.speak("I am searching for " + utterance)
         query = utterance
         if parsed_question:
-            query = "%s %s %s" % (parsed_question.get('QuestionWord'),
-                                  parsed_question.get('QuestionVerb'),
-                                  parsed_question.get('Query'))
+            # Try to store pieces of utterance (None if not parsed_question)
+            utt_word = parsed_question.get('QuestionWord')
+            utt_verb = parsed_question.get('QuestionVerb')
+            utt_query = parsed_question.get('Query')
+            query = "%s %s %s" % (utt_word, utt_verb, utt_query)
+            phrase = "know %s %s %s" % (utt_word, utt_query, utt_verb)
+        else:  # TODO: Localization
+            phrase = "understand the phrase " + utterance
 
         try:
             res = self.client.query(query)
@@ -163,7 +164,7 @@ def handle_fallback(self, message):
             return
         except Exception as e:
             logger.exception(e)
-            self.speak_dialog("not.understood")
+            self.speak_dialog("not.understood", data={'phrase': phrase})
             return
 
         if result:
@@ -194,7 +195,7 @@ def handle_fallback(self, message):
                                              metadata={'utterance':
                                                        others[0]}))
             else:
-                self.speak_dialog("not.understood")
+                self.speak_dialog("not.understood", data={'phrase': phrase})
 
     @staticmethod
     def __find_pod_id(pods, pod_id):


Test patch: