Repository: jenkinsci/prometheus-plugin (java)

Issue:
Telegraf Unable to Parse Metrics - second TYPE line for metric name "jenkins_node_builds", or TYPE reported after samples
### Jenkins and plugins versions report

<details>
  <summary>Environment</summary>

  <!-- Paste your environment details below -->
  ```text
Jenkins: 2.319.3
OS: Linux - 3.10.0-1160.59.1.el7.x86_64
---
ace-editor:1.1
active-directory:2.25.1
ansicolor:1.0.0
ant:1.11
antisamy-markup-formatter:2.1
apache-httpcomponents-client-4-api:4.5.13-1.0
authentication-tokens:1.4
blueocean:1.24.8
blueocean-autofavorite:1.2.4
blueocean-bitbucket-pipeline:1.24.8
blueocean-commons:1.24.8
blueocean-config:1.24.8
blueocean-core-js:1.24.8
blueocean-dashboard:1.24.8
blueocean-display-url:2.4.1
blueocean-events:1.24.8
blueocean-git-pipeline:1.24.8
blueocean-github-pipeline:1.24.8
blueocean-i18n:1.24.8
blueocean-jira:1.24.8
blueocean-jwt:1.24.8
blueocean-personalization:1.24.8
blueocean-pipeline-api-impl:1.24.8
blueocean-pipeline-editor:1.24.8
blueocean-pipeline-scm-api:1.24.8
blueocean-rest:1.24.8
blueocean-rest-impl:1.24.8
blueocean-web:1.24.8
bootstrap4-api:4.6.0-3
bootstrap5-api:5.1.3-6
bouncycastle-api:2.22
branch-api:2.7.0
build-blocker-plugin:1.7.7
build-name-setter:2.1.0
build-timeout:1.20
build-user-vars-plugin:1.7
build-with-parameters:1.5.1
caffeine-api:2.9.2-29.v717aac953ff3
checks-api:1.7.2
cloudbees-bitbucket-branch-source:757.vddedc5f2589a_
cloudbees-disk-usage-simple:0.10
cloudbees-folder:6.708.ve61636eb_65a_5
command-launcher:1.6
conditional-buildstep:1.4.1
configuration-as-code:1414.v878271fc496f
credentials:1087.v16065d268466
credentials-binding:1.27
display-url-api:2.3.5
docker-commons:1.19
docker-workflow:1.26
durable-task:1.37
echarts-api:5.3.0-2
email-ext:2.83
extended-choice-parameter:0.82
extended-read-permission:3.2
extensible-choice-parameter:1.8.0
external-monitor-job:1.7
favorite:2.3.2
font-awesome-api:6.0.0-1
git:4.10.3
git-client:3.11.0
git-server:1.9
github:1.33.1
github-api:1.123
github-branch-source:2.9.9
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-1.0
hidden-parameter:0.0.4
htmlpublisher:1.25
jackson2-api:2.13.2-260.v43d711474c77
jacoco:3.3.1
javadoc:1.6
javax-activation-api:1.2.0-2
javax-mail-api:1.6.2-5
jdk-tool:1.5
jenkins-design-language:1.24.8
jira:3.3
jjwt-api:0.11.2-9.c8b45b8bb173
jquery:1.12.4-1
jquery-detached:1.2.1
jquery3-api:3.6.0-2
jsch:0.1.55.2
junit:1.54
ldap:2.0
lockable-resources:2.11
logstash:2.3.1-SNAPSHOT (private-261f22ce-root)
mailer:408.vd726a_1130320
matrix-auth:2.6.8
matrix-project:1.20
maven-plugin:3.8
mercurial:2.15
metrics:4.1.6.1
momentjs:1.1.1
nodelabelparameter:1.9.0
okhttp-api:3.14.9
pam-auth:1.6
parameterized-trigger:2.39
pipeline-build-step:2.15
pipeline-graph-analysis:188.v3a01e7973f2c
pipeline-input-step:446.vf27b_0b_83500e
pipeline-milestone-step:1.3.2
pipeline-model-api:1.9.1
pipeline-model-definition:1.9.1
pipeline-model-extensions:1.9.1
pipeline-rest-api:2.23
pipeline-stage-step:291.vf0a8a7aeeb50
pipeline-stage-tags-metadata:1.9.1
pipeline-stage-view:2.19
pipeline-utility-steps:2.8.0
plain-credentials:1.7
plugin-util-api:2.15.0
popper-api:1.16.1-2
popper2-api:2.11.4-1
prometheus:2.0.10
pubsub-light:1.13
rebuild:1.32
resource-disposer:0.16
role-strategy:3.2.0
run-condition:1.5
saml:1.1.7
scm-api:595.vd5a_df5eb_0e39
script-security:1138.v8e727069a_025
simple-theme-plugin:0.7
slack:2.48
snakeyaml-api:1.29.1
sse-gateway:1.24
ssh-credentials:1.19
ssh-slaves:1.31.5
sshd:3.0.3
structs:308.v852b473a2b8c
timestamper:1.13
token-macro:280.v97a_82642793c
trilead-api:1.0.13
validating-string-parameter:2.8
variant:1.4
view-job-filters:2.3
wavefront:1.1.1
windows-slaves:1.8
workflow-aggregator:2.6
workflow-api:1143.v2d42f1e9dea_5
workflow-basic-steps:2.24
workflow-cps:2.94
workflow-cps-global-lib:2.19
workflow-durable-task-step:2.39
workflow-job:1145.v7f2433caa07f
workflow-multibranch:2.24
workflow-scm-step:2.13
workflow-step-api:622.vb_8e7c15b_c95a_
workflow-support:813.vb_d7c3d2984a_0
ws-cleanup:0.39
  ```

</details>


### What Operating System are you using (both controller, and any agents involved in the problem)?

Controller & All Agents Running CentOS 7

### Reproduction steps

* Install Jenkins Prometheus Plugin 2.0.11
* Configure Plugin as indicated in the screenshot below 
![image](https://user-images.githubusercontent.com/33108028/163724736-70c85923-5666-4e34-8233-b3abf2bcf568.png)
* Configure Telegraf to access prometheus endpoint.

### Expected Results

Telegraf should be able to parse the Prometheus Export.

### Actual Results

`2022-04-17T16:49:40Z E! [inputs.prometheus] Error in plugin: error reading metrics for https://localhost:8443/prometheus/: reading text format failed: text format parsing error in line 1803: second TYPE line for metric name "jenkins_node_builds", or TYPE reported after samples`
This is what we see over and over again in our Telegraf logs.


### Anything else?

```
[[inputs.prometheus]]
  ## Prefix to attach to the measurement name
  name_prefix = "jenkins_"

  ## URL of each Jenkins server
  urls = ["https://localhost:8443/prometheus/"]

  ## Specify timeout duration for slower clients (default is 3s)
  # response_timeout = "3s"

  ## Use TLS but skip chain & host verification
  insecure_skip_verify = true
```
Here's what our Telegraf plugin config looks like.

Pull request:
Fix duplicate TYPE lines for jenkins_node_builds metric
Fixes #385 #375 #245 #250 #314

### Changes proposed

- Fixes duplicate TYPE entries for jenkins_node_builds by bundling all nodes under the same type. 
- See https://github.com/jenkinsci/prometheus-plugin/issues/385#issuecomment-1131888705

Example Output:
```
# HELP jenkins_node_builds Generated from Dropwizard metric import (metric=jenkins.node.builds, type=com.codahale.metrics.Timer)
# TYPE jenkins_node_builds summary
jenkins_node_builds{node="master",quantile="0.5",} 0.9963810470000001
jenkins_node_builds{node="master",quantile="0.75",} 0.9963810470000001
jenkins_node_builds{node="master",quantile="0.95",} 0.9963810470000001
jenkins_node_builds{node="master",quantile="0.98",} 0.9963810470000001
jenkins_node_builds{node="master",quantile="0.99",} 0.9963810470000001
jenkins_node_builds{node="master",quantile="0.999",} 0.9963810470000001
jenkins_node_builds_count{node="master",} 1.0
jenkins_node_builds{node="jknn2t-tjglue",quantile="0.5",} 0.0
jenkins_node_builds{node="jknn2t-tjglue",quantile="0.75",} 0.0
jenkins_node_builds{node="jknn2t-tjglue",quantile="0.95",} 0.0
jenkins_node_builds{node="jknn2t-tjglue",quantile="0.98",} 0.0
jenkins_node_builds{node="jknn2t-tjglue",quantile="0.99",} 0.0
jenkins_node_builds{node="jknn2t-tjglue",quantile="0.999",} 0.0
jenkins_node_builds_count{node="jknn2t-tjglue",} 0.0
jenkins_node_builds{node="jknn2t-vnzdet",quantile="0.5",} 33.632608852000004
jenkins_node_builds{node="jknn2t-vnzdet",quantile="0.75",} 33.632608852000004
jenkins_node_builds{node="jknn2t-vnzdet",quantile="0.95",} 33.632608852000004
jenkins_node_builds{node="jknn2t-vnzdet",quantile="0.98",} 33.632608852000004
jenkins_node_builds{node="jknn2t-vnzdet",quantile="0.99",} 33.632608852000004
jenkins_node_builds{node="jknn2t-vnzdet",quantile="0.999",} 33.632608852000004
jenkins_node_builds_count{node="jknn2t-vnzdet",} 1.0
```

### Checklist

- [x] Includes tests covering the new functionality?
- [x] Ready for review
- [x] Follows CONTRIBUTING rules

### Notify

@markyjackson-taulia


Patch:
diff --git a/src/main/java/org/jenkinsci/plugins/prometheus/service/DefaultPrometheusMetrics.java b/src/main/java/org/jenkinsci/plugins/prometheus/service/DefaultPrometheusMetrics.java
index e0b6f34..f4840b4 100644
--- a/src/main/java/org/jenkinsci/plugins/prometheus/service/DefaultPrometheusMetrics.java
+++ b/src/main/java/org/jenkinsci/plugins/prometheus/service/DefaultPrometheusMetrics.java
@@ -11,7 +11,7 @@ import org.jenkinsci.plugins.prometheus.DiskUsageCollector;
 import org.jenkinsci.plugins.prometheus.ExecutorCollector;
 import org.jenkinsci.plugins.prometheus.JenkinsStatusCollector;
 import org.jenkinsci.plugins.prometheus.JobCollector;
-import org.jenkinsci.plugins.prometheus.util.MetricsFormatter;
+import org.jenkinsci.plugins.prometheus.util.JenkinsNodeBuildsSampleBuilder;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -30,7 +30,7 @@ public class DefaultPrometheusMetrics implements PrometheusMetrics {
         CollectorRegistry collectorRegistry = CollectorRegistry.defaultRegistry;
         collectorRegistry.register(new JobCollector());
         collectorRegistry.register(new JenkinsStatusCollector());
-        collectorRegistry.register(new DropwizardExports(Metrics.metricRegistry()));
+        collectorRegistry.register(new DropwizardExports(Metrics.metricRegistry(), new JenkinsNodeBuildsSampleBuilder()));
         collectorRegistry.register(new DiskUsageCollector());
         collectorRegistry.register(new ExecutorCollector());
 
@@ -52,7 +52,7 @@ public class DefaultPrometheusMetrics implements PrometheusMetrics {
     public void collectMetrics() {
         try (StringWriter buffer = new StringWriter()) {
             TextFormat.write004(buffer, collectorRegistry.metricFamilySamples());
-            cachedMetrics.set(MetricsFormatter.formatMetrics(buffer.toString()));
+            cachedMetrics.set(buffer.toString());
         } catch (IOException e) {
             logger.debug("Unable to collect metrics");
         }
diff --git a/src/main/java/org/jenkinsci/plugins/prometheus/util/JenkinsNodeBuildsSampleBuilder.java b/src/main/java/org/jenkinsci/plugins/prometheus/util/JenkinsNodeBuildsSampleBuilder.java
new file mode 100644
index 0000000..43b905e
--- /dev/null
+++ b/src/main/java/org/jenkinsci/plugins/prometheus/util/JenkinsNodeBuildsSampleBuilder.java
@@ -0,0 +1,45 @@
+package org.jenkinsci.plugins.prometheus.util;
+
+import io.prometheus.client.Collector;
+import io.prometheus.client.dropwizard.samplebuilder.DefaultSampleBuilder;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * A class that converts jenkins.node[.<node_name>].builds to jenkins.node.builds with a label of node=<node_name> (or master if not set)
+ * before creating a Sample
+ */
+public class JenkinsNodeBuildsSampleBuilder extends DefaultSampleBuilder {
+    // Note that nodes can have '.' in their name
+    static Pattern PATTERN = Pattern.compile("jenkins\\.node(\\.(?<node>.*))?\\.builds");
+
+    @Override
+    public Collector.MetricFamilySamples.Sample createSample(String dropwizardName, String nameSuffix, List<String> additionalLabelNames, List<String> additionalLabelValues, double value) {
+        Matcher matcher = PATTERN.matcher(dropwizardName);
+
+        if (matcher.matches()) {
+            String processedDropwizardName = "jenkins.node.builds";
+            String node = matcher.group("node");
+
+            if (node == null) {
+                node = "master";
+            }
+
+            List<String> processedAdditionalLabelNames = new ArrayList<String>();
+            List<String> processedAdditionalLabelValues = new ArrayList<String>();
+
+            processedAdditionalLabelNames.add("node");
+            processedAdditionalLabelNames.addAll(additionalLabelNames);
+
+            processedAdditionalLabelValues.add(node);
+            processedAdditionalLabelValues.addAll(additionalLabelValues);
+
+            return super.createSample(processedDropwizardName, nameSuffix, processedAdditionalLabelNames, processedAdditionalLabelValues, value);
+        } else {
+            return super.createSample(dropwizardName, nameSuffix, additionalLabelNames, additionalLabelValues, value);
+        }
+    }
+}
diff --git a/src/main/java/org/jenkinsci/plugins/prometheus/util/MetricsFormatter.java b/src/main/java/org/jenkinsci/plugins/prometheus/util/MetricsFormatter.java
deleted file mode 100644
index 5ed0f12..0000000
--- a/src/main/java/org/jenkinsci/plugins/prometheus/util/MetricsFormatter.java
+++ /dev/null
@@ -1,16 +0,0 @@
-package org.jenkinsci.plugins.prometheus.util;
-
-public class MetricsFormatter {
-
-    public static String formatMetrics(String formatString) {
-        //node specific build counts
-        formatString = formatString.replaceAll("jenkins_node_builds_count (.*)", "jenkins_node_builds_count{node=\"master\"} $1");
-        formatString = formatString.replaceAll("jenkins_node_(.*)_builds_count (.*)", "jenkins_node_builds_count{node=\"$1\"} $2");
-
-        //node specific histograms
-        formatString = formatString.replaceAll("jenkins_node_builds\\{(.*)} (.*)", "jenkins_node_builds{node=\"master\", $1} $2");
-        formatString = formatString.replaceAll("jenkins_node_(.*)_builds\\{(.*)} (.*)", "jenkins_node_builds{node=\"$1\", $2} $3");
-
-        return formatString;
-    }
-}


Test patch:
diff --git a/src/test/java/org/jenkinsci/plugins/prometheus/util/JenkinsNodeBuildsSampleBuilderTest.java b/src/test/java/org/jenkinsci/plugins/prometheus/util/JenkinsNodeBuildsSampleBuilderTest.java
new file mode 100644
index 0000000..5301f21
--- /dev/null
+++ b/src/test/java/org/jenkinsci/plugins/prometheus/util/JenkinsNodeBuildsSampleBuilderTest.java
@@ -0,0 +1,85 @@
+package org.jenkinsci.plugins.prometheus.util;
+
+import io.prometheus.client.Collector;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.Arrays;
+
+public class JenkinsNodeBuildsSampleBuilderTest {
+    @Test
+    public void master_node_count_format() {
+        Assert.assertEquals(
+                new Collector.MetricFamilySamples.Sample(
+                        "jenkins_node_builds_count",
+                        Arrays.asList("node", "quantile"),
+                        Arrays.asList("master", "0.5"),
+                        0.091670452
+                ),
+                new JenkinsNodeBuildsSampleBuilder().createSample(
+                        "jenkins.node.builds",
+                        "_count",
+                        Arrays.asList("quantile"),
+                        Arrays.asList("0.5"),
+                        0.091670452
+                )
+        );
+    }
+
+    @Test
+    public void master_node_histogram_format() {
+        Assert.assertEquals(
+                new Collector.MetricFamilySamples.Sample(
+                        "jenkins_node_builds",
+                        Arrays.asList("node", "quantile"),
+                        Arrays.asList("master", "0.999"),
+                        0.091670452
+                ),
+                new JenkinsNodeBuildsSampleBuilder().createSample(
+                        "jenkins.node.builds",
+                        "",
+                        Arrays.asList("quantile"),
+                        Arrays.asList("0.999"),
+                        0.091670452
+                )
+        );
+    }
+
+    @Test
+    public void named_node_count_format() {
+        Assert.assertEquals(
+                new Collector.MetricFamilySamples.Sample(
+                        "jenkins_node_builds_count",
+                        Arrays.asList("node", "quantile"),
+                        Arrays.asList("evil node_name.com", "0.5"),
+                        0.091670452
+                ),
+                new JenkinsNodeBuildsSampleBuilder().createSample(
+                        "jenkins.node.evil node_name.com.builds",
+                        "_count",
+                        Arrays.asList("quantile"),
+                        Arrays.asList("0.5"),
+                        0.091670452
+                )
+        );
+    }
+
+    @Test
+    public void named_node_histogram_format() {
+        Assert.assertEquals(
+                new Collector.MetricFamilySamples.Sample(
+                        "jenkins_node_builds",
+                        Arrays.asList("node", "quantile"),
+                        Arrays.asList("evil node_name.com", "0.999"),
+                        0.091670452
+                ),
+                new JenkinsNodeBuildsSampleBuilder().createSample(
+                        "jenkins.node.evil node_name.com.builds",
+                        "",
+                        Arrays.asList("quantile"),
+                        Arrays.asList("0.999"),
+                        0.091670452
+                )
+        );
+    }
+}
diff --git a/src/test/java/org/jenkinsci/plugins/prometheus/util/MetricsFormatterTest.java b/src/test/java/org/jenkinsci/plugins/prometheus/util/MetricsFormatterTest.java
deleted file mode 100644
index 86f3d9a..0000000
--- a/src/test/java/org/jenkinsci/plugins/prometheus/util/MetricsFormatterTest.java
+++ /dev/null
@@ -1,50 +0,0 @@
-package org.jenkinsci.plugins.prometheus.util;
-
-import org.junit.Assert;
-import org.junit.Test;
-
-public class MetricsFormatterTest {
-
-    private final String inString = "jenkins_node_builds{quantile=\"0.5\",} 0.091670452\n" +
-            "jenkins_node_builds{quantile=\"0.75\",} 0.091670452\n" +
-            "jenkins_node_builds{quantile=\"0.95\",} 0.091670452\n" +
-            "jenkins_node_builds{quantile=\"0.98\",} 0.091670452\n" +
-            "jenkins_node_builds{quantile=\"0.99\",} 0.091670452\n" +
-            "jenkins_node_builds{quantile=\"0.999\",} 0.091670452\n" +
-            "jenkins_node_my_node_1_builds{quantile=\"0.99\",} 0.091670452\n" +
-            "jenkins_node_my_node_4_builds{quantile=\"0.999\",} 0.091670452\n" +
-            "jenkins_node_builds_count 28458.0\n" +
-            "jenkins_node_my_node_1_builds_count 12345";
-
-    @Test
-    public void master_node_count_format() {
-        String formatString = MetricsFormatter.formatMetrics(inString);
-        //master node count
-        Assert.assertFalse(formatString.contains("jenkins_node_builds_count 28458.0\n"));
-        Assert.assertTrue(formatString.contains("jenkins_node_builds_count{node=\"master\"} 28458.0\n"));
-    }
-
-    @Test
-    public void master_node_histogram_format() {
-        String formatString = MetricsFormatter.formatMetrics(inString);
-        //master node histogram
-        Assert.assertFalse(formatString.contains("jenkins_node_builds{quantile=\"0.999\",} 0.091670452\n"));
-        Assert.assertTrue(formatString.contains("jenkins_node_builds{node=\"master\", quantile=\"0.999\",} 0.091670452\n"));
-    }
-
-    @Test
-    public void named_node_count_format() {
-        String formatString = MetricsFormatter.formatMetrics(inString);
-        //named node count
-        Assert.assertFalse(formatString.contains("jenkins_node_my_node_1_builds_count"));
-        Assert.assertTrue(formatString.contains("jenkins_node_builds_count{node=\"my_node_1\"} 12345"));
-    }
-
-    @Test
-    public void named_node_histogram_format() {
-        String formatString = MetricsFormatter.formatMetrics(inString);
-        //named node histogram
-        Assert.assertFalse(formatString.contains("jenkins_node_my_node_4_builds{quantile=\"0.999\",} 0.091670452\n"));
-        Assert.assertTrue(formatString.contains("jenkins_node_builds{node=\"my_node_4\", quantile=\"0.999\",} 0.091670452\n"));
-    }
-}