Repository: open-telemetry/opentelemetry-python

Issue:
Add Prometheus exporter into main
### Discussed in https://github.com/open-telemetry/opentelemetry-python/discussions/2290

<div type='discussions-op-text'>

<sup>Originally posted by **adriangonz** November 24, 2021</sup>
Hey everyone,

I've been trying to find the source code for the OpenTelemetry Prometheus exporter (i.e. the `opentelemetry-prometheus-exporter` package), however I've been unlucky so far. I was expecting to find it on the `./exporter` folder (https://github.com/open-telemetry/opentelemetry-python/tree/main/exporter), but it doesn't seem to be there any more.

Trying to track its Git history, I can see that the packaged was moved to the `opentelemetry-python-contrib` repo a while back (https://github.com/open-telemetry/opentelemetry-python-contrib/pull/81). However, this move was then reverted shortly after (https://github.com/open-telemetry/opentelemetry-python-contrib/pull/88). After this though, I'm not able to see where the package went any more.

It would be great if someone could provide some pointers on where can we find the source code for the Prometheus exporter!</div>


Discussion:
@rltosd the previous Prometheus exporter code was using the previous Metrics implementation found in the metrics branch. After several spec changes, we are now working in reimplementing metrics, the current work is now in the main branch in a still non-public directory.
We do want a Prometheus exporter, it would be great to use the previous AWS Prometheus implementation, but we first need to complete the metrics SDK prototype (work is being tracked in the metrics-labeled PRs`) and then we would need to refactor the AWS Prometheus exporter to match the new implementation.
Hope this answers your question! :v:

Resolving patch:
diff --git a/exporter/opentelemetry-exporter-prometheus/setup.py b/exporter/opentelemetry-exporter-prometheus/setup.py
new file mode 100644
--- /dev/null
+++ b/exporter/opentelemetry-exporter-prometheus/setup.py
@@ -0,0 +1,26 @@
+# Copyright The OpenTelemetry Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import os
+
+import setuptools
+
+_BASE_DIR = os.path.dirname(__file__)
+_VERSION_FILENAME = os.path.join(
+    _BASE_DIR, "src", "opentelemetry", "exporter", "prometheus", "version.py"
+)
+_PACKAGE_INFO = {}
+with open(_VERSION_FILENAME, encoding="utf-8") as f:
+    exec(f.read(), _PACKAGE_INFO)
+
+setuptools.setup(version=_PACKAGE_INFO["__version__"])
diff --git a/exporter/opentelemetry-exporter-prometheus/src/opentelemetry/exporter/prometheus/__init__.py b/exporter/opentelemetry-exporter-prometheus/src/opentelemetry/exporter/prometheus/__init__.py
new file mode 100644
--- /dev/null
+++ b/exporter/opentelemetry-exporter-prometheus/src/opentelemetry/exporter/prometheus/__init__.py
@@ -0,0 +1,208 @@
+# Copyright The OpenTelemetry Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+This library allows export of metrics data to `Prometheus <https://prometheus.io/>`_.
+
+Usage
+-----
+
+The **OpenTelemetry Prometheus Exporter** allows export of `OpenTelemetry`_
+metrics to `Prometheus`_.
+
+
+.. _Prometheus: https://prometheus.io/
+.. _OpenTelemetry: https://github.com/open-telemetry/opentelemetry-python/
+
+.. code:: python
+
+    from prometheus_client import start_http_server
+
+    from opentelemetry._metrics import get_meter_provider, set_meter_provider
+    from opentelemetry.exporter.prometheus import PrometheusMetricReader
+    from opentelemetry.sdk._metrics import MeterProvider
+
+    # Start Prometheus client
+    start_http_server(port=8000, addr="localhost")
+
+    # Exporter to export metrics to Prometheus
+    prefix = "MyAppPrefix"
+    reader = PrometheusMetricReader(prefix)
+
+    # Meter is responsible for creating and recording metrics
+    set_meter_provider(MeterProvider(metric_readers=[reader]))
+    meter = get_meter_provider().get_meter("myapp", "0.1.2")
+
+    counter = meter.create_counter(
+        "requests",
+        "requests",
+        "number of requests",
+    )
+
+    # Labels are used to identify key-values that are associated with a specific
+    # metric that you want to record. These are useful for pre-aggregation and can
+    # be used to store custom dimensions pertaining to a metric
+    labels = {"environment": "staging"}
+
+    counter.add(25, labels)
+    input("Press any key to exit...")
+
+API
+---
+"""
+
+import collections
+import logging
+import re
+from typing import Iterable, Optional, Sequence, Tuple
+
+from prometheus_client import core
+
+from opentelemetry.sdk._metrics.export import MetricReader
+from opentelemetry.sdk._metrics.point import Gauge, Histogram, Metric, Sum
+
+_logger = logging.getLogger(__name__)
+
+
+def _convert_buckets(metric: Metric) -> Sequence[Tuple[str, int]]:
+    buckets = []
+    total_count = 0
+    for index, value in enumerate(metric.point.bucket_counts):
+        total_count += value
+        buckets.append(
+            (
+                f"{metric.point.explicit_bounds[index]}",
+                total_count,
+            )
+        )
+    return buckets
+
+
+class PrometheusMetricReader(MetricReader):
+    """Prometheus metric exporter for OpenTelemetry.
+
+    Args:
+        prefix: single-word application prefix relevant to the domain
+            the metric belongs to.
+    """
+
+    def __init__(self, prefix: str = "") -> None:
+        super().__init__()
+        self._collector = _CustomCollector(prefix)
+        core.REGISTRY.register(self._collector)
+        self._collector._callback = self.collect
+
+    def _receive_metrics(self, metrics: Iterable[Metric]) -> None:
+        if metrics is None:
+            return
+        self._collector.add_metrics_data(metrics)
+
+    def shutdown(self) -> bool:
+        core.REGISTRY.unregister(self._collector)
+        return True
+
+
+class _CustomCollector:
+    """_CustomCollector represents the Prometheus Collector object
+
+    See more:
+    https://github.com/prometheus/client_python#custom-collectors
+    """
+
+    def __init__(self, prefix: str = ""):
+        self._prefix = prefix
+        self._callback = None
+        self._metrics_to_export = collections.deque()
+        self._non_letters_digits_underscore_re = re.compile(
+            r"[^\w]", re.UNICODE | re.IGNORECASE
+        )
+
+    def add_metrics_data(self, export_records: Sequence[Metric]) -> None:
+        """Add metrics to Prometheus data"""
+        self._metrics_to_export.append(export_records)
+
+    def collect(self) -> None:
+        """Collect fetches the metrics from OpenTelemetry
+        and delivers them as Prometheus Metrics.
+        Collect is invoked every time a ``prometheus.Gatherer`` is run
+        for example when the HTTP endpoint is invoked by Prometheus.
+        """
+        if self._callback is not None:
+            self._callback()
+
+        while self._metrics_to_export:
+            for export_record in self._metrics_to_export.popleft():
+                prometheus_metric = self._translate_to_prometheus(
+                    export_record
+                )
+                if prometheus_metric is not None:
+                    yield prometheus_metric
+
+    def _translate_to_prometheus(
+        self, metric: Metric
+    ) -> Optional[core.Metric]:
+        prometheus_metric = None
+        label_values = []
+        label_keys = []
+        for key, value in metric.attributes.items():
+            label_keys.append(self._sanitize(key))
+            label_values.append(str(value))
+
+        metric_name = ""
+        if self._prefix != "":
+            metric_name = self._prefix + "_"
+        metric_name += self._sanitize(metric.name)
+
+        description = metric.description or ""
+        if isinstance(metric.point, Sum):
+            prometheus_metric = core.CounterMetricFamily(
+                name=metric_name,
+                documentation=description,
+                labels=label_keys,
+                unit=metric.unit,
+            )
+            prometheus_metric.add_metric(
+                labels=label_values, value=metric.point.value
+            )
+        elif isinstance(metric.point, Gauge):
+            prometheus_metric = core.GaugeMetricFamily(
+                name=metric_name,
+                documentation=description,
+                labels=label_keys,
+                unit=metric.unit,
+            )
+            prometheus_metric.add_metric(
+                labels=label_values, value=metric.point.value
+            )
+        elif isinstance(metric.point, Histogram):
+            value = metric.point.sum
+            prometheus_metric = core.HistogramMetricFamily(
+                name=metric_name,
+                documentation=description,
+                labels=label_keys,
+                unit=metric.unit,
+            )
+            buckets = _convert_buckets(metric)
+            prometheus_metric.add_metric(
+                labels=label_values, buckets=buckets, sum_value=value
+            )
+        else:
+            _logger.warning("Unsupported metric type. %s", type(metric.point))
+        return prometheus_metric
+
+    def _sanitize(self, key: str) -> str:
+        """sanitize the given metric name or label according to Prometheus rule.
+        Replace all characters other than [A-Za-z0-9_] with '_'.
+        """
+        return self._non_letters_digits_underscore_re.sub("_", key)
diff --git a/exporter/opentelemetry-exporter-prometheus/src/opentelemetry/exporter/prometheus/version.py b/exporter/opentelemetry-exporter-prometheus/src/opentelemetry/exporter/prometheus/version.py
new file mode 100644
--- /dev/null
+++ b/exporter/opentelemetry-exporter-prometheus/src/opentelemetry/exporter/prometheus/version.py
@@ -0,0 +1,15 @@
+# Copyright The OpenTelemetry Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+__version__ = "0.28b1"


Test patch:
diff --git a/exporter/opentelemetry-exporter-prometheus/tests/__init__.py b/exporter/opentelemetry-exporter-prometheus/tests/__init__.py
new file mode 100644
--- /dev/null
+++ b/exporter/opentelemetry-exporter-prometheus/tests/__init__.py
@@ -0,0 +1,13 @@
+# Copyright The OpenTelemetry Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
diff --git a/exporter/opentelemetry-exporter-prometheus/tests/test_prometheus_exporter.py b/exporter/opentelemetry-exporter-prometheus/tests/test_prometheus_exporter.py
new file mode 100644
--- /dev/null
+++ b/exporter/opentelemetry-exporter-prometheus/tests/test_prometheus_exporter.py
@@ -0,0 +1,154 @@
+# Copyright The OpenTelemetry Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import unittest
+from unittest import mock
+
+from prometheus_client import generate_latest
+from prometheus_client.core import CounterMetricFamily, GaugeMetricFamily
+
+from opentelemetry.exporter.prometheus import (
+    PrometheusMetricReader,
+    _CustomCollector,
+)
+from opentelemetry.sdk._metrics.point import AggregationTemporality, Histogram
+from opentelemetry.test.metrictestutil import (
+    _generate_gauge,
+    _generate_metric,
+    _generate_sum,
+    _generate_unsupported_metric,
+)
+
+
+class TestPrometheusMetricReader(unittest.TestCase):
+    def setUp(self):
+        self._mock_registry_register = mock.Mock()
+        self._registry_register_patch = mock.patch(
+            "prometheus_client.core.REGISTRY.register",
+            side_effect=self._mock_registry_register,
+        )
+
+    # pylint: disable=protected-access
+    def test_constructor(self):
+        """Test the constructor."""
+        with self._registry_register_patch:
+            exporter = PrometheusMetricReader("testprefix")
+            self.assertEqual(exporter._collector._prefix, "testprefix")
+            self.assertTrue(self._mock_registry_register.called)
+
+    def test_shutdown(self):
+        with mock.patch(
+            "prometheus_client.core.REGISTRY.unregister"
+        ) as registry_unregister_patch:
+            exporter = PrometheusMetricReader()
+            exporter.shutdown()
+            self.assertTrue(registry_unregister_patch.called)
+
+    def test_histogram_to_prometheus(self):
+        record = _generate_metric(
+            "test@name",
+            Histogram(
+                time_unix_nano=1641946016139533244,
+                start_time_unix_nano=1641946016139533244,
+                bucket_counts=[1, 1],
+                sum=579.0,
+                explicit_bounds=[123.0, 456.0],
+                aggregation_temporality=AggregationTemporality.DELTA,
+            ),
+            attributes={"histo": 1},
+        )
+
+        collector = _CustomCollector("testprefix")
+        collector.add_metrics_data([record])
+        result_bytes = generate_latest(collector)
+        result = result_bytes.decode("utf-8")
+        self.assertIn('testprefix_test_name_s_sum{histo="1"} 579.0', result)
+        self.assertIn('testprefix_test_name_s_count{histo="1"} 2.0', result)
+
+    def test_sum_to_prometheus(self):
+        labels = {"environment@": "staging", "os": "Windows"}
+        record = _generate_sum(
+            "test@sum",
+            123,
+            attributes=labels,
+            description="testdesc",
+            unit="testunit",
+        )
+        collector = _CustomCollector("testprefix")
+        collector.add_metrics_data([record])
+
+        for prometheus_metric in collector.collect():
+            self.assertEqual(type(prometheus_metric), CounterMetricFamily)
+            self.assertEqual(
+                prometheus_metric.name, "testprefix_test_sum_testunit"
+            )
+            self.assertEqual(prometheus_metric.documentation, "testdesc")
+            self.assertTrue(len(prometheus_metric.samples) == 1)
+            self.assertEqual(prometheus_metric.samples[0].value, 123)
+            self.assertTrue(len(prometheus_metric.samples[0].labels) == 2)
+            self.assertEqual(
+                prometheus_metric.samples[0].labels["environment_"], "staging"
+            )
+            self.assertEqual(
+                prometheus_metric.samples[0].labels["os"], "Windows"
+            )
+
+    def test_gauge_to_prometheus(self):
+        labels = {"environment@": "dev", "os": "Unix"}
+        record = _generate_gauge(
+            "test@gauge",
+            123,
+            attributes=labels,
+            description="testdesc",
+            unit="testunit",
+        )
+        collector = _CustomCollector("testprefix")
+        collector.add_metrics_data([record])
+
+        for prometheus_metric in collector.collect():
+            self.assertEqual(type(prometheus_metric), GaugeMetricFamily)
+            self.assertEqual(
+                prometheus_metric.name, "testprefix_test_gauge_testunit"
+            )
+            self.assertEqual(prometheus_metric.documentation, "testdesc")
+            self.assertTrue(len(prometheus_metric.samples) == 1)
+            self.assertEqual(prometheus_metric.samples[0].value, 123)
+            self.assertTrue(len(prometheus_metric.samples[0].labels) == 2)
+            self.assertEqual(
+                prometheus_metric.samples[0].labels["environment_"], "dev"
+            )
+            self.assertEqual(prometheus_metric.samples[0].labels["os"], "Unix")
+
+    def test_invalid_metric(self):
+        labels = {"environment": "staging"}
+        record = _generate_unsupported_metric(
+            "tesname",
+            attributes=labels,
+            description="testdesc",
+            unit="testunit",
+        )
+        collector = _CustomCollector("testprefix")
+        collector.add_metrics_data([record])
+        collector.collect()
+        self.assertLogs("opentelemetry.exporter.prometheus", level="WARNING")
+
+    def test_sanitize(self):
+        collector = _CustomCollector("testprefix")
+        self.assertEqual(
+            collector._sanitize("1!2@3#4$5%6^7&8*9(0)_-"),
+            "1_2_3_4_5_6_7_8_9_0___",
+        )
+        self.assertEqual(collector._sanitize(",./?;:[]{}"), "__________")
+        self.assertEqual(collector._sanitize("TestString"), "TestString")
+        self.assertEqual(collector._sanitize("aAbBcC_12_oi"), "aAbBcC_12_oi")