Repository: prometheus/prometheus (language: go)

Issue:
otlp writer: colons (':') in metric names replaced with underscores ('_')
### What did you do?

When using the OTLP collector to scrape metrics from a Prometheus `/federate` API endpoint and write them to a central Prometheus instance via otlphttp, the metric names are unexpectedly modified.
Steps:
1. Set up an OTLP collector with a prometheusreceiver configured to scrape the `/federate` API of an existing Prometheus instance.
2. Configure the OTLP collector to write metrics to a central Prometheus instance using the otlphttp exporter.
3. Observe the metric names in the central Prometheus instance.

### What did you expect to see?

Metric names should be preserved exactly as they appear in the source Prometheus instance, including any colons (':') in the names.

### What did you see instead? Under which circumstances?

Colons (':') in metric names are replaced with underscores ('_').
Example:
Original metric name: `node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate`
Modified metric name: `node_namespace_pod_container_container_cpu_usage_seconds_total_sum_rate`

### System information

_No response_

### Prometheus version

```text
prometheus, version 3.0.0-beta.0 (branch: HEAD, revision: 91608c002f08039c7c0f5a3fbebaf2cf78c3b1f2)
  build user:       root@9c2eedf0b49d
  build date:       20240910-19:12:25
  go version:       go1.23.1
  platform:         linux/arm64
  tags:             netgo,builtinassets,stringlabels
```


### Prometheus configuration file

_No response_

### Alertmanager version

_No response_

### Alertmanager configuration file

_No response_

### Logs

_No response_


Discussion / hints:
The issue stems from the `AddMetricSuffixes` setting, which is always enabled. This setting triggers the `normalizeName` function, which will "remove all non-alphanumerics".
https://github.com/prometheus/prometheus/blob/7c7116fea8343795cae6da42960cacd0207a2af8/storage/remote/otlptranslator/prometheus/normalize_name.go#L114
https://github.com/prometheus/prometheus/blob/7c7116fea8343795cae6da42960cacd0207a2af8/storage/remote/write_handler.go#L515

> When using the OTLP collector to scrape metrics from a Prometheus /federate API endpoint and write them to a central Prometheus instance via otlphttp, the metric names are unexpectedly modified.

~Can I ask how metrics from your /federate Prometheus endpoint have ever `:` in names and labels? Do you process and change them to add those in your Otel Collector?~
~Anyway, this happens because Prometheus does not support UTF-8 by default, you need to enable it (in recent version): https://prometheus.io/docs/prometheus/latest/feature_flags/#utf-8-name-support~
~See also related discussion on what to do in Prometheus 3.0: https://github.com/prometheus/prometheus/issues/14990~
~Also related PromCon talk: https://www.youtube.com/watch?v=POSFqCBLH-o~
~Hope that helps, closing for now, we can re-open if something is still not clear here!~

EDIT: Reopening, my response had little sense, `:` are supported characters so it looks like a bug in normalization.

cc @ArthurSens 
Interesting, I THINK it will be fixed (accidentally) by https://github.com/prometheus/prometheus/pull/14974 because we're changing the escaping method to use `model.EscapeName()`.

I can double check later :)
I'm still looking into this, but it's strange that we're seeing this bug at all. We have tests for building compliant names and the tests show that colons aren't replaced.

https://github.com/prometheus/prometheus/blob/d0eecb122393b72e763053586a64bc6972dd8c6b/storage/remote/otlptranslator/prometheus/normalize_name_test.go#L201-L202

I can't find tests that simulate the whole e2e flow, like creating an OTLP HTTP handler, sending OTLP metrics and verifying if they are translated correctly. It would be nice to have them
> I'm still looking into this, but it's strange that we're seeing this bug at all. We have tests for building compliant names and the tests show that colons aren't replaced.
> 
> https://github.com/prometheus/prometheus/blob/d0eecb122393b72e763053586a64bc6972dd8c6b/storage/remote/otlptranslator/prometheus/normalize_name_test.go#L201-L202
> 
> I can't find tests that simulate the whole e2e flow, like creating an OTLP HTTP handler, sending OTLP metrics and verifying if they are translated correctly. It would be nice to have them

The behavior is actually specified when `AddMetricSuffixes` is enabled. There's an existing test case that specifically demonstrates this colon replacement:
https://github.com/prometheus/prometheus/blob/d0eecb122393b72e763053586a64bc6972dd8c6b/storage/remote/otlptranslator/prometheus/normalize_name_test.go#L187
Oh good catch! Our [HTTP handler also hardcodes the addition of metric suffixes](https://github.com/prometheus/prometheus/blob/d0eecb122393b72e763053586a64bc6972dd8c6b/storage/remote/write_handler.go#L515), which means we're indeed always replacing colons.

I'll open another PR today, thanks for the help with investigation :)

Fix patch:
diff --git a/storage/remote/otlptranslator/prometheus/normalize_name.go b/storage/remote/otlptranslator/prometheus/normalize_name.go
index 0f472b80a09..63fd7ceca31 100644
--- a/storage/remote/otlptranslator/prometheus/normalize_name.go
+++ b/storage/remote/otlptranslator/prometheus/normalize_name.go
@@ -114,7 +114,7 @@ func normalizeName(metric pmetric.Metric, namespace string) string {
 	// Split metric name into "tokens" (remove all non-alphanumerics)
 	nameTokens := strings.FieldsFunc(
 		metric.Name(),
-		func(r rune) bool { return !unicode.IsLetter(r) && !unicode.IsDigit(r) },
+		func(r rune) bool { return !unicode.IsLetter(r) && !unicode.IsDigit(r) && r != ':' },
 	)
 
 	// Split unit at the '/' if any


Test patch:
diff --git a/storage/remote/otlptranslator/prometheus/normalize_name_test.go b/storage/remote/otlptranslator/prometheus/normalize_name_test.go
index 07b9b0a784b..fca32a0ec2d 100644
--- a/storage/remote/otlptranslator/prometheus/normalize_name_test.go
+++ b/storage/remote/otlptranslator/prometheus/normalize_name_test.go
@@ -184,8 +184,8 @@ func TestBuildCompliantNameWithNormalize(t *testing.T) {
 	require.Equal(t, "system_network_io_bytes_total", BuildCompliantName(createCounter("network.io", "By"), "system", true))
 	require.Equal(t, "_3_14_digits", BuildCompliantName(createGauge("3.14 digits", ""), "", true))
 	require.Equal(t, "envoy_rule_engine_zlib_buf_error", BuildCompliantName(createGauge("envoy__rule_engine_zlib_buf_error", ""), "", true))
-	require.Equal(t, "foo_bar", BuildCompliantName(createGauge(":foo::bar", ""), "", true))
-	require.Equal(t, "foo_bar_total", BuildCompliantName(createCounter(":foo::bar", ""), "", true))
+	require.Equal(t, ":foo::bar", BuildCompliantName(createGauge(":foo::bar", ""), "", true))
+	require.Equal(t, ":foo::bar_total", BuildCompliantName(createCounter(":foo::bar", ""), "", true))
 	// Gauges with unit 1 are considered ratios.
 	require.Equal(t, "foo_bar_ratio", BuildCompliantName(createGauge("foo.bar", "1"), "", true))
 	// Slashes in units are converted.