Repository: prometheus/prometheus (language: go)

Issue:
Native histogram federation not working
### What did you do?

I've added native histograms to some code of mine. The code is running on 40 servers, each one is running a prometheus instance which is scraped by a master server. "--enable-feature" native-histograms is enabled on all instances.

On each instance, I can see the values of the native histogram being properly ingested by prometheus. However on the master instance I can only see the values ingested locally, the entire series are missing for all servers being scraped. All other series from these servers is present.

### What did you expect to see?

Full time series from all servers being scrapped.

### What did you see instead? Under which circumstances?

Only locally ingested data present.

### System information

Linux 6.1.12-zen1+ #19 ZEN SMP PREEMPT Wed Mar 15 19:32:47 CET 2023 x86_64 x86_64 x86_64 GNU/Linux

### Prometheus version

```text
prometheus, version 2.44.0-rc.0 (branch: HEAD, revision: d70688038e5d444ab8beac53722df15d244038e4)
  build user:       root@c3e52c6dd464
  build date:       20230426-09:46:54
  go version:       go1.20.3
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels
```


### Prometheus configuration file

```yaml
This is from the master server. As mentioned, each target node does have the data available, so I assume the problem is not on their side. 

  - job_name: 'federate'
    scrape_interval: 15s

    honor_labels: true
    metrics_path: '/federate'

    params:
      'match[]':
        - '{job=~".*"}'

    static_configs:
      - targets:
        [...]
```


### Alertmanager version

_No response_

### Alertmanager configuration file

_No response_

### Logs

_No response_


Discussion / hints:
Two more observations:

- I turned on debug logging, but it showed nothing of interest, barely anything at all.
- I deleted the TSDB on all servers, including the master, and restarted prometheus as well as my own program. Did not change anything.
I tried to reproduce the bug with a minimal setup. But everything seems to work fine.

Can you double-checked that your master server also runs with `--enable-feature=native-histograms`?

~~I would also suspect that the query you use for the federate endpoint (`'{job=~".*"}'`) doesn't work. It violates the "vector selector must contain at least one non-empty matcher" thing.~~ (I tried it out, it (weirdly) works.)
Thanks for your response. Just double-checked:

$ ps ax|grep prometheus
2704193 ?        Ssl  284:04 /usr/bin/prometheus --storage.tsdb.retention.time 7d --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus/metrics2/ --enable-feature native-histograms

I can also see it being enabled in the prometheus Web UI.

Not sure where to continue looking. All seems fine, the web UI on each server shows the time series being ingested, it just doesn't arrive on the master. The locally scraped one does however.

Regarding the query, that is a relict of when I installed prometheus two years ago and didn't really understand too much of how it works. I never had a reason to change it afterwards as so far things have been running smoothly.
Can you minimize your setup until everything works and then gradually add more things to see when it precisely breaks?

My (working) setup is the following:

- I run the example target from client_golang ( https://github.com/prometheus/client_golang/blob/main/examples/random/main.go ).

- This is the scrape config on the directly scraping Prometheus:

```
scrape_configs:
  - job_name: 'demo'
    static_configs:
      - targets:
        - localhost:8080
```

- This is the scrape config on the federating Prometheus (note that it is not scraping anything else!):

```
scrape_configs:
  - job_name: 'federation'
    honor_labels: true
    metrics_path: /federate
    params:
      'match[]':
        - '{job="demo"}'
    static_configs:
      - targets:
        - localhost:9090
```

The native histogram is called `rpc_durations_histogram_seconds`. It arrives at both Prometheus servers.
I will try, thanks.

Just one more note in case it matters, all series are being relabeled, specifically they have the server name and the program instance (their are multiple running in parallel) added as labels. This happens on both the master and the worker nodes for the locally running instances while ingesting the data.
I'd definitely try to remove all those extra things to see if federation works in a vanilla configuration. Maybe your relabeling creates some duplicates or something?

@kaber2 has any now evidence come up? I'll close this for now as I still cannot reproduce the issue. Feel free to follow up with any updates.
My apology for the delay. It appears the reason is the use of NativeHistogramZeroThreshold: prometheus.NativeHistogramZeroThresholdZero in my histograms. Without it, they are federated properly.

Let me dig down a bit more and verify.
Ok I've confirmed that this is the cause, simply removing it without any further changes makes it work properly
OK, that's a smoking gun then. So native histograms with a zero threshold of zero aren't federated properly. I'll need to look into that. Thank you very much.

Fix patch:
diff --git a/model/textparse/protobufparse.go b/model/textparse/protobufparse.go
index 2ef52da5c43..29ccdb84df0 100644
--- a/model/textparse/protobufparse.go
+++ b/model/textparse/protobufparse.go
@@ -105,7 +105,7 @@ func (p *ProtobufParser) Series() ([]byte, *int64, float64) {
 			v = float64(s.GetSampleCount())
 		case -1:
 			v = s.GetSampleSum()
-			// Need to detect a summaries without quantile here.
+			// Need to detect summaries without quantile here.
 			if len(s.GetQuantile()) == 0 {
 				p.fieldsDone = true
 			}
@@ -564,8 +564,10 @@ func formatOpenMetricsFloat(f float64) string {
 // deciding if a histogram should be ingested as a conventional one or a native
 // one.
 func isNativeHistogram(h *dto.Histogram) bool {
-	return len(h.GetNegativeDelta()) > 0 ||
-		len(h.GetPositiveDelta()) > 0 ||
+	return h.GetZeroThreshold() > 0 ||
 		h.GetZeroCount() > 0 ||
-		h.GetZeroThreshold() > 0
+		len(h.GetNegativeDelta()) > 0 ||
+		len(h.GetPositiveDelta()) > 0 ||
+		len(h.GetNegativeCount()) > 0 ||
+		len(h.GetPositiveCount()) > 0
 }


Test patch:
diff --git a/model/textparse/protobufparse_test.go b/model/textparse/protobufparse_test.go
index 88ad9f22138..07f84f4aeb2 100644
--- a/model/textparse/protobufparse_test.go
+++ b/model/textparse/protobufparse_test.go
@@ -463,6 +463,24 @@ metric: <
   >
 >
 
+`,
+		`name: "test_float_histogram_with_zerothreshold_zero"
+help: "Test float histogram with a zero threshold of zero."
+type: HISTOGRAM
+metric: <
+  histogram: <
+    sample_count_float: 5.0
+    sample_sum: 12.1
+    schema: 3
+    positive_span: <
+      offset: 8
+      length: 2
+    >
+    positive_count: 2.0
+    positive_count: 3.0
+  >
+>
+
 `,
 		`name: "rpc_durations_seconds"
 help: "RPC latency distributions."
@@ -850,6 +868,30 @@ func TestProtobufParse(t *testing.T) {
 						"foo", "baz",
 					),
 				},
+				{
+					m:    "test_float_histogram_with_zerothreshold_zero",
+					help: "Test float histogram with a zero threshold of zero.",
+				},
+				{
+					m:   "test_float_histogram_with_zerothreshold_zero",
+					typ: MetricTypeHistogram,
+				},
+				{
+					m: "test_float_histogram_with_zerothreshold_zero",
+					fhs: &histogram.FloatHistogram{
+						Count:  5.0,
+						Sum:    12.1,
+						Schema: 3,
+						PositiveSpans: []histogram.Span{
+							{Offset: 8, Length: 2},
+						},
+						PositiveBuckets: []float64{2.0, 3.0},
+						NegativeSpans:   []histogram.Span{},
+					},
+					lset: labels.FromStrings(
+						"__name__", "test_float_histogram_with_zerothreshold_zero",
+					),
+				},
 				{
 					m:    "rpc_durations_seconds",
 					help: "RPC latency distributions.",
@@ -1550,14 +1592,38 @@ func TestProtobufParse(t *testing.T) {
 					),
 				},
 				{ // 67
+					m:    "test_float_histogram_with_zerothreshold_zero",
+					help: "Test float histogram with a zero threshold of zero.",
+				},
+				{ // 68
+					m:   "test_float_histogram_with_zerothreshold_zero",
+					typ: MetricTypeHistogram,
+				},
+				{ // 69
+					m: "test_float_histogram_with_zerothreshold_zero",
+					fhs: &histogram.FloatHistogram{
+						Count:  5.0,
+						Sum:    12.1,
+						Schema: 3,
+						PositiveSpans: []histogram.Span{
+							{Offset: 8, Length: 2},
+						},
+						PositiveBuckets: []float64{2.0, 3.0},
+						NegativeSpans:   []histogram.Span{},
+					},
+					lset: labels.FromStrings(
+						"__name__", "test_float_histogram_with_zerothreshold_zero",
+					),
+				},
+				{ // 70
 					m:    "rpc_durations_seconds",
 					help: "RPC latency distributions.",
 				},
-				{ // 68
+				{ // 71
 					m:   "rpc_durations_seconds",
 					typ: MetricTypeSummary,
 				},
-				{ // 69
+				{ // 72
 					m: "rpc_durations_seconds_count\xffservice\xffexponential",
 					v: 262,
 					lset: labels.FromStrings(
@@ -1565,7 +1631,7 @@ func TestProtobufParse(t *testing.T) {
 						"service", "exponential",
 					),
 				},
-				{ // 70
+				{ // 73
 					m: "rpc_durations_seconds_sum\xffservice\xffexponential",
 					v: 0.00025551262820703587,
 					lset: labels.FromStrings(
@@ -1573,7 +1639,7 @@ func TestProtobufParse(t *testing.T) {
 						"service", "exponential",
 					),
 				},
-				{ // 71
+				{ // 74
 					m: "rpc_durations_seconds\xffservice\xffexponential\xffquantile\xff0.5",
 					v: 6.442786329648548e-07,
 					lset: labels.FromStrings(
@@ -1582,7 +1648,7 @@ func TestProtobufParse(t *testing.T) {
 						"service", "exponential",
 					),
 				},
-				{ // 72
+				{ // 75
 					m: "rpc_durations_seconds\xffservice\xffexponential\xffquantile\xff0.9",
 					v: 1.9435742936658396e-06,
 					lset: labels.FromStrings(
@@ -1591,7 +1657,7 @@ func TestProtobufParse(t *testing.T) {
 						"service", "exponential",
 					),
 				},
-				{ // 73
+				{ // 76
 					m: "rpc_durations_seconds\xffservice\xffexponential\xffquantile\xff0.99",
 					v: 4.0471608667037015e-06,
 					lset: labels.FromStrings(
@@ -1600,22 +1666,22 @@ func TestProtobufParse(t *testing.T) {
 						"service", "exponential",
 					),
 				},
-				{ // 74
+				{ // 77
 					m:    "without_quantiles",
 					help: "A summary without quantiles.",
 				},
-				{ // 75
+				{ // 78
 					m:   "without_quantiles",
 					typ: MetricTypeSummary,
 				},
-				{ // 76
+				{ // 79
 					m: "without_quantiles_count",
 					v: 42,
 					lset: labels.FromStrings(
 						"__name__", "without_quantiles_count",
 					),
 				},
-				{ // 77
+				{ // 80
 					m: "without_quantiles_sum",
 					v: 1.234,
 					lset: labels.FromStrings(
diff --git a/web/federate_test.go b/web/federate_test.go
index 6bc0ae212a8..4fdcf8daa19 100644
--- a/web/federate_test.go
+++ b/web/federate_test.go
@@ -335,18 +335,41 @@ func TestFederationWithNativeHistograms(t *testing.T) {
 		},
 		NegativeBuckets: []int64{1, 1, -1, 0},
 	}
+	histWithoutZeroBucket := &histogram.Histogram{
+		Count:  20,
+		Sum:    99.23,
+		Schema: 1,
+		PositiveSpans: []histogram.Span{
+			{Offset: 0, Length: 2},
+			{Offset: 1, Length: 2},
+		},
+		PositiveBuckets: []int64{2, 2, -2, 0},
+		NegativeSpans: []histogram.Span{
+			{Offset: 0, Length: 2},
+			{Offset: 1, Length: 2},
+		},
+		NegativeBuckets: []int64{2, 2, -2, 0},
+	}
 	app := db.Appender(context.Background())
 	for i := 0; i < 6; i++ {
 		l := labels.FromStrings("__name__", "test_metric", "foo", fmt.Sprintf("%d", i))
 		expL := labels.FromStrings("__name__", "test_metric", "instance", "", "foo", fmt.Sprintf("%d", i))
-		if i%3 == 0 {
+		switch i {
+		case 0, 3:
 			_, err = app.Append(0, l, 100*60*1000, float64(i*100))
 			expVec = append(expVec, promql.Sample{
 				T:      100 * 60 * 1000,
 				F:      float64(i * 100),
 				Metric: expL,
 			})
-		} else {
+		case 4:
+			_, err = app.AppendHistogram(0, l, 100*60*1000, histWithoutZeroBucket.Copy(), nil)
+			expVec = append(expVec, promql.Sample{
+				T:      100 * 60 * 1000,
+				H:      histWithoutZeroBucket.ToFloat(),
+				Metric: expL,
+			})
+		default:
 			hist.ZeroCount++
 			_, err = app.AppendHistogram(0, l, 100*60*1000, hist.Copy(), nil)
 			expVec = append(expVec, promql.Sample{