Repository: prometheus/prometheus (language: go)

Issue:
Fix the `prometheus_tsdb_sample_ooo_delta` metric
### What did you do?

Run Prometheus with out-of-order support enabled

### What did you expect to see?

`prometheus_tsdb_sample_ooo_delta` to have the right buckets filled.

### What did you see instead? Under which circumstances?

`prometheus_tsdb_sample_ooo_delta` is initialized with buckets as seconds
https://github.com/prometheus/prometheus/blob/f88a0a7d83bb9ef4387ff158d43f5a0c580fffdf/tsdb/head.go#L440-L448

But Prometheus uses milliseconds as its timestamp. So observing the ooo_delta with milliseconds will lead to wrong buckets being filled. See:

https://github.com/prometheus/prometheus/blob/f88a0a7d83bb9ef4387ff158d43f5a0c580fffdf/tsdb/head_append.go#L370

Until Prometheus starts using native histograms, TSDB can compromise from its timestamp unit agnostic design for this metric and do `float64(delta)/1000` instead.

---

This issue is broken down from https://github.com/prometheus/prometheus/issues/11329



Discussion / hints:

Fix patch:
diff --git a/tsdb/head_append.go b/tsdb/head_append.go
index 33cfc0eb3ef..8a622fafe5a 100644
--- a/tsdb/head_append.go
+++ b/tsdb/head_append.go
@@ -367,7 +367,7 @@ func (a *headAppender) Append(ref storage.SeriesRef, lset labels.Labels, t int64
 	}
 	s.Unlock()
 	if delta > 0 {
-		a.head.metrics.oooHistogram.Observe(float64(delta))
+		a.head.metrics.oooHistogram.Observe(float64(delta) / 1000)
 	}
 	if err != nil {
 		switch err {


Test patch: