Repository: VictoriaMetrics/VictoriaMetrics
Pull request #3925 (2023-03-07 13:15:15)

app/vmselect: export `seriesFetched` stat for /query responses
The change adds a new field `seriesFetched` to EvalConfig object.
Since EvalConfig object can be copied inside `Exec`,
`seriesFetched` is a pointer which can be updated by all copied
objects.

The reason for having stats is that other components, like vmalert,
could benefit from this information.


codecov[bot]: # [Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3925?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) Report
Patch coverage: **`60.00`**% and project coverage change: **`+0.06`** :tada:
> Comparison is base [(`57f50ac`)](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/commit/57f50ace96541474168cac34d737b44c29ec1311?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) 56.66% compared to head [(`30d2f7c`)](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3925?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) 56.72%.

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff             @@
##           cluster    #3925      +/-   ##
===========================================
+ Coverage    56.66%   56.72%   +0.06%     
===========================================
  Files          322      322              
  Lines        55108    55128      +20     
===========================================
+ Hits         31226    31271      +45     
+ Misses       21889    21866      -23     
+ Partials      1993     1991       -2     
```


| [Impacted Files](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3925?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) | Coverage Δ | |
|---|---|---|
| [app/vmselect/prometheus/prometheus.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3925?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3Ztc2VsZWN0L3Byb21ldGhldXMvcHJvbWV0aGV1cy5nbw==) | `5.91% <0.00%> (ø)` | |
| [app/vmselect/prometheus/query\_response.qtpl](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3925?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3Ztc2VsZWN0L3Byb21ldGhldXMvcXVlcnlfcmVzcG9uc2UucXRwbA==) | `0.00% <0.00%> (ø)` | |
| [app/vmselect/promql/eval.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3925?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3Ztc2VsZWN0L3Byb21xbC9ldmFsLmdv) | `61.84% <80.00%> (+0.04%)` | :arrow_up: |
| [app/vmselect/promql/transform.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3925?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3Ztc2VsZWN0L3Byb21xbC90cmFuc2Zvcm0uZ28=) | `88.39% <0.00%> (+0.14%)` | :arrow_up: |
| [lib/storage/partition.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3925?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL3N0b3JhZ2UvcGFydGl0aW9uLmdv) | `62.85% <0.00%> (+0.71%)` | :arrow_up: |
| [lib/mergeset/table.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3925?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL21lcmdlc2V0L3RhYmxlLmdv) | `64.26% <0.00%> (+0.79%)` | :arrow_up: |
| [app/vmalert/remotewrite/remotewrite.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3925?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvcmVtb3Rld3JpdGUvcmVtb3Rld3JpdGUuZ28=) | `65.75% <0.00%> (+3.42%)` | :arrow_up: |
| [lib/storage/part.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3925?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL3N0b3JhZ2UvcGFydC5nbw==) | `75.34% <0.00%> (+6.84%)` | :arrow_up: |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics)

</details>

[:umbrella: View full report at Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3925?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).


Patch:
diff --git a/app/vmselect/prometheus/prometheus.go b/app/vmselect/prometheus/prometheus.go
index e9f5a8544..61e8c9637 100644
--- a/app/vmselect/prometheus/prometheus.go
+++ b/app/vmselect/prometheus/prometheus.go
@@ -884,7 +884,8 @@ func QueryHandler(qt *querytracer.Tracer, startTime time.Time, at *auth.Token, w
 	qtDone := func() {
 		qt.Donef("query=%s, time=%d: series=%d", query, start, len(result))
 	}
-	WriteQueryResponse(bw, ec.IsPartialResponse.Load(), result, qt, qtDone)
+
+	WriteQueryResponse(bw, ec.IsPartialResponse.Load(), result, qt, qtDone, ec.SeriesFetched())
 	if err := bw.Flush(); err != nil {
 		return fmt.Errorf("cannot flush query response to remote client: %w", err)
 	}
diff --git a/app/vmselect/prometheus/query_response.qtpl b/app/vmselect/prometheus/query_response.qtpl
index bab0cfd06..7da8655a6 100644
--- a/app/vmselect/prometheus/query_response.qtpl
+++ b/app/vmselect/prometheus/query_response.qtpl
@@ -6,7 +6,7 @@
 {% stripspace %}
 QueryResponse generates response for /api/v1/query.
 See https://prometheus.io/docs/prometheus/latest/querying/api/#instant-queries
-{% func QueryResponse(isPartial bool, rs []netstorage.Result, qt *querytracer.Tracer, qtDone func()) %}
+{% func QueryResponse(isPartial bool, rs []netstorage.Result, qt *querytracer.Tracer, qtDone func(), seriesFetched int) %}
 {
 	{% code seriesCount := len(rs) %}
 	"status":"success",
@@ -29,6 +29,9 @@ See https://prometheus.io/docs/prometheus/latest/querying/api/#instant-queries
 				{% endfor %}
 			{% endif %}
 		]
+	},
+	"stats":{
+	    "seriesFetched": "{%d seriesFetched %}"
 	}
 	{% code
 		qt.Printf("generate /api/v1/query response for series=%d", seriesCount)
diff --git a/app/vmselect/prometheus/query_response.qtpl.go b/app/vmselect/prometheus/query_response.qtpl.go
index 452d6b6a9..bc30752ad 100644
--- a/app/vmselect/prometheus/query_response.qtpl.go
+++ b/app/vmselect/prometheus/query_response.qtpl.go
@@ -26,7 +26,7 @@ var (
 )
 
 //line app/vmselect/prometheus/query_response.qtpl:9
-func StreamQueryResponse(qw422016 *qt422016.Writer, isPartial bool, rs []netstorage.Result, qt *querytracer.Tracer, qtDone func()) {
+func StreamQueryResponse(qw422016 *qt422016.Writer, isPartial bool, rs []netstorage.Result, qt *querytracer.Tracer, qtDone func(), seriesFetched int) {
 //line app/vmselect/prometheus/query_response.qtpl:9
 	qw422016.N().S(`{`)
 //line app/vmselect/prometheus/query_response.qtpl:11
@@ -81,40 +81,44 @@ func StreamQueryResponse(qw422016 *qt422016.Writer, isPartial bool, rs []netstor
 //line app/vmselect/prometheus/query_response.qtpl:30
 	}
 //line app/vmselect/prometheus/query_response.qtpl:30
-	qw422016.N().S(`]}`)
+	qw422016.N().S(`]},"stats":{"seriesFetched": "`)
 //line app/vmselect/prometheus/query_response.qtpl:34
+	qw422016.N().D(seriesFetched)
+//line app/vmselect/prometheus/query_response.qtpl:34
+	qw422016.N().S(`"}`)
+//line app/vmselect/prometheus/query_response.qtpl:37
 	qt.Printf("generate /api/v1/query response for series=%d", seriesCount)
 	qtDone()
 
-//line app/vmselect/prometheus/query_response.qtpl:37
+//line app/vmselect/prometheus/query_response.qtpl:40
 	streamdumpQueryTrace(qw422016, qt)
-//line app/vmselect/prometheus/query_response.qtpl:37
+//line app/vmselect/prometheus/query_response.qtpl:40
 	qw422016.N().S(`}`)
-//line app/vmselect/prometheus/query_response.qtpl:39
+//line app/vmselect/prometheus/query_response.qtpl:42
 }
 
-//line app/vmselect/prometheus/query_response.qtpl:39
-func WriteQueryResponse(qq422016 qtio422016.Writer, isPartial bool, rs []netstorage.Result, qt *querytracer.Tracer, qtDone func()) {
-//line app/vmselect/prometheus/query_response.qtpl:39
+//line app/vmselect/prometheus/query_response.qtpl:42
+func WriteQueryResponse(qq422016 qtio422016.Writer, isPartial bool, rs []netstorage.Result, qt *querytracer.Tracer, qtDone func(), seriesFetched int) {
+//line app/vmselect/prometheus/query_response.qtpl:42
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmselect/prometheus/query_response.qtpl:39
-	StreamQueryResponse(qw422016, isPartial, rs, qt, qtDone)
-//line app/vmselect/prometheus/query_response.qtpl:39
+//line app/vmselect/prometheus/query_response.qtpl:42
+	StreamQueryResponse(qw422016, isPartial, rs, qt, qtDone, seriesFetched)
+//line app/vmselect/prometheus/query_response.qtpl:42
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmselect/prometheus/query_response.qtpl:39
+//line app/vmselect/prometheus/query_response.qtpl:42
 }
 
-//line app/vmselect/prometheus/query_response.qtpl:39
-func QueryResponse(isPartial bool, rs []netstorage.Result, qt *querytracer.Tracer, qtDone func()) string {
-//line app/vmselect/prometheus/query_response.qtpl:39
+//line app/vmselect/prometheus/query_response.qtpl:42
+func QueryResponse(isPartial bool, rs []netstorage.Result, qt *querytracer.Tracer, qtDone func(), seriesFetched int) string {
+//line app/vmselect/prometheus/query_response.qtpl:42
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmselect/prometheus/query_response.qtpl:39
-	WriteQueryResponse(qb422016, isPartial, rs, qt, qtDone)
-//line app/vmselect/prometheus/query_response.qtpl:39
+//line app/vmselect/prometheus/query_response.qtpl:42
+	WriteQueryResponse(qb422016, isPartial, rs, qt, qtDone, seriesFetched)
+//line app/vmselect/prometheus/query_response.qtpl:42
 	qs422016 := string(qb422016.B)
-//line app/vmselect/prometheus/query_response.qtpl:39
+//line app/vmselect/prometheus/query_response.qtpl:42
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmselect/prometheus/query_response.qtpl:39
+//line app/vmselect/prometheus/query_response.qtpl:42
 	return qs422016
-//line app/vmselect/prometheus/query_response.qtpl:39
+//line app/vmselect/prometheus/query_response.qtpl:42
 }
diff --git a/app/vmselect/promql/eval.go b/app/vmselect/promql/eval.go
index e2f90fcc3..128407948 100644
--- a/app/vmselect/promql/eval.go
+++ b/app/vmselect/promql/eval.go
@@ -142,6 +142,12 @@ type EvalConfig struct {
 
 	timestamps     []int64
 	timestampsOnce sync.Once
+
+	// seriesFetched stores the number of time series fetched
+	// from the storage during the evaluation.
+	// It is defined as a pointer since EvalConfig can be forked
+	// during the evaluation but we want to keep its state.
+	seriesFetched *int
 }
 
 // copyEvalConfig returns src copy.
@@ -162,10 +168,28 @@ func copyEvalConfig(src *EvalConfig) *EvalConfig {
 	ec.DenyPartialResponse = src.DenyPartialResponse
 	ec.IsPartialResponse.Store(src.IsPartialResponse.Load())
 
+	ec.seriesFetched = src.seriesFetched
+
 	// do not copy src.timestamps - they must be generated again.
 	return &ec
 }
 
+func (ec *EvalConfig) addStats(series int) {
+	if ec.seriesFetched == nil {
+		ec.seriesFetched = new(int)
+	}
+	*ec.seriesFetched += series
+}
+
+// SeriesFetched returns the number of series fetched from storages
+// during the evaluation.
+func (ec *EvalConfig) SeriesFetched() int {
+	if ec.seriesFetched == nil {
+		return 0
+	}
+	return *ec.seriesFetched
+}
+
 func (ec *EvalConfig) updateIsPartialResponse(isPartialResponse bool) {
 	ec.IsPartialResponse.CompareAndSwap(false, isPartialResponse)
 }
@@ -1095,6 +1119,7 @@ func evalRollupFuncWithMetricExpr(qt *querytracer.Tracer, ec *EvalConfig, funcNa
 		tss := mergeTimeseries(tssCached, nil, start, ec)
 		return tss, nil
 	}
+	ec.addStats(rssLen)
 
 	// Verify timeseries fit available memory after the rollup.
 	// Take into account points from tssCached.


Test patch:
diff --git a/app/vmselect/promql/eval_test.go b/app/vmselect/promql/eval_test.go
index 76c735b21..8ade7f0ae 100644
--- a/app/vmselect/promql/eval_test.go
+++ b/app/vmselect/promql/eval_test.go
@@ -76,3 +76,18 @@ func TestValidateMaxPointsPerSeriesSuccess(t *testing.T) {
 	f(1659962171908, 1659966077742, 5000, 800)
 	f(1659962150000, 1659966070000, 10000, 393)
 }
+
+func TestEvalConfig_SeriesFetched(t *testing.T) {
+	ec := &EvalConfig{}
+	ec.addStats(1)
+
+	if ec.SeriesFetched() != 1 {
+		t.Fatalf("expected to get 1; got %d instead", ec.SeriesFetched())
+	}
+
+	ecNew := copyEvalConfig(ec)
+	ecNew.addStats(3)
+	if ec.SeriesFetched() != 4 {
+		t.Fatalf("expected to get 4; got %d instead", ec.SeriesFetched())
+	}
+}