Repository: VictoriaMetrics/VictoriaMetrics
Pull request #1587 (2021-08-31 11:28:17)

Vmalert extra params

f41gh7:   Btw, can you update README with new flag description? 
hagen1778: > Btw, can you update README with new flag description?

There is no new flag. Extra params are only allowed internally so far.
codecov[bot]: # [Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1587?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) Report
> Merging [#1587](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1587?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) (e1c5968) into [master](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/commit/5c63d69454fd89337eb0909903023e8f1b743027?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) (5c63d69) will **decrease** coverage by `0.02%`.
> The diff coverage is `25.00%`.

[![Impacted file tree graph](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1587/graphs/tree.svg?width=650&height=150&src=pr&token=YND8dYVcXn&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics)](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1587?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics)

```diff
@@            Coverage Diff             @@
##           master    #1587      +/-   ##
==========================================
- Coverage   60.72%   60.70%   -0.03%     
==========================================
  Files         232      232              
  Lines       38011    38014       +3     
==========================================
- Hits        23082    23075       -7     
- Misses      13383    13392       +9     
- Partials     1546     1547       +1     
```


| [Impacted Files](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1587?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) | Coverage Δ | |
|---|---|---|
| [app/vmalert/datasource/init.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1587/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvZGF0YXNvdXJjZS9pbml0Lmdv) | `0.00% <0.00%> (ø)` | |
| [app/vmalert/datasource/vm.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1587/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvZGF0YXNvdXJjZS92bS5nbw==) | `83.33% <ø> (ø)` | |
| [app/vmalert/main.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1587/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvbWFpbi5nbw==) | `35.12% <0.00%> (-0.18%)` | :arrow_down: |
| [app/vmalert/datasource/vm\_prom\_api.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1587/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvZGF0YXNvdXJjZS92bV9wcm9tX2FwaS5nbw==) | `86.66% <100.00%> (ø)` | |
| [lib/mergeset/table.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1587/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL21lcmdlc2V0L3RhYmxlLmdv) | `65.56% <0.00%> (-1.27%)` | :arrow_down: |
| [lib/storage/partition.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1587/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL3N0b3JhZ2UvcGFydGl0aW9uLmdv) | `65.88% <0.00%> (+0.16%)` | :arrow_up: |
| [app/vmselect/promql/aggr\_incremental.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1587/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3Ztc2VsZWN0L3Byb21xbC9hZ2dyX2luY3JlbWVudGFsLmdv) | `85.24% <0.00%> (+1.20%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1587?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1587?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics). Last update [5c63d69...e1c5968](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1587?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).


Patch:
diff --git a/app/vmalert/README.md b/app/vmalert/README.md
index 253d907f6..c5cad17f3 100644
--- a/app/vmalert/README.md
+++ b/app/vmalert/README.md
@@ -95,6 +95,13 @@ name: <string>
 extra_filter_labels:
   [ <labelname>: <labelvalue> ... ]
 
+# Optional list of labels added to every rule within a group.
+# It has priority over the external labels.
+# Labels are commonly used for adding environment
+# or tenant-specific tag.
+labels:
+  [ <labelname>: <labelvalue> ... ]
+
 rules:
   [ - <rule> ... ]
 ```
@@ -304,6 +311,8 @@ to [/query_range](https://prometheus.io/docs/prometheus/latest/querying/api/#ran
 of the configured `-datasource.url`. Returned data then processed according to the rule type and
 backfilled to `-remoteWrite.url` via [Remote Write protocol](https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations).
 Vmalert respects `evaluationInterval` value set by flag or per-group during the replay.
+Vmalert automatically disables caching on VictoriaMetrics side by sending `nocache=1` param. It allows
+to prevent cache pollution and unwanted time range boundaries adjustment during backfilling.
 
 #### Recording rules
 
diff --git a/app/vmalert/config/config.go b/app/vmalert/config/config.go
index 76d92cb15..8a21cc90b 100644
--- a/app/vmalert/config/config.go
+++ b/app/vmalert/config/config.go
@@ -31,6 +31,9 @@ type Group struct {
 	// request withing a group. Is compatible only with VM datasources.
 	// See https://docs.victoriametrics.com#prometheus-querying-api-enhancements
 	ExtraFilterLabels map[string]string `yaml:"extra_filter_labels"`
+	// Labels is a set of label value pairs, that will be added to every rule.
+	// It has priority over the external labels.
+	Labels map[string]string `yaml:"labels"`
 	// Checksum stores the hash of yaml definition for this group.
 	// May be used to detect any changes like rules re-ordering etc.
 	Checksum string
diff --git a/app/vmalert/datasource/init.go b/app/vmalert/datasource/init.go
index 671b0ee9e..95029744e 100644
--- a/app/vmalert/datasource/init.go
+++ b/app/vmalert/datasource/init.go
@@ -31,8 +31,15 @@ var (
 		`In VM "round_digits" limits the number of digits after the decimal point in response values.`)
 )
 
+// Param represents an HTTP GET param
+type Param struct {
+	Key, Value string
+}
+
 // Init creates a Querier from provided flag values.
-func Init() (QuerierBuilder, error) {
+// Provided extraParams will be added as GET params to
+// each request.
+func Init(extraParams []Param) (QuerierBuilder, error) {
 	if *addr == "" {
 		return nil, fmt.Errorf("datasource.url is empty")
 	}
@@ -43,9 +50,11 @@ func Init() (QuerierBuilder, error) {
 	}
 	tr.MaxIdleConnsPerHost = *maxIdleConnections
 
-	var rd string
 	if *roundDigits > 0 {
-		rd = fmt.Sprintf("%d", *roundDigits)
+		extraParams = append(extraParams, Param{
+			Key:   "round_digits",
+			Value: fmt.Sprintf("%d", *roundDigits),
+		})
 	}
 
 	return &VMStorage{
@@ -56,7 +65,7 @@ func Init() (QuerierBuilder, error) {
 		appendTypePrefix: *appendTypePrefix,
 		lookBack:         *lookBack,
 		queryStep:        *queryStep,
-		roundDigits:      rd,
 		dataSourceType:   NewPrometheusType(),
+		extraParams:      extraParams,
 	}, nil
 }
diff --git a/app/vmalert/datasource/vm.go b/app/vmalert/datasource/vm.go
index cda418852..389a60ba2 100644
--- a/app/vmalert/datasource/vm.go
+++ b/app/vmalert/datasource/vm.go
@@ -18,11 +18,11 @@ type VMStorage struct {
 	appendTypePrefix bool
 	lookBack         time.Duration
 	queryStep        time.Duration
-	roundDigits      string
 
 	dataSourceType     Type
 	evaluationInterval time.Duration
 	extraLabels        []string
+	extraParams        []Param
 }
 
 // Clone makes clone of VMStorage, shares http client.
diff --git a/app/vmalert/datasource/vm_prom_api.go b/app/vmalert/datasource/vm_prom_api.go
index e1e1b4937..62154c313 100644
--- a/app/vmalert/datasource/vm_prom_api.go
+++ b/app/vmalert/datasource/vm_prom_api.go
@@ -155,11 +155,11 @@ func (s *VMStorage) setPrometheusReqParams(r *http.Request, query string) {
 		// override step with user-specified value
 		q.Set("step", s.queryStep.String())
 	}
-	if s.roundDigits != "" {
-		q.Set("round_digits", s.roundDigits)
-	}
 	for _, l := range s.extraLabels {
 		q.Add("extra_label", l)
 	}
+	for _, p := range s.extraParams {
+		q.Add(p.Key, p.Value)
+	}
 	r.URL.RawQuery = q.Encode()
 }
diff --git a/app/vmalert/group.go b/app/vmalert/group.go
index f9f2b957a..24b19193b 100644
--- a/app/vmalert/group.go
+++ b/app/vmalert/group.go
@@ -18,15 +18,17 @@ import (
 
 // Group is an entity for grouping rules
 type Group struct {
-	mu                sync.RWMutex
-	Name              string
-	File              string
-	Rules             []Rule
-	Type              datasource.Type
-	Interval          time.Duration
-	Concurrency       int
-	Checksum          string
+	mu          sync.RWMutex
+	Name        string
+	File        string
+	Rules       []Rule
+	Type        datasource.Type
+	Interval    time.Duration
+	Concurrency int
+	Checksum    string
+
 	ExtraFilterLabels map[string]string
+	Labels            map[string]string
 
 	doneCh     chan struct{}
 	finishedCh chan struct{}
@@ -50,6 +52,23 @@ func newGroupMetrics(name, file string) *groupMetrics {
 	return m
 }
 
+// merges group rule labels into result map
+// set2 has priority over set1.
+func mergeLabels(groupName, ruleName string, set1, set2 map[string]string) map[string]string {
+	r := map[string]string{}
+	for k, v := range set1 {
+		r[k] = v
+	}
+	for k, v := range set2 {
+		if prevV, ok := r[k]; ok {
+			logger.Infof("label %q=%q for rule %q.%q overwritten with external label %q=%q",
+				k, prevV, groupName, ruleName, k, v)
+		}
+		r[k] = v
+	}
+	return r
+}
+
 func newGroup(cfg config.Group, qb datasource.QuerierBuilder, defaultInterval time.Duration, labels map[string]string) *Group {
 	g := &Group{
 		Type:              cfg.Type,
@@ -59,6 +78,7 @@ func newGroup(cfg config.Group, qb datasource.QuerierBuilder, defaultInterval ti
 		Concurrency:       cfg.Concurrency,
 		Checksum:          cfg.Checksum,
 		ExtraFilterLabels: cfg.ExtraFilterLabels,
+		Labels:            cfg.Labels,
 
 		doneCh:     make(chan struct{}),
 		finishedCh: make(chan struct{}),
@@ -73,17 +93,20 @@ func newGroup(cfg config.Group, qb datasource.QuerierBuilder, defaultInterval ti
 	}
 	rules := make([]Rule, len(cfg.Rules))
 	for i, r := range cfg.Rules {
-		// override rule labels with external labels
-		for k, v := range labels {
-			if prevV, ok := r.Labels[k]; ok {
-				logger.Infof("label %q=%q for rule %q.%q overwritten with external label %q=%q",
-					k, prevV, g.Name, r.Name(), k, v)
-			}
-			if r.Labels == nil {
-				r.Labels = map[string]string{}
-			}
-			r.Labels[k] = v
+		var extraLabels map[string]string
+		// apply external labels
+		if len(labels) > 0 {
+			extraLabels = labels
+		}
+		// apply group labels, it has priority on external labels
+		if len(cfg.Labels) > 0 {
+			extraLabels = mergeLabels(g.Name, r.Name(), extraLabels, g.Labels)
 		}
+		// apply rules labels, it has priority on other labels
+		if len(extraLabels) > 0 {
+			r.Labels = mergeLabels(g.Name, r.Name(), extraLabels, r.Labels)
+		}
+
 		rules[i] = g.newRule(qb, r)
 	}
 	g.Rules = rules
@@ -110,6 +133,7 @@ func (g *Group) ID() uint64 {
 
 // Restore restores alerts state for group rules
 func (g *Group) Restore(ctx context.Context, qb datasource.QuerierBuilder, lookback time.Duration, labels map[string]string) error {
+	labels = mergeLabels(g.Name, "", labels, g.Labels)
 	for _, rule := range g.Rules {
 		rr, ok := rule.(*AlertingRule)
 		if !ok {
@@ -169,6 +193,7 @@ func (g *Group) updateWith(newGroup *Group) error {
 	g.Type = newGroup.Type
 	g.Concurrency = newGroup.Concurrency
 	g.ExtraFilterLabels = newGroup.ExtraFilterLabels
+	g.Labels = newGroup.Labels
 	g.Checksum = newGroup.Checksum
 	g.Rules = newRules
 	return nil
diff --git a/app/vmalert/main.go b/app/vmalert/main.go
index db21b9900..27a47b538 100644
--- a/app/vmalert/main.go
+++ b/app/vmalert/main.go
@@ -91,7 +91,10 @@ func main() {
 		if err != nil {
 			logger.Fatalf("cannot parse configuration file: %s", err)
 		}
-		q, err := datasource.Init()
+		// prevent queries from caching and boundaries aligning
+		// when querying VictoriaMetrics datasource.
+		noCache := datasource.Param{Key: "nocache", Value: "1"}
+		q, err := datasource.Init([]datasource.Param{noCache})
 		if err != nil {
 			logger.Fatalf("failed to init datasource: %s", err)
 		}
@@ -139,7 +142,7 @@ var (
 )
 
 func newManager(ctx context.Context) (*manager, error) {
-	q, err := datasource.Init()
+	q, err := datasource.Init(nil)
 	if err != nil {
 		return nil, fmt.Errorf("failed to init datasource: %w", err)
 	}
diff --git a/app/vmalert/manager.go b/app/vmalert/manager.go
index e73d399fb..630a9a271 100644
--- a/app/vmalert/manager.go
+++ b/app/vmalert/manager.go
@@ -148,6 +148,7 @@ func (g *Group) toAPI() APIGroup {
 		Interval:          g.Interval.String(),
 		Concurrency:       g.Concurrency,
 		ExtraFilterLabels: g.ExtraFilterLabels,
+		Labels:            g.Labels,
 	}
 	for _, r := range g.Rules {
 		switch v := r.(type) {
diff --git a/app/vmalert/web_types.go b/app/vmalert/web_types.go
index 6b677ef8c..bcf673222 100644
--- a/app/vmalert/web_types.go
+++ b/app/vmalert/web_types.go
@@ -27,6 +27,7 @@ type APIGroup struct {
 	Interval          string             `json:"interval"`
 	Concurrency       int                `json:"concurrency"`
 	ExtraFilterLabels map[string]string  `json:"extra_filter_labels"`
+	Labels            map[string]string  `json:"labels,omitempty"`
 	AlertingRules     []APIAlertingRule  `json:"alerting_rules"`
 	RecordingRules    []APIRecordingRule `json:"recording_rules"`
 }
diff --git a/docs/vmalert.md b/docs/vmalert.md
index d3072bd34..5e2f51678 100644
--- a/docs/vmalert.md
+++ b/docs/vmalert.md
@@ -99,6 +99,13 @@ name: <string>
 extra_filter_labels:
   [ <labelname>: <labelvalue> ... ]
 
+# Optional list of labels added to every rule within a group.
+# It has priority over the external labels.
+# Labels are commonly used for adding environment
+# or tenant-specific tag.
+labels:
+  [ <labelname>: <labelvalue> ... ]
+
 rules:
   [ - <rule> ... ]
 ```
@@ -509,7 +516,9 @@ The shortlist of configuration flags is the following:
   -remoteWrite.tlsServerName string
     	Optional TLS server name to use for connections to -remoteWrite.url. By default the server name from -remoteWrite.url is used
   -remoteWrite.url string
-    	Optional URL to VictoriaMetrics or vminsert where to persist alerts state and recording rules results in form of timeseries. For example, if -remoteWrite.url=http://127.0.0.1:8428 is specified, then the alerts state will be written to http://127.0.0.1:8428/api/v1/write . See also -remoteWrite.disablePathAppend
+      Optional URL to VictoriaMetrics or vminsert where to persist alerts state and recording rules results in form of timeseries. For example, if -remoteWrite.url=http://127.0.0.1:8428 is specified, then the alerts state will be written to http://127.0.0.1:8428/api/v1/write . See also -remoteWrite.disablePathAppend
+  -remoteWrite.disablePathAppend
+      Whether to disable automatic appending of '/api/v1/write' path to the configured -remoteWrite.url.
   -replay.maxDatapointsPerQuery int
     	Max number of data points expected in one request. The higher the value, the less requests will be made during replay. (default 1000)
   -replay.ruleRetryAttempts int


Test patch:
diff --git a/app/vmalert/config/testdata/dir/rules-update0-good.rules b/app/vmalert/config/testdata/dir/rules-update0-good.rules
index ffbfdf30f..e91277ad0 100644
--- a/app/vmalert/config/testdata/dir/rules-update0-good.rules
+++ b/app/vmalert/config/testdata/dir/rules-update0-good.rules
@@ -3,6 +3,8 @@ groups:
     interval: 2s
     concurrency: 2
     type: prometheus
+    labels:
+      cluster: main
     rules:
       - alert: up
         expr: up == 0
diff --git a/app/vmalert/config/testdata/dir/rules1-good.rules b/app/vmalert/config/testdata/dir/rules1-good.rules
index 1d04b1a6c..7f09278ed 100644
--- a/app/vmalert/config/testdata/dir/rules1-good.rules
+++ b/app/vmalert/config/testdata/dir/rules1-good.rules
@@ -1,5 +1,7 @@
 groups:
   - name: duplicatedGroupDiffFiles
+    labels:
+       dc: gcp
     rules:
       - alert: VMRows
         for: 5m
diff --git a/app/vmalert/datasource/vm_test.go b/app/vmalert/datasource/vm_test.go
index c86deb3cd..8dbf61698 100644
--- a/app/vmalert/datasource/vm_test.go
+++ b/app/vmalert/datasource/vm_test.go
@@ -385,7 +385,7 @@ func TestRequestParams(t *testing.T) {
 			"round digits",
 			false,
 			&VMStorage{
-				roundDigits: "10",
+				extraParams: []Param{{"round_digits", "10"}},
 			},
 			func(t *testing.T, r *http.Request) {
 				exp := fmt.Sprintf("query=%s&round_digits=10&time=%d", query, timestamp.Unix())
@@ -421,6 +421,20 @@ func TestRequestParams(t *testing.T) {
 				checkEqualString(t, exp, r.URL.RawQuery)
 			},
 		},
+		{
+			"extra params",
+			false,
+			&VMStorage{
+				extraParams: []Param{
+					{Key: "nocache", Value: "1"},
+					{Key: "max_lookback", Value: "1h"},
+				},
+			},
+			func(t *testing.T, r *http.Request) {
+				exp := fmt.Sprintf("max_lookback=1h&nocache=1&query=%s&time=%d", query, timestamp.Unix())
+				checkEqualString(t, exp, r.URL.RawQuery)
+			},
+		},
 	}
 
 	for _, tc := range testCases {
diff --git a/app/vmalert/manager_test.go b/app/vmalert/manager_test.go
index bd5859fab..31d885369 100644
--- a/app/vmalert/manager_test.go
+++ b/app/vmalert/manager_test.go
@@ -148,7 +148,7 @@ func TestManagerUpdate(t *testing.T) {
 							Name:   "VMRows",
 							Expr:   "vm_rows > 0",
 							For:    5 * time.Minute,
-							Labels: map[string]string{"label": "bar"},
+							Labels: map[string]string{"dc": "gcp", "label": "bar"},
 							Annotations: map[string]string{
 								"summary":     "{{ $value }}",
 								"description": "{{$labels}}",