Repository: VictoriaMetrics/VictoriaMetrics
Pull request #5229 (2023-10-24 04:45:43)

remove vmalert-tool code from branch cluster
Follow up https://github.com/VictoriaMetrics/VictoriaMetrics/commit/130e0ea5f02ba6d6238b530fdee90fb0c36462a5.
vmalert-tool can't be easily adapted for vmcluster now, cause it needs to set up the whole vmcluster[vminsert+vmstorage+vmselect] first. 

> You can use `vmalert-tool` to run unit tests for alerting and recording rules.
> It will perform the following actions:
> * sets up an isolated VictoriaMetrics instance;
> * simulates the periodic ingestion of time series;
> * queries the ingested data for recording and alerting rules evaluation like [vmalert](https://docs.victoriametrics.com/vmalert.html);

But component packages have functions that not exported and variables with same name, so to implement this for cluster will need amount of code refactor and doesn't look like a good thing to themselves.
So I want to remove it from the cluster branch.

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5229?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) Report
Attention: `407 lines` in your changes are missing coverage. Please review.
> Comparison is base [(`1b4e7fc`)](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/commit/1b4e7fcdb3655536cf0f24ae1e7ec5b669e423c5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) 59.14% compared to head [(`5b227ef`)](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5229?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) 58.87%.
> Report is 21 commits behind head on cluster.

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff             @@
##           cluster    #5229      +/-   ##
===========================================
- Coverage    59.14%   58.87%   -0.28%     
===========================================
  Files          400      404       +4     
  Lines        75668    76055     +387     
===========================================
+ Hits         44754    44775      +21     
- Misses       28444    28772     +328     
- Partials      2470     2508      +38     
```


| [Files](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5229?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) | Coverage Δ | |
|---|---|---|
| [app/vmalert/replay.go](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5229?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvcmVwbGF5Lmdv) | `42.50% <100.00%> (-20.34%)` | :arrow_down: |
| [app/vmalert/rule/utils.go](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5229?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvcnVsZS91dGlscy5nbw==) | `96.59% <ø> (ø)` | |
| [lib/promscrape/config.go](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5229?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL3Byb21zY3JhcGUvY29uZmlnLmdv) | `34.90% <ø> (ø)` | |
| [app/vmalert/main.go](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5229?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvbWFpbi5nbw==) | `39.08% <0.00%> (ø)` | |
| [app/vmalert/manager.go](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5229?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvbWFuYWdlci5nbw==) | `88.07% <94.73%> (+0.64%)` | :arrow_up: |
| [app/vmagent/remotewrite/remotewrite.go](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5229?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWdlbnQvcmVtb3Rld3JpdGUvcmVtb3Rld3JpdGUuZ28=) | `0.00% <0.00%> (ø)` | |
| [app/vmalert/web.go](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5229?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvd2ViLmdv) | `66.66% <85.71%> (ø)` | |
| [app/vmalert/web.qtpl](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5229?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvd2ViLnF0cGw=) | `72.77% <76.19%> (+1.45%)` | :arrow_up: |
| [app/vmselect/promql/transform.go](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5229?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3Ztc2VsZWN0L3Byb21xbC90cmFuc2Zvcm0uZ28=) | `88.35% <83.78%> (+0.06%)` | :arrow_up: |
| [app/vmalert/rule/recording.go](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5229?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvcnVsZS9yZWNvcmRpbmcuZ28=) | `77.37% <71.42%> (ø)` | |
| ... and [9 more](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5229?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) | |

... and [12 files with indirect coverage changes](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5229/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5229?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).

Haleygo: I tried to implement it, not for vmalert-tool itself, but to use it's unit test as E2E tests for vmcluster covering some basic write+query scenarios, like it does for [vmsingle](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5179) now.
But the code changes become bigger for both vmalert-tool(to separate the set up process) and three components, and doesn't seems good(


Patch:
diff --git a/app/vmalert-tool/Makefile b/app/vmalert-tool/Makefile
deleted file mode 100644
index dbb6b373a..000000000
--- a/app/vmalert-tool/Makefile
+++ /dev/null
@@ -1,103 +0,0 @@
-# All these commands must run from repository root.
-
-vmalert-tool:
-	APP_NAME=vmalert-tool $(MAKE) app-local
-
-vmalert-tool-race:
-	APP_NAME=vmalert-tool RACE=-race $(MAKE) app-local
-
-vmalert-tool-prod:
-	APP_NAME=vmalert-tool $(MAKE) app-via-docker
-
-vmalert-tool-pure-prod:
-	APP_NAME=vmalert-tool $(MAKE) app-via-docker-pure
-
-vmalert-tool-linux-amd64-prod:
-	APP_NAME=vmalert-tool $(MAKE) app-via-docker-linux-amd64
-
-vmalert-tool-linux-arm-prod:
-	APP_NAME=vmalert-tool $(MAKE) app-via-docker-linux-arm
-
-vmalert-tool-linux-arm64-prod:
-	APP_NAME=vmalert-tool $(MAKE) app-via-docker-linux-arm64
-
-vmalert-tool-linux-ppc64le-prod:
-	APP_NAME=vmalert-tool $(MAKE) app-via-docker-linux-ppc64le
-
-vmalert-tool-linux-386-prod:
-	APP_NAME=vmalert-tool $(MAKE) app-via-docker-linux-386
-
-vmalert-tool-darwin-amd64-prod:
-	APP_NAME=vmalert-tool $(MAKE) app-via-docker-darwin-amd64
-
-vmalert-tool-darwin-arm64-prod:
-	APP_NAME=vmalert-tool $(MAKE) app-via-docker-darwin-arm64
-
-vmalert-tool-freebsd-amd64-prod:
-	APP_NAME=vmalert-tool $(MAKE) app-via-docker-freebsd-amd64
-
-vmalert-tool-openbsd-amd64-prod:
-	APP_NAME=vmalert-tool $(MAKE) app-via-docker-openbsd-amd64
-
-vmalert-tool-windows-amd64-prod:
-	APP_NAME=vmalert-tool $(MAKE) app-via-docker-windows-amd64
-
-package-vmalert-tool:
-	APP_NAME=vmalert-tool $(MAKE) package-via-docker
-
-package-vmalert-tool-pure:
-	APP_NAME=vmalert-tool $(MAKE) package-via-docker-pure
-
-package-vmalert-tool-amd64:
-	APP_NAME=vmalert-tool $(MAKE) package-via-docker-amd64
-
-package-vmalert-tool-arm:
-	APP_NAME=vmalert-tool $(MAKE) package-via-docker-arm
-
-package-vmalert-tool-arm64:
-	APP_NAME=vmalert-tool $(MAKE) package-via-docker-arm64
-
-package-vmalert-tool-ppc64le:
-	APP_NAME=vmalert-tool $(MAKE) package-via-docker-ppc64le
-
-package-vmalert-tool-386:
-	APP_NAME=vmalert-tool $(MAKE) package-via-docker-386
-
-publish-vmalert-tool:
-	APP_NAME=vmalert-tool $(MAKE) publish-via-docker
-
-vmalert-tool-linux-amd64:
-	APP_NAME=vmalert-tool CGO_ENABLED=1 GOOS=linux GOARCH=amd64 $(MAKE) app-local-goos-goarch
-
-vmalert-tool-linux-arm:
-	APP_NAME=vmalert-tool CGO_ENABLED=0 GOOS=linux GOARCH=arm $(MAKE) app-local-goos-goarch
-
-vmalert-tool-linux-arm64:
-	APP_NAME=vmalert-tool CGO_ENABLED=0 GOOS=linux GOARCH=arm64 $(MAKE) app-local-goos-goarch
-
-vmalert-tool-linux-ppc64le:
-	APP_NAME=vmalert-tool CGO_ENABLED=0 GOOS=linux GOARCH=ppc64le $(MAKE) app-local-goos-goarch
-
-vmalert-tool-linux-s390x:
-	APP_NAME=vmalert-tool CGO_ENABLED=0 GOOS=linux GOARCH=s390x $(MAKE) app-local-goos-goarch
-
-vmalert-tool-linux-386:
-	APP_NAME=vmalert-tool CGO_ENABLED=0 GOOS=linux GOARCH=386 $(MAKE) app-local-goos-goarch
-
-vmalert-tool-darwin-amd64:
-	APP_NAME=vmalert-tool CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 $(MAKE) app-local-goos-goarch
-
-vmalert-tool-darwin-arm64:
-	APP_NAME=vmalert-tool CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 $(MAKE) app-local-goos-goarch
-
-vmalert-tool-freebsd-amd64:
-	APP_NAME=vmalert-tool CGO_ENABLED=0 GOOS=freebsd GOARCH=amd64 $(MAKE) app-local-goos-goarch
-
-vmalert-tool-openbsd-amd64:
-	APP_NAME=vmalert-tool CGO_ENABLED=0 GOOS=openbsd GOARCH=amd64 $(MAKE) app-local-goos-goarch
-
-vmalert-tool-windows-amd64:
-	GOARCH=amd64 APP_NAME=vmalert-tool $(MAKE) app-local-windows-goarch
-
-vmalert-tool-pure:
-	APP_NAME=vmalert-tool $(MAKE) app-local-pure
diff --git a/app/vmalert-tool/main.go b/app/vmalert-tool/main.go
deleted file mode 100644
index 790cebd8b..000000000
--- a/app/vmalert-tool/main.go
+++ /dev/null
@@ -1,54 +0,0 @@
-package main
-
-import (
-	"fmt"
-	"log"
-	"os"
-	"time"
-
-	"github.com/urfave/cli/v2"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert-tool/unittest"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/buildinfo"
-)
-
-func main() {
-	start := time.Now()
-	app := &cli.App{
-		Name:      "vmalert-tool",
-		Usage:     "VMAlert command-line tool",
-		UsageText: "More info in https://docs.victoriametrics.com/vmalert-tool.html",
-		Version:   buildinfo.Version,
-		Commands: []*cli.Command{
-			{
-				Name:      "unittest",
-				Usage:     "Run unittest for alerting and recording rules.",
-				UsageText: "More info in https://docs.victoriametrics.com/vmalert-tool.html#Unit-testing-for-rules",
-				Flags: []cli.Flag{
-					&cli.StringSliceFlag{
-						Name:     "files",
-						Usage:    "files to run unittest with. Supports an array of values separated by comma or specified via multiple flags.",
-						Required: true,
-					},
-					&cli.BoolFlag{
-						Name:     "disableAlertgroupLabel",
-						Usage:    "disable adding group's Name as label to generated alerts and time series.",
-						Required: false,
-					},
-				},
-				Action: func(c *cli.Context) error {
-					if failed := unittest.UnitTest(c.StringSlice("files"), c.Bool("disableAlertgroupLabel")); failed {
-						return fmt.Errorf("unittest failed")
-					}
-					return nil
-				},
-			},
-		},
-	}
-
-	err := app.Run(os.Args)
-	if err != nil {
-		log.Fatalln(err)
-	}
-	log.Printf("Total time: %v", time.Since(start))
-}
diff --git a/app/vmalert/README.md b/app/vmalert/README.md
index f389ce554..30416928b 100644
--- a/app/vmalert/README.md
+++ b/app/vmalert/README.md
@@ -119,6 +119,13 @@ name: <string>
 # `eval_offset` can't be bigger than `interval`.
 [ eval_offset: <duration> ]
 
+# Optional
+# Adjust the `time` parameter of group evaluation requests to compensate intentional query delay from the datasource.
+# By default, the value is inherited from the `-rule.evalDelay` cmd-line flag - see its description for details.
+# If group has `latency_offset` set in `params`, then it is recommended to set `eval_delay` equal to `latency_offset`.
+# See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155 and https://docs.victoriametrics.com/keyConcepts.html#query-latency.
+[ eval_delay: <duration> ]
+
 # Limit the number of alerts an alerting rule and series a recording
 # rule can produce. 0 is no limit.
 [ limit: <int> | default = 0 ]
@@ -794,9 +801,7 @@ Try the following recommendations to reduce the chance of hitting the data delay
 [time series resolution](https://docs.victoriametrics.com/keyConcepts.html#time-series-resolution). For example,
 if expression is `rate(my_metric[2m]) > 0` then ensure that `my_metric` resolution is at least `1m` or better `30s`. 
 If you use VictoriaMetrics as datasource, `[duration]` can be omitted and VictoriaMetrics will adjust it automatically.
-* If you know in advance, that data in datasource is delayed - try changing vmalerts `-datasource.lookback`
-command-line flag to add a time shift for evaluations. Or extend `[duration]` to tolerate the delay.
-For example, `max_over_time(errors_total[10m]) > 0` will be active even if there is no data in datasource for last `9m`.
+* Extend `[duration]` in expr to help tolerate the delay. For example, `max_over_time(errors_total[10m]) > 0` will be active even if there is no data in datasource for last `9m`.
 * If [time series resolution](https://docs.victoriametrics.com/keyConcepts.html#time-series-resolution)
 in datasource is inconsistent or `>=5min` - try changing vmalerts `-datasource.queryStep` command-line flag to specify
 how far search query can lookback for the recent datapoint. The recommendation is to have the step 
@@ -804,9 +809,13 @@ at least two times bigger than the resolution.
 
 > Please note, data delay is inevitable in distributed systems. And it is better to account for it instead of ignoring.
 
-By default, recently written samples to VictoriaMetrics aren't visible for queries for up to 30s
-(see `-search.latencyOffset` command-line flag at vmselect). Such delay is needed to eliminate risk of incomplete
-data on the moment of querying, since metrics collectors won't be able to deliver the data in time.
+By default, recently written samples to VictoriaMetrics [aren't visible for queries](https://docs.victoriametrics.com/keyConcepts.html#query-latency)
+for up to 30s (see `-search.latencyOffset` command-line flag at vmselect). Such delay is needed to eliminate risk of 
+incomplete data on the moment of querying, due to chance that metrics collectors won't be able to deliver that data in time.
+To compensate the latency in timestamps for produced evaluation results, `-rule.evalDelay` is also set to 30s by default.
+If you changed the `-search.latencyOffset` (cmd-line flag configured for VictoriaMetrics single-node or vmselect) value 
+or specified custom  `latency_offset` param via [Group](#groups) and observed a delay in timestamps for produced 
+evaluation results - try changing `-rule.evalDelay` equal to `-search.latencyOffset`.
 
 ### Alerts state
 
@@ -969,7 +978,7 @@ The shortlist of configuration flags is the following:
   -datasource.headers string
      Optional HTTP extraHeaders to send with each request to the corresponding -datasource.url. For example, -datasource.headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding -datasource.url. Multiple headers must be delimited by '^^': -datasource.headers='header1:value1^^header2:value2'
   -datasource.lookback duration
-     Lookback defines how far into the past to look when evaluating queries. For example, if the datasource.lookback=5m then param "time" with value now()-5m will be added to every query.
+     Will be deprecated soon, please adjust "-search.latencyOffset"  at datasource side or specify "latency_offset" in rule group's params. Lookback defines how far into the past to look when evaluating queries. For example, if the datasource.lookback=5m then param "time" with value now()-5m will be added to every query.
   -datasource.maxIdleConnections int
      Defines the number of idle (keep-alive connections) to each configured datasource. Consider setting this value equal to the value: groups_total * group.concurrency. Too low a value may result in a high number of sockets in TIME_WAIT state. (default 100)
   -datasource.oauth2.clientID string
@@ -1291,6 +1300,8 @@ The shortlist of configuration flags is the following:
      Limits the maximum duration for automatic alert expiration, which by default is 4 times evaluationInterval of the parent group.
   -rule.resendDelay duration
      Minimum amount of time to wait before resending an alert to notifier
+  -rule.evalDelay time
+     Adjustment of the time parameter for rule evaluation requests to compensate intentional data delay from the datasource.Normally, should be equal to `-search.latencyOffset` (cmd-line flag configured for VictoriaMetrics single-node or vmselect). (default 30s)
   -rule.templates array
      Path or glob pattern to location with go template definitions
      	for rules annotations templating. Flag can be specified multiple times.
diff --git a/app/vmalert/config/config.go b/app/vmalert/config/config.go
index 219dbe00b..da02b4de4 100644
--- a/app/vmalert/config/config.go
+++ b/app/vmalert/config/config.go
@@ -19,11 +19,14 @@ import (
 // Group contains list of Rules grouped into
 // entity with one name and evaluation interval
 type Group struct {
-	Type        Type `yaml:"type,omitempty"`
-	File        string
-	Name        string              `yaml:"name"`
-	Interval    *promutils.Duration `yaml:"interval,omitempty"`
-	EvalOffset  *promutils.Duration `yaml:"eval_offset,omitempty"`
+	Type       Type `yaml:"type,omitempty"`
+	File       string
+	Name       string              `yaml:"name"`
+	Interval   *promutils.Duration `yaml:"interval,omitempty"`
+	EvalOffset *promutils.Duration `yaml:"eval_offset,omitempty"`
+	// EvalDelay will adjust the `time` parameter of rule evaluation requests to compensate intentional query delay from datasource.
+	// see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155
+	EvalDelay   *promutils.Duration `yaml:"eval_delay,omitempty"`
 	Limit       int                 `yaml:"limit,omitempty"`
 	Rules       []Rule              `yaml:"rules"`
 	Concurrency int                 `yaml:"concurrency"`
diff --git a/app/vmalert/datasource/init.go b/app/vmalert/datasource/init.go
index 2c8922acc..b8bb570f8 100644
--- a/app/vmalert/datasource/init.go
+++ b/app/vmalert/datasource/init.go
@@ -43,7 +43,8 @@ var (
 	oauth2TokenURL         = flag.String("datasource.oauth2.tokenUrl", "", "Optional OAuth2 tokenURL to use for -datasource.url.")
 	oauth2Scopes           = flag.String("datasource.oauth2.scopes", "", "Optional OAuth2 scopes to use for -datasource.url. Scopes must be delimited by ';'")
 
-	lookBack  = flag.Duration("datasource.lookback", 0, `Lookback defines how far into the past to look when evaluating queries. For example, if the datasource.lookback=5m then param "time" with value now()-5m will be added to every query.`)
+	lookBack = flag.Duration("datasource.lookback", 0, `Will be deprecated soon, please adjust "-search.latencyOffset"  at datasource side or specify "latency_offset" in rule group's params.`+
+		`Lookback defines how far into the past to look when evaluating queries. For example, if the datasource.lookback=5m then param "time" with value now()-5m will be added to every query.`)
 	queryStep = flag.Duration("datasource.queryStep", 5*time.Minute, "How far a value can fallback to when evaluating queries. "+
 		"For example, if -datasource.queryStep=15s then param \"step\" with value \"15s\" will be added to every query. "+
 		"If set to 0, rule's evaluation interval will be used instead.")
@@ -83,7 +84,10 @@ func Init(extraParams url.Values) (QuerierBuilder, error) {
 		return nil, fmt.Errorf("datasource.url is empty")
 	}
 	if !*queryTimeAlignment {
-		logger.Warnf("flag `datasource.queryTimeAlignment` is deprecated and will be removed in next releases, please use `eval_alignment` in rule group instead")
+		logger.Warnf("flag `-datasource.queryTimeAlignment` is deprecated and will be removed in next releases. Please use `eval_alignment` in rule group instead.")
+	}
+	if *lookBack != 0 {
+		logger.Warnf("flag `-datasource.lookback` will be deprecated soon. Please use `-rule.evalDelay` command-line flag instead. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155 for details.")
 	}
 
 	tr, err := utils.Transport(*addr, *tlsCertFile, *tlsKeyFile, *tlsCAFile, *tlsServerName, *tlsInsecureSkipVerify)
diff --git a/app/vmalert/rule/group.go b/app/vmalert/rule/group.go
index 29f350373..9369b43f1 100644
--- a/app/vmalert/rule/group.go
+++ b/app/vmalert/rule/group.go
@@ -32,6 +32,8 @@ var (
 	resendDelay        = flag.Duration("rule.resendDelay", 0, "MiniMum amount of time to wait before resending an alert to notifier")
 	maxResolveDuration = flag.Duration("rule.maxResolveDuration", 0, "Limits the maxiMum duration for automatic alert expiration, "+
 		"which by default is 4 times evaluationInterval of the parent ")
+	evalDelay = flag.Duration("rule.evalDelay", 30*time.Second, "Adjustment of the `time` parameter for rule evaluation requests to compensate intentional data delay from the datasource."+
+		"Normally, should be equal to `-search.latencyOffset` (cmd-line flag configured for VictoriaMetrics single-node or vmselect).")
 	disableAlertGroupLabel = flag.Bool("disableAlertgroupLabel", false, "Whether to disable adding group's Name as label to generated alerts and time series.")
 	remoteReadLookBack     = flag.Duration("remoteRead.lookback", time.Hour, "Lookback defines how far to look into past for alerts timeseries."+
 		" For example, if lookback=1h then range from now() to now()-1h will be scanned.")
@@ -39,13 +41,16 @@ var (
 
 // Group is an entity for grouping rules
 type Group struct {
-	mu             sync.RWMutex
-	Name           string
-	File           string
-	Rules          []Rule
-	Type           config.Type
-	Interval       time.Duration
-	EvalOffset     *time.Duration
+	mu         sync.RWMutex
+	Name       string
+	File       string
+	Rules      []Rule
+	Type       config.Type
+	Interval   time.Duration
+	EvalOffset *time.Duration
+	// EvalDelay will adjust timestamp for rule evaluation requests to compensate intentional query delay from datasource.
+	// see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155
+	EvalDelay      *time.Duration
 	Limit          int
 	Concurrency    int
 	Checksum       string
@@ -139,6 +144,9 @@ func NewGroup(cfg config.Group, qb datasource.QuerierBuilder, defaultInterval ti
 	if cfg.EvalOffset != nil {
 		g.EvalOffset = &cfg.EvalOffset.D
 	}
+	if cfg.EvalDelay != nil {
+		g.EvalDelay = &cfg.EvalDelay.D
+	}
 	for _, h := range cfg.Headers {
 		g.Headers[h.Key] = h.Value
 	}
@@ -331,7 +339,7 @@ func (g *Group) Start(ctx context.Context, nts func() []notifier.Notifier, rw re
 		Rw:                       rw,
 		Notifiers:                nts,
 		notifierHeaders:          g.NotifierHeaders,
-		PreviouslySentSeriesToRW: make(map[uint64]map[string][]prompbmarshal.Label),
+		previouslySentSeriesToRW: make(map[uint64]map[string][]prompbmarshal.Label),
 	}
 
 	g.infof("started")
@@ -520,7 +528,7 @@ func (g *Group) ExecOnce(ctx context.Context, nts func() []notifier.Notifier, rw
 		Rw:                       rw,
 		Notifiers:                nts,
 		notifierHeaders:          g.NotifierHeaders,
-		PreviouslySentSeriesToRW: make(map[uint64]map[string][]prompbmarshal.Label),
+		previouslySentSeriesToRW: make(map[uint64]map[string][]prompbmarshal.Label),
 	}
 	if len(g.Rules) < 1 {
 		return nil
@@ -581,10 +589,15 @@ func (g *Group) adjustReqTimestamp(timestamp time.Time) time.Time {
 			// to 10:30, to the previous evaluationInterval.
 			return ts.Add(-g.Interval)
 		}
-		// EvalOffset shouldn't interfere with evalAlignment,
-		// so we return it immediately
+		// when `eval_offset` is using, ts shouldn't be effect by `eval_alignment` and `eval_delay`
+		// since it should be always aligned.
 		return ts
 	}
+
+	// account for delay before adjusting the timestamp.
+	// otherwise, the alignment may be off if `delay!=g.Interval`
+	timestamp = timestamp.Add(-g.getEvalDelay())
+
 	if g.evalAlignment == nil || *g.evalAlignment {
 		// align query time with interval to get similar result with grafana when plotting time series.
 		// see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5049
@@ -594,6 +607,13 @@ func (g *Group) adjustReqTimestamp(timestamp time.Time) time.Time {
 	return timestamp
 }
 
+func (g *Group) getEvalDelay() time.Duration {
+	if g.EvalDelay != nil {
+		return *g.EvalDelay
+	}
+	return *evalDelay
+}
+
 // executor contains group's notify and rw configs
 type executor struct {
 	Notifiers       func() []notifier.Notifier
@@ -602,11 +622,11 @@ type executor struct {
 	Rw remotewrite.RWClient
 
 	previouslySentSeriesToRWMu sync.Mutex
-	// PreviouslySentSeriesToRW stores series sent to RW on previous iteration
+	// previouslySentSeriesToRW stores series sent to RW on previous iteration
 	// map[ruleID]map[ruleLabels][]prompb.Label
 	// where `ruleID` is ID of the Rule within a Group
 	// and `ruleLabels` is []prompb.Label marshalled to a string
-	PreviouslySentSeriesToRW map[uint64]map[string][]prompbmarshal.Label
+	previouslySentSeriesToRW map[uint64]map[string][]prompbmarshal.Label
 }
 
 // execConcurrently executes rules concurrently if concurrency>1
@@ -723,7 +743,7 @@ func (e *executor) getStaleSeries(r Rule, tss []prompbmarshal.TimeSeries, timest
 	var staleS []prompbmarshal.TimeSeries
 	// check whether there are series which disappeared and need to be marked as stale
 	e.previouslySentSeriesToRWMu.Lock()
-	for key, labels := range e.PreviouslySentSeriesToRW[rID] {
+	for key, labels := range e.previouslySentSeriesToRW[rID] {
 		if _, ok := ruleLabels[key]; ok {
 			continue
 		}
@@ -732,7 +752,7 @@ func (e *executor) getStaleSeries(r Rule, tss []prompbmarshal.TimeSeries, timest
 		staleS = append(staleS, ss)
 	}
 	// set previous series to current
-	e.PreviouslySentSeriesToRW[rID] = ruleLabels
+	e.previouslySentSeriesToRW[rID] = ruleLabels
 	e.previouslySentSeriesToRWMu.Unlock()
 
 	return staleS
@@ -750,14 +770,14 @@ func (e *executor) purgeStaleSeries(activeRules []Rule) {
 
 	for _, rule := range activeRules {
 		id := rule.ID()
-		prev, ok := e.PreviouslySentSeriesToRW[id]
+		prev, ok := e.previouslySentSeriesToRW[id]
 		if ok {
 			// keep previous series for staleness detection
 			newPreviouslySentSeriesToRW[id] = prev
 		}
 	}
-	e.PreviouslySentSeriesToRW = nil
-	e.PreviouslySentSeriesToRW = newPreviouslySentSeriesToRW
+	e.previouslySentSeriesToRW = nil
+	e.previouslySentSeriesToRW = newPreviouslySentSeriesToRW
 
 	e.previouslySentSeriesToRWMu.Unlock()
 }
diff --git a/app/vmalert/web_types.go b/app/vmalert/web_types.go
index 7099c53e0..62d17f379 100644
--- a/app/vmalert/web_types.go
+++ b/app/vmalert/web_types.go
@@ -94,6 +94,10 @@ type apiGroup struct {
 	NotifierHeaders []string `json:"notifier_headers,omitempty"`
 	// Labels is a set of label value pairs, that will be added to every rule.
 	Labels map[string]string `json:"labels,omitempty"`
+	// EvalOffset Group will be evaluated at the exact time offset on the range of [0...evaluationInterval]
+	EvalOffset float64 `json:"eval_offset,omitempty"`
+	// EvalDelay will adjust the `time` parameter of rule evaluation requests to compensate intentional query delay from datasource.
+	EvalDelay float64 `json:"eval_delay,omitempty"`
 }
 
 // groupAlerts represents a group of alerts for WEB view
@@ -309,6 +313,12 @@ func groupToAPI(g *rule.Group) apiGroup {
 
 		Labels: g.Labels,
 	}
+	if g.EvalOffset != nil {
+		ag.EvalOffset = g.EvalOffset.Seconds()
+	}
+	if g.EvalDelay != nil {
+		ag.EvalDelay = g.EvalDelay.Seconds()
+	}
 	ag.Rules = make([]apiRule, 0)
 	for _, r := range g.Rules {
 		ag.Rules = append(ag.Rules, ruleToAPI(r))
diff --git a/deployment/docker/alerts-health.yml b/deployment/docker/alerts-health.yml
index eb1e2987c..caf830d93 100644
--- a/deployment/docker/alerts-health.yml
+++ b/deployment/docker/alerts-health.yml
@@ -35,12 +35,12 @@ groups:
           Consider to increase the limit as fast as possible."
 
       - alert: TooHighMemoryUsage
-        expr: (process_resident_memory_anon_bytes / vm_available_memory_bytes) > 0.8
+        expr: (min_over_time(process_resident_memory_anon_bytes[10m]) / vm_available_memory_bytes) > 0.8
         for: 5m
         labels:
           severity: critical
         annotations:
-          summary: "It is more than 80% of memory used by \"{{ $labels.job }}\"(\"{{ $labels.instance }}\") during the last 5m"
+          summary: "It is more than 80% of memory used by \"{{ $labels.job }}\"(\"{{ $labels.instance }}\")"
           description: "Too high memory usage may result into multiple issues such as OOMs or degraded performance.
            Consider to either increase available memory or decrease the load on the process."
 
diff --git a/deployment/docker/alerts-vmagent.yml b/deployment/docker/alerts-vmagent.yml
index b4f4ee9e0..6d7201fff 100644
--- a/deployment/docker/alerts-vmagent.yml
+++ b/deployment/docker/alerts-vmagent.yml
@@ -64,8 +64,11 @@ groups:
 
       - alert: RemoteWriteConnectionIsSaturated
         expr: |
-          sum(rate(vmagent_remotewrite_send_duration_seconds_total[5m])) by(job, instance, url) 
-          > 0.9 * max(vmagent_remotewrite_queues) by(job, instance, url)
+          (
+           sum(rate(vmagent_remotewrite_send_duration_seconds_total[5m])) by(job, instance, url) 
+           / 
+           max(vmagent_remotewrite_queues) by(job, instance, url)
+          ) > 0.9
         for: 15m
         labels:
           severity: warning
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index b6976d5bd..3b60a1143 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -29,6 +29,7 @@ The sandbox cluster installation is running under the constant load generated by
 ## tip
 
 **vmalert's cmd-line flag `datasource.queryTimeAlignment` was deprecated and will have no effect anymore. It will be completely removed in next releases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5049) and more detailed changes below.**
+**vmalert's cmd-line flag `datasource.lookback` will be deprecated soon. Please use `-rule.evalDelay` command-line flag instead. It will have no effect in next release and be removed in future releases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155).**
 
 * SECURITY: upgrade Go builder from Go1.21.1 to Go1.21.3. See [the list of issues addressed in Go1.21.2](https://github.com/golang/go/issues?q=milestone%3AGo1.21.2+label%3ACherryPickApproved) and [the list of issues addressed in Go1.21.3](https://github.com/golang/go/issues?q=milestone%3AGo1.21.3+label%3ACherryPickApproved).
 
@@ -37,10 +38,12 @@ The sandbox cluster installation is running under the constant load generated by
 * FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): add `eval_alignment` attribute for [Groups](https://docs.victoriametrics.com/vmalert.html#groups), it will align group query requests timestamp with interval like `datasource.queryTimeAlignment` did.
   This also means that `datasource.queryTimeAlignment` command-line flag becomes deprecated now and will have no effect if configured. If `datasource.queryTimeAlignment` was set to `false` before, then `eval_alignment` has to be set to `false` explicitly under group.
   See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5049).
+* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): add `-rule.evalDelay` flag and `eval_delay` attribute for [Groups](https://docs.victoriametrics.com/vmalert.html#groups). The new flag and param can be used to adjust the `time` parameter for rule evaluation requests to match [intentional query delay](https://docs.victoriametrics.com/keyConcepts.html#query-latency) from the datasource. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155).
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): support data ingestion from [NewRelic infrastructure agent](https://docs.newrelic.com/docs/infrastructure/install-infrastructure-agent). See [these docs](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-send-data-from-newrelic-agent), [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3520) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4712).
 * FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup.html): add `-filestream.disableFadvise` command-line flag, which can be used for disabling `fadvise` syscall during backup upload to the remote storage. By default `vmbackup` uses `fadvise` syscall in order to prevent from eviction of recently accessed data from the [OS page cache](https://en.wikipedia.org/wiki/Page_cache) when backing up large files. Sometimes the `fadvise` syscall may take significant amounts of CPU when the backup is performed with large value of `-concurrency` command-line flag on systems with big number of CPU cores. In this case it is better to manually disable `fadvise` syscall by passing `-filestream.disableFadvise` command-line flag to `vmbackup`. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5120) for details.
 * FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup.html): add `-deleteAllObjectVersions` command-line flag, which can be used for forcing removal of all object versions in remote object storage. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5121) issue and [these docs](https://docs.victoriametrics.com/vmbackup.html#permanent-deletion-of-objects-in-s3-compatible-storages) for the details.
 * FEATURE: [Alerting rules for VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#alerts): account for `vmauth` component for alerts `ServiceDown` and `TooManyRestarts`.
+* FEATURE: [Alerting rules for VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#alerts): make `TooHighMemoryUsage` more tolerable to spikes or near-the-threshold states. The change should reduce number of false positives.
 * FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add support for functions, labels, values in autocomplete. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3006).
 * FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): retain specified time interval when executing a query from `Top Queries`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5097).
 * FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): improve repeated VMUI page load times by enabling caching of static js and css at web browser side according to [these recommendations](https://developer.chrome.com/docs/lighthouse/performance/uses-long-cache-ttl/).
diff --git a/docs/vmalert.md b/docs/vmalert.md
index 5265d565e..7da9e888c 100644
--- a/docs/vmalert.md
+++ b/docs/vmalert.md
@@ -130,6 +130,13 @@ name: <string>
 # `eval_offset` can't be bigger than `interval`.
 [ eval_offset: <duration> ]
 
+# Optional
+# Adjust the `time` parameter of group evaluation requests to compensate intentional query delay from the datasource.
+# By default, the value is inherited from the `-rule.evalDelay` cmd-line flag - see its description for details.
+# If group has `latency_offset` set in `params`, then it is recommended to set `eval_delay` equal to `latency_offset`.
+# See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155 and https://docs.victoriametrics.com/keyConcepts.html#query-latency.
+[ eval_delay: <duration> ]
+
 # Limit the number of alerts an alerting rule and series a recording
 # rule can produce. 0 is no limit.
 [ limit: <int> | default = 0 ]
@@ -805,9 +812,7 @@ Try the following recommendations to reduce the chance of hitting the data delay
 [time series resolution](https://docs.victoriametrics.com/keyConcepts.html#time-series-resolution). For example,
 if expression is `rate(my_metric[2m]) > 0` then ensure that `my_metric` resolution is at least `1m` or better `30s`. 
 If you use VictoriaMetrics as datasource, `[duration]` can be omitted and VictoriaMetrics will adjust it automatically.
-* If you know in advance, that data in datasource is delayed - try changing vmalerts `-datasource.lookback`
-command-line flag to add a time shift for evaluations. Or extend `[duration]` to tolerate the delay.
-For example, `max_over_time(errors_total[10m]) > 0` will be active even if there is no data in datasource for last `9m`.
+* Extend `[duration]` in expr to help tolerate the delay. For example, `max_over_time(errors_total[10m]) > 0` will be active even if there is no data in datasource for last `9m`.
 * If [time series resolution](https://docs.victoriametrics.com/keyConcepts.html#time-series-resolution)
 in datasource is inconsistent or `>=5min` - try changing vmalerts `-datasource.queryStep` command-line flag to specify
 how far search query can lookback for the recent datapoint. The recommendation is to have the step 
@@ -815,9 +820,13 @@ at least two times bigger than the resolution.
 
 > Please note, data delay is inevitable in distributed systems. And it is better to account for it instead of ignoring.
 
-By default, recently written samples to VictoriaMetrics aren't visible for queries for up to 30s
-(see `-search.latencyOffset` command-line flag at vmselect). Such delay is needed to eliminate risk of incomplete
-data on the moment of querying, since metrics collectors won't be able to deliver the data in time.
+By default, recently written samples to VictoriaMetrics [aren't visible for queries](https://docs.victoriametrics.com/keyConcepts.html#query-latency)
+for up to 30s (see `-search.latencyOffset` command-line flag at vmselect). Such delay is needed to eliminate risk of 
+incomplete data on the moment of querying, due to chance that metrics collectors won't be able to deliver that data in time.
+To compensate the latency in timestamps for produced evaluation results, `-rule.evalDelay` is also set to 30s by default.
+If you changed the `-search.latencyOffset` (cmd-line flag configured for VictoriaMetrics single-node or vmselect) value 
+or specified custom  `latency_offset` param via [Group](#groups) and observed a delay in timestamps for produced 
+evaluation results - try changing `-rule.evalDelay` equal to `-search.latencyOffset`.
 
 ### Alerts state
 
@@ -980,7 +989,7 @@ The shortlist of configuration flags is the following:
   -datasource.headers string
      Optional HTTP extraHeaders to send with each request to the corresponding -datasource.url. For example, -datasource.headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding -datasource.url. Multiple headers must be delimited by '^^': -datasource.headers='header1:value1^^header2:value2'
   -datasource.lookback duration
-     Lookback defines how far into the past to look when evaluating queries. For example, if the datasource.lookback=5m then param "time" with value now()-5m will be added to every query.
+     Will be deprecated soon, please adjust "-search.latencyOffset"  at datasource side or specify "latency_offset" in rule group's params. Lookback defines how far into the past to look when evaluating queries. For example, if the datasource.lookback=5m then param "time" with value now()-5m will be added to every query.
   -datasource.maxIdleConnections int
      Defines the number of idle (keep-alive connections) to each configured datasource. Consider setting this value equal to the value: groups_total * group.concurrency. Too low a value may result in a high number of sockets in TIME_WAIT state. (default 100)
   -datasource.oauth2.clientID string
@@ -1302,6 +1311,8 @@ The shortlist of configuration flags is the following:
      Limits the maximum duration for automatic alert expiration, which by default is 4 times evaluationInterval of the parent group.
   -rule.resendDelay duration
      Minimum amount of time to wait before resending an alert to notifier
+  -rule.evalDelay time
+     Adjustment of the time parameter for rule evaluation requests to compensate intentional data delay from the datasource.Normally, should be equal to `-search.latencyOffset` (cmd-line flag configured for VictoriaMetrics single-node or vmselect). (default 30s)
   -rule.templates array
      Path or glob pattern to location with go template definitions
      	for rules annotations templating. Flag can be specified multiple times.


Test patch:
diff --git a/app/vmalert-tool/unittest/alerting.go b/app/vmalert-tool/unittest/alerting.go
deleted file mode 100644
index a9ffe8a79..000000000
--- a/app/vmalert-tool/unittest/alerting.go
+++ /dev/null
@@ -1,19 +0,0 @@
-package unittest
-
-import (
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promutils"
-)
-
-// alertTestCase holds alert_rule_test cases defined in test file
-type alertTestCase struct {
-	EvalTime  *promutils.Duration `yaml:"eval_time"`
-	GroupName string              `yaml:"groupname"`
-	Alertname string              `yaml:"alertname"`
-	ExpAlerts []expAlert          `yaml:"exp_alerts"`
-}
-
-// expAlert holds exp_alerts defined in test file
-type expAlert struct {
-	ExpLabels      map[string]string `yaml:"exp_labels"`
-	ExpAnnotations map[string]string `yaml:"exp_annotations"`
-}
diff --git a/app/vmalert-tool/unittest/input.go b/app/vmalert-tool/unittest/input.go
deleted file mode 100644
index af4254a80..000000000
--- a/app/vmalert-tool/unittest/input.go
+++ /dev/null
@@ -1,182 +0,0 @@
-package unittest
-
-import (
-	"bytes"
-	"fmt"
-	"io"
-	"net/http"
-	"regexp"
-	"strconv"
-	"strings"
-	"time"
-
-	testutil "github.com/VictoriaMetrics/VictoriaMetrics/app/victoria-metrics/test"
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/decimal"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promutils"
-	"github.com/VictoriaMetrics/metricsql"
-)
-
-// series holds input_series defined in the test file
-type series struct {
-	Series string `yaml:"series"`
-	Values string `yaml:"values"`
-}
-
-// sequenceValue is an omittable value in a sequence of time series values.
-type sequenceValue struct {
-	Value   float64
-	Omitted bool
-}
-
-func httpWrite(address string, r io.Reader) {
-	resp, err := http.Post(address, "", r)
-	if err != nil {
-		logger.Fatalf("failed to send to storage: %v", err)
-	}
-	resp.Body.Close()
-}
-
-// writeInputSeries send input series to vmstorage and flush them
-func writeInputSeries(input []series, interval *promutils.Duration, startStamp time.Time, dst string) error {
-	r := testutil.WriteRequest{}
-	for _, data := range input {
-		expr, err := metricsql.Parse(data.Series)
-		if err != nil {
-			return fmt.Errorf("failed to parse series %s: %v", data.Series, err)
-		}
-		promvals, err := parseInputValue(data.Values, true)
-		if err != nil {
-			return fmt.Errorf("failed to parse input series value %s: %v", data.Values, err)
-		}
-		metricExpr, ok := expr.(*metricsql.MetricExpr)
-		if !ok {
-			return fmt.Errorf("failed to parse series %s to metric expr: %v", data.Series, err)
-		}
-		samples := make([]testutil.Sample, 0, len(promvals))
-		ts := startStamp
-		for _, v := range promvals {
-			if !v.Omitted {
-				samples = append(samples, testutil.Sample{
-					Timestamp: ts.UnixMilli(),
-					Value:     v.Value,
-				})
-			}
-			ts = ts.Add(interval.Duration())
-		}
-		var ls []testutil.Label
-		for _, filter := range metricExpr.LabelFilterss[0] {
-			ls = append(ls, testutil.Label{Name: filter.Label, Value: filter.Value})
-		}
-		r.Timeseries = append(r.Timeseries, testutil.TimeSeries{Labels: ls, Samples: samples})
-	}
-
-	data, err := testutil.Compress(r)
-	if err != nil {
-		return fmt.Errorf("failed to compress data: %v", err)
-	}
-	// write input series to vm
-	httpWrite(dst, bytes.NewBuffer(data))
-	vmstorage.Storage.DebugFlush()
-	return nil
-}
-
-// parseInputValue support input like "1", "1+1x1 _ -4 3+20x1", see more examples in test.
-func parseInputValue(input string, origin bool) ([]sequenceValue, error) {
-	var res []sequenceValue
-	items := strings.Split(input, " ")
-	reg := regexp.MustCompile(`\D?\d*\D?`)
-	for _, item := range items {
-		if item == "stale" {
-			res = append(res, sequenceValue{Value: decimal.StaleNaN})
-			continue
-		}
-		vals := reg.FindAllString(item, -1)
-		switch len(vals) {
-		case 1:
-			if vals[0] == "_" {
-				res = append(res, sequenceValue{Omitted: true})
-				continue
-			}
-			v, err := strconv.ParseFloat(vals[0], 64)
-			if err != nil {
-				return nil, err
-			}
-			res = append(res, sequenceValue{Value: v})
-			continue
-		case 2:
-			p1 := vals[0][:len(vals[0])-1]
-			v2, err := strconv.ParseInt(vals[1], 10, 64)
-			if err != nil {
-				return nil, err
-			}
-			option := vals[0][len(vals[0])-1]
-			switch option {
-			case '+':
-				v1, err := strconv.ParseFloat(p1, 64)
-				if err != nil {
-					return nil, err
-				}
-				res = append(res, sequenceValue{Value: v1 + float64(v2)})
-			case 'x':
-				for i := int64(0); i <= v2; i++ {
-					if p1 == "_" {
-						if i == 0 {
-							i = 1
-						}
-						res = append(res, sequenceValue{Omitted: true})
-						continue
-					}
-					v1, err := strconv.ParseFloat(p1, 64)
-					if err != nil {
-						return nil, err
-					}
-					if !origin || v1 == 0 {
-						res = append(res, sequenceValue{Value: v1 * float64(i)})
-						continue
-					}
-					newVal := fmt.Sprintf("%s+0x%s", p1, vals[1])
-					newRes, err := parseInputValue(newVal, false)
-					if err != nil {
-						return nil, err
-					}
-					res = append(res, newRes...)
-					break
-				}
-
-			default:
-				return nil, fmt.Errorf("got invalid operation %b", option)
-			}
-		case 3:
-			r1, err := parseInputValue(fmt.Sprintf("%s%s", vals[1], vals[2]), false)
-			if err != nil {
-				return nil, err
-			}
-			p1 := vals[0][:len(vals[0])-1]
-			v1, err := strconv.ParseFloat(p1, 64)
-			if err != nil {
-				return nil, err
-			}
-			option := vals[0][len(vals[0])-1]
-			var isAdd bool
-			if option == '+' {
-				isAdd = true
-			}
-			for _, r := range r1 {
-				if isAdd {
-					res = append(res, sequenceValue{
-						Value: r.Value + v1,
-					})
-				} else {
-					res = append(res, sequenceValue{
-						Value: v1 - r.Value,
-					})
-				}
-			}
-		default:
-			return nil, fmt.Errorf("unsupported input %s", input)
-		}
-	}
-	return res, nil
-}
diff --git a/app/vmalert-tool/unittest/input_test.go b/app/vmalert-tool/unittest/input_test.go
deleted file mode 100644
index 5f74de98b..000000000
--- a/app/vmalert-tool/unittest/input_test.go
+++ /dev/null
@@ -1,93 +0,0 @@
-package unittest
-
-import (
-	"testing"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/decimal"
-)
-
-func TestParseInputValue(t *testing.T) {
-	testCases := []struct {
-		input  string
-		exp    []sequenceValue
-		failed bool
-	}{
-		{
-			"",
-			nil,
-			true,
-		},
-		{
-			"testfailed",
-			nil,
-			true,
-		},
-		// stale doesn't support operations
-		{
-			"stalex3",
-			nil,
-			true,
-		},
-		{
-			"-4",
-			[]sequenceValue{{Value: -4}},
-			false,
-		},
-		{
-			"_",
-			[]sequenceValue{{Omitted: true}},
-			false,
-		},
-		{
-			"stale",
-			[]sequenceValue{{Value: decimal.StaleNaN}},
-			false,
-		},
-		{
-			"-4x1",
-			[]sequenceValue{{Value: -4}, {Value: -4}},
-			false,
-		},
-		{
-			"_x1",
-			[]sequenceValue{{Omitted: true}},
-			false,
-		},
-		{
-			"1+1x4",
-			[]sequenceValue{{Value: 1}, {Value: 2}, {Value: 3}, {Value: 4}, {Value: 5}},
-			false,
-		},
-		{
-			"2-1x4",
-			[]sequenceValue{{Value: 2}, {Value: 1}, {Value: 0}, {Value: -1}, {Value: -2}},
-			false,
-		},
-		{
-			"1+1x1 _ -4 stale 3+20x1",
-			[]sequenceValue{{Value: 1}, {Value: 2}, {Omitted: true}, {Value: -4}, {Value: decimal.StaleNaN}, {Value: 3}, {Value: 23}},
-			false,
-		},
-	}
-
-	for _, tc := range testCases {
-		output, err := parseInputValue(tc.input, true)
-		if err != nil != tc.failed {
-			t.Fatalf("failed to parse %s, expect %t, got %t", tc.input, tc.failed, err != nil)
-		}
-		if len(tc.exp) != len(output) {
-			t.Fatalf("expect %v, got %v", tc.exp, output)
-		}
-		for i := 0; i < len(tc.exp); i++ {
-			if tc.exp[i].Omitted != output[i].Omitted {
-				t.Fatalf("expect %v, got %v", tc.exp, output)
-			}
-			if tc.exp[i].Value != output[i].Value {
-				if decimal.IsStaleNaN(tc.exp[i].Value) && decimal.IsStaleNaN(output[i].Value) {
-					continue
-				}
-				t.Fatalf("expect %v, got %v", tc.exp, output)
-			}
-		}
-	}
-}
diff --git a/app/vmalert-tool/unittest/recording.go b/app/vmalert-tool/unittest/recording.go
deleted file mode 100644
index a47d104d6..000000000
--- a/app/vmalert-tool/unittest/recording.go
+++ /dev/null
@@ -1,100 +0,0 @@
-package unittest
-
-import (
-	"context"
-	"fmt"
-	"net/url"
-	"reflect"
-	"sort"
-	"time"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/datasource"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promutils"
-	"github.com/VictoriaMetrics/metricsql"
-)
-
-// metricsqlTestCase holds metricsql_expr_test cases defined in test file
-type metricsqlTestCase struct {
-	Expr       string              `yaml:"expr"`
-	EvalTime   *promutils.Duration `yaml:"eval_time"`
-	ExpSamples []expSample         `yaml:"exp_samples"`
-}
-
-type expSample struct {
-	Labels string  `yaml:"labels"`
-	Value  float64 `yaml:"value"`
-}
-
-// checkMetricsqlCase will check metricsql_expr_test cases
-func checkMetricsqlCase(cases []metricsqlTestCase, q datasource.QuerierBuilder) (checkErrs []error) {
-	queries := q.BuildWithParams(datasource.QuerierParams{QueryParams: url.Values{"nocache": {"1"}, "latency_offset": {"1ms"}}, DataSourceType: "prometheus"})
-Outer:
-	for _, mt := range cases {
-		result, _, err := queries.Query(context.Background(), mt.Expr, durationToTime(mt.EvalTime))
-		if err != nil {
-			checkErrs = append(checkErrs, fmt.Errorf("    expr: %q, time: %s, err: %w", mt.Expr,
-				mt.EvalTime.Duration().String(), err))
-			continue
-		}
-		var gotSamples []parsedSample
-		for _, s := range result.Data {
-			sort.Slice(s.Labels, func(i, j int) bool {
-				return s.Labels[i].Name < s.Labels[j].Name
-			})
-			gotSamples = append(gotSamples, parsedSample{
-				Labels: s.Labels,
-				Value:  s.Values[0],
-			})
-		}
-		var expSamples []parsedSample
-		for _, s := range mt.ExpSamples {
-			expLb := datasource.Labels{}
-			if s.Labels != "" {
-				metricsqlExpr, err := metricsql.Parse(s.Labels)
-				if err != nil {
-					checkErrs = append(checkErrs, fmt.Errorf("\n    expr: %q, time: %s, err: %v", mt.Expr,
-						mt.EvalTime.Duration().String(), fmt.Errorf("failed to parse labels %q: %w", s.Labels, err)))
-					continue Outer
-				}
-				metricsqlMetricExpr, ok := metricsqlExpr.(*metricsql.MetricExpr)
-				if !ok {
-					checkErrs = append(checkErrs, fmt.Errorf("\n    expr: %q, time: %s, err: %v", mt.Expr,
-						mt.EvalTime.Duration().String(), fmt.Errorf("got unsupported metricsql type")))
-					continue Outer
-				}
-				for _, l := range metricsqlMetricExpr.LabelFilterss[0] {
-					expLb = append(expLb, datasource.Label{
-						Name:  l.Label,
-						Value: l.Value,
-					})
-				}
-			}
-			sort.Slice(expLb, func(i, j int) bool {
-				return expLb[i].Name < expLb[j].Name
-			})
-			expSamples = append(expSamples, parsedSample{
-				Labels: expLb,
-				Value:  s.Value,
-			})
-		}
-		sort.Slice(expSamples, func(i, j int) bool {
-			return datasource.LabelCompare(expSamples[i].Labels, expSamples[j].Labels) <= 0
-		})
-		sort.Slice(gotSamples, func(i, j int) bool {
-			return datasource.LabelCompare(gotSamples[i].Labels, gotSamples[j].Labels) <= 0
-		})
-		if !reflect.DeepEqual(expSamples, gotSamples) {
-			checkErrs = append(checkErrs, fmt.Errorf("\n    expr: %q, time: %s,\n        exp: %v\n        got: %v", mt.Expr,
-				mt.EvalTime.Duration().String(), parsedSamplesString(expSamples), parsedSamplesString(gotSamples)))
-		}
-
-	}
-	return
-}
-
-func durationToTime(pd *promutils.Duration) time.Time {
-	if pd == nil {
-		return time.Time{}
-	}
-	return time.UnixMilli(pd.Duration().Milliseconds())
-}
diff --git a/app/vmalert-tool/unittest/testdata/disable-group-label.yaml b/app/vmalert-tool/unittest/testdata/disable-group-label.yaml
deleted file mode 100644
index 6060710b6..000000000
--- a/app/vmalert-tool/unittest/testdata/disable-group-label.yaml
+++ /dev/null
@@ -1,43 +0,0 @@
-rule_files:
-  - rules.yaml
-
-evaluation_interval: 1m
-
-tests:
-  - interval: 1m
-    input_series:
-      - series: 'up{job="vmagent2", instance="localhost:9090"}'
-        values: "0+0x1440"
-
-    metricsql_expr_test:
-      - expr: suquery_interval_test
-        eval_time: 4m
-        exp_samples:
-          - labels: '{__name__="suquery_interval_test",datacenter="dc-123", instance="localhost:9090", job="vmagent2"}'
-            value: 1
-
-    alert_rule_test:
-      - eval_time: 2h
-        alertname: InstanceDown
-        exp_alerts:
-          - exp_labels:
-              job: vmagent2
-              severity: page
-              instance: localhost:9090
-              datacenter: dc-123
-            exp_annotations:
-              summary: "Instance localhost:9090 down"
-              description: "localhost:9090 of job vmagent2 has been down for more than 5 minutes."
-
-      - eval_time: 0
-        alertname: AlwaysFiring
-        exp_alerts:
-          - exp_labels:
-              datacenter: dc-123
-
-      - eval_time: 0
-        alertname: InstanceDown
-        exp_alerts: []
-
-    external_labels:
-      datacenter: dc-123
diff --git a/app/vmalert-tool/unittest/testdata/failed-test.yaml b/app/vmalert-tool/unittest/testdata/failed-test.yaml
deleted file mode 100644
index 0a1de6365..000000000
--- a/app/vmalert-tool/unittest/testdata/failed-test.yaml
+++ /dev/null
@@ -1,49 +0,0 @@
-rule_files:
-  - rules.yaml
-
-tests:
-  - interval: 1m
-    name: "Failing test"
-    input_series:
-      - series: test
-        values: "0"
-
-    metricsql_expr_test:
-      - expr: test
-        eval_time: 0m
-        exp_samples:
-          - value: 0
-            labels: test
-
-    # will failed cause there is no "Test" group and rule defined
-    alert_rule_test:
-      - eval_time: 0m
-        groupname: Test
-        alertname: Test
-        exp_alerts:
-          - exp_labels: {}
-
-  - interval: 1m
-    name: Failing alert test
-    input_series:
-      - series: 'up{job="test"}'
-        values: 0x10
-
-    alert_rule_test:
-      # will failed cause rule is firing
-      - eval_time: 5m
-        groupname: group1
-        alertname: InstanceDown
-        exp_alerts: []
-
-  - interval: 1m
-    name: Failing alert test with missing groupname
-    input_series:
-      - series: 'up{job="test"}'
-        values: 0x10
-
-    alert_rule_test:
-      # will failed cause missing groupname
-      - eval_time: 5m
-        alertname: AlwaysFiring
-        exp_alerts: []
diff --git a/app/vmalert-tool/unittest/testdata/long-period.yaml b/app/vmalert-tool/unittest/testdata/long-period.yaml
deleted file mode 100644
index 052470b69..000000000
--- a/app/vmalert-tool/unittest/testdata/long-period.yaml
+++ /dev/null
@@ -1,30 +0,0 @@
-# can be executed successfully but will take more than 1 minute
-# not included in unit test now
-evaluation_interval: 100d
-
-rule_files:
-  - rules.yaml
-
-tests:
-  - interval: 1d
-    input_series:
-      - series: test
-        # Max time in time.Duration is 106751d from 1970 (2^63/10^9), i.e. 2262.
-        # But VictoriaMetrics supports maxTimestamp value +2 days from now. see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/827.
-        # We input series to 2024-01-01T00:00:00 here.
-        values: "0+1x19723"
-
-    metricsql_expr_test:
-      - expr: timestamp(test)
-        eval_time: 0m
-        exp_samples:
-          - value: 0
-      - expr: test
-        eval_time: 100d
-        exp_samples:
-          - labels: test
-            value: 100
-      - expr: timestamp(test)
-        eval_time: 19000d
-        exp_samples:
-          - value: 1641600000 # 19000d -> seconds.
diff --git a/app/vmalert-tool/unittest/testdata/rules.yaml b/app/vmalert-tool/unittest/testdata/rules.yaml
deleted file mode 100644
index f7455c6d8..000000000
--- a/app/vmalert-tool/unittest/testdata/rules.yaml
+++ /dev/null
@@ -1,39 +0,0 @@
-groups:
-  - name: group1
-    rules:
-      - alert: InstanceDown
-        expr: up == 0
-        for: 5m
-        labels:
-          severity: page
-        annotations:
-          summary: "Instance {{ $labels.instance }} down"
-          description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes."
-      - alert: AlwaysFiring
-        expr: 1
-      - alert: SameAlertNameWithDifferentGroup
-        expr: absent(test)
-        for: 1m
-
-  - name: group2
-    rules:
-      - record: t1
-        expr: test
-      - record: job:test:count_over_time1m
-        expr: sum without(instance) (count_over_time(test[1m]))
-      - record: suquery_interval_test
-        expr: count_over_time(up[5m:])
-
-      - alert: SameAlertNameWithDifferentGroup
-        expr: absent(test)
-        for: 5m
-
-  - name: group3
-    rules:
-      - record: t2
-        expr: t1
-
-  - name: group4
-    rules:
-      - record: t3
-        expr: t1
diff --git a/app/vmalert-tool/unittest/testdata/test1.yaml b/app/vmalert-tool/unittest/testdata/test1.yaml
deleted file mode 100644
index d60e8587f..000000000
--- a/app/vmalert-tool/unittest/testdata/test1.yaml
+++ /dev/null
@@ -1,99 +0,0 @@
-rule_files:
-  - rules.yaml
-
-evaluation_interval: 1m
-group_eval_order: ["group4", "group2", "group3"]
-
-tests:
-  - interval: 1m
-    name: "basic test"
-    input_series:
-      - series: "test"
-        values: "_x5 1x5 _ stale"
-
-    alert_rule_test:
-      - eval_time: 1m
-        groupname: group1
-        alertname: SameAlertNameWithDifferentGroup
-        exp_alerts:
-          - {}
-      - eval_time: 1m
-        groupname: group2
-        alertname: SameAlertNameWithDifferentGroup
-        exp_alerts: []
-      - eval_time: 6m
-        groupname: group1
-        alertname: SameAlertNameWithDifferentGroup
-        exp_alerts: []
-
-    metricsql_expr_test:
-      - expr: test
-        eval_time: 11m
-        exp_samples:
-          - labels: '{__name__="test"}'
-            value: 1
-      - expr: test
-        eval_time: 12m
-        exp_samples: []
-
-  - interval: 1m
-    name: "basic test2"
-    input_series:
-      - series: 'up{job="vmagent1", instance="localhost:9090"}'
-        values: "0+0x1440"
-      - series: "test"
-        values: "0+1x1440"
-
-    metricsql_expr_test:
-      - expr: count(ALERTS) by (alertgroup, alertname, alertstate)
-        eval_time: 4m
-        exp_samples:
-          - labels: '{alertgroup="group1", alertname="AlwaysFiring", alertstate="firing"}'
-            value: 1
-          - labels: '{alertgroup="group1", alertname="InstanceDown", alertstate="pending"}'
-            value: 1
-      - expr: t1
-        eval_time: 4m
-        exp_samples:
-          - value: 4
-            labels: '{__name__="t1", datacenter="dc-123"}'
-      - expr: t2
-        eval_time: 4m
-        exp_samples:
-          - value: 4
-            labels: '{__name__="t2", datacenter="dc-123"}'
-      - expr: t3
-        eval_time: 4m
-        exp_samples:
-          # t3 is 3 instead of 4 cause it's rules3 is evaluated before rules1
-          - value: 3
-            labels: '{__name__="t3", datacenter="dc-123"}'
-
-    alert_rule_test:
-      - eval_time: 10m
-        groupname: group1
-        alertname: InstanceDown
-        exp_alerts:
-          - exp_labels:
-              job: vmagent1
-              severity: page
-              instance: localhost:9090
-              datacenter: dc-123
-            exp_annotations:
-              summary: "Instance localhost:9090 down"
-              description: "localhost:9090 of job vmagent1 has been down for more than 5 minutes."
-
-      - eval_time: 0
-        groupname: group1
-        alertname: AlwaysFiring
-        exp_alerts:
-          - exp_labels:
-              datacenter: dc-123
-
-      - eval_time: 0
-        groupname: alerts
-        alertname: InstanceDown
-        exp_alerts: []
-
-    external_labels:
-      datacenter: dc-123
diff --git a/app/vmalert-tool/unittest/testdata/test2.yaml b/app/vmalert-tool/unittest/testdata/test2.yaml
deleted file mode 100644
index 2d7825742..000000000
--- a/app/vmalert-tool/unittest/testdata/test2.yaml
+++ /dev/null
@@ -1,46 +0,0 @@
-rule_files:
-  - rules.yaml
-
-evaluation_interval: 1m
-
-tests:
-  - interval: 1m
-    input_series:
-      - series: 'up{job="vmagent2", instance="localhost:9090"}'
-        values: "0+0x1440"
-
-    metricsql_expr_test:
-      - expr: suquery_interval_test
-        eval_time: 4m
-        exp_samples:
-          - labels: '{__name__="suquery_interval_test",datacenter="dc-123", instance="localhost:9090", job="vmagent2"}'
-            value: 1
-
-    alert_rule_test:
-      - eval_time: 2h
-        groupname: group1
-        alertname: InstanceDown
-        exp_alerts:
-          - exp_labels:
-              job: vmagent2
-              severity: page
-              instance: localhost:9090
-              datacenter: dc-123
-            exp_annotations:
-              summary: "Instance localhost:9090 down"
-              description: "localhost:9090 of job vmagent2 has been down for more than 5 minutes."
-
-      - eval_time: 0
-        groupname: group1
-        alertname: AlwaysFiring
-        exp_alerts:
-          - exp_labels:
-              datacenter: dc-123
-
-      - eval_time: 0
-        groupname: group1
-        alertname: InstanceDown
-        exp_alerts: []
-
-    external_labels:
-      datacenter: dc-123
diff --git a/app/vmalert-tool/unittest/type.go b/app/vmalert-tool/unittest/type.go
deleted file mode 100644
index 8915a8f5c..000000000
--- a/app/vmalert-tool/unittest/type.go
+++ /dev/null
@@ -1,83 +0,0 @@
-package unittest
-
-import (
-	"fmt"
-	"strconv"
-	"strings"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/datasource"
-)
-
-// parsedSample is a sample with parsed Labels
-type parsedSample struct {
-	Labels datasource.Labels
-	Value  float64
-}
-
-func (ps *parsedSample) String() string {
-	return ps.Labels.String() + " " + strconv.FormatFloat(ps.Value, 'E', -1, 64)
-}
-
-func parsedSamplesString(pss []parsedSample) string {
-	if len(pss) == 0 {
-		return "nil"
-	}
-	s := pss[0].String()
-	for _, ps := range pss[1:] {
-		s += ", " + ps.String()
-	}
-	return s
-}
-
-// labelAndAnnotation holds labels and annotations
-type labelAndAnnotation struct {
-	Labels      datasource.Labels
-	Annotations datasource.Labels
-}
-
-func (la *labelAndAnnotation) String() string {
-	return "Labels:" + la.Labels.String() + "\nAnnotations:" + la.Annotations.String()
-}
-
-// labelsAndAnnotations is collection of LabelAndAnnotation
-type labelsAndAnnotations []labelAndAnnotation
-
-func (la labelsAndAnnotations) Len() int { return len(la) }
-
-func (la labelsAndAnnotations) Swap(i, j int) { la[i], la[j] = la[j], la[i] }
-func (la labelsAndAnnotations) Less(i, j int) bool {
-	diff := datasource.LabelCompare(la[i].Labels, la[j].Labels)
-	if diff != 0 {
-		return diff < 0
-	}
-	return datasource.LabelCompare(la[i].Annotations, la[j].Annotations) < 0
-}
-
-func (la labelsAndAnnotations) String() string {
-	if len(la) == 0 {
-		return "[]"
-	}
-	s := "[\n0:" + indentLines("\n"+la[0].String(), "  ")
-	for i, l := range la[1:] {
-		s += ",\n" + fmt.Sprintf("%d", i+1) + ":" + indentLines("\n"+l.String(), "  ")
-	}
-	s += "\n]"
-
-	return s
-}
-
-// indentLines prefixes each line in the supplied string with the given "indent" string.
-func indentLines(lines, indent string) string {
-	sb := strings.Builder{}
-	n := strings.Split(lines, "\n")
-	for i, l := range n {
-		if i > 0 {
-			sb.WriteString(indent)
-		}
-		sb.WriteString(l)
-		if i != len(n)-1 {
-			sb.WriteRune('\n')
-		}
-	}
-	return sb.String()
-}
diff --git a/app/vmalert-tool/unittest/unittest.go b/app/vmalert-tool/unittest/unittest.go
deleted file mode 100644
index 424a6132e..000000000
--- a/app/vmalert-tool/unittest/unittest.go
+++ /dev/null
@@ -1,443 +0,0 @@
-package unittest
-
-import (
-	"context"
-	"flag"
-	"fmt"
-	"net/http"
-	"os"
-	"path/filepath"
-	"reflect"
-	"sort"
-	"time"
-
-	"gopkg.in/yaml.v2"
-
-	vmalertconfig "github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/config"
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/datasource"
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/notifier"
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/remotewrite"
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/rule"
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/templates"
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vminsert"
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vminsert/promremotewrite"
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect"
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/prometheus"
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql"
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fs"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
-	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promutils"
-	"github.com/VictoriaMetrics/metrics"
-)
-
-var (
-	storagePath    string
-	httpListenAddr = ":8880"
-	// insert series from 1970-01-01T00:00:00
-	testStartTime = time.Unix(0, 0).UTC()
-
-	testPromWriteHTTPPath = "http://127.0.0.1" + httpListenAddr + "/api/v1/write"
-	testDataSourcePath    = "http://127.0.0.1" + httpListenAddr + "/prometheus"
-	testRemoteWritePath   = "http://127.0.0.1" + httpListenAddr
-	testHealthHTTPPath    = "http://127.0.0.1" + httpListenAddr + "/health"
-
-	disableAlertgroupLabel bool
-)
-
-const (
-	testStoragePath = "vmalert-unittest"
-	testLogLevel    = "ERROR"
-)
-
-// UnitTest runs unittest for files
-func UnitTest(files []string, disableGroupLabel bool) bool {
-	if err := templates.Load([]string{}, true); err != nil {
-		logger.Fatalf("failed to load template: %v", err)
-	}
-	storagePath = filepath.Join(os.TempDir(), testStoragePath)
-	processFlags()
-	vminsert.Init()
-	vmselect.Init()
-	// storagePath will be created again when closing vmselect, so remove it again.
-	defer fs.MustRemoveAll(storagePath)
-	defer vminsert.Stop()
-	defer vmselect.Stop()
-	disableAlertgroupLabel = disableGroupLabel
-	return rulesUnitTest(files)
-}
-
-func rulesUnitTest(files []string) bool {
-	var failed bool
-	for _, f := range files {
-		if err := ruleUnitTest(f); err != nil {
-			fmt.Println("  FAILED")
-			fmt.Printf("\nfailed to run unit test for file %q: \n%v", f, err)
-			failed = true
-		} else {
-			fmt.Println("  SUCCESS")
-		}
-	}
-	return failed
-}
-
-func ruleUnitTest(filename string) []error {
-	fmt.Println("\nUnit Testing: ", filename)
-	b, err := os.ReadFile(filename)
-	if err != nil {
-		return []error{fmt.Errorf("failed to read file: %w", err)}
-	}
-
-	var unitTestInp unitTestFile
-	if err := yaml.UnmarshalStrict(b, &unitTestInp); err != nil {
-		return []error{fmt.Errorf("failed to unmarshal file: %w", err)}
-	}
-	if err := resolveAndGlobFilepaths(filepath.Dir(filename), &unitTestInp); err != nil {
-		return []error{fmt.Errorf("failed to resolve path for `rule_files`: %w", err)}
-	}
-
-	if unitTestInp.EvaluationInterval.Duration() == 0 {
-		fmt.Println("evaluation_interval set to 1m by default")
-		unitTestInp.EvaluationInterval = &promutils.Duration{D: 1 * time.Minute}
-	}
-
-	groupOrderMap := make(map[string]int)
-	for i, gn := range unitTestInp.GroupEvalOrder {
-		if _, ok := groupOrderMap[gn]; ok {
-			return []error{fmt.Errorf("group name repeated in `group_eval_order`: %s", gn)}
-		}
-		groupOrderMap[gn] = i
-	}
-
-	testGroups, err := vmalertconfig.Parse(unitTestInp.RuleFiles, nil, true)
-	if err != nil {
-		return []error{fmt.Errorf("failed to parse `rule_files`: %w", err)}
-	}
-
-	var errs []error
-	for _, t := range unitTestInp.Tests {
-		if err := verifyTestGroup(t); err != nil {
-			errs = append(errs, err)
-			continue
-		}
-		testErrs := t.test(unitTestInp.EvaluationInterval.Duration(), groupOrderMap, testGroups)
-		errs = append(errs, testErrs...)
-	}
-
-	if len(errs) > 0 {
-		return errs
-	}
-	return nil
-}
-
-func verifyTestGroup(group testGroup) error {
-	var testGroupName string
-	if group.TestGroupName != "" {
-		testGroupName = fmt.Sprintf("testGroupName: %s\n", group.TestGroupName)
-	}
-	for _, at := range group.AlertRuleTests {
-		if at.Alertname == "" {
-			return fmt.Errorf("\n%s    missing required filed \"alertname\"", testGroupName)
-		}
-		if !disableAlertgroupLabel && at.GroupName == "" {
-			return fmt.Errorf("\n%s    missing required filed \"groupname\" when flag \"disableAlertGroupLabel\" is false", testGroupName)
-		}
-		if disableAlertgroupLabel && at.GroupName != "" {
-			return fmt.Errorf("\n%s    shouldn't set filed \"groupname\" when flag \"disableAlertGroupLabel\" is true", testGroupName)
-		}
-		if at.EvalTime == nil {
-			return fmt.Errorf("\n%s    missing required filed \"eval_time\"", testGroupName)
-		}
-	}
-	for _, et := range group.MetricsqlExprTests {
-		if et.Expr == "" {
-			return fmt.Errorf("\n%s    missing required filed \"expr\"", testGroupName)
-		}
-		if et.EvalTime == nil {
-			return fmt.Errorf("\n%s    missing required filed \"eval_time\"", testGroupName)
-		}
-	}
-	return nil
-}
-
-func processFlags() {
-	flag.Parse()
-	for _, fv := range []struct {
-		flag  string
-		value string
-	}{
-		{flag: "storageDataPath", value: storagePath},
-		{flag: "loggerLevel", value: testLogLevel},
-		{flag: "search.disableCache", value: "true"},
-		// set storage retention time to 100 years, allow to store series from 1970-01-01T00:00:00.
-		{flag: "retentionPeriod", value: "100y"},
-		{flag: "datasource.url", value: testDataSourcePath},
-		{flag: "remoteWrite.url", value: testRemoteWritePath},
-	} {
-		// panics if flag doesn't exist
-		if err := flag.Lookup(fv.flag).Value.Set(fv.value); err != nil {
-			logger.Fatalf("unable to set %q with value %q, err: %v", fv.flag, fv.value, err)
-		}
-	}
-}
-
-func setUp() {
-	vmstorage.Init(promql.ResetRollupResultCacheIfNeeded)
-	go httpserver.Serve(httpListenAddr, false, func(w http.ResponseWriter, r *http.Request) bool {
-		switch r.URL.Path {
-		case "/prometheus/api/v1/query":
-			if err := prometheus.QueryHandler(nil, time.Now(), w, r); err != nil {
-				httpserver.Errorf(w, r, "%s", err)
-			}
-			return true
-		case "/prometheus/api/v1/write", "/api/v1/write":
-			if err := promremotewrite.InsertHandler(r); err != nil {
-				httpserver.Errorf(w, r, "%s", err)
-			}
-			return true
-		default:
-		}
-		return false
-	})
-	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
-	defer cancel()
-	readyCheckFunc := func() bool {
-		resp, err := http.Get(testHealthHTTPPath)
-		if err != nil {
-			return false
-		}
-		_ = resp.Body.Close()
-		return resp.StatusCode == 200
-	}
-checkCheck:
-	for {
-		select {
-		case <-ctx.Done():
-			logger.Fatalf("http server can't be ready in 30s")
-		default:
-			if readyCheckFunc() {
-				break checkCheck
-			}
-			time.Sleep(3 * time.Second)
-		}
-	}
-}
-
-func tearDown() {
-	if err := httpserver.Stop(httpListenAddr); err != nil {
-		logger.Errorf("cannot stop the webservice: %s", err)
-	}
-	vmstorage.Stop()
-	metrics.UnregisterAllMetrics()
-	fs.MustRemoveAll(storagePath)
-}
-
-// resolveAndGlobFilepaths joins all relative paths in a configuration
-// with a given base directory and replaces all globs with matching files.
-func resolveAndGlobFilepaths(baseDir string, utf *unitTestFile) error {
-	for i, rf := range utf.RuleFiles {
-		if rf != "" && !filepath.IsAbs(rf) {
-			utf.RuleFiles[i] = filepath.Join(baseDir, rf)
-		}
-	}
-
-	var globbedFiles []string
-	for _, rf := range utf.RuleFiles {
-		m, err := filepath.Glob(rf)
-		if err != nil {
-			return err
-		}
-		if len(m) == 0 {
-			fmt.Fprintln(os.Stderr, "  WARNING: no file match pattern", rf)
-		}
-		globbedFiles = append(globbedFiles, m...)
-	}
-	utf.RuleFiles = globbedFiles
-	return nil
-}
-
-func (tg *testGroup) test(evalInterval time.Duration, groupOrderMap map[string]int, testGroups []vmalertconfig.Group) (checkErrs []error) {
-	// set up vmstorage and http server for ingest and read queries
-	setUp()
-	// tear down vmstorage and clean the data dir
-	defer tearDown()
-
-	err := writeInputSeries(tg.InputSeries, tg.Interval, testStartTime, testPromWriteHTTPPath)
-	if err != nil {
-		return []error{err}
-	}
-
-	q, err := datasource.Init(nil)
-	if err != nil {
-		return []error{fmt.Errorf("failed to init datasource: %v", err)}
-	}
-	rw, err := remotewrite.NewDebugClient()
-	if err != nil {
-		return []error{fmt.Errorf("failed to init wr: %v", err)}
-	}
-
-	alertEvalTimesMap := map[time.Duration]struct{}{}
-	alertExpResultMap := map[time.Duration]map[string]map[string][]expAlert{}
-	for _, at := range tg.AlertRuleTests {
-		et := at.EvalTime.Duration()
-		alertEvalTimesMap[et] = struct{}{}
-		if _, ok := alertExpResultMap[et]; !ok {
-			alertExpResultMap[et] = make(map[string]map[string][]expAlert)
-		}
-		if _, ok := alertExpResultMap[et][at.GroupName]; !ok {
-			alertExpResultMap[et][at.GroupName] = make(map[string][]expAlert)
-		}
-		alertExpResultMap[et][at.GroupName][at.Alertname] = at.ExpAlerts
-	}
-	alertEvalTimes := make([]time.Duration, 0, len(alertEvalTimesMap))
-	for k := range alertEvalTimesMap {
-		alertEvalTimes = append(alertEvalTimes, k)
-	}
-	sort.Slice(alertEvalTimes, func(i, j int) bool {
-		return alertEvalTimes[i] < alertEvalTimes[j]
-	})
-
-	// sort group eval order according to the given "group_eval_order".
-	sort.Slice(testGroups, func(i, j int) bool {
-		return groupOrderMap[testGroups[i].Name] < groupOrderMap[testGroups[j].Name]
-	})
-
-	// create groups with given rule
-	var groups []*rule.Group
-	for _, group := range testGroups {
-		ng := rule.NewGroup(group, q, time.Minute, tg.ExternalLabels)
-		groups = append(groups, ng)
-	}
-
-	evalIndex := 0
-	maxEvalTime := testStartTime.Add(tg.maxEvalTime())
-	for ts := testStartTime; ts.Before(maxEvalTime) || ts.Equal(maxEvalTime); ts = ts.Add(evalInterval) {
-		for _, g := range groups {
-			errs := g.ExecOnce(context.Background(), func() []notifier.Notifier { return nil }, rw, ts)
-			for err := range errs {
-				if err != nil {
-					checkErrs = append(checkErrs, fmt.Errorf("\nfailed to exec group: %q, time: %s, err: %w", g.Name,
-						ts, err))
-				}
-			}
-			// flush series after each group evaluation
-			vmstorage.Storage.DebugFlush()
-		}
-
-		// check alert_rule_test case at every eval time
-		for evalIndex < len(alertEvalTimes) {
-			if ts.Sub(testStartTime) > alertEvalTimes[evalIndex] ||
-				alertEvalTimes[evalIndex] >= ts.Add(evalInterval).Sub(testStartTime) {
-				break
-			}
-			gotAlertsMap := map[string]map[string]labelsAndAnnotations{}
-			for _, g := range groups {
-				if disableAlertgroupLabel {
-					g.Name = ""
-				}
-				if _, ok := alertExpResultMap[time.Duration(ts.UnixNano())][g.Name]; !ok {
-					continue
-				}
-				if _, ok := gotAlertsMap[g.Name]; !ok {
-					gotAlertsMap[g.Name] = make(map[string]labelsAndAnnotations)
-				}
-				for _, r := range g.Rules {
-					ar, isAlertRule := r.(*rule.AlertingRule)
-					if !isAlertRule {
-						continue
-					}
-					if _, ok := alertExpResultMap[time.Duration(ts.UnixNano())][g.Name][ar.Name]; ok {
-						for _, got := range ar.GetAlerts() {
-							if got.State != notifier.StateFiring {
-								continue
-							}
-							if disableAlertgroupLabel {
-								delete(got.Labels, "alertgroup")
-							}
-							laa := labelAndAnnotation{
-								Labels:      datasource.ConvertToLabels(got.Labels),
-								Annotations: datasource.ConvertToLabels(got.Annotations),
-							}
-							gotAlertsMap[g.Name][ar.Name] = append(gotAlertsMap[g.Name][ar.Name], laa)
-						}
-					}
-
-				}
-			}
-			for groupname, gres := range alertExpResultMap[alertEvalTimes[evalIndex]] {
-				for alertname, res := range gres {
-					var expAlerts labelsAndAnnotations
-					for _, expAlert := range res {
-						if expAlert.ExpLabels == nil {
-							expAlert.ExpLabels = make(map[string]string)
-						}
-						// alertGroupNameLabel is added as additional labels when `disableAlertGroupLabel` is false
-						if !disableAlertgroupLabel {
-							expAlert.ExpLabels["alertgroup"] = groupname
-						}
-						// alertNameLabel is added as additional labels in vmalert.
-						expAlert.ExpLabels["alertname"] = alertname
-						expAlerts = append(expAlerts, labelAndAnnotation{
-							Labels:      datasource.ConvertToLabels(expAlert.ExpLabels),
-							Annotations: datasource.ConvertToLabels(expAlert.ExpAnnotations),
-						})
-					}
-					sort.Sort(expAlerts)
-
-					gotAlerts := gotAlertsMap[groupname][alertname]
-					sort.Sort(gotAlerts)
-					if !reflect.DeepEqual(expAlerts, gotAlerts) {
-						var testGroupName string
-						if tg.TestGroupName != "" {
-							testGroupName = fmt.Sprintf("testGroupName: %s,\n", tg.TestGroupName)
-						}
-						expString := indentLines(expAlerts.String(), "            ")
-						gotString := indentLines(gotAlerts.String(), "            ")
-						checkErrs = append(checkErrs, fmt.Errorf("\n%s    groupname: %s, alertname: %s, time: %s, \n        exp:%v, \n        got:%v ",
-							testGroupName, groupname, alertname, alertEvalTimes[evalIndex].String(), expString, gotString))
-					}
-				}
-			}
-			evalIndex++
-		}
-
-	}
-
-	checkErrs = append(checkErrs, checkMetricsqlCase(tg.MetricsqlExprTests, q)...)
-	return checkErrs
-}
-
-// unitTestFile holds the contents of a single unit test file
-type unitTestFile struct {
-	RuleFiles          []string            `yaml:"rule_files"`
-	EvaluationInterval *promutils.Duration `yaml:"evaluation_interval"`
-	GroupEvalOrder     []string            `yaml:"group_eval_order"`
-	Tests              []testGroup         `yaml:"tests"`
-}
-
-// testGroup is a group of input series and test cases associated with it
-type testGroup struct {
-	Interval           *promutils.Duration `yaml:"interval"`
-	InputSeries        []series            `yaml:"input_series"`
-	AlertRuleTests     []alertTestCase     `yaml:"alert_rule_test"`
-	MetricsqlExprTests []metricsqlTestCase `yaml:"metricsql_expr_test"`
-	ExternalLabels     map[string]string   `yaml:"external_labels"`
-	TestGroupName      string              `yaml:"name"`
-}
-
-// maxEvalTime returns the max eval time among all alert_rule_test and metricsql_expr_test
-func (tg *testGroup) maxEvalTime() time.Duration {
-	var maxd time.Duration
-	for _, alert := range tg.AlertRuleTests {
-		if alert.EvalTime.Duration() > maxd {
-			maxd = alert.EvalTime.Duration()
-		}
-	}
-	for _, met := range tg.MetricsqlExprTests {
-		if met.EvalTime.Duration() > maxd {
-			maxd = met.EvalTime.Duration()
-		}
-	}
-	return maxd
-}
diff --git a/app/vmalert-tool/unittest/unittest_test.go b/app/vmalert-tool/unittest/unittest_test.go
deleted file mode 100644
index c2c014d6e..000000000
--- a/app/vmalert-tool/unittest/unittest_test.go
+++ /dev/null
@@ -1,47 +0,0 @@
-package unittest
-
-import (
-	"os"
-	"testing"
-
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/templates"
-)
-
-func TestMain(m *testing.M) {
-	if err := templates.Load([]string{}, true); err != nil {
-		os.Exit(1)
-	}
-	os.Exit(m.Run())
-}
-
-func TestUnitRule(t *testing.T) {
-	testCases := []struct {
-		name              string
-		disableGroupLabel bool
-		files             []string
-		failed            bool
-	}{
-		{
-			name:   "run multi files",
-			files:  []string{"./testdata/test1.yaml", "./testdata/test2.yaml"},
-			failed: false,
-		},
-		{
-			name:              "disable group label",
-			disableGroupLabel: true,
-			files:             []string{"./testdata/disable-group-label.yaml"},
-			failed:            false,
-		},
-		{
-			name:   "failing test",
-			files:  []string{"./testdata/failed-test.yaml"},
-			failed: true,
-		},
-	}
-	for _, tc := range testCases {
-		fail := UnitTest(tc.files, tc.disableGroupLabel)
-		if fail != tc.failed {
-			t.Fatalf("failed to test %s, expect %t, got %t", tc.name, tc.failed, fail)
-		}
-	}
-}
diff --git a/app/vmalert/config/testdata/rules/rules3-good.rules b/app/vmalert/config/testdata/rules/rules3-good.rules
index eb2a78d4f..ea65a9486 100644
--- a/app/vmalert/config/testdata/rules/rules3-good.rules
+++ b/app/vmalert/config/testdata/rules/rules3-good.rules
@@ -15,6 +15,7 @@ groups:
     interval: 2s
     concurrency: 2
     type: prometheus
+    eval_delay: 30s
     rules:
       - alert: Conns
         expr: sum(vm_tcplistener_conns) by (instance) > 1
diff --git a/app/vmalert/rule/group_test.go b/app/vmalert/rule/group_test.go
index f0342d120..65bdd061d 100644
--- a/app/vmalert/rule/group_test.go
+++ b/app/vmalert/rule/group_test.go
@@ -321,7 +321,7 @@ func TestResolveDuration(t *testing.T) {
 func TestGetStaleSeries(t *testing.T) {
 	ts := time.Now()
 	e := &executor{
-		PreviouslySentSeriesToRW: make(map[uint64]map[string][]prompbmarshal.Label),
+		previouslySentSeriesToRW: make(map[uint64]map[string][]prompbmarshal.Label),
 	}
 	f := func(r Rule, labels, expLabels [][]prompbmarshal.Label) {
 		t.Helper()
@@ -414,7 +414,7 @@ func TestPurgeStaleSeries(t *testing.T) {
 	f := func(curRules, newRules, expStaleRules []Rule) {
 		t.Helper()
 		e := &executor{
-			PreviouslySentSeriesToRW: make(map[uint64]map[string][]prompbmarshal.Label),
+			previouslySentSeriesToRW: make(map[uint64]map[string][]prompbmarshal.Label),
 		}
 		// seed executor with series for
 		// current rules
@@ -424,13 +424,13 @@ func TestPurgeStaleSeries(t *testing.T) {
 
 		e.purgeStaleSeries(newRules)
 
-		if len(e.PreviouslySentSeriesToRW) != len(expStaleRules) {
+		if len(e.previouslySentSeriesToRW) != len(expStaleRules) {
 			t.Fatalf("expected to get %d stale series, got %d",
-				len(expStaleRules), len(e.PreviouslySentSeriesToRW))
+				len(expStaleRules), len(e.previouslySentSeriesToRW))
 		}
 
 		for _, exp := range expStaleRules {
-			if _, ok := e.PreviouslySentSeriesToRW[exp.ID()]; !ok {
+			if _, ok := e.previouslySentSeriesToRW[exp.ID()]; !ok {
 				t.Fatalf("expected to have rule %d; got nil instead", exp.ID())
 			}
 		}
@@ -515,7 +515,7 @@ func TestFaultyRW(t *testing.T) {
 
 	e := &executor{
 		Rw:                       &remotewrite.Client{},
-		PreviouslySentSeriesToRW: make(map[uint64]map[string][]prompbmarshal.Label),
+		previouslySentSeriesToRW: make(map[uint64]map[string][]prompbmarshal.Label),
 	}
 
 	err := e.exec(context.Background(), r, time.Now(), 0, 10)
@@ -628,6 +628,7 @@ func TestGroupStartDelay(t *testing.T) {
 
 func TestGetPrometheusReqTimestamp(t *testing.T) {
 	offset := 30 * time.Minute
+	evalDelay := 1 * time.Minute
 	disableAlign := false
 	testCases := []struct {
 		name            string
@@ -635,24 +636,24 @@ func TestGetPrometheusReqTimestamp(t *testing.T) {
 		originTS, expTS string
 	}{
 		{
-			"with query align",
+			"with query align + default evalDelay",
 			&Group{
 				Interval: time.Hour,
 			},
 			"2023-08-28T11:11:00+00:00",
-			"2023-08-28T11:00:00+00:00",
+			"2023-08-28T10:59:30+00:00",
 		},
 		{
-			"without query align",
+			"without query align + default evalDelay",
 			&Group{
 				Interval:      time.Hour,
 				evalAlignment: &disableAlign,
 			},
 			"2023-08-28T11:11:00+00:00",
-			"2023-08-28T11:11:00+00:00",
+			"2023-08-28T11:10:30+00:00",
 		},
 		{
-			"with eval_offset, find previous offset point",
+			"with eval_offset, find previous offset point + default evalDelay",
 			&Group{
 				EvalOffset: &offset,
 				Interval:   time.Hour,
@@ -661,7 +662,7 @@ func TestGetPrometheusReqTimestamp(t *testing.T) {
 			"2023-08-28T10:30:00+00:00",
 		},
 		{
-			"with eval_offset",
+			"with eval_offset + default evalDelay",
 			&Group{
 				EvalOffset: &offset,
 				Interval:   time.Hour,
@@ -669,6 +670,25 @@ func TestGetPrometheusReqTimestamp(t *testing.T) {
 			"2023-08-28T11:41:00+00:00",
 			"2023-08-28T11:30:00+00:00",
 		},
+		{
+			"with eval_delay",
+			&Group{
+				EvalDelay: &evalDelay,
+				Interval:  time.Hour,
+			},
+			"2023-08-28T11:41:00+00:00",
+			"2023-08-28T10:59:00+00:00",
+		},
+		{
+			"disable alignment with eval_delay",
+			&Group{
+				EvalDelay:     &evalDelay,
+				Interval:      time.Hour,
+				evalAlignment: &disableAlign,
+			},
+			"2023-08-28T11:41:00+00:00",
+			"2023-08-28T11:40:00+00:00",
+		},
 	}
 	for _, tc := range testCases {
 		originT, _ := time.Parse(time.RFC3339, tc.originTS)