Repository: VictoriaMetrics/VictoriaMetrics
Pull request #2544 (2022-05-06 02:51:59)

app/vmctl: add flag to skip adding the InfluxDB 'db' label
Make it possible to migrate timeseries without changing labels
at all, including not adding the now optional 'db' label.

codecov[bot]: # [Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2544?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) Report
> Merging [#2544](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2544?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) (f38a74f) into [master](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/commit/ca4ca4630bef56d10af1b3567c16900164988fc6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) (ca4ca46) will **increase** coverage by `0.01%`.
> The diff coverage is `0.00%`.

```diff
@@            Coverage Diff             @@
##           master    #2544      +/-   ##
==========================================
+ Coverage   59.45%   59.46%   +0.01%     
==========================================
  Files         254      254              
  Lines       42745    42747       +2     
==========================================
+ Hits        25414    25421       +7     
+ Misses      15647    15639       -8     
- Partials     1684     1687       +3     
```


| [Impacted Files](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2544?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) | Coverage Δ | |
|---|---|---|
| [app/vmctl/flags.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2544/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtY3RsL2ZsYWdzLmdv) | `0.00% <ø> (ø)` | |
| [app/vmctl/influx.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2544/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtY3RsL2luZmx1eC5nbw==) | `0.00% <0.00%> (ø)` | |
| [app/vmctl/main.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2544/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtY3RsL21haW4uZ28=) | `0.00% <0.00%> (ø)` | |
| [lib/storage/storage.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2544/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL3N0b3JhZ2Uvc3RvcmFnZS5nbw==) | `67.13% <0.00%> (-0.29%)` | :arrow_down: |
| [app/vmselect/promql/aggr\_incremental.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2544/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3Ztc2VsZWN0L3Byb21xbC9hZ2dyX2luY3JlbWVudGFsLmdv) | `84.03% <0.00%> (ø)` | |
| [app/vmalert/remotewrite/remotewrite.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2544/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvcmVtb3Rld3JpdGUvcmVtb3Rld3JpdGUuZ28=) | `62.75% <0.00%> (+0.68%)` | :arrow_up: |
| [lib/mergeset/table.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2544/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL21lcmdlc2V0L3RhYmxlLmdv) | `67.01% <0.00%> (+0.95%)` | :arrow_up: |
| [lib/bloomfilter/filter.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2544/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL2Jsb29tZmlsdGVyL2ZpbHRlci5nbw==) | `100.00% <0.00%> (+2.00%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2544?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2544?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics). Last update [ca4ca46...f38a74f](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2544?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).


Patch:
diff --git a/app/vmagent/remotewrite/client.go b/app/vmagent/remotewrite/client.go
index 373b7157d..65bc5d12b 100644
--- a/app/vmagent/remotewrite/client.go
+++ b/app/vmagent/remotewrite/client.go
@@ -151,7 +151,7 @@ func (c *client) init(argIdx, concurrency int, sanitizedURL string) {
 		logger.Infof("applying %d bytes per second rate limit for -remoteWrite.url=%q", bytesPerSec, sanitizedURL)
 		c.rl.perSecondLimit = int64(bytesPerSec)
 	}
-	c.rl.limitReached = metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_remote_write_rate_limit_reached_total{url=%q}`, c.sanitizedURL))
+	c.rl.limitReached = metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_remotewrite_rate_limit_reached_total{url=%q}`, c.sanitizedURL))
 
 	c.bytesSent = metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_remotewrite_bytes_sent_total{url=%q}`, c.sanitizedURL))
 	c.blocksSent = metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_remotewrite_blocks_sent_total{url=%q}`, c.sanitizedURL))
diff --git a/app/vmagent/remotewrite/remotewrite.go b/app/vmagent/remotewrite/remotewrite.go
index 981a9a3cb..e810bd2ed 100644
--- a/app/vmagent/remotewrite/remotewrite.go
+++ b/app/vmagent/remotewrite/remotewrite.go
@@ -274,6 +274,8 @@ func PushWithAuthToken(at *auth.Token, wr *prompbmarshal.WriteRequest) {
 		rctx = getRelabelCtx()
 	}
 	tss := wr.Timeseries
+	rowsCount := getRowsCount(tss)
+	globalRowsPushedBeforeRelabel.Add(rowsCount)
 	maxSamplesPerBlock := *maxRowsPerBlock
 	// Allow up to 10x of labels per each block on average.
 	maxLabelsPerBlock := 10 * maxSamplesPerBlock
@@ -298,9 +300,10 @@ func PushWithAuthToken(at *auth.Token, wr *prompbmarshal.WriteRequest) {
 			tss = nil
 		}
 		if rctx != nil {
-			tssBlockLen := len(tssBlock)
+			rowsCountBeforeRelabel := getRowsCount(tssBlock)
 			tssBlock = rctx.applyRelabeling(tssBlock, labelsGlobal, pcsGlobal)
-			globalRelabelMetricsDropped.Add(tssBlockLen - len(tssBlock))
+			rowsCountAfterRelabel := getRowsCount(tssBlock)
+			rowsDroppedByGlobalRelabel.Add(rowsCountBeforeRelabel - rowsCountAfterRelabel)
 		}
 		sortLabelsIfNeeded(tssBlock)
 		tssBlock = limitSeriesCardinality(tssBlock)
@@ -414,7 +417,10 @@ func labelsToString(labels []prompbmarshal.Label) string {
 	return string(b)
 }
 
-var globalRelabelMetricsDropped = metrics.NewCounter("vmagent_remotewrite_global_relabel_metrics_dropped_total")
+var (
+	globalRowsPushedBeforeRelabel = metrics.NewCounter("vmagent_remotewrite_global_rows_pushed_before_relabel_total")
+	rowsDroppedByGlobalRelabel    = metrics.NewCounter("vmagent_remotewrite_global_relabel_metrics_dropped_total")
+)
 
 type remoteWriteCtx struct {
 	idx        int
@@ -423,7 +429,8 @@ type remoteWriteCtx struct {
 	pss        []*pendingSeries
 	pssNextIdx uint64
 
-	relabelMetricsDropped *metrics.Counter
+	rowsPushedAfterRelabel *metrics.Counter
+	rowsDroppedByRelabel   *metrics.Counter
 }
 
 func newRemoteWriteCtx(argIdx int, at *auth.Token, remoteWriteURL *url.URL, maxInmemoryBlocks int, sanitizedURL string) *remoteWriteCtx {
@@ -467,7 +474,8 @@ func newRemoteWriteCtx(argIdx int, at *auth.Token, remoteWriteURL *url.URL, maxI
 		c:   c,
 		pss: pss,
 
-		relabelMetricsDropped: metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_remotewrite_relabel_metrics_dropped_total{path=%q, url=%q}`, queuePath, sanitizedURL)),
+		rowsPushedAfterRelabel: metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_remotewrite_rows_pushed_after_relabel_total{path=%q, url=%q}`, queuePath, sanitizedURL)),
+		rowsDroppedByRelabel:   metrics.GetOrCreateCounter(fmt.Sprintf(`vmagent_remotewrite_relabel_metrics_dropped_total{path=%q, url=%q}`, queuePath, sanitizedURL)),
 	}
 }
 
@@ -483,7 +491,8 @@ func (rwctx *remoteWriteCtx) MustStop() {
 	rwctx.fq.MustClose()
 	rwctx.fq = nil
 
-	rwctx.relabelMetricsDropped = nil
+	rwctx.rowsPushedAfterRelabel = nil
+	rwctx.rowsDroppedByRelabel = nil
 }
 
 func (rwctx *remoteWriteCtx) Push(tss []prompbmarshal.TimeSeries) {
@@ -499,12 +508,15 @@ func (rwctx *remoteWriteCtx) Push(tss []prompbmarshal.TimeSeries) {
 		// and https://github.com/VictoriaMetrics/VictoriaMetrics/issues/599
 		v = tssRelabelPool.Get().(*[]prompbmarshal.TimeSeries)
 		tss = append(*v, tss...)
-		tssLen := len(tss)
+		rowsCountBeforeRelabel := getRowsCount(tss)
 		tss = rctx.applyRelabeling(tss, nil, pcs)
-		rwctx.relabelMetricsDropped.Add(tssLen - len(tss))
+		rowsCountAfterRelabel := getRowsCount(tss)
+		rwctx.rowsDroppedByRelabel.Add(rowsCountBeforeRelabel - rowsCountAfterRelabel)
 	}
 	pss := rwctx.pss
 	idx := atomic.AddUint64(&rwctx.pssNextIdx, 1) % uint64(len(pss))
+	rowsCount := getRowsCount(tss)
+	rwctx.rowsPushedAfterRelabel.Add(rowsCount)
 	pss[idx].Push(tss)
 	if rctx != nil {
 		*v = prompbmarshal.ResetTimeSeries(tss)
@@ -519,3 +531,11 @@ var tssRelabelPool = &sync.Pool{
 		return &a
 	},
 }
+
+func getRowsCount(tss []prompbmarshal.TimeSeries) int {
+	rowsCount := 0
+	for _, ts := range tss {
+		rowsCount += len(ts.Samples)
+	}
+	return rowsCount
+}
diff --git a/app/vmbackup/README.md b/app/vmbackup/README.md
index 135ff422b..a3f960b9f 100644
--- a/app/vmbackup/README.md
+++ b/app/vmbackup/README.md
@@ -7,7 +7,7 @@ Supported storage systems for backups:
 * [GCS](https://cloud.google.com/storage/). Example: `gs://<bucket>/<path/to/backup>`
 * [S3](https://aws.amazon.com/s3/). Example: `s3://<bucket>/<path/to/backup>`
 * Any S3-compatible storage such as [MinIO](https://github.com/minio/minio), [Ceph](https://docs.ceph.com/en/pacific/radosgw/s3/) or [Swift](https://platform.swiftstack.com/docs/admin/middleware/s3_middleware.html). See [these docs](#advanced-usage) for details.
-* Local filesystem. Example: `fs://</absolute/path/to/backup>`
+* Local filesystem. Example: `fs://</absolute/path/to/backup>`. Note that `vmbackup` prevents from storing the backup into the directory pointed by `-storageDataPath` command-line flag, since this directory should be managed solely by VictoriaMetrics or `vmstorage`.
 
 `vmbackup` supports incremental and full backups. Incremental backups are created automatically if the destination path already contains data from the previous backup.
 Full backups can be sped up with `-origin` pointing to an already existing backup on the same remote storage. In this case `vmbackup` makes server-side copy for the shared
diff --git a/app/vmbackup/main.go b/app/vmbackup/main.go
index 9c53f5fca..d527157c6 100644
--- a/app/vmbackup/main.go
+++ b/app/vmbackup/main.go
@@ -4,6 +4,7 @@ import (
 	"flag"
 	"fmt"
 	"os"
+	"path/filepath"
 	"strings"
 	"time"
 
@@ -155,9 +156,28 @@ func newDstFS() (common.RemoteFS, error) {
 	if err != nil {
 		return nil, fmt.Errorf("cannot parse `-dst`=%q: %w", *dst, err)
 	}
+	if hasFilepathPrefix(*dst, *storageDataPath) {
+		return nil, fmt.Errorf("-dst=%q can not point to the directory with VictoriaMetrics data (aka -storageDataPath=%q)", *dst, *storageDataPath)
+	}
 	return fs, nil
 }
 
+func hasFilepathPrefix(path, prefix string) bool {
+	if !strings.HasPrefix(path, "fs://") {
+		return false
+	}
+	path = path[len("fs://"):]
+	pathAbs, err := filepath.Abs(path)
+	if err != nil {
+		return false
+	}
+	prefixAbs, err := filepath.Abs(prefix)
+	if err != nil {
+		return false
+	}
+	return strings.HasPrefix(pathAbs, prefixAbs)
+}
+
 func newOriginFS() (common.OriginFS, error) {
 	if len(*origin) == 0 {
 		return &fsnil.FS{}, nil
diff --git a/app/vmctl/README.md b/app/vmctl/README.md
index d668be8bd..c3d1d5bd9 100644
--- a/app/vmctl/README.md
+++ b/app/vmctl/README.md
@@ -497,15 +497,18 @@ Total: 336.75 KiB ↖ Speed: 454.46 KiB p/s
 
 Importing tips:
 
-1. Migrating all the metrics from one VM to another may collide with existing application metrics
+1. Migrating big volumes of data may result in reaching the safety limits on `src` side.
+Please verify that `-search.maxExportDuration` and `-search.maxExportSeries` were set with
+proper values for `src`. If hitting the limits, follow the recommendations [here](https://docs.victoriametrics.com/#how-to-export-data-in-native-format).
+2. Migrating all the metrics from one VM to another may collide with existing application metrics
 (prefixed with `vm_`) at destination and lead to confusion when using
 [official Grafana dashboards](https://grafana.com/orgs/victoriametrics/dashboards).
 To avoid such situation try to filter out VM process metrics via `--vm-native-filter-match` flag.
-2. Migration is a backfilling process, so it is recommended to read
+3. Migration is a backfilling process, so it is recommended to read
 [Backfilling tips](https://github.com/VictoriaMetrics/VictoriaMetrics#backfilling) section.
-3. `vmctl` doesn't provide relabeling or other types of labels management in this mode.
+4. `vmctl` doesn't provide relabeling or other types of labels management in this mode.
 Instead, use [relabeling in VictoriaMetrics](https://github.com/VictoriaMetrics/vmctl/issues/4#issuecomment-683424375).
-4. When importing in or from cluster version remember to use correct [URL format](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
+5. When importing in or from cluster version remember to use correct [URL format](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
 and specify `accountID` param.
 
 ## Verifying exported blocks from VictoriaMetrics
diff --git a/app/vmctl/flags.go b/app/vmctl/flags.go
index cf14d0d9d..0f7b89152 100644
--- a/app/vmctl/flags.go
+++ b/app/vmctl/flags.go
@@ -201,6 +201,7 @@ const (
 	influxFilterTimeStart           = "influx-filter-time-start"
 	influxFilterTimeEnd             = "influx-filter-time-end"
 	influxMeasurementFieldSeparator = "influx-measurement-field-separator"
+	influxSkipDatabaseLabel         = "influx-skip-database-label"
 )
 
 var (
@@ -258,6 +259,11 @@ var (
 			Usage: "The {separator} symbol used to concatenate {measurement} and {field} names into series name {measurement}{separator}{field}.",
 			Value: "_",
 		},
+		&cli.BoolFlag{
+			Name:  influxSkipDatabaseLabel,
+			Usage: "Wether to skip adding the label 'db' to timeseries.",
+			Value: false,
+		},
 	}
 )
 
diff --git a/app/vmctl/influx.go b/app/vmctl/influx.go
index 55da3d4e1..b05b2a48a 100644
--- a/app/vmctl/influx.go
+++ b/app/vmctl/influx.go
@@ -12,21 +12,23 @@ import (
 )
 
 type influxProcessor struct {
-	ic        *influx.Client
-	im        *vm.Importer
-	cc        int
-	separator string
+	ic          *influx.Client
+	im          *vm.Importer
+	cc          int
+	separator   string
+	skipDbLabel bool
 }
 
-func newInfluxProcessor(ic *influx.Client, im *vm.Importer, cc int, separator string) *influxProcessor {
+func newInfluxProcessor(ic *influx.Client, im *vm.Importer, cc int, separator string, skipDbLabel bool) *influxProcessor {
 	if cc < 1 {
 		cc = 1
 	}
 	return &influxProcessor{
-		ic:        ic,
-		im:        im,
-		cc:        cc,
-		separator: separator,
+		ic:          ic,
+		im:          im,
+		cc:          cc,
+		separator:   separator,
+		skipDbLabel: skipDbLabel,
 	}
 }
 
@@ -126,7 +128,7 @@ func (ip *influxProcessor) do(s *influx.Series) error {
 			Value: lp.Value,
 		}
 	}
-	if !containsDBLabel {
+	if !containsDBLabel && !ip.skipDbLabel {
 		labels = append(labels, vm.LabelPair{
 			Name:  dbLabel,
 			Value: ip.ic.Database(),
diff --git a/app/vmctl/main.go b/app/vmctl/main.go
index eefe0f6bd..26d7ad3b3 100644
--- a/app/vmctl/main.go
+++ b/app/vmctl/main.go
@@ -104,7 +104,8 @@ func main() {
 						influxClient,
 						importer,
 						c.Int(influxConcurrency),
-						c.String(influxMeasurementFieldSeparator))
+						c.String(influxMeasurementFieldSeparator),
+						c.Bool(influxSkipDatabaseLabel))
 					return processor.run(c.Bool(globalSilent), c.Bool(globalVerbose))
 				},
 			},
diff --git a/deployment/docker/docker-compose.yml b/deployment/docker/docker-compose.yml
index fe12e3e2d..29cadd58a 100644
--- a/deployment/docker/docker-compose.yml
+++ b/deployment/docker/docker-compose.yml
@@ -34,6 +34,7 @@ services:
       - "--opentsdbListenAddr=:4242"
       - "--httpListenAddr=:8428"
       - "--influxListenAddr=:8089"
+      - "--vmalert.proxyURL=http://vmalert:8880"
     networks:
       - vm_net
     restart: always
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index adfa9daaa..cd04cc24f 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -15,9 +15,15 @@ The following tip changes can be tested by building VictoriaMetrics components f
 
 ## tip
 
+* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add ability to specify filters for Availability Zones in [ec2_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#ec2_sd_config) via `az_filters` section. This section can contain AZ-specific set of filters in the same way as the existing `filters` section, which is used for filtering EC2 instances. The list of supported AZ-specific filters is available [here](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeAvailabilityZones.html).
+* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): expose `vmagent_remotewrite_global_rows_pushed_before_relabel_total` and `vmagent_remotewrite_rows_pushed_after_relabel_total` metrics at `http://vmagent:8429/metrics`, which can be used for monitoring the rate of rows (aka samples) pushed to remote storage before and after the relabeling via `-remoteWrite.relabelConfig` and `-remoteWrite.urlRelabelConfig`. See [relabeling docs](https://docs.victoriametrics.com/vmagent.html#relabeling) for details.
+
 * BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl.html): properly import InfluxDB measurements if they contain `db` tag. Previously this could result in incomplete import of measurmenet tags. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2536). Thanks to @mback2k for the bugfix.
 * BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): do not reset the selected relative time range when entering new query. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2402#issuecomment-1115817302).
 * BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): properly show the graph when clicking on `Prometheus` link in Grafana graph editor. Previously the graph wasn't shown until clicking on the `Graph` tab. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2402#issuecomment-1115830648).
+* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): rename `vmagent_remote_write_rate_limit_reached_total` metric to `vmagent_remotewrite_rate_limit_reached_total`, so its name is consistent with the rest of `vmagent_remotewrite_` metrics.
+* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): rename `promscrape_stale_samples_created_total` metric to `vm_promscrape_stale_samples_created_total`, so its name is consistent with the rest of `vm_promscrape_` metrics.
+* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup.html): disallow writing backups to `-storageDataPath` directory, since this directory is managed solely by VictoriaMetrics or `vmstorage`. Other apps shouldn't write into this directory. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2503).
 
 
 ## [v1.77.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.77.0)
diff --git a/docs/Cluster-VictoriaMetrics.md b/docs/Cluster-VictoriaMetrics.md
index a58ff9f88..61bed29c4 100644
--- a/docs/Cluster-VictoriaMetrics.md
+++ b/docs/Cluster-VictoriaMetrics.md
@@ -329,7 +329,8 @@ Some capacity planning tips for VictoriaMetrics cluster:
 - Cluster capacity for [active time series](https://docs.victoriametrics.com/FAQ.html#what-is-an-active-time-series) can be increased by increasing RAM and CPU resources per each `vmstorage` node or by by adding new `vmstorage` nodes.
 - Query latency can be reduced by increasing CPU resources per each `vmselect` node, since each incoming query is processed by a single `vmselect` node. Performance for heavy queries scales with the number of available CPU cores at `vmselect` node, since `vmselect` processes time series referred by the query on all the available CPU cores.
 - If the cluster needs to process incoming queries at a high rate, then its capacity can be increased by adding more `vmselect` nodes, so incoming queries could be spread among bigger number of `vmselect` nodes.
-- By default `vminsert` compresses the data sent to `vmstorage` nodes in order to reduce network bandwidth usage. The compression takes additional CPU resources. If `vminsert` nodes have limited CPU, then the compression can be disabled by passing `-rpc.disableCompression` command-line flag at `vminsert` nodes.
+- By default `vminsert` compresses the data it sends to `vmstorage` in order to reduce network bandwidth usage. The compression takes additional CPU resources at `vminsert`. If `vminsert` nodes have limited CPU, then the compression can be disabled by passing `-rpc.disableCompression` command-line flag at `vminsert` nodes.
+- By default `vmstorage` compresses the data it sends to `vmselect` during queries in order to reduce network bandwidth usage. The compression takes additional CPU resources at `vmstorage`. If `vmstorage` nodes have limited CPU, then the compression can be disabled by passing `-rpc.disableCompression` command-line flag at `vmstorage` nodes.
 
 ## High availability
 
@@ -601,7 +602,7 @@ Below is the output for `/path/to/vminsert -help`:
   -replicationFactor int
      Replication factor for the ingested data, i.e. how many copies to make among distinct -storageNode instances. Note that vmselect must run with -dedup.minScrapeInterval=1ms for data de-duplication when replicationFactor is greater than 1. Higher values for -dedup.minScrapeInterval at vmselect is OK (default 1)
   -rpc.disableCompression
-     Whether to disable compression of RPC traffic. This reduces CPU usage at the cost of higher network bandwidth usage
+     Whether to disable compression for the data sent from vminsert to vmstorage. This reduces CPU usage at the cost of higher network bandwidth usage
   -sortLabels
      Whether to sort labels for incoming samples before writing them to storage. This may be needed for reducing memory usage at storage when the order of labels in incoming samples is random. For example, if m{k1="v1",k2="v2"} may be sent as m{k2="v2",k1="v1"}. Enabled sorting for labels can slow down ingestion performance a bit
   -storageNode array
@@ -857,7 +858,7 @@ Below is the output for `/path/to/vmstorage -help`:
      Data with timestamps outside the retentionPeriod is automatically deleted
      The following optional suffixes are supported: h (hour), d (day), w (week), y (year). If suffix isn't set, then the duration is counted in months (default 1)
   -rpc.disableCompression
-     Disable compression of RPC traffic. This reduces CPU usage at the cost of higher network bandwidth usage
+     Whether to disable compression of the data sent from vmstorage to vmselect. This reduces CPU usage at the cost of higher network bandwidth usage
   -search.maxTagKeys int
      The maximum number of tag keys returned per search (default 100000)
   -search.maxTagValueSuffixesPerSearch int
@@ -870,8 +871,9 @@ Below is the output for `/path/to/vmstorage -help`:
      The maximum number of CPU cores to use for small merges. Default value is used if set to 0
   -snapshotAuthKey string
      authKey, which must be passed in query string to /snapshot* pages
-  -snapshotsMaxAge duration
+  -snapshotsMaxAge value
      Automatically delete snapshots older than -snapshotsMaxAge if it is set to non-zero duration. Make sure that backup process has enough time to finish the backup before the corresponding snapshot is automatically deleted
+     The following optional suffixes are supported: h (hour), d (day), w (week), y (year). If suffix isn't set, then the duration is counted in months (default 0)
   -storage.cacheSizeIndexDBDataBlocks size
      Overrides max size for indexdb/dataBlocks cache. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning
      Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 0)
diff --git a/docs/vmbackup.md b/docs/vmbackup.md
index de46ca093..64b160c49 100644
--- a/docs/vmbackup.md
+++ b/docs/vmbackup.md
@@ -11,7 +11,7 @@ Supported storage systems for backups:
 * [GCS](https://cloud.google.com/storage/). Example: `gs://<bucket>/<path/to/backup>`
 * [S3](https://aws.amazon.com/s3/). Example: `s3://<bucket>/<path/to/backup>`
 * Any S3-compatible storage such as [MinIO](https://github.com/minio/minio), [Ceph](https://docs.ceph.com/en/pacific/radosgw/s3/) or [Swift](https://platform.swiftstack.com/docs/admin/middleware/s3_middleware.html). See [these docs](#advanced-usage) for details.
-* Local filesystem. Example: `fs://</absolute/path/to/backup>`
+* Local filesystem. Example: `fs://</absolute/path/to/backup>`. Note that `vmbackup` prevents from storing the backup into the directory pointed by `-storageDataPath` command-line flag, since this directory should be managed solely by VictoriaMetrics or `vmstorage`.
 
 `vmbackup` supports incremental and full backups. Incremental backups are created automatically if the destination path already contains data from the previous backup.
 Full backups can be sped up with `-origin` pointing to an already existing backup on the same remote storage. In this case `vmbackup` makes server-side copy for the shared
diff --git a/docs/vmctl.md b/docs/vmctl.md
index 4ab32ac88..d4e599c00 100644
--- a/docs/vmctl.md
+++ b/docs/vmctl.md
@@ -501,15 +501,18 @@ Total: 336.75 KiB ↖ Speed: 454.46 KiB p/s
 
 Importing tips:
 
-1. Migrating all the metrics from one VM to another may collide with existing application metrics
+1. Migrating big volumes of data may result in reaching the safety limits on `src` side.
+Please verify that `-search.maxExportDuration` and `-search.maxExportSeries` were set with
+proper values for `src`. If hitting the limits, follow the recommendations [here](https://docs.victoriametrics.com/#how-to-export-data-in-native-format).
+2. Migrating all the metrics from one VM to another may collide with existing application metrics
 (prefixed with `vm_`) at destination and lead to confusion when using
 [official Grafana dashboards](https://grafana.com/orgs/victoriametrics/dashboards).
 To avoid such situation try to filter out VM process metrics via `--vm-native-filter-match` flag.
-2. Migration is a backfilling process, so it is recommended to read
+3. Migration is a backfilling process, so it is recommended to read
 [Backfilling tips](https://github.com/VictoriaMetrics/VictoriaMetrics#backfilling) section.
-3. `vmctl` doesn't provide relabeling or other types of labels management in this mode.
+4. `vmctl` doesn't provide relabeling or other types of labels management in this mode.
 Instead, use [relabeling in VictoriaMetrics](https://github.com/VictoriaMetrics/vmctl/issues/4#issuecomment-683424375).
-4. When importing in or from cluster version remember to use correct [URL format](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
+5. When importing in or from cluster version remember to use correct [URL format](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
 and specify `accountID` param.
 
 ## Verifying exported blocks from VictoriaMetrics
diff --git a/lib/promscrape/discovery/ec2/api.go b/lib/promscrape/discovery/ec2/api.go
index 014cea086..dc3640c97 100644
--- a/lib/promscrape/discovery/ec2/api.go
+++ b/lib/promscrape/discovery/ec2/api.go
@@ -8,9 +8,10 @@ import (
 )
 
 type apiConfig struct {
-	awsConfig *awsapi.Config
-	filters   []awsapi.Filter
-	port      int
+	awsConfig       *awsapi.Config
+	instanceFilters []awsapi.Filter
+	azFilters       []awsapi.Filter
+	port            int
 
 	// A map from AZ name to AZ id.
 	azMap     map[string]string
@@ -37,9 +38,10 @@ func newAPIConfig(sdc *SDConfig) (*apiConfig, error) {
 		return nil, err
 	}
 	cfg := &apiConfig{
-		awsConfig: awsCfg,
-		filters:   sdc.Filters,
-		port:      port,
+		awsConfig:       awsCfg,
+		instanceFilters: sdc.InstanceFilters,
+		azFilters:       sdc.AZFilters,
+		port:            port,
 	}
 	return cfg, nil
 }
diff --git a/lib/promscrape/discovery/ec2/az.go b/lib/promscrape/discovery/ec2/az.go
index ed299c454..42ce2fd84 100644
--- a/lib/promscrape/discovery/ec2/az.go
+++ b/lib/promscrape/discovery/ec2/az.go
@@ -30,7 +30,7 @@ func getAZMap(cfg *apiConfig) map[string]string {
 
 func getAvailabilityZones(cfg *apiConfig) ([]AvailabilityZone, error) {
 	// See https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeAvailabilityZones.html
-	azFilters := awsapi.GetFiltersQueryString(cfg.filters, azFiltersWhitelist)
+	azFilters := awsapi.GetFiltersQueryString(cfg.azFilters, azFiltersWhitelist)
 	data, err := cfg.awsConfig.GetEC2APIResponse("DescribeAvailabilityZones", azFilters, "")
 	if err != nil {
 		return nil, fmt.Errorf("cannot obtain availability zones: %w", err)
diff --git a/lib/promscrape/discovery/ec2/ec2.go b/lib/promscrape/discovery/ec2/ec2.go
index 5cca0bc3b..dd229ae89 100644
--- a/lib/promscrape/discovery/ec2/ec2.go
+++ b/lib/promscrape/discovery/ec2/ec2.go
@@ -27,8 +27,9 @@ type SDConfig struct {
 	RoleARN string `yaml:"role_arn,omitempty"`
 	// RefreshInterval time.Duration `yaml:"refresh_interval"`
 	// refresh_interval is obtained from `-promscrape.ec2SDCheckInterval` command-line option.
-	Port    *int            `yaml:"port,omitempty"`
-	Filters []awsapi.Filter `yaml:"filters,omitempty"`
+	Port            *int            `yaml:"port,omitempty"`
+	InstanceFilters []awsapi.Filter `yaml:"filters,omitempty"`
+	AZFilters       []awsapi.Filter `yaml:"az_filters,omitempty"`
 }
 
 // GetLabels returns ec2 labels according to sdc.
diff --git a/lib/promscrape/discovery/ec2/instance.go b/lib/promscrape/discovery/ec2/instance.go
index 89b0369e0..b7a219f0b 100644
--- a/lib/promscrape/discovery/ec2/instance.go
+++ b/lib/promscrape/discovery/ec2/instance.go
@@ -29,7 +29,7 @@ func getReservations(cfg *apiConfig) ([]Reservation, error) {
 	// See https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstances.html
 	var rs []Reservation
 	pageToken := ""
-	instanceFilters := awsapi.GetFiltersQueryString(cfg.filters, nil)
+	instanceFilters := awsapi.GetFiltersQueryString(cfg.instanceFilters, nil)
 	for {
 		data, err := cfg.awsConfig.GetEC2APIResponse("DescribeInstances", instanceFilters, pageToken)
 		if err != nil {
diff --git a/lib/promscrape/scrapework.go b/lib/promscrape/scrapework.go
index 3e20633ca..f1bfb7169 100644
--- a/lib/promscrape/scrapework.go
+++ b/lib/promscrape/scrapework.go
@@ -733,7 +733,7 @@ func (sw *scrapeWork) sendStaleSeries(lastScrape, currScrape string, timestamp i
 	sw.pushData(&wc.writeRequest)
 }
 
-var staleSamplesCreated = metrics.NewCounter(`promscrape_stale_samples_created_total`)
+var staleSamplesCreated = metrics.NewCounter(`vm_promscrape_stale_samples_created_total`)
 
 func (sw *scrapeWork) getLabelsHash(labels []prompbmarshal.Label) uint64 {
 	// It is OK if there will be hash collisions for distinct sets of labels,


Test patch:
diff --git a/app/vmbackup/main_test.go b/app/vmbackup/main_test.go
new file mode 100644
index 000000000..f3288fb49
--- /dev/null
+++ b/app/vmbackup/main_test.go
@@ -0,0 +1,29 @@
+package main
+
+import (
+	"path/filepath"
+	"testing"
+)
+
+func TestHasFilepathPrefix(t *testing.T) {
+	f := func(dst, storageDataPath string, resultExpected bool) {
+		t.Helper()
+		result := hasFilepathPrefix(dst, storageDataPath)
+		if result != resultExpected {
+			t.Errorf("unexpected hasFilepathPrefix(%q, %q); got: %v; want: %v", dst, storageDataPath, result, resultExpected)
+		}
+	}
+	pwd, err := filepath.Abs("")
+	if err != nil {
+		t.Fatalf("cannot determine working directory: %s", err)
+	}
+	f("s3://foo/bar", "foo", false)
+	f("fs://"+pwd+"/foo", "foo", true)
+	f("fs://"+pwd+"/foo", "foo/bar", false)
+	f("fs://"+pwd+"/foo/bar", "foo", true)
+	f("fs://"+pwd+"/foo", "bar", false)
+	f("fs://"+pwd+"/foo", pwd+"/foo", true)
+	f("fs://"+pwd+"/foo", pwd+"/foo/bar", false)
+	f("fs://"+pwd+"/foo/bar", pwd+"/foo", true)
+	f("fs://"+pwd+"/foo", pwd+"/bar", false)
+}