Repository: VictoriaMetrics/VictoriaMetrics
Pull request #4295 (2023-05-11 12:22:10)

vmagent:scrape config support enable_http2
fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4283


Haleygo: @hagen1778 can you give me some advice :) 

https://github.com/VictoriaMetrics/VictoriaMetrics/blob/7ea2531db0bdc3e68d2503f665f6c487a1f3cd29/lib/promscrape/client.go#L122-L134
Looks like the hc also needs to http2 config, I saw this [implementation](https://github.com/dgrr/http2) for https://github.com/valyala/fasthttp, but vm use it's own private fasthttp. Should I use this implementation or add implementation inside github.com/VictoriaMetrics/fasthttp
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4295?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) Report
Patch coverage: **`26.08`**% and project coverage change: **`-0.04`** :warning:
> Comparison is base [(`5119673`)](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/commit/51196739af72824dbe59ae909ebd23e975ad0df7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) 61.21% compared to head [(`5d3ac5d`)](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4295?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) 61.18%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master    #4295      +/-   ##
==========================================
- Coverage   61.21%   61.18%   -0.04%     
==========================================
  Files         330      330              
  Lines       58951    58973      +22     
==========================================
- Hits        36089    36083       -6     
- Misses      20885    20911      +26     
- Partials     1977     1979       +2     
```


| [Impacted Files](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4295?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) | Coverage Δ | |
|---|---|---|
| [app/vmalert/alerting.go](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4295?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvYWxlcnRpbmcuZ28=) | `76.24% <0.00%> (-0.93%)` | :arrow_down: |
| [lib/promauth/config.go](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4295?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL3Byb21hdXRoL2NvbmZpZy5nbw==) | `50.11% <ø> (ø)` | |
| [lib/promscrape/client.go](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4295?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL3Byb21zY3JhcGUvY2xpZW50Lmdv) | `0.00% <0.00%> (ø)` | |
| [lib/promscrape/scrapework.go](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4295?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL3Byb21zY3JhcGUvc2NyYXBld29yay5nbw==) | `57.43% <ø> (ø)` | |
| [lib/promscrape/config.go](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4295?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL3Byb21zY3JhcGUvY29uZmlnLmdv) | `35.84% <100.00%> (+0.38%)` | :arrow_up: |

... and [5 files with indirect coverage changes](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4295/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4295?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).

hagen1778: Ok, so looks like promscrape is using two types of HTTP clients under the hood:
1. fasthttp client - for default type of scraping, optimized to allocate as less as possible for the sake of efficiency
2. standard HTTP client - for `stream parsing mode`, where the scrape body is read in chunks. 

fasthttp is unlikely to get support of http2.

So there are two options left:
1. Either move 1st case (non-streaming) to standard HTTP client
2. Or just support `enable_http2` for streaming only. Means, automatically switch to streaming mode if `enable_http2` is set.

Wdyt?

What do @f41gh7 @Amper @valyala think?
f41gh7: @hagen1778 @Haleygo 
I think, it's better to switch for streaming mode automatically if `http2` enabled.
Haleygo: fixed, can you take another look @hagen1778 
Haleygo: fixed, can you take another look @hagen1778 
Haleygo: > @Haleygo do you think we can add a test to check if http2 param is working?

Added some ut to check `FollowRedirects` in the latest commit, but I couldn't find out how to check validity of `EnableHTTP2` in that way. Any suggestion?

Patch:
diff --git a/README.md b/README.md
index d0b071e45..91d64636b 100644
--- a/README.md
+++ b/README.md
@@ -1466,22 +1466,37 @@ with the enabled de-duplication. See [this section](#deduplication) for details.
 
 ## Deduplication
 
-VictoriaMetrics leaves a single raw sample with the biggest timestamp per each `-dedup.minScrapeInterval` discrete interval
-if `-dedup.minScrapeInterval` is set to positive duration. For example, `-dedup.minScrapeInterval=60s` would leave a single
-raw sample with the biggest timestamp per each discrete 60s interval.
+VictoriaMetrics leaves a single [raw sample](https://docs.victoriametrics.com/keyConcepts.html#raw-samples)
+with the biggest [timestamp](https://en.wikipedia.org/wiki/Unix_time) for each [time series](https://docs.victoriametrics.com/keyConcepts.html#time-series)
+per each `-dedup.minScrapeInterval` discrete interval if `-dedup.minScrapeInterval` is set to positive duration. 
+For example, `-dedup.minScrapeInterval=60s` would leave a single raw sample with the biggest timestamp per each discrete
+`60s` interval.
 This aligns with the [staleness rules in Prometheus](https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness).
 
-If multiple raw samples have the same biggest timestamp on the given `-dedup.minScrapeInterval` discrete interval, then the sample with the biggest value is left.
+If multiple raw samples have **the same timestamp** on the given `-dedup.minScrapeInterval` discrete interval, 
+then the sample with **the biggest value** is kept.
 
-The `-dedup.minScrapeInterval=D` is equivalent to `-downsampling.period=0s:D` if [downsampling](#downsampling) is enabled. So it is safe to use deduplication and downsampling simultaneously.
+Please note, [labels](https://docs.victoriametrics.com/keyConcepts.html#labels) of raw samples should be identical
+in order to be deduplicated. For example, this is why [HA pair of vmagents](https://docs.victoriametrics.com/vmagent.html#high-availability)
+needs to be identically configured. 
 
-The recommended value for `-dedup.minScrapeInterval` must equal to `scrape_interval` config from Prometheus configs. It is recommended to have a single `scrape_interval` across all the scrape targets. See [this article](https://www.robustperception.io/keep-it-simple-scrape_interval-id) for details.
+The `-dedup.minScrapeInterval=D` is equivalent to `-downsampling.period=0s:D` if [downsampling](#downsampling) is enabled.
+So it is safe to use deduplication and downsampling simultaneously.
 
-The de-duplication reduces disk space usage if multiple identically configured [vmagent](https://docs.victoriametrics.com/vmagent.html) or Prometheus instances in HA pair
-write data to the same VictoriaMetrics instance. These vmagent or Prometheus instances must have identical
-`external_labels` section in their configs, so they write data to the same time series. See also [how to set up multiple vmagent instances for scraping the same targets](https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets).
+The recommended value for `-dedup.minScrapeInterval` must equal to `scrape_interval` config from Prometheus configs. 
+It is recommended to have a single `scrape_interval` across all the scrape targets. 
+See [this article](https://www.robustperception.io/keep-it-simple-scrape_interval-id) for details.
 
-It is recommended passing different `-promscrape.cluster.name` values to HA pairs of `vmagent` instances, so the de-duplication consistently leaves samples for one `vmagent` instance and removes duplicate samples from other `vmagent` instances. See [these docs](https://docs.victoriametrics.com/vmagent.html#high-availability) for details.
+The de-duplication reduces disk space usage if multiple **identically configured** [vmagent](https://docs.victoriametrics.com/vmagent.html)
+or Prometheus instances in HA pair write data to the same VictoriaMetrics instance. 
+These vmagent or Prometheus instances must have **identical** `external_labels` section in their configs, 
+so they write data to the same time series. 
+See also [how to set up multiple vmagent instances for scraping the same targets](https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets).
+
+It is recommended passing different `-promscrape.cluster.name` values to each distinct HA pair of `vmagent` instances, 
+so the de-duplication consistently leaves samples for one `vmagent` instance and removes duplicate samples 
+from other `vmagent` instances. 
+See [these docs](https://docs.victoriametrics.com/vmagent.html#high-availability) for details.
 
 ## Storage
 
@@ -2483,6 +2498,10 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      The maximum number of time series, which can be returned from /federate. This option allows limiting memory usage (default 1000000)
   -search.maxGraphiteSeries int
      The maximum number of time series, which can be scanned during queries to Graphite Render API. See https://docs.victoriametrics.com/#graphite-render-api-usage (default 300000)
+  -search.maxGraphiteTagKeys int
+     The maximum number of tag keys returned from Graphite /tags, /tags/autoComplete/*, /tags/findSeries API (default 100000)
+  -search.maxGraphiteTagValues int
+     The maximum number of tag values returned Graphite /tags/<tag_name> API (default 100000) 
   -search.maxLookback duration
      Synonym to -search.lookback-delta from Prometheus. The value is dynamically detected from interval between time series datapoints if not set. It can be overridden on per-query basis via max_lookback arg. See also '-search.maxStalenessInterval' flag, which has the same meaning due to historical reasons
   -search.maxMemoryPerQuery size
diff --git a/app/vmagent/README.md b/app/vmagent/README.md
index 2882b3121..523fd85ad 100644
--- a/app/vmagent/README.md
+++ b/app/vmagent/README.md
@@ -752,14 +752,18 @@ See [these docs](https://docs.victoriametrics.com/#deduplication) for details.
 
 ## High availability
 
-It is possible to run multiple identically configured `vmagent` instances or `vmagent` [clusters](#scraping-big-number-of-targets),
-so they [scrape](#how-to-collect-metrics-in-prometheus-format) the same set of targets and push the collected data to the same set of VictoriaMetrics remote storage systems.
+It is possible to run multiple **identically configured** `vmagent` instances or `vmagent` 
+[clusters](#scraping-big-number-of-targets), so they [scrape](#how-to-collect-metrics-in-prometheus-format) 
+the same set of targets and push the collected data to the same set of VictoriaMetrics remote storage systems. 
+Two **identically configured** vmagent instances or clusters is usually called an HA pair.
 
-In this case the deduplication must be configured at VictoriaMetrics in order to de-duplicate samples received from multiple identically configured `vmagent` instances or clusters.
+When running HA pairs, [deduplication](https://docs.victoriametrics.com/#deduplication) must be configured 
+at VictoriaMetrics side in order to de-duplicate received samples.
 See [these docs](https://docs.victoriametrics.com/#deduplication) for details.
 
-It is also recommended passing different values to `-promscrape.cluster.name` command-line flag per each `vmagent` instance or per each `vmagent` cluster in HA setup.
-This is needed for proper data de-duplication. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2679) for details.
+It is also recommended passing different values to `-promscrape.cluster.name` command-line flag per each `vmagent` 
+instance or per each `vmagent` cluster in HA setup. This is needed for proper data de-duplication. 
+See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2679) for details.
 
 ## Scraping targets via a proxy
 
diff --git a/app/vmalert/Makefile b/app/vmalert/Makefile
index 8680e2fac..9f200f0c2 100644
--- a/app/vmalert/Makefile
+++ b/app/vmalert/Makefile
@@ -73,6 +73,7 @@ test-vmalert:
 	go test -v -race -cover ./app/vmalert/notifier
 	go test -v -race -cover ./app/vmalert/config
 	go test -v -race -cover ./app/vmalert/remotewrite
+	go test -v -race -cover ./app/vmalert/utils
 
 run-vmalert: vmalert
 	./bin/vmalert -rule=app/vmalert/config/testdata/rules/rules2-good.rules \
diff --git a/app/vmalert/datasource/vm.go b/app/vmalert/datasource/vm.go
index 2210b8e18..cf6a43d3e 100644
--- a/app/vmalert/datasource/vm.go
+++ b/app/vmalert/datasource/vm.go
@@ -29,6 +29,7 @@ func toDatasourceType(s string) datasourceType {
 }
 
 // VMStorage represents vmstorage entity with ability to read and write metrics
+// WARN: when adding a new field, remember to update Clone() method.
 type VMStorage struct {
 	c                *http.Client
 	authCfg          *promauth.Config
@@ -54,29 +55,54 @@ type keyValue struct {
 
 // Clone makes clone of VMStorage, shares http client.
 func (s *VMStorage) Clone() *VMStorage {
-	return &VMStorage{
+	ns := &VMStorage{
 		c:                s.c,
 		authCfg:          s.authCfg,
 		datasourceURL:    s.datasourceURL,
+		appendTypePrefix: s.appendTypePrefix,
 		lookBack:         s.lookBack,
 		queryStep:        s.queryStep,
-		appendTypePrefix: s.appendTypePrefix,
-		dataSourceType:   s.dataSourceType,
+
+		dataSourceType:     s.dataSourceType,
+		evaluationInterval: s.evaluationInterval,
+
+		// init map so it can be populated below
+		extraParams: url.Values{},
+
+		debug: s.debug,
+	}
+	if len(s.extraHeaders) > 0 {
+		ns.extraHeaders = make([]keyValue, len(s.extraHeaders))
+		copy(ns.extraHeaders, s.extraHeaders)
 	}
+	for k, v := range s.extraParams {
+		ns.extraParams[k] = v
+	}
+
+	return ns
 }
 
 // ApplyParams - changes given querier params.
 func (s *VMStorage) ApplyParams(params QuerierParams) *VMStorage {
 	s.dataSourceType = toDatasourceType(params.DataSourceType)
 	s.evaluationInterval = params.EvaluationInterval
-	s.extraParams = params.QueryParams
-	s.debug = params.Debug
+	if params.QueryParams != nil {
+		if s.extraParams == nil {
+			s.extraParams = url.Values{}
+		}
+		for k, vl := range params.QueryParams {
+			for _, v := range vl { // custom query params are prior to default ones
+				s.extraParams.Set(k, v)
+			}
+		}
+	}
 	if params.Headers != nil {
 		for key, value := range params.Headers {
 			kv := keyValue{key: key, value: value}
 			s.extraHeaders = append(s.extraHeaders, kv)
 		}
 	}
+	s.debug = params.Debug
 	return s
 }
 
@@ -95,6 +121,7 @@ func NewVMStorage(baseURL string, authCfg *promauth.Config, lookBack time.Durati
 		lookBack:         lookBack,
 		queryStep:        queryStep,
 		dataSourceType:   datasourcePrometheus,
+		extraParams:      url.Values{},
 	}
 }
 
diff --git a/app/vmctl/barpool/pool.go b/app/vmctl/barpool/pool.go
index 4f095d8ac..05fe8b418 100644
--- a/app/vmctl/barpool/pool.go
+++ b/app/vmctl/barpool/pool.go
@@ -3,7 +3,13 @@
 // altogether.
 package barpool
 
-import "github.com/cheggaaa/pb/v3"
+import (
+	"fmt"
+	"os"
+
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/terminal"
+	"github.com/cheggaaa/pb/v3"
+)
 
 var pool = pb.NewPool()
 
@@ -20,7 +26,22 @@ func Stop() { _ = pool.Stop() }
 // AddWithTemplate adds bar with the given template
 // to the global pool
 func AddWithTemplate(format string, total int) *pb.ProgressBar {
-	bar := pb.ProgressBarTemplate(format).New(total)
+	tpl := getTemplate(format)
+	bar := pb.ProgressBarTemplate(tpl).New(total)
 	Add(bar)
 	return bar
 }
+
+// NewSingleProgress returns progress bar with given template
+func NewSingleProgress(format string, total int) *pb.ProgressBar {
+	tpl := getTemplate(format)
+	return pb.ProgressBarTemplate(tpl).New(total)
+}
+
+func getTemplate(format string) string {
+	isTerminal := terminal.IsTerminal(int(os.Stdout.Fd()))
+	if !isTerminal {
+		format = fmt.Sprintf("%s\n", format)
+	}
+	return format
+}
diff --git a/app/vmctl/influx.go b/app/vmctl/influx.go
index 8e11d91be..41cf74083 100644
--- a/app/vmctl/influx.go
+++ b/app/vmctl/influx.go
@@ -18,9 +18,11 @@ type influxProcessor struct {
 	separator   string
 	skipDbLabel bool
 	promMode    bool
+	isSilent    bool
+	isVerbose   bool
 }
 
-func newInfluxProcessor(ic *influx.Client, im *vm.Importer, cc int, separator string, skipDbLabel bool, promMode bool) *influxProcessor {
+func newInfluxProcessor(ic *influx.Client, im *vm.Importer, cc int, separator string, skipDbLabel, promMode, silent, verbose bool) *influxProcessor {
 	if cc < 1 {
 		cc = 1
 	}
@@ -31,10 +33,12 @@ func newInfluxProcessor(ic *influx.Client, im *vm.Importer, cc int, separator st
 		separator:   separator,
 		skipDbLabel: skipDbLabel,
 		promMode:    promMode,
+		isSilent:    silent,
+		isVerbose:   verbose,
 	}
 }
 
-func (ip *influxProcessor) run(silent, verbose bool) error {
+func (ip *influxProcessor) run() error {
 	series, err := ip.ic.Explore()
 	if err != nil {
 		return fmt.Errorf("explore query failed: %s", err)
@@ -44,7 +48,7 @@ func (ip *influxProcessor) run(silent, verbose bool) error {
 	}
 
 	question := fmt.Sprintf("Found %d timeseries to import. Continue?", len(series))
-	if !silent && !prompt(question) {
+	if !ip.isSilent && !prompt(question) {
 		return nil
 	}
 
@@ -79,7 +83,7 @@ func (ip *influxProcessor) run(silent, verbose bool) error {
 		case infErr := <-errCh:
 			return fmt.Errorf("influx error: %s", infErr)
 		case vmErr := <-ip.im.Errors():
-			return fmt.Errorf("import process failed: %s", wrapErr(vmErr, verbose))
+			return fmt.Errorf("import process failed: %s", wrapErr(vmErr, ip.isVerbose))
 		case seriesCh <- s:
 		}
 	}
@@ -91,7 +95,7 @@ func (ip *influxProcessor) run(silent, verbose bool) error {
 	// drain import errors channel
 	for vmErr := range ip.im.Errors() {
 		if vmErr.Err != nil {
-			return fmt.Errorf("import process failed: %s", wrapErr(vmErr, verbose))
+			return fmt.Errorf("import process failed: %s", wrapErr(vmErr, ip.isVerbose))
 		}
 	}
 	for err := range errCh {
diff --git a/app/vmctl/main.go b/app/vmctl/main.go
index be75ceb11..e81746a76 100644
--- a/app/vmctl/main.go
+++ b/app/vmctl/main.go
@@ -16,7 +16,6 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/backoff"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/native"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/remoteread"
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/terminal"
 	"github.com/urfave/cli/v2"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/influx"
@@ -72,8 +71,8 @@ func main() {
 						return fmt.Errorf("failed to create VM importer: %s", err)
 					}
 
-					otsdbProcessor := newOtsdbProcessor(otsdbClient, importer, c.Int(otsdbConcurrency))
-					return otsdbProcessor.run(isNonInteractive(c), c.Bool(globalVerbose))
+					otsdbProcessor := newOtsdbProcessor(otsdbClient, importer, c.Int(otsdbConcurrency), c.Bool(globalSilent), c.Bool(globalVerbose))
+					return otsdbProcessor.run()
 				},
 			},
 			{
@@ -113,8 +112,10 @@ func main() {
 						c.Int(influxConcurrency),
 						c.String(influxMeasurementFieldSeparator),
 						c.Bool(influxSkipDatabaseLabel),
-						c.Bool(influxPrometheusMode))
-					return processor.run(isNonInteractive(c), c.Bool(globalVerbose))
+						c.Bool(influxPrometheusMode),
+						c.Bool(globalSilent),
+						c.Bool(globalVerbose))
+					return processor.run()
 				},
 			},
 			{
@@ -152,9 +153,11 @@ func main() {
 							timeEnd:   c.Timestamp(remoteReadFilterTimeEnd),
 							chunk:     c.String(remoteReadStepInterval),
 						},
-						cc: c.Int(remoteReadConcurrency),
+						cc:        c.Int(remoteReadConcurrency),
+						isSilent:  c.Bool(globalSilent),
+						isVerbose: c.Bool(globalVerbose),
 					}
-					return rmp.run(ctx, isNonInteractive(c), c.Bool(globalVerbose))
+					return rmp.run(ctx)
 				},
 			},
 			{
@@ -188,7 +191,7 @@ func main() {
 						im: importer,
 						cc: c.Int(promConcurrency),
 					}
-					return pp.run(isNonInteractive(c), c.Bool(globalVerbose))
+					return pp.run(c.Bool(globalSilent), c.Bool(globalVerbose))
 				},
 			},
 			{
@@ -250,8 +253,9 @@ func main() {
 						backoff:        backoff.New(),
 						cc:             c.Int(vmConcurrency),
 						disableRetries: c.Bool(vmNativeDisableRetries),
+						isSilent:       c.Bool(globalSilent),
 					}
-					return p.run(ctx, isNonInteractive(c))
+					return p.run(ctx)
 				},
 			},
 			{
@@ -324,8 +328,3 @@ func initConfigVM(c *cli.Context) vm.Config {
 		DisableProgressBar: c.Bool(vmDisableProgressBar),
 	}
 }
-
-func isNonInteractive(c *cli.Context) bool {
-	isTerminal := terminal.IsTerminal(int(os.Stdout.Fd()))
-	return c.Bool(globalSilent) || !isTerminal
-}
diff --git a/app/vmctl/opentsdb.go b/app/vmctl/opentsdb.go
index b956f0592..a20ee007f 100644
--- a/app/vmctl/opentsdb.go
+++ b/app/vmctl/opentsdb.go
@@ -12,9 +12,11 @@ import (
 )
 
 type otsdbProcessor struct {
-	oc      *opentsdb.Client
-	im      *vm.Importer
-	otsdbcc int
+	oc        *opentsdb.Client
+	im        *vm.Importer
+	otsdbcc   int
+	isSilent  bool
+	isVerbose bool
 }
 
 type queryObj struct {
@@ -24,18 +26,20 @@ type queryObj struct {
 	StartTime int64
 }
 
-func newOtsdbProcessor(oc *opentsdb.Client, im *vm.Importer, otsdbcc int) *otsdbProcessor {
+func newOtsdbProcessor(oc *opentsdb.Client, im *vm.Importer, otsdbcc int, silent, verbose bool) *otsdbProcessor {
 	if otsdbcc < 1 {
 		otsdbcc = 1
 	}
 	return &otsdbProcessor{
-		oc:      oc,
-		im:      im,
-		otsdbcc: otsdbcc,
+		oc:        oc,
+		im:        im,
+		otsdbcc:   otsdbcc,
+		isSilent:  silent,
+		isVerbose: verbose,
 	}
 }
 
-func (op *otsdbProcessor) run(silent, verbose bool) error {
+func (op *otsdbProcessor) run() error {
 	log.Println("Loading all metrics from OpenTSDB for filters: ", op.oc.Filters)
 	var metrics []string
 	for _, filter := range op.oc.Filters {
@@ -51,7 +55,7 @@ func (op *otsdbProcessor) run(silent, verbose bool) error {
 	}
 
 	question := fmt.Sprintf("Found %d metrics to import. Continue?", len(metrics))
-	if !silent && !prompt(question) {
+	if !op.isSilent && !prompt(question) {
 		return nil
 	}
 	op.im.ResetStats()
@@ -114,7 +118,7 @@ func (op *otsdbProcessor) run(silent, verbose bool) error {
 					case otsdbErr := <-errCh:
 						return fmt.Errorf("opentsdb error: %s", otsdbErr)
 					case vmErr := <-op.im.Errors():
-						return fmt.Errorf("import process failed: %s", wrapErr(vmErr, verbose))
+						return fmt.Errorf("import process failed: %s", wrapErr(vmErr, op.isVerbose))
 					case seriesCh <- queryObj{
 						Tr: tr, StartTime: startTime,
 						Series: series, Rt: opentsdb.RetentionMeta{
@@ -138,7 +142,7 @@ func (op *otsdbProcessor) run(silent, verbose bool) error {
 	op.im.Close()
 	for vmErr := range op.im.Errors() {
 		if vmErr.Err != nil {
-			return fmt.Errorf("import process failed: %s", wrapErr(vmErr, verbose))
+			return fmt.Errorf("import process failed: %s", wrapErr(vmErr, op.isVerbose))
 		}
 	}
 	log.Println("Import finished!")
diff --git a/app/vmctl/remoteread.go b/app/vmctl/remoteread.go
index 6539268f4..8531c6cd6 100644
--- a/app/vmctl/remoteread.go
+++ b/app/vmctl/remoteread.go
@@ -20,7 +20,9 @@ type remoteReadProcessor struct {
 	dst *vm.Importer
 	src *remoteread.Client
 
-	cc int
+	cc        int
+	isSilent  bool
+	isVerbose bool
 }
 
 type remoteReadFilter struct {
@@ -29,7 +31,7 @@ type remoteReadFilter struct {
 	chunk     string
 }
 
-func (rrp *remoteReadProcessor) run(ctx context.Context, silent, verbose bool) error {
+func (rrp *remoteReadProcessor) run(ctx context.Context) error {
 	rrp.dst.ResetStats()
 	if rrp.filter.timeEnd == nil {
 		t := time.Now().In(rrp.filter.timeStart.Location())
@@ -46,19 +48,19 @@ func (rrp *remoteReadProcessor) run(ctx context.Context, silent, verbose bool) e
 
 	question := fmt.Sprintf("Selected time range %q - %q will be split into %d ranges according to %q step. Continue?",
 		rrp.filter.timeStart.String(), rrp.filter.timeEnd.String(), len(ranges), rrp.filter.chunk)
-	if !silent && !prompt(question) {
+	if !rrp.isSilent && !prompt(question) {
 		return nil
 	}
 
 	var bar *pb.ProgressBar
-	if !silent {
+	if !rrp.isSilent {
 		bar = barpool.AddWithTemplate(fmt.Sprintf(barTpl, "Processing ranges"), len(ranges))
 		if err := barpool.Start(); err != nil {
 			return err
 		}
 	}
 	defer func() {
-		if !silent {
+		if !rrp.isSilent {
 			barpool.Stop()
 		}
 		log.Println("Import finished!")
@@ -90,7 +92,7 @@ func (rrp *remoteReadProcessor) run(ctx context.Context, silent, verbose bool) e
 		case infErr := <-errCh:
 			return fmt.Errorf("remote read error: %s", infErr)
 		case vmErr := <-rrp.dst.Errors():
-			return fmt.Errorf("import process failed: %s", wrapErr(vmErr, verbose))
+			return fmt.Errorf("import process failed: %s", wrapErr(vmErr, rrp.isVerbose))
 		case rangeC <- &remoteread.Filter{
 			StartTimestampMs: r[0].UnixMilli(),
 			EndTimestampMs:   r[1].UnixMilli(),
@@ -105,7 +107,7 @@ func (rrp *remoteReadProcessor) run(ctx context.Context, silent, verbose bool) e
 	// drain import errors channel
 	for vmErr := range rrp.dst.Errors() {
 		if vmErr.Err != nil {
-			return fmt.Errorf("import process failed: %s", wrapErr(vmErr, verbose))
+			return fmt.Errorf("import process failed: %s", wrapErr(vmErr, rrp.isVerbose))
 		}
 	}
 	for err := range errCh {
diff --git a/app/vmctl/utils.go b/app/vmctl/utils.go
index de1fde0c2..c53b95554 100644
--- a/app/vmctl/utils.go
+++ b/app/vmctl/utils.go
@@ -6,12 +6,17 @@ import (
 	"os"
 	"strings"
 
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/terminal"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/vm"
 )
 
 const barTpl = `{{ blue "%s:" }} {{ counters . }} {{ bar . "[" "█" (cycle . "█") "▒" "]" }} {{ percent . }}`
 
 func prompt(question string) bool {
+	isTerminal := terminal.IsTerminal(int(os.Stdout.Fd()))
+	if !isTerminal {
+		return true
+	}
 	reader := bufio.NewReader(os.Stdin)
 	fmt.Print(question, " [Y/n] ")
 	answer, err := reader.ReadString('\n')
diff --git a/app/vmctl/vm_native.go b/app/vmctl/vm_native.go
index 64b2801eb..683ad6150 100644
--- a/app/vmctl/vm_native.go
+++ b/app/vmctl/vm_native.go
@@ -10,6 +10,7 @@ import (
 	"time"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/backoff"
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/barpool"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/limiter"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/native"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/stepper"
@@ -32,6 +33,7 @@ type vmNativeProcessor struct {
 	interCluster   bool
 	cc             int
 	disableRetries bool
+	isSilent       bool
 }
 
 const (
@@ -41,7 +43,7 @@ const (
 	nativeSingleProcessTpl = `Total: {{counters . }} {{ cycle . "↖" "↗" "↘" "↙" }} Speed: {{speed . }} {{string . "suffix"}}`
 )
 
-func (p *vmNativeProcessor) run(ctx context.Context, silent bool) error {
+func (p *vmNativeProcessor) run(ctx context.Context) error {
 	if p.cc == 0 {
 		p.cc = 1
 	}
@@ -78,13 +80,13 @@ func (p *vmNativeProcessor) run(ctx context.Context, silent bool) error {
 			return fmt.Errorf("failed to get tenants: %w", err)
 		}
 		question := fmt.Sprintf("The following tenants were discovered: %s.\n Continue?", tenants)
-		if !silent && !prompt(question) {
+		if !p.isSilent && !prompt(question) {
 			return nil
 		}
 	}
 
 	for _, tenantID := range tenants {
-		err := p.runBackfilling(ctx, tenantID, ranges, silent)
+		err := p.runBackfilling(ctx, tenantID, ranges, p.isSilent)
 		if err != nil {
 			return fmt.Errorf("migration failed: %s", err)
 		}
@@ -111,7 +113,6 @@ func (p *vmNativeProcessor) do(ctx context.Context, f native.Filter, srcURL, dst
 }
 
 func (p *vmNativeProcessor) runSingle(ctx context.Context, f native.Filter, srcURL, dstURL string, bar *pb.ProgressBar) error {
-
 	reader, err := p.src.ExportPipe(ctx, srcURL, f)
 	if err != nil {
 		return fmt.Errorf("failed to init export pipe: %w", err)
@@ -218,9 +219,9 @@ func (p *vmNativeProcessor) runBackfilling(ctx context.Context, tenantID string,
 
 	var bar *pb.ProgressBar
 	if !silent {
-		bar = pb.ProgressBarTemplate(fmt.Sprintf(nativeWithBackoffTpl, barPrefix)).New(len(metrics) * len(ranges))
+		bar = barpool.NewSingleProgress(fmt.Sprintf(nativeWithBackoffTpl, barPrefix), len(metrics)*len(ranges))
 		if p.disableRetries {
-			bar = pb.ProgressBarTemplate(nativeSingleProcessTpl).New(0)
+			bar = barpool.NewSingleProgress(nativeSingleProcessTpl, 0)
 		}
 		bar.Start()
 		defer bar.Finish()
diff --git a/app/vmselect/graphite/tags_api.go b/app/vmselect/graphite/tags_api.go
index 6236aff86..adbe3a886 100644
--- a/app/vmselect/graphite/tags_api.go
+++ b/app/vmselect/graphite/tags_api.go
@@ -1,6 +1,7 @@
 package graphite
 
 import (
+	"flag"
 	"fmt"
 	"net/http"
 	"regexp"
@@ -18,6 +19,11 @@ import (
 	"github.com/VictoriaMetrics/metrics"
 )
 
+var (
+	maxGraphiteTagKeysPerSearch   = flag.Int("search.maxGraphiteTagKeys", 100e3, "The maximum number of tag keys returned from Graphite /tags, /tags/autoComplete/*, /tags/findSeries API")
+	maxGraphiteTagValuesPerSearch = flag.Int("search.maxGraphiteTagValues", 100e3, "The maximum number of tag values returned Graphite /tags/<tag_name> API")
+)
+
 // TagsDelSeriesHandler implements /tags/delSeries handler.
 //
 // See https://graphite.readthedocs.io/en/stable/tags.html#removing-series-from-the-tagdb
@@ -182,13 +188,13 @@ func TagsAutoCompleteValuesHandler(startTime time.Time, w http.ResponseWriter, r
 		// Escape special chars in tagPrefix as Graphite does.
 		// See https://github.com/graphite-project/graphite-web/blob/3ad279df5cb90b211953e39161df416e54a84948/webapp/graphite/tags/base.py#L228
 		filter := regexp.QuoteMeta(valuePrefix)
-		tagValues, err = netstorage.GraphiteTagValues(nil, tag, filter, limit, deadline)
+		tagValues, err = netstorage.GraphiteTagValues(nil, tag, filter, *maxGraphiteTagKeysPerSearch, deadline)
 		if err != nil {
 			return err
 		}
 	} else {
 		// Slow path: use netstorage.SearchMetricNames for applying `expr` filters.
-		sq, err := getSearchQueryForExprs(startTime, etfs, exprs, limit*10)
+		sq, err := getSearchQueryForExprs(startTime, etfs, exprs, *maxGraphiteTagKeysPerSearch)
 		if err != nil {
 			return err
 		}
@@ -269,13 +275,13 @@ func TagsAutoCompleteTagsHandler(startTime time.Time, w http.ResponseWriter, r *
 		// Escape special chars in tagPrefix as Graphite does.
 		// See https://github.com/graphite-project/graphite-web/blob/3ad279df5cb90b211953e39161df416e54a84948/webapp/graphite/tags/base.py#L181
 		filter := regexp.QuoteMeta(tagPrefix)
-		labels, err = netstorage.GraphiteTags(nil, filter, limit, deadline)
+		labels, err = netstorage.GraphiteTags(nil, filter, *maxGraphiteTagKeysPerSearch, deadline)
 		if err != nil {
 			return err
 		}
 	} else {
 		// Slow path: use netstorage.SearchMetricNames for applying `expr` filters.
-		sq, err := getSearchQueryForExprs(startTime, etfs, exprs, limit*10)
+		sq, err := getSearchQueryForExprs(startTime, etfs, exprs, *maxGraphiteTagKeysPerSearch)
 		if err != nil {
 			return err
 		}
@@ -343,7 +349,7 @@ func TagsFindSeriesHandler(startTime time.Time, w http.ResponseWriter, r *http.R
 	if err != nil {
 		return fmt.Errorf("cannot setup tag filters: %w", err)
 	}
-	sq, err := getSearchQueryForExprs(startTime, etfs, exprs, limit*10)
+	sq, err := getSearchQueryForExprs(startTime, etfs, exprs, *maxGraphiteTagKeysPerSearch)
 	if err != nil {
 		return err
 	}
@@ -411,11 +417,14 @@ func TagValuesHandler(startTime time.Time, tagName string, w http.ResponseWriter
 		return err
 	}
 	filter := r.FormValue("filter")
-	tagValues, err := netstorage.GraphiteTagValues(nil, tagName, filter, limit, deadline)
+	tagValues, err := netstorage.GraphiteTagValues(nil, tagName, filter, *maxGraphiteTagValuesPerSearch, deadline)
 	if err != nil {
 		return err
 	}
 
+	if limit > 0 && limit < len(tagValues) {
+		tagValues = tagValues[:limit]
+	}
 	w.Header().Set("Content-Type", "application/json")
 	bw := bufferedwriter.Get(w)
 	defer bufferedwriter.Put(bw)
@@ -439,11 +448,14 @@ func TagsHandler(startTime time.Time, w http.ResponseWriter, r *http.Request) er
 		return err
 	}
 	filter := r.FormValue("filter")
-	labels, err := netstorage.GraphiteTags(nil, filter, limit, deadline)
+	labels, err := netstorage.GraphiteTags(nil, filter, *maxGraphiteTagKeysPerSearch, deadline)
 	if err != nil {
 		return err
 	}
 
+	if limit > 0 && limit < len(labels) {
+		labels = labels[:limit]
+	}
 	w.Header().Set("Content-Type", "application/json")
 	bw := bufferedwriter.Get(w)
 	defer bufferedwriter.Put(bw)
diff --git a/app/vmselect/prometheus/prometheus.go b/app/vmselect/prometheus/prometheus.go
index 9dd67ede2..dc9749805 100644
--- a/app/vmselect/prometheus/prometheus.go
+++ b/app/vmselect/prometheus/prometheus.go
@@ -633,11 +633,7 @@ func SeriesHandler(qt *querytracer.Tracer, startTime time.Time, w http.ResponseW
 		return err
 	}
 
-	minLimit := *maxSeriesLimit
-	if limit > 0 && limit < *maxSeriesLimit {
-		minLimit = limit
-	}
-	sq := storage.NewSearchQuery(cp.start, cp.end, cp.filterss, minLimit)
+	sq := storage.NewSearchQuery(cp.start, cp.end, cp.filterss, *maxSeriesLimit)
 	metricNames, err := netstorage.SearchMetricNames(qt, sq, cp.deadline)
 	if err != nil {
 		return fmt.Errorf("cannot fetch time series for %q: %w", sq, err)
diff --git a/app/vmui/packages/vmui/src/components/Configurators/GlobalSettings/GlobalSettings.tsx b/app/vmui/packages/vmui/src/components/Configurators/GlobalSettings/GlobalSettings.tsx
index 420a048be..4e3f60cbd 100644
--- a/app/vmui/packages/vmui/src/components/Configurators/GlobalSettings/GlobalSettings.tsx
+++ b/app/vmui/packages/vmui/src/components/Configurators/GlobalSettings/GlobalSettings.tsx
@@ -16,6 +16,7 @@ import { useTimeDispatch, useTimeState } from "../../../state/time/TimeStateCont
 import ThemeControl from "../ThemeControl/ThemeControl";
 import useDeviceDetect from "../../../hooks/useDeviceDetect";
 import useBoolean from "../../../hooks/useBoolean";
+import { getTenantIdFromUrl } from "../../../utils/tenants";
 
 const title = "Settings";
 
@@ -57,6 +58,10 @@ const GlobalSettings: FC = () => {
   };
 
   const handlerApply = () => {
+    const tenantIdFromUrl = getTenantIdFromUrl(serverUrl);
+    if (tenantIdFromUrl !== "") {
+      dispatch({ type: "SET_TENANT_ID", payload: tenantIdFromUrl });
+    }
     dispatch({ type: "SET_SERVER", payload: serverUrl });
     timeDispatch({ type: "SET_TIMEZONE", payload: timezone });
     customPanelDispatch({ type: "SET_SERIES_LIMITS", payload: limits });
diff --git a/deployment/docker/docker-compose-cluster.yml b/deployment/docker/docker-compose-cluster.yml
index 4b3f709d1..89fdf9d4b 100644
--- a/deployment/docker/docker-compose-cluster.yml
+++ b/deployment/docker/docker-compose-cluster.yml
@@ -2,7 +2,7 @@ version: '3.5'
 services:
   vmagent:
     container_name: vmagent
-    image: victoriametrics/vmagent:v1.91.0
+    image: victoriametrics/vmagent:v1.91.2
     depends_on:
       - "vminsert"
     ports:
@@ -32,7 +32,7 @@ services:
 
   vmstorage-1:
     container_name: vmstorage-1
-    image: victoriametrics/vmstorage:v1.91.0-cluster
+    image: victoriametrics/vmstorage:v1.91.2-cluster
     ports:
       - 8482
       - 8400
@@ -44,7 +44,7 @@ services:
     restart: always
   vmstorage-2:
     container_name: vmstorage-2
-    image: victoriametrics/vmstorage:v1.91.0-cluster
+    image: victoriametrics/vmstorage:v1.91.2-cluster
     ports:
       - 8482
       - 8400
@@ -56,7 +56,7 @@ services:
     restart: always
   vminsert:
     container_name: vminsert
-    image: victoriametrics/vminsert:v1.91.0-cluster
+    image: victoriametrics/vminsert:v1.91.2-cluster
     depends_on:
       - "vmstorage-1"
       - "vmstorage-2"
@@ -68,7 +68,7 @@ services:
     restart: always
   vmselect:
     container_name: vmselect
-    image: victoriametrics/vmselect:v1.91.0-cluster
+    image: victoriametrics/vmselect:v1.91.2-cluster
     depends_on:
       - "vmstorage-1"
       - "vmstorage-2"
@@ -82,7 +82,7 @@ services:
 
   vmalert:
     container_name: vmalert
-    image: victoriametrics/vmalert:v1.91.0
+    image: victoriametrics/vmalert:v1.91.2
     depends_on:
       - "vmselect"
     ports:
diff --git a/deployment/docker/docker-compose.yml b/deployment/docker/docker-compose.yml
index fef488659..a9e8189b7 100644
--- a/deployment/docker/docker-compose.yml
+++ b/deployment/docker/docker-compose.yml
@@ -2,7 +2,7 @@ version: "3.5"
 services:
   vmagent:
     container_name: vmagent
-    image: victoriametrics/vmagent:v1.91.0
+    image: victoriametrics/vmagent:v1.91.2
     depends_on:
       - "victoriametrics"
     ports:
@@ -18,7 +18,7 @@ services:
     restart: always
   victoriametrics:
     container_name: victoriametrics
-    image: victoriametrics/victoria-metrics:v1.91.0
+    image: victoriametrics/victoria-metrics:v1.91.2
     ports:
       - 8428:8428
       - 8089:8089
@@ -56,7 +56,7 @@ services:
     restart: always
   vmalert:
     container_name: vmalert
-    image: victoriametrics/vmalert:v1.91.0
+    image: victoriametrics/vmalert:v1.91.2
     depends_on:
       - "victoriametrics"
       - "alertmanager"
diff --git a/deployment/marketplace/digitialocean/one-click-droplet/RELEASE_GUIDE.md b/deployment/marketplace/digitialocean/one-click-droplet/RELEASE_GUIDE.md
index 9cb24a93b..f3c57cf56 100644
--- a/deployment/marketplace/digitialocean/one-click-droplet/RELEASE_GUIDE.md
+++ b/deployment/marketplace/digitialocean/one-click-droplet/RELEASE_GUIDE.md
@@ -8,7 +8,7 @@
 4. Set variables `DIGITALOCEAN_API_TOKEN` with `VM_VERSION` for `packer` environment and run make from example below:
 
 ```console
-make release-victoria-metrics-digitalocean-oneclick-droplet DIGITALOCEAN_API_TOKEN="dop_v23_2e46f4759ceeeba0d0248" VM_VERSION="1.91.0"
+make release-victoria-metrics-digitalocean-oneclick-droplet DIGITALOCEAN_API_TOKEN="dop_v23_2e46f4759ceeeba0d0248" VM_VERSION="1.91.2"
 ```
 
 
diff --git a/deployment/marketplace/digitialocean/one-click-droplet/files/etc/update-motd.d/99-one-click b/deployment/marketplace/digitialocean/one-click-droplet/files/etc/update-motd.d/99-one-click
index 6f2b03fa8..af8b34c11 100755
--- a/deployment/marketplace/digitialocean/one-click-droplet/files/etc/update-motd.d/99-one-click
+++ b/deployment/marketplace/digitialocean/one-click-droplet/files/etc/update-motd.d/99-one-click
@@ -19,8 +19,8 @@ On the server:
   * VictoriaMetrics is running on ports: 8428, 8089, 4242, 2003 and they are bound to the local interface.
 
 ********************************************************************************
-  # This image includes 1.91.0 version of VictoriaMetrics.
-  # See Release notes https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.91.0
+  # This image includes 1.91.2 version of VictoriaMetrics.
+  # See Release notes https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.91.2
 
   # Welcome to VictoriaMetrics droplet!
 
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index e0b917f18..0e1aa29ae 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -24,8 +24,31 @@ The following tip changes can be tested by building VictoriaMetrics components f
 
 ## tip
 
+* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl.html): add verbose output for docker installations or when TTY isn't available. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4081).
+* BUGFIX: add the following command-line flags, which can be used for limiting Graphite API calls: 
+  `--search.maxGraphiteTagKeys` for limiting the number of tag keys returned from Graphite `/tags`, `/tags/autoComplete/*`, `/tags/findSeries` API. 
+  `--search.maxGraphiteTagValues` for limiting the number of tag values returned Graphite `/tags/<tag_name>` API. 
+  Remove redundant limit from [Prometheus api/v1/series](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#prometheus-querying-api-usage). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4339).
+
+## [v1.91.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.91.2)
+
+Released at 2023-06-02
+
+* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): fix nil map assignment panic in runtime introduced in this [change](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4341).
+
+## [v1.91.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.91.1)
+
+Released at 2023-06-01
+
+* FEATURE:[vmagent](https://docs.victoriametrics.com/vmagent.html): Adds `follow_redirects` at service discovery level of scrape configuration. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4282). Thanks to @Haleygo for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4286).
+* FEATURE: vmselect: Decreases startup time for vmselect with a big number of vmstorage nodes. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4364). Thanks to @Haleygo for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4366).
+
 * BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): Properly form path to static assets in WEB UI if `http.pathPrefix` set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4349).
+* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): Properly set datasource query params. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4340). Thanks to @gsakun for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4341).
+* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): properly return empty slices instead of nil for `/api/v1/rules` for groups with present name but absent `rules`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221).
 * BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth.html): Properly handle LOCAL command for proxy protocol. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335#issuecomment-1569864108).
+* BUGFIX: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager.html): Fixes crash on startup. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4378).
+* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix bug with custom URL in global settings not respecting tenantID change. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4322).
 
 ## [v1.91.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.91.0)
 
diff --git a/docs/Cluster-VictoriaMetrics.md b/docs/Cluster-VictoriaMetrics.md
index 05f4cb0a3..dc7d61965 100644
--- a/docs/Cluster-VictoriaMetrics.md
+++ b/docs/Cluster-VictoriaMetrics.md
@@ -1156,6 +1156,10 @@ Below is the output for `/path/to/vmselect -help`:
      The maximum number of time series, which can be returned from /federate. This option allows limiting memory usage (default 1000000)
   -search.maxGraphiteSeries int
      The maximum number of time series, which can be scanned during queries to Graphite Render API. See https://docs.victoriametrics.com/#graphite-render-api-usage (default 300000)
+  -search.maxGraphiteTagKeys int
+     The maximum number of tag keys returned from Graphite /tags, /tags/autoComplete/*, /tags/findSeries API (default 100000)
+  -search.maxGraphiteTagValues int
+     The maximum number of tag values returned Graphite /tags/<tag_name> API (default 100000)   
   -search.maxLookback duration
      Synonym to -search.lookback-delta from Prometheus. The value is dynamically detected from interval between time series datapoints if not set. It can be overridden on per-query basis via max_lookback arg. See also '-search.maxStalenessInterval' flag, which has the same meaining due to historical reasons
   -search.maxMemoryPerQuery size
diff --git a/docs/README.md b/docs/README.md
index f08a832e2..601bcbbe6 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -1469,22 +1469,37 @@ with the enabled de-duplication. See [this section](#deduplication) for details.
 
 ## Deduplication
 
-VictoriaMetrics leaves a single raw sample with the biggest timestamp per each `-dedup.minScrapeInterval` discrete interval
-if `-dedup.minScrapeInterval` is set to positive duration. For example, `-dedup.minScrapeInterval=60s` would leave a single
-raw sample with the biggest timestamp per each discrete 60s interval.
+VictoriaMetrics leaves a single [raw sample](https://docs.victoriametrics.com/keyConcepts.html#raw-samples)
+with the biggest [timestamp](https://en.wikipedia.org/wiki/Unix_time) for each [time series](https://docs.victoriametrics.com/keyConcepts.html#time-series)
+per each `-dedup.minScrapeInterval` discrete interval if `-dedup.minScrapeInterval` is set to positive duration. 
+For example, `-dedup.minScrapeInterval=60s` would leave a single raw sample with the biggest timestamp per each discrete
+`60s` interval.
 This aligns with the [staleness rules in Prometheus](https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness).
 
-If multiple raw samples have the same biggest timestamp on the given `-dedup.minScrapeInterval` discrete interval, then the sample with the biggest value is left.
+If multiple raw samples have **the same timestamp** on the given `-dedup.minScrapeInterval` discrete interval, 
+then the sample with **the biggest value** is kept.
 
-The `-dedup.minScrapeInterval=D` is equivalent to `-downsampling.period=0s:D` if [downsampling](#downsampling) is enabled. So it is safe to use deduplication and downsampling simultaneously.
+Please note, [labels](https://docs.victoriametrics.com/keyConcepts.html#labels) of raw samples should be identical
+in order to be deduplicated. For example, this is why [HA pair of vmagents](https://docs.victoriametrics.com/vmagent.html#high-availability)
+needs to be identically configured. 
 
-The recommended value for `-dedup.minScrapeInterval` must equal to `scrape_interval` config from Prometheus configs. It is recommended to have a single `scrape_interval` across all the scrape targets. See [this article](https://www.robustperception.io/keep-it-simple-scrape_interval-id) for details.
+The `-dedup.minScrapeInterval=D` is equivalent to `-downsampling.period=0s:D` if [downsampling](#downsampling) is enabled.
+So it is safe to use deduplication and downsampling simultaneously.
 
-The de-duplication reduces disk space usage if multiple identically configured [vmagent](https://docs.victoriametrics.com/vmagent.html) or Prometheus instances in HA pair
-write data to the same VictoriaMetrics instance. These vmagent or Prometheus instances must have identical
-`external_labels` section in their configs, so they write data to the same time series. See also [how to set up multiple vmagent instances for scraping the same targets](https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets).
+The recommended value for `-dedup.minScrapeInterval` must equal to `scrape_interval` config from Prometheus configs. 
+It is recommended to have a single `scrape_interval` across all the scrape targets. 
+See [this article](https://www.robustperception.io/keep-it-simple-scrape_interval-id) for details.
 
-It is recommended passing different `-promscrape.cluster.name` values to HA pairs of `vmagent` instances, so the de-duplication consistently leaves samples for one `vmagent` instance and removes duplicate samples from other `vmagent` instances. See [these docs](https://docs.victoriametrics.com/vmagent.html#high-availability) for details.
+The de-duplication reduces disk space usage if multiple **identically configured** [vmagent](https://docs.victoriametrics.com/vmagent.html)
+or Prometheus instances in HA pair write data to the same VictoriaMetrics instance. 
+These vmagent or Prometheus instances must have **identical** `external_labels` section in their configs, 
+so they write data to the same time series. 
+See also [how to set up multiple vmagent instances for scraping the same targets](https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets).
+
+It is recommended passing different `-promscrape.cluster.name` values to each distinct HA pair of `vmagent` instances, 
+so the de-duplication consistently leaves samples for one `vmagent` instance and removes duplicate samples 
+from other `vmagent` instances. 
+See [these docs](https://docs.victoriametrics.com/vmagent.html#high-availability) for details.
 
 ## Storage
 
@@ -2486,6 +2501,10 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      The maximum number of time series, which can be returned from /federate. This option allows limiting memory usage (default 1000000)
   -search.maxGraphiteSeries int
      The maximum number of time series, which can be scanned during queries to Graphite Render API. See https://docs.victoriametrics.com/#graphite-render-api-usage (default 300000)
+  -search.maxGraphiteTagKeys int
+     The maximum number of tag keys returned from Graphite /tags, /tags/autoComplete/*, /tags/findSeries API (default 100000)
+  -search.maxGraphiteTagValues int
+     The maximum number of tag values returned Graphite /tags/<tag_name> API (default 100000)
   -search.maxLookback duration
      Synonym to -search.lookback-delta from Prometheus. The value is dynamically detected from interval between time series datapoints if not set. It can be overridden on per-query basis via max_lookback arg. See also '-search.maxStalenessInterval' flag, which has the same meaning due to historical reasons
   -search.maxMemoryPerQuery size
diff --git a/docs/Single-server-VictoriaMetrics.md b/docs/Single-server-VictoriaMetrics.md
index 7a22a8897..bd0615035 100644
--- a/docs/Single-server-VictoriaMetrics.md
+++ b/docs/Single-server-VictoriaMetrics.md
@@ -1477,22 +1477,37 @@ with the enabled de-duplication. See [this section](#deduplication) for details.
 
 ## Deduplication
 
-VictoriaMetrics leaves a single raw sample with the biggest timestamp per each `-dedup.minScrapeInterval` discrete interval
-if `-dedup.minScrapeInterval` is set to positive duration. For example, `-dedup.minScrapeInterval=60s` would leave a single
-raw sample with the biggest timestamp per each discrete 60s interval.
+VictoriaMetrics leaves a single [raw sample](https://docs.victoriametrics.com/keyConcepts.html#raw-samples)
+with the biggest [timestamp](https://en.wikipedia.org/wiki/Unix_time) for each [time series](https://docs.victoriametrics.com/keyConcepts.html#time-series)
+per each `-dedup.minScrapeInterval` discrete interval if `-dedup.minScrapeInterval` is set to positive duration. 
+For example, `-dedup.minScrapeInterval=60s` would leave a single raw sample with the biggest timestamp per each discrete
+`60s` interval.
 This aligns with the [staleness rules in Prometheus](https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness).
 
-If multiple raw samples have the same biggest timestamp on the given `-dedup.minScrapeInterval` discrete interval, then the sample with the biggest value is left.
+If multiple raw samples have **the same timestamp** on the given `-dedup.minScrapeInterval` discrete interval, 
+then the sample with **the biggest value** is kept.
 
-The `-dedup.minScrapeInterval=D` is equivalent to `-downsampling.period=0s:D` if [downsampling](#downsampling) is enabled. So it is safe to use deduplication and downsampling simultaneously.
+Please note, [labels](https://docs.victoriametrics.com/keyConcepts.html#labels) of raw samples should be identical
+in order to be deduplicated. For example, this is why [HA pair of vmagents](https://docs.victoriametrics.com/vmagent.html#high-availability)
+needs to be identically configured. 
 
-The recommended value for `-dedup.minScrapeInterval` must equal to `scrape_interval` config from Prometheus configs. It is recommended to have a single `scrape_interval` across all the scrape targets. See [this article](https://www.robustperception.io/keep-it-simple-scrape_interval-id) for details.
+The `-dedup.minScrapeInterval=D` is equivalent to `-downsampling.period=0s:D` if [downsampling](#downsampling) is enabled.
+So it is safe to use deduplication and downsampling simultaneously.
 
-The de-duplication reduces disk space usage if multiple identically configured [vmagent](https://docs.victoriametrics.com/vmagent.html) or Prometheus instances in HA pair
-write data to the same VictoriaMetrics instance. These vmagent or Prometheus instances must have identical
-`external_labels` section in their configs, so they write data to the same time series. See also [how to set up multiple vmagent instances for scraping the same targets](https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets).
+The recommended value for `-dedup.minScrapeInterval` must equal to `scrape_interval` config from Prometheus configs. 
+It is recommended to have a single `scrape_interval` across all the scrape targets. 
+See [this article](https://www.robustperception.io/keep-it-simple-scrape_interval-id) for details.
 
-It is recommended passing different `-promscrape.cluster.name` values to HA pairs of `vmagent` instances, so the de-duplication consistently leaves samples for one `vmagent` instance and removes duplicate samples from other `vmagent` instances. See [these docs](https://docs.victoriametrics.com/vmagent.html#high-availability) for details.
+The de-duplication reduces disk space usage if multiple **identically configured** [vmagent](https://docs.victoriametrics.com/vmagent.html)
+or Prometheus instances in HA pair write data to the same VictoriaMetrics instance. 
+These vmagent or Prometheus instances must have **identical** `external_labels` section in their configs, 
+so they write data to the same time series. 
+See also [how to set up multiple vmagent instances for scraping the same targets](https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets).
+
+It is recommended passing different `-promscrape.cluster.name` values to each distinct HA pair of `vmagent` instances, 
+so the de-duplication consistently leaves samples for one `vmagent` instance and removes duplicate samples 
+from other `vmagent` instances. 
+See [these docs](https://docs.victoriametrics.com/vmagent.html#high-availability) for details.
 
 ## Storage
 
@@ -2494,6 +2509,10 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      The maximum number of time series, which can be returned from /federate. This option allows limiting memory usage (default 1000000)
   -search.maxGraphiteSeries int
      The maximum number of time series, which can be scanned during queries to Graphite Render API. See https://docs.victoriametrics.com/#graphite-render-api-usage (default 300000)
+  -search.maxGraphiteTagKeys int
+     The maximum number of tag keys returned from Graphite /tags, /tags/autoComplete/*, /tags/findSeries API (default 100000)
+  -search.maxGraphiteTagValues int
+     The maximum number of tag values returned Graphite /tags/<tag_name> API (default 100000)
   -search.maxLookback duration
      Synonym to -search.lookback-delta from Prometheus. The value is dynamically detected from interval between time series datapoints if not set. It can be overridden on per-query basis via max_lookback arg. See also '-search.maxStalenessInterval' flag, which has the same meaning due to historical reasons
   -search.maxMemoryPerQuery size
diff --git a/docs/vmagent.md b/docs/vmagent.md
index 4209bf4c2..03404db69 100644
--- a/docs/vmagent.md
+++ b/docs/vmagent.md
@@ -763,14 +763,18 @@ See [these docs](https://docs.victoriametrics.com/#deduplication) for details.
 
 ## High availability
 
-It is possible to run multiple identically configured `vmagent` instances or `vmagent` [clusters](#scraping-big-number-of-targets),
-so they [scrape](#how-to-collect-metrics-in-prometheus-format) the same set of targets and push the collected data to the same set of VictoriaMetrics remote storage systems.
+It is possible to run multiple **identically configured** `vmagent` instances or `vmagent` 
+[clusters](#scraping-big-number-of-targets), so they [scrape](#how-to-collect-metrics-in-prometheus-format) 
+the same set of targets and push the collected data to the same set of VictoriaMetrics remote storage systems. 
+Two **identically configured** vmagent instances or clusters is usually called an HA pair.
 
-In this case the deduplication must be configured at VictoriaMetrics in order to de-duplicate samples received from multiple identically configured `vmagent` instances or clusters.
+When running HA pairs, [deduplication](https://docs.victoriametrics.com/#deduplication) must be configured 
+at VictoriaMetrics side in order to de-duplicate received samples.
 See [these docs](https://docs.victoriametrics.com/#deduplication) for details.
 
-It is also recommended passing different values to `-promscrape.cluster.name` command-line flag per each `vmagent` instance or per each `vmagent` cluster in HA setup.
-This is needed for proper data de-duplication. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2679) for details.
+It is also recommended passing different values to `-promscrape.cluster.name` command-line flag per each `vmagent` 
+instance or per each `vmagent` cluster in HA setup. This is needed for proper data de-duplication. 
+See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2679) for details.
 
 ## Scraping targets via a proxy
 
diff --git a/lib/promauth/config.go b/lib/promauth/config.go
index 4c66ec359..602c2b7e0 100644
--- a/lib/promauth/config.go
+++ b/lib/promauth/config.go
@@ -126,6 +126,9 @@ type HTTPClientConfig struct {
 
 	// FollowRedirects specifies whether the client should follow HTTP 3xx redirects.
 	FollowRedirects *bool `yaml:"follow_redirects,omitempty"`
+
+	// EnableHTTP2 specifies whether the client should configure HTTP2.
+	EnableHTTP2 *bool `yaml:"enable_http2,omitempty"`
 }
 
 // ProxyClientConfig represents proxy client config.
diff --git a/lib/promscrape/client.go b/lib/promscrape/client.go
index 76cef03a6..7c156ace2 100644
--- a/lib/promscrape/client.go
+++ b/lib/promscrape/client.go
@@ -11,6 +11,8 @@ import (
 	"strings"
 	"time"
 
+	"golang.org/x/net/http2"
+
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
@@ -160,6 +162,13 @@ func newClient(ctx context.Context, sw *ScrapeWork) *client {
 		// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1017#issuecomment-767235047
 		Timeout: 30 * sw.ScrapeTimeout,
 	}
+	if sw.EnableHTTP2 {
+		_, err := http2.ConfigureTransports(sc.Transport.(*http.Transport))
+		if err != nil {
+			logger.Errorf("failed to configure HTTP/2 transport: %s", err)
+		}
+	}
+
 	if sw.DenyRedirects {
 		sc.CheckRedirect = func(req *http.Request, via []*http.Request) error {
 			return http.ErrUseLastResponse
diff --git a/lib/promscrape/config.go b/lib/promscrape/config.go
index cc1b9529a..186747c79 100644
--- a/lib/promscrape/config.go
+++ b/lib/promscrape/config.go
@@ -992,6 +992,10 @@ func getScrapeWorkConfig(sc *ScrapeConfig, baseDir string, globalCfg *GlobalConf
 	if sc.HTTPClientConfig.FollowRedirects != nil {
 		denyRedirects = !*sc.HTTPClientConfig.FollowRedirects
 	}
+	enableHTTP2 := false
+	if sc.HTTPClientConfig.EnableHTTP2 != nil {
+		enableHTTP2 = !*sc.HTTPClientConfig.EnableHTTP2
+	}
 	metricsPath := sc.MetricsPath
 	if metricsPath == "" {
 		metricsPath = "/metrics"
@@ -1044,6 +1048,7 @@ func getScrapeWorkConfig(sc *ScrapeConfig, baseDir string, globalCfg *GlobalConf
 		honorLabels:          honorLabels,
 		honorTimestamps:      honorTimestamps,
 		denyRedirects:        denyRedirects,
+		enableHTTP2:          enableHTTP2,
 		externalLabels:       externalLabels,
 		relabelConfigs:       relabelConfigs,
 		metricRelabelConfigs: metricRelabelConfigs,
@@ -1074,6 +1079,7 @@ type scrapeWorkConfig struct {
 	honorLabels          bool
 	honorTimestamps      bool
 	denyRedirects        bool
+	enableHTTP2          bool
 	externalLabels       *promutils.Labels
 	relabelConfigs       *promrelabel.ParsedConfigs
 	metricRelabelConfigs *promrelabel.ParsedConfigs
@@ -1351,6 +1357,7 @@ func (swc *scrapeWorkConfig) getScrapeWork(target string, extraLabels, metaLabel
 		HonorLabels:          swc.honorLabels,
 		HonorTimestamps:      swc.honorTimestamps,
 		DenyRedirects:        swc.denyRedirects,
+		EnableHTTP2:          swc.enableHTTP2,
 		OriginalLabels:       originalLabels,
 		Labels:               labelsCopy,
 		ExternalLabels:       swc.externalLabels,
diff --git a/lib/promscrape/discovery/azure/api.go b/lib/promscrape/discovery/azure/api.go
index bd580cec0..c81758784 100644
--- a/lib/promscrape/discovery/azure/api.go
+++ b/lib/promscrape/discovery/azure/api.go
@@ -110,7 +110,7 @@ func newAPIConfig(sdc *SDConfig, baseDir string) (*apiConfig, error) {
 	if err != nil {
 		return nil, err
 	}
-	c, err := discoveryutils.NewClient(env.ResourceManagerEndpoint, ac, sdc.ProxyURL, proxyAC, sdc.HTTPClientConfig.FollowRedirects)
+	c, err := discoveryutils.NewClient(env.ResourceManagerEndpoint, ac, sdc.ProxyURL, proxyAC, sdc.HTTPClientConfig)
 	if err != nil {
 		return nil, fmt.Errorf("cannot create client for %q: %w", env.ResourceManagerEndpoint, err)
 	}
@@ -230,7 +230,7 @@ func getRefreshTokenFunc(sdc *SDConfig, ac, proxyAC *promauth.Config, env *cloud
 		return nil, fmt.Errorf("unsupported `authentication_method: %q` only `OAuth` and `ManagedIdentity` are supported", authenticationMethod)
 	}
 
-	authClient, err := discoveryutils.NewClient(tokenEndpoint, ac, sdc.ProxyURL, proxyAC, sdc.HTTPClientConfig.FollowRedirects)
+	authClient, err := discoveryutils.NewClient(tokenEndpoint, ac, sdc.ProxyURL, proxyAC, sdc.HTTPClientConfig)
 	if err != nil {
 		return nil, fmt.Errorf("cannot build auth client: %w", err)
 	}
diff --git a/lib/promscrape/discovery/consul/api.go b/lib/promscrape/discovery/consul/api.go
index dabd10ead..af637999a 100644
--- a/lib/promscrape/discovery/consul/api.go
+++ b/lib/promscrape/discovery/consul/api.go
@@ -80,7 +80,7 @@ func newAPIConfig(sdc *SDConfig, baseDir string) (*apiConfig, error) {
 	if err != nil {
 		return nil, fmt.Errorf("cannot parse proxy auth config: %w", err)
 	}
-	client, err := discoveryutils.NewClient(apiServer, ac, sdc.ProxyURL, proxyAC, sdc.HTTPClientConfig.FollowRedirects)
+	client, err := discoveryutils.NewClient(apiServer, ac, sdc.ProxyURL, proxyAC, sdc.HTTPClientConfig)
 	if err != nil {
 		return nil, fmt.Errorf("cannot create HTTP client for %q: %w", apiServer, err)
 	}
diff --git a/lib/promscrape/discovery/consulagent/api.go b/lib/promscrape/discovery/consulagent/api.go
index 9d5c4761e..c9c9456df 100644
--- a/lib/promscrape/discovery/consulagent/api.go
+++ b/lib/promscrape/discovery/consulagent/api.go
@@ -74,7 +74,7 @@ func newAPIConfig(sdc *SDConfig, baseDir string) (*apiConfig, error) {
 	if err != nil {
 		return nil, fmt.Errorf("cannot parse proxy auth config: %w", err)
 	}
-	client, err := discoveryutils.NewClient(apiServer, ac, sdc.ProxyURL, proxyAC, sdc.HTTPClientConfig.FollowRedirects)
+	client, err := discoveryutils.NewClient(apiServer, ac, sdc.ProxyURL, proxyAC, sdc.HTTPClientConfig)
 	if err != nil {
 		return nil, fmt.Errorf("cannot create HTTP client for %q: %w", apiServer, err)
 	}
diff --git a/lib/promscrape/discovery/digitalocean/api.go b/lib/promscrape/discovery/digitalocean/api.go
index 7d4cad4bb..220fb2b55 100644
--- a/lib/promscrape/discovery/digitalocean/api.go
+++ b/lib/promscrape/discovery/digitalocean/api.go
@@ -36,7 +36,7 @@ func newAPIConfig(sdc *SDConfig, baseDir string) (*apiConfig, error) {
 	if err != nil {
 		return nil, fmt.Errorf("cannot parse proxy auth config: %w", err)
 	}
-	client, err := discoveryutils.NewClient(apiServer, ac, sdc.ProxyURL, proxyAC, sdc.HTTPClientConfig.FollowRedirects)
+	client, err := discoveryutils.NewClient(apiServer, ac, sdc.ProxyURL, proxyAC, sdc.HTTPClientConfig)
 	if err != nil {
 		return nil, fmt.Errorf("cannot create HTTP client for %q: %w", apiServer, err)
 	}
diff --git a/lib/promscrape/discovery/docker/api.go b/lib/promscrape/discovery/docker/api.go
index 8301c3fd5..dd0ff7ed0 100644
--- a/lib/promscrape/discovery/docker/api.go
+++ b/lib/promscrape/discovery/docker/api.go
@@ -50,7 +50,7 @@ func newAPIConfig(sdc *SDConfig, baseDir string) (*apiConfig, error) {
 	if err != nil {
 		return nil, fmt.Errorf("cannot parse proxy auth config: %w", err)
 	}
-	client, err := discoveryutils.NewClient(sdc.Host, ac, sdc.ProxyURL, proxyAC, sdc.HTTPClientConfig.FollowRedirects)
+	client, err := discoveryutils.NewClient(sdc.Host, ac, sdc.ProxyURL, proxyAC, sdc.HTTPClientConfig)
 	if err != nil {
 		return nil, fmt.Errorf("cannot create HTTP client for %q: %w", sdc.Host, err)
 	}
diff --git a/lib/promscrape/discovery/dockerswarm/api.go b/lib/promscrape/discovery/dockerswarm/api.go
index 160e00d3d..0a76e644c 100644
--- a/lib/promscrape/discovery/dockerswarm/api.go
+++ b/lib/promscrape/discovery/dockerswarm/api.go
@@ -49,7 +49,7 @@ func newAPIConfig(sdc *SDConfig, baseDir string) (*apiConfig, error) {
 	if err != nil {
 		return nil, fmt.Errorf("cannot parse proxy auth config: %w", err)
 	}
-	client, err := discoveryutils.NewClient(sdc.Host, ac, sdc.ProxyURL, proxyAC, sdc.HTTPClientConfig.FollowRedirects)
+	client, err := discoveryutils.NewClient(sdc.Host, ac, sdc.ProxyURL, proxyAC, sdc.HTTPClientConfig)
 	if err != nil {
 		return nil, fmt.Errorf("cannot create HTTP client for %q: %w", sdc.Host, err)
 	}
diff --git a/lib/promscrape/discovery/eureka/api.go b/lib/promscrape/discovery/eureka/api.go
index 03670870a..9373e7edb 100644
--- a/lib/promscrape/discovery/eureka/api.go
+++ b/lib/promscrape/discovery/eureka/api.go
@@ -34,7 +34,7 @@ func newAPIConfig(sdc *SDConfig, baseDir string) (*apiConfig, error) {
 	if err != nil {
 		return nil, fmt.Errorf("cannot parse proxy auth config: %w", err)
 	}
-	client, err := discoveryutils.NewClient(apiServer, ac, sdc.ProxyURL, proxyAC, sdc.HTTPClientConfig.FollowRedirects)
+	client, err := discoveryutils.NewClient(apiServer, ac, sdc.ProxyURL, proxyAC, sdc.HTTPClientConfig)
 	if err != nil {
 		return nil, fmt.Errorf("cannot create HTTP client for %q: %w", apiServer, err)
 	}
diff --git a/lib/promscrape/discovery/http/api.go b/lib/promscrape/discovery/http/api.go
index 107167faf..fcccdf929 100644
--- a/lib/promscrape/discovery/http/api.go
+++ b/lib/promscrape/discovery/http/api.go
@@ -44,7 +44,7 @@ func newAPIConfig(sdc *SDConfig, baseDir string) (*apiConfig, error) {
 	if err != nil {
 		return nil, fmt.Errorf("cannot parse proxy auth config: %w", err)
 	}
-	client, err := discoveryutils.NewClient(apiServer, ac, sdc.ProxyURL, proxyAC, sdc.HTTPClientConfig.FollowRedirects)
+	client, err := discoveryutils.NewClient(apiServer, ac, sdc.ProxyURL, proxyAC, sdc.HTTPClientConfig)
 	if err != nil {
 		return nil, fmt.Errorf("cannot create HTTP client for %q: %w", apiServer, err)
 	}
diff --git a/lib/promscrape/discovery/kuma/api.go b/lib/promscrape/discovery/kuma/api.go
index c160d15e6..0a30818b8 100644
--- a/lib/promscrape/discovery/kuma/api.go
+++ b/lib/promscrape/discovery/kuma/api.go
@@ -60,7 +60,7 @@ func newAPIConfig(sdc *SDConfig, baseDir string) (*apiConfig, error) {
 	if err != nil {
 		return nil, fmt.Errorf("cannot parse proxy auth config: %w", err)
 	}
-	client, err := discoveryutils.NewClient(apiServer, ac, sdc.ProxyURL, proxyAC, sdc.HTTPClientConfig.FollowRedirects)
+	client, err := discoveryutils.NewClient(apiServer, ac, sdc.ProxyURL, proxyAC, sdc.HTTPClientConfig)
 	if err != nil {
 		return nil, fmt.Errorf("cannot create HTTP client for %q: %w", apiServer, err)
 	}
diff --git a/lib/promscrape/discovery/nomad/api.go b/lib/promscrape/discovery/nomad/api.go
index 33aefa295..2b8208ca6 100644
--- a/lib/promscrape/discovery/nomad/api.go
+++ b/lib/promscrape/discovery/nomad/api.go
@@ -68,7 +68,7 @@ func newAPIConfig(sdc *SDConfig, baseDir string) (*apiConfig, error) {
 	if err != nil {
 		return nil, fmt.Errorf("cannot parse proxy auth config: %w", err)
 	}
-	client, err := discoveryutils.NewClient(apiServer, ac, sdc.ProxyURL, proxyAC, sdc.HTTPClientConfig.FollowRedirects)
+	client, err := discoveryutils.NewClient(apiServer, ac, sdc.ProxyURL, proxyAC, sdc.HTTPClientConfig)
 	if err != nil {
 		return nil, fmt.Errorf("cannot create HTTP client for %q: %w", apiServer, err)
 	}
diff --git a/lib/promscrape/discoveryutils/client.go b/lib/promscrape/discoveryutils/client.go
index e9d1bde1f..ba52a51d2 100644
--- a/lib/promscrape/discoveryutils/client.go
+++ b/lib/promscrape/discoveryutils/client.go
@@ -13,6 +13,8 @@ import (
 	"sync"
 	"time"
 
+	"golang.org/x/net/http2"
+
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promauth"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/proxy"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/timerpool"
@@ -83,7 +85,7 @@ type HTTPClient struct {
 var defaultDialer = &net.Dialer{}
 
 // NewClient returns new Client for the given args.
-func NewClient(apiServer string, ac *promauth.Config, proxyURL *proxy.URL, proxyAC *promauth.Config, followRedirects *bool) (*Client, error) {
+func NewClient(apiServer string, ac *promauth.Config, proxyURL *proxy.URL, proxyAC *promauth.Config, httpCfg promauth.HTTPClientConfig) (*Client, error) {
 	u, err := url.Parse(apiServer)
 	if err != nil {
 		return nil, fmt.Errorf("cannot parse apiServer=%q: %w", apiServer, err)
@@ -139,7 +141,7 @@ func NewClient(apiServer string, ac *promauth.Config, proxyURL *proxy.URL, proxy
 			ac.SetHeaders(req, true)
 		}
 	}
-	if followRedirects != nil && !*followRedirects {
+	if httpCfg.FollowRedirects != nil && !*httpCfg.FollowRedirects {
 		client.CheckRedirect = func(req *http.Request, via []*http.Request) error {
 			return http.ErrUseLastResponse
 		}
@@ -153,6 +155,12 @@ func NewClient(apiServer string, ac *promauth.Config, proxyURL *proxy.URL, proxy
 			proxyURL.SetHeaders(proxyAC, req)
 		}
 	}
+	if httpCfg.EnableHTTP2 != nil && *httpCfg.EnableHTTP2 {
+		_, err := http2.ConfigureTransports(client.Transport.(*http.Transport))
+		if err != nil {
+			return nil, fmt.Errorf("failed to configure HTTP/2 transport: %s", err)
+		}
+	}
 	ctx, cancel := context.WithCancel(context.Background())
 
 	c := &Client{
diff --git a/lib/promscrape/scrapework.go b/lib/promscrape/scrapework.go
index 961b69c2b..444805164 100644
--- a/lib/promscrape/scrapework.go
+++ b/lib/promscrape/scrapework.go
@@ -66,6 +66,9 @@ type ScrapeWork struct {
 	// Whether to deny redirects during requests to scrape config.
 	DenyRedirects bool
 
+	// Whether to configure HTTP2.
+	EnableHTTP2 bool
+
 	// OriginalLabels contains original labels before relabeling.
 	//
 	// These labels are needed for relabeling troubleshooting at /targets page.
@@ -399,7 +402,8 @@ func (sw *scrapeWork) mustSwitchToStreamParseMode(responseSize int) bool {
 
 // getTargetResponse() fetches response from sw target in the same way as when scraping the target.
 func (sw *scrapeWork) getTargetResponse() ([]byte, error) {
-	if *streamParse || sw.Config.StreamParse || sw.mustSwitchToStreamParseMode(sw.prevBodyLen) {
+	// use stream reader when stream mode enabled or http2 enabled
+	if *streamParse || sw.Config.StreamParse || sw.mustSwitchToStreamParseMode(sw.prevBodyLen) || sw.Config.EnableHTTP2 {
 		// Read the response in stream mode.
 		sr, err := sw.GetStreamReader()
 		if err != nil {


Test patch:
diff --git a/app/vmalert/datasource/vm_test.go b/app/vmalert/datasource/vm_test.go
index 3ae1bf173..3df8805c2 100644
--- a/app/vmalert/datasource/vm_test.go
+++ b/app/vmalert/datasource/vm_test.go
@@ -378,6 +378,9 @@ func TestRequestParams(t *testing.T) {
 	}
 	query := "up"
 	timestamp := time.Date(2001, 2, 3, 4, 5, 6, 0, time.UTC)
+	storage := VMStorage{
+		extraParams: url.Values{"round_digits": {"10"}},
+	}
 	testCases := []struct {
 		name       string
 		queryRange bool
@@ -574,6 +577,17 @@ func TestRequestParams(t *testing.T) {
 				checkEqualString(t, exp, r.URL.RawQuery)
 			},
 		},
+		{
+			"custom params overrides the original params",
+			false,
+			storage.Clone().ApplyParams(QuerierParams{
+				QueryParams: url.Values{"round_digits": {"2"}},
+			}),
+			func(t *testing.T, r *http.Request) {
+				exp := fmt.Sprintf("query=%s&round_digits=2&time=%d", query, timestamp.Unix())
+				checkEqualString(t, exp, r.URL.RawQuery)
+			},
+		},
 		{
 			"graphite extra params",
 			false,
diff --git a/app/vmctl/remote_read_test.go b/app/vmctl/remote_read_test.go
index bfb19b2cd..dcb6c6889 100644
--- a/app/vmctl/remote_read_test.go
+++ b/app/vmctl/remote_read_test.go
@@ -154,10 +154,12 @@ func TestRemoteRead(t *testing.T) {
 					timeEnd:   &end,
 					chunk:     tt.chunk,
 				},
-				cc: 1,
+				cc:        1,
+				isSilent:  true,
+				isVerbose: false,
 			}
 
-			err = rmp.run(ctx, true, false)
+			err = rmp.run(ctx)
 			if err != nil {
 				t.Fatalf("failed to run remote read processor: %s", err)
 			}
@@ -307,10 +309,12 @@ func TestSteamRemoteRead(t *testing.T) {
 					timeEnd:   &end,
 					chunk:     tt.chunk,
 				},
-				cc: 1,
+				cc:        1,
+				isSilent:  true,
+				isVerbose: false,
 			}
 
-			err = rmp.run(ctx, true, false)
+			err = rmp.run(ctx)
 			if err != nil {
 				t.Fatalf("failed to run remote read processor: %s", err)
 			}
diff --git a/app/vmctl/vm_native_test.go b/app/vmctl/vm_native_test.go
index 72313cf1e..457437f75 100644
--- a/app/vmctl/vm_native_test.go
+++ b/app/vmctl/vm_native_test.go
@@ -227,9 +227,10 @@ func Test_vmNativeProcessor_run(t *testing.T) {
 				rateLimit:    tt.fields.rateLimit,
 				interCluster: tt.fields.interCluster,
 				cc:           tt.fields.cc,
+				isSilent:     tt.args.silent,
 			}
 
-			if err := p.run(tt.args.ctx, tt.args.silent); (err != nil) != tt.wantErr {
+			if err := p.run(tt.args.ctx); (err != nil) != tt.wantErr {
 				t.Errorf("run() error = %v, wantErr %v", err, tt.wantErr)
 			}
 			deleted, err := deleteSeries(tt.fields.matchName, tt.fields.matchValue)
diff --git a/lib/promscrape/discovery/azure/machine_test.go b/lib/promscrape/discovery/azure/machine_test.go
index 90141999e..8d778f65e 100644
--- a/lib/promscrape/discovery/azure/machine_test.go
+++ b/lib/promscrape/discovery/azure/machine_test.go
@@ -9,6 +9,7 @@ import (
 	"testing"
 	"time"
 
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promauth"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discoveryutils"
 )
 
@@ -66,7 +67,7 @@ func TestGetVirtualMachinesSuccess(t *testing.T) {
 				}
 			}))
 			defer testServer.Close()
-			c, err := discoveryutils.NewClient(testServer.URL, nil, nil, nil, nil)
+			c, err := discoveryutils.NewClient(testServer.URL, nil, nil, nil, promauth.HTTPClientConfig{})
 			if err != nil {
 				t.Fatalf("unexpected error at client create: %s", err)
 			}
diff --git a/lib/promscrape/discoveryutils/client_test.go b/lib/promscrape/discoveryutils/client_test.go
new file mode 100644
index 000000000..5882f2359
--- /dev/null
+++ b/lib/promscrape/discoveryutils/client_test.go
@@ -0,0 +1,111 @@
+package discoveryutils
+
+import (
+	"fmt"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promauth"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/proxy"
+)
+
+func newTestServer(handler func(w http.ResponseWriter, r *http.Request)) (*httptest.Server, error) {
+	testServer := httptest.NewUnstartedServer(http.HandlerFunc(handler))
+	testServer.Start()
+	return testServer, nil
+}
+
+func TestNewClientFromConfig(t *testing.T) {
+	allowed := true
+	notAllowed := false
+	newClientValidConfig := []struct {
+		httpCfg         promauth.HTTPClientConfig
+		handler         func(w http.ResponseWriter, r *http.Request)
+		expectedMessage string
+	}{
+		{
+			httpCfg: promauth.HTTPClientConfig{
+				FollowRedirects: &allowed,
+			},
+			handler: func(w http.ResponseWriter, r *http.Request) {
+				switch r.URL.Path {
+				case "/redirected":
+					fmt.Fprint(w, "I'm here to serve you!!!")
+				default:
+					w.Header().Set("Location", "/redirected")
+					w.WriteHeader(http.StatusFound)
+					fmt.Fprint(w, "It should follow the redirect.")
+				}
+			},
+			expectedMessage: "I'm here to serve you!!!",
+		},
+		{
+			httpCfg: promauth.HTTPClientConfig{
+				FollowRedirects: &notAllowed,
+			},
+			handler: func(w http.ResponseWriter, r *http.Request) {
+				switch r.URL.Path {
+				case "/redirected":
+					fmt.Fprint(w, "The redirection was followed.")
+				default:
+					w.Header().Set("Location", "/redirected")
+					w.WriteHeader(http.StatusFound)
+					fmt.Fprint(w, "I'm before redirect")
+				}
+			},
+			expectedMessage: "I'm before redirect",
+		},
+		{
+			httpCfg: promauth.HTTPClientConfig{
+				EnableHTTP2: &allowed,
+			},
+			handler: func(w http.ResponseWriter, r *http.Request) {
+				switch r.URL.Path {
+				case "/redirected":
+					fmt.Fprint(w, "I'm here to serve you!!!")
+				default:
+					w.Header().Set("Location", "/redirected")
+					w.WriteHeader(http.StatusFound)
+					fmt.Fprint(w, "I'm before redirect")
+				}
+			},
+			expectedMessage: "I'm here to serve you!!!",
+		},
+	}
+
+	for _, validConfig := range newClientValidConfig {
+		testServer, err := newTestServer(validConfig.handler)
+		if err != nil {
+			t.Fatal(err.Error())
+		}
+		defer testServer.Close()
+
+		client, err := NewClient("http://0.0.0.0:1234", nil, &proxy.URL{}, nil, validConfig.httpCfg)
+		if err != nil {
+			t.Errorf("Can't create a client from this config: %+v", validConfig.httpCfg)
+			continue
+		}
+
+		response, err := client.client.client.Get(testServer.URL)
+		if err != nil {
+			t.Errorf("Can't connect to the test server using this config: %+v: %v", validConfig.httpCfg, err)
+			continue
+		}
+
+		message, err := io.ReadAll(response.Body)
+		response.Body.Close()
+		if err != nil {
+			t.Errorf("Can't read the server response body using this config: %+v", validConfig.httpCfg)
+			continue
+		}
+
+		trimMessage := strings.TrimSpace(string(message))
+		if validConfig.expectedMessage != trimMessage {
+			t.Errorf("The expected message (%s) differs from the obtained message (%s) using this config: %+v",
+				validConfig.expectedMessage, trimMessage, validConfig.httpCfg)
+		}
+	}
+}
diff --git a/lib/promscrape/testdata/prometheus.yml b/lib/promscrape/testdata/prometheus.yml
index ecae9e7a3..62e2d76d5 100644
--- a/lib/promscrape/testdata/prometheus.yml
+++ b/lib/promscrape/testdata/prometheus.yml
@@ -7,6 +7,7 @@ scrape_configs:
     honor_labels: true
     honor_timestamps: false
     follow_redirects: false
+    enable_http2: true
     static_configs:
       - targets: ["foo.bar", "aaa"]
         labels:
@@ -21,6 +22,7 @@ scrape_configs:
       - role: endpoints
         api_server: "https://localhost:1234"
         follow_redirects: true
+        enable_http2: true
         tls_config:
           cert_file: valid_cert_file
           key_file: valid_key_file