Repository: VictoriaMetrics/VictoriaMetrics
Pull request #5004 (2023-09-13 14:45:13)

vmui: update information about tsdb usage in cluster version
This improvement adds explanations and expectations to the number of label values on Cardinality Explorer. This explanation is only visible for cluster installation because only in the cluster version the same label could have different values on each node.
Information was added to the documentation module. 

<img width="1678" alt="Screenshot 2023-09-19 at 12 18 02" src="https://github.com/VictoriaMetrics/VictoriaMetrics/assets/12189570/7aadbe57-60a7-4134-9669-d8f825900a9b">


Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3070


 Signed-off-by: dmitryk-dk [d.kozlov@victoriametrics.com](mailto:d.kozlov@victoriametrics.com)

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5004?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) Report
Patch and project coverage have no change.
> Comparison is base [(`cbcfbaf`)](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/commit/cbcfbaf4887b9ffca3e30e1abb03cf8acc9e2b98?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) 60.40% compared to head [(`8e8009e`)](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5004?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) 60.40%.
> Report is 2 commits behind head on master.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##           master    #5004   +/-   ##
=======================================
  Coverage   60.40%   60.40%           
=======================================
  Files         388      388           
  Lines       72732    72732           
=======================================
+ Hits        43932    43937    +5     
+ Misses      26402    26398    -4     
+ Partials     2398     2397    -1     
```


[see 7 files with indirect coverage changes](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5004/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5004?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).


Patch:
diff --git a/README.md b/README.md
index 1418cf658..aab85c42e 100644
--- a/README.md
+++ b/README.md
@@ -110,6 +110,7 @@ Case studies:
 * [Brandwatch](https://docs.victoriametrics.com/CaseStudies.html#brandwatch)
 * [CERN](https://docs.victoriametrics.com/CaseStudies.html#cern)
 * [COLOPL](https://docs.victoriametrics.com/CaseStudies.html#colopl)
+* [Criteo](https://docs.victoriametrics.com/CaseStudies.html#criteo)
 * [Dig Security](https://docs.victoriametrics.com/CaseStudies.html#dig-security)
 * [Fly.io](https://docs.victoriametrics.com/CaseStudies.html#flyio)
 * [German Research Center for Artificial Intelligence](https://docs.victoriametrics.com/CaseStudies.html#german-research-center-for-artificial-intelligence)
@@ -404,14 +405,16 @@ matching the specified [series selector](https://prometheus.io/docs/prometheus/l
 
 Cardinality explorer is built on top of [/api/v1/status/tsdb](#tsdb-stats).
 
+See [cardinality explorer playground](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/cardinality).
+See the example of using the cardinality explorer [here](https://victoriametrics.com/blog/cardinality-explorer/).
+
+## Cardinality explorer statistic inaccuracy
+
 In [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html) each vmstorage tracks the stored time series individually.
 vmselect requests stats via [/api/v1/status/tsdb](#tsdb-stats) API from each vmstorage node and merges the results by summing per-series stats.
 This may lead to inflated values when samples for the same time series are spread across multiple vmstorage nodes
 due to [replication](#replication) or [rerouting](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html?highlight=re-routes#cluster-availability).
 
-See [cardinality explorer playground](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/cardinality).
-See the example of using the cardinality explorer [here](https://victoriametrics.com/blog/cardinality-explorer/).
-
 ## How to apply new config to VictoriaMetrics
 
 VictoriaMetrics is configured via command-line flags, so it must be restarted when new command-line flags should be applied:
@@ -616,6 +619,28 @@ Some plugins for Telegraf such as [fluentd](https://github.com/fangli/fluent-plu
 or [Juniper/jitmon](https://github.com/Juniper/jtimon) send `SHOW DATABASES` query to `/query` and expect a particular database name in the response.
 Comma-separated list of expected databases can be passed to VictoriaMetrics via `-influx.databaseNames` command-line flag.
 
+### How to send data in InfluxDB v2 format
+
+VictoriaMetrics exposes endpoint for InfluxDB v2 HTTP API at `/influx/api/v2/write` and `/api/v2/write`.
+
+
+In order to write data with InfluxDB line protocol to local VictoriaMetrics using `curl`:
+
+<div class="with-copy" markdown="1">
+
+```console
+curl -d 'measurement,tag1=value1,tag2=value2 field1=123,field2=1.23' -X POST 'http://localhost:8428/api/v2/write'
+```
+
+</div>
+
+The `/api/v1/export` endpoint should return the following response:
+
+```json
+{"metric":{"__name__":"measurement_field1","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1695902762311]}
+{"metric":{"__name__":"measurement_field2","tag1":"value1","tag2":"value2"},"values":[1.23],"timestamps":[1695902762311]}
+```
+
 ## How to send data from Graphite-compatible agents such as [StatsD](https://github.com/etsy/statsd)
 
 Enable Graphite receiver in VictoriaMetrics by setting `-graphiteListenAddr` command line flag. For instance,
@@ -1694,6 +1719,10 @@ Downsampling happens during [background merges](https://docs.victoriametrics.com
 and can't be performed if there is not enough of free disk space or if vmstorage 
 is in [read-only mode](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#readonly-mode).
 
+Please, note that intervals of `-downsampling.period` must be multiples of each other. 
+In case [deduplication](https://docs.victoriametrics.com/#deduplication) is enabled value of `-dedup.minScrapeInterval` must also be multiple of `-downsampling.period` intervals.
+This is required to ensure consistency of deduplication and downsampling results.
+
 The downsampling can be evaluated for free by downloading and using enterprise binaries from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
 
 ## Multi-tenancy
@@ -1933,9 +1962,6 @@ and [cardinality explorer docs](#cardinality-explorer).
   has at least 20% of free space. The remaining amount of free space
   can be [monitored](#monitoring) via `vm_free_disk_space_bytes` metric. The total size of data
   stored on the disk can be monitored via sum of `vm_data_size_bytes` metrics.
-  See also `vm_merge_need_free_disk_space` metrics, which are set to values higher than 0
-  if background merge cannot be initiated due to free disk space shortage. The value shows the number of per-month partitions,
-  which would start background merge if they had more free disk space.
 
 * VictoriaMetrics buffers incoming data in memory for up to a few seconds before flushing it to persistent storage.
   This may lead to the following "issues":
diff --git a/app/vlinsert/elasticsearch/elasticsearch.go b/app/vlinsert/elasticsearch/elasticsearch.go
index 60ae8ddeb..a3ae68f19 100644
--- a/app/vlinsert/elasticsearch/elasticsearch.go
+++ b/app/vlinsert/elasticsearch/elasticsearch.go
@@ -109,6 +109,12 @@ func RequestHandler(path string, w http.ResponseWriter, r *http.Request) bool {
 		defer bufferedwriter.Put(bw)
 		WriteBulkResponse(bw, n, tookMs)
 		_ = bw.Flush()
+
+		// update bulkRequestDuration only for successfully parsed requests
+		// There is no need in updating bulkRequestDuration for request errors,
+		// since their timings are usually much smaller than the timing for successful request parsing.
+		bulkRequestDuration.UpdateDuration(startTime)
+
 		return true
 	default:
 		return false
@@ -116,7 +122,9 @@ func RequestHandler(path string, w http.ResponseWriter, r *http.Request) bool {
 }
 
 var (
-	bulkRequestsTotal = metrics.NewCounter(`vl_http_requests_total{path="/insert/elasticsearch/_bulk"}`)
+	bulkRequestsTotal   = metrics.NewCounter(`vl_http_requests_total{path="/insert/elasticsearch/_bulk"}`)
+	rowsIngestedTotal   = metrics.NewCounter(`vl_rows_ingested_total{type="elasticsearch_bulk"}`)
+	bulkRequestDuration = metrics.NewHistogram(`vl_http_request_duration_seconds{path="/insert/elasticsearch/_bulk"}`)
 )
 
 func readBulkRequest(r io.Reader, isGzip bool, timeField, msgField string,
@@ -162,8 +170,6 @@ func readBulkRequest(r io.Reader, isGzip bool, timeField, msgField string,
 
 var lineBufferPool bytesutil.ByteBufferPool
 
-var rowsIngestedTotal = metrics.NewCounter(`vl_rows_ingested_total{type="elasticsearch_bulk"}`)
-
 func readBulkLine(sc *bufio.Scanner, timeField, msgField string,
 	processLogMessage func(timestamp int64, fields []logstorage.Field),
 ) (bool, error) {
diff --git a/app/vlinsert/insertutils/common_params.go b/app/vlinsert/insertutils/common_params.go
index 97bf5c280..23f100775 100644
--- a/app/vlinsert/insertutils/common_params.go
+++ b/app/vlinsert/insertutils/common_params.go
@@ -3,12 +3,13 @@ package insertutils
 import (
 	"net/http"
 
+	"github.com/VictoriaMetrics/metrics"
+
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlstorage"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httputils"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"
-	"github.com/VictoriaMetrics/metrics"
 )
 
 // CommonParams contains common HTTP parameters used by log ingestion APIs.
@@ -73,12 +74,19 @@ func GetCommonParams(r *http.Request) (*CommonParams, error) {
 // GetProcessLogMessageFunc returns a function, which adds parsed log messages to lr.
 func (cp *CommonParams) GetProcessLogMessageFunc(lr *logstorage.LogRows) func(timestamp int64, fields []logstorage.Field) {
 	return func(timestamp int64, fields []logstorage.Field) {
+		if len(fields) > *MaxFieldsPerLine {
+			rf := logstorage.RowFormatter(fields)
+			logger.Warnf("dropping log line with %d fields; it exceeds -insert.maxFieldsPerLine=%d; %s", len(fields), *MaxFieldsPerLine, rf)
+			rowsDroppedTotalTooManyFields.Inc()
+			return
+		}
+
 		lr.MustAdd(cp.TenantID, timestamp, fields)
 		if cp.Debug {
 			s := lr.GetRowString(0)
 			lr.ResetKeepSettings()
 			logger.Infof("remoteAddr=%s; requestURI=%s; ignoring log entry because of `debug` query arg: %s", cp.DebugRemoteAddr, cp.DebugRequestURI, s)
-			rowsDroppedTotal.Inc()
+			rowsDroppedTotalDebug.Inc()
 			return
 		}
 		if lr.NeedFlush() {
@@ -88,4 +96,5 @@ func (cp *CommonParams) GetProcessLogMessageFunc(lr *logstorage.LogRows) func(ti
 	}
 }
 
-var rowsDroppedTotal = metrics.NewCounter(`vl_rows_dropped_total{reason="debug"}`)
+var rowsDroppedTotalDebug = metrics.NewCounter(`vl_rows_dropped_total{reason="debug"}`)
+var rowsDroppedTotalTooManyFields = metrics.NewCounter(`vl_rows_dropped_total{reason="too_many_fields"}`)
diff --git a/app/vlinsert/insertutils/flags.go b/app/vlinsert/insertutils/flags.go
index 8f070488c..403e315f2 100644
--- a/app/vlinsert/insertutils/flags.go
+++ b/app/vlinsert/insertutils/flags.go
@@ -1,10 +1,15 @@
 package insertutils
 
 import (
+	"flag"
+
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
 )
 
 var (
 	// MaxLineSizeBytes is the maximum length of a single line for /insert/* handlers
 	MaxLineSizeBytes = flagutil.NewBytes("insert.maxLineSizeBytes", 256*1024, "The maximum size of a single line, which can be read by /insert/* handlers")
+
+	// MaxFieldsPerLine is the maximum number of fields per line for /insert/* handlers
+	MaxFieldsPerLine = flag.Int("insert.maxFieldsPerLine", 1000, "The maximum number of log fields per line, which can be read by /insert/* handlers")
 )
diff --git a/app/vlinsert/jsonline/jsonline.go b/app/vlinsert/jsonline/jsonline.go
index 1b901db36..bf8d4760e 100644
--- a/app/vlinsert/jsonline/jsonline.go
+++ b/app/vlinsert/jsonline/jsonline.go
@@ -21,6 +21,7 @@ import (
 
 // RequestHandler processes jsonline insert requests
 func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
+	startTime := time.Now()
 	w.Header().Add("Content-Type", "application/json")
 
 	if r.Method != "POST" {
@@ -77,6 +78,11 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 	vlstorage.MustAddRows(lr)
 	logstorage.PutLogRows(lr)
 
+	// update jsonlineRequestDuration only for successfully parsed requests.
+	// There is no need in updating jsonlineRequestDuration for request errors,
+	// since their timings are usually much smaller than the timing for successful request parsing.
+	jsonlineRequestDuration.UpdateDuration(startTime)
+
 	return true
 }
 
@@ -144,6 +150,7 @@ func parseISO8601Timestamp(s string) (int64, error) {
 var lineBufferPool bytesutil.ByteBufferPool
 
 var (
-	requestsTotal     = metrics.NewCounter(`vl_http_requests_total{path="/insert/jsonline"}`)
-	rowsIngestedTotal = metrics.NewCounter(`vl_rows_ingested_total{type="jsonline"}`)
+	requestsTotal           = metrics.NewCounter(`vl_http_requests_total{path="/insert/jsonline"}`)
+	rowsIngestedTotal       = metrics.NewCounter(`vl_rows_ingested_total{type="jsonline"}`)
+	jsonlineRequestDuration = metrics.NewHistogram(`vl_http_request_duration_seconds{path="/insert/jsonline"}`)
 )
diff --git a/app/vlinsert/loki/loki.go b/app/vlinsert/loki/loki.go
index 09faa742c..e7a3c0b7d 100644
--- a/app/vlinsert/loki/loki.go
+++ b/app/vlinsert/loki/loki.go
@@ -3,17 +3,10 @@ package loki
 import (
 	"net/http"
 
-	"github.com/VictoriaMetrics/metrics"
-
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vlinsert/insertutils"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logstorage"
 )
 
-var (
-	lokiRequestsJSONTotal     = metrics.NewCounter(`vl_http_requests_total{path="/insert/loki/api/v1/push",format="json"}`)
-	lokiRequestsProtobufTotal = metrics.NewCounter(`vl_http_requests_total{path="/insert/loki/api/v1/push",format="protobuf"}`)
-)
-
 // RequestHandler processes Loki insert requests
 func RequestHandler(path string, w http.ResponseWriter, r *http.Request) bool {
 	switch path {
@@ -34,11 +27,9 @@ func handleInsert(r *http.Request, w http.ResponseWriter) bool {
 	contentType := r.Header.Get("Content-Type")
 	switch contentType {
 	case "application/json":
-		lokiRequestsJSONTotal.Inc()
 		return handleJSON(r, w)
 	default:
-		// Protobuf request body should be handled by default accoring to https://grafana.com/docs/loki/latest/api/#push-log-entries-to-loki
-		lokiRequestsProtobufTotal.Inc()
+		// Protobuf request body should be handled by default according to https://grafana.com/docs/loki/latest/api/#push-log-entries-to-loki
 		return handleProtobuf(r, w)
 	}
 }
diff --git a/app/vlinsert/loki/loki_json.go b/app/vlinsert/loki/loki_json.go
index 381aa3436..88a75df1d 100644
--- a/app/vlinsert/loki/loki_json.go
+++ b/app/vlinsert/loki/loki_json.go
@@ -18,12 +18,11 @@ import (
 	"github.com/valyala/fastjson"
 )
 
-var (
-	rowsIngestedJSONTotal = metrics.NewCounter(`vl_rows_ingested_total{type="loki",format="json"}`)
-	parserPool            fastjson.ParserPool
-)
+var parserPool fastjson.ParserPool
 
 func handleJSON(r *http.Request, w http.ResponseWriter) bool {
+	startTime := time.Now()
+	lokiRequestsJSONTotal.Inc()
 	reader := r.Body
 	if r.Header.Get("Content-Encoding") == "gzip" {
 		zr, err := common.GetGzipReader(reader)
@@ -58,9 +57,21 @@ func handleJSON(r *http.Request, w http.ResponseWriter) bool {
 		return true
 	}
 	rowsIngestedJSONTotal.Add(n)
+
+	// update lokiRequestJSONDuration only for successfully parsed requests
+	// There is no need in updating lokiRequestJSONDuration for request errors,
+	// since their timings are usually much smaller than the timing for successful request parsing.
+	lokiRequestJSONDuration.UpdateDuration(startTime)
+
 	return true
 }
 
+var (
+	lokiRequestsJSONTotal   = metrics.NewCounter(`vl_http_requests_total{path="/insert/loki/api/v1/push",format="json"}`)
+	rowsIngestedJSONTotal   = metrics.NewCounter(`vl_rows_ingested_total{type="loki",format="json"}`)
+	lokiRequestJSONDuration = metrics.NewHistogram(`vl_http_request_duration_seconds{path="/insert/loki/api/v1/push",format="json"}`)
+)
+
 func parseJSONRequest(data []byte, processLogMessage func(timestamp int64, fields []logstorage.Field)) (int, error) {
 	p := parserPool.Get()
 	defer parserPool.Put(p)
diff --git a/app/vlinsert/loki/loki_protobuf.go b/app/vlinsert/loki/loki_protobuf.go
index a75bf93d0..aa4e6b592 100644
--- a/app/vlinsert/loki/loki_protobuf.go
+++ b/app/vlinsert/loki/loki_protobuf.go
@@ -19,12 +19,13 @@ import (
 )
 
 var (
-	rowsIngestedProtobufTotal = metrics.NewCounter(`vl_rows_ingested_total{type="loki",format="protobuf"}`)
-	bytesBufPool              bytesutil.ByteBufferPool
-	pushReqsPool              sync.Pool
+	bytesBufPool bytesutil.ByteBufferPool
+	pushReqsPool sync.Pool
 )
 
 func handleProtobuf(r *http.Request, w http.ResponseWriter) bool {
+	startTime := time.Now()
+	lokiRequestsProtobufTotal.Inc()
 	wcr := writeconcurrencylimiter.GetReader(r.Body)
 	data, err := io.ReadAll(wcr)
 	writeconcurrencylimiter.PutReader(wcr)
@@ -47,10 +48,23 @@ func handleProtobuf(r *http.Request, w http.ResponseWriter) bool {
 		httpserver.Errorf(w, r, "cannot parse loki request: %s", err)
 		return true
 	}
+
 	rowsIngestedProtobufTotal.Add(n)
+
+	// update lokiRequestProtobufDuration only for successfully parsed requests
+	// There is no need in updating lokiRequestProtobufDuration for request errors,
+	// since their timings are usually much smaller than the timing for successful request parsing.
+	lokiRequestProtobufDuration.UpdateDuration(startTime)
+
 	return true
 }
 
+var (
+	lokiRequestsProtobufTotal   = metrics.NewCounter(`vl_http_requests_total{path="/insert/loki/api/v1/push",format="protobuf"}`)
+	rowsIngestedProtobufTotal   = metrics.NewCounter(`vl_rows_ingested_total{type="loki",format="protobuf"}`)
+	lokiRequestProtobufDuration = metrics.NewHistogram(`vl_http_request_duration_seconds{path="/insert/loki/api/v1/push",format="protobuf"}`)
+)
+
 func parseProtobufRequest(data []byte, processLogMessage func(timestamp int64, fields []logstorage.Field)) (int, error) {
 	bb := bytesBufPool.Get()
 	defer bytesBufPool.Put(bb)
diff --git a/app/vmagent/README.md b/app/vmagent/README.md
index aafd707d0..7f9fd0b8a 100644
--- a/app/vmagent/README.md
+++ b/app/vmagent/README.md
@@ -3,7 +3,8 @@
 `vmagent` is a tiny agent which helps you collect metrics from various sources,
 [relabel and filter the collected metrics](#relabeling)
 and store them in [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics)
-or any other storage systems via Prometheus `remote_write` protocol.
+or any other storage systems via Prometheus `remote_write` protocol
+or via [VictoriaMetrics `remote_write` protocol](#victoriametrics-remote-write-protocol).
 
 See [Quick Start](#quick-start) for details.
 
diff --git a/app/vmauth/README.md b/app/vmauth/README.md
index 610462412..71a627e01 100644
--- a/app/vmauth/README.md
+++ b/app/vmauth/README.md
@@ -25,6 +25,7 @@ The auth config can be reloaded via the following ways:
   and apply new changes every 5 seconds.
 
 Docker images for `vmauth` are available [here](https://hub.docker.com/r/victoriametrics/vmauth/tags).
+See how `vmauth` used in [docker-compose env](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/README.md#victoriametrics-cluster).
 
 Pass `-help` to `vmauth` in order to see all the supported command-line flags with their descriptions.
 
diff --git a/app/vmauth/auth_config.go b/app/vmauth/auth_config.go
index 43b40bc94..fbb9bb414 100644
--- a/app/vmauth/auth_config.go
+++ b/app/vmauth/auth_config.go
@@ -1,6 +1,7 @@
 package main
 
 import (
+	"bytes"
 	"encoding/base64"
 	"flag"
 	"fmt"
@@ -290,6 +291,13 @@ func (sp *SrcPath) MarshalYAML() (interface{}, error) {
 	return sp.sOriginal, nil
 }
 
+var (
+	configReloads      = metrics.NewCounter(`vmauth_config_last_reload_total`)
+	configReloadErrors = metrics.NewCounter(`vmauth_config_last_reload_errors_total`)
+	configSuccess      = metrics.NewCounter(`vmauth_config_last_reload_successful`)
+	configTimestamp    = metrics.NewCounter(`vmauth_config_last_reload_success_timestamp_seconds`)
+)
+
 func initAuthConfig() {
 	if len(*authConfigPath) == 0 {
 		logger.Fatalf("missing required `-auth.config` command-line flag")
@@ -300,11 +308,14 @@ func initAuthConfig() {
 	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240
 	sighupCh := procutil.NewSighupChan()
 
-	err := loadAuthConfig()
+	_, err := loadAuthConfig()
 	if err != nil {
 		logger.Fatalf("cannot load auth config: %s", err)
 	}
 
+	configSuccess.Set(1)
+	configTimestamp.Set(fasttime.UnixTimestamp())
+
 	stopCh = make(chan struct{})
 	authConfigWG.Add(1)
 	go func() {
@@ -327,52 +338,75 @@ func authConfigReloader(sighupCh <-chan os.Signal) {
 		refreshCh = ticker.C
 	}
 
+	updateFn := func() {
+		configReloads.Inc()
+		updated, err := loadAuthConfig()
+		if err != nil {
+			logger.Errorf("failed to load auth config; using the last successfully loaded config; error: %s", err)
+			configSuccess.Set(0)
+			configReloadErrors.Inc()
+			return
+		}
+		configSuccess.Set(1)
+		if updated {
+			configTimestamp.Set(fasttime.UnixTimestamp())
+		}
+	}
+
 	for {
 		select {
 		case <-stopCh:
 			return
 		case <-refreshCh:
-			procutil.SelfSIGHUP()
+			updateFn()
 		case <-sighupCh:
 			logger.Infof("SIGHUP received; loading -auth.config=%q", *authConfigPath)
-			err := loadAuthConfig()
-			if err != nil {
-				logger.Errorf("failed to load auth config; using the last successfully loaded config; error: %s", err)
-				continue
-			}
+			updateFn()
 		}
 	}
 }
 
+// authConfigData stores the yaml definition for this config.
+// authConfigData needs to be updated each time authConfig is updated.
+var authConfigData atomic.Pointer[[]byte]
+
 var authConfig atomic.Pointer[AuthConfig]
 var authUsers atomic.Pointer[map[string]*UserInfo]
 var authConfigWG sync.WaitGroup
 var stopCh chan struct{}
 
-func loadAuthConfig() error {
-	ac, err := readAuthConfig(*authConfigPath)
+// loadAuthConfig loads and applies the config from *authConfigPath.
+// It returns bool value to identify if new config was applied.
+// The config can be not applied if there is a parsing error
+// or if there are no changes to the current authConfig.
+func loadAuthConfig() (bool, error) {
+	data, err := fs.ReadFileOrHTTP(*authConfigPath)
+	if err != nil {
+		return false, fmt.Errorf("failed to read -auth.config=%q: %w", *authConfigPath, err)
+	}
+
+	oldData := authConfigData.Load()
+	if oldData != nil && bytes.Equal(data, *oldData) {
+		// there are no updates in the config - skip reloading.
+		return false, nil
+	}
+
+	ac, err := parseAuthConfig(data)
 	if err != nil {
-		return fmt.Errorf("failed to load -auth.config=%q: %s", *authConfigPath, err)
+		return false, fmt.Errorf("failed to parse -auth.config=%q: %w", *authConfigPath, err)
 	}
 
 	m, err := parseAuthConfigUsers(ac)
 	if err != nil {
-		return fmt.Errorf("failed to parse users from -auth.config=%q: %s", *authConfigPath, err)
+		return false, fmt.Errorf("failed to parse users from -auth.config=%q: %w", *authConfigPath, err)
 	}
 	logger.Infof("loaded information about %d users from -auth.config=%q", len(m), *authConfigPath)
 
 	authConfig.Store(ac)
+	authConfigData.Store(&data)
 	authUsers.Store(&m)
 
-	return nil
-}
-
-func readAuthConfig(path string) (*AuthConfig, error) {
-	data, err := fs.ReadFileOrHTTP(path)
-	if err != nil {
-		return nil, err
-	}
-	return parseAuthConfig(data)
+	return true, nil
 }
 
 func parseAuthConfig(data []byte) (*AuthConfig, error) {
diff --git a/app/vmbackup/README.md b/app/vmbackup/README.md
index 9daad41f1..340c99d6b 100644
--- a/app/vmbackup/README.md
+++ b/app/vmbackup/README.md
@@ -143,20 +143,23 @@ See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-
 
 ## Advanced usage
 
-* Obtaining credentials from a file.
 
-  Add flag `-credsFilePath=/etc/credentials` with the following content:
+### Providing credentials as a file
 
-    for s3 (aws, minio or other s3 compatible storages):
+Obtaining credentials from a file.
 
+Add flag `-credsFilePath=/etc/credentials` with the following content:
+
+- for S3 (AWS, MinIO or other S3 compatible storages):
+    
      ```console
      [default]
      aws_access_key_id=theaccesskey
      aws_secret_access_key=thesecretaccesskeyvalue
     ```
 
-    for gce cloud storage:
-
+- for GCP cloud storage:
+    
     ```json
     {
            "type": "service_account",
@@ -171,24 +174,99 @@ See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-
            "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-email"
     }
     ```
-* Obtaining credentials from env variables.
-    - For AWS S3 compatible storages set env variable `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
-      Also you can set env variable `AWS_SHARED_CREDENTIALS_FILE` with path to credentials file.
-    - For GCE cloud storage set env variable `GOOGLE_APPLICATION_CREDENTIALS` with path to credentials file.
-    - For Azure storage either set env variables `AZURE_STORAGE_ACCOUNT_NAME` and `AZURE_STORAGE_ACCOUNT_KEY`, or `AZURE_STORAGE_ACCOUNT_CONNECTION_STRING`.
 
-* Usage with s3 custom url endpoint. It is possible to use `vmbackup` with s3 compatible storages like minio, cloudian, etc.
-  You have to add a custom url endpoint via flag:
+### Providing credentials via env variables 
 
-```console
-  # for minio
-  -customS3Endpoint=http://localhost:9000
+Obtaining credentials from env variables.
+- For AWS S3 compatible storages set env variable `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. 
+  Also you can set env variable `AWS_SHARED_CREDENTIALS_FILE` with path to credentials file.
+- For GCE cloud storage set env variable `GOOGLE_APPLICATION_CREDENTIALS` with path to credentials file.
+- For Azure storage either set env variables `AZURE_STORAGE_ACCOUNT_NAME` and `AZURE_STORAGE_ACCOUNT_KEY`, or `AZURE_STORAGE_ACCOUNT_CONNECTION_STRING`.
+
+Please, note that `vmbackup` will use credentials provided by cloud providers metadata service [when applicable](https://docs.victoriametrics.com/vmbackup.html#using-cloud-providers-metadata-service).
+
+### Using cloud providers metadata service
+
+`vmbackup` and `vmbackupmanager` will automatically use cloud providers metadata service in order to obtain credentials if they are running in cloud environment
+and credentials are not explicitly provided via flags or env variables.
+
+### Providing credentials in Kubernetes
+
+The simplest way to provide credentials in Kubernetes is to use [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/)
+and inject them into the pod as environment variables. For example, the following secret can be used for AWS S3 credentials:
+```yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: vmbackup-credentials
+data:
+  access_key: key
+  secret_key: secret
+```
+And then it can be injected into the pod as environment variables:
+```yaml
+...
+env:
+- name: AWS_ACCESS_KEY_ID
+  valueFrom:
+    secretKeyRef:
+      key: access_key
+      name: vmbackup-credentials
+- name: AWS_SECRET_ACCESS_KEY
+  valueFrom:
+    secretKeyRef:
+      key: secret_key
+      name: vmbackup-credentials
+...
+```
 
-  # for aws gov region
-  -customS3Endpoint=https://s3-fips.us-gov-west-1.amazonaws.com
+A more secure way is to use IAM roles to provide tokens for pods instead of managing credentials manually. 
+
+For AWS deployments it will be required to configure [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html).
+In order to use IAM roles for service accounts with `vmbackup` or `vmbackupmanager` it is required to create ServiceAccount with IAM role mapping:
+```yaml
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: monitoring-backups
+  annotations:
+    eks.amazonaws.com/role-arn: arn:aws:iam::{ACCOUNT_ID}:role/{ROLE_NAME}
+```
+And [configure pod to use service account](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/).
+After this `vmbackup` and `vmbackupmanager` will automatically use IAM role for service account in order to obtain credentials.
+
+For GCP deployments it will be required to configure [Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity).
+In order to use Workload Identity with `vmbackup` or `vmbackupmanager` it is required to create ServiceAccount with Workload Identity annotation:
+```yaml
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: monitoring-backups
+  annotations:
+    iam.gke.io/gcp-service-account: {sa_name}@{project_name}.iam.gserviceaccount.com
 ```
+And [configure pod to use service account](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/).
+After this `vmbackup` and `vmbackupmanager` will automatically use Workload Identity for servicpe account in order to obtain credentials.
+
+### Using custom S3 endpoint
+
+Usage with s3 custom url endpoint. It is possible to use `vmbackup` with s3 compatible storages like minio, cloudian, etc.
+You have to add a custom url endpoint via flag:
+
+- for MinIO
+    ```console
+      -customS3Endpoint=http://localhost:9000
+    ```
+
+- for aws gov region
+    ```console
+      -customS3Endpoint=https://s3-fips.us-gov-west-1.amazonaws.com
+    ```
+
+### Command-line flags
 
-* Run `vmbackup -help` in order to see all the available options:
+Run `vmbackup -help` in order to see all the available options:
 
 ```console
   -concurrency int
diff --git a/app/vmbackupmanager/README.md b/app/vmbackupmanager/README.md
index 4d0c1937a..f56cac238 100644
--- a/app/vmbackupmanager/README.md
+++ b/app/vmbackupmanager/README.md
@@ -110,6 +110,9 @@ The result on the GCS bucket
 
   <img alt="latest folder" src="vmbackupmanager_latest_folder.png">
 
+Please, see [vmbackup docs](https://docs.victoriametrics.com/vmbackup.html#advanced-usage) for more examples of authentication with different
+storage types.
+
 ## Backup Retention Policy
 
 Backup retention policy is controlled by:
diff --git a/app/vmselect/graphite/render_response.qtpl b/app/vmselect/graphite/render_response.qtpl
index 308a378ad..2ea588e64 100644
--- a/app/vmselect/graphite/render_response.qtpl
+++ b/app/vmselect/graphite/render_response.qtpl
@@ -47,7 +47,7 @@ See https://graphite.readthedocs.io/en/stable/render_api.html#json
 			{% code timestamps := s.Timestamps %}
 			{% for i, v := range s.Values %}
 				[
-					{% if math.IsNaN(v) %}null{% else %}{%f= v %}{% endif %},
+					{% if math.IsNaN(v) || math.IsInf(v, 0) %}null{% else %}{%f= v %}{% endif %},
 					{%dl= timestamps[i]/1e3 %}
 				]
 				{% if i+1 < len(timestamps) %},{% endif %}
diff --git a/app/vmselect/graphite/render_response.qtpl.go b/app/vmselect/graphite/render_response.qtpl.go
index 213403fb1..77a87998e 100644
--- a/app/vmselect/graphite/render_response.qtpl.go
+++ b/app/vmselect/graphite/render_response.qtpl.go
@@ -148,7 +148,7 @@ func streamrenderSeriesJSON(qw422016 *qt422016.Writer, s *series) {
 //line app/vmselect/graphite/render_response.qtpl:48
 		qw422016.N().S(`[`)
 //line app/vmselect/graphite/render_response.qtpl:50
-		if math.IsNaN(v) {
+		if math.IsNaN(v) || math.IsInf(v, 0) {
 //line app/vmselect/graphite/render_response.qtpl:50
 			qw422016.N().S(`null`)
 //line app/vmselect/graphite/render_response.qtpl:50
diff --git a/app/vmselect/promql/exec.go b/app/vmselect/promql/exec.go
index acbaa59d5..ab14cb14c 100644
--- a/app/vmselect/promql/exec.go
+++ b/app/vmselect/promql/exec.go
@@ -111,6 +111,12 @@ func maySortResults(e metricsql.Expr) bool {
 			"bottomk_max", "bottomk_min", "bottomk_avg", "bottomk_median", "bottomk_last":
 			return false
 		}
+	case *metricsql.BinaryOpExpr:
+		if strings.ToLower(v.Op) == "or" {
+			// Do not sort results for `a or b` in the same way as Prometheus does.
+			// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4763
+			return false
+		}
 	}
 	return true
 }
diff --git a/app/vmselect/promql/rollup.go b/app/vmselect/promql/rollup.go
index 2cf827344..9f84f74a7 100644
--- a/app/vmselect/promql/rollup.go
+++ b/app/vmselect/promql/rollup.go
@@ -7,11 +7,12 @@ import (
 	"strings"
 	"sync"
 
+	"github.com/VictoriaMetrics/metrics"
+	"github.com/VictoriaMetrics/metricsql"
+
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/decimal"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage"
-	"github.com/VictoriaMetrics/metrics"
-	"github.com/VictoriaMetrics/metricsql"
 )
 
 var minStalenessInterval = flag.Duration("search.minStalenessInterval", 0, "The minimum interval for staleness calculations. "+
@@ -58,6 +59,7 @@ var rollupFuncs = map[string]newRollupFunc{
 	"lifetime":                newRollupFuncOneArg(rollupLifetime),
 	"mad_over_time":           newRollupFuncOneArg(rollupMAD),
 	"max_over_time":           newRollupFuncOneArg(rollupMax),
+	"median_over_time":        newRollupFuncOneArg(rollupMedian),
 	"min_over_time":           newRollupFuncOneArg(rollupMin),
 	"mode_over_time":          newRollupFuncOneArg(rollupModeOverTime),
 	"predict_linear":          newRollupPredictLinear,
@@ -125,6 +127,7 @@ var rollupAggrFuncs = map[string]rollupFunc{
 	"lifetime":                rollupLifetime,
 	"mad_over_time":           rollupMAD,
 	"max_over_time":           rollupMax,
+	"median_over_time":        rollupMedian,
 	"min_over_time":           rollupMin,
 	"mode_over_time":          rollupModeOverTime,
 	"present_over_time":       rollupPresent,
@@ -224,6 +227,7 @@ var rollupFuncsKeepMetricName = map[string]bool{
 	"holt_winters":          true,
 	"last_over_time":        true,
 	"max_over_time":         true,
+	"median_over_time":      true,
 	"min_over_time":         true,
 	"mode_over_time":        true,
 	"predict_linear":        true,
@@ -1396,6 +1400,10 @@ func rollupMax(rfa *rollupFuncArg) float64 {
 	return maxValue
 }
 
+func rollupMedian(rfa *rollupFuncArg) float64 {
+	return quantile(0.5, rfa.values)
+}
+
 func rollupTmin(rfa *rollupFuncArg) float64 {
 	// There is no need in handling NaNs here, since they must be cleaned up
 	// before calling rollup funcs.
diff --git a/app/vmstorage/main.go b/app/vmstorage/main.go
index f3be38ff9..e81e38825 100644
--- a/app/vmstorage/main.go
+++ b/app/vmstorage/main.go
@@ -585,11 +585,6 @@ func registerStorageMetrics(strg *storage.Storage) {
 		return float64(idbm().ItemsAddedSizeBytes)
 	})
 
-	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686
-	metrics.NewGauge(`vm_merge_need_free_disk_space`, func() float64 {
-		return float64(tm().MergeNeedFreeDiskSpace)
-	})
-
 	metrics.NewGauge(`vm_pending_rows{type="storage"}`, func() float64 {
 		return float64(tm().PendingRows)
 	})
diff --git a/app/vmui/packages/vmui/src/components/Main/Hyperlink/Hyperlink.tsx b/app/vmui/packages/vmui/src/components/Main/Hyperlink/Hyperlink.tsx
index c1cd58e3f..32dac0b3a 100644
--- a/app/vmui/packages/vmui/src/components/Main/Hyperlink/Hyperlink.tsx
+++ b/app/vmui/packages/vmui/src/components/Main/Hyperlink/Hyperlink.tsx
@@ -8,6 +8,7 @@ interface Hyperlink {
   children?: ReactNode;
   colored?: boolean;
   underlined?: boolean;
+  withIcon?: boolean;
 }
 
 const Hyperlink: FC<Hyperlink> = ({
@@ -15,14 +16,16 @@ const Hyperlink: FC<Hyperlink> = ({
   href,
   children,
   colored = true,
-  underlined = false
+  underlined = false,
+  withIcon = false,
 }) => (
   <a
     href={href}
     className={classNames({
       "vm-link": true,
       "vm-link_colored": colored,
-      "vm-link_underlined": underlined
+      "vm-link_underlined": underlined,
+      "vm-link_with-icon": withIcon,
     })}
     target="_blank"
     rel="noreferrer"
diff --git a/app/vmui/packages/vmui/src/pages/CardinalityPanel/CardinalityConfigurator/CardinalityConfigurator.tsx b/app/vmui/packages/vmui/src/pages/CardinalityPanel/CardinalityConfigurator/CardinalityConfigurator.tsx
index 3149ee3d2..661c2afab 100644
--- a/app/vmui/packages/vmui/src/pages/CardinalityPanel/CardinalityConfigurator/CardinalityConfigurator.tsx
+++ b/app/vmui/packages/vmui/src/pages/CardinalityPanel/CardinalityConfigurator/CardinalityConfigurator.tsx
@@ -11,8 +11,9 @@ import { useSearchParams } from "react-router-dom";
 import CardinalityTotals, { CardinalityTotalsProps } from "../CardinalityTotals/CardinalityTotals";
 import useSearchParamsFromObject from "../../../hooks/useSearchParamsFromObject";
 import useStateSearchParams from "../../../hooks/useStateSearchParams";
+import Hyperlink from "../../../components/Main/Hyperlink/Hyperlink";
 
-const CardinalityConfigurator: FC<CardinalityTotalsProps> = ({ isPrometheus, ...props }) => {
+const CardinalityConfigurator: FC<CardinalityTotalsProps> = ({ isPrometheus, isCluster, ...props }) => {
   const { isMobile } = useDeviceDetect();
   const [searchParams] = useSearchParams();
   const { setSearchParamsFromKeys } = useSearchParamsFromObject();
@@ -105,19 +106,29 @@ const CardinalityConfigurator: FC<CardinalityTotalsProps> = ({ isPrometheus, ...
     <div className="vm-cardinality-configurator-bottom">
       <CardinalityTotals
         isPrometheus={isPrometheus}
+        isCluster={isCluster}
         {...props}
       />
+      {isCluster &&
+        <div className="vm-cardinality-configurator-bottom-helpful">
+          <Hyperlink
+            href="https://docs.victoriametrics.com/#cardinality-explorer-statistic-inaccurancy"
+            withIcon={true}
+          >
+            <WikiIcon/>
+          Statistic inaccuracy explanation
+          </Hyperlink>
+        </div>
+      }
 
       <div className="vm-cardinality-configurator-bottom-helpful">
-        <a
-          className="vm-link vm-link_with-icon"
-          target="_blank"
+        <Hyperlink
           href="https://docs.victoriametrics.com/#cardinality-explorer"
-          rel="help noreferrer"
+          withIcon={true}
         >
           <WikiIcon/>
           Documentation
-        </a>
+        </Hyperlink>
       </div>
 
       <div className="vm-cardinality-configurator-bottom__execute">
diff --git a/app/vmui/packages/vmui/src/pages/CardinalityPanel/CardinalityTotals/CardinalityTotals.tsx b/app/vmui/packages/vmui/src/pages/CardinalityPanel/CardinalityTotals/CardinalityTotals.tsx
index 0c1f53e1a..63845403a 100644
--- a/app/vmui/packages/vmui/src/pages/CardinalityPanel/CardinalityTotals/CardinalityTotals.tsx
+++ b/app/vmui/packages/vmui/src/pages/CardinalityPanel/CardinalityTotals/CardinalityTotals.tsx
@@ -14,6 +14,7 @@ export interface CardinalityTotalsProps {
   totalLabelValuePairs: number;
   seriesCountByMetricName: TopHeapEntry[];
   isPrometheus?: boolean;
+  isCluster: boolean;
 }
 
 const CardinalityTotals: FC<CardinalityTotalsProps> = ({
@@ -21,7 +22,7 @@ const CardinalityTotals: FC<CardinalityTotalsProps> = ({
   totalSeriesPrev = 0,
   totalSeriesAll = 0,
   seriesCountByMetricName = [],
-  isPrometheus
+  isPrometheus,
 }) => {
   const { isMobile } = useDeviceDetect();
 
@@ -50,7 +51,7 @@ const CardinalityTotals: FC<CardinalityTotalsProps> = ({
       value: isNaN(progress) ? "-" : `${progress.toFixed(2)}%`,
       display: isMetric,
       info: "The share of these series in the total number of time series."
-    }
+    },
   ].filter(t => t.display);
 
   if (!totals.length) {
diff --git a/app/vmui/packages/vmui/src/pages/CardinalityPanel/hooks/useCardinalityFetch.ts b/app/vmui/packages/vmui/src/pages/CardinalityPanel/hooks/useCardinalityFetch.ts
index 585ebf245..c1c88b6e7 100644
--- a/app/vmui/packages/vmui/src/pages/CardinalityPanel/hooks/useCardinalityFetch.ts
+++ b/app/vmui/packages/vmui/src/pages/CardinalityPanel/hooks/useCardinalityFetch.ts
@@ -7,12 +7,14 @@ import AppConfigurator from "../appConfigurator";
 import { useSearchParams } from "react-router-dom";
 import dayjs from "dayjs";
 import { DATE_FORMAT } from "../../../constants/date";
+import { getTenantIdFromUrl } from "../../../utils/tenants";
 
 export const useFetchQuery = (): {
   fetchUrl?: string[],
   isLoading: boolean,
   error?: ErrorTypes | string
   appConfigurator: AppConfigurator,
+  isCluster: boolean,
 } => {
   const appConfigurator = new AppConfigurator();
 
@@ -26,6 +28,7 @@ export const useFetchQuery = (): {
   const [isLoading, setIsLoading] = useState(false);
   const [error, setError] = useState<ErrorTypes | string>();
   const [tsdbStatus, setTSDBStatus] = useState<TSDBStatus>(appConfigurator.defaultTSDBStatus);
+  const [isCluster, setIsCluster] = useState<boolean>(false);
 
   const getResponseJson = async (url: string) => {
     const response = await fetch(url);
@@ -115,6 +118,12 @@ export const useFetchQuery = (): {
     }
   }, [error]);
 
+  useEffect(() => {
+    const id = getTenantIdFromUrl(serverUrl);
+    setIsCluster(!!id);
+  }, [serverUrl]);
+
+
   appConfigurator.tsdbStatusData = tsdbStatus;
-  return { isLoading, appConfigurator: appConfigurator, error };
+  return { isLoading, appConfigurator: appConfigurator, error, isCluster };
 };
diff --git a/app/vmui/packages/vmui/src/pages/CardinalityPanel/index.tsx b/app/vmui/packages/vmui/src/pages/CardinalityPanel/index.tsx
index 631441041..777b2f398 100644
--- a/app/vmui/packages/vmui/src/pages/CardinalityPanel/index.tsx
+++ b/app/vmui/packages/vmui/src/pages/CardinalityPanel/index.tsx
@@ -31,7 +31,7 @@ const CardinalityPanel: FC = () => {
   const match = searchParams.get("match") || "";
   const focusLabel = searchParams.get("focusLabel") || "";
 
-  const { isLoading, appConfigurator, error } = useFetchQuery();
+  const { isLoading, appConfigurator, error, isCluster } = useFetchQuery();
   const { tsdbStatusData, getDefaultState, tablesHeaders, sectionsTips } = appConfigurator;
   const defaultState = getDefaultState(match, focusLabel);
 
@@ -62,6 +62,7 @@ const CardinalityPanel: FC = () => {
         totalSeriesAll={tsdbStatusData.totalSeriesByAll}
         totalLabelValuePairs={tsdbStatusData.totalLabelValuePairs}
         seriesCountByMetricName={tsdbStatusData.seriesCountByMetricName}
+        isCluster={isCluster}
       />
 
       {showTips && (
@@ -69,7 +70,7 @@ const CardinalityPanel: FC = () => {
           {!match && !focusLabel && <TipHighNumberOfSeries/>}
           {match && !focusLabel && <TipCardinalityOfSingle/>}
           {!match && !focusLabel && <TipHighNumberOfValues/>}
-          {focusLabel && <TipCardinalityOfLabel/>}
+          {focusLabel && <TipCardinalityOfLabel />}
         </div>
       )}
 
diff --git a/dashboards/victorialogs.json b/dashboards/victorialogs.json
index 629638e3d..e2bc28168 100644
--- a/dashboards/victorialogs.json
+++ b/dashboards/victorialogs.json
@@ -18,13 +18,37 @@
           "type": "dashboard"
         },
         "type": "dashboard"
+      },
+      {
+        "datasource": {
+          "type": "prometheus",
+          "uid": "${ds}"
+        },
+        "enable": true,
+        "expr": "sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"}) by(short_version) unless (sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"} offset 20m) by(short_version))",
+        "hide": true,
+        "iconColor": "blue",
+        "name": "version change",
+        "textFormat": "{{short_version}}",
+        "titleFormat": "Version change"
+      },
+      {
+        "datasource": {
+          "type": "prometheus",
+          "uid": "${ds}"
+        },
+        "enable": true,
+        "expr": "sum(changes(vm_app_start_timestamp{job=~\"$job\", instance=~\"$instance\"})) by(job)",
+        "hide": true,
+        "iconColor": "orange",
+        "name": "restarts",
+        "textFormat": "{{job}} restarted"
       }
     ]
   },
   "editable": true,
   "fiscalYearStartMonth": 0,
   "graphTooltip": 0,
-  "id": 4,
   "links": [],
   "liveNow": false,
   "panels": [
@@ -2077,8 +2101,7 @@
             "mode": "absolute",
             "steps": [
               {
-                "color": "green",
-                "value": null
+                "color": "green"
               },
               {
                 "color": "red",
@@ -2182,8 +2205,7 @@
             "mode": "absolute",
             "steps": [
               {
-                "color": "green",
-                "value": null
+                "color": "green"
               },
               {
                 "color": "red",
@@ -2318,8 +2340,7 @@
             "mode": "absolute",
             "steps": [
               {
-                "color": "green",
-                "value": null
+                "color": "green"
               },
               {
                 "color": "red",
@@ -2425,8 +2446,7 @@
             "mode": "absolute",
             "steps": [
               {
-                "color": "green",
-                "value": null
+                "color": "green"
               },
               {
                 "color": "red",
@@ -2561,6 +2581,10 @@
         "type": "query"
       },
       {
+        "datasource": {
+          "type": "prometheus",
+          "uid": "$ds"
+        },
         "filters": [],
         "hide": 0,
         "name": "adhoc",
@@ -2577,6 +2601,6 @@
   "timezone": "",
   "title": "VictoriaLogs",
   "uid": "OqPIZTX4z",
-  "version": 7,
+  "version": 3,
   "weekStart": ""
 }
\ No newline at end of file
diff --git a/dashboards/victoriametrics.json b/dashboards/victoriametrics.json
index 69a0d93b5..2d6313523 100644
--- a/dashboards/victoriametrics.json
+++ b/dashboards/victoriametrics.json
@@ -76,7 +76,7 @@
           "uid": "$ds"
         },
         "enable": true,
-        "expr": "sum(vm_app_version{job=~\"$job\"}) by(short_version) unless (sum(vm_app_version{job=~\"$job\"} offset 20m) by(short_version))",
+        "expr": "sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"}) by(short_version) unless (sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"} offset 20m) by(short_version))",
         "hide": true,
         "iconColor": "dark-blue",
         "name": "version",
diff --git a/dashboards/vm/victoriametrics.json b/dashboards/vm/victoriametrics.json
index 52c04c949..161308ade 100644
--- a/dashboards/vm/victoriametrics.json
+++ b/dashboards/vm/victoriametrics.json
@@ -77,7 +77,7 @@
           "uid": "$ds"
         },
         "enable": true,
-        "expr": "sum(vm_app_version{job=~\"$job\"}) by(short_version) unless (sum(vm_app_version{job=~\"$job\"} offset 20m) by(short_version))",
+        "expr": "sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"}) by(short_version) unless (sum(vm_app_version{job=~\"$job\", instance=~\"$instance\"} offset 20m) by(short_version))",
         "hide": true,
         "iconColor": "dark-blue",
         "name": "version",
diff --git a/dashboards/vm/vmagent.json b/dashboards/vm/vmagent.json
index adfb37735..3928c3248 100644
--- a/dashboards/vm/vmagent.json
+++ b/dashboards/vm/vmagent.json
@@ -2373,7 +2373,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "green"
+                    "color": "green",
+                    "value": null
                   },
                   {
                     "color": "red",
@@ -2389,7 +2390,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 4
+            "y": 36
           },
           "id": 92,
           "options": {
@@ -2475,7 +2476,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "green"
+                    "color": "green",
+                    "value": null
                   },
                   {
                     "color": "red",
@@ -2491,7 +2493,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 4
+            "y": 36
           },
           "id": 95,
           "options": {
@@ -2580,7 +2582,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "transparent"
+                    "color": "transparent",
+                    "value": null
                   },
                   {
                     "color": "red",
@@ -2596,7 +2599,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 12
+            "y": 44
           },
           "id": 98,
           "options": {
@@ -2685,7 +2688,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "transparent"
+                    "color": "transparent",
+                    "value": null
                   },
                   {
                     "color": "red",
@@ -2701,7 +2705,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 12
+            "y": 44
           },
           "id": 99,
           "options": {
@@ -2789,7 +2793,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "green"
+                    "color": "green",
+                    "value": null
                   },
                   {
                     "color": "red",
@@ -2805,7 +2810,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 20
+            "y": 52
           },
           "id": 79,
           "links": [],
@@ -2894,7 +2899,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "green"
+                    "color": "green",
+                    "value": null
                   },
                   {
                     "color": "red",
@@ -2910,7 +2916,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 20
+            "y": 52
           },
           "id": 18,
           "links": [
@@ -3004,7 +3010,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "green"
+                    "color": "green",
+                    "value": null
                   },
                   {
                     "color": "red",
@@ -3020,7 +3027,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 28
+            "y": 60
           },
           "id": 127,
           "links": [],
@@ -3107,7 +3114,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "green"
+                    "color": "green",
+                    "value": null
                   },
                   {
                     "color": "red",
@@ -3123,7 +3131,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 28
+            "y": 60
           },
           "id": 50,
           "options": {
@@ -3161,6 +3169,123 @@
           "title": "Invalid datapoints rate ($instance)",
           "type": "timeseries"
         },
+        {
+          "datasource": {
+            "type": "victoriametrics-datasource",
+            "uid": "$ds"
+          },
+          "description": "Shows how many concurrent inserts (parsing and processing of scraped or ingested data) are taking place.\n\nIf the number of concurrent inserts hits the `limit` or is close to the `limit` constantly - it might be a sign of a resource shortage.\n\nIf vmagent's CPU usage and remote write connection saturation are at normal level, it might be that `-maxConcurrentInserts` cmd-line flag needs to be increased.",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "axisCenteredZero": false,
+                "axisColorMode": "text",
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 0,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": false,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "links": [],
+              "mappings": [],
+              "min": 0,
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              },
+              "unit": "short"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 12,
+            "x": 0,
+            "y": 68
+          },
+          "id": 130,
+          "links": [],
+          "options": {
+            "legend": {
+              "calcs": [
+                "mean",
+                "lastNotNull",
+                "max"
+              ],
+              "displayMode": "table",
+              "placement": "bottom",
+              "showLegend": true
+            },
+            "tooltip": {
+              "mode": "multi",
+              "sort": "desc"
+            }
+          },
+          "pluginVersion": "9.2.6",
+          "targets": [
+            {
+              "datasource": {
+                "type": "victoriametrics-datasource",
+                "uid": "$ds"
+              },
+              "editorMode": "code",
+              "exemplar": true,
+              "expr": "max_over_time(vm_concurrent_insert_current{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])",
+              "interval": "",
+              "legendFormat": "{{instance}} ({{job}})",
+              "range": true,
+              "refId": "A"
+            },
+            {
+              "datasource": {
+                "type": "victoriametrics-datasource",
+                "uid": "$ds"
+              },
+              "editorMode": "code",
+              "exemplar": true,
+              "expr": "min(vm_concurrent_insert_capacity{job=~\"$job\", instance=~\"$instance\"}) by(job)",
+              "interval": "",
+              "legendFormat": "limit ({{job}})",
+              "range": true,
+              "refId": "B"
+            }
+          ],
+          "title": "Concurrent inserts ($instance)",
+          "type": "timeseries"
+        },
         {
           "datasource": {
             "type": "victoriametrics-datasource",
@@ -3181,7 +3306,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "green"
+                    "color": "green",
+                    "value": null
                   },
                   {
                     "color": "red",
@@ -3221,7 +3347,7 @@
             "h": 7,
             "w": 24,
             "x": 0,
-            "y": 36
+            "y": 76
           },
           "id": 129,
           "options": {
@@ -3240,7 +3366,7 @@
               }
             ]
           },
-          "pluginVersion": "9.2.6",
+          "pluginVersion": "9.2.7",
           "targets": [
             {
               "datasource": {
@@ -4063,7 +4189,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 38
+            "y": 85
           },
           "id": 73,
           "links": [],
@@ -4180,7 +4306,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 38
+            "y": 85
           },
           "id": 131,
           "links": [],
@@ -4219,123 +4345,6 @@
           "title": "Rows rate  ($instance)",
           "type": "timeseries"
         },
-        {
-          "datasource": {
-            "type": "victoriametrics-datasource",
-            "uid": "$ds"
-          },
-          "description": "Shows how many concurrent inserts are taking place.\n\nIf the number of concurrent inserts hitting the `limit` or is close to the `limit` constantly - it might be a sign of a resource shortage.\n\n If vmagent's CPU usage and remote write connection saturation are at normal level, it might be that `-maxConcurrentInserts` cmd-line flag need to be increased.",
-          "fieldConfig": {
-            "defaults": {
-              "color": {
-                "mode": "palette-classic"
-              },
-              "custom": {
-                "axisCenteredZero": false,
-                "axisColorMode": "text",
-                "axisLabel": "",
-                "axisPlacement": "auto",
-                "barAlignment": 0,
-                "drawStyle": "line",
-                "fillOpacity": 0,
-                "gradientMode": "none",
-                "hideFrom": {
-                  "legend": false,
-                  "tooltip": false,
-                  "viz": false
-                },
-                "lineInterpolation": "linear",
-                "lineWidth": 1,
-                "pointSize": 5,
-                "scaleDistribution": {
-                  "type": "linear"
-                },
-                "showPoints": "never",
-                "spanNulls": false,
-                "stacking": {
-                  "group": "A",
-                  "mode": "none"
-                },
-                "thresholdsStyle": {
-                  "mode": "off"
-                }
-              },
-              "links": [],
-              "mappings": [],
-              "min": 0,
-              "thresholds": {
-                "mode": "absolute",
-                "steps": [
-                  {
-                    "color": "green",
-                    "value": null
-                  },
-                  {
-                    "color": "red",
-                    "value": 80
-                  }
-                ]
-              },
-              "unit": "short"
-            },
-            "overrides": []
-          },
-          "gridPos": {
-            "h": 8,
-            "w": 12,
-            "x": 0,
-            "y": 46
-          },
-          "id": 130,
-          "links": [],
-          "options": {
-            "legend": {
-              "calcs": [
-                "mean",
-                "lastNotNull",
-                "max"
-              ],
-              "displayMode": "table",
-              "placement": "bottom",
-              "showLegend": true
-            },
-            "tooltip": {
-              "mode": "multi",
-              "sort": "desc"
-            }
-          },
-          "pluginVersion": "9.2.6",
-          "targets": [
-            {
-              "datasource": {
-                "type": "victoriametrics-datasource",
-                "uid": "$ds"
-              },
-              "editorMode": "code",
-              "exemplar": true,
-              "expr": "max_over_time(vm_concurrent_insert_current{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])",
-              "interval": "",
-              "legendFormat": "{{instance}} ({{job}})",
-              "range": true,
-              "refId": "A"
-            },
-            {
-              "datasource": {
-                "type": "victoriametrics-datasource",
-                "uid": "$ds"
-              },
-              "editorMode": "code",
-              "exemplar": true,
-              "expr": "min(vm_concurrent_insert_capacity{job=~\"$job\", instance=~\"$instance\"}) by(job)",
-              "interval": "",
-              "legendFormat": "limit ({{job}})",
-              "range": true,
-              "refId": "B"
-            }
-          ],
-          "title": "Concurrent inserts ($instance)",
-          "type": "timeseries"
-        },
         {
           "datasource": {
             "type": "victoriametrics-datasource",
@@ -4400,8 +4409,8 @@
           "gridPos": {
             "h": 8,
             "w": 12,
-            "x": 12,
-            "y": 46
+            "x": 0,
+            "y": 93
           },
           "id": 77,
           "links": [],
diff --git a/dashboards/vmagent.json b/dashboards/vmagent.json
index 4c66b13b9..de0502665 100644
--- a/dashboards/vmagent.json
+++ b/dashboards/vmagent.json
@@ -2372,7 +2372,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "green"
+                    "color": "green",
+                    "value": null
                   },
                   {
                     "color": "red",
@@ -2388,7 +2389,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 4
+            "y": 36
           },
           "id": 92,
           "options": {
@@ -2474,7 +2475,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "green"
+                    "color": "green",
+                    "value": null
                   },
                   {
                     "color": "red",
@@ -2490,7 +2492,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 4
+            "y": 36
           },
           "id": 95,
           "options": {
@@ -2579,7 +2581,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "transparent"
+                    "color": "transparent",
+                    "value": null
                   },
                   {
                     "color": "red",
@@ -2595,7 +2598,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 12
+            "y": 44
           },
           "id": 98,
           "options": {
@@ -2684,7 +2687,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "transparent"
+                    "color": "transparent",
+                    "value": null
                   },
                   {
                     "color": "red",
@@ -2700,7 +2704,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 12
+            "y": 44
           },
           "id": 99,
           "options": {
@@ -2788,7 +2792,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "green"
+                    "color": "green",
+                    "value": null
                   },
                   {
                     "color": "red",
@@ -2804,7 +2809,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 20
+            "y": 52
           },
           "id": 79,
           "links": [],
@@ -2893,7 +2898,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "green"
+                    "color": "green",
+                    "value": null
                   },
                   {
                     "color": "red",
@@ -2909,7 +2915,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 20
+            "y": 52
           },
           "id": 18,
           "links": [
@@ -3003,7 +3009,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "green"
+                    "color": "green",
+                    "value": null
                   },
                   {
                     "color": "red",
@@ -3019,7 +3026,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 28
+            "y": 60
           },
           "id": 127,
           "links": [],
@@ -3106,7 +3113,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "green"
+                    "color": "green",
+                    "value": null
                   },
                   {
                     "color": "red",
@@ -3122,7 +3130,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 28
+            "y": 60
           },
           "id": 50,
           "options": {
@@ -3160,6 +3168,123 @@
           "title": "Invalid datapoints rate ($instance)",
           "type": "timeseries"
         },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "$ds"
+          },
+          "description": "Shows how many concurrent inserts (parsing and processing of scraped or ingested data) are taking place.\n\nIf the number of concurrent inserts hits the `limit` or is close to the `limit` constantly - it might be a sign of a resource shortage.\n\nIf vmagent's CPU usage and remote write connection saturation are at normal level, it might be that `-maxConcurrentInserts` cmd-line flag needs to be increased.",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "axisCenteredZero": false,
+                "axisColorMode": "text",
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 0,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": false,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "links": [],
+              "mappings": [],
+              "min": 0,
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              },
+              "unit": "short"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 12,
+            "x": 0,
+            "y": 68
+          },
+          "id": 130,
+          "links": [],
+          "options": {
+            "legend": {
+              "calcs": [
+                "mean",
+                "lastNotNull",
+                "max"
+              ],
+              "displayMode": "table",
+              "placement": "bottom",
+              "showLegend": true
+            },
+            "tooltip": {
+              "mode": "multi",
+              "sort": "desc"
+            }
+          },
+          "pluginVersion": "9.2.6",
+          "targets": [
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "$ds"
+              },
+              "editorMode": "code",
+              "exemplar": true,
+              "expr": "max_over_time(vm_concurrent_insert_current{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])",
+              "interval": "",
+              "legendFormat": "{{instance}} ({{job}})",
+              "range": true,
+              "refId": "A"
+            },
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "$ds"
+              },
+              "editorMode": "code",
+              "exemplar": true,
+              "expr": "min(vm_concurrent_insert_capacity{job=~\"$job\", instance=~\"$instance\"}) by(job)",
+              "interval": "",
+              "legendFormat": "limit ({{job}})",
+              "range": true,
+              "refId": "B"
+            }
+          ],
+          "title": "Concurrent inserts ($instance)",
+          "type": "timeseries"
+        },
         {
           "datasource": {
             "type": "prometheus",
@@ -3180,7 +3305,8 @@
                 "mode": "absolute",
                 "steps": [
                   {
-                    "color": "green"
+                    "color": "green",
+                    "value": null
                   },
                   {
                     "color": "red",
@@ -3220,7 +3346,7 @@
             "h": 7,
             "w": 24,
             "x": 0,
-            "y": 36
+            "y": 76
           },
           "id": 129,
           "options": {
@@ -3239,7 +3365,7 @@
               }
             ]
           },
-          "pluginVersion": "9.2.6",
+          "pluginVersion": "9.2.7",
           "targets": [
             {
               "datasource": {
@@ -4062,7 +4188,7 @@
             "h": 8,
             "w": 12,
             "x": 0,
-            "y": 38
+            "y": 85
           },
           "id": 73,
           "links": [],
@@ -4179,7 +4305,7 @@
             "h": 8,
             "w": 12,
             "x": 12,
-            "y": 38
+            "y": 85
           },
           "id": 131,
           "links": [],
@@ -4218,123 +4344,6 @@
           "title": "Rows rate  ($instance)",
           "type": "timeseries"
         },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "$ds"
-          },
-          "description": "Shows how many concurrent inserts are taking place.\n\nIf the number of concurrent inserts hitting the `limit` or is close to the `limit` constantly - it might be a sign of a resource shortage.\n\n If vmagent's CPU usage and remote write connection saturation are at normal level, it might be that `-maxConcurrentInserts` cmd-line flag need to be increased.",
-          "fieldConfig": {
-            "defaults": {
-              "color": {
-                "mode": "palette-classic"
-              },
-              "custom": {
-                "axisCenteredZero": false,
-                "axisColorMode": "text",
-                "axisLabel": "",
-                "axisPlacement": "auto",
-                "barAlignment": 0,
-                "drawStyle": "line",
-                "fillOpacity": 0,
-                "gradientMode": "none",
-                "hideFrom": {
-                  "legend": false,
-                  "tooltip": false,
-                  "viz": false
-                },
-                "lineInterpolation": "linear",
-                "lineWidth": 1,
-                "pointSize": 5,
-                "scaleDistribution": {
-                  "type": "linear"
-                },
-                "showPoints": "never",
-                "spanNulls": false,
-                "stacking": {
-                  "group": "A",
-                  "mode": "none"
-                },
-                "thresholdsStyle": {
-                  "mode": "off"
-                }
-              },
-              "links": [],
-              "mappings": [],
-              "min": 0,
-              "thresholds": {
-                "mode": "absolute",
-                "steps": [
-                  {
-                    "color": "green",
-                    "value": null
-                  },
-                  {
-                    "color": "red",
-                    "value": 80
-                  }
-                ]
-              },
-              "unit": "short"
-            },
-            "overrides": []
-          },
-          "gridPos": {
-            "h": 8,
-            "w": 12,
-            "x": 0,
-            "y": 46
-          },
-          "id": 130,
-          "links": [],
-          "options": {
-            "legend": {
-              "calcs": [
-                "mean",
-                "lastNotNull",
-                "max"
-              ],
-              "displayMode": "table",
-              "placement": "bottom",
-              "showLegend": true
-            },
-            "tooltip": {
-              "mode": "multi",
-              "sort": "desc"
-            }
-          },
-          "pluginVersion": "9.2.6",
-          "targets": [
-            {
-              "datasource": {
-                "type": "prometheus",
-                "uid": "$ds"
-              },
-              "editorMode": "code",
-              "exemplar": true,
-              "expr": "max_over_time(vm_concurrent_insert_current{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])",
-              "interval": "",
-              "legendFormat": "{{instance}} ({{job}})",
-              "range": true,
-              "refId": "A"
-            },
-            {
-              "datasource": {
-                "type": "prometheus",
-                "uid": "$ds"
-              },
-              "editorMode": "code",
-              "exemplar": true,
-              "expr": "min(vm_concurrent_insert_capacity{job=~\"$job\", instance=~\"$instance\"}) by(job)",
-              "interval": "",
-              "legendFormat": "limit ({{job}})",
-              "range": true,
-              "refId": "B"
-            }
-          ],
-          "title": "Concurrent inserts ($instance)",
-          "type": "timeseries"
-        },
         {
           "datasource": {
             "type": "prometheus",
@@ -4399,8 +4408,8 @@
           "gridPos": {
             "h": 8,
             "w": 12,
-            "x": 12,
-            "y": 46
+            "x": 0,
+            "y": 93
           },
           "id": 77,
           "links": [],
diff --git a/deployment/docker/README.md b/deployment/docker/README.md
index ed9364557..fe368e880 100644
--- a/deployment/docker/README.md
+++ b/deployment/docker/README.md
@@ -42,30 +42,36 @@ The communication scheme between components is the following:
   and recording rules back to it;
 * [alertmanager](#alertmanager) is configured to receive notifications from `vmalert`.
 
-To access `vmalert` use link [http://localhost:8428/vmalert](http://localhost:8428/vmalert/).
+To access Grafana use link [http://localhost:3000](http://localhost:3000).
 
 To access [vmui](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#vmui)
 use link [http://localhost:8428/vmui](http://localhost:8428/vmui).
 
+To access `vmalert` use link [http://localhost:8428/vmalert](http://localhost:8428/vmalert/).
+
+
 ## VictoriaMetrics cluster
 
 VictoriaMetrics cluster environment consists of `vminsert`, `vmstorage` and `vmselect` components.
-`vmselect` has exposed port `:8481`, `vminsert` has exposed port `:8480` and the rest of components 
-are available only inside the environment.
+`vminsert` has exposed port `:8480`, access to `vmselect` components goes through `vmauth` on port `:8427`,
+and the rest of components are available only inside the environment.
 
 The communication scheme between components is the following:
 * [vmagent](#vmagent) sends scraped metrics to `vminsert`;
 * `vminsert` forwards data to `vmstorage`;
-* `vmselect` is connected to `vmstorage` for querying data;
-* [grafana](#grafana) is configured with datasource pointing to `vmselect`;
-* [vmalert](#vmalert) is configured to query `vmselect` and send alerts state
+* `vmselect`s are connected to `vmstorage` for querying data;
+* [vmauth](#vmauth) balances incoming read requests among `vmselect`s;
+* [grafana](#grafana) is configured with datasource pointing to `vmauth`;
+* [vmalert](#vmalert) is configured to query `vmselect`s via `vmauth` and send alerts state
   and recording rules to `vminsert`;
 * [alertmanager](#alertmanager) is configured to receive notifications from `vmalert`.
 
-To access `vmalert` use link [http://localhost:8481/select/0/prometheus/vmalert](http://localhost:8481/select/0/prometheus/vmalert/).
+To access Grafana use link [http://localhost:3000](http://localhost:3000).
 
-To access [vmui](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#vmui) 
-use link [http://localhost:8481/select/0/prometheus/vmui](http://localhost:8481/select/0/prometheus/vmui).
+To access [vmui](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#vmui)
+use link [http://localhost:8427/select/0/prometheus/vmui/](http://localhost:8427/select/0/prometheus/vmui/).
+
+To access `vmalert` use link [http://localhost:8427/select/0/prometheus/vmalert/](http://localhost:8427/select/0/prometheus/vmalert/).
 
 ## vmagent
 
@@ -75,6 +81,13 @@ with listed targets for scraping.
 
 [Web interface link](http://localhost:8429/).
 
+## vmauth
+
+[vmauth](https://docs.victoriametrics.com/vmauth.html) acts as a [balancer](https://docs.victoriametrics.com/vmauth.html#load-balancing)
+to spread the load across `vmselect`'s. [Grafana](#grafana) and [vmalert](#vmalert) use vmauth for read queries.
+vmauth config is available [here](ttps://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/auth-cluster.yml)
+
+
 ## vmalert
 
 vmalert evaluates alerting rules [alerts.yml](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts.yml)
diff --git a/deployment/docker/auth-cluster.yml b/deployment/docker/auth-cluster.yml
new file mode 100644
index 000000000..820fa0f40
--- /dev/null
+++ b/deployment/docker/auth-cluster.yml
@@ -0,0 +1,6 @@
+# balance load among vmselects
+# see https://docs.victoriametrics.com/vmauth.html#load-balancing
+unauthorized_user:
+  url_prefix:
+    - http://vmselect-1:8481
+    - http://vmselect-2:8481
\ No newline at end of file
diff --git a/deployment/docker/docker-compose-cluster.yml b/deployment/docker/docker-compose-cluster.yml
index 307765cad..35945cf1c 100644
--- a/deployment/docker/docker-compose-cluster.yml
+++ b/deployment/docker/docker-compose-cluster.yml
@@ -2,7 +2,7 @@ version: '3.5'
 services:
   vmagent:
     container_name: vmagent
-    image: victoriametrics/vmagent:v1.93.4
+    image: victoriametrics/vmagent:v1.93.5
     depends_on:
       - "vminsert"
     ports:
@@ -19,7 +19,8 @@ services:
     container_name: grafana
     image: grafana/grafana:9.2.7
     depends_on:
-      - "vmselect"
+      - "vmselect-1"
+      - "vmselect-2"
     ports:
       - 3000:3000
     restart: always
@@ -32,7 +33,7 @@ services:
 
   vmstorage-1:
     container_name: vmstorage-1
-    image: victoriametrics/vmstorage:v1.93.4-cluster
+    image: victoriametrics/vmstorage:v1.93.5-cluster
     ports:
       - 8482
       - 8400
@@ -44,7 +45,7 @@ services:
     restart: always
   vmstorage-2:
     container_name: vmstorage-2
-    image: victoriametrics/vmstorage:v1.93.4-cluster
+    image: victoriametrics/vmstorage:v1.93.5-cluster
     ports:
       - 8482
       - 8400
@@ -54,9 +55,10 @@ services:
     command:
       - '--storageDataPath=/storage'
     restart: always
+
   vminsert:
     container_name: vminsert
-    image: victoriametrics/vminsert:v1.93.4-cluster
+    image: victoriametrics/vminsert:v1.93.5-cluster
     depends_on:
       - "vmstorage-1"
       - "vmstorage-2"
@@ -66,9 +68,24 @@ services:
     ports:
       - 8480:8480
     restart: always
-  vmselect:
-    container_name: vmselect
-    image: victoriametrics/vmselect:v1.93.4-cluster
+
+  vmselect-1:
+    container_name: vmselect-1
+    image: victoriametrics/vmselect:v1.93.5-cluster
+    depends_on:
+      - "vmstorage-1"
+      - "vmstorage-2"
+    command:
+      - '--storageNode=vmstorage-1:8401'
+      - '--storageNode=vmstorage-2:8401'
+      - '--vmalert.proxyURL=http://vmalert:8880'
+    ports:
+      - 8481
+    restart: always
+
+  vmselect-2:
+    container_name: vmselect-2
+    image: victoriametrics/vmselect:v1.93.5-cluster
     depends_on:
       - "vmstorage-1"
       - "vmstorage-2"
@@ -77,14 +94,30 @@ services:
       - '--storageNode=vmstorage-2:8401'
       - '--vmalert.proxyURL=http://vmalert:8880'
     ports:
-      - 8481:8481
+      - 8481
+    restart: always
+
+  vmauth:
+    container_name: vmauth
+    image: victoriametrics/vmauth:v1.93.5
+    depends_on:
+      - "vmselect-1"
+      - "vmselect-2"
+    volumes:
+      - ./auth-cluster.yml:/etc/auth.yml
+#      - /var/run/docker.sock:/var/run/docker.sock
+    command:
+      - '--auth.config=/etc/auth.yml'
+    ports:
+      - 8427:8427
     restart: always
 
   vmalert:
     container_name: vmalert
-    image: victoriametrics/vmalert:v1.93.4
+    image: victoriametrics/vmalert:v1.93.5
     depends_on:
-      - "vmselect"
+      - "vmselect-1"
+      - "vmselect-2"
     ports:
       - 8880:8880
     volumes:
@@ -93,8 +126,8 @@ services:
       - ./alerts-vmagent.yml:/etc/alerts/alerts-vmagent.yml
       - ./alerts-vmalert.yml:/etc/alerts/alerts-vmalert.yml
     command:
-      - '--datasource.url=http://vmselect:8481/select/0/prometheus'
-      - '--remoteRead.url=http://vmselect:8481/select/0/prometheus'
+      - '--datasource.url=http://vmauth:8427/select/0/prometheus'
+      - '--remoteRead.url=http://vmauth:8427/select/0/prometheus'
       - '--remoteWrite.url=http://vminsert:8480/insert/0/prometheus'
       - '--notifier.url=http://alertmanager:9093/'
       - '--rule=/etc/alerts/*.yml'
diff --git a/deployment/docker/prometheus-cluster.yml b/deployment/docker/prometheus-cluster.yml
index 32336929b..e765b0860 100644
--- a/deployment/docker/prometheus-cluster.yml
+++ b/deployment/docker/prometheus-cluster.yml
@@ -13,7 +13,7 @@ scrape_configs:
       - targets: ['vminsert:8480']
   - job_name: 'vmselect'
     static_configs:
-      - targets: ['vmselect:8481']
+      - targets: ['vmselect-1:8481', 'vmselect-2:8481']
   - job_name: 'vmstorage'
     static_configs:
       - targets: ['vmstorage-1:8482', 'vmstorage-2:8482']
\ No newline at end of file
diff --git a/deployment/docker/provisioning/datasources/datasource.yml b/deployment/docker/provisioning/datasources/datasource.yml
index e16c273c4..c0a7a20c9 100644
--- a/deployment/docker/provisioning/datasources/datasource.yml
+++ b/deployment/docker/provisioning/datasources/datasource.yml
@@ -10,5 +10,5 @@ datasources:
     - name: VictoriaMetrics - cluster
       type: prometheus
       access: proxy
-      url: http://vmselect:8481/select/0/prometheus
+      url: http://vmauth:8427/select/0/prometheus
       isDefault: false
\ No newline at end of file
diff --git a/docs/Articles.md b/docs/Articles.md
index 9e7e679ea..96e9c7a9c 100644
--- a/docs/Articles.md
+++ b/docs/Articles.md
@@ -78,6 +78,7 @@ See also [case studies](https://docs.victoriametrics.com/CaseStudies.html).
 * [VictoriaMetrics: an overview and its use instead of Prometheus](https://rtfm.co.ua/en/victoriametrics-an-overview-and-its-use-instead-of-prometheus/)
 * [VictoriaMetrics: deploying a Kubernetes monitoring stack](https://rtfm.co.ua/en/victoriametrics-deploying-a-kubernetes-monitoring-stack/)
 * [Better, Faster, Cheaper: How Grammarly Improved Monitoring by Over 10x with VictoriaMetrics](https://www.grammarly.com/blog/engineering/monitoring-with-victoriametrics/)
+* [VictoriaMetrics, a stress-free Prometheus Remote Storage for 1 Billion metrics](https://medium.com/criteo-engineering/victoriametrics-a-prometheus-remote-storage-solution-57081a3d8e61)
 
 ## Our articles
 
@@ -136,3 +137,4 @@ See also [case studies](https://docs.victoriametrics.com/CaseStudies.html).
 * [VictoriaMetrics Meetup December 2022](https://www.youtube.com/watch?v=Mesc6JBFNhQ). See also [slides for "VictoriaMetrics 2022: new features" talk](https://docs.google.com/presentation/d/1jI7XZoodmuzLymdu4MToG9onAKQjzCNwMO2NDupyUkQ/edit?usp=sharing).
 * [Comparing Thanos to VictoriaMetrics cluster](https://faun.pub/comparing-thanos-to-victoriametrics-cluster-b193bea1683)
 * [Evaluation performance and correctness: VictoriaMetrics response](https://valyala.medium.com/evaluating-performance-and-correctness-victoriametrics-response-e27315627e87)
+* [How to reduce expenses on monitoring slides](https://www.slideshare.net/RomanKhavronenko/how-to-reduce-expenses-on-monitoringpdf)
\ No newline at end of file
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index 115b61ac2..bc5b82fcb 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -21,6 +21,10 @@ The following `tip` changes can be tested by building VictoriaMetrics components
 * [How to build vmauth](https://docs.victoriametrics.com/vmauth.html#how-to-build-from-sources)
 * [How to build vmctl](https://docs.victoriametrics.com/vmctl.html#how-to-build)
 
+Metrics of the latest version of VictoriaMetrics cluster are available for viewing at our 
+[sandbox](https://play-grafana.victoriametrics.com/d/oS7Bi_0Wz_vm/victoriametrics-cluster-vm).
+The sandbox cluster installation is running under the constant load generated by 
+[prometheus-benchmark](https://github.com/VictoriaMetrics/prometheus-benchmark) and used for testing latest releases.
 
 ## tip
 
@@ -37,15 +41,40 @@ The following `tip` changes can be tested by building VictoriaMetrics components
 * FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add button for auto-formatting PromQL/MetricsQL queries. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4681). Thanks to @aramattamara for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4694).
 * FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): improve accessibility score to 100 according to [Google's Lighthouse](https://developer.chrome.com/docs/lighthouse/accessibility/) tests.
 * FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): organize `min`, `max`, `median` values on the chart legend and tooltips for better visibility.
+* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add explanation about statistic inaccuracy in a cluster version. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3070).
 * FEATURE: dashboards: provide copies of Grafana dashboards alternated with VictoriaMetrics datasource at [dashboards/vm](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/dashboards/vm).
-* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth.html): added ability to set, override and clear request and response headers on a per-user and per-path basis. See [this i
-ssue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4825) and [these docs](https://docs.victoriametrics.com/vmauth.html#auth-config) for details.
+* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth.html): added ability to set, override and clear request and response headers on a per-user and per-path basis. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4825) and [these docs](https://docs.victoriametrics.com/vmauth.html#auth-config) for details.
 * FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth.html): add ability to retry requests to the [remaining backends](https://docs.victoriametrics.com/vmauth.html#load-balancing) if they return response status codes specified in the `retry_status_codes` list. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4893).
+* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth.html): expose metrics `vmauth_config_last_reload_*` for tracking the state of config reloads, similarly to vmagent/vmalert components.
+* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth.html): do not print logs like `SIGHUP received...` once per configured `-configCheckInterval` cmd-line flag. This log will be printed only if config reload was invoked manually. 
 * FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): add `eval_offset` attribute for [Groups](https://docs.victoriametrics.com/vmalert.html#groups). If specified, Group will be evaluated at the exact time offset on the range of [0...evaluationInterval]. The setting might be useful for cron-like rules which must be evaluated at specific moments of time. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3409) for details.
+* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): validate [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html) function names in alerting and recording rules when `vmalert` runs with `-dryRun` command-line flag. Previously it was allowed to use unknown (aka invalid) MetricsQL function names there. For example, `foo()` was counted as a valid query. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4933).
 * FEATURE: limit the length of string params in log messages to 500 chars. Longer string params are replaced with the `first_250_chars..last_250_chars`. This prevents from too long log lines, which can be emitted by VictoriaMetrics components.
+* FEATURE: [docker compose environment](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker): add `vmauth` component to cluster's docker-compose example for balancing load among multiple `vmselect` components.
+* FEATURE: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): make sure that `q2` series are returned after `q1` series in the results of `q1 or q2` query, in the same way as Prometheus does. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4763).
+* FEATURE: stop exposing `vm_merge_need_free_disk_space` metric, since it has been appeared that it confuses users while doesn't bring any useful information. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686#issuecomment-1733844128).
 
 * BUGFIX: [Official Grafana dashboards for VictoriaMetrics](https://grafana.com/orgs/victoriametrics): fix display of ingested rows rate for `Samples ingested/s` and `Samples rate` panels for vmagent's dasbhoard. Previously, not all ingested protocols were accounted in these panels. An extra panel `Rows rate` was added to `Ingestion` section to display the split for rows ingested rate by protocol.
+* BUGFIX: [Official Grafana dashboards for VictoriaMetrics](https://grafana.com/orgs/victoriametrics): move vmagent's `Concurrent inserts` panel to Troubleshooting section from `Ingestion` section because this panel is related to both: scraped and ingested data. Before, it could have give a misleading impression that it is related to ingested metrics only. 
 * BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix the bug causing render looping when switching to heatmap.
+* BUGFIX: [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise.html) validate `-dedup.minScrapeInterval` value and `-downsampling.period` intervals are multiples of each other. See [these docs](https://docs.victoriametrics.com/#downsampling).
+* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup.html): properly copy `appliedRetention.txt` files inside `<-storageDataPath>/{data}` folders during [incremental backups](https://docs.victoriametrics.com/vmbackup.html#incremental-backups). Previously the new `appliedRetention.txt` could be skipped during incremental backups, which could lead to increased load on storage after restoring from backup. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5005).
+* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): suppress `context canceled` error messages in logs when `vmagent` is reloading service discovery config. This error could appear starting from [v1.93.5](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.5). See [this PR](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5048). 
+* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): allow passing [median_over_time](https://docs.victoriametrics.com/MetricsQL.html#median_over_time) to [aggr_over_time](https://docs.victoriametrics.com/MetricsQL.html#aggr_over_time). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5034).
+* BUGFIX: [vminsert](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html): fixed ingestion via multitenant url for opentsdbhttp. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5061).
+
+## [v1.93.5](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.5)
+
+Released at 2023-09-19
+
+**v1.93.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes.
+The v1.93.x line will be supported for at least 12 months since [v1.93.0](https://docs.victoriametrics.com/CHANGELOG.html#v1930) release**
+
+* BUGFIX: [storage](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html): prevent from livelock when [forced merge](https://docs.victoriametrics.com/#forced-merge) is called under high data ingestion. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4987).
+* BUGFIX: [Graphite Render API](https://docs.victoriametrics.com/#graphite-render-api-usage): correctly return `null` instead of `Inf` in JSON query responses. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3783).
+* BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup.html): properly copy `parts.json` files inside `<-storageDataPath>/{data,indexdb}` folders during [incremental backups](https://docs.victoriametrics.com/vmbackup.html#incremental-backups). Previously the new `parts.json` could be skipped during incremental backups, which could lead to inability to restore from the backup. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5005). This issue has been introduced in [v1.90.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.90.0).
+* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): properly close connections to Kubernetes API server after the change in `selectors` or `namespaces` sections of [kubernetes_sd_configs](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs). Previously `vmagent` could continue polling Kubernetes API server with the old `selectors` or `namespaces` configs additionally to polling new configs. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4850).
+* BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth.html): prevent configuration reloading if there were no changes in config. This improves memory usage when `-configCheckInterval` cmd-line flag is configured and config has extensive list of regexp expressions requiring additional memory on parsing. 
 
 ## [v1.93.4](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.4)
 
diff --git a/docs/CaseStudies.md b/docs/CaseStudies.md
index 3ab0d3279..0e94fdd2b 100644
--- a/docs/CaseStudies.md
+++ b/docs/CaseStudies.md
@@ -23,6 +23,7 @@ where you can chat with VictoriaMetrics users to get additional references, revi
   - [Brandwatch](#brandwatch)
   - [CERN](#cern)
   - [COLOPL](#colopl)
+  - [Criteo](#criteo)
   - [Dig Security](#dig-security)
   - [Fly.io](#flyio)
   - [German Research Center for Artificial Intelligence](#german-research-center-for-artificial-intelligence)
@@ -242,6 +243,13 @@ after evaulating the following remote storage solutions for Prometheus:
 See [slides](https://speakerdeck.com/inletorder/monitoring-platform-with-victoria-metrics) and [video](https://www.youtube.com/watch?v=hUpHIluxw80)
 from `Large-scale, super-load system monitoring platform built with VictoriaMetrics` talk at [Prometheus Meetup Tokyo #3](https://prometheus.connpass.com/event/157721/).
 
+## Criteo
+
+[Criteo](https://www.criteo.com/) is a global technology company that helps marketers and media owners reach their goals through the world’s leading Commerce Media Platform.
+
+See [this blog post](https://medium.com/criteo-engineering/victoriametrics-a-prometheus-remote-storage-solution-57081a3d8e61) on how Criteo started using VictoriaMetrics
+and why they prefer VictoriaMetrics over competing solutions.
+
 ## Dig Security
 
 [Dig Security](https://www.dig.security) is a cloud data security startup with 50+ employees that provides real-time visibility, control, and protection of data assets.
diff --git a/docs/Cluster-VictoriaMetrics.md b/docs/Cluster-VictoriaMetrics.md
index c69ba299d..4c24ca044 100644
--- a/docs/Cluster-VictoriaMetrics.md
+++ b/docs/Cluster-VictoriaMetrics.md
@@ -821,15 +821,15 @@ Below is the output for `/path/to/vminsert -help`:
   -cacheExpireDuration duration
      Items are removed from in-memory caches after they aren't accessed for this duration. Lower values may reduce memory usage at the cost of higher CPU usage. See also -prevCacheRemovalPercent (default 30m0s)
   -cluster.tls
-     Whether to use TLS for connections to -storageNode. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection . This flag is available only in enterprise version of VictoriaMetrics
+     Whether to use TLS for connections to -storageNode. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -cluster.tlsCAFile string
-     Path to TLS CA file to use for verifying certificates provided by -storageNode if -cluster.tls flag is set. By default system CA is used. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection . This flag is available only in enterprise version of VictoriaMetrics
+     Path to TLS CA file to use for verifying certificates provided by -storageNode if -cluster.tls flag is set. By default system CA is used. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -cluster.tlsCertFile string
-     Path to client-side TLS certificate file to use when connecting to -storageNode if -cluster.tls flag is set. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection . This flag is available only in enterprise version of VictoriaMetrics
+     Path to client-side TLS certificate file to use when connecting to -storageNode if -cluster.tls flag is set. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -cluster.tlsInsecureSkipVerify
-     Whether to skip verification of TLS certificates provided by -storageNode nodes if -cluster.tls flag is set. Note that disabled TLS certificate verification breaks security. This flag is available only in enterprise version of VictoriaMetrics
+     Whether to skip verification of TLS certificates provided by -storageNode nodes if -cluster.tls flag is set. Note that disabled TLS certificate verification breaks security. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -cluster.tlsKeyFile string
-     Path to client-side TLS key file to use when connecting to -storageNode if -cluster.tls flag is set. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection . This flag is available only in enterprise version of VictoriaMetrics
+     Path to client-side TLS key file to use when connecting to -storageNode if -cluster.tls flag is set. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -clusternativeListenAddr string
      TCP address to listen for data from other vminsert nodes in multi-level cluster setup. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multi-level-cluster-setup . Usually :8400 should be set to match default vmstorage port for vminsert. Disabled work if empty
   -csvTrimTimestamp duration
@@ -937,7 +937,7 @@ Below is the output for `/path/to/vminsert -help`:
   -loggerWarnsPerSecondLimit int
      Per-second limit on the number of WARN messages. If more than the given number of warns are emitted per second, then the remaining warns are suppressed. Zero values disable the rate limit
   -maxConcurrentInserts int
-     The maximum number of concurrent insert requests. The default value should work for most cases, since it minimizes memory usage. The default value can be increased when clients send data over slow networks. See also -insert.maxQueueDuration (default 8)
+     The maximum number of concurrent insert requests. Default value should work for most cases, since it minimizes the memory usage. The default value can be increased when clients send data over slow networks. See also -insert.maxQueueDuration (default 8)
   -maxInsertRequestSize size
      The maximum size in bytes of a single Prometheus remote_write API request
      Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 33554432)
@@ -1025,15 +1025,15 @@ Below is the output for `/path/to/vmselect -help`:
   -cacheExpireDuration duration
      Items are removed from in-memory caches after they aren't accessed for this duration. Lower values may reduce memory usage at the cost of higher CPU usage. See also -prevCacheRemovalPercent (default 30m0s)
   -cluster.tls
-     Whether to use TLS for connections to -storageNode. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection . This flag is available only in enterprise version of VictoriaMetrics
+     Whether to use TLS for connections to -storageNode. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -cluster.tlsCAFile string
-     Path to TLS CA file to use for verifying certificates provided by -storageNode if -cluster.tls flag is set. By default system CA is used. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection . This flag is available only in enterprise version of VictoriaMetrics
+     Path to TLS CA file to use for verifying certificates provided by -storageNode if -cluster.tls flag is set. By default system CA is used. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -cluster.tlsCertFile string
-     Path to client-side TLS certificate file to use when connecting to -storageNode if -cluster.tls flag is set. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection . This flag is available only in enterprise version of VictoriaMetrics
+     Path to client-side TLS certificate file to use when connecting to -storageNode if -cluster.tls flag is set. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -cluster.tlsInsecureSkipVerify
-     Whether to skip verification of TLS certificates provided by -storageNode nodes if -cluster.tls flag is set. Note that disabled TLS certificate verification breaks security. This flag is available only in enterprise version of VictoriaMetrics
+     Whether to skip verification of TLS certificates provided by -storageNode nodes if -cluster.tls flag is set. Note that disabled TLS certificate verification breaks security. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -cluster.tlsKeyFile string
-     Path to client-side TLS key file to use when connecting to -storageNode if -cluster.tls flag is set. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection . This flag is available only in enterprise version of VictoriaMetrics
+     Path to client-side TLS key file to use when connecting to -storageNode if -cluster.tls flag is set. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -clusternative.disableCompression
      Whether to disable compression of the data sent to vmselect via -clusternativeListenAddr. This reduces CPU usage at the cost of higher network bandwidth usage
   -clusternative.maxConcurrentRequests int
@@ -1047,18 +1047,18 @@ Below is the output for `/path/to/vmselect -help`:
   -clusternative.maxTagValues int
      The maximum number of tag values returned per search at -clusternativeListenAddr (default 100000)
   -clusternative.tls
-     Whether to use TLS when accepting connections at -clusternativeListenAddr. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection
+     Whether to use TLS when accepting connections at -clusternativeListenAddr. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -clusternative.tlsCAFile string
-     Path to TLS CA file to use for verifying certificates provided by vmselect, which connects at -clusternativeListenAddr if -clusternative.tls flag is set. By default system CA is used. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection
+     Path to TLS CA file to use for verifying certificates provided by vmselect, which connects at -clusternativeListenAddr if -clusternative.tls flag is set. By default system CA is used. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -clusternative.tlsCertFile string
-     Path to server-side TLS certificate file to use when accepting connections at -clusternativeListenAddr if -clusternative.tls flag is set. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection
+     Path to server-side TLS certificate file to use when accepting connections at -clusternativeListenAddr if -clusternative.tls flag is set. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -clusternative.tlsCipherSuites array
-     Optional list of TLS cipher suites used for connections at -clusternativeListenAddr if -clusternative.tls flag is set. See the list of supported cipher suites at https://pkg.go.dev/crypto/tls#pkg-constants
+     Optional list of TLS cipher suites used for connections at -clusternativeListenAddr if -clusternative.tls flag is set. See the list of supported cipher suites at https://pkg.go.dev/crypto/tls#pkg-constants . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
      Supports an array of values separated by comma or specified via multiple flags.
   -clusternative.tlsInsecureSkipVerify
-     Whether to skip verification of TLS certificates provided by vmselect, which connects to -clusternativeListenAddr if -clusternative.tls flag is set. Note that disabled TLS certificate verification breaks security
+     Whether to skip verification of TLS certificates provided by vmselect, which connects to -clusternativeListenAddr if -clusternative.tls flag is set. Note that disabled TLS certificate verification breaks security. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -clusternative.tlsKeyFile string
-     Path to server-side TLS key file to use when accepting connections at -clusternativeListenAddr if -clusternative.tls flag is set. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection
+     Path to server-side TLS key file to use when accepting connections at -clusternativeListenAddr if -clusternative.tls flag is set. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection . This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise.html
   -clusternativeListenAddr string
      TCP address to listen for requests from other vmselect nodes in multi-level cluster setup. See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multi-level-cluster-setup . Usually :8401 should be set to match default vmstorage port for vmselect. Disabled work if empty
   -dedup.minScrapeInterval duration
@@ -1367,7 +1367,7 @@ Below is the output for `/path/to/vmstorage -help`:
   -loggerWarnsPerSecondLimit int
      Per-second limit on the number of WARN messages. If more than the given number of warns are emitted per second, then the remaining warns are suppressed. Zero values disable the rate limit
   -maxConcurrentInserts int
-     The maximum number of concurrent insert requests. The default value should work for most cases, since it minimizes memory usage. The default value can be increased when clients send data over slow networks. See also -insert.maxQueueDuration (default 8)
+     The maximum number of concurrent insert requests. Default value should work for most cases, since it minimizes the memory usage. The default value can be increased when clients send data over slow networks. See also -insert.maxQueueDuration (default 8)
   -memory.allowedBytes size
      Allowed size of system memory VictoriaMetrics caches may occupy. This option overrides -memory.allowedPercent if set to a non-zero value. Too low a value may increase the cache miss rate usually resulting in higher CPU and disk IO usage. Too high a value may evict too much data from the OS page cache resulting in higher disk IO usage
      Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 0)
diff --git a/docs/README.md b/docs/README.md
index ce3e62819..fb316fa7d 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -113,6 +113,7 @@ Case studies:
 * [Brandwatch](https://docs.victoriametrics.com/CaseStudies.html#brandwatch)
 * [CERN](https://docs.victoriametrics.com/CaseStudies.html#cern)
 * [COLOPL](https://docs.victoriametrics.com/CaseStudies.html#colopl)
+* [Criteo](https://docs.victoriametrics.com/CaseStudies.html#criteo)
 * [Dig Security](https://docs.victoriametrics.com/CaseStudies.html#dig-security)
 * [Fly.io](https://docs.victoriametrics.com/CaseStudies.html#flyio)
 * [German Research Center for Artificial Intelligence](https://docs.victoriametrics.com/CaseStudies.html#german-research-center-for-artificial-intelligence)
@@ -407,14 +408,16 @@ matching the specified [series selector](https://prometheus.io/docs/prometheus/l
 
 Cardinality explorer is built on top of [/api/v1/status/tsdb](#tsdb-stats).
 
+See [cardinality explorer playground](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/cardinality).
+See the example of using the cardinality explorer [here](https://victoriametrics.com/blog/cardinality-explorer/).
+
+## Cardinality explorer statistic inaccuracy
+
 In [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html) each vmstorage tracks the stored time series individually.
 vmselect requests stats via [/api/v1/status/tsdb](#tsdb-stats) API from each vmstorage node and merges the results by summing per-series stats.
 This may lead to inflated values when samples for the same time series are spread across multiple vmstorage nodes
 due to [replication](#replication) or [rerouting](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html?highlight=re-routes#cluster-availability).
 
-See [cardinality explorer playground](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/cardinality).
-See the example of using the cardinality explorer [here](https://victoriametrics.com/blog/cardinality-explorer/).
-
 ## How to apply new config to VictoriaMetrics
 
 VictoriaMetrics is configured via command-line flags, so it must be restarted when new command-line flags should be applied:
@@ -619,6 +622,28 @@ Some plugins for Telegraf such as [fluentd](https://github.com/fangli/fluent-plu
 or [Juniper/jitmon](https://github.com/Juniper/jtimon) send `SHOW DATABASES` query to `/query` and expect a particular database name in the response.
 Comma-separated list of expected databases can be passed to VictoriaMetrics via `-influx.databaseNames` command-line flag.
 
+### How to send data in InfluxDB v2 format
+
+VictoriaMetrics exposes endpoint for InfluxDB v2 HTTP API at `/influx/api/v2/write` and `/api/v2/write`.
+
+
+In order to write data with InfluxDB line protocol to local VictoriaMetrics using `curl`:
+
+<div class="with-copy" markdown="1">
+
+```console
+curl -d 'measurement,tag1=value1,tag2=value2 field1=123,field2=1.23' -X POST 'http://localhost:8428/api/v2/write'
+```
+
+</div>
+
+The `/api/v1/export` endpoint should return the following response:
+
+```json
+{"metric":{"__name__":"measurement_field1","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1695902762311]}
+{"metric":{"__name__":"measurement_field2","tag1":"value1","tag2":"value2"},"values":[1.23],"timestamps":[1695902762311]}
+```
+
 ## How to send data from Graphite-compatible agents such as [StatsD](https://github.com/etsy/statsd)
 
 Enable Graphite receiver in VictoriaMetrics by setting `-graphiteListenAddr` command line flag. For instance,
@@ -1697,6 +1722,10 @@ Downsampling happens during [background merges](https://docs.victoriametrics.com
 and can't be performed if there is not enough of free disk space or if vmstorage 
 is in [read-only mode](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#readonly-mode).
 
+Please, note that intervals of `-downsampling.period` must be multiples of each other. 
+In case [deduplication](https://docs.victoriametrics.com/#deduplication) is enabled value of `-dedup.minScrapeInterval` must also be multiple of `-downsampling.period` intervals.
+This is required to ensure consistency of deduplication and downsampling results.
+
 The downsampling can be evaluated for free by downloading and using enterprise binaries from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
 
 ## Multi-tenancy
@@ -1936,9 +1965,6 @@ and [cardinality explorer docs](#cardinality-explorer).
   has at least 20% of free space. The remaining amount of free space
   can be [monitored](#monitoring) via `vm_free_disk_space_bytes` metric. The total size of data
   stored on the disk can be monitored via sum of `vm_data_size_bytes` metrics.
-  See also `vm_merge_need_free_disk_space` metrics, which are set to values higher than 0
-  if background merge cannot be initiated due to free disk space shortage. The value shows the number of per-month partitions,
-  which would start background merge if they had more free disk space.
 
 * VictoriaMetrics buffers incoming data in memory for up to a few seconds before flushing it to persistent storage.
   This may lead to the following "issues":
diff --git a/docs/Single-server-VictoriaMetrics.md b/docs/Single-server-VictoriaMetrics.md
index 0d99caf0c..d7c0b73f3 100644
--- a/docs/Single-server-VictoriaMetrics.md
+++ b/docs/Single-server-VictoriaMetrics.md
@@ -121,6 +121,7 @@ Case studies:
 * [Brandwatch](https://docs.victoriametrics.com/CaseStudies.html#brandwatch)
 * [CERN](https://docs.victoriametrics.com/CaseStudies.html#cern)
 * [COLOPL](https://docs.victoriametrics.com/CaseStudies.html#colopl)
+* [Criteo](https://docs.victoriametrics.com/CaseStudies.html#criteo)
 * [Dig Security](https://docs.victoriametrics.com/CaseStudies.html#dig-security)
 * [Fly.io](https://docs.victoriametrics.com/CaseStudies.html#flyio)
 * [German Research Center for Artificial Intelligence](https://docs.victoriametrics.com/CaseStudies.html#german-research-center-for-artificial-intelligence)
@@ -415,13 +416,16 @@ matching the specified [series selector](https://prometheus.io/docs/prometheus/l
 
 Cardinality explorer is built on top of [/api/v1/status/tsdb](#tsdb-stats).
 
+See [cardinality explorer playground](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/cardinality).
+See the example of using the cardinality explorer [here](https://victoriametrics.com/blog/cardinality-explorer/).
+
+## Cardinality explorer statistic inaccuracy
+
 In [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html) each vmstorage tracks the stored time series individually.
 vmselect requests stats via [/api/v1/status/tsdb](#tsdb-stats) API from each vmstorage node and merges the results by summing per-series stats.
 This may lead to inflated values when samples for the same time series are spread across multiple vmstorage nodes
 due to [replication](#replication) or [rerouting](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html?highlight=re-routes#cluster-availability).
 
-See [cardinality explorer playground](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/cardinality).
-See the example of using the cardinality explorer [here](https://victoriametrics.com/blog/cardinality-explorer/).
 
 ## How to apply new config to VictoriaMetrics
 
@@ -627,6 +631,28 @@ Some plugins for Telegraf such as [fluentd](https://github.com/fangli/fluent-plu
 or [Juniper/jitmon](https://github.com/Juniper/jtimon) send `SHOW DATABASES` query to `/query` and expect a particular database name in the response.
 Comma-separated list of expected databases can be passed to VictoriaMetrics via `-influx.databaseNames` command-line flag.
 
+### How to send data in InfluxDB v2 format
+
+VictoriaMetrics exposes endpoint for InfluxDB v2 HTTP API at `/influx/api/v2/write` and `/api/v2/write`.
+
+
+In order to write data with InfluxDB line protocol to local VictoriaMetrics using `curl`:
+
+<div class="with-copy" markdown="1">
+
+```console
+curl -d 'measurement,tag1=value1,tag2=value2 field1=123,field2=1.23' -X POST 'http://localhost:8428/api/v2/write'
+```
+
+</div>
+
+The `/api/v1/export` endpoint should return the following response:
+
+```json
+{"metric":{"__name__":"measurement_field1","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1695902762311]}
+{"metric":{"__name__":"measurement_field2","tag1":"value1","tag2":"value2"},"values":[1.23],"timestamps":[1695902762311]}
+```
+
 ## How to send data from Graphite-compatible agents such as [StatsD](https://github.com/etsy/statsd)
 
 Enable Graphite receiver in VictoriaMetrics by setting `-graphiteListenAddr` command line flag. For instance,
@@ -1705,6 +1731,10 @@ Downsampling happens during [background merges](https://docs.victoriametrics.com
 and can't be performed if there is not enough of free disk space or if vmstorage 
 is in [read-only mode](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#readonly-mode).
 
+Please, note that intervals of `-downsampling.period` must be multiples of each other. 
+In case [deduplication](https://docs.victoriametrics.com/#deduplication) is enabled value of `-dedup.minScrapeInterval` must also be multiple of `-downsampling.period` intervals.
+This is required to ensure consistency of deduplication and downsampling results.
+
 The downsampling can be evaluated for free by downloading and using enterprise binaries from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
 
 ## Multi-tenancy
@@ -1944,9 +1974,6 @@ and [cardinality explorer docs](#cardinality-explorer).
   has at least 20% of free space. The remaining amount of free space
   can be [monitored](#monitoring) via `vm_free_disk_space_bytes` metric. The total size of data
   stored on the disk can be monitored via sum of `vm_data_size_bytes` metrics.
-  See also `vm_merge_need_free_disk_space` metrics, which are set to values higher than 0
-  if background merge cannot be initiated due to free disk space shortage. The value shows the number of per-month partitions,
-  which would start background merge if they had more free disk space.
 
 * VictoriaMetrics buffers incoming data in memory for up to a few seconds before flushing it to persistent storage.
   This may lead to the following "issues":
diff --git a/docs/VictoriaLogs/CHANGELOG.md b/docs/VictoriaLogs/CHANGELOG.md
index e4977cc51..f2bb6d29a 100644
--- a/docs/VictoriaLogs/CHANGELOG.md
+++ b/docs/VictoriaLogs/CHANGELOG.md
@@ -9,6 +9,8 @@ according to [these docs](https://docs.victoriametrics.com/VictoriaLogs/QuickSta
 * FEATURE: expose the following metrics at [/metrics](monitoring) page:
   * `vl_data_size_bytes{type="storage"}` - on-disk size for data excluding [log stream](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields) indexes.
   * `vl_data_size_bytes{type="indexdb"}` - on-disk size for [log stream](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields) indexes.
+* FEATURE: add `-insert.maxFieldsPerLine` command-line flag, which can be used for limiting the number of fields per line in logs sent to VictoriaLogs via ingestion protocols. This helps to avoid issues like [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4762).
+* FEATURE: expose `vl_http_request_duration_seconds` histogram at the [/metrics](https://docs.victoriametrics.com/VictoriaLogs/#monitoring) page. Thanks to @crossoverJie for [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4934).
 
 * BUGFIX: fix possible panic when no data is written to VictoriaLogs for a long time. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4895). Thanks to @crossoverJie for filing and fixing the issue.
 * BUGFIX: add `/insert/loky/ready` endpoint, which is used by Promtail for healthchecks. This should remove `unsupported path requested: /insert/loki/ready` warning logs. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4762#issuecomment-1690966722).
diff --git a/docs/VictoriaLogs/README.md b/docs/VictoriaLogs/README.md
index 2fb4d919c..22c722cc1 100644
--- a/docs/VictoriaLogs/README.md
+++ b/docs/VictoriaLogs/README.md
@@ -171,6 +171,8 @@ Pass `-help` to VictoriaLogs in order to see the list of supported command-line
     	Whether to use proxy protocol for connections accepted at -httpListenAddr . See https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt . With enabled proxy protocol http server cannot serve regular /metrics endpoint. Use -pushmetrics.url for metrics pushing
   -inmemoryDataFlushInterval duration
     	The interval for guaranteed saving of in-memory data to disk. The saved data survives unclean shutdowns such as OOM crash, hardware reset, SIGKILL, etc. Bigger intervals may help increase the lifetime of flash storage with limited write cycles (e.g. Raspberry PI). Smaller intervals increase disk IO load. Minimum supported value is 1s (default 5s)
+  -insert.maxFieldsPerLine int
+    	The maximum number of log fields per line, which can be read by /insert/* handlers (default 1000)
   -insert.maxLineSizeBytes size
     	The maximum size of a single line, which can be read by /insert/* handlers
     	Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 262144)
diff --git a/docs/VictoriaLogs/data-ingestion/README.md b/docs/VictoriaLogs/data-ingestion/README.md
index 9876edacb..6d657b2af 100644
--- a/docs/VictoriaLogs/data-ingestion/README.md
+++ b/docs/VictoriaLogs/data-ingestion/README.md
@@ -81,6 +81,8 @@ The response by default contains [`_msg`](https://docs.victoriametrics.com/Victo
 [`_time`](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#time-field) fields plus the explicitly mentioned fields.
 See [these docs](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#querying-specific-fields) for details.
 
+The duration of requests to `/insert/elasticsearch/_bulk` can be monitored with `vl_http_request_duration_seconds{path="/insert/elasticsearch/_bulk"}` metric.
+
 See also:
 
 - [How to debug data ingestion](#troubleshooting).
@@ -133,6 +135,8 @@ The response by default contains [`_msg`](https://docs.victoriametrics.com/Victo
 [`_time`](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#time-field) fields plus the explicitly mentioned fields.
 See [these docs](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#querying-specific-fields) for details.
 
+The duration of requests to `/insert/jsonline` can be monitored with `vl_http_request_duration_seconds{path="/insert/jsonline"}` metric.
+
 See also:
 
 - [How to debug data ingestion](#troubleshooting).
@@ -172,6 +176,8 @@ The response by default contains [`_msg`](https://docs.victoriametrics.com/Victo
 [`_time`](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#time-field) fields plus the explicitly mentioned fields.
 See [these docs](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#querying-specific-fields) for details.
 
+The duration of requests to `/insert/loki/api/v1/push` can be monitored with `vl_http_request_duration_seconds{path="/insert/loki/api/v1/push"}` metric.
+
 See also:
 
 - [How to debug data ingestion](#troubleshooting).
diff --git a/docs/operator/CHANGELOG.md b/docs/operator/CHANGELOG.md
index 6341ddcbc..c9d2022ca 100644
--- a/docs/operator/CHANGELOG.md
+++ b/docs/operator/CHANGELOG.md
@@ -9,6 +9,8 @@
 ### Fixes
 
 - [vmcluster](https://docs.victoriametrics.com/operator/api.html#vmcluster): remove redundant annotation `operator.victoriametrics/last-applied-spec` from created workloads like vmstorage statefulset.
+- [vmoperator](https://docs.victoriametrics.com/operator/): properly resize statefulset's multiple pvc when needed and allowable, before they could be updated with wrong size.
+- [vmoperator](https://docs.victoriametrics.com/operator/): fix wrong api group of endpointsices, before vmagent won't able to access endpointsices resources with default rbac rule.
 
 <a name="v0.38.0"></a>
 ## [v0.38.0](https://github.com/VictoriaMetrics/operator/releases/tag/v0.38.0) - 11 Sep 2023
diff --git a/docs/operator/vars.md b/docs/operator/vars.md
index dad003d18..5142a05d1 100644
--- a/docs/operator/vars.md
+++ b/docs/operator/vars.md
@@ -10,7 +10,7 @@ aliases:
 - /operator/vars.html
 ---
 # Auto Generated vars for package config 
- updated at Wed Sep 13 14:05:24 UTC 2023 
+ updated at Wed Sep 27 00:09:29 UTC 2023 
 
 
 | varible name | variable default value | variable required | variable description |
@@ -20,7 +20,7 @@ aliases:
 | VM_CUSTOMCONFIGRELOADERIMAGE | victoriametrics/operator:config-reloader-v0.32.0 | false | - |
 | VM_PSPAUTOCREATEENABLED | false | false | - |
 | VM_VMALERTDEFAULT_IMAGE | victoriametrics/vmalert | false | - |
-| VM_VMALERTDEFAULT_VERSION | v1.93.4 | false | - |
+| VM_VMALERTDEFAULT_VERSION | v1.93.5 | false | - |
 | VM_VMALERTDEFAULT_PORT | 8080 | false | - |
 | VM_VMALERTDEFAULT_USEDEFAULTRESOURCES | true | false | - |
 | VM_VMALERTDEFAULT_RESOURCE_LIMIT_MEM | 500Mi | false | - |
@@ -31,7 +31,7 @@ aliases:
 | VM_VMALERTDEFAULT_CONFIGRELOADERMEMORY | 25Mi | false | - |
 | VM_VMALERTDEFAULT_CONFIGRELOADIMAGE | jimmidyson/configmap-reload:v0.3.0 | false | - |
 | VM_VMAGENTDEFAULT_IMAGE | victoriametrics/vmagent | false | - |
-| VM_VMAGENTDEFAULT_VERSION | v1.93.4 | false | - |
+| VM_VMAGENTDEFAULT_VERSION | v1.93.5 | false | - |
 | VM_VMAGENTDEFAULT_CONFIGRELOADIMAGE | quay.io/prometheus-operator/prometheus-config-reloader:v0.68.0 | false | - |
 | VM_VMAGENTDEFAULT_PORT | 8429 | false | - |
 | VM_VMAGENTDEFAULT_USEDEFAULTRESOURCES | true | false | - |
@@ -42,7 +42,7 @@ aliases:
 | VM_VMAGENTDEFAULT_CONFIGRELOADERCPU | 100m | false | - |
 | VM_VMAGENTDEFAULT_CONFIGRELOADERMEMORY | 25Mi | false | - |
 | VM_VMSINGLEDEFAULT_IMAGE | victoriametrics/victoria-metrics | false | - |
-| VM_VMSINGLEDEFAULT_VERSION | v1.93.4 | false | - |
+| VM_VMSINGLEDEFAULT_VERSION | v1.93.5 | false | - |
 | VM_VMSINGLEDEFAULT_PORT | 8429 | false | - |
 | VM_VMSINGLEDEFAULT_USEDEFAULTRESOURCES | true | false | - |
 | VM_VMSINGLEDEFAULT_RESOURCE_LIMIT_MEM | 1500Mi | false | - |
@@ -53,14 +53,14 @@ aliases:
 | VM_VMSINGLEDEFAULT_CONFIGRELOADERMEMORY | 25Mi | false | - |
 | VM_VMCLUSTERDEFAULT_USEDEFAULTRESOURCES | true | false | - |
 | VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_IMAGE | victoriametrics/vmselect | false | - |
-| VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_VERSION | v1.93.4-cluster | false | - |
+| VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_VERSION | v1.93.5-cluster | false | - |
 | VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_PORT | 8481 | false | - |
 | VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_RESOURCE_LIMIT_MEM | 1000Mi | false | - |
 | VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_RESOURCE_LIMIT_CPU | 500m | false | - |
 | VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_RESOURCE_REQUEST_MEM | 500Mi | false | - |
 | VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_RESOURCE_REQUEST_CPU | 100m | false | - |
 | VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_IMAGE | victoriametrics/vmstorage | false | - |
-| VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_VERSION | v1.93.4-cluster | false | - |
+| VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_VERSION | v1.93.5-cluster | false | - |
 | VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_VMINSERTPORT | 8400 | false | - |
 | VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_VMSELECTPORT | 8401 | false | - |
 | VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_PORT | 8482 | false | - |
@@ -69,7 +69,7 @@ aliases:
 | VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_RESOURCE_REQUEST_MEM | 500Mi | false | - |
 | VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_RESOURCE_REQUEST_CPU | 250m | false | - |
 | VM_VMCLUSTERDEFAULT_VMINSERTDEFAULT_IMAGE | victoriametrics/vminsert | false | - |
-| VM_VMCLUSTERDEFAULT_VMINSERTDEFAULT_VERSION | v1.93.4-cluster | false | - |
+| VM_VMCLUSTERDEFAULT_VMINSERTDEFAULT_VERSION | v1.93.5-cluster | false | - |
 | VM_VMCLUSTERDEFAULT_VMINSERTDEFAULT_PORT | 8480 | false | - |
 | VM_VMCLUSTERDEFAULT_VMINSERTDEFAULT_RESOURCE_LIMIT_MEM | 500Mi | false | - |
 | VM_VMCLUSTERDEFAULT_VMINSERTDEFAULT_RESOURCE_LIMIT_CPU | 500m | false | - |
@@ -88,7 +88,7 @@ aliases:
 | VM_VMALERTMANAGER_RESOURCE_REQUEST_CPU | 30m | false | - |
 | VM_DISABLESELFSERVICESCRAPECREATION | false | false | - |
 | VM_VMBACKUP_IMAGE | victoriametrics/vmbackupmanager | false | - |
-| VM_VMBACKUP_VERSION | v1.93.4-enterprise | false | - |
+| VM_VMBACKUP_VERSION | v1.93.5-enterprise | false | - |
 | VM_VMBACKUP_PORT | 8300 | false | - |
 | VM_VMBACKUP_USEDEFAULTRESOURCES | true | false | - |
 | VM_VMBACKUP_RESOURCE_LIMIT_MEM | 500Mi | false | - |
@@ -97,7 +97,7 @@ aliases:
 | VM_VMBACKUP_RESOURCE_REQUEST_CPU | 150m | false | - |
 | VM_VMBACKUP_LOGLEVEL | INFO | false | - |
 | VM_VMAUTHDEFAULT_IMAGE | victoriametrics/vmauth | false | - |
-| VM_VMAUTHDEFAULT_VERSION | v1.93.4 | false | - |
+| VM_VMAUTHDEFAULT_VERSION | v1.93.5 | false | - |
 | VM_VMAUTHDEFAULT_CONFIGRELOADIMAGE | quay.io/prometheus-operator/prometheus-config-reloader:v0.68.0 | false | - |
 | VM_VMAUTHDEFAULT_PORT | 8427 | false | - |
 | VM_VMAUTHDEFAULT_USEDEFAULTRESOURCES | true | false | - |
diff --git a/docs/vmagent.md b/docs/vmagent.md
index 0e918cc38..0ab5d9514 100644
--- a/docs/vmagent.md
+++ b/docs/vmagent.md
@@ -14,7 +14,8 @@ aliases:
 `vmagent` is a tiny agent which helps you collect metrics from various sources,
 [relabel and filter the collected metrics](#relabeling)
 and store them in [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics)
-or any other storage systems via Prometheus `remote_write` protocol.
+or any other storage systems via Prometheus `remote_write` protocol
+or via [VictoriaMetrics `remote_write` protocol](#victoriametrics-remote-write-protocol).
 
 See [Quick Start](#quick-start) for details.
 
diff --git a/docs/vmauth.md b/docs/vmauth.md
index d6e631a7f..65e8d929c 100644
--- a/docs/vmauth.md
+++ b/docs/vmauth.md
@@ -36,6 +36,7 @@ The auth config can be reloaded via the following ways:
   and apply new changes every 5 seconds.
 
 Docker images for `vmauth` are available [here](https://hub.docker.com/r/victoriametrics/vmauth/tags).
+See how `vmauth` used in [docker-compose env](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/README.md#victoriametrics-cluster).
 
 Pass `-help` to `vmauth` in order to see all the supported command-line flags with their descriptions.
 
diff --git a/docs/vmbackup.md b/docs/vmbackup.md
index 39c4b3e95..bff38459a 100644
--- a/docs/vmbackup.md
+++ b/docs/vmbackup.md
@@ -154,20 +154,23 @@ See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-
 
 ## Advanced usage
 
-* Obtaining credentials from a file.
 
-  Add flag `-credsFilePath=/etc/credentials` with the following content:
+### Providing credentials as a file
 
-    for s3 (aws, minio or other s3 compatible storages):
+Obtaining credentials from a file.
 
+Add flag `-credsFilePath=/etc/credentials` with the following content:
+
+- for S3 (AWS, MinIO or other S3 compatible storages):
+    
      ```console
      [default]
      aws_access_key_id=theaccesskey
      aws_secret_access_key=thesecretaccesskeyvalue
     ```
 
-    for gce cloud storage:
-
+- for GCP cloud storage:
+    
     ```json
     {
            "type": "service_account",
@@ -182,24 +185,99 @@ See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-
            "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-email"
     }
     ```
-* Obtaining credentials from env variables.
-    - For AWS S3 compatible storages set env variable `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
-      Also you can set env variable `AWS_SHARED_CREDENTIALS_FILE` with path to credentials file.
-    - For GCE cloud storage set env variable `GOOGLE_APPLICATION_CREDENTIALS` with path to credentials file.
-    - For Azure storage either set env variables `AZURE_STORAGE_ACCOUNT_NAME` and `AZURE_STORAGE_ACCOUNT_KEY`, or `AZURE_STORAGE_ACCOUNT_CONNECTION_STRING`.
 
-* Usage with s3 custom url endpoint. It is possible to use `vmbackup` with s3 compatible storages like minio, cloudian, etc.
-  You have to add a custom url endpoint via flag:
+### Providing credentials via env variables 
 
-```console
-  # for minio
-  -customS3Endpoint=http://localhost:9000
+Obtaining credentials from env variables.
+- For AWS S3 compatible storages set env variable `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. 
+  Also you can set env variable `AWS_SHARED_CREDENTIALS_FILE` with path to credentials file.
+- For GCE cloud storage set env variable `GOOGLE_APPLICATION_CREDENTIALS` with path to credentials file.
+- For Azure storage either set env variables `AZURE_STORAGE_ACCOUNT_NAME` and `AZURE_STORAGE_ACCOUNT_KEY`, or `AZURE_STORAGE_ACCOUNT_CONNECTION_STRING`.
+
+Please, note that `vmbackup` will use credentials provided by cloud providers metadata service [when applicable](https://docs.victoriametrics.com/vmbackup.html#using-cloud-providers-metadata-service).
+
+### Using cloud providers metadata service
+
+`vmbackup` and `vmbackupmanager` will automatically use cloud providers metadata service in order to obtain credentials if they are running in cloud environment
+and credentials are not explicitly provided via flags or env variables.
+
+### Providing credentials in Kubernetes
+
+The simplest way to provide credentials in Kubernetes is to use [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/)
+and inject them into the pod as environment variables. For example, the following secret can be used for AWS S3 credentials:
+```yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: vmbackup-credentials
+data:
+  access_key: key
+  secret_key: secret
+```
+And then it can be injected into the pod as environment variables:
+```yaml
+...
+env:
+- name: AWS_ACCESS_KEY_ID
+  valueFrom:
+    secretKeyRef:
+      key: access_key
+      name: vmbackup-credentials
+- name: AWS_SECRET_ACCESS_KEY
+  valueFrom:
+    secretKeyRef:
+      key: secret_key
+      name: vmbackup-credentials
+...
+```
 
-  # for aws gov region
-  -customS3Endpoint=https://s3-fips.us-gov-west-1.amazonaws.com
+A more secure way is to use IAM roles to provide tokens for pods instead of managing credentials manually. 
+
+For AWS deployments it will be required to configure [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html).
+In order to use IAM roles for service accounts with `vmbackup` or `vmbackupmanager` it is required to create ServiceAccount with IAM role mapping:
+```yaml
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: monitoring-backups
+  annotations:
+    eks.amazonaws.com/role-arn: arn:aws:iam::{ACCOUNT_ID}:role/{ROLE_NAME}
 ```
+And [configure pod to use service account](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/).
+After this `vmbackup` and `vmbackupmanager` will automatically use IAM role for service account in order to obtain credentials.
+
+For GCP deployments it will be required to configure [Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity).
+In order to use Workload Identity with `vmbackup` or `vmbackupmanager` it is required to create ServiceAccount with Workload Identity annotation:
+```yaml
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: monitoring-backups
+  annotations:
+    iam.gke.io/gcp-service-account: {sa_name}@{project_name}.iam.gserviceaccount.com
+```
+And [configure pod to use service account](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/).
+After this `vmbackup` and `vmbackupmanager` will automatically use Workload Identity for servicpe account in order to obtain credentials.
+
+### Using custom S3 endpoint
+
+Usage with s3 custom url endpoint. It is possible to use `vmbackup` with s3 compatible storages like minio, cloudian, etc.
+You have to add a custom url endpoint via flag:
+
+- for MinIO
+    ```console
+      -customS3Endpoint=http://localhost:9000
+    ```
+
+- for aws gov region
+    ```console
+      -customS3Endpoint=https://s3-fips.us-gov-west-1.amazonaws.com
+    ```
+
+### Command-line flags
 
-* Run `vmbackup -help` in order to see all the available options:
+Run `vmbackup -help` in order to see all the available options:
 
 ```console
   -concurrency int
diff --git a/docs/vmbackupmanager.md b/docs/vmbackupmanager.md
index 7f912ff9e..051c87656 100644
--- a/docs/vmbackupmanager.md
+++ b/docs/vmbackupmanager.md
@@ -121,6 +121,9 @@ The result on the GCS bucket
 
   <img alt="latest folder" src="vmbackupmanager_latest_folder.png">
 
+Please, see [vmbackup docs](https://docs.victoriametrics.com/vmbackup.html#advanced-usage) for more examples of authentication with different
+storage types.
+
 ## Backup Retention Policy
 
 Backup retention policy is controlled by:
diff --git a/go.mod b/go.mod
index 603c4a567..6d1a47169 100644
--- a/go.mod
+++ b/go.mod
@@ -12,7 +12,7 @@ require (
 	// like https://github.com/valyala/fasthttp/commit/996610f021ff45fdc98c2ce7884d5fa4e7f9199b
 	github.com/VictoriaMetrics/fasthttp v1.2.0
 	github.com/VictoriaMetrics/metrics v1.24.0
-	github.com/VictoriaMetrics/metricsql v0.64.0
+	github.com/VictoriaMetrics/metricsql v0.66.0
 	github.com/aws/aws-sdk-go-v2 v1.21.0
 	github.com/aws/aws-sdk-go-v2/config v1.18.39
 	github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.83
diff --git a/go.sum b/go.sum
index 0b5db09a0..aebec4c96 100644
--- a/go.sum
+++ b/go.sum
@@ -70,8 +70,8 @@ github.com/VictoriaMetrics/fasthttp v1.2.0 h1:nd9Wng4DlNtaI27WlYh5mGXCJOmee/2c2b
 github.com/VictoriaMetrics/fasthttp v1.2.0/go.mod h1:zv5YSmasAoSyv8sBVexfArzFDIGGTN4TfCKAtAw7IfE=
 github.com/VictoriaMetrics/metrics v1.24.0 h1:ILavebReOjYctAGY5QU2F9X0MYvkcrG3aEn2RKa1Zkw=
 github.com/VictoriaMetrics/metrics v1.24.0/go.mod h1:eFT25kvsTidQFHb6U0oa0rTrDRdz4xTYjpL8+UPohys=
-github.com/VictoriaMetrics/metricsql v0.64.0 h1:uty6AXQFY3OpQ+eopo1jDjCcTctuqkqYLnRbQVhukW8=
-github.com/VictoriaMetrics/metricsql v0.64.0/go.mod h1:k4UaP/+CjuZslIjd+kCigNG9TQmUqh5v0TP/nMEy90I=
+github.com/VictoriaMetrics/metricsql v0.66.0 h1:2TaBEM7L5L67Ho65FdJVZ/qvjWmC/+f17nujL6dgtmE=
+github.com/VictoriaMetrics/metricsql v0.66.0/go.mod h1:k4UaP/+CjuZslIjd+kCigNG9TQmUqh5v0TP/nMEy90I=
 github.com/VividCortex/ewma v1.2.0 h1:f58SaIzcDXrSy3kWaHNvuJgJ3Nmz59Zji6XoJR/q1ow=
 github.com/VividCortex/ewma v1.2.0/go.mod h1:nz4BbCtbLyFDeC9SUHbtcT5644juEuWfUAUnGx7j5l4=
 github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
diff --git a/lib/backup/common/part.go b/lib/backup/common/part.go
index 1b58b66b1..8b9c68e31 100644
--- a/lib/backup/common/part.go
+++ b/lib/backup/common/part.go
@@ -2,10 +2,12 @@ package common
 
 import (
 	"fmt"
+	"path/filepath"
 	"regexp"
 	"sort"
 	"strconv"
 	"strings"
+	"sync/atomic"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 )
@@ -15,6 +17,9 @@ import (
 // Each source file can be split into parts with up to MaxPartSize sizes.
 type Part struct {
 	// Path is the path to file for backup.
+	//
+	// Path must consistently use `/` as directory separator.
+	// Use ToCanonicalPath() function for converting local directory separators to `/`.
 	Path string
 
 	// FileSize is the size of the whole file for the given part.
@@ -33,11 +38,23 @@ type Part struct {
 	ActualSize uint64
 }
 
+// key returns a string, which uniquely identifies p.
 func (p *Part) key() string {
+	if strings.HasSuffix(p.Path, "/parts.json") ||
+		strings.HasSuffix(p.Path, "/appliedRetention.txt") {
+		// parts.json and appliedRetention.txt files contents changes over time,
+		// so it must have an unique key in order to always copy it during
+		// backup, restore and server-side copy.
+		// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5005
+		id := atomic.AddUint64(&uniqueKeyID, 1)
+		return fmt.Sprintf("unique-%016X", id)
+	}
 	// Do not use p.FileSize in the key, since it cannot be properly initialized when resuming the restore for partially restored file
 	return fmt.Sprintf("%s%016X%016X%016X", p.Path, p.Offset, p.Size, p.ActualSize)
 }
 
+var uniqueKeyID uint64
+
 // String returns human-readable representation of the part.
 func (p *Part) String() string {
 	return fmt.Sprintf("part{path: %q, file_size: %d, offset: %d, size: %d}", p.Path, p.FileSize, p.Offset, p.Size)
@@ -51,11 +68,30 @@ func (p *Part) RemotePath(prefix string) string {
 	return fmt.Sprintf("%s/%s/%016X_%016X_%016X", prefix, p.Path, p.FileSize, p.Offset, p.Size)
 }
 
+// LocalPath returns local path for p at the given dir.
+func (p *Part) LocalPath(dir string) string {
+	path := p.Path
+	if filepath.Separator != '/' {
+		path = strings.ReplaceAll(path, "/", string(filepath.Separator))
+	}
+	return filepath.Join(dir, path)
+}
+
+// ToCanonicalPath returns canonical path by replacing local directory separators with `/`.
+func ToCanonicalPath(path string) string {
+	if filepath.Separator == '/' {
+		return path
+	}
+	return strings.ReplaceAll(path, string(filepath.Separator), "/")
+}
+
 var partNameRegexp = regexp.MustCompile(`^(.+)[/\\]([0-9A-F]{16})_([0-9A-F]{16})_([0-9A-F]{16})$`)
 
 // ParseFromRemotePath parses p from remotePath.
 //
 // Returns true on success.
+//
+// remotePath must be in canonical form received from ToCanonicalPath().
 func (p *Part) ParseFromRemotePath(remotePath string) bool {
 	tmp := partNameRegexp.FindStringSubmatch(remotePath)
 	if len(tmp) != 5 {
diff --git a/lib/backup/fscommon/fscommon.go b/lib/backup/fscommon/fscommon.go
index 0ba3e856a..3d52bbcc9 100644
--- a/lib/backup/fscommon/fscommon.go
+++ b/lib/backup/fscommon/fscommon.go
@@ -10,9 +10,10 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 )
 
-// AppendFiles appends all the files from dir to dst.
+// AppendFiles appends paths to all the files from local dir to dst.
 //
-// All the appended files will have dir prefix.
+// All the appended filepaths will have dir prefix.
+// The returned paths have local OS-specific directory separators.
 func AppendFiles(dst []string, dir string) ([]string, error) {
 	d, err := os.Open(dir)
 	if err != nil {
diff --git a/lib/backup/fslocal/fslocal.go b/lib/backup/fslocal/fslocal.go
index 1136cee74..d9d853da4 100644
--- a/lib/backup/fslocal/fslocal.go
+++ b/lib/backup/fslocal/fslocal.go
@@ -77,7 +77,7 @@ func (fs *FS) ListParts() ([]common.Part, error) {
 		if err != nil {
 			return nil, fmt.Errorf("cannot stat %q: %w", file, err)
 		}
-		path := file[len(dir):]
+		path := common.ToCanonicalPath(file[len(dir):])
 		size := uint64(fi.Size())
 		if size == 0 {
 			parts = append(parts, common.Part{
@@ -146,8 +146,9 @@ func (fs *FS) NewWriteCloser(p common.Part) (io.WriteCloser, error) {
 	return blwc, nil
 }
 
-// DeletePath deletes the given path from fs and returns the size
-// for the deleted file.
+// DeletePath deletes the given path from fs and returns the size for the deleted file.
+//
+// The path must be in canonical form, e.g. it must have `/` directory separators
 func (fs *FS) DeletePath(path string) (uint64, error) {
 	p := common.Part{
 		Path: path,
@@ -187,7 +188,7 @@ func (fs *FS) mkdirAll(filePath string) error {
 }
 
 func (fs *FS) path(p common.Part) string {
-	return filepath.Join(fs.Dir, p.Path)
+	return p.LocalPath(fs.Dir)
 }
 
 type limitedReadCloser struct {
diff --git a/lib/backup/fsremote/fsremote.go b/lib/backup/fsremote/fsremote.go
index cd2d81014..f52442adb 100644
--- a/lib/backup/fsremote/fsremote.go
+++ b/lib/backup/fsremote/fsremote.go
@@ -58,7 +58,8 @@ func (fs *FS) ListParts() ([]common.Part, error) {
 			continue
 		}
 		var p common.Part
-		if !p.ParseFromRemotePath(file[len(dir):]) {
+		remotePath := common.ToCanonicalPath(file[len(dir):])
+		if !p.ParseFromRemotePath(remotePath) {
 			logger.Infof("skipping unknown file %s", file)
 			continue
 		}
@@ -68,7 +69,6 @@ func (fs *FS) ListParts() ([]common.Part, error) {
 			return nil, fmt.Errorf("cannot stat file %q for part %q: %w", file, p.Path, err)
 		}
 		p.ActualSize = uint64(fi.Size())
-		p.Path = pathToCanonical(p.Path)
 		parts = append(parts, p)
 	}
 	return parts, nil
@@ -76,7 +76,6 @@ func (fs *FS) ListParts() ([]common.Part, error) {
 
 // DeletePart deletes the given part p from fs.
 func (fs *FS) DeletePart(p common.Part) error {
-	p.Path = canonicalPathToLocal(p.Path)
 	path := fs.path(p)
 	if err := os.Remove(path); err != nil {
 		return fmt.Errorf("cannot remove %q: %w", path, err)
@@ -97,7 +96,6 @@ func (fs *FS) CopyPart(srcFS common.OriginFS, p common.Part) error {
 	if !ok {
 		return fmt.Errorf("cannot perform server-side copying from %s to %s: both of them must be fsremote", srcFS, fs)
 	}
-	p.Path = canonicalPathToLocal(p.Path)
 	srcPath := src.path(p)
 	dstPath := fs.path(p)
 	if err := fs.mkdirAll(dstPath); err != nil {
@@ -142,7 +140,6 @@ func (fs *FS) CopyPart(srcFS common.OriginFS, p common.Part) error {
 
 // DownloadPart download part p from fs to w.
 func (fs *FS) DownloadPart(p common.Part, w io.Writer) error {
-	p.Path = canonicalPathToLocal(p.Path)
 	path := fs.path(p)
 	r, err := os.Open(path)
 	if err != nil {
@@ -198,14 +195,13 @@ func (fs *FS) mkdirAll(filePath string) error {
 }
 
 func (fs *FS) path(p common.Part) string {
-	return filepath.Join(fs.Dir, p.Path, fmt.Sprintf("%016X_%016X_%016X", p.FileSize, p.Offset, p.Size))
+	return filepath.Join(p.LocalPath(fs.Dir), fmt.Sprintf("%016X_%016X_%016X", p.FileSize, p.Offset, p.Size))
 }
 
 // DeleteFile deletes filePath at fs.
 //
 // The function does nothing if the filePath doesn't exist.
 func (fs *FS) DeleteFile(filePath string) error {
-	filePath = canonicalPathToLocal(filePath)
 	path := filepath.Join(fs.Dir, filePath)
 	err := os.Remove(path)
 	if err != nil && !os.IsNotExist(err) {
@@ -247,6 +243,5 @@ func (fs *FS) HasFile(filePath string) (bool, error) {
 // ReadFile returns the content of filePath at fs.
 func (fs *FS) ReadFile(filePath string) ([]byte, error) {
 	path := filepath.Join(fs.Dir, filePath)
-
 	return os.ReadFile(path)
 }
diff --git a/lib/backup/fsremote/normilize_path_bsd.go b/lib/backup/fsremote/normilize_path_bsd.go
deleted file mode 100644
index 4fb42fee9..000000000
--- a/lib/backup/fsremote/normilize_path_bsd.go
+++ /dev/null
@@ -1,12 +0,0 @@
-//go:build freebsd || openbsd || dragonfly || netbsd
-// +build freebsd openbsd dragonfly netbsd
-
-package fsremote
-
-func pathToCanonical(path string) string {
-	return path
-}
-
-func canonicalPathToLocal(path string) string {
-	return path
-}
diff --git a/lib/backup/fsremote/normilize_path_darwin.go b/lib/backup/fsremote/normilize_path_darwin.go
deleted file mode 100644
index 5e9590684..000000000
--- a/lib/backup/fsremote/normilize_path_darwin.go
+++ /dev/null
@@ -1,9 +0,0 @@
-package fsremote
-
-func pathToCanonical(path string) string {
-	return path
-}
-
-func canonicalPathToLocal(path string) string {
-	return path
-}
diff --git a/lib/backup/fsremote/normilize_path_linux.go b/lib/backup/fsremote/normilize_path_linux.go
deleted file mode 100644
index 5e9590684..000000000
--- a/lib/backup/fsremote/normilize_path_linux.go
+++ /dev/null
@@ -1,9 +0,0 @@
-package fsremote
-
-func pathToCanonical(path string) string {
-	return path
-}
-
-func canonicalPathToLocal(path string) string {
-	return path
-}
diff --git a/lib/backup/fsremote/normilize_path_solaris.go b/lib/backup/fsremote/normilize_path_solaris.go
deleted file mode 100644
index 5e9590684..000000000
--- a/lib/backup/fsremote/normilize_path_solaris.go
+++ /dev/null
@@ -1,9 +0,0 @@
-package fsremote
-
-func pathToCanonical(path string) string {
-	return path
-}
-
-func canonicalPathToLocal(path string) string {
-	return path
-}
diff --git a/lib/backup/fsremote/normilize_path_windows.go b/lib/backup/fsremote/normilize_path_windows.go
deleted file mode 100644
index c1dfe71af..000000000
--- a/lib/backup/fsremote/normilize_path_windows.go
+++ /dev/null
@@ -1,11 +0,0 @@
-package fsremote
-
-import "strings"
-
-func pathToCanonical(path string) string {
-	return strings.ReplaceAll(path, "\\", "/")
-}
-
-func canonicalPathToLocal(path string) string {
-	return strings.ReplaceAll(path, "/", "\\")
-}
diff --git a/lib/logstorage/datadb.go b/lib/logstorage/datadb.go
index d23c05eed..41f9c31a6 100644
--- a/lib/logstorage/datadb.go
+++ b/lib/logstorage/datadb.go
@@ -149,7 +149,18 @@ func mustOpenDatadb(pt *partition, path string, flushInterval time.Duration) *da
 
 	pws := make([]*partWrapper, len(partNames))
 	for i, partName := range partNames {
+		// Make sure the partName exists on disk.
+		// If it is missing, then manual action from the user is needed,
+		// since this is unexpected state, which cannot occur under normal operation,
+		// including unclean shutdown.
 		partPath := filepath.Join(path, partName)
+		if !fs.IsPathExist(partPath) {
+			partsFile := filepath.Join(path, partsFilename)
+			logger.Panicf("FATAL: part %q is listed in %q, but is missing on disk; "+
+				"ensure %q contents is not corrupted; remove %q to rebuild its' content from the list of existing parts",
+				partPath, partsFile, partsFile, partsFile)
+		}
+
 		p := mustOpenFilePart(pt, partPath)
 		pws[i] = newPartWrapper(p, nil, time.Time{})
 	}
diff --git a/lib/mergeset/table.go b/lib/mergeset/table.go
index 8b916770e..bf7274a8e 100644
--- a/lib/mergeset/table.go
+++ b/lib/mergeset/table.go
@@ -1349,6 +1349,18 @@ func mustOpenParts(path string) []*partWrapper {
 	des := fs.MustReadDir(path)
 	m := make(map[string]struct{}, len(partNames))
 	for _, partName := range partNames {
+		// Make sure the partName exists on disk.
+		// If it is missing, then manual action from the user is needed,
+		// since this is unexpected state, which cannot occur under normal operation,
+		// including unclean shutdown.
+		partPath := filepath.Join(path, partName)
+		if !fs.IsPathExist(partPath) {
+			partsFile := filepath.Join(path, partsFilename)
+			logger.Panicf("FATAL: part %q is listed in %q, but is missing on disk; "+
+				"ensure %q contents is not corrupted; remove %q to rebuild its' content from the list of existing parts",
+				partPath, partsFile, partsFile, partsFile)
+		}
+
 		m[partName] = struct{}{}
 	}
 	for _, de := range des {
diff --git a/lib/promscrape/discovery/kubernetes/api_watcher.go b/lib/promscrape/discovery/kubernetes/api_watcher.go
index 78bb7ddd7..b677bd241 100644
--- a/lib/promscrape/discovery/kubernetes/api_watcher.go
+++ b/lib/promscrape/discovery/kubernetes/api_watcher.go
@@ -1,6 +1,7 @@
 package kubernetes
 
 import (
+	"context"
 	"encoding/json"
 	"errors"
 	"flag"
@@ -16,11 +17,13 @@ import (
 	"sync/atomic"
 	"time"
 
+	"github.com/VictoriaMetrics/metrics"
+
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/cgroup"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promauth"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promutils"
-	"github.com/VictoriaMetrics/metrics"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/timerpool"
 )
 
 var apiServerTimeout = flag.Duration("promscrape.kubernetes.apiServerTimeout", 30*time.Minute, "How frequently to reload the full state from Kubernetes API server")
@@ -269,7 +272,11 @@ func selectorsKey(selectors []Selector) string {
 
 var (
 	groupWatchersLock sync.Mutex
-	groupWatchers     = make(map[string]*groupWatcher)
+	groupWatchers     = func() map[string]*groupWatcher {
+		gws := make(map[string]*groupWatcher)
+		go groupWatchersCleaner(gws)
+		return gws
+	}()
 
 	_ = metrics.NewGauge(`vm_promscrape_discovery_kubernetes_group_watchers`, func() float64 {
 		groupWatchersLock.Lock()
@@ -279,6 +286,21 @@ var (
 	})
 )
 
+func groupWatchersCleaner(gws map[string]*groupWatcher) {
+	for {
+		time.Sleep(7 * time.Second)
+		groupWatchersLock.Lock()
+		for key, gw := range gws {
+			gw.mu.Lock()
+			if len(gw.m) == 0 {
+				delete(gws, key)
+			}
+			gw.mu.Unlock()
+		}
+		groupWatchersLock.Unlock()
+	}
+}
+
 type swosByKeyWithLock struct {
 	mu        sync.Mutex
 	swosByKey map[string][]interface{}
@@ -378,31 +400,14 @@ func (gw *groupWatcher) startWatchersForRole(role string, aw *apiWatcher) {
 				// This should guarantee that the ScrapeWork objects for these objects are properly updated
 				// as soon as the objects they depend on are updated.
 				// This should fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240 .
-				go func() {
-					const minSleepTime = 5 * time.Second
-					sleepTime := minSleepTime
-					for {
-						time.Sleep(sleepTime)
-						startTime := time.Now()
-						gw.mu.Lock()
-						if uw.needRecreateScrapeWorks {
-							uw.needRecreateScrapeWorks = false
-							uw.recreateScrapeWorksLocked(uw.objectsByKey, uw.aws)
-							sleepTime = time.Since(startTime)
-							if sleepTime < minSleepTime {
-								sleepTime = minSleepTime
-							}
-						}
-						gw.mu.Unlock()
-					}
-				}()
+				go uw.recreateScrapeWorks()
 			}
 		}
 	}
 }
 
 // doRequest performs http request to the given requestURL.
-func (gw *groupWatcher) doRequest(requestURL string) (*http.Response, error) {
+func (gw *groupWatcher) doRequest(ctx context.Context, requestURL string) (*http.Response, error) {
 	if strings.Contains(requestURL, "/apis/networking.k8s.io/v1/") && atomic.LoadUint32(&gw.useNetworkingV1Beta1) == 1 {
 		// Update networking URL for old Kubernetes API, which supports only v1beta1 path.
 		requestURL = strings.Replace(requestURL, "/apis/networking.k8s.io/v1/", "/apis/networking.k8s.io/v1beta1/", 1)
@@ -411,7 +416,7 @@ func (gw *groupWatcher) doRequest(requestURL string) (*http.Response, error) {
 		// Update discovery URL for old Kubernetes API, which supports only v1beta1 path.
 		requestURL = strings.Replace(requestURL, "/apis/discovery.k8s.io/v1/", "/apis/discovery.k8s.io/v1beta1/", 1)
 	}
-	req, err := http.NewRequest(http.MethodGet, requestURL, nil)
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, requestURL, nil)
 	if err != nil {
 		logger.Fatalf("cannot create a request for %q: %s", requestURL, err)
 	}
@@ -423,11 +428,11 @@ func (gw *groupWatcher) doRequest(requestURL string) (*http.Response, error) {
 	if resp.StatusCode == http.StatusNotFound {
 		if strings.Contains(requestURL, "/apis/networking.k8s.io/v1/") && atomic.LoadUint32(&gw.useNetworkingV1Beta1) == 0 {
 			atomic.StoreUint32(&gw.useNetworkingV1Beta1, 1)
-			return gw.doRequest(requestURL)
+			return gw.doRequest(ctx, requestURL)
 		}
 		if strings.Contains(requestURL, "/apis/discovery.k8s.io/v1/") && atomic.LoadUint32(&gw.useDiscoveryV1Beta1) == 0 {
 			atomic.StoreUint32(&gw.useDiscoveryV1Beta1, 1)
-			return gw.doRequest(requestURL)
+			return gw.doRequest(ctx, requestURL)
 		}
 	}
 	return resp, nil
@@ -446,6 +451,9 @@ func (gw *groupWatcher) unsubscribeAPIWatcher(aw *apiWatcher) {
 	defer gw.mu.Unlock()
 	for _, uw := range gw.m {
 		uw.unsubscribeAPIWatcherLocked(aw)
+		if len(uw.aws)+len(uw.awsPending) == 0 {
+			time.AfterFunc(10*time.Second, uw.stopIfNoUsers)
+		}
 	}
 }
 
@@ -458,6 +466,9 @@ type urlWatcher struct {
 	apiURL    string
 	gw        *groupWatcher
 
+	ctx    context.Context
+	cancel context.CancelFunc
+
 	parseObject     parseObjectFunc
 	parseObjectList parseObjectListFunc
 
@@ -488,11 +499,16 @@ type urlWatcher struct {
 func newURLWatcher(role, apiURL string, gw *groupWatcher) *urlWatcher {
 	parseObject, parseObjectList := getObjectParsersForRole(role)
 	metrics.GetOrCreateCounter(fmt.Sprintf(`vm_promscrape_discovery_kubernetes_url_watchers{role=%q}`, role)).Inc()
+
+	ctx, cancel := context.WithCancel(context.Background())
 	uw := &urlWatcher{
 		role:   role,
 		apiURL: apiURL,
 		gw:     gw,
 
+		ctx:    ctx,
+		cancel: cancel,
+
 		parseObject:     parseObject,
 		parseObjectList: parseObjectList,
 
@@ -510,6 +526,44 @@ func newURLWatcher(role, apiURL string, gw *groupWatcher) *urlWatcher {
 	return uw
 }
 
+func (uw *urlWatcher) stopIfNoUsers() {
+	gw := uw.gw
+	gw.mu.Lock()
+	if len(uw.aws)+len(uw.awsPending) == 0 {
+		uw.cancel()
+		delete(gw.m, uw.apiURL)
+	}
+	gw.mu.Unlock()
+}
+
+func (uw *urlWatcher) recreateScrapeWorks() {
+	const minSleepTime = 5 * time.Second
+	sleepTime := minSleepTime
+	gw := uw.gw
+	stopCh := uw.ctx.Done()
+	for {
+		t := timerpool.Get(sleepTime)
+		select {
+		case <-stopCh:
+			timerpool.Put(t)
+			return
+		case <-t.C:
+			timerpool.Put(t)
+		}
+		startTime := time.Now()
+		gw.mu.Lock()
+		if uw.needRecreateScrapeWorks {
+			uw.needRecreateScrapeWorks = false
+			uw.recreateScrapeWorksLocked(uw.objectsByKey, uw.aws)
+			sleepTime = time.Since(startTime)
+			if sleepTime < minSleepTime {
+				sleepTime = minSleepTime
+			}
+		}
+		gw.mu.Unlock()
+	}
+}
+
 func (uw *urlWatcher) subscribeAPIWatcherLocked(aw *apiWatcher) {
 	if _, ok := uw.aws[aw]; !ok {
 		if _, ok := uw.awsPending[aw]; !ok {
@@ -587,9 +641,11 @@ func (uw *urlWatcher) reloadObjects() string {
 	// and https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4855 .
 	delimiter := getQueryArgsDelimiter(apiURL)
 	requestURL := apiURL + delimiter + "resourceVersion=0&resourceVersionMatch=NotOlderThan"
-	resp, err := uw.gw.doRequest(requestURL)
+	resp, err := uw.gw.doRequest(uw.ctx, requestURL)
 	if err != nil {
-		logger.Errorf("cannot perform request to %q: %s", requestURL, err)
+		if !errors.Is(err, context.Canceled) {
+			logger.Errorf("cannot perform request to %q: %s", requestURL, err)
+		}
 		return ""
 	}
 	if resp.StatusCode != http.StatusOK {
@@ -653,10 +709,18 @@ func (uw *urlWatcher) reloadObjects() string {
 //
 // See https://kubernetes.io/docs/reference/using-api/api-concepts/#efficient-detection-of-changes
 func (uw *urlWatcher) watchForUpdates() {
+	stopCh := uw.ctx.Done()
 	backoffDelay := time.Second
 	maxBackoffDelay := 30 * time.Second
 	backoffSleep := func() {
-		time.Sleep(backoffDelay)
+		t := timerpool.Get(backoffDelay)
+		select {
+		case <-stopCh:
+			timerpool.Put(t)
+			return
+		case <-t.C:
+			timerpool.Put(t)
+		}
 		backoffDelay *= 2
 		if backoffDelay > maxBackoffDelay {
 			backoffDelay = maxBackoffDelay
@@ -667,16 +731,26 @@ func (uw *urlWatcher) watchForUpdates() {
 	timeoutSeconds := time.Duration(0.9 * float64(uw.gw.client.Timeout)).Seconds()
 	apiURL += delimiter + "watch=1&allowWatchBookmarks=true&timeoutSeconds=" + strconv.Itoa(int(timeoutSeconds))
 	for {
+		select {
+		case <-stopCh:
+			metrics.GetOrCreateCounter(fmt.Sprintf(`vm_promscrape_discovery_kubernetes_url_watchers{role=%q}`, uw.role)).Dec()
+			logger.Infof("stopped %s watcher for %q", uw.role, uw.apiURL)
+			return
+		default:
+		}
+
 		resourceVersion := uw.reloadObjects()
 		if resourceVersion == "" {
 			backoffSleep()
 			continue
 		}
 		requestURL := apiURL + "&resourceVersion=" + url.QueryEscape(resourceVersion)
-		resp, err := uw.gw.doRequest(requestURL)
+		resp, err := uw.gw.doRequest(uw.ctx, requestURL)
 		if err != nil {
-			logger.Errorf("cannot perform request to %q: %s", requestURL, err)
-			backoffSleep()
+			if !errors.Is(err, context.Canceled) {
+				logger.Errorf("cannot perform request to %q: %s", requestURL, err)
+				backoffSleep()
+			}
 			continue
 		}
 		if resp.StatusCode != http.StatusOK {
@@ -697,7 +771,7 @@ func (uw *urlWatcher) watchForUpdates() {
 		err = uw.readObjectUpdateStream(resp.Body)
 		_ = resp.Body.Close()
 		if err != nil {
-			if !errors.Is(err, io.EOF) {
+			if !(errors.Is(err, io.EOF) || errors.Is(err, context.Canceled)) {
 				logger.Errorf("error when reading WatchEvent stream from %q: %s", requestURL, err)
 				uw.resourceVersion = ""
 			}
diff --git a/lib/storage/index_db.go b/lib/storage/index_db.go
index 7c99809c6..62777d7fa 100644
--- a/lib/storage/index_db.go
+++ b/lib/storage/index_db.go
@@ -668,7 +668,8 @@ func (is *indexSearch) searchLabelNamesWithFiltersOnDate(qt *querytracer.Tracer,
 		// This would help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2978
 		metricIDs := filter.AppendTo(nil)
 		qt.Printf("sort %d metricIDs", len(metricIDs))
-		return is.getLabelNamesForMetricIDs(qt, metricIDs, lns, maxLabelNames)
+		is.getLabelNamesForMetricIDs(qt, metricIDs, lns, maxLabelNames)
+		return nil
 	}
 	var prevLabelName []byte
 	ts := &is.ts
@@ -732,39 +733,34 @@ func (is *indexSearch) searchLabelNamesWithFiltersOnDate(qt *querytracer.Tracer,
 	return nil
 }
 
-func (is *indexSearch) getLabelNamesForMetricIDs(qt *querytracer.Tracer, metricIDs []uint64, lns map[string]struct{}, maxLabelNames int) error {
+func (is *indexSearch) getLabelNamesForMetricIDs(qt *querytracer.Tracer, metricIDs []uint64, lns map[string]struct{}, maxLabelNames int) {
 	lns["__name__"] = struct{}{}
 	var mn MetricName
 	foundLabelNames := 0
 	var buf []byte
 	for _, metricID := range metricIDs {
-		var err error
-		buf, err = is.searchMetricNameWithCache(buf[:0], metricID)
-		if err != nil {
-			if err == io.EOF {
-				// It is likely the metricID->metricName entry didn't propagate to inverted index yet.
-				// Skip this metricID for now.
-				continue
-			}
-			return fmt.Errorf("cannot find metricName by metricID %d: %w", metricID, err)
+		var ok bool
+		buf, ok = is.searchMetricNameWithCache(buf[:0], metricID)
+		if !ok {
+			// It is likely the metricID->metricName entry didn't propagate to inverted index yet.
+			// Skip this metricID for now.
+			continue
 		}
 		if err := mn.Unmarshal(buf); err != nil {
-			return fmt.Errorf("cannot unmarshal metricName %q: %w", buf, err)
+			logger.Panicf("FATAL: cannot unmarshal metricName %q: %w", buf, err)
 		}
 		for _, tag := range mn.Tags {
-			_, ok := lns[string(tag.Key)]
-			if !ok {
+			if _, ok := lns[string(tag.Key)]; !ok {
 				foundLabelNames++
 				lns[string(tag.Key)] = struct{}{}
 				if len(lns) >= maxLabelNames {
 					qt.Printf("hit the limit on the number of unique label names: %d", maxLabelNames)
-					return nil
+					return
 				}
 			}
 		}
 	}
 	qt.Printf("get %d distinct label names from %d metricIDs", foundLabelNames, len(metricIDs))
-	return nil
 }
 
 // SearchLabelValuesWithFiltersOnTimeRange returns label values for the given labelName, tfss and tr.
@@ -868,7 +864,8 @@ func (is *indexSearch) searchLabelValuesWithFiltersOnDate(qt *querytracer.Tracer
 		// This would help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2978
 		metricIDs := filter.AppendTo(nil)
 		qt.Printf("sort %d metricIDs", len(metricIDs))
-		return is.getLabelValuesForMetricIDs(qt, lvs, labelName, metricIDs, maxLabelValues)
+		is.getLabelValuesForMetricIDs(qt, lvs, labelName, metricIDs, maxLabelValues)
+		return nil
 	}
 	if labelName == "__name__" {
 		// __name__ label is encoded as empty string in indexdb.
@@ -927,7 +924,7 @@ func (is *indexSearch) searchLabelValuesWithFiltersOnDate(qt *querytracer.Tracer
 	return nil
 }
 
-func (is *indexSearch) getLabelValuesForMetricIDs(qt *querytracer.Tracer, lvs map[string]struct{}, labelName string, metricIDs []uint64, maxLabelValues int) error {
+func (is *indexSearch) getLabelValuesForMetricIDs(qt *querytracer.Tracer, lvs map[string]struct{}, labelName string, metricIDs []uint64, maxLabelValues int) {
 	if labelName == "" {
 		labelName = "__name__"
 	}
@@ -935,32 +932,27 @@ func (is *indexSearch) getLabelValuesForMetricIDs(qt *querytracer.Tracer, lvs ma
 	foundLabelValues := 0
 	var buf []byte
 	for _, metricID := range metricIDs {
-		var err error
-		buf, err = is.searchMetricNameWithCache(buf[:0], metricID)
-		if err != nil {
-			if err == io.EOF {
-				// It is likely the metricID->metricName entry didn't propagate to inverted index yet.
-				// Skip this metricID for now.
-				continue
-			}
-			return fmt.Errorf("cannot find metricName by metricID %d: %w", metricID, err)
+		var ok bool
+		buf, ok = is.searchMetricNameWithCache(buf[:0], metricID)
+		if !ok {
+			// It is likely the metricID->metricName entry didn't propagate to inverted index yet.
+			// Skip this metricID for now.
+			continue
 		}
 		if err := mn.Unmarshal(buf); err != nil {
-			return fmt.Errorf("cannot unmarshal metricName %q: %w", buf, err)
+			logger.Panicf("FATAL: cannot unmarshal metricName %q: %s", buf, err)
 		}
 		tagValue := mn.GetTagValue(labelName)
-		_, ok := lvs[string(tagValue)]
-		if !ok {
+		if _, ok := lvs[string(tagValue)]; !ok {
 			foundLabelValues++
 			lvs[string(tagValue)] = struct{}{}
 			if len(lvs) >= maxLabelValues {
 				qt.Printf("hit the limit on the number of unique label values for label %q: %d", labelName, maxLabelValues)
-				return nil
+				return
 			}
 		}
 	}
 	qt.Printf("get %d distinct values for label %q from %d metricIDs", foundLabelValues, labelName, len(metricIDs))
-	return nil
 }
 
 // SearchTagValueSuffixes returns all the tag value suffixes for the given tagKey and tagValuePrefix on the given tr.
@@ -1442,38 +1434,35 @@ func (th *topHeap) Pop() interface{} {
 
 // searchMetricNameWithCache appends metric name for the given metricID to dst
 // and returns the result.
-func (db *indexDB) searchMetricNameWithCache(dst []byte, metricID uint64) ([]byte, error) {
+func (db *indexDB) searchMetricNameWithCache(dst []byte, metricID uint64) ([]byte, bool) {
 	metricName := db.getMetricNameFromCache(dst, metricID)
 	if len(metricName) > len(dst) {
-		return metricName, nil
+		return metricName, true
 	}
 
 	is := db.getIndexSearch(noDeadline)
-	var err error
-	dst, err = is.searchMetricName(dst, metricID)
+	var ok bool
+	dst, ok = is.searchMetricName(dst, metricID)
 	db.putIndexSearch(is)
-	if err == nil {
+	if ok {
 		// There is no need in verifying whether the given metricID is deleted,
 		// since the filtering must be performed before calling this func.
 		db.putMetricNameToCache(metricID, dst)
-		return dst, nil
-	}
-	if err != io.EOF {
-		return dst, err
+		return dst, true
 	}
 
 	// Try searching in the external indexDB.
 	if db.doExtDB(func(extDB *indexDB) {
 		is := extDB.getIndexSearch(noDeadline)
-		dst, err = is.searchMetricName(dst, metricID)
+		dst, ok = is.searchMetricName(dst, metricID)
 		extDB.putIndexSearch(is)
-		if err == nil {
+		if ok {
 			// There is no need in verifying whether the given metricID is deleted,
 			// since the filtering must be performed before calling this func.
 			extDB.putMetricNameToCache(metricID, dst)
 		}
-	}) {
-		return dst, err
+	}) && ok {
+		return dst, true
 	}
 
 	// Cannot find MetricName for the given metricID. This may be the case
@@ -1484,7 +1473,7 @@ func (db *indexDB) searchMetricNameWithCache(dst []byte, metricID uint64) ([]byt
 	// Mark the metricID as deleted, so it will be created again when new data point
 	// for the given time series will arrive.
 	db.deleteMetricIDs([]uint64{metricID})
-	return dst, io.EOF
+	return dst, false
 }
 
 // DeleteTSIDs marks as deleted all the TSIDs matching the given tfss.
@@ -1695,6 +1684,8 @@ func (db *indexDB) getTSIDsFromMetricIDs(qt *querytracer.Tracer, metricIDs []uin
 	if len(metricIDs) == 0 {
 		return nil, nil
 	}
+
+	// Search for TSIDs in the current indexdb
 	tsids := make([]TSID, len(metricIDs))
 	var extMetricIDs []uint64
 	i := 0
@@ -1719,13 +1710,10 @@ func (db *indexDB) getTSIDsFromMetricIDs(qt *querytracer.Tracer, metricIDs []uin
 			if err != io.EOF {
 				return err
 			}
-			if err := is.getTSIDByMetricID(tsid, metricID); err != nil {
-				if err == io.EOF {
-					// Postpone searching for the metricID in the extDB.
-					extMetricIDs = append(extMetricIDs, metricID)
-					continue
-				}
-				return fmt.Errorf("cannot find tsid %d out of %d for metricID %d: %w", i, len(metricIDs), metricID, err)
+			if !is.getTSIDByMetricID(tsid, metricID) {
+				// Postpone searching for the missing metricID in the extDB.
+				extMetricIDs = append(extMetricIDs, metricID)
+				continue
 			}
 			is.db.putToMetricIDCache(metricID, tsid)
 			i++
@@ -1738,40 +1726,37 @@ func (db *indexDB) getTSIDsFromMetricIDs(qt *querytracer.Tracer, metricIDs []uin
 	tsidsFound := i
 	qt.Printf("found %d tsids for %d metricIDs in the current indexdb", tsidsFound, len(metricIDs))
 
-	// Search for extMetricIDs in the extDB.
-	db.doExtDB(func(extDB *indexDB) {
-		is := extDB.getIndexSearch(deadline)
-		defer extDB.putIndexSearch(is)
-		for loopsPaceLimiter, metricID := range extMetricIDs {
-			if loopsPaceLimiter&paceLimiterSlowIterationsMask == 0 {
-				if err = checkSearchDeadlineAndPace(is.deadline); err != nil {
-					return
+	if len(extMetricIDs) > 0 {
+		// Search for extMetricIDs in the previous indexdb (aka extDB)
+		db.doExtDB(func(extDB *indexDB) {
+			is := extDB.getIndexSearch(deadline)
+			defer extDB.putIndexSearch(is)
+			for loopsPaceLimiter, metricID := range extMetricIDs {
+				if loopsPaceLimiter&paceLimiterSlowIterationsMask == 0 {
+					if err = checkSearchDeadlineAndPace(is.deadline); err != nil {
+						return
+					}
 				}
-			}
-			// There is no need in searching for TSIDs in MetricID->TSID cache, since
-			// this has been already done in the loop above (the MetricID->TSID cache is global).
-			tsid := &tsids[i]
-			if err = is.getTSIDByMetricID(tsid, metricID); err != nil {
-				if err == io.EOF {
+				// There is no need in searching for TSIDs in MetricID->TSID cache, since
+				// this has been already done in the loop above (the MetricID->TSID cache is global).
+				tsid := &tsids[i]
+				if !is.getTSIDByMetricID(tsid, metricID) {
 					// Cannot find TSID for the given metricID.
 					// This may be the case on incomplete indexDB
 					// due to snapshot or due to unflushed entries.
 					// Just increment errors counter and skip it for now.
 					atomic.AddUint64(&is.db.missingTSIDsForMetricID, 1)
-					err = nil
 					continue
 				}
-				err = fmt.Errorf("cannot find tsid for metricID=%d: %w", metricID, err)
-				return
+				is.db.putToMetricIDCache(metricID, tsid)
+				i++
 			}
-			is.db.putToMetricIDCache(metricID, tsid)
-			i++
+		})
+		if err != nil {
+			return nil, fmt.Errorf("error when searching for TSIDs by metricIDs in the previous indexdb: %w", err)
 		}
-	})
-	if err != nil {
-		return nil, fmt.Errorf("error when searching for TSIDs by metricIDs in the previous indexdb: %w", err)
+		qt.Printf("found %d tsids for %d metricIDs in the previous indexdb", i-tsidsFound, len(extMetricIDs))
 	}
-	qt.Printf("found %d tsids for %d metricIDs in the previous indexdb", i-tsidsFound, len(extMetricIDs))
 
 	tsids = tsids[:i]
 	qt.Printf("load %d tsids for %d metricIDs from both current and previous indexdb", len(tsids), len(metricIDs))
@@ -1824,36 +1809,36 @@ func (is *indexSearch) getTSIDByMetricNameNoExtDB(dst *TSID, metricName []byte,
 	return false
 }
 
-func (is *indexSearch) searchMetricNameWithCache(dst []byte, metricID uint64) ([]byte, error) {
+func (is *indexSearch) searchMetricNameWithCache(dst []byte, metricID uint64) ([]byte, bool) {
 	metricName := is.db.getMetricNameFromCache(dst, metricID)
 	if len(metricName) > len(dst) {
-		return metricName, nil
+		return metricName, true
 	}
-	var err error
-	dst, err = is.searchMetricName(dst, metricID)
-	if err == nil {
+	var ok bool
+	dst, ok = is.searchMetricName(dst, metricID)
+	if ok {
 		// There is no need in verifying whether the given metricID is deleted,
 		// since the filtering must be performed before calling this func.
 		is.db.putMetricNameToCache(metricID, dst)
-		return dst, nil
+		return dst, true
 	}
-	return dst, err
+	return dst, false
 }
 
-func (is *indexSearch) searchMetricName(dst []byte, metricID uint64) ([]byte, error) {
+func (is *indexSearch) searchMetricName(dst []byte, metricID uint64) ([]byte, bool) {
 	ts := &is.ts
 	kb := &is.kb
 	kb.B = is.marshalCommonPrefix(kb.B[:0], nsPrefixMetricIDToMetricName)
 	kb.B = encoding.MarshalUint64(kb.B, metricID)
 	if err := ts.FirstItemWithPrefix(kb.B); err != nil {
 		if err == io.EOF {
-			return dst, err
+			return dst, false
 		}
-		return dst, fmt.Errorf("error when searching metricName by metricID; searchPrefix %q: %w", kb.B, err)
+		logger.Panicf("FATAL: error when searching metricName by metricID; searchPrefix %q: %w", kb.B, err)
 	}
 	v := ts.Item[len(kb.B):]
 	dst = append(dst, v...)
-	return dst, nil
+	return dst, true
 }
 
 func (is *indexSearch) containsTimeRange(tr TimeRange) (bool, error) {
@@ -1884,7 +1869,7 @@ func (is *indexSearch) containsTimeRange(tr TimeRange) (bool, error) {
 	return true, nil
 }
 
-func (is *indexSearch) getTSIDByMetricID(dst *TSID, metricID uint64) error {
+func (is *indexSearch) getTSIDByMetricID(dst *TSID, metricID uint64) bool {
 	// There is no need in checking for deleted metricIDs here, since they
 	// must be checked by the caller.
 	ts := &is.ts
@@ -1893,19 +1878,19 @@ func (is *indexSearch) getTSIDByMetricID(dst *TSID, metricID uint64) error {
 	kb.B = encoding.MarshalUint64(kb.B, metricID)
 	if err := ts.FirstItemWithPrefix(kb.B); err != nil {
 		if err == io.EOF {
-			return err
+			return false
 		}
-		return fmt.Errorf("error when searching TSID by metricID; searchPrefix %q: %w", kb.B, err)
+		logger.Panicf("FATAL: error when searching TSID by metricID=%d; searchPrefix %q: %s", metricID, kb.B, err)
 	}
 	v := ts.Item[len(kb.B):]
 	tail, err := dst.Unmarshal(v)
 	if err != nil {
-		return fmt.Errorf("cannot unmarshal TSID=%X: %w", v, err)
+		logger.Panicf("FATAL: cannot unmarshal the found TSID=%X for metricID=%d: %s", v, metricID, err)
 	}
 	if len(tail) > 0 {
-		return fmt.Errorf("unexpected non-zero tail left after unmarshaling TSID: %X", tail)
+		logger.Panicf("FATAL: unexpected non-zero tail left after unmarshaling TSID for metricID=%d: %X", metricID, tail)
 	}
-	return nil
+	return true
 }
 
 // updateMetricIDsByMetricNameMatch matches metricName values for the given srcMetricIDs against tfs
@@ -1932,18 +1917,15 @@ func (is *indexSearch) updateMetricIDsByMetricNameMatch(qt *querytracer.Tracer,
 				return err
 			}
 		}
-		var err error
-		metricName.B, err = is.searchMetricNameWithCache(metricName.B[:0], metricID)
-		if err != nil {
-			if err == io.EOF {
-				// It is likely the metricID->metricName entry didn't propagate to inverted index yet.
-				// Skip this metricID for now.
-				continue
-			}
-			return fmt.Errorf("cannot find metricName by metricID %d: %w", metricID, err)
+		var ok bool
+		metricName.B, ok = is.searchMetricNameWithCache(metricName.B[:0], metricID)
+		if !ok {
+			// It is likely the metricID->metricName entry didn't propagate to inverted index yet.
+			// Skip this metricID for now.
+			continue
 		}
 		if err := mn.Unmarshal(metricName.B); err != nil {
-			return fmt.Errorf("cannot unmarshal metricName %q: %w", metricName.B, err)
+			logger.Panicf("FATAL: cannot unmarshal metricName %q: %s", metricName.B, err)
 		}
 
 		// Match the mn against tfs.
diff --git a/lib/storage/partition.go b/lib/storage/partition.go
index da66c9c88..c9d7860c1 100644
--- a/lib/storage/partition.go
+++ b/lib/storage/partition.go
@@ -119,8 +119,6 @@ type partition struct {
 	inmemoryAssistedMerges uint64
 	smallAssistedMerges    uint64
 
-	mergeNeedFreeDiskSpace uint64
-
 	mergeIdx uint64
 
 	smallPartsPath string
@@ -354,8 +352,6 @@ type partitionMetrics struct {
 
 	InmemoryAssistedMerges uint64
 	SmallAssistedMerges    uint64
-
-	MergeNeedFreeDiskSpace uint64
 }
 
 // TotalRowsCount returns total number of rows in tm.
@@ -421,8 +417,6 @@ func (pt *partition) UpdateMetrics(m *partitionMetrics) {
 
 	m.InmemoryAssistedMerges += atomic.LoadUint64(&pt.inmemoryAssistedMerges)
 	m.SmallAssistedMerges += atomic.LoadUint64(&pt.smallAssistedMerges)
-
-	m.MergeNeedFreeDiskSpace += atomic.LoadUint64(&pt.mergeNeedFreeDiskSpace)
 }
 
 // AddRows adds the given rows to the partition pt.
@@ -936,29 +930,25 @@ func (pt *partition) ForceMergeAllParts() error {
 		// Nothing to merge.
 		return nil
 	}
-	for {
-		// Check whether there is enough disk space for merging pws.
-		newPartSize := getPartsSize(pws)
-		maxOutBytes := fs.MustGetFreeSpace(pt.bigPartsPath)
-		if newPartSize > maxOutBytes {
-			freeSpaceNeededBytes := newPartSize - maxOutBytes
-			forceMergeLogger.Warnf("cannot initiate force merge for the partition %s; additional space needed: %d bytes", pt.name, freeSpaceNeededBytes)
-			pt.releasePartsToMerge(pws)
-			return nil
-		}
 
-		// If len(pws) == 1, then the merge must run anyway.
-		// This allows applying the configured retention, removing the deleted series
-		// and performing de-duplication if needed.
-		if err := pt.mergePartsOptimal(pws, pt.stopCh); err != nil {
-			return fmt.Errorf("cannot force merge %d parts from partition %q: %w", len(pws), pt.name, err)
-		}
-		pws = pt.getAllPartsForMerge()
-		if len(pws) <= 1 {
-			pt.releasePartsToMerge(pws)
-			return nil
-		}
+	// Check whether there is enough disk space for merging pws.
+	newPartSize := getPartsSize(pws)
+	maxOutBytes := fs.MustGetFreeSpace(pt.bigPartsPath)
+	if newPartSize > maxOutBytes {
+		freeSpaceNeededBytes := newPartSize - maxOutBytes
+		forceMergeLogger.Warnf("cannot initiate force merge for the partition %s; additional space needed: %d bytes", pt.name, freeSpaceNeededBytes)
+		pt.releasePartsToMerge(pws)
+		return nil
 	}
+
+	// If len(pws) == 1, then the merge must run anyway.
+	// This allows applying the configured retention, removing the deleted series
+	// and performing de-duplication if needed.
+	if err := pt.mergePartsOptimal(pws, pt.stopCh); err != nil {
+		return fmt.Errorf("cannot force merge %d parts from partition %q: %w", len(pws), pt.name, err)
+	}
+
+	return nil
 }
 
 var forceMergeLogger = logger.WithThrottler("forceMerge", time.Minute)
@@ -1123,7 +1113,9 @@ func (pt *partition) getMaxSmallPartSize() uint64 {
 }
 
 func (pt *partition) getMaxBigPartSize() uint64 {
-	workersCount := getDefaultMergeConcurrency(4)
+	// Always use 4 workers for big merges due to historical reasons.
+	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4915#issuecomment-1733922830
+	workersCount := 4
 	return getMaxOutBytes(pt.bigPartsPath, workersCount)
 }
 
@@ -1150,10 +1142,9 @@ func (pt *partition) mergeInmemoryParts() error {
 	maxOutBytes := pt.getMaxBigPartSize()
 
 	pt.partsLock.Lock()
-	pws, needFreeSpace := getPartsToMerge(pt.inmemoryParts, maxOutBytes, false)
+	pws := getPartsToMerge(pt.inmemoryParts, maxOutBytes, false)
 	pt.partsLock.Unlock()
 
-	atomicSetBool(&pt.mergeNeedFreeDiskSpace, needFreeSpace)
 	return pt.mergeParts(pws, pt.stopCh, false)
 }
 
@@ -1170,10 +1161,9 @@ func (pt *partition) mergeExistingParts(isFinal bool) error {
 	dst = append(dst, pt.inmemoryParts...)
 	dst = append(dst, pt.smallParts...)
 	dst = append(dst, pt.bigParts...)
-	pws, needFreeSpace := getPartsToMerge(dst, maxOutBytes, isFinal)
+	pws := getPartsToMerge(dst, maxOutBytes, isFinal)
 	pt.partsLock.Unlock()
 
-	atomicSetBool(&pt.mergeNeedFreeDiskSpace, needFreeSpace)
 	return pt.mergeParts(pws, pt.stopCh, isFinal)
 }
 
@@ -1190,14 +1180,6 @@ func (pt *partition) releasePartsToMerge(pws []*partWrapper) {
 
 var errNothingToMerge = fmt.Errorf("nothing to merge")
 
-func atomicSetBool(p *uint64, b bool) {
-	v := uint64(0)
-	if b {
-		v = 1
-	}
-	atomic.StoreUint64(p, v)
-}
-
 func (pt *partition) runFinalDedup() error {
 	requiredDedupInterval, actualDedupInterval := pt.getRequiredDedupInterval()
 	t := time.Now()
@@ -1633,8 +1615,7 @@ func (pt *partition) removeStaleParts() {
 // getPartsToMerge returns optimal parts to merge from pws.
 //
 // The summary size of the returned parts must be smaller than maxOutBytes.
-// The function returns true if pws contains parts, which cannot be merged because of maxOutBytes limit.
-func getPartsToMerge(pws []*partWrapper, maxOutBytes uint64, isFinal bool) ([]*partWrapper, bool) {
+func getPartsToMerge(pws []*partWrapper, maxOutBytes uint64, isFinal bool) []*partWrapper {
 	pwsRemaining := make([]*partWrapper, 0, len(pws))
 	for _, pw := range pws {
 		if !pw.isInMerge {
@@ -1643,14 +1624,13 @@ func getPartsToMerge(pws []*partWrapper, maxOutBytes uint64, isFinal bool) ([]*p
 	}
 	maxPartsToMerge := defaultPartsToMerge
 	var pms []*partWrapper
-	needFreeSpace := false
 	if isFinal {
 		for len(pms) == 0 && maxPartsToMerge >= finalPartsToMerge {
-			pms, needFreeSpace = appendPartsToMerge(pms[:0], pwsRemaining, maxPartsToMerge, maxOutBytes)
+			pms = appendPartsToMerge(pms[:0], pwsRemaining, maxPartsToMerge, maxOutBytes)
 			maxPartsToMerge--
 		}
 	} else {
-		pms, needFreeSpace = appendPartsToMerge(pms[:0], pwsRemaining, maxPartsToMerge, maxOutBytes)
+		pms = appendPartsToMerge(pms[:0], pwsRemaining, maxPartsToMerge, maxOutBytes)
 	}
 	for _, pw := range pms {
 		if pw.isInMerge {
@@ -1658,7 +1638,7 @@ func getPartsToMerge(pws []*partWrapper, maxOutBytes uint64, isFinal bool) ([]*p
 		}
 		pw.isInMerge = true
 	}
-	return pms, needFreeSpace
+	return pms
 }
 
 // minMergeMultiplier is the minimum multiplier for the size of the output part
@@ -1669,13 +1649,11 @@ func getPartsToMerge(pws []*partWrapper, maxOutBytes uint64, isFinal bool) ([]*p
 // The 1.7 is good enough for production workloads.
 const minMergeMultiplier = 1.7
 
-// appendPartsToMerge finds optimal parts to merge from src, appends
-// them to dst and returns the result.
-// The function returns true if src contains parts, which cannot be merged because of maxOutBytes limit.
-func appendPartsToMerge(dst, src []*partWrapper, maxPartsToMerge int, maxOutBytes uint64) ([]*partWrapper, bool) {
+// appendPartsToMerge finds optimal parts to merge from src, appends them to dst and returns the result.
+func appendPartsToMerge(dst, src []*partWrapper, maxPartsToMerge int, maxOutBytes uint64) []*partWrapper {
 	if len(src) < 2 {
 		// There is no need in merging zero or one part :)
-		return dst, false
+		return dst
 	}
 	if maxPartsToMerge < 2 {
 		logger.Panicf("BUG: maxPartsToMerge cannot be smaller than 2; got %d", maxPartsToMerge)
@@ -1683,18 +1661,15 @@ func appendPartsToMerge(dst, src []*partWrapper, maxPartsToMerge int, maxOutByte
 
 	// Filter out too big parts.
 	// This should reduce N for O(N^2) algorithm below.
-	skippedBigParts := 0
 	maxInPartBytes := uint64(float64(maxOutBytes) / minMergeMultiplier)
 	tmp := make([]*partWrapper, 0, len(src))
 	for _, pw := range src {
 		if pw.p.size > maxInPartBytes {
-			skippedBigParts++
 			continue
 		}
 		tmp = append(tmp, pw)
 	}
 	src = tmp
-	needFreeSpace := skippedBigParts > 1
 
 	sortPartsForOptimalMerge(src)
 
@@ -1713,15 +1688,12 @@ func appendPartsToMerge(dst, src []*partWrapper, maxPartsToMerge int, maxOutByte
 	for i := minSrcParts; i <= maxSrcParts; i++ {
 		for j := 0; j <= len(src)-i; j++ {
 			a := src[j : j+i]
-			outSize := getPartsSize(a)
-			if outSize > maxOutBytes {
-				needFreeSpace = true
-			}
 			if a[0].p.size*uint64(len(a)) < a[len(a)-1].p.size {
 				// Do not merge parts with too big difference in size,
 				// since this results in unbalanced merges.
 				continue
 			}
+			outSize := getPartsSize(a)
 			if outSize > maxOutBytes {
 				// There is no need in verifying remaining parts with bigger sizes.
 				break
@@ -1742,9 +1714,9 @@ func appendPartsToMerge(dst, src []*partWrapper, maxPartsToMerge int, maxOutByte
 	if maxM < minM {
 		// There is no sense in merging parts with too small m,
 		// since this leads to high disk write IO.
-		return dst, needFreeSpace
+		return dst
 	}
-	return append(dst, pws...), needFreeSpace
+	return append(dst, pws...)
 }
 
 func sortPartsForOptimalMerge(pws []*partWrapper) {
@@ -1783,6 +1755,18 @@ func mustOpenParts(path string, partNames []string) []*partWrapper {
 	des := fs.MustReadDir(path)
 	m := make(map[string]struct{}, len(partNames))
 	for _, partName := range partNames {
+		// Make sure the partName exists on disk.
+		// If it is missing, then manual action from the user is needed,
+		// since this is unexpected state, which cannot occur under normal operation,
+		// including unclean shutdown.
+		partPath := filepath.Join(path, partName)
+		if !fs.IsPathExist(partPath) {
+			partsFile := filepath.Join(path, partsFilename)
+			logger.Panicf("FATAL: part %q is listed in %q, but is missing on disk; "+
+				"ensure %q contents is not corrupted; remove %q to rebuild its' content from the list of existing parts",
+				partPath, partsFile, partsFile, partsFile)
+		}
+
 		m[partName] = struct{}{}
 	}
 	for _, de := range des {
diff --git a/lib/storage/search.go b/lib/storage/search.go
index 1873939f5..f5b4ab7f8 100644
--- a/lib/storage/search.go
+++ b/lib/storage/search.go
@@ -211,16 +211,12 @@ func (s *Search) NextMetricBlock() bool {
 				// Skip the block, since it contains only data outside the configured retention.
 				continue
 			}
-			var err error
-			s.MetricBlockRef.MetricName, err = s.idb.searchMetricNameWithCache(s.MetricBlockRef.MetricName[:0], tsid.MetricID)
-			if err != nil {
-				if err == io.EOF {
-					// Skip missing metricName for tsid.MetricID.
-					// It should be automatically fixed. See indexDB.searchMetricNameWithCache for details.
-					continue
-				}
-				s.err = err
-				return false
+			var ok bool
+			s.MetricBlockRef.MetricName, ok = s.idb.searchMetricNameWithCache(s.MetricBlockRef.MetricName[:0], tsid.MetricID)
+			if !ok {
+				// Skip missing metricName for tsid.MetricID.
+				// It should be automatically fixed. See indexDB.searchMetricNameWithCache for details.
+				continue
 			}
 			s.prevMetricID = tsid.MetricID
 		}
diff --git a/lib/storage/storage.go b/lib/storage/storage.go
index 18b0ba34a..7d88d36c5 100644
--- a/lib/storage/storage.go
+++ b/lib/storage/storage.go
@@ -1114,15 +1114,12 @@ func (s *Storage) SearchMetricNames(qt *querytracer.Tracer, tfss []*TagFilters,
 				return nil, err
 			}
 		}
-		var err error
-		metricName, err = idb.searchMetricNameWithCache(metricName[:0], metricID)
-		if err != nil {
-			if err == io.EOF {
-				// Skip missing metricName for metricID.
-				// It should be automatically fixed. See indexDB.searchMetricName for details.
-				continue
-			}
-			return nil, fmt.Errorf("error when searching metricName for metricID=%d: %w", metricID, err)
+		var ok bool
+		metricName, ok = idb.searchMetricNameWithCache(metricName[:0], metricID)
+		if !ok {
+			// Skip missing metricName for metricID.
+			// It should be automatically fixed. See indexDB.searchMetricName for details.
+			continue
 		}
 		if _, ok := metricNamesSeen[string(metricName)]; ok {
 			// The given metric name was already seen; skip it
@@ -1156,7 +1153,7 @@ func (s *Storage) prefetchMetricNames(qt *querytracer.Tracer, srcMetricIDs []uin
 	qt.Printf("%d out of %d metric names must be pre-fetched", len(metricIDs), len(srcMetricIDs))
 	if len(metricIDs) < 500 {
 		// It is cheaper to skip pre-fetching and obtain metricNames inline.
-		qt.Printf("skip pre-fetching metric names for low number of metrid ids=%d", len(metricIDs))
+		qt.Printf("skip pre-fetching metric names for low number of metric ids=%d", len(metricIDs))
 		return nil
 	}
 	atomic.AddUint64(&s.slowMetricNameLoads, uint64(len(metricIDs)))
@@ -1175,13 +1172,11 @@ func (s *Storage) prefetchMetricNames(qt *querytracer.Tracer, srcMetricIDs []uin
 				return err
 			}
 		}
-		metricName, err = is.searchMetricNameWithCache(metricName[:0], metricID)
-		if err != nil {
-			if err == io.EOF {
-				missingMetricIDs = append(missingMetricIDs, metricID)
-				continue
-			}
-			return fmt.Errorf("error in pre-fetching metricName for metricID=%d: %w", metricID, err)
+		var ok bool
+		metricName, ok = is.searchMetricNameWithCache(metricName[:0], metricID)
+		if !ok {
+			missingMetricIDs = append(missingMetricIDs, metricID)
+			continue
 		}
 	}
 	idb.doExtDB(func(extDB *indexDB) {
@@ -1193,11 +1188,7 @@ func (s *Storage) prefetchMetricNames(qt *querytracer.Tracer, srcMetricIDs []uin
 					return
 				}
 			}
-			metricName, err = is.searchMetricNameWithCache(metricName[:0], metricID)
-			if err != nil && err != io.EOF {
-				err = fmt.Errorf("error in pre-fetching metricName for metricID=%d in extDB: %w", metricID, err)
-				return
-			}
+			metricName, _ = is.searchMetricNameWithCache(metricName[:0], metricID)
 		}
 	})
 	if err != nil && err != io.EOF {
diff --git a/vendor/github.com/VictoriaMetrics/metricsql/aggr.go b/vendor/github.com/VictoriaMetrics/metricsql/aggr.go
index b8c77c349..51bc36a0f 100644
--- a/vendor/github.com/VictoriaMetrics/metricsql/aggr.go
+++ b/vendor/github.com/VictoriaMetrics/metricsql/aggr.go
@@ -43,7 +43,8 @@ var aggrFuncs = map[string]bool{
 	"zscore":         true,
 }
 
-func isAggrFunc(s string) bool {
+// IsAggrFunc returns whether funcName is a known aggregate function.
+func IsAggrFunc(s string) bool {
 	s = strings.ToLower(s)
 	return aggrFuncs[s]
 }
diff --git a/vendor/github.com/VictoriaMetrics/metricsql/optimizer.go b/vendor/github.com/VictoriaMetrics/metricsql/optimizer.go
index 8072667cd..046a8f1b8 100644
--- a/vendor/github.com/VictoriaMetrics/metricsql/optimizer.go
+++ b/vendor/github.com/VictoriaMetrics/metricsql/optimizer.go
@@ -362,7 +362,7 @@ func getFuncArgIdxForOptimization(funcName string, args []Expr) int {
 	if IsTransformFunc(funcName) {
 		return getTransformArgIdxForOptimization(funcName, args)
 	}
-	if isAggrFunc(funcName) {
+	if IsAggrFunc(funcName) {
 		return getAggrArgIdxForOptimization(funcName, args)
 	}
 	return -1
diff --git a/vendor/github.com/VictoriaMetrics/metricsql/parser.go b/vendor/github.com/VictoriaMetrics/metricsql/parser.go
index bc3f83e7f..8458e5f71 100644
--- a/vendor/github.com/VictoriaMetrics/metricsql/parser.go
+++ b/vendor/github.com/VictoriaMetrics/metricsql/parser.go
@@ -31,6 +31,9 @@ func Parse(s string) (Expr, error) {
 	}
 	e = removeParensExpr(e)
 	e = simplifyConstants(e)
+	if err := checkSupportedFunctions(e); err != nil {
+		return nil, err
+	}
 	return e, nil
 }
 
@@ -52,7 +55,6 @@ func getDefaultWithArgExprs() []*withArgExpr {
 				clamp_max(step()/300, 1)
 			)`,
 
-			`median_over_time(m) = quantile_over_time(0.5, m)`,
 			`range_median(q) = range_quantile(0.5, q)`,
 			`alias(q, name) = label_set(q, "__name__", name)`,
 		})
@@ -596,7 +598,7 @@ func (p *parser) parseParensExpr() (*parensExpr, error) {
 }
 
 func (p *parser) parseAggrFuncExpr() (*AggrFuncExpr, error) {
-	if !isAggrFunc(p.lex.Token) {
+	if !IsAggrFunc(p.lex.Token) {
 		return nil, fmt.Errorf(`AggrFuncExpr: unexpected token %q; want aggregate func`, p.lex.Token)
 	}
 
@@ -1608,7 +1610,7 @@ func (p *parser) parseIdentExpr() (Expr, error) {
 	}
 	if isIdentPrefix(p.lex.Token) {
 		p.lex.Prev()
-		if isAggrFunc(p.lex.Token) {
+		if IsAggrFunc(p.lex.Token) {
 			return p.parseAggrFuncExpr()
 		}
 		return p.parseMetricExpr()
@@ -1620,7 +1622,7 @@ func (p *parser) parseIdentExpr() (Expr, error) {
 	switch p.lex.Token {
 	case "(":
 		p.lex.Prev()
-		if isAggrFunc(p.lex.Token) {
+		if IsAggrFunc(p.lex.Token) {
 			return p.parseAggrFuncExpr()
 		}
 		return p.parseFuncExpr()
diff --git a/vendor/github.com/VictoriaMetrics/metricsql/rollup.go b/vendor/github.com/VictoriaMetrics/metricsql/rollup.go
index da3204adc..99d8f56bc 100644
--- a/vendor/github.com/VictoriaMetrics/metricsql/rollup.go
+++ b/vendor/github.com/VictoriaMetrics/metricsql/rollup.go
@@ -44,6 +44,7 @@ var rollupFuncs = map[string]bool{
 	"lifetime":                true,
 	"mad_over_time":           true,
 	"max_over_time":           true,
+	"median_over_time":        true,
 	"min_over_time":           true,
 	"mode_over_time":          true,
 	"predict_linear":          true,
diff --git a/vendor/github.com/VictoriaMetrics/metricsql/utils.go b/vendor/github.com/VictoriaMetrics/metricsql/utils.go
index 11828f39f..2405b1fef 100644
--- a/vendor/github.com/VictoriaMetrics/metricsql/utils.go
+++ b/vendor/github.com/VictoriaMetrics/metricsql/utils.go
@@ -1,5 +1,10 @@
 package metricsql
 
+import (
+	"fmt"
+	"strings"
+)
+
 // ExpandWithExprs expands WITH expressions inside q and returns the resulting
 // PromQL without WITH expressions.
 func ExpandWithExprs(q string) (string, error) {
@@ -14,7 +19,7 @@ func ExpandWithExprs(q string) (string, error) {
 // VisitAll recursively calls f for all the Expr children in e.
 //
 // It visits leaf children at first and then visits parent nodes.
-// It is safe modifying expr in f.
+// It is safe modifying e in f.
 func VisitAll(e Expr, f func(expr Expr)) {
 	switch expr := e.(type) {
 	case *BinaryOpExpr:
@@ -36,3 +41,38 @@ func VisitAll(e Expr, f func(expr Expr)) {
 	}
 	f(e)
 }
+
+// IsSupportedFunction returns true if funcName contains supported MetricsQL function
+func IsSupportedFunction(funcName string) bool {
+	funcName = strings.ToLower(funcName)
+	if IsRollupFunc(funcName) {
+		return true
+	}
+	if IsTransformFunc(funcName) {
+		return true
+	}
+	if IsAggrFunc(funcName) {
+		return true
+	}
+	return false
+}
+
+func checkSupportedFunctions(e Expr) error {
+	var err error
+	VisitAll(e, func(expr Expr) {
+		if err != nil {
+			return
+		}
+		switch t := expr.(type) {
+		case *FuncExpr:
+			if !IsRollupFunc(t.Name) && !IsTransformFunc(t.Name) {
+				err = fmt.Errorf("unsupported function %q", t.Name)
+			}
+		case *AggrFuncExpr:
+			if !IsAggrFunc(t.Name) {
+				err = fmt.Errorf("unsupported aggregate function %q", t.Name)
+			}
+		}
+	})
+	return err
+}
diff --git a/vendor/modules.txt b/vendor/modules.txt
index 1dbceba9a..37d0db292 100644
--- a/vendor/modules.txt
+++ b/vendor/modules.txt
@@ -99,7 +99,7 @@ github.com/VictoriaMetrics/fasthttp/stackless
 # github.com/VictoriaMetrics/metrics v1.24.0
 ## explicit; go 1.20
 github.com/VictoriaMetrics/metrics
-# github.com/VictoriaMetrics/metricsql v0.64.0
+# github.com/VictoriaMetrics/metricsql v0.66.0
 ## explicit; go 1.13
 github.com/VictoriaMetrics/metricsql
 github.com/VictoriaMetrics/metricsql/binaryop


Test patch:
diff --git a/app/vmselect/promql/exec_test.go b/app/vmselect/promql/exec_test.go
index 90d31e349..60430bdf1 100644
--- a/app/vmselect/promql/exec_test.go
+++ b/app/vmselect/promql/exec_test.go
@@ -45,7 +45,7 @@ func TestEscapeDotsInRegexpLabelFilters(t *testing.T) {
 	f("2", "2")
 	f(`foo.bar + 123`, `foo.bar + 123`)
 	f(`foo{bar=~"baz.xx.yyy"}`, `foo{bar=~"baz\\.xx\\.yyy"}`)
-	f(`foo(a.b{c="d.e",x=~"a.b.+[.a]",y!~"aaa.bb|cc.dd"}) + x.y(1,sum({x=~"aa.bb"}))`, `foo(a.b{c="d.e",x=~"a\\.b.+[\\.a]",y!~"aaa\\.bb|cc\\.dd"}) + x.y(1, sum({x=~"aa\\.bb"}))`)
+	f(`sum(a.b{c="d.e",x=~"a.b.+[.a]",y!~"aaa.bb|cc.dd"}) + avg_over_time(1,sum({x=~"aa.bb"}))`, `sum(a.b{c="d.e",x=~"a\\.b.+[\\.a]",y!~"aaa\\.bb|cc\\.dd"}) + avg_over_time(1, sum({x=~"aa\\.bb"}))`)
 }
 
 func TestExecSuccess(t *testing.T) {
@@ -7669,7 +7669,7 @@ func TestExecSuccess(t *testing.T) {
 	})
 	t.Run(`aggr_over_time(multi-func)`, func(t *testing.T) {
 		t.Parallel()
-		q := `sort(aggr_over_time(("min_over_time", "count_over_time", "max_over_time"), round(rand(0),0.1)[:10s]))`
+		q := `sort(aggr_over_time(("min_over_time", "median_over_time", "max_over_time"), round(rand(0),0.1)[:10s]))`
 		r1 := netstorage.Result{
 			MetricName: metricNameExpected,
 			Values:     []float64{0, 0, 0, 0, 0, 0},
@@ -7681,21 +7681,21 @@ func TestExecSuccess(t *testing.T) {
 		}}
 		r2 := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{0.8, 0.9, 1, 0.9, 1, 0.9},
+			Values:     []float64{0.4, 0.5, 0.5, 0.75, 0.6, 0.45},
 			Timestamps: timestampsExpected,
 		}
 		r2.MetricName.Tags = []storage.Tag{{
 			Key:   []byte("rollup"),
-			Value: []byte("max_over_time"),
+			Value: []byte("median_over_time"),
 		}}
 		r3 := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{20, 20, 20, 20, 20, 20},
+			Values:     []float64{0.8, 0.9, 1, 0.9, 1, 0.9},
 			Timestamps: timestampsExpected,
 		}
 		r3.MetricName.Tags = []storage.Tag{{
 			Key:   []byte("rollup"),
-			Value: []byte("count_over_time"),
+			Value: []byte("max_over_time"),
 		}}
 		resultExpected := []netstorage.Result{r1, r2, r3}
 		f(q, resultExpected)
@@ -8479,11 +8479,11 @@ func TestExecSuccess(t *testing.T) {
 	})
 	t.Run(`result sorting`, func(t *testing.T) {
 		t.Parallel()
-		q := `label_set(1, "instance", "localhost:1001", "type", "free")
-			or label_set(1, "instance", "localhost:1001", "type", "buffers")
-			or label_set(1, "instance", "localhost:1000", "type", "buffers")
-			or label_set(1, "instance", "localhost:1000", "type", "free")
-`
+		q := `(label_set(1, "instance", "localhost:1001", "type", "free"),
+			label_set(1, "instance", "localhost:1001", "type", "buffers"),
+			label_set(1, "instance", "localhost:1000", "type", "buffers"),
+			label_set(1, "instance", "localhost:1000", "type", "free"),
+		)`
 		r1 := netstorage.Result{
 			MetricName: metricNameExpected,
 			Values:     []float64{1, 1, 1, 1, 1, 1},
@@ -8515,6 +8515,34 @@ func TestExecSuccess(t *testing.T) {
 		resultExpected := []netstorage.Result{r1, r2, r3, r4}
 		f(q, resultExpected)
 	})
+	t.Run(`no_sorting_for_or`, func(t *testing.T) {
+		t.Parallel()
+		q := `label_set(2, "foo", "bar") or label_set(1, "foo", "baz")`
+		r1 := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{2, 2, 2, 2, 2, 2},
+			Timestamps: timestampsExpected,
+		}
+		r1.MetricName.Tags = []storage.Tag{
+			{
+				Key:   []byte("foo"),
+				Value: []byte("bar"),
+			},
+		}
+		r2 := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{1, 1, 1, 1, 1, 1},
+			Timestamps: timestampsExpected,
+		}
+		r2.MetricName.Tags = []storage.Tag{
+			{
+				Key:   []byte("foo"),
+				Value: []byte("baz"),
+			},
+		}
+		resultExpected := []netstorage.Result{r1, r2}
+		f(q, resultExpected)
+	})
 	t.Run(`sort_by_label_numeric(multiple_labels_only_string)`, func(t *testing.T) {
 		t.Parallel()
 		q := `sort_by_label_numeric((
diff --git a/lib/storage/index_db_test.go b/lib/storage/index_db_test.go
index 28c01b72e..16f5f2dec 100644
--- a/lib/storage/index_db_test.go
+++ b/lib/storage/index_db_test.go
@@ -3,7 +3,6 @@ package storage
 import (
 	"bytes"
 	"fmt"
-	"io"
 	"math/rand"
 	"os"
 	"reflect"
@@ -655,19 +654,19 @@ func testIndexDBCheckTSIDByName(db *indexDB, mns []MetricName, tsids []TSID, isC
 		}
 
 		// Search for metric name for the given metricID.
-		var err error
-		metricNameCopy, err = db.searchMetricNameWithCache(metricNameCopy[:0], genTSID.TSID.MetricID)
-		if err != nil {
-			return fmt.Errorf("error in searchMetricNameWithCache for metricID=%d; i=%d: %w", genTSID.TSID.MetricID, i, err)
+		var ok bool
+		metricNameCopy, ok = db.searchMetricNameWithCache(metricNameCopy[:0], genTSID.TSID.MetricID)
+		if !ok {
+			return fmt.Errorf("cannot find metricName for metricID=%d; i=%d", genTSID.TSID.MetricID, i)
 		}
 		if !bytes.Equal(metricName, metricNameCopy) {
 			return fmt.Errorf("unexpected mn for metricID=%d;\ngot\n%q\nwant\n%q", genTSID.TSID.MetricID, metricNameCopy, metricName)
 		}
 
 		// Try searching metric name for non-existent MetricID.
-		buf, err := db.searchMetricNameWithCache(nil, 1)
-		if err != io.EOF {
-			return fmt.Errorf("expecting io.EOF error when searching for non-existing metricID; got %v", err)
+		buf, found := db.searchMetricNameWithCache(nil, 1)
+		if found {
+			return fmt.Errorf("unexpected metricName found for non-existing metricID; got %X", buf)
 		}
 		if len(buf) > 0 {
 			return fmt.Errorf("expecting empty buf when searching for non-existent metricID; got %X", buf)
diff --git a/lib/storage/partition_test.go b/lib/storage/partition_test.go
index 931ada012..2643347a5 100644
--- a/lib/storage/partition_test.go
+++ b/lib/storage/partition_test.go
@@ -34,24 +34,6 @@ func TestAppendPartsToMerge(t *testing.T) {
 	testAppendPartsToMerge(t, 3, []uint64{11, 1, 10, 100, 10}, []uint64{10, 10, 11})
 }
 
-func TestAppendPartsToMergeNeedFreeSpace(t *testing.T) {
-	f := func(sizes []uint64, maxOutBytes int, expectedNeedFreeSpace bool) {
-		t.Helper()
-		pws := newTestPartWrappersForSizes(sizes)
-		_, needFreeSpace := appendPartsToMerge(nil, pws, defaultPartsToMerge, uint64(maxOutBytes))
-		if needFreeSpace != expectedNeedFreeSpace {
-			t.Fatalf("unexpected needFreeSpace; got %v; want %v", needFreeSpace, expectedNeedFreeSpace)
-		}
-	}
-	f(nil, 1000, false)
-	f([]uint64{1000}, 100, false)
-	f([]uint64{1000}, 1100, false)
-	f([]uint64{120, 200}, 180, true)
-	f([]uint64{100, 200}, 310, false)
-	f([]uint64{100, 110, 109, 1}, 300, true)
-	f([]uint64{100, 110, 109, 1}, 330, false)
-}
-
 func TestAppendPartsToMergeManyParts(t *testing.T) {
 	// Verify that big number of parts are merged into minimal number of parts
 	// using minimum merges.
@@ -69,7 +51,7 @@ func TestAppendPartsToMergeManyParts(t *testing.T) {
 	iterationsCount := 0
 	sizeMergedTotal := uint64(0)
 	for {
-		pms, _ := appendPartsToMerge(nil, pws, defaultPartsToMerge, maxOutSize)
+		pms := appendPartsToMerge(nil, pws, defaultPartsToMerge, maxOutSize)
 		if len(pms) == 0 {
 			break
 		}
@@ -118,7 +100,7 @@ func testAppendPartsToMerge(t *testing.T, maxPartsToMerge int, initialSizes, exp
 	pws := newTestPartWrappersForSizes(initialSizes)
 
 	// Verify appending to nil.
-	pms, _ := appendPartsToMerge(nil, pws, maxPartsToMerge, 1e9)
+	pms := appendPartsToMerge(nil, pws, maxPartsToMerge, 1e9)
 	sizes := newTestSizesFromPartWrappers(pms)
 	if !reflect.DeepEqual(sizes, expectedSizes) {
 		t.Fatalf("unexpected size for maxPartsToMerge=%d, initialSizes=%d; got\n%d; want\n%d",
@@ -135,7 +117,7 @@ func testAppendPartsToMerge(t *testing.T, maxPartsToMerge int, initialSizes, exp
 		{},
 		{},
 	}
-	pms, _ = appendPartsToMerge(prefix, pws, maxPartsToMerge, 1e9)
+	pms = appendPartsToMerge(prefix, pws, maxPartsToMerge, 1e9)
 	if !reflect.DeepEqual(pms[:len(prefix)], prefix) {
 		t.Fatalf("unexpected prefix for maxPartsToMerge=%d, initialSizes=%d; got\n%+v; want\n%+v",
 			maxPartsToMerge, initialSizes, pms[:len(prefix)], prefix)