Repository: VictoriaMetrics/VictoriaMetrics
Pull request #1040 (2021-01-28 15:06:49)

Adds windows build

codecov[bot]: # [Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1040?src=pr&el=h1) Report
> Merging [#1040](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1040?src=pr&el=desc) (3d406c1) into [master](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/commit/803a00102a8d42d572561a08807bfaa1afe684ad?el=desc) (803a001) will **decrease** coverage by `0.07%`.
> The diff coverage is `3.57%`.

[![Impacted file tree graph](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1040/graphs/tree.svg?width=650&height=150&src=pr&token=YND8dYVcXn)](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1040?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1040      +/-   ##
==========================================
- Coverage   59.99%   59.92%   -0.08%     
==========================================
  Files         212      212              
  Lines       24290    24292       +2     
==========================================
- Hits        14573    14556      -17     
- Misses       8350     8381      +31     
+ Partials     1367     1355      -12     
```


| [Impacted Files](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1040?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [lib/fs/fs.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1040/diff?src=pr&el=tree#diff-bGliL2ZzL2ZzLmdv) | `5.17% <ø> (+0.17%)` | :arrow_up: |
| [lib/fs/reader\_at.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1040/diff?src=pr&el=tree#diff-bGliL2ZzL3JlYWRlcl9hdC5nbw==) | `17.43% <0.00%> (ø)` | |
| [lib/mergeset/table.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1040/diff?src=pr&el=tree#diff-bGliL21lcmdlc2V0L3RhYmxlLmdv) | `64.82% <ø> (-0.62%)` | :arrow_down: |
| [lib/persistentqueue/persistentqueue.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1040/diff?src=pr&el=tree#diff-bGliL3BlcnNpc3RlbnRxdWV1ZS9wZXJzaXN0ZW50cXVldWUuZ28=) | `66.18% <ø> (ø)` | |
| [lib/fs/fs\_nix.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1040/diff?src=pr&el=tree#diff-bGliL2ZzL2ZzX25peC5nbw==) | `3.70% <3.84%> (+3.70%)` | :arrow_up: |
| [lib/mergeset/block\_stream\_reader.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1040/diff?src=pr&el=tree#diff-bGliL21lcmdlc2V0L2Jsb2NrX3N0cmVhbV9yZWFkZXIuZ28=) | `49.30% <0.00%> (-13.20%)` | :arrow_down: |
| [app/vmselect/promql/aggr\_incremental.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1040/diff?src=pr&el=tree#diff-YXBwL3Ztc2VsZWN0L3Byb21xbC9hZ2dyX2luY3JlbWVudGFsLmdv) | `87.98% <0.00%> (+1.55%)` | :arrow_up: |
| [app/vmalert/remotewrite/remotewrite.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1040/diff?src=pr&el=tree#diff-YXBwL3ZtYWxlcnQvcmVtb3Rld3JpdGUvcmVtb3Rld3JpdGUuZ28=) | `60.37% <0.00%> (+1.88%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1040?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1040?src=pr&el=footer). Last update [9b9cb04...3d406c1](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1040?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/README.md b/README.md
index 54526ccdb..7aba95a44 100644
--- a/README.md
+++ b/README.md
@@ -605,7 +605,8 @@ and it is easier to use when migrating from Graphite to VictoriaMetrics.
 ### Graphite Render API usage
 
 [VictoriaMetrics Enterprise](https://victoriametrics.com/enterprise.html) supports [Graphite Render API](https://graphite.readthedocs.io/en/stable/render_api.html) subset
-at `/render` endpoint. This subset is required for [Graphite datasource in Grafana](https://grafana.com/docs/grafana/latest/datasources/graphite/).
+at `/render` endpoint, which is used by [Graphite datasource in Grafana](https://grafana.com/docs/grafana/latest/datasources/graphite/).
+It supports `Storage-Step` http request header, which must be set to a step between data points stored in VictoriaMetrics when configuring Graphite datasource in Grafana.
 
 
 ### Graphite Metrics API usage
diff --git a/app/vmagent/Makefile b/app/vmagent/Makefile
index 0ee29504d..e0be998ab 100644
--- a/app/vmagent/Makefile
+++ b/app/vmagent/Makefile
@@ -73,6 +73,10 @@ vmagent-ppc64le:
 vmagent-386:
 	CGO_ENABLED=0 GOARCH=386 $(MAKE) vmagent-local-with-goarch
 
+vmagent-windows:
+	GOOS=windows CGO_ENABLED=0 $(MAKE) vmagent-local-with-goarch
+
+
 vmagent-local-with-goarch:
 	APP_NAME=vmagent $(MAKE) app-local-with-goarch
 
diff --git a/app/vmselect/promql/rollup_result_cache.go b/app/vmselect/promql/rollup_result_cache.go
index 8ef54398e..e518324f6 100644
--- a/app/vmselect/promql/rollup_result_cache.go
+++ b/app/vmselect/promql/rollup_result_cache.go
@@ -178,7 +178,7 @@ func (rrc *rollupResultCache) Get(ec *EvalConfig, expr metricsql.Expr, window in
 	bb := bbPool.Get()
 	defer bbPool.Put(bb)
 
-	bb.B = marshalRollupResultCacheKey(bb.B[:0], expr, window, ec.Step)
+	bb.B = marshalRollupResultCacheKey(bb.B[:0], expr, window, ec.Step, ec.EnforcedTagFilters)
 	metainfoBuf := rrc.c.Get(nil, bb.B)
 	if len(metainfoBuf) == 0 {
 		return nil, ec.Start
@@ -198,7 +198,7 @@ func (rrc *rollupResultCache) Get(ec *EvalConfig, expr metricsql.Expr, window in
 	if len(compressedResultBuf.B) == 0 {
 		mi.RemoveKey(key)
 		metainfoBuf = mi.Marshal(metainfoBuf[:0])
-		bb.B = marshalRollupResultCacheKey(bb.B[:0], expr, window, ec.Step)
+		bb.B = marshalRollupResultCacheKey(bb.B[:0], expr, window, ec.Step, ec.EnforcedTagFilters)
 		rrc.c.Set(bb.B, metainfoBuf)
 		return nil, ec.Start
 	}
@@ -301,7 +301,7 @@ func (rrc *rollupResultCache) Put(ec *EvalConfig, expr metricsql.Expr, window in
 	bb.B = key.Marshal(bb.B[:0])
 	rrc.c.SetBig(bb.B, compressedResultBuf.B)
 
-	bb.B = marshalRollupResultCacheKey(bb.B[:0], expr, window, ec.Step)
+	bb.B = marshalRollupResultCacheKey(bb.B[:0], expr, window, ec.Step, ec.EnforcedTagFilters)
 	metainfoBuf := rrc.c.Get(nil, bb.B)
 	var mi rollupResultCacheMetainfo
 	if len(metainfoBuf) > 0 {
@@ -331,11 +331,14 @@ var tooBigRollupResults = metrics.NewCounter("vm_too_big_rollup_results_total")
 // Increment this value every time the format of the cache changes.
 const rollupResultCacheVersion = 7
 
-func marshalRollupResultCacheKey(dst []byte, expr metricsql.Expr, window, step int64) []byte {
+func marshalRollupResultCacheKey(dst []byte, expr metricsql.Expr, window, step int64, filters []storage.TagFilter) []byte {
 	dst = append(dst, rollupResultCacheVersion)
 	dst = encoding.MarshalInt64(dst, window)
 	dst = encoding.MarshalInt64(dst, step)
 	dst = expr.AppendString(dst)
+	for _, f := range filters {
+		dst = f.Marshal(dst)
+	}
 	return dst
 }
 
diff --git a/app/vmselect/promql/transform.go b/app/vmselect/promql/transform.go
index 20961e409..714f3cf80 100644
--- a/app/vmselect/promql/transform.go
+++ b/app/vmselect/promql/transform.go
@@ -19,6 +19,7 @@ import (
 
 var transformFuncsKeepMetricGroup = map[string]bool{
 	"ceil":               true,
+	"clamp":              true,
 	"clamp_max":          true,
 	"clamp_min":          true,
 	"floor":              true,
@@ -44,6 +45,7 @@ var transformFuncs = map[string]transformFunc{
 	"abs":                newTransformFuncOneArg(transformAbs),
 	"absent":             transformAbsent,
 	"ceil":               newTransformFuncOneArg(transformCeil),
+	"clamp":              transformClamp,
 	"clamp_max":          transformClampMax,
 	"clamp_min":          transformClampMin,
 	"day_of_month":       newTransformFuncDateTime(transformDayOfMonth),
@@ -61,6 +63,7 @@ var transformFuncs = map[string]transformFunc{
 	"minute":             newTransformFuncDateTime(transformMinute),
 	"month":              newTransformFuncDateTime(transformMonth),
 	"round":              transformRound,
+	"sign":               transformSign,
 	"scalar":             transformScalar,
 	"sort":               newTransformFuncSort(false),
 	"sort_desc":          newTransformFuncSort(true),
@@ -215,6 +218,31 @@ func transformCeil(v float64) float64 {
 	return math.Ceil(v)
 }
 
+func transformClamp(tfa *transformFuncArg) ([]*timeseries, error) {
+	args := tfa.args
+	if err := expectTransformArgsNum(args, 3); err != nil {
+		return nil, err
+	}
+	mins, err := getScalar(args[1], 1)
+	if err != nil {
+		return nil, err
+	}
+	maxs, err := getScalar(args[2], 2)
+	if err != nil {
+		return nil, err
+	}
+	tf := func(values []float64) {
+		for i, v := range values {
+			if v > maxs[i] {
+				values[i] = maxs[i]
+			} else if v < mins[i] {
+				values[i] = mins[i]
+			}
+		}
+	}
+	return doTransformValues(args[0], tf, tfa.fe)
+}
+
 func transformClampMax(tfa *transformFuncArg) ([]*timeseries, error) {
 	args := tfa.args
 	if err := expectTransformArgsNum(args, 2); err != nil {
@@ -315,6 +343,11 @@ func transformBucketsLimit(tfa *transformFuncArg) ([]*timeseries, error) {
 	if limit <= 0 {
 		return nil, nil
 	}
+	if limit < 3 {
+		// Preserve the first and the last bucket for better accuracy,
+		// since these buckets are usually `[0...leMin]` and `(leMax ... +Inf]`
+		limit = 3
+	}
 	tss := vmrangeBucketsToLE(args[1])
 	if len(tss) == 0 {
 		return nil, nil
@@ -376,15 +409,18 @@ func transformBucketsLimit(tfa *transformFuncArg) ([]*timeseries, error) {
 			}
 		}
 		for len(leGroup) > limit {
+			// Preserve the first and the last bucket for better accuracy,
+			// since these buckets are usually `[0...leMin]` and `(leMax ... +Inf]`
 			xxMinIdx := 0
-			for i, xx := range leGroup {
+			for i, xx := range leGroup[1 : len(leGroup)-1] {
 				if xx.hits < leGroup[xxMinIdx].hits {
 					xxMinIdx = i
 				}
 			}
+			xxMinIdx++
 			// Merge the leGroup[xxMinIdx] bucket with the smallest adjacent bucket in order to preserve
 			// the maximum accuracy.
-			if xxMinIdx+1 == len(leGroup) || (xxMinIdx > 0 && leGroup[xxMinIdx-1].hits < leGroup[xxMinIdx+1].hits) {
+			if xxMinIdx > 1 && leGroup[xxMinIdx-1].hits < leGroup[xxMinIdx+1].hits {
 				xxMinIdx--
 			}
 			leGroup[xxMinIdx+1].hits += leGroup[xxMinIdx].hits
@@ -550,7 +586,6 @@ func transformHistogramShare(tfa *transformFuncArg) ([]*timeseries, error) {
 	m := groupLeTimeseries(tss)
 
 	// Calculate share for les
-
 	share := func(i int, les []float64, xss []leTimeseries) (q, lower, upper float64) {
 		leReq := les[i]
 		if math.IsNaN(leReq) || len(xss) == 0 {
@@ -1569,6 +1604,25 @@ func transformRound(tfa *transformFuncArg) ([]*timeseries, error) {
 	return doTransformValues(args[0], tf, tfa.fe)
 }
 
+func transformSign(tfa *transformFuncArg) ([]*timeseries, error) {
+	args := tfa.args
+	if err := expectTransformArgsNum(args, 1); err != nil {
+		return nil, err
+	}
+	tf := func(values []float64) {
+		for i, v := range values {
+			sign := float64(0)
+			if v < 0 {
+				sign = -1
+			} else if v > 0 {
+				sign = 1
+			}
+			values[i] = sign
+		}
+	}
+	return doTransformValues(args[0], tf, tfa.fe)
+}
+
 func transformScalar(tfa *transformFuncArg) ([]*timeseries, error) {
 	args := tfa.args
 	if err := expectTransformArgsNum(args, 1); err != nil {
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index 192a75ff5..8ae0db448 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -2,6 +2,7 @@
 
 # tip
 
+* FEATURE: add `sign(q)` and `clamp(q, min, max)` functions, which are planned to be added in [the upcoming Prometheus release](https://twitter.com/roidelapluie/status/1363428376162295811) . The `last_over_time(m[d])` function is already supported in [MetricsQL](https://victoriametrics.github.io/MetricsQL.html).
 * FEATURE: vmagent: add `scrape_align_interval` config option, which can be used for aligning scrapes to the beginning of the configured interval. See [these docs](https://victoriametrics.github.io/vmagent.html#troubleshooting) for details.
 * FEATURE: expose io-related metrics at `/metrics` page for every VictoriaMetrics component:
   * `process_io_read_bytes_total` - the number of bytes read via io syscalls such as read and pread
@@ -10,15 +11,18 @@
   * `process_io_write_syscalls_total` - the number of write syscalls such as write and pwrite
   * `process_io_storage_read_bytes_total` - the number of bytes read from storage layer
   * `process_io_storage_written_bytes_total` - the number of bytes written to storage layer
+* FEATURE: vmagent: use watch API for Kuberntes service discovery. This should reduce load on Kuberntes API server when it tracks big number of objects (for example, 10K pods). This should also reduce the time needed for k8s targets discovery.
 * FEATURE: vmagent: export `vm_promscrape_target_relabel_duration_seconds` metric, which can be used for monitoring the time spend on relabeling for discovered targets.
 * FEATURE: vmagent: optimize [relabeling](https://victoriametrics.github.io/vmagent.html#relabeling) performance for common cases.
 * FEATURE: add `increase_pure(m[d])` function to MetricsQL. It works the same as `increase(m[d])` except of various edge cases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962) for details.
-
+* FEATURE: increase accuracy for `buckets_limit(limit, buckets)` results for small `limit` values. See [MetricsQL docs](https://victoriametrics.github.io/MetricsQL.html) for details.
 
 * BUGFIX: vmagent: properly perform graceful shutdown on `SIGINT` and `SIGTERM` signals. The graceful shutdown has been broken in `v1.54.0`. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1065
 * BUGFIX: reduce the probability of `duplicate time series` errors when querying Kubernetes metrics.
 * BUGFIX: properly calculate `histogram_quantile()` over time series with only a single non-zero bucket with `{le="+Inf"}`. Previously `NaN` was returned, now the value for the last bucket before `{le="+Inf"}` is returned like Prometheus does.
 * BUGFIX: vmselect: do not cache partial query results on timeout when receiving data from `vmstorage` nodes. See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1085
+* BUGFIX: properly handle `stale NFS file handle` error.
+* BUGFIX: properly cache query results when `extra_label` query arg is used. Previously the cached results could clash for different `extra_label` values. See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1095
 
 
 # [v1.54.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.54.1)
diff --git a/docs/CaseStudies.md b/docs/CaseStudies.md
index 99e22456b..c9c9a8e01 100644
--- a/docs/CaseStudies.md
+++ b/docs/CaseStudies.md
@@ -1,9 +1,9 @@
 # Case studies and talks
 
-Below are approved public case studies and talks from VictoriaMetrics users. Join our [community Slack channel](http://slack.victoriametrics.com/)
-and feel free asking for references, reviews and additional case studies from real VictoriaMetrics users there.
+Below please find public case studies and talks from VictoriaMetrics users. You can also join our [community Slack channel](http://slack.victoriametrics.com/)
+where you can chat with VictoriaMetrics users to get additional references, reviews and case studies.
 
-See also [articles about VictoriaMetrics from our users](https://victoriametrics.github.io/Articles.html#third-party-articles-and-slides).
+You can also read [articles about VictoriaMetrics from our users](https://victoriametrics.github.io/Articles.html#third-party-articles-and-slides).
 
 Alphabetically sorted links to case studies:
 
@@ -23,156 +23,185 @@ Alphabetically sorted links to case studies:
 * [zhihu](#zhihu)
 
 
-## zhihu
-
-[zhihu](https://www.zhihu.com) is the largest chinese question-and-answer website. We use VictoriaMetrics to store and use Graphite metrics, and we shared the [promate](https://github.com/zhihu/promate) solution in our [单机 20 亿指标，知乎 Graphite 极致优化！](https://qcon.infoq.cn/2020/shenzhen/presentation/2881)([slides](https://static001.geekbang.org/con/76/pdf/828698018/file/%E5%8D%95%E6%9C%BA%2020%20%E4%BA%BF%E6%8C%87%E6%A0%87%EF%BC%8C%E7%9F%A5%E4%B9%8E%20Graphite%20%E6%9E%81%E8%87%B4%E4%BC%98%E5%8C%96%EF%BC%81-%E7%86%8A%E8%B1%B9.pdf)) talk at [QCon 2020](https://qcon.infoq.cn/2020/shenzhen/).
-
-Numbers:
-
-- Active time series: ~2500 Million
-- Datapoints: ~20 Trillion
-- Ingestion rate: ~1800k/s
-- Disk usage: ~20 TiB
-- Index size: ~600 GiB
-- The average query rate is ~3k per second (mostly alert queries).
-- Query duration: median is ~40ms, 99th percentile is ~100ms.
-
-
 ## adidas
 
-See [slides](https://promcon.io/2019-munich/slides/remote-write-storage-wars.pdf) and [video](https://youtu.be/OsH6gPdxR4s)
+See our [slides](https://promcon.io/2019-munich/slides/remote-write-storage-wars.pdf) and [video](https://youtu.be/OsH6gPdxR4s)
 from [Remote Write Storage Wars](https://promcon.io/2019-munich/talks/remote-write-storage-wars/) talk at [PromCon 2019](https://promcon.io/2019-munich/).
 VictoriaMetrics is compared to Thanos, Corex and M3DB in the talk.
 
+## Adsterra
 
-## CERN
+[Adsterra Network](https://adsterra.com) is a leading digital advertising agency that offers
+performance-based solutions for advertisers and media partners worldwide.
 
-The European Organization for Nuclear Research known as [CERN](https://home.cern/) uses VictoriaMetrics for real-time monitoring
-of the [CMS](https://home.cern/science/experiments/cms) detector system.
-According to [published talk](https://indico.cern.ch/event/877333/contributions/3696707/attachments/1972189/3281133/CMS_mon_RD_for_opInt.pdf)
-VictoriaMetrics is used for the following purposes as a part of "CMS Monitoring cluster":
+We used to collect and store our metrics with Prometheus. Over time, the data volume on our servers
+and metrics increased to the point that we were forced to gradually reduce what we were retaining. When our retention got as low as 7 days
+we looked for alternative solutions. We chose between Thanos, VictoriaMetrics and Prometheus federation.
 
-* As long-term storage for messages consumed from the [NATS messaging system](https://nats.io/). Consumed messages are pushed directly to VictoriaMetrics via HTTP protocol
-* As long-term storage for Prometheus monitoring system (30 days retention policy, there are plans to increase it up to ½ year)
-* As a data source for visualizing metrics in Grafana.
+We ended up with the following configuration:
 
-R&D topic: Evaluate VictoraMetrics vs InfluxDB for large cardinality data.
+- Local instances of Prometheus with VictoriaMetrics as the remote storage on our backend servers.
+- A single Prometheus on our monitoring server scrapes metrics from other servers and writes to VictoriaMetrics.
+- A separate Prometheus that federates from other instances of Prometheus and processes alerts.
 
-See also [The CMS monitoring infrastructure and applications](https://arxiv.org/pdf/2007.03630.pdf) publication from CERN with details about VictoriaMetrics usage.
+We learned that remote write protocol generated too much traffic and connections so after 8 months we started looking for alternatives.
 
+Around the same time, VictoriaMetrics released [vmagent](https://victoriametrics.github.io/vmagent.html).
+We tried to scrape all the metrics via a single instance of vmagent but it that didn't work because vmgent wasn't able to catch up with writes
+into VictoriaMetrics. We tested different options and end up with the following scheme:
 
-## COLOPL
+- We removed Prometheus from our setup.
+- VictoriaMetrics [can scrape targets](https://victoriametrics.github.io/Single-server-VictoriaMetrics.html#how-to-scrape-prometheus-exporters-such-as-node-exporter) as well
+so we removed vmagent. Now, VictoriaMetrics scrapes all the metrics from 110 jobs and 5531 targets.
+- We use [Promxy](https://github.com/jacksontj/promxy) for alerting.
 
-[COLOPL](http://www.colopl.co.jp/en/) is Japaneese Game Development company. It started using VictoriaMetrics
-after evaulating the following remote storage solutions for Prometheus:
+Such a scheme has generated the following benefits compared with Prometheus:
 
-* Cortex
-* Thanos
-* M3DB
-* VictoriaMetrics
+- We can store more metrics.
+- We need less RAM and CPU for the same workload.
 
-See [slides](https://speakerdeck.com/inletorder/monitoring-platform-with-victoria-metrics) and [video](https://www.youtube.com/watch?v=hUpHIluxw80)
-from `Large-scale, super-load system monitoring platform built with VictoriaMetrics` talk at [Prometheus Meetup Tokyo #3](https://prometheus.connpass.com/event/157721/).
+Cons are the following:
 
+- VictoriaMetrics didn't support replication (it [supports replication now](https://victoriametrics.github.io/Cluster-VictoriaMetrics.html#replication-and-data-safety)) - we run an extra instance of VictoriaMetrics and Promxy in front of a VictoriaMetrics pair for high availability.
+- VictoriaMetrics stores 1 extra month for defined retention (if retention is set to N months, then VM stores N+1 months of data), but this is still better than other solutions.
 
-## Zerodha
+Here are some numbers from our single-node VictoriaMetrics setup:
+
+- active time series: 10M
+- ingestion rate: 800K samples/sec
+- total number of datapoints: more than 2 trillion
+- total number of entries in inverted index: more than 1 billion
+- daily time series churn rate: 2.6M
+- data size on disk: 1.5 TB
+- index size on disk: 27 GB
+- average datapoint size on disk: 0.75 bytes
+- range query rate: 16 rps
+- instant query rate: 25 rps
+- range query duration: max: 0.5s; median: 0.05s; 97th percentile: 0.29s
+- instant query duration: max: 2.1s; median: 0.04s; 97th percentile: 0.15s
 
-[Zerodha](https://zerodha.com/) is India's largest stock broker. Monitoring team at Zerodha faced with the following requirements:
+VictoriaMetrics consumes about 50GB of RAM.
 
-* Multiple K8s clusters to monitor
-* Consistent monitoring infra for each cluster across the fleet
-* Ability to handle billions of timeseries events at any point of time
-* Easier to operate and cost effective
+Setup:
 
-Thanos, Cortex and VictoriaMetrics were evaluated as a long-term storage for Prometheus. VictoriaMetrics has been selected due to the following reasons:
+We have 2 single-node instances of VictoriaMetrics. The first instance collects and stores high-resolution metrics (10s scrape interval) for a month.
+The second instance collects and stores low-resolution metrics (300s scrape interval) for a month.
+We use Promxy + Alertmanager for global view and alerts evaluation.
 
-* Blazing fast benchmarks for a single node setup.
-* Single binary mode. Easy to scale vertically, very less operational headache.
-* Considerable [improvements on creating Histograms](https://medium.com/@valyala/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350).
-* [MetricsQL](https://victoriametrics.github.io/MetricsQL.html) gives us the ability to extend PromQL with more aggregation operators.
-* API is compatible with Prometheus, almost all standard PromQL queries just work out of the box.
-* Handles storage well, with periodic compaction. Makes it easy to take snapshots.
 
-See [Monitoring K8S with VictoriaMetrics](https://docs.google.com/presentation/d/1g7yUyVEaAp4tPuRy-MZbPXKqJ1z78_5VKuV841aQfsg/edit) slides,
-[video](https://youtu.be/ZJQYW-cFOms) and [Infrastructure monitoring with Prometheus at Zerodha](https://zerodha.tech/blog/infra-monitoring-at-zerodha/) blog post for more details.
+## ARNES
 
+[The Academic and Research Network of Slovenia](https://www.arnes.si/en/) (ARNES) is a public institute that provides network services to research,
+educational and cultural organizations enabling connections and cooperation with each other and with related organizations worldwide.
 
-## Wix.com
+After using Cacti, Graphite and StatsD for years, we wanted to upgrade our monitoring stack to something that:
 
-[Wix.com](https://en.wikipedia.org/wiki/Wix.com) is the leading web development platform.
+- has native alerting support
+- can be run on-prem
+- has multi-dimensional metrics
+- has lower hardware requirements
+- is scalable
+- has a simple client that allows for provisioning and discovery with Puppet
 
-> We needed to redesign metric infrastructure from the ground up after the move to Kubernethes. A few approaches/designs have been tried before the one that works great has been chosen: Prometheus instance in every datacenter with 2 hours retention for local storage and remote write into [HA pair of single-node VictoriaMetrics instances](https://victoriametrics.github.io/Single-server-VictoriaMetrics.html#high-availability).
+We hed been running Prometheus for about a year in a test environment and it was working well but there was a need/wish for a few more years of retention than the old system provided. We tested Thanos which was a bit resource hungry but worked great for about half a year.
+Then we discovered VictoriaMetrics. Our scale isn't that big so we don't have on-prem S3 and no Kubernetes. VM's single node instance provided
+the same result with far less maintenance overhead and lower hardware requirements.
+
+After testing it a few months and with great support from the maintainers on [Slack](http://slack.victoriametrics.com/),
+we decided to go with it. VM's support for the ingestion of InfluxDB metrics was an additional bonus as our hardware team uses
+SNMPCollector to collect metrics from network devices and switching from InfluxDB to VictoriaMetrics required just a simple change in the config file. 
 
 Numbers:
 
-* The number of active time series per VictoriaMetrics instance is 50 millions.
-* The total number of time series per VictoriaMetrics instance is 5000 millions.
-* Ingestion rate per VictoriaMetrics instance is 1.1 millions data points per second.
-* The total number of datapoints per VictoriaMetrics instance is 8.5 trillions.
-* The average churn rate is 150 millions new time series per day.
-* The average query rate is ~150 per second (mostly alert queries).
-* Query duration: median is ~1ms, 99th percentile is ~1sec.
-* Retention: 3 months.
+- 2 single node instances per DC (one for Prometheus and one for InfluxDB metrics)
+- Active time series per VictoriaMetrics instance: ~500k (Prometheus) + ~320k (InfluxDB)
+- Ingestion rate per VictoriaMetrics instance: 45k/s (Prometheus) / 30k/s (InfluxDB)
+- Query duration: median ~5ms, 99th percentile ~45ms
+- Total number of datapoints per instance: 390B (Prometheus), 110B (InfluxDB)
+- Average datapoint size on drive: 0.4 bytes
+- Disk usage per VictoriaMetrics instance: 125GB (Prometheus), 185GB (InfluxDB)
+- Index size per VictoriaMetrics instance: 1.6GB (Prometheus), 1.2GB (InfluxDB)
 
-> Alternatives that we’ve played with before choosing VictoriaMetrics are: federated Prometheus, Cortex, IronDB and Thanos.
-> Points that were critical to us when we were choosing a central tsdb, in order of importance:
+We are running 1 Prometheus, 1 VictoriaMetrics and 1 Grafana server in each datacenter on baremetal servers, scraping 350+ targets
+(and 3k+ devices collected via SNMPCollector sending metrics directly to VM). Each Prometheus is scraping all targets
+so we have all metrics in both VictoriaMetrics instances. We are using [Promxy](https://github.com/jacksontj/promxy) to deduplicate metrics from both instances.
+Grafana has an LB infront so if one DC has problems we can still view all metrics from both DCs on the other Grafana instance.
 
-* At least 3 month worth of history.
-* Raw data, no aggregation, no sampling.
-* High query speed.
-* Clean fail state for HA (multi-node clusters may return partial data resulting in false alerts).
-* Enough head room/scaling capacity for future growth, up to 100M active time series.
-* Ability to split DB replicas per workload. Alert queries go to one replica, user queries go to another (speed for users, effective cache).
+We are still in the process of migration, but we are really happy with the whole stack. It has proven to be an essential tool
+for gathering insights into our services during COVID-19 and has enabled us to provide better service and identify problems faster.
 
-> Optimizing for those points and our specific workload VictoriaMetrics proved to be the best option. As an icing on a cake we’ve got [PromQL extensions](https://victoriametrics.github.io/MetricsQL.html) - `default 0` and `histogram` are my favorite ones, for example. What we specially like is having a lot of tsdb params easily available via config options, that makes tsdb easy to tune for specific use case. Also worth noting is a great community in [Slack channel](http://slack.victoriametrics.com/) and of course maintainer support.
+## Brandwatch
 
-Alex Ulstein, Head of Monitoring, Wix.com
+[Brandwatch](https://www.brandwatch.com/) is the world's pioneering digital consumer intelligence suite,
+helping over 2,000 of the world's most admired brands and agencies to make insightful, data-driven business decisions.
 
+The engineering department at Brandwatch has been using InfluxDB to store application metrics for many years
+but when End-of-Life of InfluxDB version 1.x was announced we decided to re-evaluate our entire metrics collection and storage stack.
 
-## Wedos.com
+The main goals for the new metrics stack were:
+- improved performance
+- lower maintenance
+- support for native clustering in open source version
+- the less metrics shipment had to change, the better
+- longer data retention time period would be great but not critical
 
-> [Wedos](https://www.wedos.com/) is the Biggest Czech Hosting. We have our own private data center, that holds only our servers and technologies. The second data center, where the servers will be cooled in an oil bath, is being built. We started using [cluster VictoriaMetrics](https://victoriametrics.github.io/Cluster-VictoriaMetrics.html) to store Prometheus metrics from all our infrastructure after receiving positive references from our friends who successfully use VictoriaMetrics.
+We initially tested CrateDB and TimescaleDB wand found that both had limitations or requirements in their open source versions
+that made them unfit for our use case. Prometheus was also considered but it's push vs. pull metrics was a big change we did not want
+to include in the already significant change.
 
-Numbers:
+Once we found VictoriaMetrics it solved the following problems:
+- it is very lightweight and we can now run virtual machines instead of dedicated hardware machines for metrics storage
+- very short startup time and any possible gaps in data can easily be filled in using Promxy
+- we could continue using Telegraf as our metrics agent and ship identical metrics to both InfluxDB and VictoriaMetrics during the migration period (migration just about to start)
+- compression im VM is really good. We can store more metrics and we can easily spin up new VictoriaMetrics instances
+for new data and keep read-only nodes with older data if we need to extend our retention period further
+than single virtual machine disks allow and we can aggregate all the data from VictoriaMetrics with Promxy
 
-* The number of acitve time series: 5M.
-* Ingestion rate: 170K data points per second.
-* Query duration: median is ~2ms, 99th percentile is ~50ms.
+High availability is done the same way we did with InfluxDB by running parallel single nodes of VictoriaMetrics.
 
-> We like configuration simplicity and zero maintenance for VictoriaMetrics - once installed and forgot about it. It works out of the box without any issues.
+Numbers:
 
+- active time series: up to 25 million
+- ingestion rate: ~300 000
+- total number of datapoints: 380 billion and growing
+- total number of entries in inverted index: 575 million and growing
+- daily time series churn rate: ~550 000
+- data size on disk: ~660GB and growing
+- index size on disk: ~9,3GB and growing
+- average datapoint size on disk: ~1.75 bytes
 
-## Synthesio
+Query rates are insignificant as we have concentrated on data ingestion so far.
 
-[Synthesio](https://www.synthesio.com/) is the leading social intelligence tool for social media monitoring & social analytics.
+Anders Bomberg, Monitoring and Infrastructure Team Lead, brandwatch.com
 
-> We fully migrated from [Metrictank](https://grafana.com/oss/metrictank/)  to Victoria Metrics
+## CERN
 
-Numbers:
-- Single node
-- Active time series - 5 Million
-- Datapoints: 1.25 Trillion
-- Ingestion rate - 550k datapoints per second
-- Disk usage - 150gb
-- Index size - 3gb
-- Query duration 99th percentile - 147ms
-- Churn rate - 100 new time series per hour
+The European Organization for Nuclear Research better known as [CERN](https://home.cern/) uses VictoriaMetrics for real-time monitoring
+of the [CMS](https://home.cern/science/experiments/cms) detector system.
+According to [published talk](https://indico.cern.ch/event/877333/contributions/3696707/attachments/1972189/3281133/CMS_mon_RD_for_opInt.pdf)
+VictoriaMetrics is used for the following purposes as a part of the "CMS Monitoring cluster":
 
+* As a long-term storage for messages ingested from the [NATS messaging system](https://nats.io/). Ingested messages are pushed directly to VictoriaMetrics via HTTP protocol
+* As a long-term storage for Prometheus monitoring system (30 days retention policy. There are plans to increase it up to ½ year)
+* As a data source for visualizing metrics in Grafana.
 
-## MHI Vestas Offshore Wind
+R&D topic: Evaluate VictoraMetrics vs InfluxDB for large cardinality data.
 
-The mission of [MHI Vestas Offshore Wind](http://www.mhivestasoffshore.com) is to co-develop offshore wind as an economically viable and sustainable energy resource to benefit future generations.
+Please also see [The CMS monitoring infrastructure and applications](https://arxiv.org/pdf/2007.03630.pdf) publication from CERN with details about their VictoriaMetrics usage.
 
-MHI Vestas Offshore Wind is using VictoriaMetrics to ingest and visualize sensor data from offshore wind turbines. The very efficient storage and ability to backfill was key in chosing VictoriaMetrics. MHI Vestas Offshore Wind is running the cluster version of VictoriaMetrics on Kubernetes using the Helm charts for deployment to be able to scale up capacity as the solution will be rolled out.
 
-Numbers with current limited roll out:
+## COLOPL
 
-- Active time series: 270K
-- Ingestion rate: 70K/sec
-- Total number of datapoints: 850 billions
-- Data size on disk: 800 GiB
-- Retention time: 3 years
+[COLOPL](http://www.colopl.co.jp/en/) is Japanese game development company. It started using VictoriaMetrics
+after evaulating the following remote storage solutions for Prometheus:
+
+* Cortex
+* Thanos
+* M3DB
+* VictoriaMetrics
 
+See [slides](https://speakerdeck.com/inletorder/monitoring-platform-with-victoria-metrics) and [video](https://www.youtube.com/watch?v=hUpHIluxw80)
+from `Large-scale, super-load system monitoring platform built with VictoriaMetrics` talk at [Prometheus Meetup Tokyo #3](https://prometheus.connpass.com/event/157721/).
 
 ## Dreamteam
 
@@ -182,175 +211,137 @@ Numbers:
 
 * Active time series: from 350K to 725K.
 * Total number of time series: from 100M to 320M.
-* Total number of datapoints: from 120 billions to 155 billions.
-* Retention: 3 months.
+* Total number of datapoints: from 120 billion to 155 billion.
+* Retention period: 3 months.
 
 VictoriaMetrics in production environment runs on 2 M5 EC2 instances in "HA" mode, managed by Terraform and Ansible TF module.
 2 Prometheus instances are writing to both VMs, with 2 [Promxy](https://github.com/jacksontj/promxy) replicas
-as load balancer for reads.
-
-
-## Brandwatch
-
-[Brandwatch](https://www.brandwatch.com/) is the world's pioneering digital consumer intelligence suite,
-helping over 2,000 of the world's most admired brands and agencies to make insightful, data-driven business decisions.
-
-The engineering department at Brandwatch has been using InfluxDB for storing application metrics for many years
-and when End-of-Life of InfluxDB version 1.x was announced we decided to re-evaluate our whole metrics collection and storage stack.
-
-Main goals for the new metrics stack were:
-- improved performance
-- lower maintenance
-- support for native clustering in open source version
-- the less metrics shipment had to change, the better
-- achieving longer data retention would be great but not critical
+as the load balancer for reads.
 
-We initially looked at CrateDB and TimescaleDB which both turned out to have limitations or requirements in the open source versions
-that made them unfit for our use case. Prometheus was also considered but push vs. pull metrics was a big change we did not want
-to include in the already significant change.
+## Idealo.de
 
-Once we found VictoriaMetrics it solved the following problems:
-- it is very light weight and we can now run virtual machines instead of dedicated hardware machines for metrics storage
-- very short startup time and any possible gaps in data can easily be filled in by using Promxy
-- we could continue using Telegraf as our metrics agent and ship identical metrics to both InfluxDB and VictoriaMetrics during a migration period (migration just about to start)
-- compression is really good so we can store more metrics and we can spin up new VictoriaMetrics instances
-for new data and keep read-only nodes with older data if we need to extend our retention period further
-than single virtual machine disks allow and we can aggregate all the data from VictoriaMetrics with Promxy
+[idealo.de](https://www.idealo.de/) is the leading price comparison website in Germany. We use Prometheus for metrics on our container platform.
+When we introduced Prometheus at idealo we started with m3db as our longterm storage. In our setup, m3db was quite unstable and consumed a lot of resources.
 
-High availability is done the same way we did with InfluxDB, by running parallel single nodes of VictoriaMetrics.
+VictoriaMetrics in poroduction is very stable for us and uses only a fraction of the resources even though we also increased our retention period from 1 month to 13 months.
 
 Numbers:
 
-- active time series: up to 25 million
-- ingestion rate: ~300 000
-- total number of datapoints: 380 billion and growing
-- total number of entries in inverted index: 575 million and growing
-- daily time series churn rate: ~550 000
-- data size on disk: ~660GB and growing
-- index size on disk: ~9,3GB and growing
-- average datapoint size on disk: ~1.75 bytes
-
-Query rates are insignificant as we have concentrated on data ingestion so far.
+- The number of active time series per VictoriaMetrics instance is 21M.
+- Total ingestion rate 120k metrics per second.
+- The total number of datapoints 3.1 trillion.
+- The average time series churn rate is ~9M per day.
+- The average query rate is ~20 per second. Response time for 99th quantile is 120ms.
+- Retention: 13 months.
+- Size of all datapoints: 3.5 TB
 
-Anders Bomberg, Monitoring and Infrastructure Team Lead, brandwatch.com
 
+## MHI Vestas Offshore Wind
 
-## Adsterra
+The mission of [MHI Vestas Offshore Wind](http://www.mhivestasoffshore.com) is to co-develop offshore wind as an economically viable and sustainable energy resource to benefit future generations.
 
-[Adsterra Network](https://adsterra.com) is a leading digital advertising company that offers
-performance-based solutions for advertisers and media partners worldwide.
+MHI Vestas Offshore Wind is using VictoriaMetrics to ingest and visualize sensor data from offshore wind turbines. The very efficient storage and ability to backfill was key in choosing VictoriaMetrics. MHI Vestas Offshore Wind is running the cluster version of VictoriaMetrics on Kubernetes using the Helm charts for deployment to be able to scale up capacity as the solution is rolled out.
 
-We used to collect and store our metrics via Prometheus. Over time the amount of our servers
-and metrics increased so we were gradually reducing the retention. When retention became 7 days
-we started to look for alternative solutions. We were choosing among Thanos, VictoriaMetrics and Prometheus federation.
+Numbers with current, limited roll out:
 
-We end up with the following configuration:
+- Active time series: 270K
+- Ingestion rate: 70K/sec
+- Total number of datapoints: 850 billion
+- Data size on disk: 800 GiB
+- Retention period: 3 years
 
-- Local Prometheus'es with VictoriaMetrics as remote storage on our backend servers.
-- A single Prometheus on our monitoring server scrapes metrics from other servers and writes to VictoriaMetrics.
-- A separate Prometheus that federates from other Prometheus'es and processes alerts.
 
-Turns out that remote write protocol generates too much traffic and connections. So after 8 months we started to look for alternatives.
+## Synthesio
 
-Around the same time VictoriaMetrics released [vmagent](https://victoriametrics.github.io/vmagent.html).
-We tried to scrape all the metrics via a single insance of vmagent. But that didn't work - vmgent wasn't able to catch up with writes
-into VictoriaMetrics. We tested different options and end up with the following scheme:
+[Synthesio](https://www.synthesio.com/) is the leading social intelligence tool for social media monitoring and analytics.
 
-- We removed Prometheus from our setup.
-- VictoriaMetrics [can scrape targets](https://victoriametrics.github.io/Single-server-VictoriaMetrics.html#how-to-scrape-prometheus-exporters-such-as-node-exporter) as well,
-so we removed vmagent. Now VictoriaMetrics scrapes all the metrics from 110 jobs and 5531 targets.
-- We use [Promxy](https://github.com/jacksontj/promxy) for alerting.
+> We fully migrated from [Metrictank](https://grafana.com/oss/metrictank/)  to VictoriaMetrics
 
-Such a scheme has the following benefits comparing to Prometheus:
+Numbers:
+- Single node
+- Active time series - 5 Million
+- Datapoints: 1.25 Trillion
+- Ingestion rate - 550k datapoints per second
+- Disk usage - 150gb
+- Index size - 3gb
+- Query duration 99th percentile - 147ms
+- Churn rate - 100 new time series per hour
 
-- We can store more metrics.
-- We need less RAM and CPU for the same workload.
+## Wedos.com
 
-Cons are the following:
+> [Wedos](https://www.wedos.com/) is the biggest hosting provider in the Czech Republic. We have our own private data center that holds our servers and technologies. We are in the process of building a second, stae of the art data center where the servers will be cooled in an oil bath. We started using [cluster VictoriaMetrics](https://victoriametrics.github.io/Cluster-VictoriaMetrics.html) to store Prometheus metrics from all our infrastructure after receiving positive references from people who had successfully used VictoriaMetrics.
 
-- VictoriaMetrics didn't support replication (it [supports replication now](https://victoriametrics.github.io/Cluster-VictoriaMetrics.html#replication-and-data-safety)) - we run extra instance of VictoriaMetrics and Promxy in front of VictoriaMetrics pair for high availability.
-- VictoriaMetrics stores 1 extra month for defined retention (if retention is set to N months, then VM stores N+1 months of data), but this is still better than other solutions.
+Numbers:
 
-Some numbers from our single-node VictoriaMetrics setup:
+* The number of acitve time series: 5M.
+* Ingestion rate: 170K data points per second.
+* Query duration: median is ~2ms, 99th percentile is ~50ms.
 
-- active time series: 10M
-- ingestion rate: 800K samples/sec
-- total number of datapoints: more than 2 trillion
-- total number of entries in inverted index: more than 1 billion
-- daily time series churn rate: 2.6M
-- data size on disk: 1.5 TB
-- index size on disk: 27 GB
-- average datapoint size on disk: 0.75 bytes
-- range query rate: 16 rps
-- instant query rate: 25 rps
-- range query duration: max: 0.5s; median: 0.05s; 97th percentile: 0.29s
-- instant query duration: max: 2.1s; median: 0.04s; 97th percentile: 0.15s
+> We like that VictoriaMetrics is simple to configuree and requires zero maintenance. It works right out of the box and once it's set up you can just forget about it. 
 
-VictoriaMetrics consumes about 50GiB of RAM.
+## Wix.com
 
-Setup:
+[Wix.com](https://en.wikipedia.org/wiki/Wix.com) is the leading web development platform.
 
-We have 2 single-node instances of VictoriaMetircs. The first instance collects and stores high-resolution metrics (10s scrape interval) for a month.
-The second instance collects and stores low-resolution metrics (300s scrape interval) for a month.
-We use Promxy + Alertmanager for global view and alerts evaluation.
+> We needed to redesign our metrics infrastructure from the ground up after the move to Kubernetes. We had tried out a few different options before landing on this solution which is working great. We have a Prometheus instance in every datacenter with 2 hours retention for local storage and remote write into [HA pair of single-node VictoriaMetrics instances](https://victoriametrics.github.io/Single-server-VictoriaMetrics.html#high-availability).
 
+Numbers:
 
-## ARNES
+* The number of active time series per VictoriaMetrics instance is 50 millios.
+* The total number of time series per VictoriaMetrics instance is 5000 million.
+* Ingestion rate per VictoriaMetrics instance is 1.1 millions data points per second.
+* The total number of datapoints per VictoriaMetrics instance is 8.5 trillion.
+* The average churn rate is 150 millions new time series per day.
+* The average query rate is ~150 per second (mostly alert queries).
+* Query duration: median is ~1ms, 99th percentile is ~1sec.
+* Retention period: 3 months.
 
-[The Academic and Research Network of Slovenia](https://www.arnes.si/en/) (ARNES) is a public institute that provides network services to research,
-educational and cultural organizations, and enables them to establish connections and cooperation with each other and with related organizations abroad.
+> The alternatives that we tested prior to choosing VictoriaMetrics were: Prometheus federated, Cortex, IronDB and Thanos.
+> The items that were critical to us central tsdb, in order of importance were as follows:
 
-After using Cacti, Graphite and StatsD for years, we wanted to upgrade our monitoring stack to something that:
+* At least 3 month worth of retention.
+* Raw data, no aggregation, no sampling.
+* High query speed.
+* Clean fail state for HA (multi-node clusters may return partial data resulting in false alerts).
+* Enough headroom/scaling capacity for future growth which is planned to be up to 100M active time series.
+* Ability to split DB replicas per workload. Alert queries go to one replica and user queries go to another (speed for users, effective cache).
 
-- has native alerting support
-- can run on-prem
-- has multi-dimension metrics
-- lower hardware requirements
-- is scalable
-- simple client provisioning and discovery with Puppet
+> Optimizing for those points and our specific workload, VictoriaMetrics proved to be the best option. As icing on the cake we’ve got [PromQL extensions](https://victoriametrics.github.io/MetricsQL.html) - `default 0` and `histogram` are my favorite ones. We really like having a lot of tsdb params easily available via config options which makes tsdb easy to tune for each specific use case. We've also found a great community in [Slack channel](http://slack.victoriametrics.com/) and responsive and helpful maintainer support.
 
-We were running Prometheus for about a year in a test environment and it worked great. But there was a need/wish for a few years of retention time,
-like the old systems provided. We tested Thanos, which was a bit resource hungry back then, but it worked great for about half a year
-until we discovered VictoriaMetrics. As our scale is not that big, we don't have on-prem S3 and no Kubernetes, VM's single node instance provided
-the same result with less maintenance overhead and lower hardware requirements.
+Alex Ulstein, Head of Monitoring, Wix.com
 
-After testing it a few months and having great support from the maintainers on [Slack](http://slack.victoriametrics.com/),
-we decided to go with it. VM's support for ingesting InfluxDB metrics was an additional bonus, since our hardware team uses
-SNMPCollector to collect metrics from network devices and switching from InfluxDB to VictoriaMetrics was a simple change in the config file for them. 
+## Zerodha
 
-Numbers:
+[Zerodha](https://zerodha.com/) is India's largest stock broker. The monitoring team at Zerodha had the following requirements:
 
-- 2 single node instances per DC (one for prometheus and one for influxdb metrics)
-- Active time series per VictoriaMetrics instance: ~500k (prometheus) + ~320k (influxdb)
-- Ingestion rate per VictoriaMetrics instance: 45k/s (prometheus) / 30k/s (influxdb)
-- Query duration: median is ~5ms, 99th percentile is ~45ms
-- Total number of datapoints per instance: 390B (prometheus), 110B (influxdb)
-- Average datapoint size on drive: 0.4 bytes
-- Disk usage per VictoriaMetrics instance: 125GB (prometheus), 185GB (influxdb)
-- Index size per VictoriaMetrics instance: 1.6GB (prometheus), 1.2GB (influcdb)
+* Multiple K8s clusters to monitor
+* Consistent monitoring infra for each cluster across the fleet
+* The ability to handle billions of timeseries events at any point of time
+* Easy to operate and cost effective
 
-We are running 1 Prometheus, 1 VictoriaMetrics and 1 Grafana server in each datacenter on baremetal servers, scraping 350+ targets
-(and 3k+ devices collected via SNMPCollector sending metrics directly to VM). Each Prometheus is scraping all targets,
-so we have all metrics in both VictoriaMetrics instances. We are using [Promxy](https://github.com/jacksontj/promxy) to deduplicate metrics from both instances.
-Grafana has a LB infront, so if one DC has problems, we can still view all metrics from both DCs on the other Grafana instance.
+Thanos, Cortex and VictoriaMetrics were evaluated as a long-term storage for Prometheus. VictoriaMetrics has been selected for the following reasons:
 
-We are still in the process of migration, but we are really happy with the whole stack. It has proven as an essential piece
-for insight into our services during COVID-19 and has enabled us to provide better service and spot problems faster.
+* Blazingly fast benchmarks for a single node setup.
+* Single binary mode. Easy to scale vertically with far fewer operational headaches.
+* Considerable [improvements on creating Histograms](https://medium.com/@valyala/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350).
+* [MetricsQL](https://victoriametrics.github.io/MetricsQL.html) gives us the ability to extend PromQL with more aggregation operators.
+* The API is compatible with Prometheus and nearly all standard PromQL queries work well out of the box.
+* Handles storage well, with periodic compaction which makes it easy to take snapshots.
 
+Please see [Monitoring K8S with VictoriaMetrics](https://docs.google.com/presentation/d/1g7yUyVEaAp4tPuRy-MZbPXKqJ1z78_5VKuV841aQfsg/edit) slides,
+[video](https://youtu.be/ZJQYW-cFOms) and [Infrastructure monitoring with Prometheus at Zerodha](https://zerodha.tech/blog/infra-monitoring-at-zerodha/) blog post for more details.
 
-## Idealo.de
 
-[idealo.de](https://www.idealo.de/) is the leading price comparison website in Germany. We use Prometheus for metrics on our container platform.
-When we introduced Prometheus at idealo we started with m3db as a longterm storage. In our setup m3db was quite unstable and consumed a lot of resources.
+## zhihu
 
-VictoriaMetrics runs very stable for us and uses only a fraction of the resources. Although we also increased our retention time from 1 month to 13 months.
+[zhihu](https://www.zhihu.com) is the largest Chinese question-and-answer website. We use VictoriaMetrics to store and use Graphite metrics. We shared the [promate](https://github.com/zhihu/promate) solution in our [单机 20 亿指标，知乎 Graphite 极致优化！](https://qcon.infoq.cn/2020/shenzhen/presentation/2881)([slides](https://static001.geekbang.org/con/76/pdf/828698018/file/%E5%8D%95%E6%9C%BA%2020%20%E4%BA%BF%E6%8C%87%E6%A0%87%EF%BC%8C%E7%9F%A5%E4%B9%8E%20Graphite%20%E6%9E%81%E8%87%B4%E4%BC%98%E5%8C%96%EF%BC%81-%E7%86%8A%E8%B1%B9.pdf)) talk at [QCon 2020](https://qcon.infoq.cn/2020/shenzhen/).
 
 Numbers:
 
-- The number of active time series per VictoriaMetrics instance is 21M.
-- Total ingestion rate 120k metrics per second.
-- The total number of datapoints 3.1 trillion.
-- The average time series churn rate is ~9M per day.
-- The average query rate is ~20 per second. Response time for 99th quantile is 120ms.
-- Retention: 13 months.
-- Size of all datapoints: 3.5 TB
+- Active time series: ~2500 Million
+- Datapoints: ~20 Trillion
+- Ingestion rate: ~1800k/s
+- Disk usage: ~20 TB
+- Index size: ~600 GB
+- The average query rate is ~3k per second (mostly alert queries).
+- Query duration: median is ~40ms, 99th percentile is ~100ms.
diff --git a/docs/Quick-Start.md b/docs/Quick-Start.md
index df49809d3..e2f0f085c 100644
--- a/docs/Quick-Start.md
+++ b/docs/Quick-Start.md
@@ -1,31 +1,31 @@
 # Quick Start
 
-1. If you run Ubuntu, then just run `snap install victoriametrics` command in order to install and start VictoriaMetrics, then read [these docs](https://snapcraft.io/victoriametrics).
-   Otherwise download the latest VictoriaMetrics release from [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases),
-   from [Docker hub](https://hub.docker.com/r/victoriametrics/victoria-metrics/)
-   or [build it from sources](https://victoriametrics.github.io/Single-server-VictoriaMetrics.html#how-to-build-from-sources).
+1. If you run Ubuntu please run the `snap install victoriametrics` command to install and start VictoriaMetrics. Then read [these docs](https://snapcraft.io/victoriametrics).
+   Otherwise you can download the latest VictoriaMetrics release from [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases),
+   or [Docker hub](https://hub.docker.com/r/victoriametrics/victoria-metrics/)
+   or [build it from sources](https://victoriametrics.github.io/Single-server-VictoriaMetrics.html#how-to-build-from-sources). 
 
 2. This step isn't needed if you run VictoriaMetrics via `snap install victoriametrics` as described above.
-   Otherwise run the binary or Docker image with the desired command-line flags. Pass `-help` in order to see description for all the available flags
-   and their default values. Default flag values should fit the majoirty of cases. The minimum required flags to configure are:
+   Otherwise, please run the binary or Docker image with your desired command-line flags. You can look at `-help` to see descriptions of all available flags
+   and their default values. The default flag values should fit the majority of cases. The minimum required flags that must be configured are:
 
-   * `-storageDataPath` - path to directory where VictoriaMetrics stores all the data.
+   * `-storageDataPath` - the path to directory where VictoriaMetrics stores your data.
    * `-retentionPeriod` - data retention.
 
-   For instance:
+   For example:
 
    `./victoria-metrics-prod -storageDataPath=/var/lib/victoria-metrics-data -retentionPeriod=3`
 
-   See [these instructions](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/43) in order to configure VictoriaMetrics as OS service.
-   It is recommended setting up [VictoriaMetrics monitoring](https://victoriametrics.github.io/Single-server-VictoriaMetrics.html#monitoring).
+   Check [these instructions](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/43) to configure VictoriaMetrics as an OS service.
+   We recommended setting up [VictoriaMetrics monitoring](https://victoriametrics.github.io/Single-server-VictoriaMetrics.html#monitoring).
 
-3. Configure [vmagent](https://victoriametrics.github.io/vmagent.html) or Prometheus to write data to VictoriaMetrics.
-   It is recommended to use `vmagent` instead of Prometheus, since it is more resource efficient. If you still prefer Prometheus, then
+3. Configure either [vmagent](https://victoriametrics.github.io/vmagent.html) or Prometheus to write data to VictoriaMetrics.
+   We recommended using `vmagent` instead of Prometheus because it is more resource efficient. If you still prefer Prometheus
    see [these instructions](https://victoriametrics.github.io/Single-server-VictoriaMetrics.html#prometheus-setup)
-   for details on how to configure Prometheus.
+   for details on how it may be properly configured.
 
-4. Configure Grafana to query VictoriaMetrics instead of Prometheus.
-   See [these instructions](https://victoriametrics.github.io/Single-server-VictoriaMetrics.html#grafana-setup).
+4. To configure Grafana to query VictoriaMetrics instead of Prometheus
+   please see [these instructions](https://victoriametrics.github.io/Single-server-VictoriaMetrics.html#grafana-setup).
 
 
 There is also [cluster version](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster) and [SaaS playground](https://play.victoriametrics.com/signIn).
diff --git a/go.mod b/go.mod
index fe007c0dd..fa4503144 100644
--- a/go.mod
+++ b/go.mod
@@ -9,7 +9,7 @@ require (
 	// like https://github.com/valyala/fasthttp/commit/996610f021ff45fdc98c2ce7884d5fa4e7f9199b
 	github.com/VictoriaMetrics/fasthttp v1.0.12
 	github.com/VictoriaMetrics/metrics v1.15.0
-	github.com/VictoriaMetrics/metricsql v0.11.0
+	github.com/VictoriaMetrics/metricsql v0.12.0
 	github.com/aws/aws-sdk-go v1.37.12
 	github.com/cespare/xxhash/v2 v2.1.1
 	github.com/cheggaaa/pb/v3 v3.0.6
diff --git a/go.sum b/go.sum
index 343c504a0..8dfcad85f 100644
--- a/go.sum
+++ b/go.sum
@@ -87,8 +87,8 @@ github.com/VictoriaMetrics/fasthttp v1.0.12/go.mod h1:3SeUL4zwB/p/a9aEeRc6gdlbrt
 github.com/VictoriaMetrics/metrics v1.12.2/go.mod h1:Z1tSfPfngDn12bTfZSCqArT3OPY3u88J12hSoOhuiRE=
 github.com/VictoriaMetrics/metrics v1.15.0 h1:HGmGaILioC4vNk6UhkcwLIaDlg5y4MVganq1verl5js=
 github.com/VictoriaMetrics/metrics v1.15.0/go.mod h1:Z1tSfPfngDn12bTfZSCqArT3OPY3u88J12hSoOhuiRE=
-github.com/VictoriaMetrics/metricsql v0.11.0 h1:85zbY8NxWNALppctOAkWfLDC7dDFcvTEn5IHMKwOGag=
-github.com/VictoriaMetrics/metricsql v0.11.0/go.mod h1:ylO7YITho/Iw6P71oEaGyHbO94bGoGtzWfLGqFhMIg8=
+github.com/VictoriaMetrics/metricsql v0.12.0 h1:NMIu0MPBmGP34g4RUjI1U0xW5XYp7IVNXe9KtZI3PFQ=
+github.com/VictoriaMetrics/metricsql v0.12.0/go.mod h1:ylO7YITho/Iw6P71oEaGyHbO94bGoGtzWfLGqFhMIg8=
 github.com/VividCortex/ewma v1.1.1 h1:MnEK4VOv6n0RSY4vtRe3h11qjxL3+t0B8yOL8iMXdcM=
 github.com/VividCortex/ewma v1.1.1/go.mod h1:2Tkkvm3sRDVXaiyucHiACn4cqf7DpdyLvmxzcbUokwA=
 github.com/VividCortex/gohistogram v1.0.0/go.mod h1:Pf5mBqqDxYaXu3hDrrU+w6nw50o/4+TcAqDqk/vUH7g=
diff --git a/lib/fs/dir_remover.go b/lib/fs/dir_remover.go
index aefcd77eb..fae5701bb 100644
--- a/lib/fs/dir_remover.go
+++ b/lib/fs/dir_remover.go
@@ -13,7 +13,7 @@ import (
 
 func mustRemoveAll(path string, done func()) bool {
 	err := os.RemoveAll(path)
-	if err == nil {
+	if err == nil || isStaleNFSFileHandleError(err) {
 		// Make sure the parent directory doesn't contain references
 		// to the current directory.
 		mustSyncParentDirIfExists(path)
@@ -87,6 +87,11 @@ func dirRemover() {
 	}
 }
 
+func isStaleNFSFileHandleError(err error) bool {
+	errStr := err.Error()
+	return strings.Contains(errStr, "stale NFS file handle")
+}
+
 func isTemporaryNFSError(err error) bool {
 	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/61 for details.
 	errStr := err.Error()
diff --git a/lib/fs/fs.go b/lib/fs/fs.go
index 229e0266b..8f8c65943 100644
--- a/lib/fs/fs.go
+++ b/lib/fs/fs.go
@@ -13,26 +13,15 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/filestream"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
-	"golang.org/x/sys/unix"
 )
 
+var tmpFileNum uint64
+
 // MustSyncPath syncs contents of the given path.
 func MustSyncPath(path string) {
-	d, err := os.Open(path)
-	if err != nil {
-		logger.Panicf("FATAL: cannot open %q: %s", path, err)
-	}
-	if err := d.Sync(); err != nil {
-		_ = d.Close()
-		logger.Panicf("FATAL: cannot flush %q to storage: %s", path, err)
-	}
-	if err := d.Close(); err != nil {
-		logger.Panicf("FATAL: cannot close %q: %s", path, err)
-	}
+	mustSyncPath(path)
 }
 
-var tmpFileNum uint64
-
 // WriteFileAtomically atomically writes data to the given file path.
 //
 // WriteFileAtomically returns only after the file is fully written and synced
@@ -332,15 +321,7 @@ func MustWriteData(w io.Writer, data []byte) {
 // CreateFlockFile creates flock.lock file in the directory dir
 // and returns the handler to the file.
 func CreateFlockFile(dir string) (*os.File, error) {
-	flockFile := dir + "/flock.lock"
-	flockF, err := os.Create(flockFile)
-	if err != nil {
-		return nil, fmt.Errorf("cannot create lock file %q: %w", flockFile, err)
-	}
-	if err := unix.Flock(int(flockF.Fd()), unix.LOCK_EX|unix.LOCK_NB); err != nil {
-		return nil, fmt.Errorf("cannot acquire lock on file %q: %w", flockFile, err)
-	}
-	return flockF, nil
+	return createFlockFile(dir)
 }
 
 // MustGetFreeSpace returns free space for the given directory path.
@@ -372,18 +353,3 @@ type freeSpaceEntry struct {
 	updateTime uint64
 	freeSpace  uint64
 }
-
-func mustGetFreeSpace(path string) uint64 {
-	d, err := os.Open(path)
-	if err != nil {
-		logger.Panicf("FATAL: cannot determine free disk space on %q: %s", path, err)
-	}
-	defer MustClose(d)
-
-	fd := d.Fd()
-	var stat unix.Statfs_t
-	if err := unix.Fstatfs(int(fd), &stat); err != nil {
-		logger.Panicf("FATAL: cannot determine free disk space on %q: %s", path, err)
-	}
-	return freeSpace(stat)
-}
diff --git a/lib/fs/fs_nix.go b/lib/fs/fs_nix.go
index a1e126127..df7c51fef 100644
--- a/lib/fs/fs_nix.go
+++ b/lib/fs/fs_nix.go
@@ -2,7 +2,27 @@
 
 package fs
 
-import "golang.org/x/sys/unix"
+import (
+	"os"
+
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
+	"golang.org/x/sys/unix"
+)
+
+func mustGetFreeSpace(path string) uint64 {
+	d, err := os.Open(path)
+	if err != nil {
+		logger.Panicf("FATAL: cannot determine free disk space on %q: %s", path, err)
+	}
+	defer MustClose(d)
+
+	fd := d.Fd()
+	var stat unix.Statfs_t
+	if err := unix.Fstatfs(int(fd), &stat); err != nil {
+		logger.Panicf("FATAL: cannot determine free disk space on %q: %s", path, err)
+	}
+	return freeSpace(stat)
+}
 
 func freeSpace(stat unix.Statfs_t) uint64 {
 	return uint64(stat.Bavail) * uint64(stat.Bsize)
diff --git a/lib/fs/fs_openbsd.go b/lib/fs/fs_openbsd.go
index 1ab4b9358..9d48a9ee1 100644
--- a/lib/fs/fs_openbsd.go
+++ b/lib/fs/fs_openbsd.go
@@ -1,9 +1,27 @@
 package fs
 
 import (
+	"os"
+
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"golang.org/x/sys/unix"
 )
 
+func mustGetFreeSpace(path string) uint64 {
+	d, err := os.Open(path)
+	if err != nil {
+		logger.Panicf("FATAL: cannot determine free disk space on %q: %s", path, err)
+	}
+	defer MustClose(d)
+
+	fd := d.Fd()
+	var stat unix.Statfs_t
+	if err := unix.Fstatfs(int(fd), &stat); err != nil {
+		logger.Panicf("FATAL: cannot determine free disk space on %q: %s", path, err)
+	}
+	return freeSpace(stat)
+}
+
 func freeSpace(stat unix.Statfs_t) uint64 {
 	return uint64(stat.F_bavail) * uint64(stat.F_bsize)
 }
diff --git a/lib/fs/fs_unix.go b/lib/fs/fs_unix.go
new file mode 100644
index 000000000..ceb27d3a3
--- /dev/null
+++ b/lib/fs/fs_unix.go
@@ -0,0 +1,45 @@
+// +build  linux darwin freebsd openbsd
+
+package fs
+
+import (
+	"fmt"
+	"os"
+
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
+	"golang.org/x/sys/unix"
+)
+
+func mmap(fd int, offset int64, length int) (data []byte, err error) {
+	return unix.Mmap(fd, 0, length, unix.PROT_READ, unix.MAP_SHARED)
+
+}
+func mUnmap(data []byte) error {
+	return unix.Munmap(data)
+}
+
+func mustSyncPath(path string) {
+	d, err := os.Open(path)
+	if err != nil {
+		logger.Panicf("FATAL: cannot open %q: %s", path, err)
+	}
+	if err := d.Sync(); err != nil {
+		_ = d.Close()
+		logger.Panicf("FATAL: cannot flush %q to storage: %s", path, err)
+	}
+	if err := d.Close(); err != nil {
+		logger.Panicf("FATAL: cannot close %q: %s", path, err)
+	}
+}
+
+func createFlockFile(dir string) (*os.File, error) {
+	flockFile := dir + "/flock.lock"
+	flockF, err := os.Create(flockFile)
+	if err != nil {
+		return nil, fmt.Errorf("cannot create lock file %q: %w", flockFile, err)
+	}
+	if err := unix.Flock(int(flockF.Fd()), unix.LOCK_EX|unix.LOCK_NB); err != nil {
+		return nil, fmt.Errorf("cannot acquire lock on file %q: %w", flockFile, err)
+	}
+	return flockF, nil
+}
diff --git a/lib/fs/fs_windows.go b/lib/fs/fs_windows.go
new file mode 100644
index 000000000..e92e3391f
--- /dev/null
+++ b/lib/fs/fs_windows.go
@@ -0,0 +1,149 @@
+package fs
+
+import (
+	"fmt"
+	"os"
+	"unsafe"
+
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
+	"golang.org/x/sys/windows"
+)
+
+var (
+	kernelDLL            = windows.MustLoadDLL("kernel32.dll")
+	procLock             = kernelDLL.MustFindProc("LockFileEx")
+	procEvent            = kernelDLL.MustFindProc("CreateEventW")
+	procDisk             = kernelDLL.MustFindProc("GetDiskFreeSpaceExW")
+	ntDLL                = windows.MustLoadDLL("ntdll.dll")
+	ntSetInformationProc = ntDLL.MustFindProc("NtSetInformationFile")
+)
+
+// panic at windows, if file already open by another process.
+// one of possible solutions - change files opening process with correct flags.
+// https://github.com/dgraph-io/badger/issues/699
+// https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-flushfilebuffers
+func mustSyncPath(string) {
+}
+
+const (
+	lockfileExclusiveLock = 2
+	fileFlagNormal        = 0x00000080
+	// https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/ntddk/ns-ntddk-_file_disposition_information_ex
+	fileDispositionPosixSemantics          = 0x00000002
+	fileDispositionIgnoreReadonlyAttribute = 0x00000010
+)
+
+// createFlockFile creates flock.lock file in the directory dir
+// and returns the handler to the file.
+// https://github.com/juju/fslock/blob/master/fslock_windows.go
+func createFlockFile(dir string) (*os.File, error) {
+	flockFile := dir + "/flock.lock"
+	name, err := windows.UTF16PtrFromString(flockFile)
+	if err != nil {
+		return nil, err
+	}
+	handle, err := windows.CreateFile(
+		name,
+		windows.GENERIC_READ|windows.DELETE,
+		windows.FILE_SHARE_READ|windows.FILE_SHARE_DELETE,
+		nil,
+		windows.OPEN_ALWAYS,
+		windows.FILE_FLAG_OVERLAPPED|fileFlagNormal,
+		0)
+	if err != nil {
+		return nil, fmt.Errorf("cannot create lock file %q: %w", flockFile, err)
+	}
+	ol, err := newOverlapped()
+	if err != nil {
+		return nil, fmt.Errorf("cannot create Overlapped handler: %w", err)
+	}
+	// https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-lockfileex
+	// overlapped is dropped?
+	r1, _, err := procLock.Call(uintptr(handle), uintptr(lockfileExclusiveLock), uintptr(0), uintptr(1), uintptr(0), uintptr(unsafe.Pointer(ol)))
+	if r1 == 0 {
+		return nil, err
+	}
+	return os.NewFile(uintptr(handle), flockFile), nil
+}
+
+// stub
+func mmap(fd int, offset int64, length int) ([]byte, error) {
+	return nil, nil
+}
+
+// stub
+func mUnmap([]byte) error {
+	return nil
+}
+
+func mustGetFreeSpace(path string) uint64 {
+	var freeBytes int64
+	r, _, err := procDisk.Call(uintptr(unsafe.Pointer(windows.StringToUTF16Ptr(path))),
+		uintptr(unsafe.Pointer(&freeBytes)))
+	if r == 0 {
+		logger.Errorf("cannot get free space: %v", err)
+		return 0
+	}
+	return uint64(freeBytes)
+}
+
+// stub
+func fadviseSequentialRead(f *os.File, prefetch bool) error {
+	return nil
+}
+
+// copied from https://github.com/juju/fslock/blob/master/fslock_windows.go
+// https://docs.microsoft.com/en-us/windows/win32/api/minwinbase/ns-minwinbase-overlapped
+func newOverlapped() (*windows.Overlapped, error) {
+	event, err := createEvent(nil, nil)
+	if err != nil {
+		return nil, err
+	}
+	return &windows.Overlapped{HEvent: event}, nil
+}
+
+// copied from https://github.com/juju/fslock/blob/master/fslock_windows.go
+// https://docs.microsoft.com/en-us/windows/win32/api/synchapi/nf-synchapi-createeventa
+func createEvent(sa *windows.SecurityAttributes, name *uint16) (windows.Handle, error) {
+	r0, _, err := procEvent.Call(uintptr(unsafe.Pointer(sa)), uintptr(1), uintptr(1), uintptr(unsafe.Pointer(name)))
+	handle := windows.Handle(r0)
+	if handle == windows.InvalidHandle {
+		return 0, err
+	}
+	return handle, nil
+}
+
+// https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/ntddk/ns-ntddk-_file_disposition_information_ex
+type fileDispositionInformationEx struct {
+	Flags uint32
+}
+
+// https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/ns-wdm-_io_status_block
+type ioStatusBlock struct {
+	Status, Information uintptr
+}
+
+// UpdateFileHandle - changes file deletion semantic at windows to posix-like.
+func UpdateFileHandle(path string) error {
+	handle, err := windows.Open(path, windows.GENERIC_READ|windows.DELETE, windows.FILE_SHARE_READ|windows.FILE_SHARE_DELETE)
+	if err != nil {
+		return err
+	}
+	return setPosixDelete(handle)
+}
+
+// supported starting with Windows 10, version 1709.
+// supported by NTFS only.
+func setPosixDelete(handle windows.Handle) error {
+	var iosb ioStatusBlock
+	flags := fileDispositionInformationEx{
+		Flags: fileDispositionPosixSemantics | fileDispositionIgnoreReadonlyAttribute,
+	}
+	// class FileDispositionInformationEx,                   // 64
+	// https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/ne-wdm-_file_information_class
+	r0, _, err := ntSetInformationProc.Call(uintptr(handle), uintptr(unsafe.Pointer(&iosb)), uintptr(unsafe.Pointer(&flags)), unsafe.Sizeof(flags), uintptr(64))
+	if r0 == 0 {
+		return nil
+	}
+	return fmt.Errorf("cannot set file disposition information: NT_STATUS: 0x%X, error: %w", r0, err)
+}
diff --git a/lib/fs/reader_at.go b/lib/fs/reader_at.go
index 219bbcbb1..13850695c 100644
--- a/lib/fs/reader_at.go
+++ b/lib/fs/reader_at.go
@@ -7,7 +7,6 @@ import (
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/metrics"
-	"golang.org/x/sys/unix"
 )
 
 var disableMmap = flag.Bool("fs.disableMmap", is32BitPtr, "Whether to use pread() instead of mmap() for reading data files. "+
@@ -65,7 +64,7 @@ func (r *ReaderAt) MustReadAt(p []byte, off int64) {
 func (r *ReaderAt) MustClose() {
 	fname := r.f.Name()
 	if len(r.mmapData) > 0 {
-		if err := unix.Munmap(r.mmapData[:cap(r.mmapData)]); err != nil {
+		if err := mUnmap(r.mmapData[:cap(r.mmapData)]); err != nil {
 			logger.Panicf("FATAL: cannot unmap data for file %q: %s", fname, err)
 		}
 		r.mmapData = nil
@@ -135,7 +134,7 @@ func mmapFile(f *os.File, size int64) ([]byte, error) {
 	if size%4096 != 0 {
 		size += 4096 - size%4096
 	}
-	data, err := unix.Mmap(int(f.Fd()), 0, int(size), unix.PROT_READ, unix.MAP_SHARED)
+	data, err := mmap(int(f.Fd()), 0, int(size))
 	if err != nil {
 		return nil, fmt.Errorf("cannot mmap file with size %d: %w", size, err)
 	}
diff --git a/lib/procutil/signal.go b/lib/procutil/signal.go
index ade5acd2b..7076b0b71 100644
--- a/lib/procutil/signal.go
+++ b/lib/procutil/signal.go
@@ -1,3 +1,5 @@
+// +build !windows
+
 package procutil
 
 import (
diff --git a/lib/procutil/signal_windows.go b/lib/procutil/signal_windows.go
new file mode 100644
index 000000000..6b16aee73
--- /dev/null
+++ b/lib/procutil/signal_windows.go
@@ -0,0 +1,60 @@
+// +build windows
+
+package procutil
+
+import (
+	"os"
+	"os/signal"
+	"sync"
+	"syscall"
+)
+
+// WaitForSigterm waits for either SIGTERM or SIGINT
+//
+// Returns the caught signal.
+//
+// Windows dont have SIGHUP syscall.
+func WaitForSigterm() os.Signal {
+	ch := make(chan os.Signal, 1)
+	signal.Notify(ch, os.Interrupt, syscall.SIGTERM)
+	sig := <-ch
+	return sig
+}
+
+type sigHUPNotifier struct {
+	lock        sync.Mutex
+	subscribers []chan<- os.Signal
+}
+
+var notifier sigHUPNotifier
+
+// https://golang.org/pkg/os/signal/#hdr-Windows
+// https://github.com/golang/go/issues/6948
+// SelfSIGHUP sends SIGHUP signal to the subscribed listeners.
+func SelfSIGHUP() {
+	notifier.notify(syscall.SIGHUP)
+}
+
+// NewSighupChan returns a channel, which is triggered on every SelfSIGHUP.
+func NewSighupChan() <-chan os.Signal {
+	ch := make(chan os.Signal, 1)
+	notifier.subscribe(ch)
+	return ch
+}
+
+func (sn *sigHUPNotifier) subscribe(sub chan<- os.Signal) {
+	sn.lock.Lock()
+	defer sn.lock.Unlock()
+	sn.subscribers = append(sn.subscribers, sub)
+}
+
+func (sn *sigHUPNotifier) notify(sig os.Signal) {
+	sn.lock.Lock()
+	defer sn.lock.Unlock()
+	for _, sub := range sn.subscribers {
+		select {
+		case sub <- sig:
+		default:
+		}
+	}
+}
diff --git a/lib/promrelabel/relabel.go b/lib/promrelabel/relabel.go
index a161e55cc..477cf2bea 100644
--- a/lib/promrelabel/relabel.go
+++ b/lib/promrelabel/relabel.go
@@ -334,7 +334,21 @@ func (prc *parsedRelabelConfig) matchString(s string) bool {
 	if complete {
 		return prefix == s
 	}
-	return strings.HasPrefix(s, prefix) && prc.Regex.MatchString(s)
+	if !strings.HasPrefix(s, prefix) {
+		return false
+	}
+	reStr := prc.regexOriginal.String()
+	if strings.HasPrefix(reStr, prefix) {
+		// Fast path for `foo.*` and `bar.+` regexps
+		reSuffix := reStr[len(prefix):]
+		switch reSuffix {
+		case ".+", "(.+)":
+			return len(s) > len(prefix)
+		case ".*", "(.*)":
+			return true
+		}
+	}
+	return prc.Regex.MatchString(s)
 }
 
 func (prc *parsedRelabelConfig) expandCaptureGroups(template, source string, match []int) string {
diff --git a/lib/promscrape/config.go b/lib/promscrape/config.go
index 4f97506b6..641adf8f1 100644
--- a/lib/promscrape/config.go
+++ b/lib/promscrape/config.go
@@ -6,11 +6,15 @@ import (
 	"io/ioutil"
 	"net/url"
 	"path/filepath"
+	"sort"
+	"strconv"
 	"strings"
 	"sync"
 	"time"
 
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/cgroup"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/envtemplate"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promauth"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
@@ -196,7 +200,7 @@ func (cfg *Config) getKubernetesSDScrapeWork(prev []*ScrapeWork) []*ScrapeWork {
 		for j := range sc.KubernetesSDConfigs {
 			sdc := &sc.KubernetesSDConfigs[j]
 			var okLocal bool
-			dst, okLocal = appendKubernetesScrapeWork(dst, sdc, cfg.baseDir, sc.swc)
+			dst, okLocal = appendSDScrapeWork(dst, sdc, cfg.baseDir, sc.swc, "kubernetes_sd_config")
 			if ok {
 				ok = okLocal
 			}
@@ -224,7 +228,7 @@ func (cfg *Config) getOpenStackSDScrapeWork(prev []*ScrapeWork) []*ScrapeWork {
 		for j := range sc.OpenStackSDConfigs {
 			sdc := &sc.OpenStackSDConfigs[j]
 			var okLocal bool
-			dst, okLocal = appendOpenstackScrapeWork(dst, sdc, cfg.baseDir, sc.swc)
+			dst, okLocal = appendSDScrapeWork(dst, sdc, cfg.baseDir, sc.swc, "openstack_sd_config")
 			if ok {
 				ok = okLocal
 			}
@@ -252,7 +256,7 @@ func (cfg *Config) getDockerSwarmSDScrapeWork(prev []*ScrapeWork) []*ScrapeWork
 		for j := range sc.DockerSwarmConfigs {
 			sdc := &sc.DockerSwarmConfigs[j]
 			var okLocal bool
-			dst, okLocal = appendDockerSwarmScrapeWork(dst, sdc, cfg.baseDir, sc.swc)
+			dst, okLocal = appendSDScrapeWork(dst, sdc, cfg.baseDir, sc.swc, "dockerswarm_sd_config")
 			if ok {
 				ok = okLocal
 			}
@@ -280,7 +284,7 @@ func (cfg *Config) getConsulSDScrapeWork(prev []*ScrapeWork) []*ScrapeWork {
 		for j := range sc.ConsulSDConfigs {
 			sdc := &sc.ConsulSDConfigs[j]
 			var okLocal bool
-			dst, okLocal = appendConsulScrapeWork(dst, sdc, cfg.baseDir, sc.swc)
+			dst, okLocal = appendSDScrapeWork(dst, sdc, cfg.baseDir, sc.swc, "consul_sd_config")
 			if ok {
 				ok = okLocal
 			}
@@ -308,7 +312,7 @@ func (cfg *Config) getEurekaSDScrapeWork(prev []*ScrapeWork) []*ScrapeWork {
 		for j := range sc.EurekaSDConfigs {
 			sdc := &sc.EurekaSDConfigs[j]
 			var okLocal bool
-			dst, okLocal = appendEurekaScrapeWork(dst, sdc, cfg.baseDir, sc.swc)
+			dst, okLocal = appendSDScrapeWork(dst, sdc, cfg.baseDir, sc.swc, "eureka_sd_config")
 			if ok {
 				ok = okLocal
 			}
@@ -336,7 +340,7 @@ func (cfg *Config) getDNSSDScrapeWork(prev []*ScrapeWork) []*ScrapeWork {
 		for j := range sc.DNSSDConfigs {
 			sdc := &sc.DNSSDConfigs[j]
 			var okLocal bool
-			dst, okLocal = appendDNSScrapeWork(dst, sdc, sc.swc)
+			dst, okLocal = appendSDScrapeWork(dst, sdc, cfg.baseDir, sc.swc, "dns_sd_config")
 			if ok {
 				ok = okLocal
 			}
@@ -364,7 +368,7 @@ func (cfg *Config) getEC2SDScrapeWork(prev []*ScrapeWork) []*ScrapeWork {
 		for j := range sc.EC2SDConfigs {
 			sdc := &sc.EC2SDConfigs[j]
 			var okLocal bool
-			dst, okLocal = appendEC2ScrapeWork(dst, sdc, sc.swc)
+			dst, okLocal = appendSDScrapeWork(dst, sdc, cfg.baseDir, sc.swc, "ec2_sd_config")
 			if ok {
 				ok = okLocal
 			}
@@ -392,7 +396,7 @@ func (cfg *Config) getGCESDScrapeWork(prev []*ScrapeWork) []*ScrapeWork {
 		for j := range sc.GCESDConfigs {
 			sdc := &sc.GCESDConfigs[j]
 			var okLocal bool
-			dst, okLocal = appendGCEScrapeWork(dst, sdc, sc.swc)
+			dst, okLocal = appendSDScrapeWork(dst, sdc, cfg.baseDir, sc.swc, "gce_sd_config")
 			if ok {
 				ok = okLocal
 			}
@@ -509,6 +513,8 @@ func getScrapeWorkConfig(sc *ScrapeConfig, baseDir string, globalCfg *GlobalConf
 		disableKeepAlive:     sc.DisableKeepAlive,
 		streamParse:          sc.StreamParse,
 		scrapeAlignInterval:  sc.ScrapeAlignInterval,
+
+		cache: newScrapeWorkCache(),
 	}
 	return swc, nil
 }
@@ -532,92 +538,112 @@ type scrapeWorkConfig struct {
 	disableKeepAlive     bool
 	streamParse          bool
 	scrapeAlignInterval  time.Duration
-}
 
-func appendKubernetesScrapeWork(dst []*ScrapeWork, sdc *kubernetes.SDConfig, baseDir string, swc *scrapeWorkConfig) ([]*ScrapeWork, bool) {
-	targetLabels, err := kubernetes.GetLabels(sdc, baseDir)
-	if err != nil {
-		logger.Errorf("error when discovering kubernetes targets for `job_name` %q: %s; skipping it", swc.jobName, err)
-		return dst, false
-	}
-	return appendScrapeWorkForTargetLabels(dst, swc, targetLabels, "kubernetes_sd_config"), true
+	cache *scrapeWorkCache
 }
 
-func appendOpenstackScrapeWork(dst []*ScrapeWork, sdc *openstack.SDConfig, baseDir string, swc *scrapeWorkConfig) ([]*ScrapeWork, bool) {
-	targetLabels, err := openstack.GetLabels(sdc, baseDir)
-	if err != nil {
-		logger.Errorf("error when discovering openstack targets for `job_name` %q: %s; skipping it", swc.jobName, err)
-		return dst, false
-	}
-	return appendScrapeWorkForTargetLabels(dst, swc, targetLabels, "openstack_sd_config"), true
+type scrapeWorkCache struct {
+	mu              sync.Mutex
+	m               map[string]*scrapeWorkEntry
+	lastCleanupTime uint64
 }
 
-func appendDockerSwarmScrapeWork(dst []*ScrapeWork, sdc *dockerswarm.SDConfig, baseDir string, swc *scrapeWorkConfig) ([]*ScrapeWork, bool) {
-	targetLabels, err := dockerswarm.GetLabels(sdc, baseDir)
-	if err != nil {
-		logger.Errorf("error when discovering dockerswarm targets for `job_name` %q: %s; skipping it", swc.jobName, err)
-		return dst, false
-	}
-	return appendScrapeWorkForTargetLabels(dst, swc, targetLabels, "dockerswarm_sd_config"), true
+type scrapeWorkEntry struct {
+	sw             *ScrapeWork
+	lastAccessTime uint64
 }
 
-func appendConsulScrapeWork(dst []*ScrapeWork, sdc *consul.SDConfig, baseDir string, swc *scrapeWorkConfig) ([]*ScrapeWork, bool) {
-	targetLabels, err := consul.GetLabels(sdc, baseDir)
-	if err != nil {
-		logger.Errorf("error when discovering consul targets for `job_name` %q: %s; skipping it", swc.jobName, err)
-		return dst, false
+func newScrapeWorkCache() *scrapeWorkCache {
+	return &scrapeWorkCache{
+		m: make(map[string]*scrapeWorkEntry),
 	}
-	return appendScrapeWorkForTargetLabels(dst, swc, targetLabels, "consul_sd_config"), true
 }
 
-func appendEurekaScrapeWork(dst []*ScrapeWork, sdc *eureka.SDConfig, baseDir string, swc *scrapeWorkConfig) ([]*ScrapeWork, bool) {
-	targetLabels, err := eureka.GetLabels(sdc, baseDir)
-	if err != nil {
-		logger.Errorf("error when discovering eureka targets for `job_name` %q: %s; skipping it", swc.jobName, err)
-		return dst, false
+func (swc *scrapeWorkCache) Get(key string) *ScrapeWork {
+	currentTime := fasttime.UnixTimestamp()
+	swc.mu.Lock()
+	swe := swc.m[key]
+	if swe != nil {
+		swe.lastAccessTime = currentTime
 	}
-	return appendScrapeWorkForTargetLabels(dst, swc, targetLabels, "eureka_sd_config"), true
+	swc.mu.Unlock()
+	if swe == nil {
+		return nil
+	}
+	return swe.sw
 }
 
-func appendDNSScrapeWork(dst []*ScrapeWork, sdc *dns.SDConfig, swc *scrapeWorkConfig) ([]*ScrapeWork, bool) {
-	targetLabels, err := dns.GetLabels(sdc)
-	if err != nil {
-		logger.Errorf("error when discovering dns targets for `job_name` %q: %s; skipping it", swc.jobName, err)
-		return dst, false
+func (swc *scrapeWorkCache) Set(key string, sw *ScrapeWork) {
+	currentTime := fasttime.UnixTimestamp()
+	swc.mu.Lock()
+	swc.m[key] = &scrapeWorkEntry{
+		sw:             sw,
+		lastAccessTime: currentTime,
 	}
-	return appendScrapeWorkForTargetLabels(dst, swc, targetLabels, "dns_sd_config"), true
+	if currentTime > swc.lastCleanupTime+10*60 {
+		for k, swe := range swc.m {
+			if currentTime > swe.lastAccessTime+2*60 {
+				delete(swc.m, k)
+			}
+		}
+		swc.lastCleanupTime = currentTime
+	}
+	swc.mu.Unlock()
 }
 
-func appendEC2ScrapeWork(dst []*ScrapeWork, sdc *ec2.SDConfig, swc *scrapeWorkConfig) ([]*ScrapeWork, bool) {
-	targetLabels, err := ec2.GetLabels(sdc)
-	if err != nil {
-		logger.Errorf("error when discovering ec2 targets for `job_name` %q: %s; skipping it", swc.jobName, err)
-		return dst, false
-	}
-	return appendScrapeWorkForTargetLabels(dst, swc, targetLabels, "ec2_sd_config"), true
+type targetLabelsGetter interface {
+	GetLabels(baseDir string) ([]map[string]string, error)
 }
 
-func appendGCEScrapeWork(dst []*ScrapeWork, sdc *gce.SDConfig, swc *scrapeWorkConfig) ([]*ScrapeWork, bool) {
-	targetLabels, err := gce.GetLabels(sdc)
+func appendSDScrapeWork(dst []*ScrapeWork, sdc targetLabelsGetter, baseDir string, swc *scrapeWorkConfig, discoveryType string) ([]*ScrapeWork, bool) {
+	targetLabels, err := sdc.GetLabels(baseDir)
 	if err != nil {
-		logger.Errorf("error when discovering gce targets for `job_name` %q: %s; skippint it", swc.jobName, err)
+		logger.Errorf("skipping %s targets for job_name %q because of error: %s", discoveryType, swc.jobName, err)
 		return dst, false
 	}
-	return appendScrapeWorkForTargetLabels(dst, swc, targetLabels, "gce_sd_config"), true
+	return appendScrapeWorkForTargetLabels(dst, swc, targetLabels, discoveryType), true
 }
 
-func appendScrapeWorkForTargetLabels(dst []*ScrapeWork, swc *scrapeWorkConfig, targetLabels []map[string]string, sectionName string) []*ScrapeWork {
+func appendScrapeWorkForTargetLabels(dst []*ScrapeWork, swc *scrapeWorkConfig, targetLabels []map[string]string, discoveryType string) []*ScrapeWork {
 	startTime := time.Now()
+	// Process targetLabels in parallel in order to reduce processing time for big number of targetLabels.
+	type result struct {
+		sw  *ScrapeWork
+		err error
+	}
+	resultCh := make(chan result)
+	workCh := make(chan map[string]string)
+	goroutines := cgroup.AvailableCPUs()
+	for i := 0; i < goroutines; i++ {
+		go func() {
+			for metaLabels := range workCh {
+				target := metaLabels["__address__"]
+				sw, err := swc.getScrapeWork(target, nil, metaLabels)
+				if err != nil {
+					err = fmt.Errorf("skipping %s target %q for job_name %q because of error: %w", discoveryType, target, swc.jobName, err)
+				}
+				resultCh <- result{
+					sw:  sw,
+					err: err,
+				}
+			}
+		}()
+	}
 	for _, metaLabels := range targetLabels {
-		target := metaLabels["__address__"]
-		var err error
-		dst, err = appendScrapeWork(dst, swc, target, nil, metaLabels)
-		if err != nil {
-			logger.Errorf("error when parsing `%s` target %q for `job_name` %q: %s; skipping it", sectionName, target, swc.jobName, err)
+		workCh <- metaLabels
+	}
+	close(workCh)
+	for range targetLabels {
+		r := <-resultCh
+		if r.err != nil {
+			logger.Errorf("%s", r.err)
 			continue
 		}
+		if r.sw != nil {
+			dst = append(dst, r.sw)
+		}
 	}
-	metrics.GetOrCreateHistogram(fmt.Sprintf("vm_promscrape_target_relabel_duration_seconds{type=%q}", sectionName)).UpdateDuration(startTime)
+	metrics.GetOrCreateHistogram(fmt.Sprintf("vm_promscrape_target_relabel_duration_seconds{type=%q}", discoveryType)).UpdateDuration(startTime)
 	return dst
 }
 
@@ -673,18 +699,55 @@ func (stc *StaticConfig) appendScrapeWork(dst []*ScrapeWork, swc *scrapeWorkConf
 			logger.Errorf("`static_configs` target for `job_name` %q cannot be empty; skipping it", swc.jobName)
 			continue
 		}
-		var err error
-		dst, err = appendScrapeWork(dst, swc, target, stc.Labels, metaLabels)
+		sw, err := swc.getScrapeWork(target, stc.Labels, metaLabels)
 		if err != nil {
 			// Do not return this error, since other targets may be valid
 			logger.Errorf("error when parsing `static_configs` target %q for `job_name` %q: %s; skipping it", target, swc.jobName, err)
 			continue
 		}
+		if sw != nil {
+			dst = append(dst, sw)
+		}
+	}
+	return dst
+}
+
+func (swc *scrapeWorkConfig) getScrapeWork(target string, extraLabels, metaLabels map[string]string) (*ScrapeWork, error) {
+	key := getScrapeWorkKey(extraLabels, metaLabels)
+	if sw := swc.cache.Get(key); sw != nil {
+		return sw, nil
+	}
+	sw, err := swc.getScrapeWorkReal(target, extraLabels, metaLabels)
+	if err != nil {
+		swc.cache.Set(key, sw)
 	}
+	return sw, err
+}
+
+func getScrapeWorkKey(extraLabels, metaLabels map[string]string) string {
+	var b []byte
+	b = appendSortedKeyValuePairs(b, extraLabels)
+	b = appendSortedKeyValuePairs(b, metaLabels)
+	return string(b)
+}
+
+func appendSortedKeyValuePairs(dst []byte, m map[string]string) []byte {
+	keys := make([]string, 0, len(m))
+	for k := range m {
+		keys = append(keys, k)
+	}
+	sort.Strings(keys)
+	for _, k := range keys {
+		dst = strconv.AppendQuote(dst, k)
+		dst = append(dst, ':')
+		dst = strconv.AppendQuote(dst, m[k])
+		dst = append(dst, ',')
+	}
+	dst = append(dst, '\n')
 	return dst
 }
 
-func appendScrapeWork(dst []*ScrapeWork, swc *scrapeWorkConfig, target string, extraLabels, metaLabels map[string]string) ([]*ScrapeWork, error) {
+func (swc *scrapeWorkConfig) getScrapeWorkReal(target string, extraLabels, metaLabels map[string]string) (*ScrapeWork, error) {
 	labels := mergeLabels(swc.jobName, swc.scheme, target, swc.metricsPath, extraLabels, swc.externalLabels, metaLabels, swc.params)
 	var originalLabels []prompbmarshal.Label
 	if !*dropOriginalLabels {
@@ -703,7 +766,7 @@ func appendScrapeWork(dst []*ScrapeWork, swc *scrapeWorkConfig, target string, e
 	if len(labels) == 0 {
 		// Drop target without labels.
 		droppedTargetsMap.Register(originalLabels)
-		return dst, nil
+		return nil, nil
 	}
 	// See https://www.robustperception.io/life-of-a-label
 	schemeRelabeled := promrelabel.GetLabelValueByName(labels, "__scheme__")
@@ -714,12 +777,12 @@ func appendScrapeWork(dst []*ScrapeWork, swc *scrapeWorkConfig, target string, e
 	if len(addressRelabeled) == 0 {
 		// Drop target without scrape address.
 		droppedTargetsMap.Register(originalLabels)
-		return dst, nil
+		return nil, nil
 	}
 	if strings.Contains(addressRelabeled, "/") {
 		// Drop target with '/'
 		droppedTargetsMap.Register(originalLabels)
-		return dst, nil
+		return nil, nil
 	}
 	addressRelabeled = addMissingPort(schemeRelabeled, addressRelabeled)
 	metricsPathRelabeled := promrelabel.GetLabelValueByName(labels, "__metrics_path__")
@@ -737,7 +800,7 @@ func appendScrapeWork(dst []*ScrapeWork, swc *scrapeWorkConfig, target string, e
 	paramsStr := url.Values(paramsRelabeled).Encode()
 	scrapeURL := fmt.Sprintf("%s://%s%s%s%s", schemeRelabeled, addressRelabeled, metricsPathRelabeled, optionalQuestion, paramsStr)
 	if _, err := url.Parse(scrapeURL); err != nil {
-		return dst, fmt.Errorf("invalid url %q for scheme=%q (%q), target=%q (%q), metrics_path=%q (%q) for `job_name` %q: %w",
+		return nil, fmt.Errorf("invalid url %q for scheme=%q (%q), target=%q (%q), metrics_path=%q (%q) for `job_name` %q: %w",
 			scrapeURL, swc.scheme, schemeRelabeled, target, addressRelabeled, swc.metricsPath, metricsPathRelabeled, swc.jobName, err)
 	}
 	// Set missing "instance" label according to https://www.robustperception.io/life-of-a-label
@@ -750,7 +813,7 @@ func appendScrapeWork(dst []*ScrapeWork, swc *scrapeWorkConfig, target string, e
 	}
 	// Reduce memory usage by interning all the strings in labels.
 	internLabelStrings(labels)
-	dst = append(dst, &ScrapeWork{
+	sw := &ScrapeWork{
 		ScrapeURL:            scrapeURL,
 		ScrapeInterval:       swc.scrapeInterval,
 		ScrapeTimeout:        swc.scrapeTimeout,
@@ -768,8 +831,8 @@ func appendScrapeWork(dst []*ScrapeWork, swc *scrapeWorkConfig, target string, e
 		ScrapeAlignInterval:  swc.scrapeAlignInterval,
 
 		jobNameOriginal: swc.jobName,
-	})
-	return dst, nil
+	}
+	return sw, nil
 }
 
 func internLabelStrings(labels []prompbmarshal.Label) {
diff --git a/lib/promscrape/discovery/consul/consul.go b/lib/promscrape/discovery/consul/consul.go
index 9c980f779..b0e81d062 100644
--- a/lib/promscrape/discovery/consul/consul.go
+++ b/lib/promscrape/discovery/consul/consul.go
@@ -29,7 +29,7 @@ type SDConfig struct {
 }
 
 // GetLabels returns Consul labels according to sdc.
-func GetLabels(sdc *SDConfig, baseDir string) ([]map[string]string, error) {
+func (sdc *SDConfig) GetLabels(baseDir string) ([]map[string]string, error) {
 	cfg, err := getAPIConfig(sdc, baseDir)
 	if err != nil {
 		return nil, fmt.Errorf("cannot get API config: %w", err)
diff --git a/lib/promscrape/discovery/dns/dns.go b/lib/promscrape/discovery/dns/dns.go
index 40d8c580d..7d330a2d5 100644
--- a/lib/promscrape/discovery/dns/dns.go
+++ b/lib/promscrape/discovery/dns/dns.go
@@ -24,7 +24,7 @@ type SDConfig struct {
 }
 
 // GetLabels returns DNS labels according to sdc.
-func GetLabels(sdc *SDConfig) ([]map[string]string, error) {
+func (sdc *SDConfig) GetLabels(baseDir string) ([]map[string]string, error) {
 	if len(sdc.Names) == 0 {
 		return nil, fmt.Errorf("`names` cannot be empty in `dns_sd_config`")
 	}
diff --git a/lib/promscrape/discovery/dockerswarm/dockerswarm.go b/lib/promscrape/discovery/dockerswarm/dockerswarm.go
index bd9564306..b58f68fcd 100644
--- a/lib/promscrape/discovery/dockerswarm/dockerswarm.go
+++ b/lib/promscrape/discovery/dockerswarm/dockerswarm.go
@@ -31,7 +31,7 @@ type Filter struct {
 }
 
 // GetLabels returns dockerswarm labels according to sdc.
-func GetLabels(sdc *SDConfig, baseDir string) ([]map[string]string, error) {
+func (sdc *SDConfig) GetLabels(baseDir string) ([]map[string]string, error) {
 	cfg, err := getAPIConfig(sdc, baseDir)
 	if err != nil {
 		return nil, fmt.Errorf("cannot get API config: %w", err)
diff --git a/lib/promscrape/discovery/ec2/ec2.go b/lib/promscrape/discovery/ec2/ec2.go
index 8ffa1697e..6a830e163 100644
--- a/lib/promscrape/discovery/ec2/ec2.go
+++ b/lib/promscrape/discovery/ec2/ec2.go
@@ -31,7 +31,7 @@ type Filter struct {
 }
 
 // GetLabels returns ec2 labels according to sdc.
-func GetLabels(sdc *SDConfig) ([]map[string]string, error) {
+func (sdc *SDConfig) GetLabels(baseDir string) ([]map[string]string, error) {
 	cfg, err := getAPIConfig(sdc)
 	if err != nil {
 		return nil, fmt.Errorf("cannot get API config: %w", err)
diff --git a/lib/promscrape/discovery/eureka/eureka.go b/lib/promscrape/discovery/eureka/eureka.go
index c8ebc55ef..c9759e6e0 100644
--- a/lib/promscrape/discovery/eureka/eureka.go
+++ b/lib/promscrape/discovery/eureka/eureka.go
@@ -82,7 +82,7 @@ type DataCenterInfo struct {
 }
 
 // GetLabels returns Eureka labels according to sdc.
-func GetLabels(sdc *SDConfig, baseDir string) ([]map[string]string, error) {
+func (sdc *SDConfig) GetLabels(baseDir string) ([]map[string]string, error) {
 	cfg, err := getAPIConfig(sdc, baseDir)
 	if err != nil {
 		return nil, fmt.Errorf("cannot get API config: %w", err)
diff --git a/lib/promscrape/discovery/gce/gce.go b/lib/promscrape/discovery/gce/gce.go
index f0629d2ef..4234c7655 100644
--- a/lib/promscrape/discovery/gce/gce.go
+++ b/lib/promscrape/discovery/gce/gce.go
@@ -48,7 +48,7 @@ func (z *ZoneYAML) UnmarshalYAML(unmarshal func(interface{}) error) error {
 }
 
 // GetLabels returns gce labels according to sdc.
-func GetLabels(sdc *SDConfig) ([]map[string]string, error) {
+func (sdc *SDConfig) GetLabels(baseDir string) ([]map[string]string, error) {
 	cfg, err := getAPIConfig(sdc)
 	if err != nil {
 		return nil, fmt.Errorf("cannot get API config: %w", err)
diff --git a/lib/promscrape/discovery/kubernetes/api.go b/lib/promscrape/discovery/kubernetes/api.go
index 2c0463214..a11e528c7 100644
--- a/lib/promscrape/discovery/kubernetes/api.go
+++ b/lib/promscrape/discovery/kubernetes/api.go
@@ -1,19 +1,32 @@
 package kubernetes
 
 import (
+	"encoding/json"
+	"errors"
+	"flag"
 	"fmt"
+	"io"
+	"io/ioutil"
 	"net"
+	"net/http"
+	"net/url"
 	"os"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
 
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promauth"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discoveryutils"
 )
 
+var apiServerTimeout = flag.Duration("promscrape.kubernetes.apiServerTimeout", 2*time.Minute, "Timeout for requests to Kuberntes API server")
+
 // apiConfig contains config for API server
 type apiConfig struct {
-	client     *discoveryutils.Client
-	namespaces []string
-	selectors  []Selector
+	aw *apiWatcher
 }
 
 var configMap = discoveryutils.NewConfigMap()
@@ -56,22 +69,437 @@ func newAPIConfig(sdc *SDConfig, baseDir string) (*apiConfig, error) {
 		}
 		ac = acNew
 	}
-	client, err := discoveryutils.NewClient(apiServer, ac, sdc.ProxyURL)
-	if err != nil {
-		return nil, fmt.Errorf("cannot create HTTP client for %q: %w", apiServer, err)
+	if !strings.Contains(apiServer, "://") {
+		proto := "http"
+		if sdc.TLSConfig != nil {
+			proto = "https"
+		}
+		apiServer = proto + "://" + apiServer
+	}
+	for strings.HasSuffix(apiServer, "/") {
+		apiServer = apiServer[:len(apiServer)-1]
+	}
+	var proxy func(*http.Request) (*url.URL, error)
+	if proxyURL := sdc.ProxyURL.URL(); proxyURL != nil {
+		proxy = http.ProxyURL(proxyURL)
+	}
+	client := &http.Client{
+		Transport: &http.Transport{
+			TLSClientConfig:     ac.NewTLSConfig(),
+			Proxy:               proxy,
+			TLSHandshakeTimeout: 10 * time.Second,
+			IdleConnTimeout:     *apiServerTimeout,
+		},
+		Timeout: *apiServerTimeout,
 	}
+	aw := newAPIWatcher(client, apiServer, ac.Authorization, sdc.Namespaces.Names, sdc.Selectors)
 	cfg := &apiConfig{
-		client:     client,
-		namespaces: sdc.Namespaces.Names,
-		selectors:  sdc.Selectors,
+		aw: aw,
 	}
 	return cfg, nil
 }
 
-func getAPIResponse(cfg *apiConfig, role, path string) ([]byte, error) {
-	query := joinSelectors(role, cfg.namespaces, cfg.selectors)
+// WatchEvent is a watch event returned from API server endpoints if `watch=1` query arg is set.
+//
+// See https://kubernetes.io/docs/reference/using-api/api-concepts/#efficient-detection-of-changes
+type WatchEvent struct {
+	Type   string
+	Object json.RawMessage
+}
+
+// object is any Kubernetes object.
+type object interface {
+	key() string
+	getTargetLabels(aw *apiWatcher) []map[string]string
+}
+
+// parseObjectFunc must parse object from the given data.
+type parseObjectFunc func(data []byte) (object, error)
+
+// parseObjectListFunc must parse objectList from the given data.
+type parseObjectListFunc func(data []byte) (map[string]object, ListMeta, error)
+
+// apiWatcher is used for watching for Kuberntes object changes and caching their latest states.
+type apiWatcher struct {
+	// The client used for watching for object changes
+	client *http.Client
+
+	// Kubenetes API server address in the form http://api-server
+	apiServer string
+
+	// The contents for `Authorization` HTTP request header
+	authorization string
+
+	// Namespaces to watch
+	namespaces []string
+
+	// Selectors to apply during watch
+	selectors []Selector
+
+	// mu protects watchersByURL and lastAccessTime
+	mu sync.Mutex
+
+	// a map of watchers keyed by request paths
+	watchersByURL map[string]*urlWatcher
+
+	// The last time the apiWatcher was queried for cached objects.
+	// It is used for stopping unused watchers.
+	lastAccessTime uint64
+}
+
+func newAPIWatcher(client *http.Client, apiServer, authorization string, namespaces []string, selectors []Selector) *apiWatcher {
+	return &apiWatcher{
+		apiServer:     apiServer,
+		authorization: authorization,
+		client:        client,
+		namespaces:    namespaces,
+		selectors:     selectors,
+
+		watchersByURL: make(map[string]*urlWatcher),
+
+		lastAccessTime: fasttime.UnixTimestamp(),
+	}
+}
+
+// getLabelsForRole returns all the sets of labels for the given role.
+func (aw *apiWatcher) getLabelsForRole(role string) []map[string]string {
+	var ms []map[string]string
+	aw.mu.Lock()
+	for _, uw := range aw.watchersByURL {
+		if uw.role != role {
+			continue
+		}
+		uw.mu.Lock()
+		for _, labels := range uw.labelsByKey {
+			ms = append(ms, labels...)
+		}
+		uw.mu.Unlock()
+	}
+	aw.lastAccessTime = fasttime.UnixTimestamp()
+	aw.mu.Unlock()
+	return ms
+}
+
+// getObjectByRole returns an object with the given (namespace, name) key and the given role.
+func (aw *apiWatcher) getObjectByRole(role, namespace, name string) object {
+	if aw == nil {
+		return nil
+	}
+	key := namespace + "/" + name
+	aw.startWatchersForRole(role)
+	var o object
+	aw.mu.Lock()
+	for _, uw := range aw.watchersByURL {
+		if uw.role != role {
+			continue
+		}
+		uw.mu.Lock()
+		o = uw.objectsByKey[key]
+		uw.mu.Unlock()
+		if o != nil {
+			break
+		}
+	}
+	aw.lastAccessTime = fasttime.UnixTimestamp()
+	aw.mu.Unlock()
+	return o
+}
+
+func (aw *apiWatcher) startWatchersForRole(role string) {
+	parseObject, parseObjectList := getObjectParsersForRole(role)
+	paths := getAPIPaths(role, aw.namespaces, aw.selectors)
+	for _, path := range paths {
+		apiURL := aw.apiServer + path
+		aw.startWatcherForURL(role, apiURL, parseObject, parseObjectList)
+	}
+}
+
+func (aw *apiWatcher) startWatcherForURL(role, apiURL string, parseObject parseObjectFunc, parseObjectList parseObjectListFunc) {
+	aw.mu.Lock()
+	defer aw.mu.Unlock()
+	if aw.watchersByURL[apiURL] != nil {
+		// Watcher for the given path already exists.
+		return
+	}
+	uw := aw.newURLWatcher(role, apiURL, parseObject, parseObjectList)
+	resourceVersion := uw.reloadObjects()
+	aw.watchersByURL[apiURL] = uw
+	go func() {
+		uw.watchForUpdates(resourceVersion)
+		aw.mu.Lock()
+		delete(aw.watchersByURL, apiURL)
+		aw.mu.Unlock()
+	}()
+}
+
+// needStop returns true if aw wasn't used for long time.
+func (aw *apiWatcher) needStop() bool {
+	aw.mu.Lock()
+	defer aw.mu.Unlock()
+	return fasttime.UnixTimestamp() > aw.lastAccessTime+5*60
+}
+
+// doRequest performs http request to the given requestURL.
+func (aw *apiWatcher) doRequest(requestURL string) (*http.Response, error) {
+	req, err := http.NewRequest("GET", requestURL, nil)
+	if err != nil {
+		logger.Fatalf("cannot create a request for %q: %s", requestURL, err)
+	}
+	if aw.authorization != "" {
+		req.Header.Set("Authorization", aw.authorization)
+	}
+	return aw.client.Do(req)
+}
+
+// urlWatcher watches for an apiURL and updates object states in objectsByKey.
+type urlWatcher struct {
+	role   string
+	apiURL string
+
+	parseObject     parseObjectFunc
+	parseObjectList parseObjectListFunc
+
+	// mu protects objectsByKey and labelsByKey
+	mu sync.Mutex
+
+	// objectsByKey contains the latest state for objects obtained from apiURL
+	objectsByKey map[string]object
+	labelsByKey  map[string][]map[string]string
+
+	// the parent apiWatcher
+	aw *apiWatcher
+}
+
+func (aw *apiWatcher) newURLWatcher(role, apiURL string, parseObject parseObjectFunc, parseObjectList parseObjectListFunc) *urlWatcher {
+	return &urlWatcher{
+		role:   role,
+		apiURL: apiURL,
+
+		parseObject:     parseObject,
+		parseObjectList: parseObjectList,
+
+		objectsByKey: make(map[string]object),
+		labelsByKey:  make(map[string][]map[string]string),
+
+		aw: aw,
+	}
+}
+
+// reloadObjects reloads objects to the latest state and returns resourceVersion for the latest state.
+func (uw *urlWatcher) reloadObjects() string {
+	requestURL := uw.apiURL
+	resp, err := uw.aw.doRequest(requestURL)
+	if err != nil {
+		logger.Errorf("error when performing a request to %q: %s", requestURL, err)
+		return ""
+	}
+	body, _ := ioutil.ReadAll(resp.Body)
+	_ = resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		logger.Errorf("unexpected status code for request to %q: %d; want %d; response: %q", requestURL, resp.StatusCode, http.StatusOK, body)
+		return ""
+	}
+	objectsByKey, metadata, err := uw.parseObjectList(body)
+	if err != nil {
+		logger.Errorf("cannot parse response from %q: %s", requestURL, err)
+		return ""
+	}
+	uw.mu.Lock()
+	uw.objectsByKey = objectsByKey
+	uw.mu.Unlock()
+	return metadata.ResourceVersion
+}
+
+// watchForUpdates watches for object updates starting from resourceVersion and updates the corresponding objects to the latest state.
+//
+// See https://kubernetes.io/docs/reference/using-api/api-concepts/#efficient-detection-of-changes
+func (uw *urlWatcher) watchForUpdates(resourceVersion string) {
+	aw := uw.aw
+	backoffDelay := time.Second
+	maxBackoffDelay := 30 * time.Second
+	backoffSleep := func() {
+		time.Sleep(backoffDelay)
+		backoffDelay *= 2
+		if backoffDelay > maxBackoffDelay {
+			backoffDelay = maxBackoffDelay
+		}
+	}
+	apiURL := uw.apiURL
+	delimiter := "?"
+	if strings.Contains(apiURL, "?") {
+		delimiter = "&"
+	}
+	timeoutSeconds := time.Duration(0.9 * float64(aw.client.Timeout)).Seconds()
+	apiURL += delimiter + "watch=1&timeoutSeconds=" + strconv.Itoa(int(timeoutSeconds))
+	logger.Infof("started watcher for %q", apiURL)
+	for {
+		if aw.needStop() {
+			logger.Infof("stopped unused watcher for %q", apiURL)
+			return
+		}
+		requestURL := apiURL
+		if resourceVersion != "" {
+			requestURL += "&resourceVersion=" + url.QueryEscape(resourceVersion) + "&resourceVersionMatch=NotOlderThan"
+		}
+		resp, err := aw.doRequest(requestURL)
+		if err != nil {
+			logger.Errorf("error when performing a request to %q: %s", requestURL, err)
+			backoffSleep()
+			// There is no sense in reloading resources on non-http errors.
+			continue
+		}
+		if resp.StatusCode != http.StatusOK {
+			body, _ := ioutil.ReadAll(resp.Body)
+			_ = resp.Body.Close()
+			logger.Errorf("unexpected status code for request to %q: %d; want %d; response: %q", requestURL, resp.StatusCode, http.StatusOK, body)
+			if resp.StatusCode == 410 {
+				// Update stale resourceVersion. See https://kubernetes.io/docs/reference/using-api/api-concepts/#410-gone-responses
+				resourceVersion = uw.reloadObjects()
+				backoffDelay = time.Second
+			} else {
+				backoffSleep()
+				// There is no sense in reloading resources on non-410 status codes.
+			}
+			continue
+		}
+		backoffDelay = time.Second
+		err = uw.readObjectUpdateStream(resp.Body)
+		_ = resp.Body.Close()
+		if err != nil {
+			if errors.Is(err, io.EOF) {
+				// The stream has been closed (probably due to timeout)
+				backoffSleep()
+				continue
+			}
+			logger.Errorf("error when reading WatchEvent stream from %q: %s", requestURL, err)
+			backoffSleep()
+			// There is no sense in reloading resources on non-http errors.
+			continue
+		}
+	}
+}
+
+// readObjectUpdateStream reads Kuberntes watch events from r and updates locally cached objects according to the received events.
+func (uw *urlWatcher) readObjectUpdateStream(r io.Reader) error {
+	d := json.NewDecoder(r)
+	var we WatchEvent
+	for {
+		if err := d.Decode(&we); err != nil {
+			return err
+		}
+		o, err := uw.parseObject(we.Object)
+		if err != nil {
+			return err
+		}
+		key := o.key()
+		switch we.Type {
+		case "ADDED", "MODIFIED":
+			uw.mu.Lock()
+			uw.objectsByKey[key] = o
+			uw.mu.Unlock()
+			labels := o.getTargetLabels(uw.aw)
+			uw.mu.Lock()
+			uw.labelsByKey[key] = labels
+			uw.mu.Unlock()
+		case "DELETED":
+			uw.mu.Lock()
+			delete(uw.objectsByKey, key)
+			delete(uw.labelsByKey, key)
+			uw.mu.Unlock()
+		default:
+			return fmt.Errorf("unexpected WatchEvent type %q for role %q", we.Type, uw.role)
+		}
+	}
+}
+
+func getAPIPaths(role string, namespaces []string, selectors []Selector) []string {
+	objectName := getObjectNameByRole(role)
+	if objectName == "nodes" || len(namespaces) == 0 {
+		query := joinSelectors(role, selectors)
+		path := getAPIPath(objectName, "", query)
+		return []string{path}
+	}
+	query := joinSelectors(role, selectors)
+	paths := make([]string, len(namespaces))
+	for i, namespace := range namespaces {
+		paths[i] = getAPIPath(objectName, namespace, query)
+	}
+	return paths
+}
+
+func getAPIPath(objectName, namespace, query string) string {
+	suffix := objectName
+	if namespace != "" {
+		suffix = "namespaces/" + namespace + "/" + objectName
+	}
 	if len(query) > 0 {
-		path += "?" + query
+		suffix += "?" + query
+	}
+	if objectName == "endpointslices" {
+		return "/apis/discovery.k8s.io/v1beta1/" + suffix
+	}
+	return "/api/v1/" + suffix
+}
+
+func joinSelectors(role string, selectors []Selector) string {
+	var labelSelectors, fieldSelectors []string
+	for _, s := range selectors {
+		if s.Role != role {
+			continue
+		}
+		if s.Label != "" {
+			labelSelectors = append(labelSelectors, s.Label)
+		}
+		if s.Field != "" {
+			fieldSelectors = append(fieldSelectors, s.Field)
+		}
+	}
+	var args []string
+	if len(labelSelectors) > 0 {
+		args = append(args, "labelSelector="+url.QueryEscape(strings.Join(labelSelectors, ",")))
+	}
+	if len(fieldSelectors) > 0 {
+		args = append(args, "fieldSelector="+url.QueryEscape(strings.Join(fieldSelectors, ",")))
+	}
+	return strings.Join(args, "&")
+}
+
+func getObjectNameByRole(role string) string {
+	switch role {
+	case "node":
+		return "nodes"
+	case "pod":
+		return "pods"
+	case "service":
+		return "services"
+	case "endpoints":
+		return "endpoints"
+	case "endpointslices":
+		return "endpointslices"
+	case "ingress":
+		return "ingresses"
+	default:
+		logger.Panicf("BUG: unknonw role=%q", role)
+		return ""
+	}
+}
+
+func getObjectParsersForRole(role string) (parseObjectFunc, parseObjectListFunc) {
+	switch role {
+	case "node":
+		return parseNode, parseNodeList
+	case "pod":
+		return parsePod, parsePodList
+	case "service":
+		return parseService, parseServiceList
+	case "endpoints":
+		return parseEndpoints, parseEndpointsList
+	case "endpointslices":
+		return parseEndpointSlice, parseEndpointSliceList
+	case "ingress":
+		return parseIngress, parseIngressList
+	default:
+		logger.Panicf("BUG: unsupported role=%q", role)
+		return nil, nil
 	}
-	return cfg.client.GetAPIResponse(path)
 }
diff --git a/lib/promscrape/discovery/kubernetes/common_types.go b/lib/promscrape/discovery/kubernetes/common_types.go
index d1bc21203..be93bbb4a 100644
--- a/lib/promscrape/discovery/kubernetes/common_types.go
+++ b/lib/promscrape/discovery/kubernetes/common_types.go
@@ -1,9 +1,6 @@
 package kubernetes
 
 import (
-	"net/url"
-	"strings"
-
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discoveryutils"
 )
 
@@ -19,6 +16,16 @@ type ObjectMeta struct {
 	OwnerReferences []OwnerReference
 }
 
+func (om *ObjectMeta) key() string {
+	return om.Namespace + "/" + om.Name
+}
+
+// ListMeta is a Kubernetes list metadata
+// https://v1-17.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#listmeta-v1-meta
+type ListMeta struct {
+	ResourceVersion string
+}
+
 func (om *ObjectMeta) registerLabelsAndAnnotations(prefix string, m map[string]string) {
 	for _, lb := range om.Labels {
 		ln := discoveryutils.SanitizeLabelName(lb.Name)
@@ -47,29 +54,3 @@ type OwnerReference struct {
 type DaemonEndpoint struct {
 	Port int
 }
-
-func joinSelectors(role string, namespaces []string, selectors []Selector) string {
-	var labelSelectors, fieldSelectors []string
-	for _, ns := range namespaces {
-		fieldSelectors = append(fieldSelectors, "metadata.namespace="+ns)
-	}
-	for _, s := range selectors {
-		if s.Role != role {
-			continue
-		}
-		if s.Label != "" {
-			labelSelectors = append(labelSelectors, s.Label)
-		}
-		if s.Field != "" {
-			fieldSelectors = append(fieldSelectors, s.Field)
-		}
-	}
-	var args []string
-	if len(labelSelectors) > 0 {
-		args = append(args, "labelSelector="+url.QueryEscape(strings.Join(labelSelectors, ",")))
-	}
-	if len(fieldSelectors) > 0 {
-		args = append(args, "fieldSelector="+url.QueryEscape(strings.Join(fieldSelectors, ",")))
-	}
-	return strings.Join(args, "&")
-}
diff --git a/lib/promscrape/discovery/kubernetes/endpoints.go b/lib/promscrape/discovery/kubernetes/endpoints.go
index 6f95a2a9d..003cceca5 100644
--- a/lib/promscrape/discovery/kubernetes/endpoints.go
+++ b/lib/promscrape/discovery/kubernetes/endpoints.go
@@ -7,66 +7,36 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discoveryutils"
 )
 
-// getEndpointsLabels returns labels for k8s endpoints obtained from the given cfg.
-func getEndpointsLabels(cfg *apiConfig) ([]map[string]string, error) {
-	eps, err := getEndpoints(cfg)
-	if err != nil {
-		return nil, err
-	}
-	pods, err := getPods(cfg)
-	if err != nil {
-		return nil, err
-	}
-	svcs, err := getServices(cfg)
-	if err != nil {
-		return nil, err
-	}
-	var ms []map[string]string
-	for _, ep := range eps {
-		ms = ep.appendTargetLabels(ms, pods, svcs)
-	}
-	return ms, nil
+func (eps *Endpoints) key() string {
+	return eps.Metadata.key()
 }
 
-func getEndpoints(cfg *apiConfig) ([]Endpoints, error) {
-	if len(cfg.namespaces) == 0 {
-		return getEndpointsByPath(cfg, "/api/v1/endpoints")
+func parseEndpointsList(data []byte) (map[string]object, ListMeta, error) {
+	var epsl EndpointsList
+	if err := json.Unmarshal(data, &epsl); err != nil {
+		return nil, epsl.Metadata, fmt.Errorf("cannot unmarshal EndpointsList from %q: %w", data, err)
 	}
-	// Query /api/v1/namespaces/* for each namespace.
-	// This fixes authorization issue at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/432
-	cfgCopy := *cfg
-	namespaces := cfgCopy.namespaces
-	cfgCopy.namespaces = nil
-	cfg = &cfgCopy
-	var result []Endpoints
-	for _, ns := range namespaces {
-		path := fmt.Sprintf("/api/v1/namespaces/%s/endpoints", ns)
-		eps, err := getEndpointsByPath(cfg, path)
-		if err != nil {
-			return nil, err
-		}
-		result = append(result, eps...)
+	objectsByKey := make(map[string]object)
+	for _, eps := range epsl.Items {
+		objectsByKey[eps.key()] = eps
 	}
-	return result, nil
+	return objectsByKey, epsl.Metadata, nil
 }
 
-func getEndpointsByPath(cfg *apiConfig, path string) ([]Endpoints, error) {
-	data, err := getAPIResponse(cfg, "endpoints", path)
-	if err != nil {
-		return nil, fmt.Errorf("cannot obtain endpoints data from API server: %w", err)
-	}
-	epl, err := parseEndpointsList(data)
-	if err != nil {
-		return nil, fmt.Errorf("cannot parse endpoints response from API server: %w", err)
+func parseEndpoints(data []byte) (object, error) {
+	var eps Endpoints
+	if err := json.Unmarshal(data, &eps); err != nil {
+		return nil, err
 	}
-	return epl.Items, nil
+	return &eps, nil
 }
 
 // EndpointsList implements k8s endpoints list.
 //
 // See https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#endpointslist-v1-core
 type EndpointsList struct {
-	Items []Endpoints
+	Metadata ListMeta
+	Items    []*Endpoints
 }
 
 // Endpoints implements k8s endpoints.
@@ -115,25 +85,20 @@ type EndpointPort struct {
 	Protocol    string
 }
 
-// parseEndpointsList parses EndpointsList from data.
-func parseEndpointsList(data []byte) (*EndpointsList, error) {
-	var esl EndpointsList
-	if err := json.Unmarshal(data, &esl); err != nil {
-		return nil, fmt.Errorf("cannot unmarshal EndpointsList from %q: %w", data, err)
-	}
-	return &esl, nil
-}
-
-// appendTargetLabels appends labels for each endpoint in eps to ms and returns the result.
+// getTargetLabels returns labels for each endpoint in eps.
 //
 // See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#endpoints
-func (eps *Endpoints) appendTargetLabels(ms []map[string]string, pods []Pod, svcs []Service) []map[string]string {
-	svc := getService(svcs, eps.Metadata.Namespace, eps.Metadata.Name)
+func (eps *Endpoints) getTargetLabels(aw *apiWatcher) []map[string]string {
+	var svc *Service
+	if o := aw.getObjectByRole("service", eps.Metadata.Namespace, eps.Metadata.Name); o != nil {
+		svc = o.(*Service)
+	}
 	podPortsSeen := make(map[*Pod][]int)
+	var ms []map[string]string
 	for _, ess := range eps.Subsets {
 		for _, epp := range ess.Ports {
-			ms = appendEndpointLabelsForAddresses(ms, podPortsSeen, eps, ess.Addresses, epp, pods, svc, "true")
-			ms = appendEndpointLabelsForAddresses(ms, podPortsSeen, eps, ess.NotReadyAddresses, epp, pods, svc, "false")
+			ms = appendEndpointLabelsForAddresses(ms, aw, podPortsSeen, eps, ess.Addresses, epp, svc, "true")
+			ms = appendEndpointLabelsForAddresses(ms, aw, podPortsSeen, eps, ess.NotReadyAddresses, epp, svc, "false")
 		}
 	}
 
@@ -168,10 +133,13 @@ func (eps *Endpoints) appendTargetLabels(ms []map[string]string, pods []Pod, svc
 	return ms
 }
 
-func appendEndpointLabelsForAddresses(ms []map[string]string, podPortsSeen map[*Pod][]int, eps *Endpoints, eas []EndpointAddress, epp EndpointPort,
-	pods []Pod, svc *Service, ready string) []map[string]string {
+func appendEndpointLabelsForAddresses(ms []map[string]string, aw *apiWatcher, podPortsSeen map[*Pod][]int, eps *Endpoints,
+	eas []EndpointAddress, epp EndpointPort, svc *Service, ready string) []map[string]string {
 	for _, ea := range eas {
-		p := getPod(pods, ea.TargetRef.Namespace, ea.TargetRef.Name)
+		var p *Pod
+		if o := aw.getObjectByRole("pod", ea.TargetRef.Namespace, ea.TargetRef.Name); o != nil {
+			p = o.(*Pod)
+		}
 		m := getEndpointLabelsForAddressAndPort(podPortsSeen, eps, ea, epp, p, svc, ready)
 		ms = append(ms, m)
 	}
diff --git a/lib/promscrape/discovery/kubernetes/endpointslices.go b/lib/promscrape/discovery/kubernetes/endpointslices.go
index f1f9da645..cec87ce13 100644
--- a/lib/promscrape/discovery/kubernetes/endpointslices.go
+++ b/lib/promscrape/discovery/kubernetes/endpointslices.go
@@ -8,85 +8,48 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discoveryutils"
 )
 
-// getEndpointSlicesLabels returns labels for k8s endpointSlices obtained from the given cfg.
-func getEndpointSlicesLabels(cfg *apiConfig) ([]map[string]string, error) {
-	eps, err := getEndpointSlices(cfg)
-	if err != nil {
-		return nil, err
-	}
-	pods, err := getPods(cfg)
-	if err != nil {
-		return nil, err
-	}
-	svcs, err := getServices(cfg)
-	if err != nil {
-		return nil, err
-	}
-	var ms []map[string]string
-	for _, ep := range eps {
-		ms = ep.appendTargetLabels(ms, pods, svcs)
-	}
-
-	return ms, nil
+func (eps *EndpointSlice) key() string {
+	return eps.Metadata.key()
 }
 
-// getEndpointSlices retrieves endpointSlice with given apiConfig
-func getEndpointSlices(cfg *apiConfig) ([]EndpointSlice, error) {
-	if len(cfg.namespaces) == 0 {
-		return getEndpointSlicesByPath(cfg, "/apis/discovery.k8s.io/v1beta1/endpointslices")
+func parseEndpointSliceList(data []byte) (map[string]object, ListMeta, error) {
+	var epsl EndpointSliceList
+	if err := json.Unmarshal(data, &epsl); err != nil {
+		return nil, epsl.Metadata, fmt.Errorf("cannot unmarshal EndpointSliceList from %q: %w", data, err)
 	}
-	// Query /api/v1/namespaces/* for each namespace.
-	// This fixes authorization issue at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/432
-	cfgCopy := *cfg
-	namespaces := cfgCopy.namespaces
-	cfgCopy.namespaces = nil
-	cfg = &cfgCopy
-	var result []EndpointSlice
-	for _, ns := range namespaces {
-		path := fmt.Sprintf("/apis/discovery.k8s.io/v1beta1/namespaces/%s/endpointslices", ns)
-		eps, err := getEndpointSlicesByPath(cfg, path)
-		if err != nil {
-			return nil, err
-		}
-		result = append(result, eps...)
+	objectsByKey := make(map[string]object)
+	for _, eps := range epsl.Items {
+		objectsByKey[eps.key()] = eps
 	}
-	return result, nil
+	return objectsByKey, epsl.Metadata, nil
 }
 
-// getEndpointSlicesByPath retrieves endpointSlices from k8s api by given path
-func getEndpointSlicesByPath(cfg *apiConfig, path string) ([]EndpointSlice, error) {
-	data, err := getAPIResponse(cfg, "endpointslices", path)
-	if err != nil {
-		return nil, fmt.Errorf("cannot obtain endpointslices data from API server: %w", err)
-	}
-	epl, err := parseEndpointSlicesList(data)
-	if err != nil {
-		return nil, fmt.Errorf("cannot parse endpointslices response from API server: %w", err)
+func parseEndpointSlice(data []byte) (object, error) {
+	var eps EndpointSlice
+	if err := json.Unmarshal(data, &eps); err != nil {
+		return nil, err
 	}
-	return epl.Items, nil
-
+	return &eps, nil
 }
 
-// parseEndpointsList parses EndpointSliceList from data.
-func parseEndpointSlicesList(data []byte) (*EndpointSliceList, error) {
-	var esl EndpointSliceList
-	if err := json.Unmarshal(data, &esl); err != nil {
-		return nil, fmt.Errorf("cannot unmarshal EndpointSliceList from %q: %w", data, err)
+// getTargetLabels returns labels for eps.
+//
+// See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#endpointslices
+func (eps *EndpointSlice) getTargetLabels(aw *apiWatcher) []map[string]string {
+	var svc *Service
+	if o := aw.getObjectByRole("service", eps.Metadata.Namespace, eps.Metadata.Name); o != nil {
+		svc = o.(*Service)
 	}
-
-	return &esl, nil
-}
-
-// appendTargetLabels injects labels for endPointSlice to slice map
-// follows TargetRef for enrich labels with pod and service metadata
-func (eps *EndpointSlice) appendTargetLabels(ms []map[string]string, pods []Pod, svcs []Service) []map[string]string {
-	svc := getService(svcs, eps.Metadata.Namespace, eps.Metadata.Name)
 	podPortsSeen := make(map[*Pod][]int)
+	var ms []map[string]string
 	for _, ess := range eps.Endpoints {
-		pod := getPod(pods, ess.TargetRef.Namespace, ess.TargetRef.Name)
+		var p *Pod
+		if o := aw.getObjectByRole("pod", ess.TargetRef.Namespace, ess.TargetRef.Name); o != nil {
+			p = o.(*Pod)
+		}
 		for _, epp := range eps.Ports {
 			for _, addr := range ess.Addresses {
-				ms = append(ms, getEndpointSliceLabelsForAddressAndPort(podPortsSeen, addr, eps, ess, epp, pod, svc))
+				ms = append(ms, getEndpointSliceLabelsForAddressAndPort(podPortsSeen, addr, eps, ess, epp, p, svc))
 			}
 
 		}
@@ -121,13 +84,12 @@ func (eps *EndpointSlice) appendTargetLabels(ms []map[string]string, pods []Pod,
 		}
 	}
 	return ms
-
 }
 
 // getEndpointSliceLabelsForAddressAndPort gets labels for endpointSlice
 // from  address, Endpoint and EndpointPort
 // enriches labels with TargetRef
-// pod appended to seen Ports
+// p appended to seen Ports
 // if TargetRef matches
 func getEndpointSliceLabelsForAddressAndPort(podPortsSeen map[*Pod][]int, addr string, eps *EndpointSlice, ea Endpoint, epp EndpointPort, p *Pod, svc *Service) map[string]string {
 	m := getEndpointSliceLabels(eps, addr, ea, epp)
@@ -186,7 +148,8 @@ func getEndpointSliceLabels(eps *EndpointSlice, addr string, ea Endpoint, epp En
 // that groups service endpoints slices.
 // https://v1-17.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#endpointslice-v1beta1-discovery-k8s-io
 type EndpointSliceList struct {
-	Items []EndpointSlice
+	Metadata ListMeta
+	Items    []*EndpointSlice
 }
 
 // EndpointSlice - implements kubernetes endpoint slice.
diff --git a/lib/promscrape/discovery/kubernetes/ingress.go b/lib/promscrape/discovery/kubernetes/ingress.go
index 5aad6d903..38aef79d3 100644
--- a/lib/promscrape/discovery/kubernetes/ingress.go
+++ b/lib/promscrape/discovery/kubernetes/ingress.go
@@ -5,58 +5,36 @@ import (
 	"fmt"
 )
 
-// getIngressesLabels returns labels for k8s ingresses obtained from the given cfg.
-func getIngressesLabels(cfg *apiConfig) ([]map[string]string, error) {
-	igs, err := getIngresses(cfg)
-	if err != nil {
-		return nil, err
-	}
-	var ms []map[string]string
-	for _, ig := range igs {
-		ms = ig.appendTargetLabels(ms)
-	}
-	return ms, nil
+func (ig *Ingress) key() string {
+	return ig.Metadata.key()
 }
 
-func getIngresses(cfg *apiConfig) ([]Ingress, error) {
-	if len(cfg.namespaces) == 0 {
-		return getIngressesByPath(cfg, "/apis/extensions/v1beta1/ingresses")
+func parseIngressList(data []byte) (map[string]object, ListMeta, error) {
+	var igl IngressList
+	if err := json.Unmarshal(data, &igl); err != nil {
+		return nil, igl.Metadata, fmt.Errorf("cannot unmarshal IngressList from %q: %w", data, err)
 	}
-	// Query /api/v1/namespaces/* for each namespace.
-	// This fixes authorization issue at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/432
-	cfgCopy := *cfg
-	namespaces := cfgCopy.namespaces
-	cfgCopy.namespaces = nil
-	cfg = &cfgCopy
-	var result []Ingress
-	for _, ns := range namespaces {
-		path := fmt.Sprintf("/apis/extensions/v1beta1/namespaces/%s/ingresses", ns)
-		igs, err := getIngressesByPath(cfg, path)
-		if err != nil {
-			return nil, err
-		}
-		result = append(result, igs...)
+	objectsByKey := make(map[string]object)
+	for _, ig := range igl.Items {
+		objectsByKey[ig.key()] = ig
 	}
-	return result, nil
+	return objectsByKey, igl.Metadata, nil
 }
 
-func getIngressesByPath(cfg *apiConfig, path string) ([]Ingress, error) {
-	data, err := getAPIResponse(cfg, "ingress", path)
-	if err != nil {
-		return nil, fmt.Errorf("cannot obtain ingresses data from API server: %w", err)
-	}
-	igl, err := parseIngressList(data)
-	if err != nil {
-		return nil, fmt.Errorf("cannot parse ingresses response from API server: %w", err)
+func parseIngress(data []byte) (object, error) {
+	var ig Ingress
+	if err := json.Unmarshal(data, &ig); err != nil {
+		return nil, err
 	}
-	return igl.Items, nil
+	return &ig, nil
 }
 
 // IngressList represents ingress list in k8s.
 //
 // See https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#ingresslist-v1beta1-extensions
 type IngressList struct {
-	Items []Ingress
+	Metadata ListMeta
+	Items    []*Ingress
 }
 
 // Ingress represents ingress in k8s.
@@ -104,25 +82,17 @@ type HTTPIngressPath struct {
 	Path string
 }
 
-// parseIngressList parses IngressList from data.
-func parseIngressList(data []byte) (*IngressList, error) {
-	var il IngressList
-	if err := json.Unmarshal(data, &il); err != nil {
-		return nil, fmt.Errorf("cannot unmarshal IngressList from %q: %w", data, err)
-	}
-	return &il, nil
-}
-
-// appendTargetLabels appends labels for Ingress ig to ms and returns the result.
+// getTargetLabels returns labels for ig.
 //
 // See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#ingress
-func (ig *Ingress) appendTargetLabels(ms []map[string]string) []map[string]string {
+func (ig *Ingress) getTargetLabels(aw *apiWatcher) []map[string]string {
 	tlsHosts := make(map[string]bool)
 	for _, tls := range ig.Spec.TLS {
 		for _, host := range tls.Hosts {
 			tlsHosts[host] = true
 		}
 	}
+	var ms []map[string]string
 	for _, r := range ig.Spec.Rules {
 		paths := getIngressRulePaths(r.HTTP.Paths)
 		scheme := "http"
diff --git a/lib/promscrape/discovery/kubernetes/kubernetes.go b/lib/promscrape/discovery/kubernetes/kubernetes.go
index 2c7a48995..504fef616 100644
--- a/lib/promscrape/discovery/kubernetes/kubernetes.go
+++ b/lib/promscrape/discovery/kubernetes/kubernetes.go
@@ -38,25 +38,15 @@ type Selector struct {
 }
 
 // GetLabels returns labels for the given sdc and baseDir.
-func GetLabels(sdc *SDConfig, baseDir string) ([]map[string]string, error) {
+func (sdc *SDConfig) GetLabels(baseDir string) ([]map[string]string, error) {
 	cfg, err := getAPIConfig(sdc, baseDir)
 	if err != nil {
 		return nil, fmt.Errorf("cannot create API config: %w", err)
 	}
 	switch sdc.Role {
-	case "node":
-		return getNodesLabels(cfg)
-	case "service":
-		return getServicesLabels(cfg)
-	case "pod":
-		return getPodsLabels(cfg)
-	case "endpoints":
-		return getEndpointsLabels(cfg)
-	case "endpointslices":
-		return getEndpointSlicesLabels(cfg)
-	case "ingress":
-		return getIngressesLabels(cfg)
+	case "node", "pod", "service", "endpoints", "endpointslices", "ingress":
+		return cfg.aw.getLabelsForRole(sdc.Role), nil
 	default:
-		return nil, fmt.Errorf("unexpected `role`: %q; must be one of `node`, `service`, `pod`, `endpoints` or `ingress`; skipping it", sdc.Role)
+		return nil, fmt.Errorf("unexpected `role`: %q; must be one of `node`, `pod`, `service`, `endpoints`, `endpointslices` or `ingress`; skipping it", sdc.Role)
 	}
 }
diff --git a/lib/promscrape/discovery/kubernetes/node.go b/lib/promscrape/discovery/kubernetes/node.go
index 9a584c67e..653a99e73 100644
--- a/lib/promscrape/discovery/kubernetes/node.go
+++ b/lib/promscrape/discovery/kubernetes/node.go
@@ -7,29 +7,37 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discoveryutils"
 )
 
-// getNodesLabels returns labels for k8s nodes obtained from the given cfg.
-func getNodesLabels(cfg *apiConfig) ([]map[string]string, error) {
-	data, err := getAPIResponse(cfg, "node", "/api/v1/nodes")
-	if err != nil {
-		return nil, fmt.Errorf("cannot obtain nodes data from API server: %w", err)
-	}
-	nl, err := parseNodeList(data)
-	if err != nil {
-		return nil, fmt.Errorf("cannot parse nodes response from API server: %w", err)
+// getNodesLabels returns labels for k8s nodes obtained from the given cfg
+func (n *Node) key() string {
+	return n.Metadata.key()
+}
+
+func parseNodeList(data []byte) (map[string]object, ListMeta, error) {
+	var nl NodeList
+	if err := json.Unmarshal(data, &nl); err != nil {
+		return nil, nl.Metadata, fmt.Errorf("cannot unmarshal NodeList from %q: %w", data, err)
 	}
-	var ms []map[string]string
+	objectsByKey := make(map[string]object)
 	for _, n := range nl.Items {
-		// Do not apply namespaces, since they are missing in nodes.
-		ms = n.appendTargetLabels(ms)
+		objectsByKey[n.key()] = n
 	}
-	return ms, nil
+	return objectsByKey, nl.Metadata, nil
+}
+
+func parseNode(data []byte) (object, error) {
+	var n Node
+	if err := json.Unmarshal(data, &n); err != nil {
+		return nil, err
+	}
+	return &n, nil
 }
 
 // NodeList represents NodeList from k8s API.
 //
 // See https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#nodelist-v1-core
 type NodeList struct {
-	Items []Node
+	Metadata ListMeta
+	Items    []*Node
 }
 
 // Node represents Node from k8s API.
@@ -63,23 +71,14 @@ type NodeDaemonEndpoints struct {
 	KubeletEndpoint DaemonEndpoint
 }
 
-// parseNodeList parses NodeList from data.
-func parseNodeList(data []byte) (*NodeList, error) {
-	var nl NodeList
-	if err := json.Unmarshal(data, &nl); err != nil {
-		return nil, fmt.Errorf("cannot unmarshal NodeList from %q: %w", data, err)
-	}
-	return &nl, nil
-}
-
-// appendTargetLabels appends labels for the given Node n to ms and returns the result.
+// getTargetLabels returs labels for the given n.
 //
 // See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#node
-func (n *Node) appendTargetLabels(ms []map[string]string) []map[string]string {
+func (n *Node) getTargetLabels(aw *apiWatcher) []map[string]string {
 	addr := getNodeAddr(n.Status.Addresses)
 	if len(addr) == 0 {
 		// Skip node without address
-		return ms
+		return nil
 	}
 	addr = discoveryutils.JoinHostPort(addr, n.Status.DaemonEndpoints.KubeletEndpoint.Port)
 	m := map[string]string{
@@ -97,8 +96,7 @@ func (n *Node) appendTargetLabels(ms []map[string]string) []map[string]string {
 		ln := discoveryutils.SanitizeLabelName(a.Type)
 		m["__meta_kubernetes_node_address_"+ln] = a.Address
 	}
-	ms = append(ms, m)
-	return ms
+	return []map[string]string{m}
 }
 
 func getNodeAddr(nas []NodeAddress) string {
diff --git a/lib/promscrape/discovery/kubernetes/pod.go b/lib/promscrape/discovery/kubernetes/pod.go
index 3bc3495b0..80864d25e 100644
--- a/lib/promscrape/discovery/kubernetes/pod.go
+++ b/lib/promscrape/discovery/kubernetes/pod.go
@@ -9,58 +9,36 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discoveryutils"
 )
 
-// getPodsLabels returns labels for k8s pods obtained from the given cfg
-func getPodsLabels(cfg *apiConfig) ([]map[string]string, error) {
-	pods, err := getPods(cfg)
-	if err != nil {
-		return nil, err
-	}
-	var ms []map[string]string
-	for _, p := range pods {
-		ms = p.appendTargetLabels(ms)
-	}
-	return ms, nil
+func (p *Pod) key() string {
+	return p.Metadata.key()
 }
 
-func getPods(cfg *apiConfig) ([]Pod, error) {
-	if len(cfg.namespaces) == 0 {
-		return getPodsByPath(cfg, "/api/v1/pods")
+func parsePodList(data []byte) (map[string]object, ListMeta, error) {
+	var pl PodList
+	if err := json.Unmarshal(data, &pl); err != nil {
+		return nil, pl.Metadata, fmt.Errorf("cannot unmarshal PodList from %q: %w", data, err)
 	}
-	// Query /api/v1/namespaces/* for each namespace.
-	// This fixes authorization issue at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/432
-	cfgCopy := *cfg
-	namespaces := cfgCopy.namespaces
-	cfgCopy.namespaces = nil
-	cfg = &cfgCopy
-	var result []Pod
-	for _, ns := range namespaces {
-		path := fmt.Sprintf("/api/v1/namespaces/%s/pods", ns)
-		pods, err := getPodsByPath(cfg, path)
-		if err != nil {
-			return nil, err
-		}
-		result = append(result, pods...)
+	objectsByKey := make(map[string]object)
+	for _, p := range pl.Items {
+		objectsByKey[p.key()] = p
 	}
-	return result, nil
+	return objectsByKey, pl.Metadata, nil
 }
 
-func getPodsByPath(cfg *apiConfig, path string) ([]Pod, error) {
-	data, err := getAPIResponse(cfg, "pod", path)
-	if err != nil {
-		return nil, fmt.Errorf("cannot obtain pods data from API server: %w", err)
-	}
-	pl, err := parsePodList(data)
-	if err != nil {
-		return nil, fmt.Errorf("cannot parse pods response from API server: %w", err)
+func parsePod(data []byte) (object, error) {
+	var p Pod
+	if err := json.Unmarshal(data, &p); err != nil {
+		return nil, err
 	}
-	return pl.Items, nil
+	return &p, nil
 }
 
 // PodList implements k8s pod list.
 //
 // See https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#podlist-v1-core
 type PodList struct {
-	Items []Pod
+	Metadata ListMeta
+	Items    []*Pod
 }
 
 // Pod implements k8s pod.
@@ -114,23 +92,15 @@ type PodCondition struct {
 	Status string
 }
 
-// parsePodList parses PodList from data.
-func parsePodList(data []byte) (*PodList, error) {
-	var pl PodList
-	if err := json.Unmarshal(data, &pl); err != nil {
-		return nil, fmt.Errorf("cannot unmarshal PodList from %q: %w", data, err)
-	}
-	return &pl, nil
-}
-
-// appendTargetLabels appends labels for each port of the given Pod p to ms and returns the result.
+// getTargetLabels returns labels for each port of the given p.
 //
 // See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#pod
-func (p *Pod) appendTargetLabels(ms []map[string]string) []map[string]string {
+func (p *Pod) getTargetLabels(aw *apiWatcher) []map[string]string {
 	if len(p.Status.PodIP) == 0 {
 		// Skip pod without IP
-		return ms
+		return nil
 	}
+	var ms []map[string]string
 	ms = appendPodLabels(ms, p, p.Spec.Containers, "false")
 	ms = appendPodLabels(ms, p, p.Spec.InitContainers, "true")
 	return ms
@@ -210,13 +180,3 @@ func getPodReadyStatus(conds []PodCondition) string {
 	}
 	return "unknown"
 }
-
-func getPod(pods []Pod, namespace, name string) *Pod {
-	for i := range pods {
-		pod := &pods[i]
-		if pod.Metadata.Name == name && pod.Metadata.Namespace == namespace {
-			return pod
-		}
-	}
-	return nil
-}
diff --git a/lib/promscrape/discovery/kubernetes/service.go b/lib/promscrape/discovery/kubernetes/service.go
index 5ce94a4c9..c129816e6 100644
--- a/lib/promscrape/discovery/kubernetes/service.go
+++ b/lib/promscrape/discovery/kubernetes/service.go
@@ -7,58 +7,36 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discoveryutils"
 )
 
-// getServicesLabels returns labels for k8s services obtained from the given cfg.
-func getServicesLabels(cfg *apiConfig) ([]map[string]string, error) {
-	svcs, err := getServices(cfg)
-	if err != nil {
-		return nil, err
-	}
-	var ms []map[string]string
-	for _, svc := range svcs {
-		ms = svc.appendTargetLabels(ms)
-	}
-	return ms, nil
+func (s *Service) key() string {
+	return s.Metadata.key()
 }
 
-func getServices(cfg *apiConfig) ([]Service, error) {
-	if len(cfg.namespaces) == 0 {
-		return getServicesByPath(cfg, "/api/v1/services")
+func parseServiceList(data []byte) (map[string]object, ListMeta, error) {
+	var sl ServiceList
+	if err := json.Unmarshal(data, &sl); err != nil {
+		return nil, sl.Metadata, fmt.Errorf("cannot unmarshal ServiceList from %q: %w", data, err)
 	}
-	// Query /api/v1/namespaces/* for each namespace.
-	// This fixes authorization issue at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/432
-	cfgCopy := *cfg
-	namespaces := cfgCopy.namespaces
-	cfgCopy.namespaces = nil
-	cfg = &cfgCopy
-	var result []Service
-	for _, ns := range namespaces {
-		path := fmt.Sprintf("/api/v1/namespaces/%s/services", ns)
-		svcs, err := getServicesByPath(cfg, path)
-		if err != nil {
-			return nil, err
-		}
-		result = append(result, svcs...)
+	objectsByKey := make(map[string]object)
+	for _, s := range sl.Items {
+		objectsByKey[s.key()] = s
 	}
-	return result, nil
+	return objectsByKey, sl.Metadata, nil
 }
 
-func getServicesByPath(cfg *apiConfig, path string) ([]Service, error) {
-	data, err := getAPIResponse(cfg, "service", path)
-	if err != nil {
-		return nil, fmt.Errorf("cannot obtain services data from API server: %w", err)
-	}
-	sl, err := parseServiceList(data)
-	if err != nil {
-		return nil, fmt.Errorf("cannot parse services response from API server: %w", err)
+func parseService(data []byte) (object, error) {
+	var s Service
+	if err := json.Unmarshal(data, &s); err != nil {
+		return nil, err
 	}
-	return sl.Items, nil
+	return &s, nil
 }
 
 // ServiceList is k8s service list.
 //
 // See https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#servicelist-v1-core
 type ServiceList struct {
-	Items []Service
+	Metadata ListMeta
+	Items    []*Service
 }
 
 // Service is k8s service.
@@ -88,20 +66,12 @@ type ServicePort struct {
 	Port     int
 }
 
-// parseServiceList parses ServiceList from data.
-func parseServiceList(data []byte) (*ServiceList, error) {
-	var sl ServiceList
-	if err := json.Unmarshal(data, &sl); err != nil {
-		return nil, fmt.Errorf("cannot unmarshal ServiceList from %q: %w", data, err)
-	}
-	return &sl, nil
-}
-
-// appendTargetLabels appends labels for each port of the given Service s to ms and returns the result.
+// getTargetLabels returns labels for each port of the given s.
 //
 // See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#service
-func (s *Service) appendTargetLabels(ms []map[string]string) []map[string]string {
+func (s *Service) getTargetLabels(aw *apiWatcher) []map[string]string {
 	host := fmt.Sprintf("%s.%s.svc", s.Metadata.Name, s.Metadata.Namespace)
+	var ms []map[string]string
 	for _, sp := range s.Spec.Ports {
 		addr := discoveryutils.JoinHostPort(host, sp.Port)
 		m := map[string]string{
@@ -126,13 +96,3 @@ func (s *Service) appendCommonLabels(m map[string]string) {
 	}
 	s.Metadata.registerLabelsAndAnnotations("__meta_kubernetes_service", m)
 }
-
-func getService(svcs []Service, namespace, name string) *Service {
-	for i := range svcs {
-		svc := &svcs[i]
-		if svc.Metadata.Name == name && svc.Metadata.Namespace == namespace {
-			return svc
-		}
-	}
-	return nil
-}
diff --git a/lib/promscrape/discovery/openstack/openstack.go b/lib/promscrape/discovery/openstack/openstack.go
index 0a1e9bd1d..f4b17bac3 100644
--- a/lib/promscrape/discovery/openstack/openstack.go
+++ b/lib/promscrape/discovery/openstack/openstack.go
@@ -31,8 +31,8 @@ type SDConfig struct {
 	Availability string              `yaml:"availability,omitempty"`
 }
 
-// GetLabels returns gce labels according to sdc.
-func GetLabels(sdc *SDConfig, baseDir string) ([]map[string]string, error) {
+// GetLabels returns OpenStack labels according to sdc.
+func (sdc *SDConfig) GetLabels(baseDir string) ([]map[string]string, error) {
 	cfg, err := getAPIConfig(sdc, baseDir)
 	if err != nil {
 		return nil, fmt.Errorf("cannot get API config: %w", err)
diff --git a/vendor/github.com/VictoriaMetrics/metricsql/transform.go b/vendor/github.com/VictoriaMetrics/metricsql/transform.go
index c1196badf..f8dd7cc84 100644
--- a/vendor/github.com/VictoriaMetrics/metricsql/transform.go
+++ b/vendor/github.com/VictoriaMetrics/metricsql/transform.go
@@ -10,6 +10,7 @@ var transformFuncs = map[string]bool{
 	"abs":                true,
 	"absent":             true,
 	"ceil":               true,
+	"clamp":              true,
 	"clamp_max":          true,
 	"clamp_min":          true,
 	"day_of_month":       true,
@@ -28,6 +29,7 @@ var transformFuncs = map[string]bool{
 	"month":              true,
 	"round":              true,
 	"scalar":             true,
+	"sign":               true,
 	"sort":               true,
 	"sort_desc":          true,
 	"sqrt":               true,
diff --git a/vendor/modules.txt b/vendor/modules.txt
index fa5b55f16..dcecec4b1 100644
--- a/vendor/modules.txt
+++ b/vendor/modules.txt
@@ -16,7 +16,7 @@ github.com/VictoriaMetrics/fasthttp/fasthttputil
 github.com/VictoriaMetrics/fasthttp/stackless
 # github.com/VictoriaMetrics/metrics v1.15.0
 github.com/VictoriaMetrics/metrics
-# github.com/VictoriaMetrics/metricsql v0.11.0
+# github.com/VictoriaMetrics/metricsql v0.12.0
 github.com/VictoriaMetrics/metricsql
 github.com/VictoriaMetrics/metricsql/binaryop
 # github.com/VividCortex/ewma v1.1.1


Test patch:
diff --git a/app/vmselect/promql/exec_test.go b/app/vmselect/promql/exec_test.go
index f12fbbda0..9a3664064 100644
--- a/app/vmselect/promql/exec_test.go
+++ b/app/vmselect/promql/exec_test.go
@@ -673,6 +673,17 @@ func TestExecSuccess(t *testing.T) {
 		resultExpected := []netstorage.Result{r}
 		f(q, resultExpected)
 	})
+	t.Run("clamp(time(), 1400, 1800)", func(t *testing.T) {
+		t.Parallel()
+		q := `clamp(time(), 1400, 1800)`
+		r := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{1400, 1400, 1400, 1600, 1800, 1800},
+			Timestamps: timestampsExpected,
+		}
+		resultExpected := []netstorage.Result{r}
+		f(q, resultExpected)
+	})
 	t.Run("clamp_max(time(), 1400)", func(t *testing.T) {
 		t.Parallel()
 		q := `clamp_max(time(), 1400)`
@@ -1716,6 +1727,17 @@ func TestExecSuccess(t *testing.T) {
 		resultExpected := []netstorage.Result{r1, r2}
 		f(q, resultExpected)
 	})
+	t.Run(`sign(time()-1400)`, func(t *testing.T) {
+		t.Parallel()
+		q := `sign(time()-1400)`
+		r := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{-1, -1, 0, 1, 1, 1},
+			Timestamps: timestampsExpected,
+		}
+		resultExpected := []netstorage.Result{r}
+		f(q, resultExpected)
+	})
 	t.Run(`round(time()/1e3)`, func(t *testing.T) {
 		t.Parallel()
 		q := `round(time()/1e3)`
@@ -2763,7 +2785,7 @@ func TestExecSuccess(t *testing.T) {
 		resultExpected := []netstorage.Result{}
 		f(q, resultExpected)
 	})
-	t.Run(`histogram_quantile(single-value-inf-le)`, func(t *testing.T) {
+	t.Run(`histogram_quantile(zero-value-inf-le)`, func(t *testing.T) {
 		t.Parallel()
 		q := `histogram_quantile(0.6, (
 			label_set(100, "le", "+Inf"),
@@ -3315,14 +3337,14 @@ func TestExecSuccess(t *testing.T) {
 		)))`
 		r1 := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{52, 52, 52, 52, 52, 52},
+			Values:     []float64{9, 9, 9, 9, 9, 9},
 			Timestamps: timestampsExpected,
 		}
 		r1.MetricName.MetricGroup = []byte("metric")
 		r1.MetricName.Tags = []storage.Tag{
 			{
 				Key:   []byte("le"),
-				Value: []byte("200"),
+				Value: []byte("10"),
 			},
 			{
 				Key:   []byte("x"),
@@ -3331,11 +3353,27 @@ func TestExecSuccess(t *testing.T) {
 		}
 		r2 := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{100, 100, 100, 100, 100, 100},
+			Values:     []float64{98, 98, 98, 98, 98, 98},
 			Timestamps: timestampsExpected,
 		}
 		r2.MetricName.MetricGroup = []byte("metric")
 		r2.MetricName.Tags = []storage.Tag{
+			{
+				Key:   []byte("le"),
+				Value: []byte("300"),
+			},
+			{
+				Key:   []byte("x"),
+				Value: []byte("y"),
+			},
+		}
+		r3 := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{100, 100, 100, 100, 100, 100},
+			Timestamps: timestampsExpected,
+		}
+		r3.MetricName.MetricGroup = []byte("metric")
+		r3.MetricName.Tags = []storage.Tag{
 			{
 				Key:   []byte("le"),
 				Value: []byte("inf"),
@@ -3345,7 +3383,7 @@ func TestExecSuccess(t *testing.T) {
 				Value: []byte("y"),
 			},
 		}
-		resultExpected := []netstorage.Result{r1, r2}
+		resultExpected := []netstorage.Result{r1, r2, r3}
 		f(q, resultExpected)
 	})
 	t.Run(`prometheus_buckets(missing-vmrange)`, func(t *testing.T) {
@@ -4153,11 +4191,11 @@ func TestExecSuccess(t *testing.T) {
 	})
 	t.Run(`sum(histogram_over_time) by (vmrange)`, func(t *testing.T) {
 		t.Parallel()
-		q := `sort_desc(
+		q := `sort_by_label(
 			buckets_limit(
 				3,
 				sum(histogram_over_time(alias(label_set(rand(0)*1.3+1.1, "foo", "bar"), "xxx")[200s:5s])) by (vmrange)
-			)
+			), "le"
 		)`
 		r1 := netstorage.Result{
 			MetricName: metricNameExpected,
@@ -4172,24 +4210,24 @@ func TestExecSuccess(t *testing.T) {
 		}
 		r2 := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{24, 22, 26, 25, 24, 24},
+			Values:     []float64{0, 0, 0, 0, 0, 0},
 			Timestamps: timestampsExpected,
 		}
 		r2.MetricName.Tags = []storage.Tag{
 			{
 				Key:   []byte("le"),
-				Value: []byte("1.896e+00"),
+				Value: []byte("1.000e+00"),
 			},
 		}
 		r3 := netstorage.Result{
 			MetricName: metricNameExpected,
-			Values:     []float64{11, 12, 11, 7, 11, 13},
+			Values:     []float64{40, 40, 40, 40, 40, 40},
 			Timestamps: timestampsExpected,
 		}
 		r3.MetricName.Tags = []storage.Tag{
 			{
 				Key:   []byte("le"),
-				Value: []byte("1.468e+00"),
+				Value: []byte("2.448e+00"),
 			},
 		}
 		resultExpected := []netstorage.Result{r1, r2, r3}
@@ -6307,6 +6345,7 @@ func TestExecError(t *testing.T) {
 	f(`abs()`)
 	f(`abs(1,2)`)
 	f(`absent(1, 2)`)
+	f(`clamp()`)
 	f(`clamp_max()`)
 	f(`clamp_min(1,2,3)`)
 	f(`hour(1,2)`)
@@ -6323,6 +6362,7 @@ func TestExecError(t *testing.T) {
 	f(`label_mismatch()`)
 	f(`round()`)
 	f(`round(1,2,3)`)
+	f(`sign()`)
 	f(`scalar()`)
 	f(`sort(1,2)`)
 	f(`sort_desc()`)
diff --git a/lib/promrelabel/relabel_test.go b/lib/promrelabel/relabel_test.go
index cc06887e7..c8033877a 100644
--- a/lib/promrelabel/relabel_test.go
+++ b/lib/promrelabel/relabel_test.go
@@ -713,7 +713,7 @@ func TestApplyRelabelConfigs(t *testing.T) {
 			},
 		})
 	})
-	t.Run("labeldrop-regexp", func(t *testing.T) {
+	t.Run("labeldrop-prefix", func(t *testing.T) {
 		f(`
 - action: labeldrop
   regex: "dropme.*"
@@ -730,7 +730,49 @@ func TestApplyRelabelConfigs(t *testing.T) {
 		})
 		f(`
 - action: labeldrop
-  regex: "dropme.*"
+  regex: "dropme(.+)"
+`, []prompbmarshal.Label{
+			{
+				Name:  "xxx",
+				Value: "yyy",
+			},
+			{
+				Name:  "dropme-please",
+				Value: "aaa",
+			},
+			{
+				Name:  "foo",
+				Value: "bar",
+			},
+		}, false, []prompbmarshal.Label{
+			{
+				Name:  "foo",
+				Value: "bar",
+			},
+			{
+				Name:  "xxx",
+				Value: "yyy",
+			},
+		})
+	})
+	t.Run("labeldrop-regexp", func(t *testing.T) {
+		f(`
+- action: labeldrop
+  regex: ".*dropme.*"
+`, []prompbmarshal.Label{
+			{
+				Name:  "aaa",
+				Value: "bbb",
+			},
+		}, true, []prompbmarshal.Label{
+			{
+				Name:  "aaa",
+				Value: "bbb",
+			},
+		})
+		f(`
+- action: labeldrop
+  regex: ".*dropme.*"
 `, []prompbmarshal.Label{
 			{
 				Name:  "xxx",
diff --git a/lib/promrelabel/relabel_timing_test.go b/lib/promrelabel/relabel_timing_test.go
index 4904bb0ef..bdaf05986 100644
--- a/lib/promrelabel/relabel_timing_test.go
+++ b/lib/promrelabel/relabel_timing_test.go
@@ -495,10 +495,44 @@ func BenchmarkApplyRelabelConfigs(b *testing.B) {
 			}
 		})
 	})
-	b.Run("labeldrop-match-regexp", func(b *testing.B) {
+	b.Run("labeldrop-match-prefix", func(b *testing.B) {
 		pcs := mustParseRelabelConfigs(`
 - action: labeldrop
   regex: "id.*"
+`)
+		labelsOrig := []prompbmarshal.Label{
+			{
+				Name:  "__name__",
+				Value: "metric",
+			},
+			{
+				Name:  "id",
+				Value: "foobar-random-string-here",
+			},
+		}
+		b.ReportAllocs()
+		b.SetBytes(1)
+		b.RunParallel(func(pb *testing.PB) {
+			var labels []prompbmarshal.Label
+			for pb.Next() {
+				labels = append(labels[:0], labelsOrig...)
+				labels = pcs.Apply(labels, 0, true)
+				if len(labels) != 1 {
+					panic(fmt.Errorf("unexpected number of labels; got %d; want %d; labels:\n%#v", len(labels), 1, labels))
+				}
+				if labels[0].Name != "__name__" {
+					panic(fmt.Errorf("unexpected label name; got %q; want %q", labels[0].Name, "__name__"))
+				}
+				if labels[0].Value != "metric" {
+					panic(fmt.Errorf("unexpected label value; got %q; want %q", labels[0].Value, "metric"))
+				}
+			}
+		})
+	})
+	b.Run("labeldrop-match-regexp", func(b *testing.B) {
+		pcs := mustParseRelabelConfigs(`
+- action: labeldrop
+  regex: ".*id.*"
 `)
 		labelsOrig := []prompbmarshal.Label{
 			{
@@ -591,10 +625,44 @@ func BenchmarkApplyRelabelConfigs(b *testing.B) {
 			}
 		})
 	})
-	b.Run("labelkeep-match-regexp", func(b *testing.B) {
+	b.Run("labelkeep-match-prefix", func(b *testing.B) {
 		pcs := mustParseRelabelConfigs(`
 - action: labelkeep
   regex: "id.*"
+`)
+		labelsOrig := []prompbmarshal.Label{
+			{
+				Name:  "__name__",
+				Value: "metric",
+			},
+			{
+				Name:  "id",
+				Value: "foobar-random-string-here",
+			},
+		}
+		b.ReportAllocs()
+		b.SetBytes(1)
+		b.RunParallel(func(pb *testing.PB) {
+			var labels []prompbmarshal.Label
+			for pb.Next() {
+				labels = append(labels[:0], labelsOrig...)
+				labels = pcs.Apply(labels, 0, true)
+				if len(labels) != 1 {
+					panic(fmt.Errorf("unexpected number of labels; got %d; want %d; labels:\n%#v", len(labels), 1, labels))
+				}
+				if labels[0].Name != "id" {
+					panic(fmt.Errorf("unexpected label name; got %q; want %q", labels[0].Name, "id"))
+				}
+				if labels[0].Value != "foobar-random-string-here" {
+					panic(fmt.Errorf("unexpected label value; got %q; want %q", labels[0].Value, "foobar-random-string-here"))
+				}
+			}
+		})
+	})
+	b.Run("labelkeep-match-regexp", func(b *testing.B) {
+		pcs := mustParseRelabelConfigs(`
+- action: labelkeep
+  regex: ".*id.*"
 `)
 		labelsOrig := []prompbmarshal.Label{
 			{
diff --git a/lib/promscrape/discovery/kubernetes/api_test.go b/lib/promscrape/discovery/kubernetes/api_test.go
new file mode 100644
index 000000000..1a21eb70d
--- /dev/null
+++ b/lib/promscrape/discovery/kubernetes/api_test.go
@@ -0,0 +1,162 @@
+package kubernetes
+
+import (
+	"reflect"
+	"testing"
+)
+
+func TestGetAPIPaths(t *testing.T) {
+	f := func(role string, namespaces []string, selectors []Selector, expectedPaths []string) {
+		t.Helper()
+		paths := getAPIPaths(role, namespaces, selectors)
+		if !reflect.DeepEqual(paths, expectedPaths) {
+			t.Fatalf("unexpected paths; got\n%q\nwant\n%q", paths, expectedPaths)
+		}
+	}
+
+	// role=node
+	f("node", nil, nil, []string{"/api/v1/nodes"})
+	f("node", []string{"foo", "bar"}, nil, []string{"/api/v1/nodes"})
+	f("node", nil, []Selector{
+		{
+			Role:  "pod",
+			Label: "foo",
+			Field: "bar",
+		},
+	}, []string{"/api/v1/nodes"})
+	f("node", nil, []Selector{
+		{
+			Role:  "node",
+			Label: "foo",
+			Field: "bar",
+		},
+	}, []string{"/api/v1/nodes?labelSelector=foo&fieldSelector=bar"})
+	f("node", []string{"x", "y"}, []Selector{
+		{
+			Role:  "node",
+			Label: "foo",
+			Field: "bar",
+		},
+	}, []string{"/api/v1/nodes?labelSelector=foo&fieldSelector=bar"})
+
+	// role=pod
+	f("pod", nil, nil, []string{"/api/v1/pods"})
+	f("pod", []string{"foo", "bar"}, nil, []string{
+		"/api/v1/namespaces/foo/pods",
+		"/api/v1/namespaces/bar/pods",
+	})
+	f("pod", nil, []Selector{
+		{
+			Role:  "node",
+			Label: "foo",
+		},
+	}, []string{"/api/v1/pods"})
+	f("pod", nil, []Selector{
+		{
+			Role:  "pod",
+			Label: "foo",
+		},
+		{
+			Role:  "pod",
+			Label: "x",
+			Field: "y",
+		},
+	}, []string{"/api/v1/pods?labelSelector=foo%2Cx&fieldSelector=y"})
+	f("pod", []string{"x", "y"}, []Selector{
+		{
+			Role:  "pod",
+			Label: "foo",
+		},
+		{
+			Role:  "pod",
+			Label: "x",
+			Field: "y",
+		},
+	}, []string{
+		"/api/v1/namespaces/x/pods?labelSelector=foo%2Cx&fieldSelector=y",
+		"/api/v1/namespaces/y/pods?labelSelector=foo%2Cx&fieldSelector=y",
+	})
+
+	// role=service
+	f("service", nil, nil, []string{"/api/v1/services"})
+	f("service", []string{"x", "y"}, nil, []string{
+		"/api/v1/namespaces/x/services",
+		"/api/v1/namespaces/y/services",
+	})
+	f("service", nil, []Selector{
+		{
+			Role:  "node",
+			Label: "foo",
+		},
+		{
+			Role:  "service",
+			Field: "bar",
+		},
+	}, []string{"/api/v1/services?fieldSelector=bar"})
+	f("service", []string{"x", "y"}, []Selector{
+		{
+			Role:  "service",
+			Label: "abc=de",
+		},
+	}, []string{
+		"/api/v1/namespaces/x/services?labelSelector=abc%3Dde",
+		"/api/v1/namespaces/y/services?labelSelector=abc%3Dde",
+	})
+
+	// role=endpoints
+	f("endpoints", nil, nil, []string{"/api/v1/endpoints"})
+	f("endpoints", []string{"x", "y"}, nil, []string{
+		"/api/v1/namespaces/x/endpoints",
+		"/api/v1/namespaces/y/endpoints",
+	})
+	f("endpoints", []string{"x", "y"}, []Selector{
+		{
+			Role:  "endpoints",
+			Label: "bbb",
+		},
+		{
+			Role:  "node",
+			Label: "aa",
+		},
+	}, []string{
+		"/api/v1/namespaces/x/endpoints?labelSelector=bbb",
+		"/api/v1/namespaces/y/endpoints?labelSelector=bbb",
+	})
+
+	// role=endpointslices
+	f("endpointslices", nil, nil, []string{"/apis/discovery.k8s.io/v1beta1/endpointslices"})
+	f("endpointslices", []string{"x", "y"}, []Selector{
+		{
+			Role:  "endpointslices",
+			Field: "field",
+			Label: "label",
+		},
+	}, []string{
+		"/apis/discovery.k8s.io/v1beta1/namespaces/x/endpointslices?labelSelector=label&fieldSelector=field",
+		"/apis/discovery.k8s.io/v1beta1/namespaces/y/endpointslices?labelSelector=label&fieldSelector=field",
+	})
+
+	// role=ingress
+	f("ingress", nil, nil, []string{"/api/v1/ingresses"})
+	f("ingress", []string{"x", "y"}, []Selector{
+		{
+			Role:  "node",
+			Field: "xyay",
+		},
+		{
+			Role:  "ingress",
+			Field: "abc",
+		},
+		{
+			Role:  "ingress",
+			Label: "cde",
+		},
+		{
+			Role:  "ingress",
+			Label: "baaa",
+		},
+	}, []string{
+		"/api/v1/namespaces/x/ingresses?labelSelector=cde%2Cbaaa&fieldSelector=abc",
+		"/api/v1/namespaces/y/ingresses?labelSelector=cde%2Cbaaa&fieldSelector=abc",
+	})
+}
diff --git a/lib/promscrape/discovery/kubernetes/endpoints_test.go b/lib/promscrape/discovery/kubernetes/endpoints_test.go
index e5f0f4251..05a74183b 100644
--- a/lib/promscrape/discovery/kubernetes/endpoints_test.go
+++ b/lib/promscrape/discovery/kubernetes/endpoints_test.go
@@ -11,12 +11,12 @@ import (
 func TestParseEndpointsListFailure(t *testing.T) {
 	f := func(s string) {
 		t.Helper()
-		els, err := parseEndpointsList([]byte(s))
+		objectsByKey, _, err := parseEndpointsList([]byte(s))
 		if err == nil {
 			t.Fatalf("expecting non-nil error")
 		}
-		if els != nil {
-			t.Fatalf("unexpected non-nil EnpointsList: %v", els)
+		if len(objectsByKey) != 0 {
+			t.Fatalf("unexpected non-empty objectsByKey: %v", objectsByKey)
 		}
 	}
 	f(``)
@@ -79,21 +79,16 @@ func TestParseEndpointsListSuccess(t *testing.T) {
   ]
 }
 `
-	els, err := parseEndpointsList([]byte(data))
+	objectsByKey, meta, err := parseEndpointsList([]byte(data))
 	if err != nil {
 		t.Fatalf("unexpected error: %s", err)
 	}
-	if len(els.Items) != 1 {
-		t.Fatalf("unexpected length of EndpointsList.Items; got %d; want %d", len(els.Items), 1)
+	expectedResourceVersion := "128055"
+	if meta.ResourceVersion != expectedResourceVersion {
+		t.Fatalf("unexpected resource version; got %s; want %s", meta.ResourceVersion, expectedResourceVersion)
 	}
-	endpoint := els.Items[0]
 
-	// Check endpoint.appendTargetLabels()
-	labelss := endpoint.appendTargetLabels(nil, nil, nil)
-	var sortedLabelss [][]prompbmarshal.Label
-	for _, labels := range labelss {
-		sortedLabelss = append(sortedLabelss, discoveryutils.GetSortedLabels(labels))
-	}
+	sortedLabelss := getSortedLabelss(objectsByKey)
 	expectedLabelss := [][]prompbmarshal.Label{
 		discoveryutils.GetSortedLabels(map[string]string{
 			"__address__": "172.17.0.2:8443",
diff --git a/lib/promscrape/discovery/kubernetes/endpointslices_test.go b/lib/promscrape/discovery/kubernetes/endpointslices_test.go
index 900751b74..d587fd93f 100644
--- a/lib/promscrape/discovery/kubernetes/endpointslices_test.go
+++ b/lib/promscrape/discovery/kubernetes/endpointslices_test.go
@@ -8,14 +8,14 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discoveryutils"
 )
 
-func Test_parseEndpointSlicesListFail(t *testing.T) {
+func TestParseEndpointSliceListFail(t *testing.T) {
 	f := func(data string) {
-		eslList, err := parseEndpointSlicesList([]byte(data))
+		objectsByKey, _, err := parseEndpointSliceList([]byte(data))
 		if err == nil {
 			t.Errorf("unexpected result, test must fail! data: %s", data)
 		}
-		if eslList != nil {
-			t.Errorf("endpointSliceList must be nil, got: %v", eslList)
+		if len(objectsByKey) != 0 {
+			t.Errorf("EndpointSliceList must be emptry, got: %v", objectsByKey)
 		}
 	}
 
@@ -28,7 +28,7 @@ func Test_parseEndpointSlicesListFail(t *testing.T) {
 
 }
 
-func Test_parseEndpointSlicesListSuccess(t *testing.T) {
+func TestParseEndpointSliceListSuccess(t *testing.T) {
 	data := `{
   "kind": "EndpointSliceList",
   "apiVersion": "discovery.k8s.io/v1beta1",
@@ -176,22 +176,17 @@ func Test_parseEndpointSlicesListSuccess(t *testing.T) {
     }
   ]
 }`
-	esl, err := parseEndpointSlicesList([]byte(data))
+	objectsByKey, meta, err := parseEndpointSliceList([]byte(data))
 	if err != nil {
 		t.Errorf("cannot parse data for EndpointSliceList: %v", err)
 		return
 	}
-	if len(esl.Items) != 2 {
-		t.Fatalf("expected 2 items at endpointSliceList, got: %d", len(esl.Items))
+	expectedResourceVersion := "1177"
+	if meta.ResourceVersion != expectedResourceVersion {
+		t.Fatalf("unexpected resource version; got %s; want %s", meta.ResourceVersion, expectedResourceVersion)
 	}
-
-	firstEsl := esl.Items[0]
-	got := firstEsl.appendTargetLabels(nil, nil, nil)
-	sortedLables := [][]prompbmarshal.Label{}
-	for _, labels := range got {
-		sortedLables = append(sortedLables, discoveryutils.GetSortedLabels(labels))
-	}
-	expectedLabels := [][]prompbmarshal.Label{
+	sortedLabelss := getSortedLabelss(objectsByKey)
+	expectedLabelss := [][]prompbmarshal.Label{
 		discoveryutils.GetSortedLabels(map[string]string{
 			"__address__": "172.18.0.2:6443",
 			"__meta_kubernetes_endpointslice_address_type":              "IPv4",
@@ -201,253 +196,94 @@ func Test_parseEndpointSlicesListSuccess(t *testing.T) {
 			"__meta_kubernetes_endpointslice_port_name":                 "https",
 			"__meta_kubernetes_endpointslice_port_protocol":             "TCP",
 			"__meta_kubernetes_namespace":                               "default",
-		})}
-	if !reflect.DeepEqual(sortedLables, expectedLabels) {
-		t.Fatalf("unexpected labels,\ngot:\n%v,\nwant:\n%v", sortedLables, expectedLabels)
-	}
-
-}
-
-func TestEndpointSlice_appendTargetLabels(t *testing.T) {
-	type fields struct {
-		Metadata    ObjectMeta
-		Endpoints   []Endpoint
-		AddressType string
-		Ports       []EndpointPort
-	}
-	type args struct {
-		ms   []map[string]string
-		pods []Pod
-		svcs []Service
+		}),
+		discoveryutils.GetSortedLabels(map[string]string{
+			"__address__": "10.244.0.3:53",
+			"__meta_kubernetes_endpointslice_address_target_kind":                              "Pod",
+			"__meta_kubernetes_endpointslice_address_target_name":                              "coredns-66bff467f8-z8czk",
+			"__meta_kubernetes_endpointslice_address_type":                                     "IPv4",
+			"__meta_kubernetes_endpointslice_endpoint_conditions_ready":                        "true",
+			"__meta_kubernetes_endpointslice_endpoint_topology_kubernetes_io_hostname":         "kind-control-plane",
+			"__meta_kubernetes_endpointslice_endpoint_topology_present_kubernetes_io_hostname": "true",
+			"__meta_kubernetes_endpointslice_name":                                             "kube-dns-22mvb",
+			"__meta_kubernetes_endpointslice_port":                                             "53",
+			"__meta_kubernetes_endpointslice_port_name":                                        "dns-tcp",
+			"__meta_kubernetes_endpointslice_port_protocol":                                    "TCP",
+			"__meta_kubernetes_namespace":                                                      "kube-system",
+		}),
+		discoveryutils.GetSortedLabels(map[string]string{
+			"__address__": "10.244.0.3:9153",
+			"__meta_kubernetes_endpointslice_address_target_kind":                              "Pod",
+			"__meta_kubernetes_endpointslice_address_target_name":                              "coredns-66bff467f8-z8czk",
+			"__meta_kubernetes_endpointslice_address_type":                                     "IPv4",
+			"__meta_kubernetes_endpointslice_endpoint_conditions_ready":                        "true",
+			"__meta_kubernetes_endpointslice_endpoint_topology_kubernetes_io_hostname":         "kind-control-plane",
+			"__meta_kubernetes_endpointslice_endpoint_topology_present_kubernetes_io_hostname": "true",
+			"__meta_kubernetes_endpointslice_name":                                             "kube-dns-22mvb",
+			"__meta_kubernetes_endpointslice_port":                                             "9153",
+			"__meta_kubernetes_endpointslice_port_name":                                        "metrics",
+			"__meta_kubernetes_endpointslice_port_protocol":                                    "TCP",
+			"__meta_kubernetes_namespace":                                                      "kube-system",
+		}),
+		discoveryutils.GetSortedLabels(map[string]string{
+			"__address__": "10.244.0.3:53",
+			"__meta_kubernetes_endpointslice_address_target_kind":                              "Pod",
+			"__meta_kubernetes_endpointslice_address_target_name":                              "coredns-66bff467f8-z8czk",
+			"__meta_kubernetes_endpointslice_address_type":                                     "IPv4",
+			"__meta_kubernetes_endpointslice_endpoint_conditions_ready":                        "true",
+			"__meta_kubernetes_endpointslice_endpoint_topology_kubernetes_io_hostname":         "kind-control-plane",
+			"__meta_kubernetes_endpointslice_endpoint_topology_present_kubernetes_io_hostname": "true",
+			"__meta_kubernetes_endpointslice_name":                                             "kube-dns-22mvb",
+			"__meta_kubernetes_endpointslice_port":                                             "53",
+			"__meta_kubernetes_endpointslice_port_name":                                        "dns",
+			"__meta_kubernetes_endpointslice_port_protocol":                                    "UDP",
+			"__meta_kubernetes_namespace":                                                      "kube-system",
+		}),
+		discoveryutils.GetSortedLabels(map[string]string{
+			"__address__": "10.244.0.4:53",
+			"__meta_kubernetes_endpointslice_address_target_kind":                              "Pod",
+			"__meta_kubernetes_endpointslice_address_target_name":                              "coredns-66bff467f8-kpbhk",
+			"__meta_kubernetes_endpointslice_address_type":                                     "IPv4",
+			"__meta_kubernetes_endpointslice_endpoint_conditions_ready":                        "true",
+			"__meta_kubernetes_endpointslice_endpoint_topology_kubernetes_io_hostname":         "kind-control-plane",
+			"__meta_kubernetes_endpointslice_endpoint_topology_present_kubernetes_io_hostname": "true",
+			"__meta_kubernetes_endpointslice_name":                                             "kube-dns-22mvb",
+			"__meta_kubernetes_endpointslice_port":                                             "53",
+			"__meta_kubernetes_endpointslice_port_name":                                        "dns-tcp",
+			"__meta_kubernetes_endpointslice_port_protocol":                                    "TCP",
+			"__meta_kubernetes_namespace":                                                      "kube-system",
+		}),
+		discoveryutils.GetSortedLabels(map[string]string{
+			"__address__": "10.244.0.4:9153",
+			"__meta_kubernetes_endpointslice_address_target_kind":                              "Pod",
+			"__meta_kubernetes_endpointslice_address_target_name":                              "coredns-66bff467f8-kpbhk",
+			"__meta_kubernetes_endpointslice_address_type":                                     "IPv4",
+			"__meta_kubernetes_endpointslice_endpoint_conditions_ready":                        "true",
+			"__meta_kubernetes_endpointslice_endpoint_topology_kubernetes_io_hostname":         "kind-control-plane",
+			"__meta_kubernetes_endpointslice_endpoint_topology_present_kubernetes_io_hostname": "true",
+			"__meta_kubernetes_endpointslice_name":                                             "kube-dns-22mvb",
+			"__meta_kubernetes_endpointslice_port":                                             "9153",
+			"__meta_kubernetes_endpointslice_port_name":                                        "metrics",
+			"__meta_kubernetes_endpointslice_port_protocol":                                    "TCP",
+			"__meta_kubernetes_namespace":                                                      "kube-system",
+		}),
+		discoveryutils.GetSortedLabels(map[string]string{
+			"__address__": "10.244.0.4:53",
+			"__meta_kubernetes_endpointslice_address_target_kind":                              "Pod",
+			"__meta_kubernetes_endpointslice_address_target_name":                              "coredns-66bff467f8-kpbhk",
+			"__meta_kubernetes_endpointslice_address_type":                                     "IPv4",
+			"__meta_kubernetes_endpointslice_endpoint_conditions_ready":                        "true",
+			"__meta_kubernetes_endpointslice_endpoint_topology_kubernetes_io_hostname":         "kind-control-plane",
+			"__meta_kubernetes_endpointslice_endpoint_topology_present_kubernetes_io_hostname": "true",
+			"__meta_kubernetes_endpointslice_name":                                             "kube-dns-22mvb",
+			"__meta_kubernetes_endpointslice_port":                                             "53",
+			"__meta_kubernetes_endpointslice_port_name":                                        "dns",
+			"__meta_kubernetes_endpointslice_port_protocol":                                    "UDP",
+			"__meta_kubernetes_namespace":                                                      "kube-system",
+		}),
 	}
-	tests := []struct {
-		name   string
-		fields fields
-		args   args
-		want   [][]prompbmarshal.Label
-	}{
-		{
-			name: "simple eps",
-			args: args{},
-			fields: fields{
-				Metadata: ObjectMeta{
-					Name:      "fake-esl",
-					Namespace: "default",
-				},
-				AddressType: "ipv4",
-				Endpoints: []Endpoint{
-					{Addresses: []string{"127.0.0.1"},
-						Hostname:   "node-1",
-						Topology:   map[string]string{"kubernetes.topoligy.io/zone": "gce-1"},
-						Conditions: EndpointConditions{Ready: true},
-						TargetRef: ObjectReference{
-							Kind:      "Pod",
-							Namespace: "default",
-							Name:      "main-pod",
-						},
-					},
-				},
-				Ports: []EndpointPort{
-					{
-						Name:        "http",
-						Port:        8085,
-						AppProtocol: "http",
-						Protocol:    "tcp",
-					},
-				},
-			},
-			want: [][]prompbmarshal.Label{
-				discoveryutils.GetSortedLabels(map[string]string{
-					"__address__": "127.0.0.1:8085",
-					"__meta_kubernetes_endpointslice_address_target_kind":                                   "Pod",
-					"__meta_kubernetes_endpointslice_address_target_name":                                   "main-pod",
-					"__meta_kubernetes_endpointslice_address_type":                                          "ipv4",
-					"__meta_kubernetes_endpointslice_endpoint_conditions_ready":                             "true",
-					"__meta_kubernetes_endpointslice_endpoint_topology_kubernetes_topoligy_io_zone":         "gce-1",
-					"__meta_kubernetes_endpointslice_endpoint_topology_present_kubernetes_topoligy_io_zone": "true",
-					"__meta_kubernetes_endpointslice_endpoint_hostname":                                     "node-1",
-					"__meta_kubernetes_endpointslice_name":                                                  "fake-esl",
-					"__meta_kubernetes_endpointslice_port":                                                  "8085",
-					"__meta_kubernetes_endpointslice_port_app_protocol":                                     "http",
-					"__meta_kubernetes_endpointslice_port_name":                                             "http",
-					"__meta_kubernetes_endpointslice_port_protocol":                                         "tcp",
-					"__meta_kubernetes_namespace":                                                           "default",
-				}),
-			},
-		},
-		{
-			name: "eps with pods and services",
-			args: args{
-				pods: []Pod{
-					{
-						Metadata: ObjectMeta{
-							UID:       "some-pod-uuid",
-							Namespace: "monitoring",
-							Name:      "main-pod",
-							Labels: discoveryutils.GetSortedLabels(map[string]string{
-								"pod-label-1": "pod-value-1",
-								"pod-label-2": "pod-value-2",
-							}),
-							Annotations: discoveryutils.GetSortedLabels(map[string]string{
-								"pod-annotations-1": "annotation-value-1",
-							}),
-						},
-						Status: PodStatus{PodIP: "192.168.11.5", HostIP: "172.15.1.1"},
-						Spec: PodSpec{NodeName: "node-2", Containers: []Container{
-							{
-								Name: "container-1",
-								Ports: []ContainerPort{
-									{
-										ContainerPort: 8085,
-										Protocol:      "tcp",
-										Name:          "http",
-									},
-									{
-										ContainerPort: 8011,
-										Protocol:      "udp",
-										Name:          "dns",
-									},
-								},
-							},
-						}},
-					},
-				},
-				svcs: []Service{
-					{
-						Spec: ServiceSpec{Type: "ClusterIP", Ports: []ServicePort{
-							{
-								Name:     "http",
-								Protocol: "tcp",
-								Port:     8085,
-							},
-						}},
-						Metadata: ObjectMeta{
-							Name:      "custom-esl",
-							Namespace: "monitoring",
-							Labels: discoveryutils.GetSortedLabels(map[string]string{
-								"service-label-1": "value-1",
-								"service-label-2": "value-2",
-							}),
-						},
-					},
-				},
-			},
-			fields: fields{
-				Metadata: ObjectMeta{
-					Name:      "custom-esl",
-					Namespace: "monitoring",
-				},
-				AddressType: "ipv4",
-				Endpoints: []Endpoint{
-					{Addresses: []string{"127.0.0.1"},
-						Hostname:   "node-1",
-						Topology:   map[string]string{"kubernetes.topoligy.io/zone": "gce-1"},
-						Conditions: EndpointConditions{Ready: true},
-						TargetRef: ObjectReference{
-							Kind:      "Pod",
-							Namespace: "monitoring",
-							Name:      "main-pod",
-						},
-					},
-				},
-				Ports: []EndpointPort{
-					{
-						Name:        "http",
-						Port:        8085,
-						AppProtocol: "http",
-						Protocol:    "tcp",
-					},
-				},
-			},
-			want: [][]prompbmarshal.Label{
-				discoveryutils.GetSortedLabels(map[string]string{
-					"__address__": "127.0.0.1:8085",
-					"__meta_kubernetes_endpointslice_address_target_kind":                                   "Pod",
-					"__meta_kubernetes_endpointslice_address_target_name":                                   "main-pod",
-					"__meta_kubernetes_endpointslice_address_type":                                          "ipv4",
-					"__meta_kubernetes_endpointslice_endpoint_conditions_ready":                             "true",
-					"__meta_kubernetes_endpointslice_endpoint_topology_kubernetes_topoligy_io_zone":         "gce-1",
-					"__meta_kubernetes_endpointslice_endpoint_topology_present_kubernetes_topoligy_io_zone": "true",
-					"__meta_kubernetes_endpointslice_endpoint_hostname":                                     "node-1",
-					"__meta_kubernetes_endpointslice_name":                                                  "custom-esl",
-					"__meta_kubernetes_endpointslice_port":                                                  "8085",
-					"__meta_kubernetes_endpointslice_port_app_protocol":                                     "http",
-					"__meta_kubernetes_endpointslice_port_name":                                             "http",
-					"__meta_kubernetes_endpointslice_port_protocol":                                         "tcp",
-					"__meta_kubernetes_namespace":                                                           "monitoring",
-					"__meta_kubernetes_pod_annotation_pod_annotations_1":                                    "annotation-value-1",
-					"__meta_kubernetes_pod_annotationpresent_pod_annotations_1":                             "true",
-					"__meta_kubernetes_pod_container_name":                                                  "container-1",
-					"__meta_kubernetes_pod_container_port_name":                                             "http",
-					"__meta_kubernetes_pod_container_port_number":                                           "8085",
-					"__meta_kubernetes_pod_container_port_protocol":                                         "tcp",
-					"__meta_kubernetes_pod_host_ip":                                                         "172.15.1.1",
-					"__meta_kubernetes_pod_ip":                                                              "192.168.11.5",
-					"__meta_kubernetes_pod_label_pod_label_1":                                               "pod-value-1",
-					"__meta_kubernetes_pod_label_pod_label_2":                                               "pod-value-2",
-					"__meta_kubernetes_pod_labelpresent_pod_label_1":                                        "true",
-					"__meta_kubernetes_pod_labelpresent_pod_label_2":                                        "true",
-					"__meta_kubernetes_pod_name":                                                            "main-pod",
-					"__meta_kubernetes_pod_node_name":                                                       "node-2",
-					"__meta_kubernetes_pod_phase":                                                           "",
-					"__meta_kubernetes_pod_ready":                                                           "unknown",
-					"__meta_kubernetes_pod_uid":                                                             "some-pod-uuid",
-					"__meta_kubernetes_service_cluster_ip":                                                  "",
-					"__meta_kubernetes_service_label_service_label_1":                                       "value-1",
-					"__meta_kubernetes_service_label_service_label_2":                                       "value-2",
-					"__meta_kubernetes_service_labelpresent_service_label_1":                                "true",
-					"__meta_kubernetes_service_labelpresent_service_label_2":                                "true",
-					"__meta_kubernetes_service_name":                                                        "custom-esl",
-					"__meta_kubernetes_service_type":                                                        "ClusterIP",
-				}),
-				discoveryutils.GetSortedLabels(map[string]string{
-					"__address__":                 "192.168.11.5:8011",
-					"__meta_kubernetes_namespace": "monitoring",
-					"__meta_kubernetes_pod_annotation_pod_annotations_1":        "annotation-value-1",
-					"__meta_kubernetes_pod_annotationpresent_pod_annotations_1": "true",
-					"__meta_kubernetes_pod_container_name":                      "container-1",
-					"__meta_kubernetes_pod_container_port_name":                 "dns",
-					"__meta_kubernetes_pod_container_port_number":               "8011",
-					"__meta_kubernetes_pod_container_port_protocol":             "udp",
-					"__meta_kubernetes_pod_host_ip":                             "172.15.1.1",
-					"__meta_kubernetes_pod_ip":                                  "192.168.11.5",
-					"__meta_kubernetes_pod_label_pod_label_1":                   "pod-value-1",
-					"__meta_kubernetes_pod_label_pod_label_2":                   "pod-value-2",
-					"__meta_kubernetes_pod_labelpresent_pod_label_1":            "true",
-					"__meta_kubernetes_pod_labelpresent_pod_label_2":            "true",
-					"__meta_kubernetes_pod_name":                                "main-pod",
-					"__meta_kubernetes_pod_node_name":                           "node-2",
-					"__meta_kubernetes_pod_phase":                               "",
-					"__meta_kubernetes_pod_ready":                               "unknown",
-					"__meta_kubernetes_pod_uid":                                 "some-pod-uuid",
-					"__meta_kubernetes_service_cluster_ip":                      "",
-					"__meta_kubernetes_service_label_service_label_1":           "value-1",
-					"__meta_kubernetes_service_label_service_label_2":           "value-2",
-					"__meta_kubernetes_service_labelpresent_service_label_1":    "true",
-					"__meta_kubernetes_service_labelpresent_service_label_2":    "true",
-					"__meta_kubernetes_service_name":                            "custom-esl",
-					"__meta_kubernetes_service_type":                            "ClusterIP",
-				}),
-			},
-		},
+	if !reflect.DeepEqual(sortedLabelss, expectedLabelss) {
+		t.Fatalf("unexpected labels,\ngot:\n%v,\nwant:\n%v", sortedLabelss, expectedLabelss)
 	}
-	for _, tt := range tests {
-		t.Run(tt.name, func(t *testing.T) {
-			eps := &EndpointSlice{
-				Metadata:    tt.fields.Metadata,
-				Endpoints:   tt.fields.Endpoints,
-				AddressType: tt.fields.AddressType,
-				Ports:       tt.fields.Ports,
-			}
-			got := eps.appendTargetLabels(tt.args.ms, tt.args.pods, tt.args.svcs)
-			var sortedLabelss [][]prompbmarshal.Label
-			for _, labels := range got {
-				sortedLabelss = append(sortedLabelss, discoveryutils.GetSortedLabels(labels))
-			}
 
-			if !reflect.DeepEqual(sortedLabelss, tt.want) {
-				t.Errorf("got unxpected labels: \ngot:\n %v, \nexpect:\n %v", sortedLabelss, tt.want)
-			}
-		})
-	}
 }
diff --git a/lib/promscrape/discovery/kubernetes/ingress_test.go b/lib/promscrape/discovery/kubernetes/ingress_test.go
index 575abb5f8..8046d8a24 100644
--- a/lib/promscrape/discovery/kubernetes/ingress_test.go
+++ b/lib/promscrape/discovery/kubernetes/ingress_test.go
@@ -11,12 +11,12 @@ import (
 func TestParseIngressListFailure(t *testing.T) {
 	f := func(s string) {
 		t.Helper()
-		nls, err := parseIngressList([]byte(s))
+		objectsByKey, _, err := parseIngressList([]byte(s))
 		if err == nil {
 			t.Fatalf("expecting non-nil error")
 		}
-		if nls != nil {
-			t.Fatalf("unexpected non-nil IngressList: %v", nls)
+		if len(objectsByKey) != 0 {
+			t.Fatalf("unexpected non-empty IngressList: %v", objectsByKey)
 		}
 	}
 	f(``)
@@ -71,21 +71,15 @@ func TestParseIngressListSuccess(t *testing.T) {
     }
   ]
 }`
-	igs, err := parseIngressList([]byte(data))
+	objectsByKey, meta, err := parseIngressList([]byte(data))
 	if err != nil {
 		t.Fatalf("unexpected error: %s", err)
 	}
-	if len(igs.Items) != 1 {
-		t.Fatalf("unexpected length of IngressList.Items; got %d; want %d", len(igs.Items), 1)
-	}
-	ig := igs.Items[0]
-
-	// Check ig.appendTargetLabels()
-	labelss := ig.appendTargetLabels(nil)
-	var sortedLabelss [][]prompbmarshal.Label
-	for _, labels := range labelss {
-		sortedLabelss = append(sortedLabelss, discoveryutils.GetSortedLabels(labels))
+	expectedResourceVersion := "351452"
+	if meta.ResourceVersion != expectedResourceVersion {
+		t.Fatalf("unexpected resource version; got %s; want %s", meta.ResourceVersion, expectedResourceVersion)
 	}
+	sortedLabelss := getSortedLabelss(objectsByKey)
 	expectedLabelss := [][]prompbmarshal.Label{
 		discoveryutils.GetSortedLabels(map[string]string{
 			"__address__": "foobar",
diff --git a/lib/promscrape/discovery/kubernetes/node_test.go b/lib/promscrape/discovery/kubernetes/node_test.go
index c632b8123..e63cbe4b4 100644
--- a/lib/promscrape/discovery/kubernetes/node_test.go
+++ b/lib/promscrape/discovery/kubernetes/node_test.go
@@ -4,18 +4,19 @@ import (
 	"reflect"
 	"testing"
 
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discoveryutils"
 )
 
 func TestParseNodeListFailure(t *testing.T) {
 	f := func(s string) {
 		t.Helper()
-		nls, err := parseNodeList([]byte(s))
+		objectsByKey, _, err := parseNodeList([]byte(s))
 		if err == nil {
 			t.Fatalf("expecting non-nil error")
 		}
-		if nls != nil {
-			t.Fatalf("unexpected non-nil NodeList: %v", nls)
+		if len(objectsByKey) != 0 {
+			t.Fatalf("unexpected non-empty objectsByKey: %v", objectsByKey)
 		}
 	}
 	f(``)
@@ -226,97 +227,67 @@ func TestParseNodeListSuccess(t *testing.T) {
   ]
 }
 `
-	nls, err := parseNodeList([]byte(data))
+	objectsByKey, meta, err := parseNodeList([]byte(data))
 	if err != nil {
 		t.Fatalf("unexpected error: %s", err)
 	}
-	if len(nls.Items) != 1 {
-		t.Fatalf("unexpected length of NodeList.Items; got %d; want %d", len(nls.Items), 1)
-	}
-	node := nls.Items[0]
-	meta := node.Metadata
-	if meta.Name != "m01" {
-		t.Fatalf("unexpected ObjectMeta.Name; got %q; want %q", meta.Name, "m01")
-	}
-	expectedLabels := discoveryutils.GetSortedLabels(map[string]string{
-		"beta.kubernetes.io/arch":        "amd64",
-		"beta.kubernetes.io/os":          "linux",
-		"kubernetes.io/arch":             "amd64",
-		"kubernetes.io/hostname":         "m01",
-		"kubernetes.io/os":               "linux",
-		"minikube.k8s.io/commit":         "eb13446e786c9ef70cb0a9f85a633194e62396a1",
-		"minikube.k8s.io/name":           "minikube",
-		"minikube.k8s.io/updated_at":     "2020_03_16T22_44_27_0700",
-		"minikube.k8s.io/version":        "v1.8.2",
-		"node-role.kubernetes.io/master": "",
-	})
-	if !reflect.DeepEqual(meta.Labels, expectedLabels) {
-		t.Fatalf("unexpected ObjectMeta.Labels\ngot\n%v\nwant\n%v", meta.Labels, expectedLabels)
-	}
-	expectedAnnotations := discoveryutils.GetSortedLabels(map[string]string{
-		"kubeadm.alpha.kubernetes.io/cri-socket":                 "/var/run/dockershim.sock",
-		"node.alpha.kubernetes.io/ttl":                           "0",
-		"volumes.kubernetes.io/controller-managed-attach-detach": "true",
-	})
-	if !reflect.DeepEqual(meta.Annotations, expectedAnnotations) {
-		t.Fatalf("unexpected ObjectMeta.Annotations\ngot\n%v\nwant\n%v", meta.Annotations, expectedAnnotations)
-	}
-	status := node.Status
-	expectedAddresses := []NodeAddress{
-		{
-			Type:    "InternalIP",
-			Address: "172.17.0.2",
-		},
-		{
-			Type:    "Hostname",
-			Address: "m01",
-		},
-	}
-	if !reflect.DeepEqual(status.Addresses, expectedAddresses) {
-		t.Fatalf("unexpected addresses\ngot\n%v\nwant\n%v", status.Addresses, expectedAddresses)
+	expectedResourceVersion := "22627"
+	if meta.ResourceVersion != expectedResourceVersion {
+		t.Fatalf("unexpected resource version; got %s; want %s", meta.ResourceVersion, expectedResourceVersion)
 	}
+	sortedLabelss := getSortedLabelss(objectsByKey)
+	expectedLabelss := [][]prompbmarshal.Label{
+		discoveryutils.GetSortedLabels(map[string]string{
+			"instance":                    "m01",
+			"__address__":                 "172.17.0.2:10250",
+			"__meta_kubernetes_node_name": "m01",
 
-	// Check node.appendTargetLabels()
-	labels := discoveryutils.GetSortedLabels(node.appendTargetLabels(nil)[0])
-	expectedLabels = discoveryutils.GetSortedLabels(map[string]string{
-		"instance":                    "m01",
-		"__address__":                 "172.17.0.2:10250",
-		"__meta_kubernetes_node_name": "m01",
+			"__meta_kubernetes_node_label_beta_kubernetes_io_arch":        "amd64",
+			"__meta_kubernetes_node_label_beta_kubernetes_io_os":          "linux",
+			"__meta_kubernetes_node_label_kubernetes_io_arch":             "amd64",
+			"__meta_kubernetes_node_label_kubernetes_io_hostname":         "m01",
+			"__meta_kubernetes_node_label_kubernetes_io_os":               "linux",
+			"__meta_kubernetes_node_label_minikube_k8s_io_commit":         "eb13446e786c9ef70cb0a9f85a633194e62396a1",
+			"__meta_kubernetes_node_label_minikube_k8s_io_name":           "minikube",
+			"__meta_kubernetes_node_label_minikube_k8s_io_updated_at":     "2020_03_16T22_44_27_0700",
+			"__meta_kubernetes_node_label_minikube_k8s_io_version":        "v1.8.2",
+			"__meta_kubernetes_node_label_node_role_kubernetes_io_master": "",
 
-		"__meta_kubernetes_node_label_beta_kubernetes_io_arch":        "amd64",
-		"__meta_kubernetes_node_label_beta_kubernetes_io_os":          "linux",
-		"__meta_kubernetes_node_label_kubernetes_io_arch":             "amd64",
-		"__meta_kubernetes_node_label_kubernetes_io_hostname":         "m01",
-		"__meta_kubernetes_node_label_kubernetes_io_os":               "linux",
-		"__meta_kubernetes_node_label_minikube_k8s_io_commit":         "eb13446e786c9ef70cb0a9f85a633194e62396a1",
-		"__meta_kubernetes_node_label_minikube_k8s_io_name":           "minikube",
-		"__meta_kubernetes_node_label_minikube_k8s_io_updated_at":     "2020_03_16T22_44_27_0700",
-		"__meta_kubernetes_node_label_minikube_k8s_io_version":        "v1.8.2",
-		"__meta_kubernetes_node_label_node_role_kubernetes_io_master": "",
+			"__meta_kubernetes_node_labelpresent_beta_kubernetes_io_arch":        "true",
+			"__meta_kubernetes_node_labelpresent_beta_kubernetes_io_os":          "true",
+			"__meta_kubernetes_node_labelpresent_kubernetes_io_arch":             "true",
+			"__meta_kubernetes_node_labelpresent_kubernetes_io_hostname":         "true",
+			"__meta_kubernetes_node_labelpresent_kubernetes_io_os":               "true",
+			"__meta_kubernetes_node_labelpresent_minikube_k8s_io_commit":         "true",
+			"__meta_kubernetes_node_labelpresent_minikube_k8s_io_name":           "true",
+			"__meta_kubernetes_node_labelpresent_minikube_k8s_io_updated_at":     "true",
+			"__meta_kubernetes_node_labelpresent_minikube_k8s_io_version":        "true",
+			"__meta_kubernetes_node_labelpresent_node_role_kubernetes_io_master": "true",
 
-		"__meta_kubernetes_node_labelpresent_beta_kubernetes_io_arch":        "true",
-		"__meta_kubernetes_node_labelpresent_beta_kubernetes_io_os":          "true",
-		"__meta_kubernetes_node_labelpresent_kubernetes_io_arch":             "true",
-		"__meta_kubernetes_node_labelpresent_kubernetes_io_hostname":         "true",
-		"__meta_kubernetes_node_labelpresent_kubernetes_io_os":               "true",
-		"__meta_kubernetes_node_labelpresent_minikube_k8s_io_commit":         "true",
-		"__meta_kubernetes_node_labelpresent_minikube_k8s_io_name":           "true",
-		"__meta_kubernetes_node_labelpresent_minikube_k8s_io_updated_at":     "true",
-		"__meta_kubernetes_node_labelpresent_minikube_k8s_io_version":        "true",
-		"__meta_kubernetes_node_labelpresent_node_role_kubernetes_io_master": "true",
+			"__meta_kubernetes_node_annotation_kubeadm_alpha_kubernetes_io_cri_socket":                 "/var/run/dockershim.sock",
+			"__meta_kubernetes_node_annotation_node_alpha_kubernetes_io_ttl":                           "0",
+			"__meta_kubernetes_node_annotation_volumes_kubernetes_io_controller_managed_attach_detach": "true",
 
-		"__meta_kubernetes_node_annotation_kubeadm_alpha_kubernetes_io_cri_socket":                 "/var/run/dockershim.sock",
-		"__meta_kubernetes_node_annotation_node_alpha_kubernetes_io_ttl":                           "0",
-		"__meta_kubernetes_node_annotation_volumes_kubernetes_io_controller_managed_attach_detach": "true",
+			"__meta_kubernetes_node_annotationpresent_kubeadm_alpha_kubernetes_io_cri_socket":                 "true",
+			"__meta_kubernetes_node_annotationpresent_node_alpha_kubernetes_io_ttl":                           "true",
+			"__meta_kubernetes_node_annotationpresent_volumes_kubernetes_io_controller_managed_attach_detach": "true",
 
-		"__meta_kubernetes_node_annotationpresent_kubeadm_alpha_kubernetes_io_cri_socket":                 "true",
-		"__meta_kubernetes_node_annotationpresent_node_alpha_kubernetes_io_ttl":                           "true",
-		"__meta_kubernetes_node_annotationpresent_volumes_kubernetes_io_controller_managed_attach_detach": "true",
+			"__meta_kubernetes_node_address_InternalIP": "172.17.0.2",
+			"__meta_kubernetes_node_address_Hostname":   "m01",
+		}),
+	}
+	if !reflect.DeepEqual(sortedLabelss, expectedLabelss) {
+		t.Fatalf("unexpected labels:\ngot\n%v\nwant\n%v", sortedLabelss, expectedLabelss)
+	}
+}
 
-		"__meta_kubernetes_node_address_InternalIP": "172.17.0.2",
-		"__meta_kubernetes_node_address_Hostname":   "m01",
-	})
-	if !reflect.DeepEqual(labels, expectedLabels) {
-		t.Fatalf("unexpected labels:\ngot\n%v\nwant\n%v", labels, expectedLabels)
+func getSortedLabelss(objectsByKey map[string]object) [][]prompbmarshal.Label {
+	var result [][]prompbmarshal.Label
+	for _, o := range objectsByKey {
+		labelss := o.getTargetLabels(nil)
+		for _, labels := range labelss {
+			result = append(result, discoveryutils.GetSortedLabels(labels))
+		}
 	}
+	return result
 }
diff --git a/lib/promscrape/discovery/kubernetes/pod_test.go b/lib/promscrape/discovery/kubernetes/pod_test.go
index 7462f0014..204a8f102 100644
--- a/lib/promscrape/discovery/kubernetes/pod_test.go
+++ b/lib/promscrape/discovery/kubernetes/pod_test.go
@@ -11,12 +11,12 @@ import (
 func TestParsePodListFailure(t *testing.T) {
 	f := func(s string) {
 		t.Helper()
-		nls, err := parsePodList([]byte(s))
+		objectsByKey, _, err := parsePodList([]byte(s))
 		if err == nil {
 			t.Fatalf("expecting non-nil error")
 		}
-		if nls != nil {
-			t.Fatalf("unexpected non-nil PodList: %v", nls)
+		if len(objectsByKey) != 0 {
+			t.Fatalf("unexpected non-empty objectsByKey: %v", objectsByKey)
 		}
 	}
 	f(``)
@@ -228,22 +228,16 @@ func TestParsePodListSuccess(t *testing.T) {
   ]
 }
 `
-	pls, err := parsePodList([]byte(data))
+	objectsByKey, meta, err := parsePodList([]byte(data))
 	if err != nil {
 		t.Fatalf("unexpected error: %s", err)
 	}
-	if len(pls.Items) != 1 {
-		t.Fatalf("unexpected length of PodList.Items; got %d; want %d", len(pls.Items), 1)
+	expectedResourceVersion := "72425"
+	if meta.ResourceVersion != expectedResourceVersion {
+		t.Fatalf("unexpected resource version; got %s; want %s", meta.ResourceVersion, expectedResourceVersion)
 	}
-	pod := pls.Items[0]
-
-	// Check pod.appendTargetLabels()
-	labelss := pod.appendTargetLabels(nil)
-	var sortedLabelss [][]prompbmarshal.Label
-	for _, labels := range labelss {
-		sortedLabelss = append(sortedLabelss, discoveryutils.GetSortedLabels(labels))
-	}
-	expectedLabels := [][]prompbmarshal.Label{
+	sortedLabelss := getSortedLabelss(objectsByKey)
+	expectedLabelss := [][]prompbmarshal.Label{
 		discoveryutils.GetSortedLabels(map[string]string{
 			"__address__": "172.17.0.2:1234",
 
@@ -280,7 +274,7 @@ func TestParsePodListSuccess(t *testing.T) {
 			"__meta_kubernetes_pod_annotationpresent_kubernetes_io_config_source": "true",
 		}),
 	}
-	if !reflect.DeepEqual(sortedLabelss, expectedLabels) {
-		t.Fatalf("unexpected labels:\ngot\n%v\nwant\n%v", sortedLabelss, expectedLabels)
+	if !reflect.DeepEqual(sortedLabelss, expectedLabelss) {
+		t.Fatalf("unexpected labels:\ngot\n%v\nwant\n%v", sortedLabelss, expectedLabelss)
 	}
 }
diff --git a/lib/promscrape/discovery/kubernetes/service_test.go b/lib/promscrape/discovery/kubernetes/service_test.go
index 2f3c1f7e7..56508e5f3 100644
--- a/lib/promscrape/discovery/kubernetes/service_test.go
+++ b/lib/promscrape/discovery/kubernetes/service_test.go
@@ -11,12 +11,12 @@ import (
 func TestParseServiceListFailure(t *testing.T) {
 	f := func(s string) {
 		t.Helper()
-		nls, err := parseServiceList([]byte(s))
+		objectsByKey, _, err := parseServiceList([]byte(s))
 		if err == nil {
 			t.Fatalf("expecting non-nil error")
 		}
-		if nls != nil {
-			t.Fatalf("unexpected non-nil ServiceList: %v", nls)
+		if len(objectsByKey) != 0 {
+			t.Fatalf("unexpected non-empty objectsByKey: %v", objectsByKey)
 		}
 	}
 	f(``)
@@ -89,68 +89,15 @@ func TestParseServiceListSuccess(t *testing.T) {
   ]
 }
 `
-	sls, err := parseServiceList([]byte(data))
+	objectsByKey, meta, err := parseServiceList([]byte(data))
 	if err != nil {
 		t.Fatalf("unexpected error: %s", err)
 	}
-	if len(sls.Items) != 1 {
-		t.Fatalf("unexpected length of ServiceList.Items; got %d; want %d", len(sls.Items), 1)
-	}
-	service := sls.Items[0]
-	meta := service.Metadata
-	if meta.Name != "kube-dns" {
-		t.Fatalf("unexpected ObjectMeta.Name; got %q; want %q", meta.Name, "kube-dns")
-	}
-	expectedLabels := discoveryutils.GetSortedLabels(map[string]string{
-		"k8s-app":                       "kube-dns",
-		"kubernetes.io/cluster-service": "true",
-		"kubernetes.io/name":            "KubeDNS",
-	})
-	if !reflect.DeepEqual(meta.Labels, expectedLabels) {
-		t.Fatalf("unexpected ObjectMeta.Labels\ngot\n%v\nwant\n%v", meta.Labels, expectedLabels)
-	}
-	expectedAnnotations := discoveryutils.GetSortedLabels(map[string]string{
-		"prometheus.io/port":   "9153",
-		"prometheus.io/scrape": "true",
-	})
-	if !reflect.DeepEqual(meta.Annotations, expectedAnnotations) {
-		t.Fatalf("unexpected ObjectMeta.Annotations\ngot\n%v\nwant\n%v", meta.Annotations, expectedAnnotations)
-	}
-	spec := service.Spec
-	expectedClusterIP := "10.96.0.10"
-	if spec.ClusterIP != expectedClusterIP {
-		t.Fatalf("unexpected clusterIP; got %q; want %q", spec.ClusterIP, expectedClusterIP)
-	}
-	if spec.Type != "ClusterIP" {
-		t.Fatalf("unexpected type; got %q; want %q", spec.Type, "ClusterIP")
-	}
-	expectedPorts := []ServicePort{
-		{
-			Name:     "dns",
-			Protocol: "UDP",
-			Port:     53,
-		},
-		{
-			Name:     "dns-tcp",
-			Protocol: "TCP",
-			Port:     53,
-		},
-		{
-			Name:     "metrics",
-			Protocol: "TCP",
-			Port:     9153,
-		},
-	}
-	if !reflect.DeepEqual(spec.Ports, expectedPorts) {
-		t.Fatalf("unexpected ports\ngot\n%v\nwant\n%v", spec.Ports, expectedPorts)
-	}
-
-	// Check service.appendTargetLabels()
-	labelss := service.appendTargetLabels(nil)
-	var sortedLabelss [][]prompbmarshal.Label
-	for _, labels := range labelss {
-		sortedLabelss = append(sortedLabelss, discoveryutils.GetSortedLabels(labels))
+	expectedResourceVersion := "60485"
+	if meta.ResourceVersion != expectedResourceVersion {
+		t.Fatalf("unexpected resource version; got %s; want %s", meta.ResourceVersion, expectedResourceVersion)
 	}
+	sortedLabelss := getSortedLabelss(objectsByKey)
 	expectedLabelss := [][]prompbmarshal.Label{
 		discoveryutils.GetSortedLabels(map[string]string{
 			"__address__":                             "kube-dns.kube-system.svc:53",