Repository: VictoriaMetrics/VictoriaMetrics
Pull request #2492 (2022-04-22 14:22:08)

Single dashboards

codecov[bot]: # [Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2492?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) Report
> Merging [#2492](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2492?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) (9697826) into [master](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/commit/8d0fb4d69d4ecfe3d9e47b230cf92aacd1432c7f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) (8d0fb4d) will **increase** coverage by `0.07%`.
> The diff coverage is `64.17%`.

> :exclamation: Current head 9697826 differs from pull request most recent head 44082fd. Consider uploading reports for the commit 44082fd to get more accurate results

```diff
@@            Coverage Diff             @@
##           master    #2492      +/-   ##
==========================================
+ Coverage   60.43%   60.50%   +0.07%     
==========================================
  Files         245      245              
  Lines       41881    41915      +34     
==========================================
+ Hits        25310    25360      +50     
+ Misses      14904    14887      -17     
- Partials     1667     1668       +1     
```


| [Impacted Files](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2492?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) | Coverage Δ | |
|---|---|---|
| [lib/promscrape/discovery/kubernetes/api\_watcher.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2492/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL3Byb21zY3JhcGUvZGlzY292ZXJ5L2t1YmVybmV0ZXMvYXBpX3dhdGNoZXIuZ28=) | `67.32% <59.61%> (-0.08%)` | :arrow_down: |
| [lib/promscrape/config.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2492/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL3Byb21zY3JhcGUvY29uZmlnLmdv) | `46.87% <80.00%> (+0.77%)` | :arrow_up: |
| [lib/mergeset/table.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2492/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL21lcmdlc2V0L3RhYmxlLmdv) | `67.01% <0.00%> (+0.95%)` | :arrow_up: |
| [lib/bloomfilter/filter.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2492/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL2Jsb29tZmlsdGVyL2ZpbHRlci5nbw==) | `100.00% <0.00%> (+2.00%)` | :arrow_up: |
| [app/vmselect/promql/aggr\_incremental.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2492/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3Ztc2VsZWN0L3Byb21xbC9hZ2dyX2luY3JlbWVudGFsLmdv) | `87.65% <0.00%> (+3.61%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2492?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2492?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics). Last update [57a0aa2...44082fd](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2492?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).


Patch:
diff --git a/dashboards/clusterbytenant.json b/dashboards/clusterbytenant.json
index 1d684c704..d09cde83e 100644
--- a/dashboards/clusterbytenant.json
+++ b/dashboards/clusterbytenant.json
@@ -172,7 +172,7 @@
       "steppedLine": false,
       "targets": [
         {
-          "expr": "sum(rate(vm_tenant_select_requests_total{job=~\"$job\", instance=~\"$instance.*\",accountID=~\"$accountID\", projectID=~\"$projectID\"}[5m])) by (accountID,projectID) ",
+          "expr": "sum(rate(vm_tenant_select_requests_total{job=~\"$job\", instance=~\"$instance.*\",accountID=~\"$accountID\", projectID=~\"$projectID\"}[$__rate_interval])) by (accountID,projectID) ",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 1,
diff --git a/dashboards/victoriametrics.json b/dashboards/victoriametrics.json
index 3bd3a4eaa..911e68666 100644
--- a/dashboards/victoriametrics.json
+++ b/dashboards/victoriametrics.json
@@ -1,5 +1,6 @@
 {
   "__inputs": [],
+  "__elements": [],
   "__requires": [
     {
       "type": "grafana",
@@ -51,13 +52,13 @@
       }
     ]
   },
-  "description": "Overview for single node VictoriaMetrics v1.73.0 or higher",
+  "description": "Overview for single node VictoriaMetrics v1.77.0 or higher",
   "editable": true,
   "fiscalYearStartMonth": 0,
   "gnetId": 10229,
   "graphTooltip": 0,
   "id": null,
-  "iteration": 1645085513510,
+  "iteration": 1650637176348,
   "links": [
     {
       "icon": "doc",
@@ -189,6 +190,7 @@
     },
     {
       "datasource": {
+        "type": "prometheus",
         "uid": "$ds"
       },
       "description": "Total amount of used disk space",
@@ -238,8 +240,12 @@
       "pluginVersion": "8.3.5",
       "targets": [
         {
-          "exemplar": true,
-          "expr": "sum(vm_data_size_bytes{job=~\"$job\", type!=\"indexdb\"})",
+          "datasource": {
+            "type": "prometheus",
+            "uid": "$ds"
+          },
+          "exemplar": false,
+          "expr": "sum(vm_data_size_bytes{job=~\"$job\"})",
           "format": "time_series",
           "instant": true,
           "interval": "",
@@ -1402,7 +1408,7 @@
                 "uid": "$ds"
               },
               "exemplar": true,
-              "expr": "1 - (\n sum(rate(vm_cache_misses_total{job=~\"$job\", instance=~\"$instance\"}[5m])) by (type) /\n sum(rate(vm_cache_requests_total{job=~\"$job\", instance=~\"$instance\"}[5m])) by (type)\n)",
+              "expr": "1 - (\n sum(rate(vm_cache_misses_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) by (type) /\n sum(rate(vm_cache_requests_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) by (type)\n)",
               "interval": "",
               "legendFormat": "{{type}}",
               "refId": "A"
@@ -2405,7 +2411,7 @@
           "steppedLine": false,
           "targets": [
             {
-              "expr": "sum(rate(vm_rows_merged_total{job=~\"$job\", instance=~\"$instance\"}[5m])) by(type)",
+              "expr": "sum(rate(vm_rows_merged_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) by(type)",
               "legendFormat": "{{type}}",
               "refId": "A"
             }
@@ -2500,7 +2506,7 @@
           "steppedLine": false,
           "targets": [
             {
-              "expr": "sum(rate(vm_log_messages_total{job=~\"$job\", instance=~\"$instance\"}[5m])) by (level) ",
+              "expr": "sum(rate(vm_log_messages_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) by (level) ",
               "format": "time_series",
               "hide": false,
               "intervalFactor": 1,
@@ -2616,7 +2622,7 @@
           "steppedLine": false,
           "targets": [
             {
-              "expr": "sum(rate(vm_new_timeseries_created_total{job=~\"$job\", instance=~\"$instance\"}[5m]))",
+              "expr": "sum(rate(vm_new_timeseries_created_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))",
               "interval": "",
               "legendFormat": "churn rate",
               "refId": "A"
@@ -2666,9 +2672,10 @@
           "dashLength": 10,
           "dashes": false,
           "datasource": {
+            "type": "prometheus",
             "uid": "$ds"
           },
-          "description": "Slow queries rate according to `search.logSlowQueryDuration` flag, which is `5s` by default.",
+          "description": "Shows the rate of adding new items to the index. It should correlate with `Slow inserts` and `Churn rate` graphs and could help to determine the pressure on indexdb.",
           "fieldConfig": {
             "defaults": {
               "links": []
@@ -2684,7 +2691,7 @@
             "y": 33
           },
           "hiddenSeries": false,
-          "id": 60,
+          "id": 96,
           "legend": {
             "alignAsTable": true,
             "avg": true,
@@ -2699,7 +2706,6 @@
           },
           "lines": true,
           "linewidth": 1,
-          "links": [],
           "nullPointMode": "null",
           "options": {
             "alertThreshold": true
@@ -2715,17 +2721,20 @@
           "steppedLine": false,
           "targets": [
             {
-              "expr": "sum(rate(vm_slow_queries_total{job=~\"$job\", instance=~\"$instance\"}[5m]))",
-              "format": "time_series",
-              "hide": false,
-              "intervalFactor": 1,
-              "legendFormat": "slow queries rate",
+              "datasource": {
+                "type": "prometheus",
+                "uid": "$ds"
+              },
+              "exemplar": true,
+              "expr": "sum(rate(vm_indexdb_items_added_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))",
+              "interval": "",
+              "legendFormat": "items",
               "refId": "A"
             }
           ],
           "thresholds": [],
           "timeRegions": [],
-          "title": "Slow queries rate ($instance)",
+          "title": "IndexDB items rate ($instance)",
           "tooltip": {
             "shared": true,
             "sort": 0,
@@ -2739,12 +2748,14 @@
           },
           "yaxes": [
             {
+              "$$hashKey": "object:76",
               "format": "short",
               "logBase": 1,
               "min": "0",
               "show": true
             },
             {
+              "$$hashKey": "object:77",
               "format": "short",
               "logBase": 1,
               "min": "0",
@@ -2773,7 +2784,7 @@
           "fill": 1,
           "fillGradient": 0,
           "gridPos": {
-            "h": 9,
+            "h": 8,
             "w": 12,
             "x": 0,
             "y": 41
@@ -2810,7 +2821,7 @@
           "steppedLine": false,
           "targets": [
             {
-              "expr": "sum(rate(vm_slow_row_inserts_total{job=~\"$job\", instance=~\"$instance\"}[5m])) / sum(rate(vm_rows_inserted_total{job=~\"$job\", instance=~\"$instance\"}[5m]))",
+              "expr": "sum(rate(vm_slow_row_inserts_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) / sum(rate(vm_rows_inserted_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))",
               "format": "time_series",
               "hide": false,
               "interval": "",
@@ -2870,7 +2881,7 @@
           "datasource": {
             "uid": "$ds"
           },
-          "description": "VictoriaMetrics limits the number of labels per each metric with `-maxLabelsPerTimeseries` command-line flag.\n\nThis prevents from ingesting metrics with too many labels. The value of `maxLabelsPerTimeseries` must be adjusted for your workload.\n\nWhen limit is exceeded (graph is > 0) - extra labels are dropped, which could result in unexpected identical time series.",
+          "description": "Slow queries rate according to `search.logSlowQueryDuration` flag, which is `5s` by default.",
           "fieldConfig": {
             "defaults": {
               "links": []
@@ -2880,20 +2891,22 @@
           "fill": 1,
           "fillGradient": 0,
           "gridPos": {
-            "h": 9,
+            "h": 8,
             "w": 12,
             "x": 12,
             "y": 41
           },
           "hiddenSeries": false,
-          "id": 74,
+          "id": 60,
           "legend": {
             "alignAsTable": true,
             "avg": true,
             "current": true,
             "max": true,
             "min": false,
-            "show": false,
+            "show": true,
+            "sort": "current",
+            "sortDesc": true,
             "total": false,
             "values": true
           },
@@ -2915,19 +2928,17 @@
           "steppedLine": false,
           "targets": [
             {
-              "exemplar": true,
-              "expr": "sum(increase(vm_metrics_with_dropped_labels_total{job=~\"$job\", instance=~\"$instance\"}[5m]))",
+              "expr": "sum(rate(vm_slow_queries_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))",
               "format": "time_series",
               "hide": false,
-              "interval": "",
               "intervalFactor": 1,
-              "legendFormat": "limit exceeded",
+              "legendFormat": "slow queries rate",
               "refId": "A"
             }
           ],
           "thresholds": [],
           "timeRegions": [],
-          "title": "Labels limit exceeded ($instance)",
+          "title": "Slow queries rate ($instance)",
           "tooltip": {
             "shared": true,
             "sort": 0,
@@ -2941,7 +2952,6 @@
           },
           "yaxes": [
             {
-              "decimals": 2,
               "format": "short",
               "logBase": 1,
               "min": "0",
@@ -2967,36 +2977,28 @@
             "type": "prometheus",
             "uid": "$ds"
           },
-          "description": "Shows whether background merges were deferred because of not enough of disk space. \n\nBefore starting the data parts merge (improves compression and reduces number of files), VictoriaMetrics estimates if there is enough of disk space for the operation. If panel shows values higher than zero, it means no disk space left for merging operation.\n\nConsider extending the disk size or decreasing retention. ",
-          "fieldConfig": {
-            "defaults": {
-              "links": []
-            },
-            "overrides": []
-          },
-          "fill": 1,
+          "description": "Shows the percentage of used cache size from the allowed size by type. \nValues close to 100% show the maximum potential utilization.\nValues close to 0% show that cache is underutilized.",
+          "fill": 0,
           "fillGradient": 0,
           "gridPos": {
-            "h": 9,
+            "h": 8,
             "w": 12,
             "x": 0,
-            "y": 50
+            "y": 49
           },
           "hiddenSeries": false,
-          "id": 88,
+          "id": 90,
           "legend": {
-            "alignAsTable": true,
-            "avg": true,
-            "current": true,
-            "max": true,
+            "avg": false,
+            "current": false,
+            "max": false,
             "min": false,
             "show": false,
             "total": false,
-            "values": true
+            "values": false
           },
           "lines": true,
           "linewidth": 1,
-          "links": [],
           "nullPointMode": "null",
           "options": {
             "alertThreshold": true
@@ -3017,21 +3019,18 @@
                 "uid": "$ds"
               },
               "exemplar": true,
-              "expr": "sum(vm_merge_need_free_disk_space{job=~\"$job\", instance=~\"$instance\"}) by(type)",
-              "format": "time_series",
-              "hide": false,
+              "expr": "vm_cache_size_bytes{job=~\"$job\", instance=~\"$instance\"} / vm_cache_size_max_bytes{job=~\"$job\", instance=~\"$instance\"}",
               "interval": "",
-              "intervalFactor": 1,
-              "legendFormat": "{{ type }} deferred",
+              "legendFormat": "{{type}}",
               "refId": "A"
             }
           ],
           "thresholds": [],
           "timeRegions": [],
-          "title": "Merges deferred ($instance)",
+          "title": "Cache usage % ($instance)",
           "tooltip": {
             "shared": true,
-            "sort": 0,
+            "sort": 2,
             "value_type": "individual"
           },
           "type": "graph",
@@ -3042,16 +3041,15 @@
           },
           "yaxes": [
             {
-              "decimals": 2,
-              "format": "short",
+              "$$hashKey": "object:235",
+              "format": "percentunit",
               "logBase": 1,
-              "min": "0",
               "show": true
             },
             {
+              "$$hashKey": "object:236",
               "format": "short",
               "logBase": 1,
-              "min": "0",
               "show": true
             }
           ],
@@ -3065,31 +3063,38 @@
           "dashLength": 10,
           "dashes": false,
           "datasource": {
-            "type": "prometheus",
             "uid": "$ds"
           },
-          "description": "Shows the percentage of used cache size from the allowed size by type. \nValues close to 100% show the maximum potential utilization.\nValues close to 0% show that cache is underutilized.",
-          "fill": 0,
+          "description": "VictoriaMetrics limits the number of labels per each metric with `-maxLabelsPerTimeseries` command-line flag.\n\nThis prevents from ingesting metrics with too many labels. The value of `maxLabelsPerTimeseries` must be adjusted for your workload.\n\nWhen limit is exceeded (graph is > 0) - extra labels are dropped, which could result in unexpected identical time series.",
+          "fieldConfig": {
+            "defaults": {
+              "links": []
+            },
+            "overrides": []
+          },
+          "fill": 1,
           "fillGradient": 0,
           "gridPos": {
-            "h": 9,
+            "h": 8,
             "w": 12,
             "x": 12,
-            "y": 50
+            "y": 49
           },
           "hiddenSeries": false,
-          "id": 90,
+          "id": 74,
           "legend": {
-            "avg": false,
-            "current": false,
-            "max": false,
+            "alignAsTable": true,
+            "avg": true,
+            "current": true,
+            "max": true,
             "min": false,
             "show": false,
             "total": false,
-            "values": false
+            "values": true
           },
           "lines": true,
           "linewidth": 1,
+          "links": [],
           "nullPointMode": "null",
           "options": {
             "alertThreshold": true
@@ -3105,23 +3110,22 @@
           "steppedLine": false,
           "targets": [
             {
-              "datasource": {
-                "type": "prometheus",
-                "uid": "$ds"
-              },
               "exemplar": true,
-              "expr": "vm_cache_size_bytes{job=~\"$job\", instance=~\"$instance\"} / vm_cache_size_max_bytes{job=~\"$job\", instance=~\"$instance\"}",
+              "expr": "sum(increase(vm_metrics_with_dropped_labels_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))",
+              "format": "time_series",
+              "hide": false,
               "interval": "",
-              "legendFormat": "{{type}}",
+              "intervalFactor": 1,
+              "legendFormat": "limit exceeded",
               "refId": "A"
             }
           ],
           "thresholds": [],
           "timeRegions": [],
-          "title": "Cache usage % ($instance)",
+          "title": "Labels limit exceeded ($instance)",
           "tooltip": {
             "shared": true,
-            "sort": 2,
+            "sort": 0,
             "value_type": "individual"
           },
           "type": "graph",
@@ -3132,15 +3136,16 @@
           },
           "yaxes": [
             {
-              "$$hashKey": "object:235",
-              "format": "percentunit",
+              "decimals": 2,
+              "format": "short",
               "logBase": 1,
+              "min": "0",
               "show": true
             },
             {
-              "$$hashKey": "object:236",
               "format": "short",
               "logBase": 1,
+              "min": "0",
               "show": true
             }
           ],
@@ -3361,7 +3366,7 @@
                 "type": "prometheus",
                 "uid": "$ds"
               },
-              "expr": "rate(process_cpu_seconds_total{job=~\"$job\", instance=~\"$instance\"}[5m])",
+              "expr": "rate(process_cpu_seconds_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])",
               "format": "time_series",
               "interval": "",
               "intervalFactor": 1,
@@ -3584,7 +3589,7 @@
           "steppedLine": false,
           "targets": [
             {
-              "expr": "sum(rate(process_io_storage_read_bytes_total{job=~\"$job\", instance=~\"$instance\"}[5m]))",
+              "expr": "sum(rate(process_io_storage_read_bytes_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))",
               "format": "time_series",
               "hide": false,
               "interval": "",
@@ -3593,7 +3598,7 @@
               "refId": "A"
             },
             {
-              "expr": "sum(rate(process_io_storage_written_bytes_total{job=~\"$job\", instance=~\"$instance\"}[5m]))",
+              "expr": "sum(rate(process_io_storage_written_bytes_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))",
               "format": "time_series",
               "hide": false,
               "interval": "",
@@ -3782,7 +3787,7 @@
           "steppedLine": false,
           "targets": [
             {
-              "expr": "sum(rate(go_gc_duration_seconds_sum{job=~\"$job\", instance=~\"$instance\"}[5m]))\n/\nsum(rate(go_gc_duration_seconds_count{job=~\"$job\", instance=~\"$instance\"}[5m]))",
+              "expr": "sum(rate(go_gc_duration_seconds_sum{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))\n/\nsum(rate(go_gc_duration_seconds_count{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))",
               "format": "time_series",
               "intervalFactor": 2,
               "legendFormat": "avg gc duration",
@@ -4208,11 +4213,22 @@
         "tagsQuery": "",
         "type": "query",
         "useTags": false
+      },
+      {
+        "datasource": {
+          "type": "prometheus",
+          "uid": "${ds}"
+        },
+        "filters": [],
+        "hide": 0,
+        "name": "adhoc",
+        "skipUrlSync": false,
+        "type": "adhoc"
       }
     ]
   },
   "time": {
-    "from": "now-30m",
+    "from": "now-3h",
     "to": "now"
   },
   "timepicker": {
diff --git a/dashboards/vmagent.json b/dashboards/vmagent.json
index 5465a4fc9..b09a658e7 100644
--- a/dashboards/vmagent.json
+++ b/dashboards/vmagent.json
@@ -1306,7 +1306,7 @@
       "targets": [
         {
           "exemplar": true,
-          "expr": "sum(rate(vm_log_messages_total{job=~\"$job\", instance=~\"$instance\"}[5m])) by (level) ",
+          "expr": "sum(rate(vm_log_messages_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) by (level) ",
           "format": "time_series",
           "hide": false,
           "interval": "",
@@ -3247,7 +3247,7 @@
                 "uid": "$ds"
               },
               "exemplar": false,
-              "expr": "topk(5, sum(sum_over_time(scrape_series_added[5m])) by (job))",
+              "expr": "topk(5, sum(sum_over_time(scrape_series_added[$__rate_interval])) by (job))",
               "interval": "",
               "legendFormat": "{{ job }}",
               "refId": "A"
@@ -3336,7 +3336,7 @@
                 "uid": "$ds"
               },
               "exemplar": false,
-              "expr": "topk(5, sum(sum_over_time(scrape_series_added[5m])) by (instance))",
+              "expr": "topk(5, sum(sum_over_time(scrape_series_added[$__rate_interval])) by (instance))",
               "interval": "",
               "legendFormat": "{{ instance }}",
               "refId": "A"
@@ -3900,7 +3900,7 @@
           "steppedLine": false,
           "targets": [
             {
-              "expr": "sum(rate(process_io_storage_read_bytes_total{job=~\"$job\", instance=~\"$instance\"}[5m]))",
+              "expr": "sum(rate(process_io_storage_read_bytes_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))",
               "format": "time_series",
               "hide": false,
               "interval": "",
@@ -3909,7 +3909,7 @@
               "refId": "A"
             },
             {
-              "expr": "sum(rate(process_io_storage_written_bytes_total{job=~\"$job\", instance=~\"$instance\"}[5m]))",
+              "expr": "sum(rate(process_io_storage_written_bytes_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))",
               "format": "time_series",
               "hide": false,
               "interval": "",
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index 0130a17fe..49c9b5941 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -18,6 +18,9 @@ The following tip changes can be tested by building VictoriaMetrics components f
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): allow filtering targets by target url and by target labels with [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) on `http://vmagent:8429/targets` page. This may be useful when `vmagent` scrapes big number of targets. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1796).
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): reduce `-promscrape.config` reload duration when the config contains big number of jobs (aka [scrape_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) sections) and only a few of them are changed. Previously all the jobs were restarted. Now only the jobs with changed configs are restarted. This should reduce the probability of data miss because of slow config reload. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2270).
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): improve service discovery speed for big number of scrape targets. This should help when `vmagent` discovers big number of targets (e.g. thousands) in Kubernetes cluster. The service discovery speed now should scale with the number of CPU cores available to `vmagent`.
+* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add ability to attach node-level labels and annotations to discovered Kubernetes pod targets in the same way as Prometheus 2.35 does. See [this feature request](https://github.com/prometheus/prometheus/issues/9510) and [this pull request](https://github.com/prometheus/prometheus/pull/10080).
+* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add support for `tls_config` and `proxy_url` options at `oauth2` section in the same way as Prometheus does. See [oauth2 docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#oauth2).
+* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add support for `min_version` option at `tls_config` section in the same way as Prometheus does. See [tls_config docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config).
 * FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): add support for DNS-based discovery for notifiers in the same way as Prometheus does. See [these docs](https://docs.victoriametrics.com/vmalert.html#notifier-configuration-file) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2460).
 * FEATURE: allow specifying TLS cipher suites for incoming https requests via `-tlsCipherSuites` command-line flag. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2404).
 * FEATURE: allow specifying TLS cipher suites for mTLS connections between cluster components via `-cluster.tlsCipherSuites` command-line flag. See [these docs](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection).
@@ -27,7 +30,7 @@ The following tip changes can be tested by building VictoriaMetrics components f
 
 * BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl.html): return non-zero exit code on error. This allows handling `vmctl` errors in shell scripts. Previously `vmctl` was returning 0 exit code on error. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2322).
 * BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): properly show `scrape_timeout` and `scrape_interval` options at `http://vmagent:8429/config` page. Previously these options weren't displayed even if they were set in `-promscrape.config`.
-* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): properly handle joins on time series filtered by values. For example, `kube_pod_container_resource_requests{resource="cpu"} * on (namespace,pod) group_left() (kube_pod_status_phase{phase=~"Pending|Running"}==1)`. This query may result in `duplicate time series on the right side` error even if `==1` filter leaves only a single time series per `(namespace,pod)` labels. Now such query is properly executed.
+* BUGFIX: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): properly handle joins on time series filtered by values. For example, `kube_pod_container_resource_requests{resource="cpu"} * on (namespace,pod) group_left() (kube_pod_status_phase{phase=~"Pending|Running"}==1)`. This query could result in `duplicate time series on the right side` error even if `==1` filter leaves only a single time series per `(namespace,pod)` labels. Now such query is properly executed.
 * BUGFIX: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): properly handle `scalar default vector`, `scalar if vector` and `scalar ifnot vector` queries. Previously such queries could return unexpected results from the `vector` part.
 
 
diff --git a/lib/promauth/config.go b/lib/promauth/config.go
index b92ea185e..16528b1df 100644
--- a/lib/promauth/config.go
+++ b/lib/promauth/config.go
@@ -7,7 +7,9 @@ import (
 	"crypto/x509"
 	"encoding/base64"
 	"fmt"
+	"net/http"
 	"net/url"
+	"strings"
 	"sync"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
@@ -71,6 +73,16 @@ type TLSConfig struct {
 	KeyFile            string `yaml:"key_file,omitempty"`
 	ServerName         string `yaml:"server_name,omitempty"`
 	InsecureSkipVerify bool   `yaml:"insecure_skip_verify,omitempty"`
+	MinVersion         string `yaml:"min_version,omitempty"`
+}
+
+// String returns human-readable representation of tlsConfig
+func (tlsConfig *TLSConfig) String() string {
+	if tlsConfig == nil {
+		return ""
+	}
+	return fmt.Sprintf("ca_file=%q, cert_file=%q, key_file=%q, server_name=%q, insecure_skip_verify=%v, min_version=%q",
+		tlsConfig.CAFile, tlsConfig.CertFile, tlsConfig.KeyFile, tlsConfig.ServerName, tlsConfig.InsecureSkipVerify, tlsConfig.MinVersion)
 }
 
 // Authorization represents generic authorization config.
@@ -116,12 +128,14 @@ type OAuth2Config struct {
 	Scopes           []string          `yaml:"scopes,omitempty"`
 	TokenURL         string            `yaml:"token_url"`
 	EndpointParams   map[string]string `yaml:"endpoint_params,omitempty"`
+	TLSConfig        *TLSConfig        `yaml:"tls_config,omitempty"`
+	ProxyURL         string            `yaml:"proxy_url,omitempty"`
 }
 
 // String returns string representation of o.
 func (o *OAuth2Config) String() string {
-	return fmt.Sprintf("clientID=%q, clientSecret=%q, clientSecretFile=%q, Scopes=%q, tokenURL=%q, endpointParams=%q",
-		o.ClientID, o.ClientSecret, o.ClientSecretFile, o.Scopes, o.TokenURL, o.EndpointParams)
+	return fmt.Sprintf("clientID=%q, clientSecret=%q, clientSecretFile=%q, Scopes=%q, tokenURL=%q, endpointParams=%q, tlsConfig={%s}, proxyURL=%q",
+		o.ClientID, o.ClientSecret, o.ClientSecretFile, o.Scopes, o.TokenURL, o.EndpointParams, o.TLSConfig.String(), o.ProxyURL)
 }
 
 func (o *OAuth2Config) validate() error {
@@ -144,6 +158,7 @@ type oauth2ConfigInternal struct {
 	mu               sync.Mutex
 	cfg              *clientcredentials.Config
 	clientSecretFile string
+	ctx              context.Context
 	tokenSource      oauth2.TokenSource
 }
 
@@ -168,7 +183,27 @@ func newOAuth2ConfigInternal(baseDir string, o *OAuth2Config) (*oauth2ConfigInte
 		}
 		oi.cfg.ClientSecret = secret
 	}
-	oi.tokenSource = oi.cfg.TokenSource(context.Background())
+	ac, err := o.NewConfig(baseDir)
+	if err != nil {
+		return nil, fmt.Errorf("cannot initialize TLS config for OAuth2: %w", err)
+	}
+	tlsCfg := ac.NewTLSConfig()
+	var proxyURLFunc func(*http.Request) (*url.URL, error)
+	if o.ProxyURL != "" {
+		u, err := url.Parse(o.ProxyURL)
+		if err != nil {
+			return nil, fmt.Errorf("cannot parse proxy_url=%q: %w", o.ProxyURL, err)
+		}
+		proxyURLFunc = http.ProxyURL(u)
+	}
+	c := &http.Client{
+		Transport: &http.Transport{
+			TLSClientConfig: tlsCfg,
+			Proxy:           proxyURLFunc,
+		},
+	}
+	oi.ctx = context.WithValue(context.Background(), oauth2.HTTPClient, c)
+	oi.tokenSource = oi.cfg.TokenSource(oi.ctx)
 	return oi, nil
 }
 
@@ -195,7 +230,7 @@ func (oi *oauth2ConfigInternal) getTokenSource() (oauth2.TokenSource, error) {
 		return oi.tokenSource, nil
 	}
 	oi.cfg.ClientSecret = newSecret
-	oi.tokenSource = oi.cfg.TokenSource(context.Background())
+	oi.tokenSource = oi.cfg.TokenSource(oi.ctx)
 	return oi.tokenSource, nil
 }
 
@@ -205,6 +240,7 @@ type Config struct {
 	TLSRootCA             *x509.CertPool
 	TLSServerName         string
 	TLSInsecureSkipVerify bool
+	TLSMinVersion         uint16
 
 	getTLSCert    func(*tls.CertificateRequestInfo) (*tls.Certificate, error)
 	tlsCertDigest string
@@ -235,8 +271,8 @@ func (ac *Config) GetAuthHeader() string {
 
 // String returns human-readable representation for ac.
 func (ac *Config) String() string {
-	return fmt.Sprintf("AuthDigest=%s, TLSRootCA=%s, TLSCertificate=%s, TLSServerName=%s, TLSInsecureSkipVerify=%v",
-		ac.authDigest, ac.tlsRootCAString(), ac.tlsCertDigest, ac.TLSServerName, ac.TLSInsecureSkipVerify)
+	return fmt.Sprintf("AuthDigest=%s, TLSRootCA=%s, TLSCertificate=%s, TLSServerName=%s, TLSInsecureSkipVerify=%v, TLSMinVersion=%d",
+		ac.authDigest, ac.tlsRootCAString(), ac.tlsCertDigest, ac.TLSServerName, ac.TLSInsecureSkipVerify, ac.TLSMinVersion)
 }
 
 func (ac *Config) tlsRootCAString() string {
@@ -278,6 +314,7 @@ func (ac *Config) NewTLSConfig() *tls.Config {
 	tlsCfg.RootCAs = ac.TLSRootCA
 	tlsCfg.ServerName = ac.TLSServerName
 	tlsCfg.InsecureSkipVerify = ac.TLSInsecureSkipVerify
+	tlsCfg.MinVersion = ac.TLSMinVersion
 	return tlsCfg
 }
 
@@ -291,6 +328,11 @@ func (pcc *ProxyClientConfig) NewConfig(baseDir string) (*Config, error) {
 	return NewConfig(baseDir, pcc.Authorization, pcc.BasicAuth, pcc.BearerToken.String(), pcc.BearerTokenFile, nil, pcc.TLSConfig)
 }
 
+// NewConfig creates auth config for the given o.
+func (o *OAuth2Config) NewConfig(baseDir string) (*Config, error) {
+	return NewConfig(baseDir, nil, nil, "", "", nil, o.TLSConfig)
+}
+
 // NewConfig creates auth config from the given args.
 func NewConfig(baseDir string, az *Authorization, basicAuth *BasicAuthConfig, bearerToken, bearerTokenFile string, o *OAuth2Config, tlsConfig *TLSConfig) (*Config, error) {
 	var getAuthHeader func() string
@@ -410,6 +452,7 @@ func NewConfig(baseDir string, az *Authorization, basicAuth *BasicAuthConfig, be
 	tlsCertDigest := ""
 	tlsServerName := ""
 	tlsInsecureSkipVerify := false
+	tlsMinVersion := uint16(0)
 	if tlsConfig != nil {
 		tlsServerName = tlsConfig.ServerName
 		tlsInsecureSkipVerify = tlsConfig.InsecureSkipVerify
@@ -441,11 +484,19 @@ func NewConfig(baseDir string, az *Authorization, basicAuth *BasicAuthConfig, be
 				return nil, fmt.Errorf("cannot parse data from `ca_file` %q", tlsConfig.CAFile)
 			}
 		}
+		if tlsConfig.MinVersion != "" {
+			v, err := parseTLSVersion(tlsConfig.MinVersion)
+			if err != nil {
+				return nil, fmt.Errorf("cannot parse `min_version`: %w", err)
+			}
+			tlsMinVersion = v
+		}
 	}
 	ac := &Config{
 		TLSRootCA:             tlsRootCA,
 		TLSServerName:         tlsServerName,
 		TLSInsecureSkipVerify: tlsInsecureSkipVerify,
+		TLSMinVersion:         tlsMinVersion,
 
 		getTLSCert:    getTLSCert,
 		tlsCertDigest: tlsCertDigest,
@@ -455,3 +506,18 @@ func NewConfig(baseDir string, az *Authorization, basicAuth *BasicAuthConfig, be
 	}
 	return ac, nil
 }
+
+func parseTLSVersion(s string) (uint16, error) {
+	switch strings.ToUpper(s) {
+	case "TLS13":
+		return tls.VersionTLS13, nil
+	case "TLS12":
+		return tls.VersionTLS12, nil
+	case "TLS11":
+		return tls.VersionTLS11, nil
+	case "TLS10":
+		return tls.VersionTLS10, nil
+	default:
+		return 0, fmt.Errorf("unsupported TLS version %q", s)
+	}
+}
diff --git a/lib/promscrape/discovery/kubernetes/api_watcher.go b/lib/promscrape/discovery/kubernetes/api_watcher.go
index 3f7e013a5..2fff3f249 100644
--- a/lib/promscrape/discovery/kubernetes/api_watcher.go
+++ b/lib/promscrape/discovery/kubernetes/api_watcher.go
@@ -74,8 +74,9 @@ func newAPIWatcher(apiServer string, ac *promauth.Config, sdc *SDConfig, swcFunc
 		}
 	}
 	selectors := sdc.Selectors
+	attachNodeMetadata := sdc.AttachMetadata.Node
 	proxyURL := sdc.ProxyURL.URL()
-	gw := getGroupWatcher(apiServer, ac, namespaces, selectors, proxyURL)
+	gw := getGroupWatcher(apiServer, ac, namespaces, selectors, attachNodeMetadata, proxyURL)
 	role := sdc.role()
 	return &apiWatcher{
 		role:             role,
@@ -163,7 +164,7 @@ func (aw *apiWatcher) getScrapeWorkObjects() []interface{} {
 }
 
 // groupWatcher watches for Kubernetes objects on the given apiServer with the given namespaces,
-// selectors using the given client.
+// selectors and attachNodeMetadata using the given client.
 type groupWatcher struct {
 	// Old Kubernetes doesn't support /apis/networking.k8s.io/v1/, so /apis/networking.k8s.io/v1beta1/ must be used instead.
 	// This flag is used for automatic substitution of v1 API path with v1beta1 API path during requests to apiServer.
@@ -173,9 +174,11 @@ type groupWatcher struct {
 	// This flag is used for automatic substitution of v1 API path with v1beta1 API path during requests to apiServer.
 	useDiscoveryV1Beta1 uint32
 
-	apiServer     string
-	namespaces    []string
-	selectors     []Selector
+	apiServer          string
+	namespaces         []string
+	selectors          []Selector
+	attachNodeMetadata bool
+
 	getAuthHeader func() string
 	client        *http.Client
 
@@ -183,7 +186,7 @@ type groupWatcher struct {
 	m  map[string]*urlWatcher
 }
 
-func newGroupWatcher(apiServer string, ac *promauth.Config, namespaces []string, selectors []Selector, proxyURL *url.URL) *groupWatcher {
+func newGroupWatcher(apiServer string, ac *promauth.Config, namespaces []string, selectors []Selector, attachNodeMetadata bool, proxyURL *url.URL) *groupWatcher {
 	var proxy func(*http.Request) (*url.URL, error)
 	if proxyURL != nil {
 		proxy = http.ProxyURL(proxyURL)
@@ -199,26 +202,28 @@ func newGroupWatcher(apiServer string, ac *promauth.Config, namespaces []string,
 		Timeout: *apiServerTimeout,
 	}
 	return &groupWatcher{
-		apiServer:     apiServer,
+		apiServer:          apiServer,
+		namespaces:         namespaces,
+		selectors:          selectors,
+		attachNodeMetadata: attachNodeMetadata,
+
 		getAuthHeader: ac.GetAuthHeader,
-		namespaces:    namespaces,
-		selectors:     selectors,
 		client:        client,
 		m:             make(map[string]*urlWatcher),
 	}
 }
 
-func getGroupWatcher(apiServer string, ac *promauth.Config, namespaces []string, selectors []Selector, proxyURL *url.URL) *groupWatcher {
+func getGroupWatcher(apiServer string, ac *promauth.Config, namespaces []string, selectors []Selector, attachNodeMetadata bool, proxyURL *url.URL) *groupWatcher {
 	proxyURLStr := "<nil>"
 	if proxyURL != nil {
 		proxyURLStr = proxyURL.String()
 	}
-	key := fmt.Sprintf("apiServer=%s, namespaces=%s, selectors=%s, proxyURL=%s, authConfig=%s",
-		apiServer, namespaces, selectorsKey(selectors), proxyURLStr, ac.String())
+	key := fmt.Sprintf("apiServer=%s, namespaces=%s, selectors=%s, attachNodeMetadata=%v, proxyURL=%s, authConfig=%s",
+		apiServer, namespaces, selectorsKey(selectors), attachNodeMetadata, proxyURLStr, ac.String())
 	groupWatchersLock.Lock()
 	gw := groupWatchers[key]
 	if gw == nil {
-		gw = newGroupWatcher(apiServer, ac, namespaces, selectors, proxyURL)
+		gw = newGroupWatcher(apiServer, ac, namespaces, selectors, attachNodeMetadata, proxyURL)
 		groupWatchers[key] = gw
 	}
 	groupWatchersLock.Unlock()
@@ -246,9 +251,9 @@ var (
 )
 
 func (gw *groupWatcher) getObjectByRoleLocked(role, namespace, name string) object {
-	if gw == nil {
-		// this is needed for testing
-		return nil
+	if role == "node" {
+		// Node objects have no namespace
+		namespace = ""
 	}
 	key := namespace + "/" + name
 	for _, uw := range gw.m {
@@ -256,7 +261,7 @@ func (gw *groupWatcher) getObjectByRoleLocked(role, namespace, name string) obje
 			// Role mismatch
 			continue
 		}
-		if uw.namespace != "" && uw.namespace != namespace {
+		if namespace != "" && uw.namespace != "" && uw.namespace != namespace {
 			// Namespace mismatch
 			continue
 		}
@@ -273,6 +278,9 @@ func (gw *groupWatcher) startWatchersForRole(role string, aw *apiWatcher) {
 		gw.startWatchersForRole("pod", nil)
 		gw.startWatchersForRole("service", nil)
 	}
+	if gw.attachNodeMetadata && role == "pod" {
+		gw.startWatchersForRole("node", nil)
+	}
 	paths := getAPIPathsWithNamespaces(role, gw.namespaces, gw.selectors)
 	for _, path := range paths {
 		apiURL := gw.apiServer + path
@@ -290,11 +298,14 @@ func (gw *groupWatcher) startWatchersForRole(role string, aw *apiWatcher) {
 		if needStart {
 			uw.reloadObjects()
 			go uw.watchForUpdates()
-			if role == "endpoints" || role == "endpointslice" {
-				// Refresh endpoints and enpointslices targets in background, since they depend on other object types such as pod and service.
+			if role == "endpoints" || role == "endpointslice" || (gw.attachNodeMetadata && role == "pod") {
+				// Refresh targets in background, since they depend on other object types such as pod, service or node.
+				// This should guarantee that the ScrapeWork objects for these objects are properly updated
+				// as soon as the objects they depend on are updated.
 				// This should fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240 .
 				go func() {
-					sleepTime := 20 * time.Second
+					const minSleepTime = 5 * time.Second
+					sleepTime := minSleepTime
 					for {
 						time.Sleep(sleepTime)
 						startTime := time.Now()
@@ -302,9 +313,12 @@ func (gw *groupWatcher) startWatchersForRole(role string, aw *apiWatcher) {
 						if uw.needUpdateScrapeWorks {
 							uw.needUpdateScrapeWorks = false
 							uw.updateScrapeWorksLocked(uw.objectsByKey, uw.aws)
+							sleepTime = time.Since(startTime)
+							if sleepTime < minSleepTime {
+								sleepTime = minSleepTime
+							}
 						}
 						gw.mu.Unlock()
-						sleepTime = time.Since(startTime)
 					}
 				}()
 			}
@@ -562,7 +576,7 @@ func (uw *urlWatcher) reloadObjects() string {
 	uw.updateScrapeWorksLocked(objectsAdded, uw.aws)
 	uw.needUpdateScrapeWorks = false
 	if len(objectsRemoved) > 0 || len(objectsUpdated) > 0 || len(objectsAdded) > 0 {
-		uw.maybeUpdateEndpointsScrapeWorksLocked()
+		uw.maybeUpdateDependedScrapeWorksLocked()
 	}
 	uw.gw.mu.Unlock()
 
@@ -711,7 +725,7 @@ func (uw *urlWatcher) updateObjectLocked(key string, o object) {
 			aw.setScrapeWorks(uw, key, labels)
 		}
 	}
-	uw.maybeUpdateEndpointsScrapeWorksLocked()
+	uw.maybeUpdateDependedScrapeWorksLocked()
 }
 
 func (uw *urlWatcher) removeObjectLocked(key string) {
@@ -724,23 +738,32 @@ func (uw *urlWatcher) removeObjectLocked(key string) {
 	for aw := range uw.aws {
 		aw.removeScrapeWorks(uw, key)
 	}
-	uw.maybeUpdateEndpointsScrapeWorksLocked()
+	uw.maybeUpdateDependedScrapeWorksLocked()
 }
 
-func (uw *urlWatcher) maybeUpdateEndpointsScrapeWorksLocked() {
-	if uw.role != "pod" && uw.role != "service" {
+func (uw *urlWatcher) maybeUpdateDependedScrapeWorksLocked() {
+	role := uw.role
+	attachNodeMetadata := uw.gw.attachNodeMetadata
+	if !(role == "pod" || role == "service" || (attachNodeMetadata && role == "node")) {
 		// Nothing to update
 		return
 	}
 	namespace := uw.namespace
 	for _, uwx := range uw.gw.m {
-		if uwx.role != "endpoints" && uwx.role != "endpointslice" {
+		if namespace != "" && uwx.namespace != "" && uwx.namespace != namespace {
+			// Namespace mismatch
+			continue
+		}
+		if (role == "pod" || role == "service") && (uwx.role == "endpoints" || uwx.role == "endpointslice") {
+			// endpoints and endpointslice objects depend on pods and service objects
+			uwx.needUpdateScrapeWorks = true
 			continue
 		}
-		if uwx.namespace != "" && uwx.namespace != namespace {
+		if attachNodeMetadata && role == "node" && uwx.role == "pod" {
+			// pod objects depend on node objects if attachNodeMetadata is set
+			uwx.needUpdateScrapeWorks = true
 			continue
 		}
-		uwx.needUpdateScrapeWorks = true
 	}
 }
 
diff --git a/lib/promscrape/discovery/kubernetes/endpoints.go b/lib/promscrape/discovery/kubernetes/endpoints.go
index f011c0c45..37bf270af 100644
--- a/lib/promscrape/discovery/kubernetes/endpoints.go
+++ b/lib/promscrape/discovery/kubernetes/endpoints.go
@@ -132,7 +132,11 @@ func (eps *Endpoints) getTargetLabels(gw *groupWatcher) []map[string]string {
 				m := map[string]string{
 					"__address__": addr,
 				}
-				p.appendCommonLabels(m)
+				if !p.appendCommonLabels(m, gw) {
+					// The corresponding node is filtered out with label or field selectors.
+					// Do not generate endpoint labels in this case.
+					continue
+				}
 				p.appendContainerLabels(m, c, &cp)
 				if svc != nil {
 					svc.appendCommonLabels(m)
@@ -153,13 +157,16 @@ func appendEndpointLabelsForAddresses(ms []map[string]string, gw *groupWatcher,
 				p = o.(*Pod)
 			}
 		}
-		m := getEndpointLabelsForAddressAndPort(podPortsSeen, eps, ea, epp, p, svc, ready)
-		ms = append(ms, m)
+		m := getEndpointLabelsForAddressAndPort(gw, podPortsSeen, eps, ea, epp, p, svc, ready)
+		if m != nil {
+			ms = append(ms, m)
+		}
 	}
 	return ms
 }
 
-func getEndpointLabelsForAddressAndPort(podPortsSeen map[*Pod][]int, eps *Endpoints, ea EndpointAddress, epp EndpointPort, p *Pod, svc *Service, ready string) map[string]string {
+func getEndpointLabelsForAddressAndPort(gw *groupWatcher, podPortsSeen map[*Pod][]int, eps *Endpoints, ea EndpointAddress, epp EndpointPort,
+	p *Pod, svc *Service, ready string) map[string]string {
 	m := getEndpointLabels(eps.Metadata, ea, epp, ready)
 	if svc != nil {
 		svc.appendCommonLabels(m)
@@ -169,7 +176,11 @@ func getEndpointLabelsForAddressAndPort(podPortsSeen map[*Pod][]int, eps *Endpoi
 	if ea.TargetRef.Kind != "Pod" || p == nil {
 		return m
 	}
-	p.appendCommonLabels(m)
+	if !p.appendCommonLabels(m, gw) {
+		// The corresponding node is filtered out with label or field selectors.
+		// Do not generate endpoint labels in this case.
+		return nil
+	}
 	// always add pod targetRef, even if epp port doesn't match container port
 	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2134
 	if _, ok := podPortsSeen[p]; !ok {
diff --git a/lib/promscrape/discovery/kubernetes/endpointslice.go b/lib/promscrape/discovery/kubernetes/endpointslice.go
index 620b03e05..bf8192843 100644
--- a/lib/promscrape/discovery/kubernetes/endpointslice.go
+++ b/lib/promscrape/discovery/kubernetes/endpointslice.go
@@ -51,7 +51,10 @@ func (eps *EndpointSlice) getTargetLabels(gw *groupWatcher) []map[string]string
 		}
 		for _, epp := range eps.Ports {
 			for _, addr := range ess.Addresses {
-				ms = append(ms, getEndpointSliceLabelsForAddressAndPort(podPortsSeen, addr, eps, ess, epp, p, svc))
+				m := getEndpointSliceLabelsForAddressAndPort(gw, podPortsSeen, addr, eps, ess, epp, p, svc)
+				if m != nil {
+					ms = append(ms, m)
+				}
 			}
 
 		}
@@ -76,7 +79,11 @@ func (eps *EndpointSlice) getTargetLabels(gw *groupWatcher) []map[string]string
 				m := map[string]string{
 					"__address__": addr,
 				}
-				p.appendCommonLabels(m)
+				if !p.appendCommonLabels(m, gw) {
+					// The corresponding node is filtered out with label or field selectors.
+					// Do not generate endpointslice labels in this case.
+					continue
+				}
 				p.appendContainerLabels(m, c, &cp)
 				if svc != nil {
 					svc.appendCommonLabels(m)
@@ -93,7 +100,8 @@ func (eps *EndpointSlice) getTargetLabels(gw *groupWatcher) []map[string]string
 // enriches labels with TargetRef
 // p appended to seen Ports
 // if TargetRef matches
-func getEndpointSliceLabelsForAddressAndPort(podPortsSeen map[*Pod][]int, addr string, eps *EndpointSlice, ea Endpoint, epp EndpointPort, p *Pod, svc *Service) map[string]string {
+func getEndpointSliceLabelsForAddressAndPort(gw *groupWatcher, podPortsSeen map[*Pod][]int, addr string, eps *EndpointSlice, ea Endpoint, epp EndpointPort,
+	p *Pod, svc *Service) map[string]string {
 	m := getEndpointSliceLabels(eps, addr, ea, epp)
 	if svc != nil {
 		svc.appendCommonLabels(m)
@@ -108,7 +116,11 @@ func getEndpointSliceLabelsForAddressAndPort(podPortsSeen map[*Pod][]int, addr s
 	if _, ok := podPortsSeen[p]; !ok {
 		podPortsSeen[p] = []int{}
 	}
-	p.appendCommonLabels(m)
+	if !p.appendCommonLabels(m, gw) {
+		// The corresponding node is filtered out with label or field selectors.
+		// Do not generate endpointslice labels in this case.
+		return nil
+	}
 	for _, c := range p.Spec.Containers {
 		for _, cp := range c.Ports {
 			if cp.ContainerPort == epp.Port {
diff --git a/lib/promscrape/discovery/kubernetes/kubernetes.go b/lib/promscrape/discovery/kubernetes/kubernetes.go
index fd2d517ce..4be5aa783 100644
--- a/lib/promscrape/discovery/kubernetes/kubernetes.go
+++ b/lib/promscrape/discovery/kubernetes/kubernetes.go
@@ -27,6 +27,7 @@ type SDConfig struct {
 	ProxyURL         *proxy.URL                `yaml:"proxy_url,omitempty"`
 	Namespaces       Namespaces                `yaml:"namespaces,omitempty"`
 	Selectors        []Selector                `yaml:"selectors,omitempty"`
+	AttachMetadata   AttachMetadata            `yaml:"attach_metadata,omitempty"`
 
 	cfg      *apiConfig
 	startErr error
@@ -41,6 +42,13 @@ func (sdc *SDConfig) role() string {
 	return sdc.Role
 }
 
+// AttachMetadata represents `attach_metadata` option at `kuberentes_sd_config`.
+//
+// See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config
+type AttachMetadata struct {
+	Node bool `yaml:"node"`
+}
+
 // Namespaces represents namespaces for SDConfig
 type Namespaces struct {
 	OwnNamespace bool     `yaml:"own_namespace"`
diff --git a/lib/promscrape/discovery/kubernetes/pod.go b/lib/promscrape/discovery/kubernetes/pod.go
index 8a88ffaca..306116197 100644
--- a/lib/promscrape/discovery/kubernetes/pod.go
+++ b/lib/promscrape/discovery/kubernetes/pod.go
@@ -103,26 +103,24 @@ func (p *Pod) getTargetLabels(gw *groupWatcher) []map[string]string {
 		return nil
 	}
 	var ms []map[string]string
-	ms = appendPodLabels(ms, p, p.Spec.Containers, "false")
-	ms = appendPodLabels(ms, p, p.Spec.InitContainers, "true")
+	ms = appendPodLabels(ms, gw, p, p.Spec.Containers, "false")
+	ms = appendPodLabels(ms, gw, p, p.Spec.InitContainers, "true")
 	return ms
 }
 
-func appendPodLabels(ms []map[string]string, p *Pod, cs []Container, isInit string) []map[string]string {
+func appendPodLabels(ms []map[string]string, gw *groupWatcher, p *Pod, cs []Container, isInit string) []map[string]string {
 	for _, c := range cs {
 		for _, cp := range c.Ports {
-			m := getPodLabels(p, c, &cp, isInit)
-			ms = append(ms, m)
+			ms = appendPodLabelsInternal(ms, gw, p, c, &cp, isInit)
 		}
 		if len(c.Ports) == 0 {
-			m := getPodLabels(p, c, nil, isInit)
-			ms = append(ms, m)
+			ms = appendPodLabelsInternal(ms, gw, p, c, nil, isInit)
 		}
 	}
 	return ms
 }
 
-func getPodLabels(p *Pod, c Container, cp *ContainerPort, isInit string) map[string]string {
+func appendPodLabelsInternal(ms []map[string]string, gw *groupWatcher, p *Pod, c Container, cp *ContainerPort, isInit string) []map[string]string {
 	addr := p.Status.PodIP
 	if cp != nil {
 		addr = discoveryutils.JoinHostPort(addr, cp.ContainerPort)
@@ -131,9 +129,13 @@ func getPodLabels(p *Pod, c Container, cp *ContainerPort, isInit string) map[str
 		"__address__":                          addr,
 		"__meta_kubernetes_pod_container_init": isInit,
 	}
-	p.appendCommonLabels(m)
+	if !p.appendCommonLabels(m, gw) {
+		// The corresponding node is filtered out with label or field selectors.
+		// Do not generate pod labels in this case.
+		return ms
+	}
 	p.appendContainerLabels(m, c, cp)
-	return m
+	return append(ms, m)
 }
 
 func (p *Pod) appendContainerLabels(m map[string]string, c Container, cp *ContainerPort) {
@@ -145,7 +147,18 @@ func (p *Pod) appendContainerLabels(m map[string]string, c Container, cp *Contai
 	}
 }
 
-func (p *Pod) appendCommonLabels(m map[string]string) {
+func (p *Pod) appendCommonLabels(m map[string]string, gw *groupWatcher) bool {
+	if gw.attachNodeMetadata {
+		o := gw.getObjectByRoleLocked("node", p.Metadata.Namespace, p.Spec.NodeName)
+		if o == nil {
+			// The node associated with the pod is filtered out with label or field selectors,
+			// so do not generate labels for the pod.
+			return false
+		}
+		n := o.(*Node)
+		m["__meta_kubernetes_node_name"] = p.Spec.NodeName
+		n.Metadata.registerLabelsAndAnnotations("__meta_kubernetes_node", m)
+	}
 	m["__meta_kubernetes_pod_name"] = p.Metadata.Name
 	m["__meta_kubernetes_pod_ip"] = p.Status.PodIP
 	m["__meta_kubernetes_pod_ready"] = getPodReadyStatus(p.Status.Conditions)
@@ -163,6 +176,7 @@ func (p *Pod) appendCommonLabels(m map[string]string) {
 		}
 	}
 	p.Metadata.registerLabelsAndAnnotations("__meta_kubernetes_pod", m)
+	return true
 }
 
 func getPodController(ors []OwnerReference) *OwnerReference {


Test patch:
diff --git a/lib/promscrape/discovery/kubernetes/endpoints_test.go b/lib/promscrape/discovery/kubernetes/endpoints_test.go
index 4687784d7..845d92001 100644
--- a/lib/promscrape/discovery/kubernetes/endpoints_test.go
+++ b/lib/promscrape/discovery/kubernetes/endpoints_test.go
@@ -56,7 +56,7 @@ func TestParseEndpointsListSuccess(t *testing.T) {
           "addresses": [
             {
 	      "hostname": "aaa.bbb",
-	      "nodeName": "foobar",
+	      "nodeName": "test-node",
               "ip": "172.17.0.2",
               "targetRef": {
                 "kind": "Pod",
@@ -97,7 +97,7 @@ func TestParseEndpointsListSuccess(t *testing.T) {
 			"__meta_kubernetes_endpoint_address_target_kind":  "Pod",
 			"__meta_kubernetes_endpoint_address_target_name":  "coredns-6955765f44-lnp6t",
 			"__meta_kubernetes_endpoint_hostname":             "aaa.bbb",
-			"__meta_kubernetes_endpoint_node_name":            "foobar",
+			"__meta_kubernetes_endpoint_node_name":            "test-node",
 			"__meta_kubernetes_endpoint_port_name":            "https",
 			"__meta_kubernetes_endpoint_port_protocol":        "TCP",
 			"__meta_kubernetes_endpoint_ready":                "true",
diff --git a/lib/promscrape/discovery/kubernetes/node_test.go b/lib/promscrape/discovery/kubernetes/node_test.go
index 5192a21db..b42b85270 100644
--- a/lib/promscrape/discovery/kubernetes/node_test.go
+++ b/lib/promscrape/discovery/kubernetes/node_test.go
@@ -289,9 +289,28 @@ func TestParseNodeListSuccess(t *testing.T) {
 }
 
 func getSortedLabelss(objectsByKey map[string]object) [][]prompbmarshal.Label {
+	var gw groupWatcher
+	gw.m = map[string]*urlWatcher{
+		"node": {
+			role: "node",
+			objectsByKey: map[string]object{
+				"/test-node": &Node{
+					Metadata: ObjectMeta{
+						Labels: []prompbmarshal.Label{
+							{
+								Name:  "node-label",
+								Value: "xyz",
+							},
+						},
+					},
+				},
+			},
+		},
+	}
+	gw.attachNodeMetadata = true
 	var result [][]prompbmarshal.Label
 	for _, o := range objectsByKey {
-		labelss := o.getTargetLabels(nil)
+		labelss := o.getTargetLabels(&gw)
 		for _, labels := range labelss {
 			result = append(result, discoveryutils.GetSortedLabels(labels))
 		}
diff --git a/lib/promscrape/discovery/kubernetes/pod_test.go b/lib/promscrape/discovery/kubernetes/pod_test.go
index f697d1b84..f6b7681c1 100644
--- a/lib/promscrape/discovery/kubernetes/pod_test.go
+++ b/lib/promscrape/discovery/kubernetes/pod_test.go
@@ -146,7 +146,7 @@ func TestParsePodListSuccess(t *testing.T) {
         "restartPolicy": "Always",
         "terminationGracePeriodSeconds": 30,
         "dnsPolicy": "ClusterFirst",
-        "nodeName": "m01",
+        "nodeName": "test-node",
         "hostNetwork": true,
         "securityContext": {
           
@@ -243,21 +243,24 @@ func TestParsePodListSuccess(t *testing.T) {
 		discoveryutils.GetSortedLabels(map[string]string{
 			"__address__": "172.17.0.2:1234",
 
-			"__meta_kubernetes_namespace":                   "kube-system",
-			"__meta_kubernetes_pod_name":                    "etcd-m01",
-			"__meta_kubernetes_pod_ip":                      "172.17.0.2",
-			"__meta_kubernetes_pod_container_name":          "etcd",
-			"__meta_kubernetes_pod_container_port_name":     "foobar",
-			"__meta_kubernetes_pod_container_port_number":   "1234",
-			"__meta_kubernetes_pod_container_port_protocol": "TCP",
-			"__meta_kubernetes_pod_ready":                   "true",
-			"__meta_kubernetes_pod_phase":                   "Running",
-			"__meta_kubernetes_pod_node_name":               "m01",
-			"__meta_kubernetes_pod_host_ip":                 "172.17.0.2",
-			"__meta_kubernetes_pod_uid":                     "9d328156-75d1-411a-bdd0-aeacb53a38de",
-			"__meta_kubernetes_pod_controller_kind":         "Node",
-			"__meta_kubernetes_pod_controller_name":         "m01",
-			"__meta_kubernetes_pod_container_init":          "false",
+			"__meta_kubernetes_namespace":                    "kube-system",
+			"__meta_kubernetes_node_label_node_label":        "xyz",
+			"__meta_kubernetes_node_labelpresent_node_label": "true",
+			"__meta_kubernetes_node_name":                    "test-node",
+			"__meta_kubernetes_pod_name":                     "etcd-m01",
+			"__meta_kubernetes_pod_ip":                       "172.17.0.2",
+			"__meta_kubernetes_pod_container_name":           "etcd",
+			"__meta_kubernetes_pod_container_port_name":      "foobar",
+			"__meta_kubernetes_pod_container_port_number":    "1234",
+			"__meta_kubernetes_pod_container_port_protocol":  "TCP",
+			"__meta_kubernetes_pod_ready":                    "true",
+			"__meta_kubernetes_pod_phase":                    "Running",
+			"__meta_kubernetes_pod_node_name":                "test-node",
+			"__meta_kubernetes_pod_host_ip":                  "172.17.0.2",
+			"__meta_kubernetes_pod_uid":                      "9d328156-75d1-411a-bdd0-aeacb53a38de",
+			"__meta_kubernetes_pod_controller_kind":          "Node",
+			"__meta_kubernetes_pod_controller_name":          "m01",
+			"__meta_kubernetes_pod_container_init":           "false",
 
 			"__meta_kubernetes_pod_label_component": "etcd",
 			"__meta_kubernetes_pod_label_tier":      "control-plane",