Repository: VictoriaMetrics/VictoriaMetrics
Pull request #8719: docs\stream-aggregation: Describe dropping unneeded labels in more details
Author: login: makasim
Created: 2025-04-14T17:54:02Z | Merged: 2025-04-24T17:26:37Z



### Describe Your Changes

Follow-up on https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8715 and https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8681#issuecomment-2796127921

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres to [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).



Diff:
diff --git a/docs/victoriametrics/stream-aggregation.md b/docs/victoriametrics/stream-aggregation.md
index e2edbdf71096b..0249889045f56 100644
--- a/docs/victoriametrics/stream-aggregation.md
+++ b/docs/victoriametrics/stream-aggregation.md
@@ -1126,27 +1126,54 @@ See also [aggregation outputs](#aggregation-outputs).
 
 ## Dropping unneeded labels
 
-If you need dropping some labels from input samples before [input relabeling](#relabeling), [de-duplication](#deduplication)
-and [stream aggregation](#aggregation-outputs), then the following options exist:
-
-- To specify comma-separated list of label names to drop in `-streamAggr.dropInputLabels` command-line flag
-  or via `-remoteWrite.streamAggr.dropInputLabels` individually per each `-remoteWrite.url`.
-  For example, `-streamAggr.dropInputLabels=replica,az` instructs to drop `replica` and `az` labels from input samples
-  before applying de-duplication and stream aggregation.
-
-- To specify `drop_input_labels` list with the labels to drop in [stream aggregation config](#stream-aggregation-config).
-  For example, the following config drops `replica` label from input samples with the name `process_resident_memory_bytes`
-  before calculating the average over one minute:
-
-  ```yaml
-  - match: process_resident_memory_bytes
-    interval: 1m
-    drop_input_labels: [replica]
-    outputs: [avg]
-    keep_metric_names: true
-  ```
+To optimize performance and reduce [churn rate](https://docs.victoriametrics.com/guides/understand-your-setup-size/#churn-rate), it's important to drop unnecessary labels from incoming samples. 
+Dropping unnecessary labels can significantly enhance efficiency. 
+There are various strategies for label dropping, which can be implemented individually or combined.
+
+**Global Label Dropping** is configured using the `-streamAggr.dropInputLabels` flag. 
+It works in conjunction with the `-streamAggr.config` flag and applies to all matching sections within it.
+The labels are dropped before [input relabeling](#relabeling), [de-duplication](#deduplication), and [stream aggregation](#aggregation-outputs) are applied.
+This flag can be used with [vmagent](https://docs.victoriametrics.com/vmagent/), vminsert, and [vmsingle](https://docs.victoriametrics.com/single-server-victoriametrics/).
+
+The following example demonstrates how to drop the `replica` and `az` labels for both `foo` and `bar` remote write targets:
+```bash
+/path/to/vmagent \
+  -remoteWrite.url="http://foo/api/v1/write" \
+  -remoteWrite.url="http://bar/api/v1/write" \
+  -streamAggr.config="aggr.yaml" \
+  -streamAggr.dropInputLabels="replica,az"
+```
+
+**Per Remote Write Label Drop** is configured using the `-remoteWrite.streamAggr.dropInputLabels` flag.  
+It should be defined as many times as there are `-remoteWrite.url` flags. 
+To drop multiple labels for a remote write, use `^^` to separate them.  
+The labels are dropped before [input relabeling](#relabeling), [de-duplication](#deduplication), and [stream aggregation](#aggregation-outputs) are applied.
+This flag is available for [vmagent](https://docs.victoriametrics.com/vmagent/) only.
+
+In the example below, `replica` and `az` are dropped for the `foo` target,  
+while `instance` is dropped for the `bar` target:
+
+```bash
+/path/to/vmagent \
+  -remoteWrite.url="http://foo/api/v1/write" \
+  -remoteWrite.url="http://bar/api/v1/write" \
+  -remoteWrite.streamAggr.config="aggr.yaml" \
+  -remoteWrite.streamAggr.dropInputLabels="replica^^az" \
+  -remoteWrite.streamAggr.dropInputLabels="instance"
+```
 
-Typical use case is to drop `replica` label from samples, which are received from high availability replicas.
+**Config based label drop** can be defined within the [stream aggregation config](#stream-aggregation-config) using the `drop_input_labels` key.
+This method applies to configurations provided via either the `-streamAggr.config` or `-remoteWrite.streamAggr.config` flag.
+When specified, `drop_input_labels` takes precedence over any label drop definitions set via flags.
+
+Below is an example of an `aggr.yaml` configuration that drops the `replica` and `az` labels from `process_resident_memory_bytes` metrics:
+```yaml
+- match: 'process_resident_memory_bytes'
+  interval: '1m'
+  drop_input_labels: ['replica', 'az']
+  outputs: ['avg']
+  keep_metric_names: true
+```
 
 # Troubleshooting
 
diff --git a/lib/flagutil/array.go b/lib/flagutil/array.go
index fb086004fee8b..e612fd726dc2b 100644
--- a/lib/flagutil/array.go
+++ b/lib/flagutil/array.go
@@ -79,6 +79,14 @@ func NewArrayBytes(name string, defaultValue int64, description string) *ArrayBy
 // Flag values may be quoted. For instance, the following arg creates an array of ("a", "b,c") items:
 //
 //	-foo='a,"b,c"'
+//
+// Multiple flags can be combined with comma-separated values.
+// In this case, values are joined into an array of ("a", "b", "c", "d", "e") in the order they are defined:
+//
+//	-foo=a,b -foo=c -foo=d,e
+//
+// While this is possible, we do not recommend mixing comma-separated and repeated flags,
+// as it may lead to ambiguity or unexpected parsing behavior.
 type ArrayString []string
 
 // String implements flag.Value interface


Reviews:
user: login: hagen1778
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-15T08:48:19Z
user: login: makasim
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-15T13:05:29Z
user: login: makasim
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-16T16:00:34Z
user: login: hagen1778
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-18T12:13:15Z
user: login: makasim
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-18T12:44:50Z
user: login: hagen1778
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-18T12:53:12Z
user: login: makasim
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-18T13:00:00Z
user: login: makasim
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-24T15:16:42Z
user: login: makasim
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-24T15:16:48Z
user: login: makasim
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-24T15:16:58Z
user: login: f41gh7
body: LGTM
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-24T15:37:34Z

Review comments:
diff_hunk: @@ -1124,27 +1124,67 @@ See also [aggregation outputs](#aggregation-outputs).
 
 ## Dropping unneeded labels
 
-If you need dropping some labels from input samples before [input relabeling](#relabeling), [de-duplication](#deduplication)
-and [stream aggregation](#aggregation-outputs), then the following options exist:
-
-- To specify comma-separated list of label names to drop in `-streamAggr.dropInputLabels` command-line flag
-  or via `-remoteWrite.streamAggr.dropInputLabels` individually per each `-remoteWrite.url`.
-  For example, `-streamAggr.dropInputLabels=replica,az` instructs to drop `replica` and `az` labels from input samples
-  before applying de-duplication and stream aggregation.
-
-- To specify `drop_input_labels` list with the labels to drop in [stream aggregation config](#stream-aggregation-config).
-  For example, the following config drops `replica` label from input samples with the name `process_resident_memory_bytes`
-  before calculating the average over one minute:
-
-  ```yaml
-  - match: process_resident_memory_bytes
-    interval: 1m
-    drop_input_labels: [replica]
-    outputs: [avg]
-    keep_metric_names: true
-  ```
+To optimize performance and reduce [churn rate](https://docs.victoriametrics.com/guides/understand-your-setup-size/#churn-rate), 
+certain labels can be dropped from incoming samples before [input relabeling](#relabeling), 
+[de-duplication](#deduplication) and [stream aggregation](#aggregation-outputs) are applied. 
+There are several ways to drop labels, which can be used independently or together.
+
+**Global Label Drop** is configured using the `-streamAggr.dropInputLabels` flag. 
+This approach drops specified labels for all remote write targets globally. 
+The flag can be set multiple times, and each instance may contain a comma-separated list of labels. 
+All specified labels across flags are merged and dropped from all remote write destinations.
+
+The following example drops the labels `replica`, `az`, `instance`, and `job` 
+before de-duplication and aggregation for both `foo` and `bar` remote write targets:
+
+```bash
+vmagent \
+  -remoteWrite.url="http://foo/api/v1/write" \
+  -remoteWrite.url="http://bar/api/v1/write" \
+  -streamAggr.config="aggr.yaml" \
+  -streamAggr.dropInputLabels="replica,az" \
+  -streamAggr.dropInputLabels="instance,job"
path: docs/victoriametrics/stream-aggregation.md
user: login: hagen1778
body: I find this example super confusing. I know it works like you described. But my initial understanding of it was that ` -streamAggr.dropInputLabels` is indiviudally applied to corresponding `-remoteWrite.url`. Which made me doubting why ` -streamAggr.dropInputLabels` is called global :)
I would suggest keeping only one usage of this flag to emphasize its "global" appliance:
```
vmagent \
  -remoteWrite.url="http://foo/api/v1/write" \
  -remoteWrite.url="http://bar/api/v1/write" \
  -streamAggr.config="aggr.yaml" \
  -streamAggr.dropInputLabels="replica,az,instance,job"
```

created_at: 2025-04-15T08:42:35Z
author_association: COLLABORATOR
diff_hunk: @@ -1124,27 +1124,67 @@ See also [aggregation outputs](#aggregation-outputs).
 
 ## Dropping unneeded labels
 
-If you need dropping some labels from input samples before [input relabeling](#relabeling), [de-duplication](#deduplication)
-and [stream aggregation](#aggregation-outputs), then the following options exist:
-
-- To specify comma-separated list of label names to drop in `-streamAggr.dropInputLabels` command-line flag
-  or via `-remoteWrite.streamAggr.dropInputLabels` individually per each `-remoteWrite.url`.
-  For example, `-streamAggr.dropInputLabels=replica,az` instructs to drop `replica` and `az` labels from input samples
-  before applying de-duplication and stream aggregation.
-
-- To specify `drop_input_labels` list with the labels to drop in [stream aggregation config](#stream-aggregation-config).
-  For example, the following config drops `replica` label from input samples with the name `process_resident_memory_bytes`
-  before calculating the average over one minute:
-
-  ```yaml
-  - match: process_resident_memory_bytes
-    interval: 1m
-    drop_input_labels: [replica]
-    outputs: [avg]
-    keep_metric_names: true
-  ```
+To optimize performance and reduce [churn rate](https://docs.victoriametrics.com/guides/understand-your-setup-size/#churn-rate), 
+certain labels can be dropped from incoming samples before [input relabeling](#relabeling), 
+[de-duplication](#deduplication) and [stream aggregation](#aggregation-outputs) are applied. 
+There are several ways to drop labels, which can be used independently or together.
+
+**Global Label Drop** is configured using the `-streamAggr.dropInputLabels` flag. 
+This approach drops specified labels for all remote write targets globally. 
+The flag can be set multiple times, and each instance may contain a comma-separated list of labels. 
+All specified labels across flags are merged and dropped from all remote write destinations.
+
+The following example drops the labels `replica`, `az`, `instance`, and `job` 
+before de-duplication and aggregation for both `foo` and `bar` remote write targets:
+
+```bash
+vmagent \
+  -remoteWrite.url="http://foo/api/v1/write" \
+  -remoteWrite.url="http://bar/api/v1/write" \
+  -streamAggr.config="aggr.yaml" \
+  -streamAggr.dropInputLabels="replica,az" \
+  -streamAggr.dropInputLabels="instance,job"
+```
+
+**Per-Remote Write Label Drop** lets you define different label sets for each -`remoteWrite.url` using the `-remoteWrite.streamAggr.dropInputLabels` flag. 
+The flag can be set multiple times. 
+Each entry should list labels separated by commas, and multiple labels per remote write target must be separated with "^^". 
path: docs/victoriametrics/stream-aggregation.md
user: login: hagen1778
body: > Each entry should list labels separated by commas

this is not true, afaik. The reason why we have `ˆˆ` as separator here is that we reuse the same `flagutil.NewArrayString` struct for a different purpose. So we should emphasize that `,` should not be used anymore here. This should reduce confusion. 

Maybe, flag description should be updated too by moving delimiters to the beginning of flag description. Just in case users read flag description and not this doc
created_at: 2025-04-15T08:46:56Z
author_association: COLLABORATOR
diff_hunk: @@ -1124,27 +1124,67 @@ See also [aggregation outputs](#aggregation-outputs).
 
 ## Dropping unneeded labels
 
-If you need dropping some labels from input samples before [input relabeling](#relabeling), [de-duplication](#deduplication)
-and [stream aggregation](#aggregation-outputs), then the following options exist:
-
-- To specify comma-separated list of label names to drop in `-streamAggr.dropInputLabels` command-line flag
-  or via `-remoteWrite.streamAggr.dropInputLabels` individually per each `-remoteWrite.url`.
-  For example, `-streamAggr.dropInputLabels=replica,az` instructs to drop `replica` and `az` labels from input samples
-  before applying de-duplication and stream aggregation.
-
-- To specify `drop_input_labels` list with the labels to drop in [stream aggregation config](#stream-aggregation-config).
-  For example, the following config drops `replica` label from input samples with the name `process_resident_memory_bytes`
-  before calculating the average over one minute:
-
-  ```yaml
-  - match: process_resident_memory_bytes
-    interval: 1m
-    drop_input_labels: [replica]
-    outputs: [avg]
-    keep_metric_names: true
-  ```
+To optimize performance and reduce [churn rate](https://docs.victoriametrics.com/guides/understand-your-setup-size/#churn-rate), 
+certain labels can be dropped from incoming samples before [input relabeling](#relabeling), 
+[de-duplication](#deduplication) and [stream aggregation](#aggregation-outputs) are applied. 
+There are several ways to drop labels, which can be used independently or together.
+
+**Global Label Drop** is configured using the `-streamAggr.dropInputLabels` flag. 
+This approach drops specified labels for all remote write targets globally. 
+The flag can be set multiple times, and each instance may contain a comma-separated list of labels. 
+All specified labels across flags are merged and dropped from all remote write destinations.
+
+The following example drops the labels `replica`, `az`, `instance`, and `job` 
+before de-duplication and aggregation for both `foo` and `bar` remote write targets:
+
+```bash
+vmagent \
+  -remoteWrite.url="http://foo/api/v1/write" \
+  -remoteWrite.url="http://bar/api/v1/write" \
+  -streamAggr.config="aggr.yaml" \
+  -streamAggr.dropInputLabels="replica,az" \
+  -streamAggr.dropInputLabels="instance,job"
+```
+
+**Per-Remote Write Label Drop** lets you define different label sets for each -`remoteWrite.url` using the `-remoteWrite.streamAggr.dropInputLabels` flag. 
+The flag can be set multiple times. 
+Each entry should list labels separated by commas, and multiple labels per remote write target must be separated with "^^". 
+Labels are matched to remote write targets in the order they appear. 
+If a remote write has no matching entry, no labels are dropped for it.
+
+In the example below, `replica` and `az` are dropped for the `foo` remote write targets, 
+`instance` is dropped for the `bar` target, and `job` is ignored since there’s no third target.
+
+```bash
+vmagent \
+  -remoteWrite.url="http://foo/api/v1/write" \
+  -remoteWrite.url="http://bar/api/v1/write" \
+  -remoteWrite.streamAggr.config="aggr.yaml" \
+  -remoteWrite.streamAggr.dropInputLabels="replica^^az" \
+  -remoteWrite.streamAggr.dropInputLabels="instance,job"
path: docs/victoriametrics/stream-aggregation.md
user: login: hagen1778
body: Oh, this is so confusing... :) While the description is correct, we should never show examples like this. We should prevent users doing this at all costs.
created_at: 2025-04-15T08:48:15Z
author_association: COLLABORATOR
diff_hunk: @@ -1124,27 +1124,67 @@ See also [aggregation outputs](#aggregation-outputs).
 
 ## Dropping unneeded labels
 
-If you need dropping some labels from input samples before [input relabeling](#relabeling), [de-duplication](#deduplication)
-and [stream aggregation](#aggregation-outputs), then the following options exist:
-
-- To specify comma-separated list of label names to drop in `-streamAggr.dropInputLabels` command-line flag
-  or via `-remoteWrite.streamAggr.dropInputLabels` individually per each `-remoteWrite.url`.
-  For example, `-streamAggr.dropInputLabels=replica,az` instructs to drop `replica` and `az` labels from input samples
-  before applying de-duplication and stream aggregation.
-
-- To specify `drop_input_labels` list with the labels to drop in [stream aggregation config](#stream-aggregation-config).
-  For example, the following config drops `replica` label from input samples with the name `process_resident_memory_bytes`
-  before calculating the average over one minute:
-
-  ```yaml
-  - match: process_resident_memory_bytes
-    interval: 1m
-    drop_input_labels: [replica]
-    outputs: [avg]
-    keep_metric_names: true
-  ```
+To optimize performance and reduce [churn rate](https://docs.victoriametrics.com/guides/understand-your-setup-size/#churn-rate), 
+certain labels can be dropped from incoming samples before [input relabeling](#relabeling), 
+[de-duplication](#deduplication) and [stream aggregation](#aggregation-outputs) are applied. 
path: docs/victoriametrics/stream-aggregation.md
user: login: makasim
body: @hagen1778 Am I correct assuming that all three ways are applied before relabeling, deduplications and aggregation ?
created_at: 2025-04-15T13:05:29Z
author_association: MEMBER
diff_hunk: @@ -1124,27 +1124,64 @@ See also [aggregation outputs](#aggregation-outputs).
 
 ## Dropping unneeded labels
 
-If you need dropping some labels from input samples before [input relabeling](#relabeling), [de-duplication](#deduplication)
-and [stream aggregation](#aggregation-outputs), then the following options exist:
-
-- To specify comma-separated list of label names to drop in `-streamAggr.dropInputLabels` command-line flag
-  or via `-remoteWrite.streamAggr.dropInputLabels` individually per each `-remoteWrite.url`.
-  For example, `-streamAggr.dropInputLabels=replica,az` instructs to drop `replica` and `az` labels from input samples
-  before applying de-duplication and stream aggregation.
-
-- To specify `drop_input_labels` list with the labels to drop in [stream aggregation config](#stream-aggregation-config).
-  For example, the following config drops `replica` label from input samples with the name `process_resident_memory_bytes`
-  before calculating the average over one minute:
-
-  ```yaml
-  - match: process_resident_memory_bytes
-    interval: 1m
-    drop_input_labels: [replica]
-    outputs: [avg]
-    keep_metric_names: true
-  ```
+To optimize performance and reduce the [churn rate](https://docs.victoriametrics.com/guides/understand-your-setup-size/#churn-rate), 
+certain labels can be dropped from incoming samples before [input relabeling](#relabeling),
+[de-duplication](#deduplication) and [stream aggregation](#aggregation-outputs) are applied.
path: docs/victoriametrics/stream-aggregation.md
user: login: makasim
body: @hagen1778  I assumed that `can be dropped from incoming samples before input relabeling,
de-duplication and stream aggregation are applied` is correct for all three ways of dropping labels. Am I correct ? 
created_at: 2025-04-16T16:00:34Z
author_association: MEMBER
diff_hunk: @@ -1124,27 +1124,63 @@ See also [aggregation outputs](#aggregation-outputs).
 
 ## Dropping unneeded labels
 
-If you need dropping some labels from input samples before [input relabeling](#relabeling), [de-duplication](#deduplication)
-and [stream aggregation](#aggregation-outputs), then the following options exist:
-
-- To specify comma-separated list of label names to drop in `-streamAggr.dropInputLabels` command-line flag
-  or via `-remoteWrite.streamAggr.dropInputLabels` individually per each `-remoteWrite.url`.
-  For example, `-streamAggr.dropInputLabels=replica,az` instructs to drop `replica` and `az` labels from input samples
-  before applying de-duplication and stream aggregation.
-
-- To specify `drop_input_labels` list with the labels to drop in [stream aggregation config](#stream-aggregation-config).
-  For example, the following config drops `replica` label from input samples with the name `process_resident_memory_bytes`
-  before calculating the average over one minute:
-
-  ```yaml
-  - match: process_resident_memory_bytes
-    interval: 1m
-    drop_input_labels: [replica]
-    outputs: [avg]
-    keep_metric_names: true
-  ```
+To optimize performance and reduce the [churn rate](https://docs.victoriametrics.com/guides/understand-your-setup-size/#churn-rate), 
+certain labels can be dropped from incoming samples before [input relabeling](#relabeling),
+[de-duplication](#deduplication) and [stream aggregation](#aggregation-outputs) are applied.
+There are several ways to drop labels, which can be used independently or together.
+
+**Global label dropping** is configured using the `-streamAggr.dropInputLabels` flag.
path: docs/victoriametrics/stream-aggregation.md
user: login: hagen1778
body: It misses a critical point: labels are dropped "before applying de-duplication and stream aggregation". Can we keep this statement?
created_at: 2025-04-18T12:09:43Z
author_association: COLLABORATOR
diff_hunk: @@ -1124,27 +1124,63 @@ See also [aggregation outputs](#aggregation-outputs).
 
 ## Dropping unneeded labels
 
-If you need dropping some labels from input samples before [input relabeling](#relabeling), [de-duplication](#deduplication)
-and [stream aggregation](#aggregation-outputs), then the following options exist:
-
-- To specify comma-separated list of label names to drop in `-streamAggr.dropInputLabels` command-line flag
-  or via `-remoteWrite.streamAggr.dropInputLabels` individually per each `-remoteWrite.url`.
-  For example, `-streamAggr.dropInputLabels=replica,az` instructs to drop `replica` and `az` labels from input samples
-  before applying de-duplication and stream aggregation.
-
-- To specify `drop_input_labels` list with the labels to drop in [stream aggregation config](#stream-aggregation-config).
-  For example, the following config drops `replica` label from input samples with the name `process_resident_memory_bytes`
-  before calculating the average over one minute:
-
-  ```yaml
-  - match: process_resident_memory_bytes
-    interval: 1m
-    drop_input_labels: [replica]
-    outputs: [avg]
-    keep_metric_names: true
-  ```
+To optimize performance and reduce the [churn rate](https://docs.victoriametrics.com/guides/understand-your-setup-size/#churn-rate), 
+certain labels can be dropped from incoming samples before [input relabeling](#relabeling),
+[de-duplication](#deduplication) and [stream aggregation](#aggregation-outputs) are applied.
+There are several ways to drop labels, which can be used independently or together.
+
+**Global label dropping** is configured using the `-streamAggr.dropInputLabels` flag.
+It works in conjunction with the `-streamAggr.config` flag and applies to all matching sections in it.
+The flag could be applied on [vmagent](https://docs.victoriametrics.com/vmagent/), vminsert and [vmsingle](https://docs.victoriametrics.com/single-server-victoriametrics/). 
+The following example drops the `replica` and `az` labels for both `foo` and `bar` remote write targets:
+
+```bash
+/path/to/vmagent \
+  -remoteWrite.url="http://foo/api/v1/write" \
+  -remoteWrite.url="http://bar/api/v1/write" \
+  -streamAggr.config="aggr.yaml" \
+  -streamAggr.dropInputLabels="replica,az"
+```
+
+**Per remote write label drop** is configured using the `-remoteWrite.streamAggr.dropInputLabels` flag.  
+This flag is available for [vmagent](https://docs.victoriametrics.com/vmagent/) only.
+
+It works similarly to `-streamAggr.dropInputLabels` when defined once.  
+In the example below, `replica` and `az` are dropped for both `foo` and `bar` remote write targets:
path: docs/victoriametrics/stream-aggregation.md
user: login: hagen1778
body: I think this example is excessive, as it can be easily covered with `-streamAggr.dropInputLabels`.
Let's drop this example to reduce confusion.
created_at: 2025-04-18T12:11:45Z
author_association: COLLABORATOR
diff_hunk: @@ -1124,27 +1124,63 @@ See also [aggregation outputs](#aggregation-outputs).
 
 ## Dropping unneeded labels
 
-If you need dropping some labels from input samples before [input relabeling](#relabeling), [de-duplication](#deduplication)
-and [stream aggregation](#aggregation-outputs), then the following options exist:
-
-- To specify comma-separated list of label names to drop in `-streamAggr.dropInputLabels` command-line flag
-  or via `-remoteWrite.streamAggr.dropInputLabels` individually per each `-remoteWrite.url`.
-  For example, `-streamAggr.dropInputLabels=replica,az` instructs to drop `replica` and `az` labels from input samples
-  before applying de-duplication and stream aggregation.
-
-- To specify `drop_input_labels` list with the labels to drop in [stream aggregation config](#stream-aggregation-config).
-  For example, the following config drops `replica` label from input samples with the name `process_resident_memory_bytes`
-  before calculating the average over one minute:
-
-  ```yaml
-  - match: process_resident_memory_bytes
-    interval: 1m
-    drop_input_labels: [replica]
-    outputs: [avg]
-    keep_metric_names: true
-  ```
+To optimize performance and reduce the [churn rate](https://docs.victoriametrics.com/guides/understand-your-setup-size/#churn-rate), 
+certain labels can be dropped from incoming samples before [input relabeling](#relabeling),
+[de-duplication](#deduplication) and [stream aggregation](#aggregation-outputs) are applied.
+There are several ways to drop labels, which can be used independently or together.
+
+**Global label dropping** is configured using the `-streamAggr.dropInputLabels` flag.
+It works in conjunction with the `-streamAggr.config` flag and applies to all matching sections in it.
+The flag could be applied on [vmagent](https://docs.victoriametrics.com/vmagent/), vminsert and [vmsingle](https://docs.victoriametrics.com/single-server-victoriametrics/). 
+The following example drops the `replica` and `az` labels for both `foo` and `bar` remote write targets:
+
+```bash
+/path/to/vmagent \
+  -remoteWrite.url="http://foo/api/v1/write" \
+  -remoteWrite.url="http://bar/api/v1/write" \
+  -streamAggr.config="aggr.yaml" \
+  -streamAggr.dropInputLabels="replica,az"
+```
+
+**Per remote write label drop** is configured using the `-remoteWrite.streamAggr.dropInputLabels` flag.  
+This flag is available for [vmagent](https://docs.victoriametrics.com/vmagent/) only.
+
+It works similarly to `-streamAggr.dropInputLabels` when defined once.  
+In the example below, `replica` and `az` are dropped for both `foo` and `bar` remote write targets:
+
+```bash
+/path/to/vmagent \
+  -remoteWrite.url="http://foo/api/v1/write" \
+  -remoteWrite.url="http://bar/api/v1/write" \
+  -remoteWrite.streamAggr.config="aggr.yaml" \
+  -remoteWrite.streamAggr.dropInputLabels="replica^^az"
+```
 
-Typical use case is to drop `replica` label from samples, which are received from high availability replicas.
+When the flag is specified multiple times, it applies to the corresponding remote write target.
path: docs/victoriametrics/stream-aggregation.md
user: login: hagen1778
body: I would emphasize on the fact that it should be applied multiple times, once per each rw flag. This rule is simple to understand and it naturally leads to the `ˆˆ` thing
created_at: 2025-04-18T12:13:00Z
author_association: COLLABORATOR
diff_hunk: @@ -1124,27 +1124,63 @@ See also [aggregation outputs](#aggregation-outputs).
 
 ## Dropping unneeded labels
 
-If you need dropping some labels from input samples before [input relabeling](#relabeling), [de-duplication](#deduplication)
-and [stream aggregation](#aggregation-outputs), then the following options exist:
-
-- To specify comma-separated list of label names to drop in `-streamAggr.dropInputLabels` command-line flag
-  or via `-remoteWrite.streamAggr.dropInputLabels` individually per each `-remoteWrite.url`.
-  For example, `-streamAggr.dropInputLabels=replica,az` instructs to drop `replica` and `az` labels from input samples
-  before applying de-duplication and stream aggregation.
-
-- To specify `drop_input_labels` list with the labels to drop in [stream aggregation config](#stream-aggregation-config).
-  For example, the following config drops `replica` label from input samples with the name `process_resident_memory_bytes`
-  before calculating the average over one minute:
-
-  ```yaml
-  - match: process_resident_memory_bytes
-    interval: 1m
-    drop_input_labels: [replica]
-    outputs: [avg]
-    keep_metric_names: true
-  ```
+To optimize performance and reduce the [churn rate](https://docs.victoriametrics.com/guides/understand-your-setup-size/#churn-rate), 
+certain labels can be dropped from incoming samples before [input relabeling](#relabeling),
+[de-duplication](#deduplication) and [stream aggregation](#aggregation-outputs) are applied.
+There are several ways to drop labels, which can be used independently or together.
+
+**Global label dropping** is configured using the `-streamAggr.dropInputLabels` flag.
+It works in conjunction with the `-streamAggr.config` flag and applies to all matching sections in it.
+The flag could be applied on [vmagent](https://docs.victoriametrics.com/vmagent/), vminsert and [vmsingle](https://docs.victoriametrics.com/single-server-victoriametrics/). 
+The following example drops the `replica` and `az` labels for both `foo` and `bar` remote write targets:
+
+```bash
+/path/to/vmagent \
+  -remoteWrite.url="http://foo/api/v1/write" \
+  -remoteWrite.url="http://bar/api/v1/write" \
+  -streamAggr.config="aggr.yaml" \
+  -streamAggr.dropInputLabels="replica,az"
+```
+
+**Per remote write label drop** is configured using the `-remoteWrite.streamAggr.dropInputLabels` flag.  
+This flag is available for [vmagent](https://docs.victoriametrics.com/vmagent/) only.
+
+It works similarly to `-streamAggr.dropInputLabels` when defined once.  
+In the example below, `replica` and `az` are dropped for both `foo` and `bar` remote write targets:
path: docs/victoriametrics/stream-aggregation.md
user: login: makasim
body: Here, I wanted to clarify the difference in behavior when one flag and several are defined. They differ a lot!. 

When only one flag is specified, it is applied to **all remote write targets** — regardless of how many are configured.
However, as soon as you define two or more flags, the behavior changes: flags are **matched positionally**, meaning the first flag applies to the first remote write target, the second flag to the second, and so on.
created_at: 2025-04-18T12:44:50Z
author_association: MEMBER
diff_hunk: @@ -1124,27 +1124,63 @@ See also [aggregation outputs](#aggregation-outputs).
 
 ## Dropping unneeded labels
 
-If you need dropping some labels from input samples before [input relabeling](#relabeling), [de-duplication](#deduplication)
-and [stream aggregation](#aggregation-outputs), then the following options exist:
-
-- To specify comma-separated list of label names to drop in `-streamAggr.dropInputLabels` command-line flag
-  or via `-remoteWrite.streamAggr.dropInputLabels` individually per each `-remoteWrite.url`.
-  For example, `-streamAggr.dropInputLabels=replica,az` instructs to drop `replica` and `az` labels from input samples
-  before applying de-duplication and stream aggregation.
-
-- To specify `drop_input_labels` list with the labels to drop in [stream aggregation config](#stream-aggregation-config).
-  For example, the following config drops `replica` label from input samples with the name `process_resident_memory_bytes`
-  before calculating the average over one minute:
-
-  ```yaml
-  - match: process_resident_memory_bytes
-    interval: 1m
-    drop_input_labels: [replica]
-    outputs: [avg]
-    keep_metric_names: true
-  ```
+To optimize performance and reduce the [churn rate](https://docs.victoriametrics.com/guides/understand-your-setup-size/#churn-rate), 
+certain labels can be dropped from incoming samples before [input relabeling](#relabeling),
+[de-duplication](#deduplication) and [stream aggregation](#aggregation-outputs) are applied.
+There are several ways to drop labels, which can be used independently or together.
+
+**Global label dropping** is configured using the `-streamAggr.dropInputLabels` flag.
+It works in conjunction with the `-streamAggr.config` flag and applies to all matching sections in it.
+The flag could be applied on [vmagent](https://docs.victoriametrics.com/vmagent/), vminsert and [vmsingle](https://docs.victoriametrics.com/single-server-victoriametrics/). 
+The following example drops the `replica` and `az` labels for both `foo` and `bar` remote write targets:
+
+```bash
+/path/to/vmagent \
+  -remoteWrite.url="http://foo/api/v1/write" \
+  -remoteWrite.url="http://bar/api/v1/write" \
+  -streamAggr.config="aggr.yaml" \
+  -streamAggr.dropInputLabels="replica,az"
+```
+
+**Per remote write label drop** is configured using the `-remoteWrite.streamAggr.dropInputLabels` flag.  
+This flag is available for [vmagent](https://docs.victoriametrics.com/vmagent/) only.
+
+It works similarly to `-streamAggr.dropInputLabels` when defined once.  
+In the example below, `replica` and `az` are dropped for both `foo` and `bar` remote write targets:
path: docs/victoriametrics/stream-aggregation.md
user: login: hagen1778
body: I see. I understand the intention, but in my personal opinion **showing** only recommended examples and **mentioning** possible examples is better than **showing** recommended and possible examples together.

I am afraid of a user in rush not reading carefully what the text says and just copying the first code snippet they find. With showing only recommended code snippets I hope to reduce chances for people even trying to use 1 flag for 2 remote writes. Yes, we can explain later how the flag works in all cases.

But that's only my opinion. If for your fresh eye this version of doc seems more appropriate and less confusing - let's proceed with how it is.
created_at: 2025-04-18T12:53:11Z
author_association: COLLABORATOR
diff_hunk: @@ -1124,27 +1124,63 @@ See also [aggregation outputs](#aggregation-outputs).
 
 ## Dropping unneeded labels
 
-If you need dropping some labels from input samples before [input relabeling](#relabeling), [de-duplication](#deduplication)
-and [stream aggregation](#aggregation-outputs), then the following options exist:
-
-- To specify comma-separated list of label names to drop in `-streamAggr.dropInputLabels` command-line flag
-  or via `-remoteWrite.streamAggr.dropInputLabels` individually per each `-remoteWrite.url`.
-  For example, `-streamAggr.dropInputLabels=replica,az` instructs to drop `replica` and `az` labels from input samples
-  before applying de-duplication and stream aggregation.
-
-- To specify `drop_input_labels` list with the labels to drop in [stream aggregation config](#stream-aggregation-config).
-  For example, the following config drops `replica` label from input samples with the name `process_resident_memory_bytes`
-  before calculating the average over one minute:
-
-  ```yaml
-  - match: process_resident_memory_bytes
-    interval: 1m
-    drop_input_labels: [replica]
-    outputs: [avg]
-    keep_metric_names: true
-  ```
+To optimize performance and reduce the [churn rate](https://docs.victoriametrics.com/guides/understand-your-setup-size/#churn-rate), 
+certain labels can be dropped from incoming samples before [input relabeling](#relabeling),
+[de-duplication](#deduplication) and [stream aggregation](#aggregation-outputs) are applied.
+There are several ways to drop labels, which can be used independently or together.
+
+**Global label dropping** is configured using the `-streamAggr.dropInputLabels` flag.
+It works in conjunction with the `-streamAggr.config` flag and applies to all matching sections in it.
+The flag could be applied on [vmagent](https://docs.victoriametrics.com/vmagent/), vminsert and [vmsingle](https://docs.victoriametrics.com/single-server-victoriametrics/). 
+The following example drops the `replica` and `az` labels for both `foo` and `bar` remote write targets:
+
+```bash
+/path/to/vmagent \
+  -remoteWrite.url="http://foo/api/v1/write" \
+  -remoteWrite.url="http://bar/api/v1/write" \
+  -streamAggr.config="aggr.yaml" \
+  -streamAggr.dropInputLabels="replica,az"
+```
+
+**Per remote write label drop** is configured using the `-remoteWrite.streamAggr.dropInputLabels` flag.  
+This flag is available for [vmagent](https://docs.victoriametrics.com/vmagent/) only.
+
+It works similarly to `-streamAggr.dropInputLabels` when defined once.  
+In the example below, `replica` and `az` are dropped for both `foo` and `bar` remote write targets:
path: docs/victoriametrics/stream-aggregation.md
user: login: makasim
body: You're right—it's better for users who take their time to read carefully, but for those in a rush, it could potentially be misleading. I'll think about ways to improve this. 

Thank you for the feedback.
created_at: 2025-04-18T13:00:00Z
author_association: MEMBER
diff_hunk: @@ -1124,27 +1124,63 @@ See also [aggregation outputs](#aggregation-outputs).
 
 ## Dropping unneeded labels
 
-If you need dropping some labels from input samples before [input relabeling](#relabeling), [de-duplication](#deduplication)
-and [stream aggregation](#aggregation-outputs), then the following options exist:
-
-- To specify comma-separated list of label names to drop in `-streamAggr.dropInputLabels` command-line flag
-  or via `-remoteWrite.streamAggr.dropInputLabels` individually per each `-remoteWrite.url`.
-  For example, `-streamAggr.dropInputLabels=replica,az` instructs to drop `replica` and `az` labels from input samples
-  before applying de-duplication and stream aggregation.
-
-- To specify `drop_input_labels` list with the labels to drop in [stream aggregation config](#stream-aggregation-config).
-  For example, the following config drops `replica` label from input samples with the name `process_resident_memory_bytes`
-  before calculating the average over one minute:
-
-  ```yaml
-  - match: process_resident_memory_bytes
-    interval: 1m
-    drop_input_labels: [replica]
-    outputs: [avg]
-    keep_metric_names: true
-  ```
+To optimize performance and reduce the [churn rate](https://docs.victoriametrics.com/guides/understand-your-setup-size/#churn-rate), 
+certain labels can be dropped from incoming samples before [input relabeling](#relabeling),
+[de-duplication](#deduplication) and [stream aggregation](#aggregation-outputs) are applied.
+There are several ways to drop labels, which can be used independently or together.
+
+**Global label dropping** is configured using the `-streamAggr.dropInputLabels` flag.
path: docs/victoriametrics/stream-aggregation.md
user: login: makasim
body: fixed
created_at: 2025-04-24T15:16:42Z
author_association: MEMBER
diff_hunk: @@ -1124,27 +1124,63 @@ See also [aggregation outputs](#aggregation-outputs).
 
 ## Dropping unneeded labels
 
-If you need dropping some labels from input samples before [input relabeling](#relabeling), [de-duplication](#deduplication)
-and [stream aggregation](#aggregation-outputs), then the following options exist:
-
-- To specify comma-separated list of label names to drop in `-streamAggr.dropInputLabels` command-line flag
-  or via `-remoteWrite.streamAggr.dropInputLabels` individually per each `-remoteWrite.url`.
-  For example, `-streamAggr.dropInputLabels=replica,az` instructs to drop `replica` and `az` labels from input samples
-  before applying de-duplication and stream aggregation.
-
-- To specify `drop_input_labels` list with the labels to drop in [stream aggregation config](#stream-aggregation-config).
-  For example, the following config drops `replica` label from input samples with the name `process_resident_memory_bytes`
-  before calculating the average over one minute:
-
-  ```yaml
-  - match: process_resident_memory_bytes
-    interval: 1m
-    drop_input_labels: [replica]
-    outputs: [avg]
-    keep_metric_names: true
-  ```
+To optimize performance and reduce the [churn rate](https://docs.victoriametrics.com/guides/understand-your-setup-size/#churn-rate), 
+certain labels can be dropped from incoming samples before [input relabeling](#relabeling),
+[de-duplication](#deduplication) and [stream aggregation](#aggregation-outputs) are applied.
+There are several ways to drop labels, which can be used independently or together.
+
+**Global label dropping** is configured using the `-streamAggr.dropInputLabels` flag.
+It works in conjunction with the `-streamAggr.config` flag and applies to all matching sections in it.
+The flag could be applied on [vmagent](https://docs.victoriametrics.com/vmagent/), vminsert and [vmsingle](https://docs.victoriametrics.com/single-server-victoriametrics/). 
+The following example drops the `replica` and `az` labels for both `foo` and `bar` remote write targets:
+
+```bash
+/path/to/vmagent \
+  -remoteWrite.url="http://foo/api/v1/write" \
+  -remoteWrite.url="http://bar/api/v1/write" \
+  -streamAggr.config="aggr.yaml" \
+  -streamAggr.dropInputLabels="replica,az"
+```
+
+**Per remote write label drop** is configured using the `-remoteWrite.streamAggr.dropInputLabels` flag.  
+This flag is available for [vmagent](https://docs.victoriametrics.com/vmagent/) only.
+
+It works similarly to `-streamAggr.dropInputLabels` when defined once.  
+In the example below, `replica` and `az` are dropped for both `foo` and `bar` remote write targets:
path: docs/victoriametrics/stream-aggregation.md
user: login: makasim
body: fixed
created_at: 2025-04-24T15:16:48Z
author_association: MEMBER
diff_hunk: @@ -1124,27 +1124,63 @@ See also [aggregation outputs](#aggregation-outputs).
 
 ## Dropping unneeded labels
 
-If you need dropping some labels from input samples before [input relabeling](#relabeling), [de-duplication](#deduplication)
-and [stream aggregation](#aggregation-outputs), then the following options exist:
-
-- To specify comma-separated list of label names to drop in `-streamAggr.dropInputLabels` command-line flag
-  or via `-remoteWrite.streamAggr.dropInputLabels` individually per each `-remoteWrite.url`.
-  For example, `-streamAggr.dropInputLabels=replica,az` instructs to drop `replica` and `az` labels from input samples
-  before applying de-duplication and stream aggregation.
-
-- To specify `drop_input_labels` list with the labels to drop in [stream aggregation config](#stream-aggregation-config).
-  For example, the following config drops `replica` label from input samples with the name `process_resident_memory_bytes`
-  before calculating the average over one minute:
-
-  ```yaml
-  - match: process_resident_memory_bytes
-    interval: 1m
-    drop_input_labels: [replica]
-    outputs: [avg]
-    keep_metric_names: true
-  ```
+To optimize performance and reduce the [churn rate](https://docs.victoriametrics.com/guides/understand-your-setup-size/#churn-rate), 
+certain labels can be dropped from incoming samples before [input relabeling](#relabeling),
+[de-duplication](#deduplication) and [stream aggregation](#aggregation-outputs) are applied.
+There are several ways to drop labels, which can be used independently or together.
+
+**Global label dropping** is configured using the `-streamAggr.dropInputLabels` flag.
+It works in conjunction with the `-streamAggr.config` flag and applies to all matching sections in it.
+The flag could be applied on [vmagent](https://docs.victoriametrics.com/vmagent/), vminsert and [vmsingle](https://docs.victoriametrics.com/single-server-victoriametrics/). 
+The following example drops the `replica` and `az` labels for both `foo` and `bar` remote write targets:
+
+```bash
+/path/to/vmagent \
+  -remoteWrite.url="http://foo/api/v1/write" \
+  -remoteWrite.url="http://bar/api/v1/write" \
+  -streamAggr.config="aggr.yaml" \
+  -streamAggr.dropInputLabels="replica,az"
+```
+
+**Per remote write label drop** is configured using the `-remoteWrite.streamAggr.dropInputLabels` flag.  
+This flag is available for [vmagent](https://docs.victoriametrics.com/vmagent/) only.
+
+It works similarly to `-streamAggr.dropInputLabels` when defined once.  
+In the example below, `replica` and `az` are dropped for both `foo` and `bar` remote write targets:
+
+```bash
+/path/to/vmagent \
+  -remoteWrite.url="http://foo/api/v1/write" \
+  -remoteWrite.url="http://bar/api/v1/write" \
+  -remoteWrite.streamAggr.config="aggr.yaml" \
+  -remoteWrite.streamAggr.dropInputLabels="replica^^az"
+```
 
-Typical use case is to drop `replica` label from samples, which are received from high availability replicas.
+When the flag is specified multiple times, it applies to the corresponding remote write target.
path: docs/victoriametrics/stream-aggregation.md
user: login: makasim
body: fixed
created_at: 2025-04-24T15:16:58Z
author_association: MEMBER

PR comments:
user: login: makasim
created_at: 2025-04-15T13:12:20Z
body: @hagen1778 please take a look 
author_association: MEMBER
user: login: makasim
created_at: 2025-04-16T16:06:57Z
body: @hagen1778 @f41gh7 please review
author_association: MEMBER
user: login: makasim
created_at: 2025-04-24T15:19:28Z
body: @hagen1778 @f41gh7 updated, please review
author_association: MEMBER
user: login: f41gh7
created_at: 2025-04-24T17:26:43Z
body: Thanks for contribution!
author_association: CONTRIBUTOR
user: login: makasim
created_at: 2025-04-25T06:21:48Z
body: @AndrewChubatiuk @f41gh7 I don't see any changes in [the document](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#dropping-unneeded-labels). Previously, the updates appeared after the merge—unless I'm mistaken. 

Could something be broken in the pipeline?
author_association: MEMBER
user: login: AndrewChubatiuk
created_at: 2025-04-25T06:24:51Z
body: there was an issue in deployment docs pipeline, it was temporary disabled. already fixed it approval [here](https://github.com/VictoriaMetrics/vmdocs/pull/136) is needed 
author_association: CONTRIBUTOR