Project: VictoriaMetrics (Observability and Analysis / Observability)
File: stream-aggregation.md

sort: 98 weight: 98 title: Streaming aggregation menu: docs: parent: 'victoriametrics' weight: 98 aliases: /stream-aggregation.html and can aggregate incoming in streaming mode by time and by labels before data is written to remote storage (or local storage for single-node VictoriaMetrics). The aggregation is applied to all the metrics received via any and/or scraped from after applying all the configured . _By default, stream aggregation ignores timestamps associated with the input . It expects that the ingested samples have timestamps close to the current time. See ._ Stream aggregation can be configured via the following command-line flags: `-streamAggr.config` at and at . `-remoteWrite.streamAggr.config` at only. This flag can be specified individually per each `-remoteWrite.url` and aggregation will happen independently for each of them. This allows writing different aggregates to different remote storage destinations. These flags must point to a file containing . The file may contain `%{ENVVAR}` placeholders which are substituted by the corresponding `ENVVAR` environment variable values. By default, the following data is written to the storage when stream aggregation is enabled: the aggregated samples; the raw input samples, which didn't match any `match` option in the provided . This behaviour can be changed via the following command-line flags: `-streamAggr.keepInput` at and . At `-remoteWrite.streamAggr.keepInput` flag can be specified individually per each `-remoteWrite.url`. If one of these flags is set, then all the input samples are written to the storage alongside the aggregated samples. `-streamAggr.dropInput` at and . At `-remoteWrite.streamAggr.dropInput` flag can be specified individually per each `-remoteWrite.url`. If one of these flags are set, then all the input samples are dropped, while only the aggregated samples are written to the storage. supports relabeling, deduplication and stream aggregation for all the received data, scraped or pushed. The processed data is then stored in local storage and can't be forwarded further. supports relabeling, deduplication and stream aggregation for all the received data, scraped or pushed. Then, the collected data will be forwarded to specified `-remoteWrite.url` destinations. The data processing order is the following: All the received data is according to specified `-remoteWrite.relabelConfig`; All the received data is according to specified `-streamAggr.dedupInterval`; All the received data is aggregated according to specified `-streamAggr.config`; The resulting data from p1 and p2 is then replicated to each `-remoteWrite.url`; Data sent to each `-remoteWrite.url` can be additionally relabeled according to the corresponding `-remoteWrite.urlRelabelConfig` (set individually per URL); Data sent to each `-remoteWrite.url` can be additionally deduplicated according to the corresponding `-remoteWrite.streamAggr.dedupInterval` (set individually per URL); Data sent to each `-remoteWrite.url` can be additionally aggregated according to the corresponding `-remoteWrite.streamAggr.config` (set individually per URL). Please note, it is not recommended to use `-streamAggr.config` and `-remoteWrite.streamAggr.config` together, unless you understand the complications. Typical scenarios for data routing with vmagent: Aggregate incoming data and replicate to N destinations. For this one should configure `-streamAggr.config` to aggregate the incoming data before replicating it to all the configured `-remoteWrite.url` destinations. Individually aggregate incoming data for each destination. For this on should configure `-remoteWrite.streamAggr.config` for each `-remoteWrite.url` destination. via `-remoteWrite.urlRelabelConfig` can be used for routing only selected metrics to each `-remoteWrite.url` destination. supports online of samples before sending them to the configured `-remoteWrite.url`. The de-duplication can be enabled via the following options: By specifying the desired de-duplication interval via `-streamAggr.dedupInterval` command-line flag for all received data or via

command-line flag for the particular `-remoteWrite.url` destination. For example, `./vmagent -remoteWrite.url=http://remote-storage/api/v1/write -remoteWrite.streamAggr.dedupInterval=30s` instructs `vmagent` to leave only the last sample per each seen per every 30 seconds. The de-deduplication is performed after applying and before performing the aggregation. If the `-remoteWrite.streamAggr.config` and / or `-streamAggr.config` is set, then the de-duplication is performed individually per each for the matching samples after applying . By specifying `dedup_interval` option individually per each in `-remoteWrite.streamAggr.config` or `-streamAggr.config` configs. supports two types of de-duplication: After storing the duplicate samples to local storage. See command-line option. Before storing the duplicate samples to local storage. This type of de-duplication can be enabled via the following options: By specifying the desired de-duplication interval via `-streamAggr.dedupInterval` command-line flag. For example, `./victoria-metrics -streamAggr.dedupInterval=30s` instructs VictoriaMetrics to leave only the last sample per each seen per every 30 seconds. The de-duplication is performed after applying `-relabelConfig` . If the `-streamAggr.config` is set, then the de-duplication is performed individually per each for the matching samples after applying . By specifying `dedup_interval` option individually per each at `-streamAggr.config`. It is possible to drop the given labels before applying the de-duplication. See . The online de-duplication uses the same logic as at VictoriaMetrics. By default, all the input samples are taken into account during stream aggregation. If samples with old timestamps outside the current must be ignored, then the following options can be used: To pass `-streamAggr.ignoreOldSamples` command-line flag to or to . At `-remoteWrite.streamAggr.ignoreOldSamples` flag can be specified individually per each `-remoteWrite.url`. This enables ignoring old samples for all the . To set `ignoreoldsamples: true` option at the particular . This enables ignoring old samples for that particular aggregation config. Stream aggregation may yield inaccurate results if it processes incomplete data. This issue can arise when data is received from clients that maintain a queue of unsent data, such as Prometheus or vmagent. If the queue isn't fully cleared within the aggregation `interval`, only a portion of the time series may be processed, leading to distorted calculations. To mitigate this, consider the following options: Set `-streamAggr.ignoreFirstIntervals=<intervalsCount>` command-line flag to or to to skip first `<intervalsCount>` from persisting to the storage. At `-remoteWrite.streamAggr.ignoreFirstIntervals=<intervalsCount>` flag can be specified individually per each `-remoteWrite.url`. It is expected that all incomplete or queued data will be processed during specified `<intervalsCount>` and all subsequent aggregation intervals will produce correct data. Set `ignorefirstintervals: <intervalsCount>` option individually per . This enables ignoring first `<intervalsCount>` aggregation intervals for that particular aggregation config. By default, the time for aggregated data flush is aligned by the `interval` option specified in . For example: if `interval: 1m` is set, then the aggregated data is flushed to the storage at the end of every minute if `interval: 1h` is set, then the aggregated data is flushed to the storage at the end of every hour If you do not need such an alignment, then set `noalignflushtointerval: true` option in the . In this case aggregated data flushes will be aligned to the `vmagent` start time or to time. The aggregated data on the first and the last interval is dropped during `vmagent` start, restart or , since the first and the last aggregation intervals are incomplete, so they usually contain incomplete confusing

If you need preserving the aggregated data on these intervals, then set `flushonshutdown: true` option in the . Stream aggregation can be used in the following cases: Stream aggregation can be used as drop-in replacement in the following cases: Sometimes may require non-trivial amounts of CPU, RAM, disk IO and network bandwidth at metrics storage side. For example, if `httprequestduration_seconds` histogram is generated by thousands of application instances, then the alerting query `histogramquantile(0.99, sum(increase(httprequestdurationseconds_bucket[5m])) without (instance)) > 0.5` can become slow, since it needs to scan too big number of unique with `httprequestdurationsecondsbucket` name. This alerting query can be accelerated by pre-calculating the `sum(increase(httprequestdurationsecondsbucket. But this recording rule may take too much time to execute too. In this case the slow recording rule can be substituted with the following : ```yaml match: 'httprequestdurationsecondsbucket' interval: 5m without: [instance] outputs: [total] ``` This stream aggregation generates `httprequestdurationsecondsbucket:5mwithoutinstance_total` output series according to . Then these series can be used in : ```metricsql histogramquantile(0.99, lastovertime(httprequestdurationsecondsbucket:5mwithoutinstancetotal[5m])) > 0.5 ``` This query is executed much faster than the original query, because it needs to scan much lower number of time series. See , which can be specified at `output` field. See also . Field `interval` is recommended to be set to a value at least several times higher than your metrics collect interval. If per- samples are ingested at high frequency, then this may result in high disk space usage, since too much data must be stored to disk. This also may result in slow queries, since too much data must be processed during queries. This can be fixed with the stream aggregation by increasing the interval between per-series samples stored in the database. For example, the following reduces the frequency of input samples to one sample per 5 minutes per each input time series (this operation is also known as downsampling): ```yaml match: '{name=~".+_total"}' interval: 5m outputs: [total] match: '{name!~".+_total"}' interval: 5m outputs: [countsamples, sumsamples, min, max] ``` The aggregated output metrics have the following names according to : ```text somemetrictotal:5m_total somemetric:5mcount_samples somemetric:5msum_samples somemetric:5mmin somemetric:5mmax ``` See , which can be specified at `output` field. See also and . Sometimes applications may generate too many . For example, the `httprequeststotal` metric may have `path` or `user` label with too big number of unique values. In this case the following stream aggregation can be used for reducing the number metrics stored in VictoriaMetrics: ```yaml match: 'httprequeststotal' interval: 30s without: [path, user] outputs: [total] ``` This config specifies labels, which must be removed from the aggregate output, in the `without` list. See for more details. The aggregated output metric has the following name according to : ```text httprequeststotal:30swithoutpathusertotal ``` See , which can be specified at `output` field. See also . If the monitored application generates event-based metrics, then it may be useful to count the number of such metrics at stream aggregation level. For example, if an advertising server generates `hits{some="labels"} 1` and `clicks{some="labels"} 1` metrics per each incoming hit and click, then the following can be used for counting these metrics per every 30 second interval: ```yaml match: '{name=~"hits|clicks"}' interval: 30s outputs: [count_samples] ``` This config generates the following output metrics for `hits` and `clicks` input metrics according to : ```text hits:30scountsamples count1 clicks:30scountsamples count2 ``` See , which can be specified at `output` field. See also

If the monitored application calculates some events and then sends the calculated number of events to VictoriaMetrics at irregular intervals or at too high frequency, then stream aggregation can be used for summing such events and writing the aggregate sums to the storage at regular intervals. For example, if an advertising server generates `hits{some="labels} N` and `clicks{some="labels"} M` metrics at irregular intervals, then the following can be used for summing these metrics per every minute: ```yaml match: '{name=~"hits|clicks"}' interval: 1m outputs: [sum_samples] ``` This config generates the following output metrics according to : ```text hits:1msumsamples sum1 clicks:1msumsamples sum2 ``` See , which can be specified at `output` field. See also . If the monitored application generates measurement metrics per each request, then it may be useful to calculate the pre-defined set of over these measurements. For example, if the monitored application generates `requestdurationseconds N` and `responsesizebytes M` metrics per each incoming request, then the following can be used for calculating 50th and 99th percentiles for these metrics every 30 seconds: ```yaml match: requestdurationseconds responsesizebytes interval: 30s outputs: ["quantiles(0.50, 0.99)"] ``` This config generates the following output metrics according to : ```text requestdurationseconds:30s_quantiles{quantile="0.50"} value1 requestdurationseconds:30s_quantiles{quantile="0.99"} value2 responsesizebytes:30s_quantiles{quantile="0.50"} value1 responsesizebytes:30s_quantiles{quantile="0.99"} value2 ``` See , which can be specified at `output` field. See also and . If the monitored application generates measurement metrics per each request, then it may be useful to calculate a over these metrics. For example, if the monitored application generates `requestdurationseconds N` and `responsesizebytes M` metrics per each incoming request, then the following can be used for calculating for these metrics every 60 seconds: ```yaml match: requestdurationseconds responsesizebytes interval: 60s outputs: [histogram_bucket] ``` This config generates the following output metrics according to . ```text requestdurationseconds:60shistogrambucket{vmrange="start1...end1"} count1 requestdurationseconds:60shistogrambucket{vmrange="start2...end2"} count2 ... requestdurationseconds:60shistogrambucket{vmrange="startN...endN"} countN responsesizebytes:60shistogrambucket{vmrange="start1...end1"} count1 responsesizebytes:60shistogrambucket{vmrange="start2...end2"} count2 ... responsesizebytes:60shistogrambucket{vmrange="startN...endN"} countN ``` The resulting histogram buckets can be queried with in the following ways: An estimated 50th and 99th of the request duration over the last hour: ```metricsql histogramquantiles("quantile", 0.50, 0.99, sum(increase(requestdurationseconds:60shistogram_bucket[1h])) by (vmrange)) ``` This query uses function. An estimated of the request duration over the last hour: ```metricsql histogramstddev(sum(increase(requestdurationseconds:60shistogram_bucket[1h])) by (vmrange)) ``` This query uses function. An estimated share of requests with the duration smaller than `0.5s` over the last hour: ```metricsql histogramshare(0.5, sum(increase(requestdurationseconds:60shistogram_bucket[1h])) by (vmrange)) ``` This query uses function. See , which can be specified at `output` field. See also and . is a set of metrics with different `vmrange` or `le` labels. As they're counters, the applicable aggregation output is : ```yaml match: 'httprequestdurationsecondsbucket' interval: 1m without: [instance] outputs: [total] ``` This config generates the following output metrics according to : ```text httprequestdurationsecondsbucket:1mwithoutinstance_total{le="0.1"} value1 httprequestdurationsecondsbucket:1mwithoutinstance_total{le="0.2"} value2 httprequestdurationsecondsbucket:1mwithoutinstance_total{le="0.4"} value3 httprequestdurationsecondsbucket:1mwithoutinstance_total{le="1"} value4 httprequestdurationsecondsbucket:1mwithoutinstance_total{le="3"} value5 httprequestdurationsecondsbucket:1mwithoutinstance_total{le="+Inf" value6 ``` The resulting metrics can be passed to function: ```metricsql histogramquantile(0.9, sum(rate(httprequestdurationsecondsbucket:1mwithoutinstancetotal[5m])) by(le)) ``` Please note, histograms can be aggregated if their `le` labels are configured identically. have no such requirement. See , which can be specified at `output` field. See also and . Output metric names for stream aggregation are constructed according to the following pattern: ```text <metricname>:<interval><output> ``` `<metric_name>` is the original metric name. `<interval>` is the interval specified in the . `<bylabels>` is ``-delimited sorted list of `by` labels specified in the

If the `by` list is missing in the config, then the `by<by_labels>` part isn't included in the output metric name. `<withoutlabels>` is an optional ``-delimited sorted list of `without` labels specified in the . If the `without` list is missing in the config, then the `without<without_labels>` part isn't included in the output metric name. `<output>` is the aggregate used for constructing the output metric. The aggregate name is taken from the `outputs` list at the corresponding . Both input and output metric names can be modified if needed via relabeling according to . It is possible to leave the original metric name after the aggregation by specifying `keepmetricnames: true` option at . The `keepmetricnames` option can be used if only a single output is set in . It is possible to apply to input and output metrics during stream aggregation via `inputrelabelconfigs` and `outputrelabelconfigs` options in . Relabeling rules inside `inputrelabelconfigs` are applied to samples matching the `match` filters before optional . Relabeling rules inside `outputrelabelconfigs` are applied to aggregated samples before sending them to the remote storage. For example, the following config removes the `:1msumsamples` suffix added : ```yaml interval: 1m outputs: [sum_samples] outputrelabelconfigs: sourcelabels: [name_] targetlabel: name_ regex: "(.+):.+" ``` Another option to remove the suffix, which is added by stream aggregation, is to add `keepmetricnames: true` to the config: ```yaml interval: 1m outputs: [sum_samples] keepmetricnames: true ``` See also . If you need dropping some labels from input samples before , and , then the following options exist: To specify comma-separated list of label names to drop in `-streamAggr.dropInputLabels` command-line flag or via `-remoteWrite.streamAggr.dropInputLabels` individually per each `-remoteWrite.url`. For example, `-streamAggr.dropInputLabels=replica,az` instructs to drop `replica` and `az` labels from input samples before applying de-duplication and stream aggregation. To specify `dropinputlabels` list with the labels to drop in . For example, the following config drops `replica` label from input samples with the name `processresidentmemory_bytes` before calculating the average over one minute: ```yaml match: processresidentmemory_bytes interval: 1m dropinputlabels: [replica] outputs: [avg] keepmetricnames: true ``` Typical use case is to drop `replica` label from samples, which are recevied from high availability replicas. The aggregations are calculated during the `interval` specified in the and then sent to the storage once per `interval`. The aggregated samples are named according to . If `by` and `without` lists are specified in the , then the is performed additionally to aggregation by `interval`. Below are aggregation functions that can be put in the `outputs` list at : `avg` returns the average over input . `avg` makes sense only for aggregating . The results of `avg` is equal to the following query: ```metricsql sum(sumovertime(somemetric[interval])) / sum(countovertime(somemetric[interval])) ``` For example, see below time series produced by config with aggregation interval `1m` and `by: ["instance"]` and the regular query: <img alt="avg aggregation" src="stream-aggregation-check-avg.webp"> See also , , and . `count_samples` counts the number of input over the given `interval`. The results of `count_samples` is equal to the following query: ```metricsql sum(countovertime(some_metric[interval])) ``` See also and . `count_series` counts the number of unique over the given `interval`. The results of `count_series` is equal to the following query: ```metricsql count(lastovertime(some_metric[interval])) ``` See also and . `increase` returns the increase of input over the given 'interval'. `increase` makes sense only for aggregating . The results of `increase` is equal to the following query: ```metricsql sum(increasepure(somecounter[interval])) ``` `increase` assumes that all the counters start from

For example, if the first seen sample for new is `10`, then `increase` assumes that the time series has been increased by `10`. If you need ignoring the first sample for new time series, then take a look at . For example, see below time series produced by config with aggregation interval `1m` and `by: ["instance"]` and the regular query: <img alt="increase aggregation" src="stream-aggregation-check-increase.webp"> Aggregating irregular and sporadic metrics (received from or ) can be controlled via option. See also and . `rate_sum` returns the sum of average per-second change of input over the given `interval`. `rate_sum` makes sense only for aggregating . The results of `rate_sum` are equal to the following query: ```metricsql sum(rate(some_counter[interval])) ``` See also and outputs. `rate_avg` returns the average of average per-second of input over the given `interval`. `rate_avg` makes sense only for aggregating . The results of `rate_avg` are equal to the following query: ```metricsql avg(rate(some_counter[interval])) ``` See also and outputs. `increase_prometheus` returns the increase of input over the given `interval`. `increase_prometheus` makes sense only for aggregating . The results of `increase_prometheus` is equal to the following query: ```metricsql sum(increaseprometheus(somecounter[interval])) ``` `increase_prometheus` skips the first seen sample value per each . If you need taking into account the first sample per time series, then take a look at . Aggregating irregular and sporadic metrics (received from or ) can be controlled via option. See also , and . `histogram_bucket` returns for the input over the given `interval`. `histogram_bucket` makes sense only for aggregating . See how to aggregate regular histograms . The results of `histogram_bucket` is equal to the following query: Aggregating irregular and sporadic metrics (received from or ) can be controlled via option. ```metricsql sum(histogramovertime(somehistogrambucket[interval])) by (vmrange) ``` See also , , and . `last` returns the last input over the given `interval`. The results of `last` is roughly equal to the the following query: ```metricsql lastovertime(some_metric[interval]) ``` See also , and . `max` returns the maximum input over the given `interval`. The results of `max` is equal to the following query: ```metricsql max(maxovertime(some_metric[interval])) ``` For example, see below time series produced by config with aggregation interval `1m` and the regular query: <img alt="total aggregation" src="stream-aggregation-check-max.webp"> See also and . `min` returns the minimum input over the given `interval`. The results of `min` is equal to the following query: ```metricsql min(minovertime(some_metric[interval])) ``` For example, see below time series produced by config with aggregation interval `1m` and the regular query: <img alt="min aggregation" src="stream-aggregation-check-min.webp"> See also and . `stddev` returns for the input over the given `interval`. `stddev` makes sense only for aggregating . The results of `stddev` is roughly equal to the following query: ```metricsql histogramstddev(sum(histogramovertime(somemetric[interval])) by (vmrange)) ``` See also and . `stdvar` returns for the input over the given `interval`. `stdvar` makes sense only for aggregating . The results of `stdvar` is roughly equal to the following query: ```metricsql histogramstdvar(sum(histogramovertime(somemetric[interval])) by (vmrange)) ``` For example, see below time series produced by config with aggregation interval `1m` and the regular query: <img alt="stdvar aggregation" src="stream-aggregation-check-stdvar.webp"> See also and . `sum_samples` sums input over the given `interval`. `sum_samples` makes sense only for aggregating

The results of `sum_samples` is equal to the following query: ```metricsql sum(sumovertime(some_metric[interval])) ``` For example, see below time series produced by config with aggregation interval `1m` and the regular query: <img alt="sum_samples aggregation" src="stream-aggregation-check-sum-samples.webp"> See also and . `total` generates output by summing the input counters over the given `interval`. `total` makes sense only for aggregating . The results of `total` is roughly equal to the the following query: ```metricsql sum(runningsum(increasepure(some_counter))) ``` `total` assumes that all the counters start from 0. For example, if the first seen sample for new is `10`, then `total` assumes that the time series has been increased by `10`. If you need ignoring the first sample for new time series, then take a look at . For example, see below time series produced by config with aggregation interval `1m` and `by: ["instance"]` and the regular query: <img alt="total aggregation" src="stream-aggregation-check-total.webp"> `total` is not affected by - it continues to increase monotonically with respect to the previous value. The counters are most often reset when the application is restarted. For example: <img alt="total aggregation counter reset" src="stream-aggregation-check-total-reset.webp"> The same behavior occurs when creating or deleting new series in an aggregation group - `total` output increases monotonically considering the values of the series set. An example of changing a set of series can be restarting a pod in the Kubernetes. This changes pod name label, but the `total` accounts for such a scenario and doesn't reset the state of aggregated metric. Aggregating irregular and sporadic metrics (received from or ) can be controlled via option. See also , and . `total_prometheus` generates output by summing the input counters over the given `interval`. `total_prometheus` makes sense only for aggregating . The results of `total_prometheus` is roughly equal to the the following query: ```metricsql sum(runningsum(increaseprometheus(some_counter))) ``` `total_prometheus` skips the first seen sample value per each . If you need taking into account the first sample per time series, then take a look at . `total_prometheus` is not affected by - it continues to increase monotonically with respect to the previous value. The counters are most often reset when the application is restarted. Aggregating irregular and sporadic metrics (received from or ) can be controlled via option. See also , and . `unique_samples` counts the number of unique sample values over the given `interval`. `unique_samples` makes sense only for aggregating . The results of `unique_samples` is equal to the following query: ```metricsql count(countvaluesovertime(somemetric[interval])) ``` See also and . `quantiles(phi1, ..., phiN)` returns for the given `phi*` over the input on the given `interval`. `phi` must be in the range `[0..1]`, where `0` means `0th` percentile, while `1` means `100th` percentile. `quantiles(...)` makes sense only for aggregating . The results of `quantiles(phi1, ..., phiN)` is equal to the following query: ```metricsql histogramquantiles("quantile", phi1, ..., phiN, sum(histogramovertime(somemetric[interval])) by (vmrange)) ``` See also , , and . All the labels for the input metrics are preserved by default in the output metrics. For example, the input metric `foo{app="bar",instance="host1"}` results to the output metric `foo:1msumsamples{app="bar",instance="host1"}` when the following is used: ```yaml interval: 1m outputs: [sum_samples] ``` The input labels can be removed via `without` list specified in the config. For example, the following config removes the `instance` label from output metrics by summing input samples across all the instances: ```yaml interval: 1m without: [instance] outputs: [sum_samples] ``` In this case the `foo{app="bar",instance="..."}` input metrics are transformed into `foo:1mwithoutinstancesumsamples{app="bar"}` output metric according to

It is possible specifying the exact list of labels in the output metrics via `by` list. For example, the following config sums input samples by the `app` label: ```yaml interval: 1m by: [app] outputs: [sum_samples] ``` In this case the `foo{app="bar",instance="..."}` input metrics are transformed into `foo:1mbyappsumsamples{app="bar"}` output metric according to . The labels used in `by` and `without` lists can be modified via `inputrelabelconfigs` section - see . See also . Below is the format for stream aggregation config file, which may be referred via `-streamAggr.config` command-line flag at and . At `-remoteWrite.streamAggr.config` command-line flag can be specified individually per each `-remoteWrite.url`: ```yaml match: 'httprequestdurationsecondsbucket{env=~"prod|staging"}' interval: 1m without: [instance] outputs: [total] inputrelabelconfigs: target_label: vmaggr replacement: before outputrelabelconfigs: target_label: vmaggr replacement: after ``` The file can contain multiple aggregation configs. The aggregation is performed independently per each specified config entry. and support the following approaches for hot reloading stream aggregation configs from `-remoteWrite.streamAggr.config` and `-streamAggr.config`: By sending `SIGHUP` signal to `vmagent` or `victoria-metrics` process: ```sh kill -SIGHUP `pidof vmagent` ``` By sending HTTP request to `/-/reload` endpoint (e.g. `http://vmagent:8429/-/reload` or `http://victoria-metrics:8428/-/reload). . . . . The following outputs track the last seen per-series values in order to properly calculate output values: The last seen per-series value is dropped if no new samples are received for the given time series during two consecutive aggregation intervals specified in via `interval` option. If a new sample for the existing time series is received after that, then it is treated as the first sample for a new time series. This may lead to the following issues: Lower than expected results for and outputs, since they ignore the first sample in a new time series. Unexpected spikes for and outputs, since they assume that new time series start from 0. These issues can be be fixed in the following ways: By increasing the `interval` option at , so it covers the expected delays in data ingestion pipelines. By specifying the `staleness_interval` option at , so it covers the expected delays in data ingestion pipelines. By default, the `staleness_interval` equals to `2 x interval`. The following solutions can help reducing memory usage and CPU usage durting streaming aggregation: To use more specific `match` filters at , so only the really needed are aggregated. To increase aggregation interval by specifying bigger duration for the `interval` option at . To generate lower number of output time series by using less specific or more specific . To drop unneeded long labels in input samples via . If you use for streaming aggregation then be careful when using or when modfying sample labels via , since incorrect usage may result in duplicates and data collision. For example, if more than one `vmagent` instance calculates for `httprequeststotal` metric with `by: [path]` option, then all the `vmagent` instances will aggregate samples to the same set of time series with different `path` labels. The proper fix would be for all the output samples produced by each `vmagent`, so they are aggregated into distinct sets of . These time series then can be aggregated later as needed during querying. If `vmagent` instances run in Docker or Kubernetes, then you can refer `POD_NAME` or `HOSTNAME` environment variables as an unique label value per each `vmagent` via `-remoteWrite.label=vmagent=%{HOSTNAME}` command-line flag. See on how to refer environment variables in VictoriaMetrics components.