Project: VictoriaMetrics (Observability and Analysis / Observability)
File: CHANGELOG_2023.md

sort: 104 weight: 104 title: CHANGELOG for the year 2023 menu: docs: parent: 'victoriametrics' weight: 104 aliases: /CHANGELOG_2023.html Released at 2023-12-13 vmalert's metrics `vmalert_alerting_rules_error` and `vmalert_recording_rules_error` were replaced with `vmalert_alerting_rules_errors_total` and `vmalert_recording_rules_errors_total`. See for details. SECURITY: upgrade base docker image (Alpine) from 3.18.4 to 3.19.0. See . SECURITY: upgrade Go builder from Go1.21.4 to Go1.21.5. See . FEATURE: : add ability to send requests to the first available backend and fall back to other `hot standby` backends when the first backend is unavailable. This allows building highly available setups as shown in . See . FEATURE: `vmselect`: allow specifying multiple groups of `vmstorage` nodes with independent `-replicationFactor` per each group. See and for details. FEATURE: `vmselect`: allow opening and investigating and when the `vmselect` is overloaded with concurrent queries (e.g. when more than `-search.maxConcurrentRequests` concurrent queries are executed). Previously an attempt to open `Top queries` or `Active queries` at `vmui` could result in `couldn't start executing the request in ... seconds, since -search.maxConcurrentRequests=... concurrent requests are executed` error, which could complicate debugging of overloaded `vmselect` or single-node VictoriaMetrics. FEATURE: : add `-enableMultitenantHandlers` command-line flag, which allows receiving data via at `vmagent` and converting to (`vmaccountid`, `vmprojectid`) labels before sending the data to the configured `-remoteWrite.url`. See for details. FEATURE: : add `-remoteWrite.disableOnDiskQueue` command-line flag, which can be used for disabling data queueing to disk when the remote storage cannot keep up with the data ingestion rate. See and . FEATURE: : add support for reading and writing samples via . See . FEATURE: : show all the dropped targets together with the reason why they are dropped at `http://vmagent:8429/service-discovery` page. Previously targets, which were dropped because of weren't displayed on this page. This could complicate service discovery debugging. See and . FEATURE: reduce the default value for `-import.maxLineLen` command-line flag from 100MB to 10MB in order to prevent excessive memory usage during data import via . FEATURE: : add `keepifcontains` and `dropifcontains` relabeling actions. See for details. FEATURE: : export `vmpromscrapescrapepooltargets` to track the number of targets each scrape job discovers. See . FEATURE: : provide `/vmalert/api/v1/rule` and `/api/v1/rule` API endpoints to get the rule object in JSON format. See for details. FEATURE: : deprecate process gauge metrics `vmalertalertingruleserror` and `vmalertrecordingruleserror` in favour of `vmalertalertingruleserrorstotal` and `vmalertrecordingruleserrorstotal` counter metrics. metric type is more suitable for error counting as it preserves the state change between the scrapes. See for details. FEATURE: : add function, which returns the day of the year for each of the given unix timestamps. See for details. Thanks to @luckyxiaoqiang for the . FEATURE: all VictoriaMetrics binaries: expose additional metrics at `/metrics` page, which may simplify debugging of VictoriaMetrics components (see ): `goschedlatencies_seconds` - the , which shows the time goroutines have spent in runnable state before actually running. Big values point to the lack of CPU time for the current workload. `gomutexwaitsecondstotal` - the , which shows the total time spent by goroutines waiting for locked mutex. Big values point to mutex contention issues. `gogccpusecondstotal` - the , which shows the total CPU time spent by Go garbage collector. `gogcmarkassistcpusecondstotal` - the , which shows the total CPU time spent by goroutines in GC mark assist state. `gogcpauses_seconds` - the , which shows the duration of GC pauses. `goscavengecpusecondstotal` - the , which shows the total CPU time spent by Go runtime for returning memory to the Operating System. `gomemlimitbytes` - the value of environment variable. FEATURE: : enhance autocomplete functionality with caching. See

FEATURE: add field `version` to the response for `/api/v1/status/buildinfo` API for using more efficient API in Grafana for receiving label values. Add additional info about setup Grafana datasource. See and for details. FEATURE: add `-search.maxResponseSeries` command-line flag for limiting the number of time series a single query to or can return. This limit can protect Grafana from high memory usage when the query returns too many series. See . FEATURE: : ease aggregation for certain alerting rules to keep more useful labels for the context. Before, all extra labels except `job` and `instance` were ignored. See this and this . Thanks to @7840vz. FEATURE: : allow reversing the migrating order from the newest to the oldest data for and modes via `--vm-native-filter-time-reverse` and `--remote-read-filter-time-reverse` command-line flags respectively. See: https://docs.victoriametrics.com/vmctl/#using-time-based-chunking-of-migration and . BUGFIX: : properly calculate values for the first point on the graph for queries, which do not use . For example, previously `count(up)` could return lower than expected values for the first point on the graph. This also could result in lower than expected values in the middle of the graph like in when the response caching isn't disabled. The issue has been introduced in . BUGFIX: : prevent from `FATAL: cannot flush metainfo` panic when command-line flag is set. See . BUGFIX: : properly decode zstd-encoded data blocks received via . See . BUGFIX: : properly add new labels at `outputrelabelconfigs` during . Previously this could lead to corrupted labels in output samples. Thanks to @ChengChung for providing . BUGFIX: : allow using arbitrary `evaltime` in case. Previously, test cases with `evaltime` not being a multiple of `evaluation_interval` would fail. BUGFIX: : sanitize label names before sending the alert notification to Alertmanager. Before, vmalert would send notifications with labels containing characters not supported by Alertmanager validator, resulting into validation errors like `msg="Failed to validate alerts" err="invalid label set: invalid name "foo.bar"`. BUGFIX: : fix `vmbackupmanager` not deleting previous object versions from S3 when applying retention policy with `-deleteAllObjectVersions` command-line flag. BUGFIX: : fix panic when ingesting data via into VictoriaMetrics cluster. See . BUGFIX: properly escape `<` character in responses returned via endpoint. See . BUGFIX: : check for Error field in response from influx client during migration. Before, only network errors were checked. Thanks to @wozz for the . Released at 2023-11-16 FEATURE: dashboards: use `version` instead of `short_version` in version change annotation for single/cluster dashboards. The update should reflect version changes even if different flavours of the same release were applied (custom builds). BUGFIX: fix a bug, which could result in improper results and/or to `cannot merge series: duplicate series found` error during execution. The issue has been introduced in . See for details. BUGFIX: improve deadline detection when using buffered connection for communication between cluster components. Before, due to nature of a buffered connection the deadline could have been exceeded while reading or writing buffered data to connection. See . Released at 2023-11-15 It is recommended upgrading to because v1.95.0 contains a bug, which can lead to incorrect query results and to `cannot merge series: duplicate series found` error. See for details. vmalert's cmd-line flag `-datasource.lookback` will be deprecated soon. Please use `-rule.evalDelay` command-line flag instead and see more details on how to use it . The flag `datasource.lookback` will have no effect in the next release and will be removed in the future releases. See . vmalert's cmd-line flag `-datasource.queryTimeAlignment` was deprecated and will have no effect anymore. It will be completely removed in next releases. See and more detailed changes related to vmalert below. SECURITY: upgrade Go builder from Go1.21.1 to Go1.21.4. See , and

FEATURE: `vmselect`: improve performance for repeated if they contain one of the following : - - - - - The optimization is enabled when these functions contain lookbehind window in square brackets bigger or equal to `6h` (the threshold can be changed via `-search.minWindowForInstantRollupOptimization` command-line flag). The optimization improves performance for SLO/SLI-like queries such as `avgovertime(up or similar projects. FEATURE: `vmselect`: improve query performance on systems with big number of CPU cores (`>=32`). Add `-search.maxWorkersPerQuery` command-line flag, which can be used for fine-tuning query performance on systems with big number of CPU cores. See . FEATURE: `vmselect`: expose `vmmemoryintensivequeriestotal` counter metric which gets increased each time `-search.logQueryMemoryUsage` memory limit is exceeded by a query. This metric should help to identify expensive and heavy queries without inspecting the logs. FEATURE: : add function, which can be used for filtering out empty series before performing additional calculations as shown in . FEATURE: : add function, which can be used for searching series with identical values for the given labels. See . FEATURE: : add and functions, which allow detecting anomalies in samples and series using . FEATURE: : add `eval_alignment` attribute for , it will align group query requests timestamp with interval like `datasource.queryTimeAlignment` did. This also means that `datasource.queryTimeAlignment` command-line flag becomes deprecated now and will have no effect if configured. If `datasource.queryTimeAlignment` was set to `false` before, then `eval_alignment` has to be set to `false` explicitly under group. See . FEATURE: : add `-rule.evalDelay` flag and `eval_delay` attribute for . The new flag and param can be used to adjust the `time` parameter for rule evaluation requests to match from the datasource. See . FEATURE: : allow specifying full url in notifier static_configs target address, like `http://alertmanager:9093/test/api/v2/alerts`. See . FEATURE: : reduce the number of queries for restoring alerts state on start-up. The change should speed up the restore process and reduce pressure on `remoteRead.url`. See . FEATURE: : add label `file` pointing to the group's filename to metrics `vmalert_recording_.` and `vmalertalerts.*`. The filename should help identifying alerting rules belonging to specific groups with identical names but different filenames. See . FEATURE: : automatically retry remote-write requests on closed connections. The change should reduce the amount of logs produced in environments with short-living connections or environments without support of keep-alive on network balancers. FEATURE: : support data ingestion from . See , and . FEATURE: : add `-remoteWrite.shardByURL.labels` command-line flag, which can be used for specifying a list of labels for sharding outgoing samples among the configured `-remoteWrite.url` destinations if `-remoteWrite.shardByURL` command-line flag is set. See and for details. FEATURE: : do not exit on startup when refer to non-existing or invalid files with auth configs, since these files may appear / updated later. See and . FEATURE: : allow loading TLS certificates from HTTP and HTTPS urls by specifying these urls at `certfile` and `keyfile` options inside `tlsconfig` and `proxytls_config` sections at . FEATURE: : reduce CPU load when big number of targets are scraped over HTTPS with the same custom TLS certificate configured via `tlsconfig->certfile` and `tlsconfig->keyfile` at . FEATURE: : add `-filestream.disableFadvise` command-line flag, which can be used for disabling `fadvise` syscall during backup upload to the remote storage. By default `vmbackup` uses `fadvise` syscall in order to prevent from eviction of recently accessed data from the when backing up large files. Sometimes the `fadvise` syscall may take significant amounts of CPU when the backup is performed with large value of `-concurrency` command-line flag on systems with big number of CPU

In this case it is better to manually disable `fadvise` syscall by passing `-filestream.disableFadvise` command-line flag to `vmbackup`. See for details. FEATURE: : add `-deleteAllObjectVersions` command-line flag, which can be used for forcing removal of all object versions in remote object storage. See issue and for the details. FEATURE: : account for `vmauth` component for alerts `ServiceDown` and `TooManyRestarts`. FEATURE: : make `TooHighMemoryUsage` more tolerable to spikes or near-the-threshold states. The change should reduce number of false positives. FEATURE: : add `TooManyMissedIterations` alerting rule for vmalert to detect groups that miss their evaulations due to slow queries. FEATURE: : add support for functions, labels, values in autocomplete. See . FEATURE: : retain specified time interval when executing a query from `Top Queries`. See . FEATURE: : improve repeated VMUI page load times by enabling caching of static js and css at web browser side according to . FEATURE: : sort legend under the graph in descending order of median values. This should simplify graph analysis, since usually the most important lines have bigger values. FEATURE: : reduce vertical space usage, so more information is visible on the screen without scrolling. FEATURE: : show query execution duration in the header of query input field. This should help optimizing query performance. FEATURE: support `Strict-Transport-Security`, `Content-Security-Policy` and `X-Frame-Options` HTTP response headers in the all VictoriaMetrics components. The values for headers can be specified via the following command-line flags: `-http.header.hsts`, `-http.header.csp` and `-http.header.frameOptions`. FEATURE: : add `unittest` command to run unittest for alerting and recording rules. See for details. FEATURE: dashboards/vmalert: add new panel `Missed evaluations` for indicating alerting groups that miss their evaluations. FEATURE: all: track requests with wrong auth key and wrong basic auth at `vmhttprequesterrorstotal` with `reason="wrongauthkey"` and `reason="wrongbasicauth"`. See . Thanks to @venkatbvc for the . FEATURE: : add ability to drop the specified number of `/`-delimited prefix parts from the request path before proxying the request to the matching backend. See . FEATURE: : add ability to skip TLS verification and to specify TLS Root CA when connecting to backends. See and . FEATURE: `vmstorage`: gradually close `vminsert` connections during 25 seconds at . This should reduce data ingestion slowdown during rolling restarts. The duration for gradual closing of `vminsert` connections can be configured via `-storage.vminsertConnsShutdownDuration` command-line flag. See and for details. FEATURE: `vmstorage`: add `-blockcache.missesBeforeCaching` command-line flag, which can be used for fine-tuning RAM usage for `indexdb/dataBlocks` cache when queries touching big number of time series are executed. FEATURE: add `-loggerMaxArgLen` command-line flag for fine-tuning the maximum lengths of logged args. BUGFIX: : strip sensitive information such as auth headers or passwords from datasource, remote-read, remote-write or notifier URLs in log messages or UI. This behavior is by default and is controlled via `-datasource.showURL`, `-remoteRead.showURL`, `remoteWrite.showURL` or `-notifier.showURL` cmd-line flags. See . BUGFIX: : fix vmalert web UI when running on 32-bit architectures machine. BUGFIX: : do not send requests to configured remote systems when `-datasource.`, `-remoteWrite.`, `-remoteRead.` or `-notifier.*` command-line flags refer files with invalid auth configs. Previously such requests were sent without properly set auth headers. Now the requests are sent only after the files are updated with valid auth configs. See . BUGFIX: : properly maintain alerts state in if alert's `for` param was bigger than replay request range (usually a couple of hours). See for details. BUGFIX: : increment `vmalertremotewriteerrors_total` metric if all retries to send remote-write request failed. Before, this metric was incremented only if remote-write client's buffer is overloaded. BUGFIX: : increment `vmalertremotewritedroppedrowstotal` metric if remote-write client's buffer is overloaded. Before, these metrics were incremented only after unsuccessful HTTP

BUGFIX: `vmselect`: improve performance and memory usage during query processing on machines with big number of CPU cores. See . BUGFIX: dashboards: fix vminsert/vmstorage/vmselect metrics filtering when dashboard is used to display data from many sub-clusters with unique job names. Before, only one specific job could have been accounted for component-specific panels, instead of all available jobs for the component. BUGFIX: dashboards: respect `job` and `instance` filters for `alerts` annotation in cluster and single-node dashboards. BUGFIX: dashboards: update description for RSS and anonymous memory panels to be consistent for single-node, cluster and vmagent dashboards. BUGFIX: dashboards/vmalert: apply `desc` sorting in tooltips for vmalert dashboard in order to improve visibility of the outliers on graph. BUGFIX: dashboards/vmalert: properly apply time series filter for panel `No data errors`. Before, the panel didn't respect `job` or `instance` filters. BUGFIX: dashboards/vmalert: fix panel `Errors rate to Alertmanager` not showing any data due to wrong label filters. BUGFIX: dashboards/cluster: fix description about `max` threshold for `Concurrent selects` panel. Before, it was mistakenly implying that `max` is equal to the double of available CPUs. BUGFIX: : bump hard-coded limit for search query size at `vmstorage` from 1MB to 5MB. The change should be more suitable for real-world scenarios and protect vmstorage from excessive memory usage. See for details BUGFIX: : fix error when creating an incremental backup with the `-origin` command-line flag. See for details. BUGFIX: : properly apply with `regex`, which start and end with `.+` or `.` and which contain alternate sub-regexps. For example, `.+;|;.+` or `.foo|bar|baz.`. Previously such regexps were improperly parsed, which could result in undexpected relabeling results. See . BUGFIX: : properly discover Kubernetes targets via . Previously some targets and some labels could be skipped during service discovery because of the bug introduced in when implementing . See for more details. BUGFIX: : fix vmagent ignoring configuration reload for streaming aggregation if it was started with empty streaming aggregation config. Thanks to @aluode99 for the . BUGFIX: : do not scrape targets if the corresponding refer to files with invalid auth configs. Previously the targets were scraped without properly set auth headers in this case. Now targets are scraped only after the files are updated with valid auth configs. See . BUGFIX: : properly parse `ca`, `cert` and `key` options at `tls_config` section inside . Previously string values couldn't be parsed for these options, since the parser was mistakenly expecting a list of `uint8` values instead. BUGFIX: : properly drop samples if `-streamAggr.dropInput` command-line flag is set and `-remoteWrite.streamAggr.config` contains an empty file. See . BUGFIX: : do not print redundant error logs when failed to scrape consul or nomad target. See . BUGFIX: : generate proper link to the main page and to `favicon.ico` at http pages served by `vmagent` such as `/targets` or `/service-discovery` when `vmagent` sits behind an http proxy with custom http path prefixes. See . BUGFIX: : properly decode Snappy-encoded data blocks received via . See . BUGFIX: : prevent deleted series to be searchable via `/api/v1/series` API if they were re-ingested with staleness markers. This situation could happen if user deletes the series from the target and from VM, and then vmagent sends stale markers for absent series. Thanks to @ilyatrefilov for the and . BUGFIX: : log warning about switching to ReadOnly mode only on state change. Before, vmstorage would log this warning every 1s. See for details. BUGFIX: : show browser authorization window for unauthorized requests to unsupported paths if the `unauthorized_user` section is specified. This allows properly authorizing the user. See for

BUGFIX: : properly proxy requests to HTTP/2.0 backends and properly pass `Host` header to backends. BUGFIX: : fix the `Disable cache` toggle at `JSON` and `Table` views. Previously response caching was always enabled and couldn't be disabled at these views. BUGFIX: : correctly display query errors on page. See for details. BUGFIX: : properly handle trailing slash in the server URL. See . BUGFIX: : correctly print error in logs when copying backup fails. Previously, error was displayed in metrics but was missing in logs. BUGFIX: fix panic, which could occur when is enabled. See . Released at 2023-10-02 FEATURE: : add support for numbers with underscore delimiters such as `1234567890` and `1.234567_890`. These numbers are easier to read than `1234567890` and `1.234567890`. FEATURE: : add support for server-side copy of existing backups. See for details. FEATURE: : add the option to see the latest 25 queries. See . FEATURE: : add ability to set `member num` label for all the metrics scraped by a particular `vmagent` instance in via `-promscrape.cluster.memberLabel` command-line flag. See and . FEATURE: : do not log `unexpected EOF` when reading incoming metrics, since this error is expected and is handled during metrics' parsing. This reduces the amounts of noisy logs. See . FEATURE: : retry failed write request on the closed connection immediately, without waiting for backoff. This should improve data delivery speed and reduce amount of error logs emitted by vmagent when using idle connections. See related . FEATURE: : reduces load on Kubernetes control plane during initial service discovery. See for details. FEATURE: : reduce the maximum recovery time at `vmselect` and `vminsert` when some of `vmstorage` nodes become unavailable because of networking issues from 60 seconds to 3 seconds by default. The recovery time can be tuned at `vmselect` and `vminsert` nodes with `-vmstorageUserTimeout` command-line flag if needed. Thanks to @wjordan for . FEATURE: : add Prometheus data support to the "Explore cardinality" page. See for details. FEATURE: : make the warning message more noticeable for text fields. See . FEATURE: : add button for auto-formatting PromQL/MetricsQL queries. See . Thanks to @aramattamara for the . FEATURE: : improve accessibility score to 100 according to tests. FEATURE: : organize `min`, `max`, `median` values on the chart legend and tooltips for better visibility. FEATURE: : add explanation about statistic inaccuracy in VictoriaMetrics cluster. See . FEATURE: : add storage of query history in `localStorage`. See . FEATURE: dashboards: provide copies of Grafana dashboards alternated with VictoriaMetrics datasource at . FEATURE: : added ability to set, override and clear request and response headers on a per-user and per-path basis. See and for details. FEATURE: : add ability to retry requests to the if they return response status codes specified in the `retrystatuscodes` list. See . FEATURE: : expose metrics `vmauth_config_last_reload_` for tracking the state of config reloads, similarly to vmagent/vmalert components. FEATURE: : do not print logs like `SIGHUP received...` once per configured `-configCheckInterval` cmd-line flag. This log will be printed only if config reload was invoked manually. FEATURE: : add `eval_offset` attribute for . If specified, Group will be evaluated at the exact time offset on the range of for details. FEATURE: : validate function names in alerting and recording rules when `vmalert` runs with `-dryRun` command-line flag. Previously it was allowed to use unknown (aka invalid) MetricsQL function names there. For example, `foo()` was counted as a valid query. See . FEATURE: limit the length of string params in log messages to 500 chars. Longer string params are replaced with the

This prevents from too long log lines, which can be emitted by VictoriaMetrics components. FEATURE: : add `vmauth` component to cluster's docker-compose example for balancing load among multiple `vmselect` components. FEATURE: : make sure that `q2` series are returned after `q1` series in the results of `q1 or q2` query, in the same way as Prometheus does. See . FEATURE: : return empty result from , and if `a` or `b` have no value at the particular timestamp. Previously `0` was returned in this case. See . FEATURE: stop exposing `vmmergeneedfreedisk_space` metric, since it has been appeared that it confuses users while doesn't bring any useful information. See . BUGFIX: : fix display of ingested rows rate for `Samples ingested/s` and `Samples rate` panels for vmagent's dasbhoard. Previously, not all ingested protocols were accounted in these panels. An extra panel `Rows rate` was added to `Ingestion` section to display the split for rows ingested rate by protocol. BUGFIX: : fix the bug causing render looping when switching to heatmap. BUGFIX: validate `-dedup.minScrapeInterval` value and `-downsampling.period` intervals are multiples of each other. See . BUGFIX: : properly copy `appliedRetention.txt` files inside `<-storageDataPath>/{data}` folders during . Previously the new `appliedRetention.txt` could be skipped during incremental backups, which could lead to increased load on storage after restoring from backup. See . BUGFIX: : suppress `context canceled` error messages in logs when `vmagent` is reloading service discovery config. This error could appear starting from . See . BUGFIX: : remove concurrency limit during parsing of scraped metrics, which was mistakenly applied to it. With this change cmd-line flag `-maxConcurrentInserts` won't have effect on scraping anymore. BUGFIX: : allow passing to . See . BUGFIX: : fix ingestion via for opentsdbhttp. See . The bug has been introduced in . BUGFIX: : fix support of legacy DataDog agent, which adds trailing slashes to urls. See . Thanks to @maxb for spotting the issue. Released at 2023-12-10 v1.93.x is a line of . It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since release SECURITY: upgrade base docker image (Alpine) from 3.18.4 to 3.19.0. See . SECURITY: upgrade Go builder from Go1.21.4 to Go1.21.5. See . BUGFIX: : prevent from `FATAL: cannot flush metainfo` panic when command-line flag is set. See . BUGFIX: : properly decode zstd-encoded data blocks received via . See . BUGFIX: : properly add new labels at `outputrelabelconfigs` during . Previously this could lead to corrupted labels in output samples. Thanks to @ChengChung for providing . BUGFIX: : sanitize label names before sending the alert notification to Alertmanager. Before, vmalert would send notifications with labels containing characters not supported by Alertmanager validator, resulting into validation errors like `msg="Failed to validate alerts" err="invalid label set: invalid name "foo.bar"`. BUGFIX: properly escape `<` character in responses returned via endpoint. See . Released at 2023-11-15 v1.93.x is a line of . It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since release SECURITY: upgrade Go builder from Go1.21.3 to Go1.21.4. See . BUGFIX: : properly apply with `regex`, which start and end with `.+` or `.` and which contain alternate sub-regexps. For example, `.+;|;.+` or `.foo|bar|baz.`. Previously such regexps were improperly parsed, which could result in undexpected relabeling results. See . BUGFIX: : properly decode Snappy-encoded data blocks received via . See . BUGFIX: fix panic, which could occur when is enabled. See . Released at 2023-11-02 v1.93.x is a line of . It contains important up-to-date bugfixes. The

line will be supported for at least 12 months since release BUGFIX: : properly discover Kubernetes targets via . Previously some targets and some labels could be skipped during service discovery because of the bug introduced in when implementing . See for more details. BUGFIX: : properly parse `ca`, `cert` and `key` options at `tls_config` section inside . Previously string values couldn't be parsed for these options, since the parser was mistakenly expecting a list of `uint8` values instead. BUGFIX: : properly drop samples if `-streamAggr.dropInput` command-line flag is set and `-remoteWrite.streamAggr.config` contains an empty file. See . BUGFIX: : do not print redundant error logs when failed to scrape consul or nomad target. See . BUGFIX: : prevent deleted series to be searchable via `/api/v1/series` API if they were re-ingested with staleness markers. This situation could happen if user deletes the series from the target and from VM, and then vmagent sends stale markers for absent series. Thanks to @ilyatrefilov for the and . BUGFIX: : log warning about switching to ReadOnly mode only on state change. Before, vmstorage would log this warning every 1s. See for details. BUGFIX: : show browser authorization window for unauthorized requests to unsupported paths if the `unauthorized_user` section is specified. This allows properly authorizing the user. See for details. Released at 2023-10-16 v1.93.x is a line of . It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since release SECURITY: upgrade Go builder from Go1.21.1 to Go1.21.3. See and . BUGFIX: : strip sensitive information such as auth headers or passwords from datasource, remote-read, remote-write or notifier URLs in log messages or UI. This behavior is by default and is controlled via `-datasource.showURL`, `-remoteRead.showURL`, `remoteWrite.showURL` or `-notifier.showURL` cmd-line flags. See . BUGFIX: `vmselect`: improve performance and memory usage during query processing on machines with big number of CPU cores. See for details. BUGFIX: : bump hard-coded limit for search query size at `vmstorage` from 1MB to 5MB. The change should be more suitable for real-world scenarios and protect vmstorage from excessive memory usage. See for details BUGFIX: : fix vmagent ignoring configuration reload for streaming aggregation if it was started with empty streaming aggregation config. Thanks to @aluode99 for the . BUGFIX: : properly copy `appliedRetention.txt` files inside `<-storageDataPath>/{data}` folders during . Previously the new `appliedRetention.txt` could be skipped during incremental backups, which could lead to increased load on storage after restoring from backup. See . BUGFIX: : suppress `context canceled` error messages in logs when `vmagent` is reloading service discovery config. This error could appear starting from . See . BUGFIX: : allow passing to . See . BUGFIX: : fix ingestion via for opentsdbhttp. See . The bug has been introduced in . BUGFIX: : fix support of legacy DataDog agent, which adds trailing slashes to urls. See . Thanks to @maxb for spotting the issue. Released at 2023-09-19 v1.93.x is a line of . It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since release BUGFIX: : prevent from livelock when is called under high data ingestion. See . BUGFIX: : correctly return `null` instead of `Inf` in JSON query responses. See . BUGFIX: : properly copy `parts.json` files inside `<-storageDataPath>/{data,indexdb}` folders during . Previously the new `parts.json` could be skipped during incremental backups, which could lead to inability to restore from the backup. See . This issue has been introduced in . BUGFIX: : properly close connections to Kubernetes API server after the change in `selectors` or `namespaces` sections of

Previously `vmagent` could continue polling Kubernetes API server with the old `selectors` or `namespaces` configs additionally to polling new configs. See . BUGFIX: : prevent configuration reloading if there were no changes in config. This improves memory usage when `-configCheckInterval` cmd-line flag is configured and config has extensive list of regexp expressions requiring additional memory on parsing. Released at 2023-09-10 v1.93.x is a line of . It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since release SECURITY: upgrade Go builder from Go1.21.0 to Go1.21.1. See . BUGFIX: : properly parse `/insert/multitenant/` urls, which have been broken since . See . BUGFIX: properly build production armv5 binaries for `GOARCH=arm`. This has been broken after the upgrading of Go builder to Go1.21.0. See . BUGFIX: : return `503 Service Unavailable` status code when are denied and some of `vmstorage` nodes are temporarily unavailable. Previously `422 Unprocessable Entiry` status code was mistakenly returned in this case, which could prevent from automatic recovery by re-sending the request to healthy cluster replica in another availability zone. BUGFIX: : fix the bug when Group's `params` fields with multiple values were overriding each other instead of adding up. The bug was introduced in starting from . See . BUGFIX: : fix possble corruption of labels in the collected samples if `-remoteWrite.label` is set toghether with multiple `-remoteWrite.url` options. The bug has been introduced in . See . Released at 2023-09-02 v1.93.x is a line of . It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since release BUGFIX: : properly close broken vmstorage connection during checks at `vmstorage`. Previously it wasn't properly closed, which prevents restoring `vmstorage` node from read-only mode. See . BUGFIX: : prevent from breaking `vmselect` -> `vmstorage` RPC communication when `vmstorage` returns an empty label name at `/api/v1/labels` request. See . Released at 2023-09-01 v1.93.x is a line of . It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since release BUGFIX: : fix Docker builds for old Docker releases. See . BUGFIX: : consistently set `User-Agent` header to `vm_promscrape` during scraping with enabled or disabled . See . BUGFIX: : consistently set timeout for scraping with enabled or disabled . See . BUGFIX: : correctly re-use HTTP request object on `EOF` retries when querying the configured datasource. Previously, there was a small chance that query retry wouldn't succeed. BUGFIX: : correctly handle requests with `/select/multitenant` prefix. Such requests must be rejected since vmselect does not support multitenancy endpoint. Previously, such requests were causing panic. See . BUGFIX: : properly check for at `vmstorage`. Previously it wasn't properly checked, which could lead to increased resource usage and data ingestion slowdown when some of `vmstorage` nodes are in read-only mode. See . Released at 2023-08-23 v1.93.x is a line of . It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since release BUGFIX: prevent from possible data loss during `indexdb` rotation. See for details. BUGFIX: do not allow starting VictoriaMetrics components with improperly set boolean command-line flags in the form `-boolFlagName value`, since this leads to silent incomplete flags' parsing. This form should be replaced with `-boolFlagName=value`. See . BUGFIX: : properly set labels from `-remoteWrite.label` command-line flag just before sending samples to the configured `-remoteWrite.url` according to . Previously these labels were incorrectly set before configured via `-remoteWrite.urlRelabelConfigs` and configured via `-remoteWrite.streamAggr.config`, so these labels could be lost or incorrectly transformed before sending the samples to remote

The fix allows using `-remoteWrite.label` for identifying `vmagent` instances in . See and for more details. BUGFIX: remove `DEBUG` logging when parsing `if` filters inside and when parsing `match` filters inside . BUGFIX: properly replace `:` chars in label names with `_` when `-usePromCompatibleNaming` command-line flag is passed to `vmagent`, `vminsert` or single-node VictoriaMetrics. This addresses . BUGFIX: : correctly check if specified `-dst` belongs to specified `-storageDataPath`. See . BUGFIX: : don't interrupt the migration process if no metrics were found for a specific tenant. See . Released at 2023-08-12 It is recommended upgrading to because v1.93.0 contains a bug, which can lead to data loss because of incorrect `indexdb` rotation. See for details. v1.93.x is a line of . It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since release Update note: starting from this release, ignores timestamps provided by scrape targets by default - it associates scraped metrics with local timestamps instead. Set `honortimestamps: true` in if timestamps provided by scrape targets must be used instead. This change helps removing gaps for metrics collected from such as `containermemoryusagebytes`. This also improves data compression and query performance over metrics collected from `cadvisor`. See more details . SECURITY: upgrade Go builder from Go1.20.6 to Go1.21.0 in order to fix . SECURITY: upgrade base docker image (Alpine) from 3.18.2 to 3.18.3. See . FEATURE: : add `shareeqover_time(m. Thanks to @Damon07 for the . FEATURE: : allow configuring deadline for a backend to be excluded from the rotation on errors via `-failTimeout` cmd-line flag. This feature could be useful when it is expected for backends to be not available for significant periods of time. See for details. Thanks to @SunKyu for . FEATURE: : remove deprecated in `-rule.configCheckInterval` command-line flag. Use `-configCheckInterval` command-line flag instead. FEATURE: : remove support of deprecated web links of `/api/v1/<groupID>/<alertID>/status` form in favour of `/api/v1/alerts?groupid=<>&alertid=<>` links. Links of `/api/v1/<groupID>/<alertID>/status` form were deprecated in v1.79.0. See for details. FEATURE: : allow disabling binary export API protocol via `-vm-native-disable-binary-protocol` cmd-line flag when . Disabling binary protocol can be useful for deduplication of the exported data before ingestion. For this, deduplication need at `-vm-native-src-addr` side and `-vm-native-disable-binary-protocol` should be set on vmctl side. FEATURE: : add support of `week` step for . See . FEATURE: : allow specifying custom full url at `--remote-read-src-addr` command-line flag if `--remote-read-disable-path-append` command-line flag is set. This allows importing data from urls, which do not end with `/api/v1/read`. For example, from Promscale. See . FEATURE: : add warning in query field of vmui for partial data responses. See . FEATURE: : allow displaying the full error message on click for trimmed error messages in vmui. See . FEATURE: : add `Concurrent inserts` panel to vmagent's dasbhoard. The new panel supposed to show whether the number of concurrent inserts processed by vmagent isn't reaching the limit. FEATURE: : add panels for absolute Mem and CPU usage by vmalert. See related issue . FEATURE: : correctly calculate `Bytes per point` value for single-server and cluster VM dashboards. Before, the calculation mistakenly accounted for the number of entries in indexdb in denominator, which could have shown lower values than expected. FEATURE: : `ConcurrentFlushesHitTheLimit` alerting rule was moved from and alerts to the as it could be related to many VictoriaMetrics components. BUGFIX: : properly set next retention time for indexDB. Previously it may enter into endless retention loop. See for details. BUGFIX: : return human readable error if opentelemetry has json encoding. Follow-up after

BUGFIX: : properly validate scheme for `proxy_url` field at the scrape config. See for details. BUGFIX: : properly apply `if` filters during . Previously the `if` filter could improperly work. See and . BUGFIX: : use local scrape timestamps for the scraped metrics unless `honor_timestamps: true` option is explicitly set at . This fixes gaps for metrics collected from or similar exporters, which export metrics with invalid timestamps. See and for details. The issue has been introduced in . BUGFIX: : fixes runtime panic at OpenTelemetry parser. Opentelemetry format allows histograms without `sum` fields. Such histogram converted as counter with `_count` suffix. See . BUGFIX: : keep unmatched series at when `-remoteWrite.streamAggr.dropInput` is set to `false` to match intended behaviour introduced at . See . BUGFIX: : properly set `vmalertconfiglastreloadsuccessful` value on configuration updates or rollbacks. The bug was introduced in in . BUGFIX: : fix `vmalertremotewritesenddurationseconds_total` value, before it didn't count in the real time spending on remote write requests. See for details. BUGFIX: : fix panic when creating a backup to a local filesystem on Windows. See . BUGFIX: : properly handle client address with `X-Forwarded-For` part at the page. See . BUGFIX: : prevent from panic when the lookbehind window in square brackets of is parsed into negative value. See . Released at 2023-07-28 BUGFIX: : revert unit test feature for alerting and recording rules introduced in . See the following . Released at 2023-07-27 Update note: this release contains backwards-incompatible change to indexdb, so rolling back to the previous versions of VictoriaMetrics may result in partial data loss of entries in indexdb. Update note: starting from this release, writes the following samples to the configured remote storage by default: aggregated samples; the original input samples, which match zero `match` options from the provided . Previously only aggregated samples were written to the storage by default. The previous behavior can be restored in the following ways: by passing `-streamAggr.dropInput` command-line flag to single-node VictoriaMetrics; by passing `-remoteWrite.streamAggr.dropInput` command-line flag per each configured `-remoteWrite.streamAggr.config` at `vmagent`. SECURITY: upgrade base docker image (alpine) from 3.18.0 to 3.18.2. See . SECURITY: upgrade Go builder from Go1.20.5 to Go1.20.6. See . FEATURE: reduce memory usage by up to 5x for setups with and long . See and for details. FEATURE: reduce spikes in CPU and disk IO usage during `indexdb` rotation (aka inverted index), which is performed once per . The new algorithm gradually pre-populates newly created `indexdb` during the last hour before the rotation. The number of pre-populated series in the newly created `indexdb` can be via `vmtimeseriesprecreated_total` metric. This should resolve . FEATURE: : allow selecting time series matching at least one of multiple `or` filters. For example, `{env="prod",job="a" or env="dev",job="b"}` selects series with either `{env="prod",job="a"}` or `{env="dev",job="b"}` labels. This functionality allows passing the selected series to without the need to use . See . FEATURE: : add ability to preserve metric names for binary operation results via `keepmetricnames` modifier. For example, `({name=~"foo|bar"} / 10) keepmetricnames` leaves `foo` and `bar` metric names in division results. See . This helps to address issues like . FEATURE: : add ability to copy all the labels from `one` side of by specifying `` inside `groupleft()` or `groupright()`. Also allow adding a prefix for copied label names via `groupleft(*) prefix "..."` syntax. For example, the following query copies Kubernetes namespace labels to `kubepodinfo` series and adds `ns` prefix for the copied label names: `kubepodinfo on(namespace) group_left() prefix "ns" kubenamespace_labels`. The labels from `on()` list aren't prefixed. This feature resolves and questions at StackOverflow. FEATURE: : add ability to specify durations via

Examples: `WITH (w = 5m) m[w]` is automatically transformed to `m[5m]` `WITH (f(window, step, off) = m[window:step] offset off) f(5m, 10s, 1h)` is automatically transformed to `m[5m:10s] offset 1h` Thanks to @lujiajing1126 for the initial idea and . See . FEATURE: : added a new page with the list of currently running queries. See and . FEATURE: : add support for data ingestion via . See , and . FEATURE: : allow sharding outgoing time series among the configured remote storage systems. This can be useful for building horizontally scalable , when samples for the same time series must be aggregated by the same `vmagent` instance at the second level. See and for details. FEATURE: : allow configuring staleness interval in config. See for details. FEATURE: : allow specifying a list of inside `if` option of relabeling rules. The corresponding relabeling rule is executed when at least a single series selector matches. See . FEATURE: : allow specifying a list of inside `match` option of . The input sample is aggregated when at least a single series selector matches. See . FEATURE: : preserve input samples, which match zero `match` options from the . Previously all the input samples were dropped by default, so only the aggregated samples are written to the output storage. The previous behavior can be restored by passing `-streamAggr.dropInput` command-line flag to single-node VictoriaMetrics or by passing `-remoteWrite.streamAggr.dropInput` command-line flag to `vmagent`. FEATURE: : add verbose output for docker installations or when TTY isn't available. See . FEATURE: : interrupt backoff retries when import process is cancelled. The change makes vmctl more responsive in case of errors during the import. See . FEATURE: : update backoff policy on retries to reduce probability of overloading for `source` or `destination` databases. See . FEATURE: vmstorage: suppress "broken pipe" and "connection reset by peer" errors for search queries on vmstorage side. See and commits. FEATURE: : add panel for tracking rate of syscalls while writing or reading from disk via `processio(read|write)syscallstotal` metrics. FEATURE: accept timestamps in milliseconds at `start`, `end` and `time` query args in . See and . FEATURE: : update retry policy for pushing data to `-remoteWrite.url`. By default, vmalert will make multiple retry attempts with exponential delay. The total time spent during retry attempts shouldn't exceed `-remoteWrite.retryMaxTime` (default is 30s). When retry time is exceeded vmalert drops the data dedicated for `-remoteWrite.url`. Before, vmalert dropped data after 5 retry attempts with 1s delay between attempts (not configurable). See `-remoteWrite.retryMinInterval` and `-remoteWrite.retryMaxTime` cmd-line flags. FEATURE: : expose `vmalertremotewritesenddurationsecondstotal` counter, which can be used for determining high saturation of every connection to remote storage with an alerting query `sum(rate(vmalertremotewritesenddurationsecondstotal. FEATUTE: : display the error message received during unsuccessful config reload in vmalert's UI. See for details. FEATUTE: : allow disabling of `step` param attached to . This might be useful for using vmalert with datasources that to not support this param, unlike VictoriaMetrics. See for details. FEATUTE: : support option for "blackholing" alerting notifications if `-notifier.blackhole` cmd-line flag is set. Enable this flag if you want vmalert to evaluate alerting rules without sending any notifications to external receivers (eg. alertmanager). See for details. Thanks to @venkatbvc for . FEATURE: : add unit test for alerting and recording rules, see more here. Thanks to @Haleygo for . FEATURE: : allow overriding default GET params for rules with `graphite` datasource type, in the same way as it happens for `prometheus` type. See . FEATUTE: : support `keepfiringfor` field for alerting rules. See docs updated and . Thanks to @Haleygo for

FEATURE: : expose `vmauthuserrequestdurationseconds` and `vmauthunauthorizeduserrequestduration_seconds` summary metrics for measuring requests latency per user. FEATURE: : show backup progress percentage in log during backup uploading. See . FEATURE: : show restoring progress percentage in log during backup downloading. See . FEATURE: add ability to fine-tune Graphite API limits via the following command-line flags: `-search.maxGraphiteTagKeys` for limiting the number of tag keys returned from `-search.maxGraphiteTagValues` for limiting the number of tag values returned from `-search.maxGraphiteSeries` for limiting the number of series (aka paths) returned from See . BUGFIX: properly return series from if it finds more than the `limit` series (`limit` is an optional query arg passed to this API). Previously the `limit exceeded error` error was returned in this case. See . BUGFIX: : fix application routing issues and problems with manual URL changes. See and . BUGFIX: add validation for invalid in query and export APIs. BUGFIX: : interrupt explore procedure in influx mode if vmctl found no numeric fields. BUGFIX: : fix panic in case `--remote-read-filter-time-start` flag is not set for remote-read mode. This flag is now required to use remote-read mode. See . BUGFIX: : fix formatting issue, which could add superflouos `s` characters at the end of `samples/s` output during data migration. For example, it could write `samples/ssssss`. See . BUGFIX: : use RFC3339 time format in query args instead of unix timestamp for all issued queries to Prometheus-like datasources. BUGFIX: : correctly calculate evaluation time for rules. Before, there was a low probability for discrepancy between actual time and rules evaluation time if evaluation interval was lower than the execution time for rules within the group. BUGFIX: : reset evaluation timestamp after modifying group interval. Before, there could have latency on rule evaluation time. BUGFIX: vmselect: fix timestamp alignment for Prometheus querying API if time argument is less than 10m from the beginning of Unix epoch. BUGFIX: : close HTTP connections to servers when they are no longer needed. This should prevent from possible connection exhasution in some cases. See . BUGFIX: : do not show links at the `/targets` page when `vmagent` runs with `-promscrape.dropOriginalLabels` command-line flag, since it has no the original labels needed for relabel debug. See . BUGFIX: vminsert: fixed decoding of label values with slash when accepting data via . This fixes Prometheus golang client compatibility. See . BUGFIX: : properly parse binary operations with reserved words on the right side such as `foo + (on{bar="baz"})`. Previously such queries could lead to panic. See . BUGFIX: : display cache usage for all components on panel `Cache usage % by type` for cluster dashboard. Before, only vmstorage caches were shown. Released at 2023-06-30 SECURITY: upgrade Go builder from Go1.20.4 to Go1.20.5. See . BUGFIX: : fix possible panic at shutdown when is enabled. See for details. BUGFIX: : fixed service name detection for in case of a difference in service name and service id. See for details. BUGFIX: : retry all errors except 4XX status codes while pushing via remote-write to the remote storage. Previously, errors like broken connection could prevent vmalert from retrying the request. BUGFIX: : properly interrupt retry attempts on vmalert shutdown. Before, vmalert could have waited for all retries to finish for shutdown. BUGFIX: : fix an issue with `vmbackupmanager` not being able to restore data from a backup stored in GCS. See for details. BUGFIX: : properly return error from and at `vmselect` when the `-search.maxSamplesPerQuery` or `-search.maxSamplesPerSeries` is exceeded. Previously incomplete response could be returned without the error if `vmselect` runs with `-replicationFactor` greater than 1. See

BUGFIX: : prevent from possible crashloop after the migration from versions below `v1.90.0` to newer versions. See for details. BUGFIX: : fix a memory leak issue associated with chart updates. See . BUGFIX: : fix removing storage data dir before restoring from backup. BUGFIX: vmselect: wait for all vmstorage nodes to respond when the `-replicationFactor` flag is set bigger than > 1. Before, vmselect could have to respond. This could have resulted in issues illustrated . Now, this optimization is disabled by default and could be re-enabled by passing `-search.skipSlowReplicas` cmd-line flag to vmselect. See more details . Released at 2023-06-02 BUGFIX: : fix nil map assignment panic in runtime introduced in this . Released at 2023-06-01 FEATURE:: Adds `follow_redirects` at service discovery level of scrape configuration. See . Thanks to @Haleygo for . FEATURE: vmselect: Decreases startup time for vmselect with a big number of vmstorage nodes. See . Thanks to @Haleygo for . BUGFIX: : Properly form path to static assets in WEB UI if `http.pathPrefix` set. See . BUGFIX: : Properly set datasource query params. See . Thanks to @gsakun for . BUGFIX: : properly return empty slices instead of nil for `/api/v1/rules` for groups with present name but absent `rules`. See . BUGFIX: : Properly handle LOCAL command for proxy protocol. See . BUGFIX: : Fixes crash on startup. See . BUGFIX: : fix bug with custom URL in global settings not respecting tenantID change. See . Released at 2023-05-18 SECURITY: upgrade Go builder from Go1.20.3 to Go1.20.4. See . SECURITY: serve `/robots.txt` content to disallow indexing of the exposed instances by search engines. See for details. FEATURE: update to V2 in respect to V1 deprecation notice from June 2023. See . FEATURE: deprecate `-bigMergeConcurrency` command-line flag, since improper configuration for this flag frequently led to uncontrolled growth of unmerged parts, which, in turn, could lead to queries slowdown and increased CPU usage. The concurrency for can be controlled via `-smallMergeConcurrency` command-line flag, though it isn't recommended to change this flag in general case. FEATURE: do not execute the incoming request if it has been canceled by the client before the execution start. See . FEATURE: support time formats with timezones. For example, `2024-01-02+02:00` means `January 2, 2024` at `+02:00` time zone. See . FEATURE: expose `process_` metrics at `/metrics` page of all the VictoriaMetrics components under Windows OS. See . FEATURE: reduce the amounts of unimportant `INFO` logging during VictoriaMetrics startup / shutdown. This should improve visibility for potentially important logs. FEATURE: upgrade base docker image (alpine) from 3.17.3 to 3.18.0. See . FEATURE: : do not pollute logs with `cannot read hello: cannot read message with size 11: EOF` messages at `vmstorage` during TCP health checks performed by or . See . FEATURE: : support the ability to filter targets in more optimal way via new `filter` option. See . FEATURE: : add support for . See . FEATURE: : emit a warning if too small value is passed to `-remoteWrite.maxDiskUsagePerURL` command-line flag. See . FEATURE: : add support of recursive globs for `-rule` and `-rule.templates` command-line flags by using `*` in the glob pattern. See . FEATURE: : add ability to specify custom per-group HTTP headers sent to the configured notifiers. See . Thanks to @Haleygo for . FEATURE: : detect alerting rules which don't match any series. See and . FEATURE: : support loading rules via HTTP URL. See . Thanks to @Haleygo for the . FEATURE: : add buttons for filtering groups/rules with errors or with no-match warning in web UI for page `/groups`. See

FEATURE: : do not retry remote-write requests for responses with 4XX status codes. This aligns with . Thanks to @MichaHoffmann for . FEATURE: : add ability to filter incoming requests by IP. See and . FEATURE: : add ability to proxy requests to the specified backends for unauthorized users. See . FEATURE: : add ability to specify default route for unmatched requests. See . FEATURE: : retry `POST` requests on the remaining backends if the currently selected backend isn't reachable. See . FEATURE: : add ability to compare the data for the previous day with the data for the current day at . See . FEATURE: : display histograms as heatmaps in . See . FEATURE: : add `WITH template` playground. See . FEATURE: : add ability to debug relabeling. See . FEATURE: : add an ability to copy and execute queries listed at page. Also make more human readable the query duration column. See and . FEATURE: : increase default font size for better readability. FEATURE: : : return back a table with labels containing the highest number of unique label values. See . FEATURE: : add notification icon for queries that do not match any time series. A warning icon appears next to the query field when the executed query does not match any time series. See . FEATURE: : add `-s3StorageClass` command-line flag for setting the storage class for AWS S3 backups. See . Thanks to @justcompile for the . FEATURE: : store backup creation and completion time in `backup_complete.ignore` file of backup contents. This allows determining the exact timestamp when the backup was created and completed. FEATURE: : add `created_at` field to the output of `/api/v1/backups` API and `vmbackupmanager backup list` command. See this for data format details. FEATURE: : add commands for locking/unlocking backups against deletion by retention policy. See this for data format details. FEATURE: : add support for for `--vm-native-filter-time-start` and `--vm-native-filter-time-end` command-line flags. See . FEATURE: : set default value for `--vm-native-step-interval` command-line flag to `month`. This enables of data based on monthly step value when using native migration mode. See . BUGFIX: reduce the probability of sudden increase in the number of small parts on systems with small number of CPU cores. BUGFIX: reduce the possibility of increased CPU usage when data with timestamps older than one hour is ingested into VictoriaMetrics. This reduces spikes for the graph `sum(rate(vmslowperdayindexinsertstotal))`. See . BUGFIX: fix possible infinite loop during `indexdb` rotation when `-retentionTimezoneOffset` command-line flag is set and the local timezone is not UTC. See . Thanks to @faceair for . BUGFIX: do not panic at Windows during . Instead, delete the snapshot on the next restart. See for details. BUGFIX: change the max allowed value for `-memory.allowedPercent` from 100 to 200. See . BUGFIX: properly limit the number of concurrent requests specified via `-maxConcurrentInserts` command-line flag. See . Thanks to @zouxiang1993 for . BUGFIX: do not ignore trailing empty field in CSV lines when . See . BUGFIX: disallow `"` chars when parsing Prometheus label names, since they aren't allowed by . Previously this could result in silent incorrect parsing of incorrect Prometheus labels such as `foo{"bar"="baz"}` or `{foo:"bar",baz="aaa"}`. See . BUGFIX: : prevent from possible panic when the number of vmstorage nodes increases when is enabled. BUGFIX: : fix a panic when the duration in the query contains uppercase `M` suffix. Such a suffix isn't allowed to use in durations, since it clashes with `a million` suffix, e.g. it isn't clear whether `rate(metric and

BUGFIX: : properly handle the `vmpromscrapeconfiglastreload_successful` metric after config reload. See . BUGFIX: : add `metakubernetesendpoints_name` label for all ports discovered from endpoint. Previously, ports not matched by `Service` did not have this label. See for details. Thanks to @thunderbird86 for discovering and the issue. BUGFIX: : retry failed read request on the closed connection one more time. This improves rules execution reliability when connection between vmalert and datasource closes unexpectedly. BUGFIX: : properly display an error when using `query` function for templating value of `-external.alert.source` flag. See . BUGFIX: : properly return empty slices instead of nil for `/api/v1/rules` and `/api/v1/alerts` API handlers. See . BUGFIX: : do not return invalid auth credentials in http response by default, since it may be logged by client. See . BUGFIX: : fix the display of the tenant selector. See . BUGFIX: : fix UI freeze when the query returns non-histogram series alongside histogram series. BUGFIX: : fix the text display on buttons in Safari 16.4. BUGFIX: : update threshold for `TooHighMemoryUsage` alert from 90% to 80%, since 90% is too high for production environments. BUGFIX: : fix compatibility with Windows OS. See . BUGFIX: : fix performance issue when migrating data from VictoriaMetrics according to . Add the ability to speed up the data migration via `--vm-native-disable-retries` command-line flag. See . BUGFIX: : fix bug with duplicated labels during stream aggregation via single-node VictoriaMetrics. See . Released at 2023-04-06 Update note: this release contains backwards-incompatible change in storage data format, so the previous versions of VictoriaMetrics will exit with the `unexpected number of substrings in the part name` error when trying to run them on the data created by v1.90.0 or newer versions. The solution is to upgrade to v1.90.0 or newer releases SECURITY: upgrade base docker image (alpine) from 3.17.2 to 3.17.3. See . SECURITY: upgrade Go builder from Go1.20.2 to Go1.20.3. See . FEATURE: open source . This API allows using VictoriaMetrics as a drop-in replacement for Graphite at both data ingestion and querying sides and reducing infrastructure costs by up to 10x comparing to Graphite. See as an example. FEATURE: release Windows binaries for , , and . See , and issues. This release of VictoriaMetrics for Windows cannot delete due to Windows constraints. See for details. This issue should be resolved in future releases. FEATURE: log metrics with truncated labels if the length of label value in the ingested metric exceeds `-maxLabelValueLen`. This should simplify debugging for this case. FEATURE: : show target URL when debugging . This should simplify target relabel debugging a bit. See . FEATURE: : add support for when . This protocol allows saving egress network bandwidth costs when sending data from `vmagent` to `Kafka` located in another datacenter or availability zone. See . FEATURE: : add `-kafka.consumer.topic.concurrency` command-line flag. It controls the number of Kafka consumer workers to use by `vmagent`. It should eliminate the need to start multiple `vmagent` instances to improve data transfer rate. See . FEATURE: : add support for on `arm64` machines. See . FEATURE: : delete unused buffered data at `-remoteWrite.tmpDataPath` directory when there is no matching `-remoteWrite.url` to send this data to. See . FEATURE: : add the ability for hot reloading of configs. See and . FEATURE: check the contents of `-relabelConfig` and `-streamAggr.config` files additionally to `-promscrape.config` when single-node VictoriaMetrics runs with `-dryRun` command-line flag. This aligns the behaviour of single-node VictoriaMetrics with behaviour for `-dryRun` command-line flag. FEATURE: : automatically draw a heatmap graph when the query selects a single . This simplifies analyzing histograms. See

FEATURE: : add support for drag'n'drop and paste from clipboard in the "Trace analyzer" page. See . FEATURE: : hide messages longer than 3 lines in the trace. You can view the full message by clicking on the `show more` button. See . FEATURE: : add the ability to manually input date and time when selecting a time range. See . FEATURE: : updated usability and the search process in cardinality explorer. Made this process straightforward for user. See . FEATURE: : add the ability to collapse/expand the legend. See . FEATURE: : add tips for working with the graph and legend. See . FEATURE: : add `apply` and `cancel` buttons to settings popup. See . FEATURE: : automatically disable progress bar when TTY isn't available. See . FEATURE: : add `-configCheckInterval` command-line flag, which can be used for automatic re-reading the `-auth.config` file. See . BUGFIX: prevent from slow under high data ingestion rate. See . BUGFIX: : suppress parsing errors in case of `EOF`. Usually, the error is caused by health checks and is not a sign of an actual error. BUGFIX: : fix displaying errors for each query. See . BUGFIX: : fix snapshot not being deleted in case of error during backup. See . BUGFIX: : suppress `series after dedup` error message in logs when `-remoteWrite.streamAggr.dedupInterval` command-line flag is set at or when `-streamAggr.dedupInterval` command-line flag is set at . BUGFIX: allow using dashes and dots in environment variables names referred in config files via `%{ENV-VAR.SYNTAX}`. See and . BUGFIX: return back query performance scalability on hosts with big number of CPU cores. The scalability has been reduced in . See . BUGFIX: : properly convert to Prometheus histogram buckets when VictoriaMetrics histogram contain zero buckets. Previously these buckets were ignored, and this could lead to missing Prometheus histogram buckets after the conversion. Thanks to @zklapow for . BUGFIX: : fix CPU and memory usage spikes when files pointed by cannot be re-read. See . BUGFIX: prevent unexpected merges on start-up when `-storage.minFreeDiskSpaceBytes` is set. See . BUGFIX: properly support comma-separated filters inside . See . BUGFIX: verify response code when fetching configuration files via HTTP. See . BUGFIX: : replace empty labels with `""` instead of `"<no value>"` during templating, as Prometheus does. See . BUGFIX: : properly pass multiple filters from `--vm-native-filter-match` command-line flag to the data source. Previously filters from `--vm-native-filter-match` were only used to discover the metric names, and the metric names like `name="metricname"` has been taken into account, while the remaining filters were ignored. For example `--vm-native-src-addr={foo="bar",baz="abc"}` may found `metricname{foo="bar",baz="abc"}` and filter was treated as `--vm-native-src-addr={name="metrics_name"}`, e.g. `foo="bar",baz="abc"` filter was ignored. See . Released at 2023-03-12 BUGFIX: prevent from possible `cannot unmarshal timeseries from rollupResultCache` panic after the upgrade to . Released at 2023-03-12 Update note: this release can crash with `cannot unmarshal timeseries from rollupResultCache` panic after the upgrade from the previous releases. This issue can be fixed by removing caches stored on disk according to . Another option is to upgrade to . SECURITY: upgrade Go builder from Go1.20.1 to Go1.20.2. See . FEATURE: : increase the default value for `--remote-read-http-timeout` command-line option from 30s (30 seconds) to 5m (5 minutes). This reduces the probability of timeout errors when migrating big number of time series. See . FEATURE: : migrate series one-by-one in . This allows better tracking the migration progress and resuming the migration process from the last migrated time series. See and . FEATURE: : add `--vm-native-src-headers` and `--vm-native-dst-headers` command-line flags, which can be used for setting custom HTTP headers during

Thanks to @baconmania for . FEATURE: : add `--vm-native-src-bearer-token` and `--vm-native-dst-bearer-token` command-line flags, which can be used for setting Bearer token headers for the source and the destination storage during . See . FEATURE: : add `--vm-native-disable-http-keep-alive` command-line flag to allow `vmctl` to use non-persistent HTTP connections in . Thanks to @baconmania for . FEATURE: : log number of configration files found for each specified `-rule` command-line flag. FEATURE: : concurrently . This significantly improves config load speed for cases when there are thousands of files to read from the object storage. BUGFIX: vmstorage: fix a bug, which could lead to incomplete or empty results for heavy queries selecting tens of thousands of time series. See . BUGFIX: vmselect: reduce memory usage and CPU usage when performing heavy queries. See . BUGFIX: prevent from possible `invalid memory address or nil pointer dereference` panic during . The issue has been introduced at . See . BUGFIX: prevent from possible `SIGBUS` crash on ARM architectures (Raspberry Pi), which deny unaligned access to 8-byte words. Thanks to @oliverpool for narrowing down the issue and for . BUGFIX: : always return `is_partial: true` in partial responses. Previously partial responses could be returned as non-partial in some cases. BUGFIX: : properly take into account `-rpc.disableCompression` command-line flag at `vmstorage`. It was ignored since . See . BUGFIX: : fix panic when . The panic has been introduced in . BUGFIX: : stop showing `Please enter a valid Query and execute it` error message on the first load of vmui. BUGFIX: : properly process `Run in VMUI` button click in . BUGFIX: : fix the display of the selected value for dropdowns on `Explore` page. BUGFIX: : do not send `step` param for instant queries. See . BUGFIX: : fix `cannot serve http` panic when plain HTTP request is sent to `vmauth` configured to accept requests over -encoded request (e.g. when `vmauth` runs with `-httpListenAddr.useProxyProtocol` command-line flag). The issue has been introduced at when implementing . BUGFIX: : properly parse RSA public key discovered via JWK endpoint. Released at 2023-02-27 FEATURE: add `-snapshotCreateTimeout` flag to allow configuring timeout for . See . FEATURE: expose `vm_http_requests_total` and `vm_http_request_errors_total` metrics for `snapshot/` at `vmstorage` and . See . FEATURE: : add the ability to discover keys for JWT verification via . See . FEATURE: add `-internStringDisableCache` command-line flag for disabling the cache for . This flag may be useful in for reducing memory usage at the cost of higher CPU usage. FEATURE: add `-internStringCacheExpireDuration` command-line flag for controlling the lifetime of cached . BUGFIX: : fix panic when executing the query `aggr_func(rollup(some_value))`. The panic has been introduced in . BUGFIX: : use the provided `-remoteWrite.` auth options when determining whether the remote storage supports . Previously the auth options were ignored. This was preventing from automatic switch to VictoriaMetrics remote write protocol. BUGFIX: : do not register `vm_promscrape_config_` metrics if `-promscrape.config` flag is not used. Previously those metrics were registered and never updated, which was confusing and could trigger false-positive alerts. BUGFIX: : skip measurements with no fields when migrating data from influxdb. See . BUGFIX: delete failed snapshot contents from disk on failed attempt to . Previously failed snapshot contents could remain on disk in incomplete state. See Released at 2023-02-24 SECURITY: upgrade base docker image (alpine) from 3.17.1 to 3.17.2. See . SECURITY: upgrade Go builder from Go1.20.0 to Go1.20.1. See . FEATURE: : add support for . This protocol allows saving egress network bandwidth costs when sending data from `vmagent` to VictoriaMetrics located in another datacenter or availability

This also allows reducing disk IO under high load when `vmagent` starts queuing the collected data to disk when the remote storage is temporarily unavailable or cannot keep up with the data ingestion rate. See . FEATURE: : add support for Control Plane targets discovery aka . See . FEATURE: : add the ability to verify JWT signature via . See . FEATURE: : add the ability to limit the number of concurrent requests on a per-user basis via `-maxConcurrentPerUserRequests` command-line flag and via `maxconcurrentrequests` config option. See and . FEATURE: : automatically retry failing `GET` requests on all . Previously the backend error has been immediately returned to the client without retrying the request on the remaining backends. FEATURE: : choose the backend with the minimum number of concurrently executed requests in a round-robin manner for serving the incoming requests. This allows spreading the load among backends more evenly, while improving the response time. FEATURE: : add ability to read alerting and recording rules from S3, GCS or S3-compatible object storage. See . FEATURE: : automatically retry requests to remote storage if up to 5 errors occur during the data migration process. This should help continuing the data migration process on temporary errors. Previously `vmctl` was stopping after the first error. See . FEATURE: : support optional 2nd argument `min`, `max` or `avg` for , , , , and function. If the second argument is passed, then the function returns only the selected aggregation type. This change can be useful for situations where only one type of rollup calculation is needed. For example, `rolluprate(requeststotal for details. FEATURE: : support optional 2nd argument `open`, `low`, `high`, `close` for function. If the second argument is passed, then the function returns only the selected aggregation type. FEATURE: : add aggregate function. FEATURE: : add `madovertime(m over raw samples on the lookbehind window `d`. See . FEATURE: : add `range_mad(q)` function for calculating the over points per each time series returned by `q`. FEATURE: : add `range_zscore(q)` function for calculating over points per each time series returned from `q`. FEATURE: : add `range_trim_outliers(k, q)` function for dropping outliers located farther than `krangemad(q)` from the `rangemedian(q)`. This should help removing outliers during query time at . FEATURE: : add `range_trim_zscore(z, q)` function for dropping outliers located farther than `zrangestddev(q)` from `rangeavg(q)`. This should help removing outliers during query time at . FEATURE: : show `median` instead of `avg` in graph tooltip and line legend, since `median` is more tolerant against spikes. See . FEATURE: add `-search.maxSeriesPerAggrFunc` command-line flag, which can be used for limiting the number of time series can return in a single query. This flag can be useful for preventing OOMs when function is improperly used. FEATURE: : small UX improvements for mobile view. See and . FEATURE: add `-search.logQueryMemoryUsage` command-line flag for logging queries, which need more memory than specified by this command-line flag. See . Thanks to @michal-kralik for the idea and the intial implementation. FEATURE: allow setting zero value for `-search.latencyOffset` command-line flag. This may be needed in . Previously the minimum supported value for `-search.latencyOffset` command-line flag was `1s`. BUGFIX: : immediately cancel in-flight scrape requests during configuration reload when is disabled. Previously `vmagent` could wait for long time until all the in-flight requests are completed before reloading the configuration. This could significantly slow down configuration reload. See . BUGFIX: : do not wait for 2 seconds after the first unsuccessful attempt to scrape the target before performing the next

This should improve scrape speed when the target closes between scrapes. See and issues. BUGFIX: : fix inside . See . Thanks to @MattiasAng for the fix! BUGFIX: do not put auxiliary directories scheduled for removal into snapshots. This should prevent from `cannot create hard links from ...must-remove...` errors when making snapshots / backups. See . BUGFIX: prevent from possible data ingestion slowdown and query performance slowdown during on systems with small number of CPU cores (1 or 2 CPU cores). The issue has been introduced in when implementing . See also . BUGFIX: properly parse timestamps in milliseconds when . Previously timestamps in milliseconds were mistakenly multiplied by 1000. Thanks to @Droxenator for the . BUGFIX: : do not add extrapolated points outside the real points when using function. See . Released at 2023-12-10 v1.87.x is a line of . It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since release SECURITY: upgrade base docker image (Alpine) from 3.18.4 to 3.19.0. See . SECURITY: upgrade Go builder from Go1.21.4 to Go1.21.5. See . BUGFIX: : sanitize label names before sending the alert notification to Alertmanager. Before, vmalert would send notifications with labels containing characters not supported by Alertmanager validator, resulting into validation errors like `msg="Failed to validate alerts" err="invalid label set: invalid name "foo.bar"`. BUGFIX: properly escape `<` character in responses returned via endpoint. See . Released at 2023-11-14 v1.87.x is a line of . It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since release SECURITY: upgrade Go builder from Go1.21.3 to Go1.21.4. . BUGFIX: : properly apply with `regex`, which start and end with `.+` or `.` and which contain alternate sub-regexps. For example, `.+;|;.+` or `.foo|bar|baz.`. Previously such regexps were improperly parsed, which could result in undexpected relabeling results. See . BUGFIX: fix panic, which could occur when is enabled. See . BUGFIX: : log warning about switching to ReadOnly mode only on state change. Before, vmstorage would log this warning every 1s. See for details. Released at 2023-10-16 v1.87.x is a line of . It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since release SECURITY: upgrade Go builder from Go1.21.1 to Go1.21.3. See and . BUGFIX: : prevent from livelock when is called under high data ingestion. See . BUGFIX: : correctly return `null` instead of `Inf` in JSON query responses. See . BUGFIX: : fix ingestion via for opentsdbhttp. See . The bug has been introduced in . BUGFIX: : fix support of legacy DataDog agent, which adds trailing slashes to urls. See . Thanks to @maxb for spotting the issue. Released at 2023-09-10 v1.87.x is a line of . It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since release SECURITY: upgrade Go builder from Go1.21.0 to Go1.21.1. See . BUGFIX: : properly parse `/insert/multitenant/` urls, which have been broken since . See . BUGFIX: properly build production armv5 binaries for `GOARCH=arm`. This has been broken after the upgrading of Go builder to Go1.21.0. See . BUGFIX: : return `503 Service Unavailable` status code when are denied and some of `vmstorage` nodes are temporarily unavailable. Previously `422 Unprocessable Entiry` status code was mistakenly returned in this case, which could prevent from automatic recovery by re-sending the request to healthy cluster replica in another availability zone. BUGFIX: : fix the bug when Group's `params` fields with multiple values were overriding each other instead of adding up. The bug was introduced in starting from . See

Released at 2023-09-01 v1.87.x is a line of . It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since release BUGFIX: : fix Docker builds for old Docker releases. See . BUGFIX: vmselect: correctly handle requests with `/select/multitenant` prefix. Such requests must be rejected since vmselect does not support multitenancy endpoint. Previously, such requests were causing panic. See . BUGFIX: : properly check for at `vmstorage`. Previously it wasn't properly checked, which could lead to increased resource usage and data ingestion slowdown when some of `vmstorage` nodes are in read-only mode. See . BUGFIX: : properly close broken vmstorage connection during checks at `vmstorage`. Previously it wasn't properly closed, which prevents restoring `vmstorage` node from read-only mode. See . BUGFIX: : prevent from breaking `vmselect` -> `vmstorage` RPC communication when `vmstorage` returns an empty label name at `/api/v1/labels` request. See . BUGFIX: do not allow starting VictoriaMetrics components with improperly set boolean command-line flags in the form `-boolFlagName value`, since this leads to silent incomplete flags' parsing. This form should be replaced with `-boolFlagName=value`. See . BUGFIX: properly replace `:` chars in label names with `_` when `-usePromCompatibleNaming` command-line flag is passed to `vmagent`, `vminsert` or single-node VictoriaMetrics. This addresses . BUGFIX: : correctly check if specified `-dst` belongs to specified `-storageDataPath`. See . Released at 2023-08-12 v1.87.x is a line of . It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since release SECURITY: upgrade Go builder from Go1.20.4 to Go1.21.0. SECURITY: upgrade base docker image (Alpine) from 3.18.2 to 3.18.3. See . BUGFIX: vmselect: fix timestamp alignment for Prometheus querying API if time argument is less than 10m from the beginning of Unix epoch. BUGFIX: vminsert: fixed decoding of label values with slash when accepting data via . This fixes Prometheus golang client compatibility. See . BUGFIX: : properly validate scheme for `proxy_url` field at the scrape config. See for details. BUGFIX: : close HTTP connections to servers when they are no longer needed. This should prevent from possible connection exhasution in some cases. See . BUGFIX: : properly apply `if` filters during . Previously the `if` filter could improperly work. See and . BUGFIX: : fix possible panic at shutdown when is enabled. See for details. BUGFIX: : use local scrape timestamps for the scraped metrics unless `honor_timestamps: true` option is explicitly set at . This fixes gaps for metrics collected from or similar exporters, which export metrics with invalid timestamps. See and for details. BUGFIX: : Properly handle LOCAL command for proxy protocol. See . BUGFIX: : properly return error from and at `vmselect` when the `-search.maxSamplesPerQuery` or `-search.maxSamplesPerSeries` is exceeded. Previously incomplete response could be returned without the error if `vmselect` runs with `-replicationFactor` greater than 1. See . BUGFIX: : correctly calculate evaluation time for rules. Before, there was a low probability for discrepancy between actual time and rules evaluation time if evaluation interval was lower than the execution time for rules within the group. BUGFIX: : reset evaluation timestamp after modifying group interval. Before, there could have latency on rule evaluation time. BUGFIX: : Properly set datasource query params. See . Thanks to @gsakun for . BUGFIX: : Properly form path to static assets in WEB UI if `http.pathPrefix` set. See . BUGFIX: : properly return empty slices instead of nil for `/api/v1/rules` for groups with present name but absent `rules`. See . BUGFIX: : interrupt explore procedure in influx mode if vmctl found no numeric

BUGFIX: : fix panic in case `--remote-read-filter-time-start` flag is not set for remote-read mode. This flag is now required to use remote-read mode. See . Released at 2023-05-18 v1.87.x is a line of . It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since release SECURITY: upgrade Go builder from Go1.20.3 to Go1.20.4. See . SECURITY: upgrade base docker image (alpine) from 3.17.3 to 3.18.0. See . SECURITY: serve `/robots.txt` content to disallow indexing of the exposed instances by search engines. See for details. BUGFIX: reduce the probability of sudden increase in the number of small parts on systems with small number of CPU cores. BUGFIX: reduce the possibility of increased CPU usage when data with timestamps older than one hour is ingested into VictoriaMetrics. This reduces spikes for the graph `sum(rate(vmslowperdayindexinsertstotal))`. See . BUGFIX: do not ignore trailing empty field in CSV lines when . See . BUGFIX: disallow `"` chars when parsing Prometheus label names, since they aren't allowed by . Previously this could result in silent incorrect parsing of incorrect Prometheus labels such as `foo{"bar"="baz"}` or `{foo:"bar",baz="aaa"}`. See . BUGFIX: : fix a panic when the duration in the query contains uppercase `M` suffix. Such a suffix isn't allowed to use in durations, since it clashes with `a million` suffix, e.g. it isn't clear whether `rate(metric and issues. BUGFIX: : prevent from possible panic when the number of vmstorage nodes increases when is enabled. BUGFIX: properly limit the number of concurrent requests specified via `-maxConcurrentInserts` command-line flag. See . Thanks to @zouxiang1993 for . BUGFIX: : properly return empty slices instead of nil for `/api/v1/rules` and `/api/v1/alerts` API handlers. See . BUGFIX: : add `metakubernetesendpoints_name` label for all ports discovered from endpoint. Previously, ports not matched by `Service` did not have this label. See for details. Thanks to @thunderbird86 for discovering and the issue. BUGFIX: fix possible infinite loop during `indexdb` rotation when `-retentionTimezoneOffset` command-line flag is set and the local timezone is not UTC. See . Thanks to @faceair for . BUGFIX: : do not return invalid auth credentials in http response by default, since it may be logged by client. See . BUGFIX: : update threshold for `TooHighMemoryUsage` alert from 90% to 80%, since 90% is too high for production environments. BUGFIX: : properly handle the `vmpromscrapeconfiglastreload_successful` metric after config reload. See . BUGFIX: : fix bug with duplicated labels during stream aggregation via single-node VictoriaMetrics. See . BUGFIX: : suppress `series after dedup` error message in logs when `-remoteWrite.streamAggr.dedupInterval` command-line flag is set at or when `-streamAggr.dedupInterval` command-line flag is set at . Released at 2023-04-06 v1.87.x is a line of . It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since release SECURITY: upgrade base docker image (alpine) from 3.17.2 to 3.17.3. See . SECURITY: upgrade Go builder from Go1.20.2 to Go1.20.3. See . BUGFIX: : properly convert to Prometheus histogram buckets when VictoriaMetrics histogram contain zero buckets. Previously these buckets were ignored, and this could lead to missing Prometheus histogram buckets after the conversion. Thanks to @zklapow for . BUGFIX: : fix CPU and memory usage spikes when files pointed by cannot be re-read. See . BUGFIX: prevent unexpected merges on start-up when `-storage.minFreeDiskSpaceBytes` is set. See . BUGFIX: properly support comma-separated filters inside . See . BUGFIX: verify response code when fetching configuration files via HTTP. See . Released at 2023-03-25 v1.87.x is a line of . It contains important up-to-date bugfixes. The

line will be supported for at least 12 months since release BUGFIX: prevent from slow under high data ingestion rate. See . BUGFIX: : suppress parsing errors in case of `EOF`. Usually, the error is caused by health checks and is not a sign of an actual error. BUGFIX: : fix snapshot not being deleted in case of error during backup. See . BUGFIX: allow using dashes and dots in environment variables names referred in config files via `%{ENV-VAR.SYNTAX}`. See and . BUGFIX: return back query performance scalability on hosts with big number of CPU cores. The scalability has been reduced in . See . Released at 2023-03-12 v1.87.x is a line of . It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since release SECURITY: upgrade Go builder from Go1.20.1 to Go1.20.2. See . BUGFIX: vmstorage: fix a bug, which could lead to incomplete or empty results for heavy queries selecting tens of thousands of time series. See . BUGFIX: vmselect: reduce memory usage and CPU usage when performing heavy queries. See . BUGFIX: prevent from possible `invalid memory address or nil pointer dereference` panic during . The issue has been introduced at . See . BUGFIX: prevent from possible `SIGBUS` crash on ARM architectures (Raspberry Pi), which deny unaligned access to 8-byte words. Thanks to @oliverpool for narrowing down the issue and for . BUGFIX: : always return `is_partial: true` in partial responses. Previously partial responses could be returned as non-partial in some cases. BUGFIX: : properly take into account `-rpc.disableCompression` command-line flag at `vmstorage`. It was ignored since . See . BUGFIX: : do not register `vm_promscrape_config_` metrics if `-promscrape.config` flag is not used. Previously those metrics were registered and never updated, which was confusing and could trigger false-positive alerts. BUGFIX: : skip measurements with no fields when migrating data from influxdb. See . BUGFIX: : fix `cannot serve http` panic when plain HTTP request is sent to `vmauth` configured to accept requests over -encoded request (e.g. when `vmauth` runs with `-httpListenAddr.useProxyProtocol` command-line flag). The issue has been introduced at when implementing . Released at 2023-02-24 v1.87.x is a line of . It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since release SECURITY: upgrade base docker image (alpine) from 3.17.1 to 3.17.2. See . SECURITY: upgrade Go builder from Go1.20.0 to Go1.20.1. See . BUGFIX: : immediately cancel in-flight scrape requests during configuration reload when is disabled. Previously `vmagent` could wait for long time until all the in-flight requests are completed before reloading the configuration. This could significantly slow down configuration reload. See . BUGFIX: : do not wait for 2 seconds after the first unsuccessful attempt to scrape the target before performing the next attempt. This should improve scrape speed when the target closes between scrapes. See and issues. BUGFIX: : fix inside . See . Thanks to @MattiasAng for the fix! BUGFIX: do not put auxiliary directories scheduled for removal into snapshots. This should prevent from `cannot create hard links from ...must-remove...` errors when making snapshots / backups. See . BUGFIX: prevent from possible data ingestion slowdown and query performance slowdown during on systems with small number of CPU cores (1 or 2 CPU cores). The issue has been introduced in when implementing . See also . BUGFIX: properly parse timestamps in milliseconds when . Previously timestamps in milliseconds were mistakenly multiplied by 1000. Thanks to @Droxenator for the . BUGFIX: : do not add extrapolated points outside the real points when using function. See . Released at 2023-02-09

is a line of . It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since release FEATURE: : alerts state restore procedure was changed to become asynchronous. It doesn't block groups start anymore which significantly improves vmalert's startup time. This also means that `-remoteRead.ignoreRestoreErrors` command-line flag becomes deprecated now and will have no effect if configured. While previously state restore attempt was made for all the loaded alerting rules, now it is called only for alerts which became active after the first evaluation. See . FEATURE: : optimize VMUI for use from smartphones and tablets. See . FEATURE: : add ability to search tenants in the drop-down list for the tenant selector. See . FEATURE: : add avg/min/max/last values to line legends and tooltips for graphs. See . FEATURE: : hide the default `per-job resource usage` dashboard if there is a custom dashboard exists at the directory specified via `-vmui.customDashboardsPath` command-line flag. See . BUGFIX: : fix panic in . Thanks to @mr-karan for the . BUGFIX: : fix display of rules number per-group for groups with identical names in UI. BUGFIX: : prevent disabling state updates tracking per rule via setting values < 1. The minimum number of update states to track is now set to 1. BUGFIX: : properly update `debug` and `updateentrieslimit` rule's params on config's hot-reload. BUGFIX: properly initialize the `vmconcurrentinsertcurrent` metric before exposing it. Previously this metric could be left uninitialized in some cases, e.g. its value was zero. This could lead to false alerts for the query `avgovertime(vmconcurrentinsertcurrent. BUGFIX: : immediately cancel in-flight scrape requests during configuration reload when using . Previously `vmagent` could wait for long time until all the in-flight requests are completed before reloading the configuration. This could significantly slow down configuration reload. See . BUGFIX: : do not validate JWT signature if no public keys are provided. Previously this could result in the `error setting up jwt verification` error. Released at 2023-02-01 v1.87.x is a line of . It contains important up-to-date bugfixes. The v1.87.x line will be supported for at least 12 months since release FEATURE: : add the ability to input samples before aggregation via `-streamAggr.dedupInterval` and `-remoteWrite.streamAggr.dedupInterval` command-line options. FEATURE: : add dark mode - it can be selected via `settings` menu in the top right corner. See . FEATURE: : improve visual appearance of the top menu. See . FEATURE: : embed fonts into binary instead of loading them from external sources. This allows using `vmui` in full from isolated networks without access to Internet. Thanks to @ScottKevill for . FEATURE: : add ability to switch between tenants by selecting the needed tenant in the drop-down list at the top right corner of the UI. See . FEATURE: : reduce memory usage when sending stale markers for targets, which expose big number of metrics. See and issues. FEATURE: : add `metakubernetespodcontainerid` meta-label to the targets discovered via . This label has been added in Prometheus starting from `v2.42.0`. See . FEATURE: : add `metaazuremachine_size` meta-label to the targets discovered via . This label has been added in Prometheus starting from `v2.42.0`. See . FEATURE: : allow limiting the number of concurrent requests sent to `vmauth` via `-maxConcurrentRequests` command-line flag. This allows controlling memory usage of `vmauth` and the resource usage of backends behind `vmauth`. See . Thanks to @dmitryk-dk for . FEATURE: allow using VictoriaMetrics components behind proxies, which communicate with the backend via . See . For example, accepts proxy protocol connections when it starts with `-httpListenAddr.useProxyProtocol` command-line

FEATURE: add `-internStringMaxLen` command-line flag, which can be used for fine-tuning RAM vs CPU usage in certain workloads. For example, if the stored time series contain long labels, then it may be useful reducing the `-internStringMaxLen` in order to reduce memory usage at the cost of increased CPU usage. See . FEATURE: provide GOARCH=386 binaries for single-node VictoriaMetrics, vmagent, vmalert, vmauth, vmbackup and vmrestore components at . See . Thanks to @denisgolius for . BUGFIX: fix a bug, which could prevent background merges for the previous partitions until restart if the storage didn't have enough disk space for final deduplication and down-sampling. BUGFIX: fix a bug, which could lead to increased CPU usage and disk IO usage when adding data to previous months and when the or is enabled. See . BUGFIX: : propagate all the timeout-related errors from `vmstorage` to `vmselect`. Previously some timeout errors weren't returned from `vmselect` to `vmstorage`. Instead, `vmstorage` could log the error and close the connection to `vmselect`, so `vmselect` was logging cryptic errors such as `cannot execute funcName="..." on vmstorage "...": EOF`. BUGFIX: : add support for time zone selection for older versions of browsers. See . BUGFIX: : update API version for to fix with missing `metaec2availabilityzoneid` attribute. BUGFIX: : properly return `200 OK` HTTP status code when importing data via . See . BUGFIX: : do not add `exported` prefix to scraped metric names, which clash with the if `honorlabels: true` option is set in the . See the and issues. BUGFIX: : allow re-entering authorization info in the web browser if the entered info was incorrect. Previously it was non-trivial to do via the web browser, since `vmauth` was returning `400 Bad Request` instead of `401 Unauthorized` http response code. BUGFIX: : always log the client address and the requested URL on proxying errors. Previously some errors could miss this information. BUGFIX: : fix snapshot not being deleted after backup completion. This issue could result in unnecessary snapshots being stored, it is required to delete unnecessary snapshots manually. See the . BUGFIX: : fix panic on top-level vmselect nodes of when the `-replicationFactor` flag is set and request contains `trace` query parameter. See . Released at 2023-01-18 SECURITY: : do not expose basic auth passwords from `-snapshot.createURL` and `-snapshot.deleteURL` command-line flags in logs. Thanks to @toanju for the . FEATURE: : add ability to show custom dashboards at vmui by specifying a path to a directory with dashboard config files via `-vmui.customDashboardsPath` command-line flag. See and . FEATURE: : apply the `step` globally to all the displayed graphs. See . FEATURE: : improve the appearance of graph lines by using more visually distinct colors. See . BUGFIX: do not slow down concurrently executed queries during assisted merges, since assisted merges already prioritize data ingestion over queries. The probability of assisted merges has been increased starting from because of internal refactoring. This could result in slowed down queries when there is a plenty of free CPU resources. See and issues. BUGFIX: reduce the increased CPU usage at `vmselect` to v1.85.3 level when processing heavy queries. See . BUGFIX: : fix `FATAL: cannot locate metric name for metricID=...: EOF` panic, which could occur when retention filters are enabled. BUGFIX: : properly cancel in-flight service discovery requests for and when the service list changes. See . BUGFIX: : : apply `filters` only to objects of the specified `role`. Previously filters were applied to all the objects, which could cause errors when different types of objects were used with filters that were not compatible with them. See

BUGFIX: : suppress all the scrape errors when `-promscrape.suppressScrapeErrors` is enabled. Previously some scrape errors were logged even if `-promscrape.suppressScrapeErrors` flag was set. BUGFIX: : consistently put the scrape url with scrape target labels to all error logs for failed scrapes. Previously some failed scrapes were logged without this information. BUGFIX: : do not send stale markers to remote storage for series exceeding the configured . See . BUGFIX: : properly apply when is disabled. BUGFIX: : reduce memory usage spikes when big number of scrape targets disappear at once. See . Thanks to @lzfhust for . BUGFIX: : properly return `200 OK` HTTP response code. See . BUGFIX: : properly parse `M` and `Mi` suffixes as `1e6` multipliers in `1M` and `1Mi` numeric constants. See . The issue has been introduced in . BUGFIX: : properly display range query results at `Table` view. For example, `up. Released at 2023-01-10 BUGFIX: return correct query results over time series with gaps. The issue has been introduced in . BUGFIX: properly take into account the timeout passed by `vmselect` to `vmstorage` during query execution. This issue could result in the following error logs at `vmstorage` under load: `cannot process vmselect request: cannot execute "search_v7": couldn't start executing the request in 0.000 seconds, since -search.maxConcurrentRequests=... concurrent requests are already executed`. The issue has been introduced in . Released at 2023-01-10 It is recommended upgrading to because v1.86.0 contains a bug, which could lead to incorrect query results over time series with gaps. Update note 1: This release changes the logic behind `-maxConcurrentInserts` command-line flag. Previously this flag was limiting the number of concurrent connections established from clients, which send data to VictoriaMetrics. Some of these connections could be temporarily idle. Such connections do not take significant CPU and memory resources, so there is no need in limiting their count. The new logic takes into account only those connections, which actively ingest new data to VictoriaMetrics and to . This means that the default `-maxConcurrentInserts` value should handle cases, which could require increasing the value in the previous releases. So it is recommended trying to remove the explicitly set `-maxConcurrentInserts` command-line flag after upgrading to this release and verifying whether this reduces CPU and memory usage. Update note 2: The `vmconcurrentaddrowscurrent` and `vmconcurrentaddrowscapacity` metrics by `vmstorage` are replaced with `vmconcurrentinsertcurrent` and `vmconcurrentinsertcapacity` metrics in order to be consistent with the corresponding metrics exported by `vminsert`. Please update queries in dahsboards and alerting rules with new metric names if old metric names are used there. FEATURE: : add support for aggregation of incoming by time and by labels. See and . FEATURE: : reduce memory usage when scraping big number of targets without the need to enable . FEATURE: : add support for Prometheus-compatible target discovery for services via . See . Thanks to @mr-karan for . FEATURE: : automatically pre-fetch `metricrelabelconfigs` and the target labels when clicking on the `debug metrics relabeling` link at the `http://vmagent:8429/targets` page at the particular target. See . FEATURE: : add ability to explore metrics exported by a particular `job` / `instance`. See and . FEATURE: allow passing partial `RFC3339` date/time to `time`, `start` and `end` query args at and . For example, `2022` is equivalent to `2022-01-01T00:00:00Z`, while `2022-01-30T14` is equivalent to `2022-01-30T14:00:00Z`. See . FEATURE: : allow using unicode letters in identifiers. For example, `{=""}` is a valid MetricsQL expression now. Previously every non-ascii letters should be escaped with `\` char when used inside MetricsQL expression: `\\\\\\\\\\\{\\\\\=""}`. Now both expressions are equivalent. Thanks to @hzwwww for the

FEATURE: : add support for `keepequal` and `dropequal` relabeling actions, which are supported by Prometheus starting from . These relabeling actions are almost identical to `keepifequal` and `dropifequal` relabeling actions supported by VictoriaMetrics since `v1.38.0` - see - so it is recommended sticking to `keepifequal` and `dropifequal` actions instead of switching to `keepequal` and `dropequal`. FEATURE: : support empty values for imported metrics. See . FEATURE: : allow configuring the default number of stored rule's update states in memory via global `-rule.updateEntriesLimit` command-line flag or per-rule via rule's `updateentrieslimit` configuration param. See and . FEATURE: improve the logic benhind `-maxConcurrentInserts` command-line flag. Previously this flag was limiting the number of concurrent connections from clients, which write data to VictoriaMetrics or . Some of these connections could be idle for some time. These connections do not need significant amounts of CPU and memory, so there is no sense in limiting their count. The updated logic behind `-maxConcurrentInserts` limits the number of active* insert requests, not counting idle connections. FEATURE: protect all the http endpoints with `-httpAuth.` command-line flag. Previously endpoints protected by `-AuthKey` command-line flags weren't protected by `-httpAuth.`. This could complicate the proper security setup. See . FEATURE: : add `-maxConcurrentInserts` and `-insert.maxQueueDuration` command-line flags to `vmstorage`, so they could be tuned if needed in the same way as at `vminsert` nodes. FEATURE: : limit the number of concurrently executed requests at `vmstorage` proportionally to the number of available CPU cores, since every request can saturate a single CPU core at `vmstorage`. Previously a single `vmstorage` could accept and start processing arbitrary number of concurrent requests received from big number of `vmselect` nodes. This could result in increased RAM, CPU and disk IO usage or event to out of memory crash at `vmstorage` side under high load. The limit can be fine-tuned if needed via `-search.maxConcurrentRequests` command-line flag at `vmstorage` according to . `vmstorage` now the following additional metrics at `http://vmstorage:8482/metrics` page: `vmvmselectconcurrentrequestscapacity` - the maximum number of requests allowed to execute concurrently `vmvmselectconcurrentrequestscurrent` - the current number of concurrently executed requests `vmvmselectconcurrentrequestslimitreachedtotal` - the total number of requests, which were put in the wait queue when `-search.maxConcurrentRequests` concurrent requests are being executed `vmvmselectconcurrentrequestslimittimeouttotal` - the total number of canceled requests because they were sitting in the wait queue for more than `-search.maxQueueDuration` BUGFIX: : properly update the `step` value in url after the `step` input field has been manually changed. This allows preserving the proper `step` when copy-n-pasting the url to another instance of web browser. See . BUGFIX: : properly update tooltip when quickly hovering multiple lines on the graph. See . BUGFIX: properly parse floating-point numbers without integer or fractional parts such as `.123` and `20.` during . See . BUGFIX: : properly parse durations with uppercase suffixes such as `10S`, `5MS`, `1W`, etc. See . BUGFIX: : fix a panic during target discovery when `vmagent` runs with `-promscrape.dropOriginalLabels` command-line flag. See . The bug has been introduced in . BUGFIX: : : properly encode `filters` field. See . BUGFIX: : fix possible resource leak after hot reload of the updated . See . BUGFIX: : fix a panic in when the discovered instance has zero labels. See . The issue has been introduced in . BUGFIX: properly return label names starting from uppercase such as `CamelCaseLabel` from . See . BUGFIX: fix `opentsdb` HTTP endpoint not respecting `-httpAuth.` flags. See BUGFIX: consistently select the sample with the biggest value out of samples with identical timestamps during querying when the is enabled according to . Previously random samples could be selected during querying. See changes for older