Project: VictoriaMetrics (Observability and Analysis / Observability)
File: CHANGELOG_2020.md

sort: 101 weight: 101 title: CHANGELOG for the year 2020 menu: docs: parent: 'victoriametrics' weight: 101 aliases: /CHANGELOG_2020.html Released at 2020-12-27 FEATURE: add `/api/v1/status/top_queries` handler, which returns the most frequently executed queries and queries that took the most time for execution. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/907> FEATURE: vmagent: add support for `proxy_url` config option in Prometheus scrape configs. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/503> FEATURE: remove parts with stale data as soon as they go outside the configured `-retentionPeriod`. Previously such parts may remain active for long periods of time. This should help reducing disk usage for `-retentionPeriod` smaller than one month. FEATURE: vmalert: allow setting multiple values for `-notifier.tlsInsecureSkipVerify` command-line flag per each `-notifier.url`. BUGFIX: vmalert: properly escape multiline queries when passing them to Grafana. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890> BUGFIX: vmagent: set missing `__meta_kubernetes_service_` labels in `kubernetessdconfig` for `endpoints` and `endpointslices` roles. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/982> BUGFIX: do not adjust `offset` value provided in MetricsQL query. Previously it could be modified in order to improve response cache hit ratio. This is unneeded, since cache hit ratio should remain good because the query time range should be already aligned to multiple of `step` values. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/976> Released at 2020-12-19 FEATURE: do not publish duplicate Docker images with `-cluster` tag suffix for , , , and , since they are identical to images without `-cluster` tag suffix. BUGFIX: vmalert: properly populate template variables. This has been broken in v1.50.0. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/974> BUGFIX: properly parse negative combined duration in MetricsQL such as `-1h3m4s`. It must be parsed as `-(1h + 3m + 4s)`. Prevsiously it was parsed as `-1h + 3m + 4s`. BUGFIX: properly parse lines in and in with whitespace after the timestamp. For example, `foo 123 456 ## some comment here`. See <https://github.com/VictoriaMetrics/VictoriaMetrics/pull/970> Released at 2020-12-15 FEATURE: vmagent: export `vmagentremotewriteblockssenttotal` and `vmagentremotewriteblockssenttotal` metrics for each `-remoteWrite.url`. BUGFIX: vmagent: properly delete unregistered scrape targets from `/targets` and `/api/v1/targets` pages. They weren't deleted due to the bug in `v1.50.0`. Released at 2020-12-15 FEATURE: automatically reset response cache when samples with timestamps older than `now - search.cacheTimestampOffset` are ingested to VictoriaMetrics. This makes unnecessary disabling response cache during data backfilling or resetting it after backfilling is complete as described . This feature applies only to single-node VictoriaMetrics. It doesn't apply to cluster version of VictoriaMetrics because `vminsert` nodes don't know about `vmselect` nodes where the response cache must be reset. FEATURE: vmalert: add `query`, `first` and `value` functions to alert templates. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/539> FEATURE: vmagent: return user-friendly HTML page when requesting `/targets` page from web browser. The page is returned in the old plaintext format when requesting via curl or similar tool. FEATURE: allow multiple whitespace chars between measurements, fields and timestamp when parsing InfluxDB line protocol. Though denies multiple whitespace chars between these entities, some apps improperly put multiple whitespace chars. This workaround allows accepting data from such apps. FEATURE: export `vmpromscrapeactivescrapers{type="<sdtype>"}` metric for tracking the number of active scrapers per each service discovery type. FEATURE: export `vmpromscrapescrapersstartedtotal{type="<sdtype>"}` and `vmpromscrapescrapersstoppedtotal{type="<sdtype>"}` metrics for tracking churn rate for scrapers per each service discovery type. FEATURE: vmagent: allow setting per-`-remoteWrite.url` command-line flags for `-remoteWrite.sendTimeout` and `-remoteWrite.tlsInsecureSkipVerify`. BUGFIX: properly handle `` and `[...]` inside curly braces in query passed to Graphite Metrics API. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/952> BUGFIX: vmagent: fix memory leak when big number of targets is discovered via service discovery. BUGFIX: vmagent: properly pass `datacenter` filter to Consul API server. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574#issuecomment-740454170> BUGFIX: properly handle CPU limits set on the host system or host

The bugfix may result in lower memory usage on systems with CPU limits. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946> BUGFIX: prevent from duplicate `name` tag returned from `/tags/autoComplete/tags` handler. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/942> BUGFIX: do not enable strict parsing for `-promscrape.config` if `-promscrape.config.dryRun` comand-line flag is set. Strict parsing can be enabled with `-promscrape.config.strictParse` command-line flag. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/944> BUGFIX: vminsert: properly update `vmrpcreroutedrowsprocessed_total` metric. Previously it wasn't updated. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/955> BUGFIX: vmagent: properly recover when opening incorrectly stored persistent queue. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/964> BUGFIX: vmagent: properly handle scrape errors when stream parsing is enabled with `-promscrape.streamParse` command-line flag or with `stream_parse: true` per-target config option. Previously such errors weren't reported at `/targets` page. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/967> BUGFIX: assume the previous value is 0 when calculating `increase()` for the first point on the graph if its value doesn't exceed 100 and the delta between two first points equals to 0. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962> Released at 2020-12-05 FEATURE: optimize Consul service discovery speed when discovering big number of services. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574> FEATURE: add `labeluppercase(q, label1, ... labelN)` and `labellowercase(q, label1, ... labelN)` function to for uppercasing and lowercasing values for the given labels. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/936> FEATURE: add `counteqovertime(m[d], N)` and `countneovertime(m[d], N)` for counting the number of samples for `m` over `d` that (equal / not equal) to `N`. FEATURE: do not print usage info for all the command-line flags when incorrect command-line flag is passed. Previously it could be hard reading the error message about incorrect command-line flag because of too big usage info for all the flags. FEATURE: upgrade Go builder from v1.15.5 to v1.15.6 . This fixes . BUGFIX: properly parse timestamps in OpenMetrics format - they are exposed as floating-point number in seconds instead of integer milliseconds unlike in Prometheus exposition format. See . BUGFIX: return `nan` for `a >bool b` query when `a` equals to `nan` like Prometheus does. Previously `0` was returned in this case. This applies to any comparison operation with `bool` modifier. See for details. BUGFIX: properly parse hex numbers in MetricsQL. Previously hex numbers with non-decimal digits such as `0x3b` couldn't be parsed. BUGFIX: handle `time() cmpop metric` like Prometheus does - i.e. return `metric` value if `cmpop` comparison is true. Previously `time()` value was returned. BUGFIX: return `nan` for `minute(m)` query when `m` equals to `nan` like Prometheus does. This applies to all the time-related functions such as `dayofmonth`, `dayofweek`, `daysinmonth`, `hour`, `month` and `year`. Released at 2020-11-26 FEATURE: added . This simplifies installation under Ubuntu to a single command: ```sh snap install victoriametrics ``` FEATURE: vmselect: add `-replicationFactor` command-line flag for reducing query duration when replication is enabled and a part of vmstorage nodes are temporarily slow and/or temporarily unavailable. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711> FEATURE: vminsert: export `vmrpcvmstorageisreachable` metric, which can be used for monitoring reachability of vmstorage nodes from vminsert nodes. FEATURE: vmagent: add service discovery (aka ). See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/851> FEATURE: add `filters` option to `dockerswarmsdconfig` like Prometheus did in v2.23.0 - see <https://github.com/prometheus/prometheus/pull/8074> FEATURE: expose `metaec2ipv6addresses` label for `ec2sd_config` like Prometheus will do in the next release. FEATURE: add `-loggerWarnsPerSecondLimit` command-line flag for rate limiting of WARN messages in logs. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/905> FEATURE: apply `loggerErrorsPerSecondLimit` and `-loggerWarnsPerSecondLimit` rate limit per caller. I.e. log messages are suppressed if the same caller logs the same message at the rate exceeding the given limit. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/905#issuecomment-729395855> FEATURE: add remoteAddr to slow query log in order to simplify identifying the client that sends slow queries to VictoriaMetrics. Slow query logging is controlled with `-search.logSlowQueryDuration` command-line flag. FEATURE: add `/tags/delSeries` handler from Graphite Tags

See <https://docs.victoriametrics.com/#graphite-tags-api-usage> FEATURE: log metric name plus all its labels when the metric timestamp is out of the configured retention. This should simplify detecting the source of metrics with unexpected timestamps. FEATURE: add `-dryRun` command-line flag to single-node VictoriaMetrics in order to check config file pointed by `-promscrape.config`. BUGFIX: properly parse Prometheus metrics with such as `foo 123 ## {bar="baz"} 1`. BUGFIX: properly parse "infinity" values in . See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/924> Released at 2020-11-16 FEATURE: vmselect: return the original error from `vmstorage` node in query response if `-search.denyPartialResponse` is set. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/891> FEATURE: vmselect: add `"isPartial":{true|false}` field in JSON output for `/api/v1/` functions from . `"isPartial":true` is set if the response contains partial data because of a part of `vmstorage` nodes were unavailable during query processing. FEATURE: improve performance for `/api/v1/series`, `/api/v1/labels` and `/api/v1/label/<labelName>/values` on time ranges exceeding one day. FEATURE: vmagent: reduce memory usage when service discovery detects big number of scrape targets and the set of discovered targets changes over time. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825> FEATURE: vmagent: add `-promscrape.dropOriginalLabels` command-line option, which can be used for reducing memory usage when scraping big number of targets. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825#issuecomment-724308361> for details. FEATURE: vmalert: explicitly set extra labels to alert entities. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/870> FEATURE: add `-search.treatDotsAsIsInRegexps` command-line flag, which can be used for automatic escaping of dots in regexp label filters used in queries. For example, if `-search.treatDotsAsIsInRegexps` is set, then the query `foo{bar=~"aaa.bb.cc|dd.eee"}` is automatically converted to `foo{bar=~"aaa\\.bb\\.cc|dd\\.eee"}`. This may be useful for querying Graphite data. FEATURE: consistently return text-based HTTP responses such as `plain/text` and `application/json` with `charset=utf-8`. See <https://github.com/VictoriaMetrics/VictoriaMetrics/pull/897> FEATURE: update Go builder from v1.15.4 to v1.15.5. This should fix . FEATURE: added `/internal/force_flush` http handler for flushing recently ingested data from in-memory buffers to persistent storage. See for more details. FEATURE: added support. See for details. BUGFIX: do not return data points in the end of the selected time range for time series ending in the middle of the selected time range. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/887> and <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845> BUGFIX: remove spikes at the end of time series gaps for `increase()` or `delta()` functions. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/894> BUGFIX: vminsert: properly return HTTP 503 status code when all the vmstorage nodes are unavailable. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/896> Released at 2020-11-07 FEATURE: optimize requests to `/api/v1/labels` and `/api/v1/label/<name>/values` when `start` and `end` args are set. FEATURE: reduce memory usage when query touches big number of time series. FEATURE: vmagent: reduce memory usage when `kubernetessdconfig` discovers big number of scrape targets (e.g. hundreds of thousands) and the majority of these targets (99%) are dropped during relabeling. Previously labels for all the dropped targets were displayed at `/api/v1/targets` page. Now only up to `-promscrape.maxDroppedTargets` such targets are displayed. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878> for details. FEATURE: vmagent: reduce memory usage when scraping big number of targets with big number of temporary labels starting with ``. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825> FEATURE: vmagent: add `/ready` HTTP endpoint, which returns 200 OK status code when all the service discovery has been initialized. This may be useful during rolling upgrades. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/875> BUGFIX: vmagent: eliminate data race when `-promscrape.streamParse` command-line is set. Previously this mode could result in scraped metrics with garbage labels. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825#issuecomment-723198247> for details. BUGFIX: properly calculate `topk_` and `bottomk_*` functions from for time series with gaps. See <https://github.com/VictoriaMetrics/VictoriaMetrics/pull/883> Released at 2020-11-02 FEATURE: allow setting `-retentionPeriod` smaller than one month. I.e. `-retentionPeriod=3d`, `-retentionPeriod=2w`, etc. is supported now. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/173> FEATURE: optimize more cases according to <https://utcc.utoronto.ca/~cks/space/blog/sysadmin/PrometheusLabelNonOptimization>

Now the following cases are optimized too: `rollupfunc(foo{filters}[d]) op bar` -> `rollupfunc(foo{filters}[d]) op bar{filters}` `transformfunc(foo{filters}) op bar` -> `transformfunc(foo{filters}) op bar{filters}` `numorscalar op foo{filters} op bar` -> `numorscalar op foo{filters} op bar{filters}` FEATURE: improve time series search for queries with multiple label filters. I.e. `foo{label1="value", label2=~"regexp"}`. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/781> FEATURE: vmagent: add `stream parse` mode. This mode allows reducing memory usage when individual scrape targets expose tens of millions of metrics. For example, during scraping Prometheus in mode. See `-promscrape.streamParse` command-line option and `streamparse: true` config option for `scrapeconfig` section in `-promscrape.config`. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825> and . FEATURE: vmalert: add `-dryRun` command-line option for validating the provided config files without the need to start `vmalert` service. FEATURE: accept optional third argument of string type at `topk_` and `bottomk_*` functions. This is label name for additional time series to return with the sum of time series outside top/bottom K. See for more details. FEATURE: vmagent: expose `/api/v1/targets` page according to . See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/643> BUGFIX: vmagent: properly handle OpenStack endpoint ending with `v3.0` such as `https://ostack.example.com:5000/v3.0` in the same way as Prometheus does. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728#issuecomment-709914803> BUGFIX: drop trailing data points for time series with a single raw sample. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748> BUGFIX: do not drop trailing data points for instant queries to `/api/v1/query`. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845> BUGFIX: vmbackup: fix panic when `-origin` isn't specified. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/856> BUGFIX: vmalert: skip automatically added labels on alerts restore. Label `alertgroup` was introduced in and automatically added to generated time series. By mistake, this new label wasn't correctly purged on restore event and affected alert's ID uniqueness. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/870> BUGFIX: vmagent: fix panic at scrape error body formating. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/864> BUGFIX: vmagent: add leading missing slash to metrics path like Prometheus does. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/835> BUGFIX: vmagent: drop packet if remote storage returns 4xx status code. This make the behaviour consistent with Prometheus. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/873> BUGFIX: vmagent: properly handle 301 redirects. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/869> Released at 2020-10-13 FEATURE: automatically add missing label filters to binary operands as described at <https://utcc.utoronto.ca/~cks/space/blog/sysadmin/PrometheusLabelNonOptimization> . This should improve performance for queries with missing label filters in binary operands. For example, the following query should work faster now, because it shouldn't fetch and discard time series for `nodefilesystemfiles_free` metric without matching labels for the left side of the expression: ``` nodefilesystemfiles{ host="$host", mountpoint="/" } - nodefilesystemfiles_free ``` FEATURE: vmagent: add Docker Swarm service discovery (aka ). See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/656> FEATURE: add ability to export data in CSV format. See for details. FEATURE: vmagent: add `-promscrape.suppressDuplicateScrapeTargetErrors` command-line flag for suppressing `duplicate scrape target` errors. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/651> and <https://docs.victoriametrics.com/vmagent/#troubleshooting> . FEATURE: vmagent: show original labels before relabeling is applied on `duplicate scrape target` errors. This should simplify debugging for incorrect relabeling. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/651> FEATURE: vmagent: `/targets` page now accepts optional `showoriginallabels=1` query arg for displaying original labels for each target before relabeling is applied. This should simplify debugging for target relabeling configs. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/651> FEATURE: add `-finalMergeDelay` command-line flag for configuring the delay before final merge for per-month partitions. The final merge is started after no new data is ingested into per-month partition during `-finalMergeDelay`. FEATURE: add `vmrowsaddedtostorage_total` metric, which shows the total number of rows added to storage since app start. The `sum(rate(vmrowsaddedtostoragetotal))` can be smaller than `sum(rate(vmrowsinsertedtotal))` if certain metrics are dropped due to . The `sum(rate(vmrowsaddedtostorage_total))` can be bigger than `sum(rate(vmrowsinserted_total))` if is enabled. FEATURE: keep metric name after applying functions, which don't change time series

The list of such functions: `keeplastvalue` `keepnextvalue` `interpolate` `running_min` `running_max` `running_avg` `range_min` `range_max` `range_avg` `range_first` `range_last` `range_quantile` `smooth_exponential` `ceil` `floor` `round` `clamp_min` `clamp_max` `maxovertime` `minovertime` `avgovertime` `quantileovertime` `modeovertime` `geomeanovertime` `holt_winters` `predict_linear` See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674> BUGFIX: properly handle stale time series after K8S deployment. Previously such time series could be double-counted. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748> BUGFIX: return a single time series at max from `absent()` function like Prometheus does. BUGFIX: vmalert: accept days, weeks and years in `for:` part of config like Prometheus does. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/817> BUGFIX: fix `modeovertime(m[d])` calculations. Previously the function could return incorrect results. Released at 2020-10-06 FEATURE: reduce CPU usage for repeated queries over sliding time window when no new time series are added to the database. Typical use cases: repeated evaluation of alerting rules in or dashboard auto-refresh in Grafana. FEATURE: vmagent: add OpenStack service discovery aka . See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728> . FEATURE: vmalert: make `-maxIdleConnections` configurable for datasource HTTP client. This option can be used for minimizing connection churn. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/795> . FEATURE: add `-influx.maxLineSize` command-line flag for configuring the maximum size for a single InfluxDB line during parsing. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/807> BUGFIX: properly handle `inf` values during . Previously `Inf` values could result in `NaN` values for adjacent samples in time series. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/805> . BUGFIX: fill gaps on graphs for `range_` and `running_*` functions. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/806> . BUGFIX: make a copy of label with new name during relabeling with `action: labelmap` in the same way as Prometheus does. Previously the original label name has been replaced. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/812> . BUGFIX: support parsing floating-point timestamp like Graphite Carbon does. Such timestmaps are truncated to seconds. Released at 2020-09-30 FEATURE: use all the available CPU cores when accepting data via a single TCP connection for . Previously data ingested via a single TCP connection could use only a single CPU core. This could limit data ingestion performance. The main benefit of this feature is that data can be imported at max speed via a single connection - there is no need to open multiple concurrent connections to VictoriaMetrics or in order to achieve the maximum data ingestion speed. FEATURE: cluster: improve performance for data ingestion path from `vminsert` to `vmstorage` nodes. The maximum data ingestion performance for a single connection between `vminsert` and `vmstorage` node scales with the number of available CPU cores on `vmstorage` side. This should help with <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/791> . FEATURE: add ability to export / import data in native format via `/api/v1/export/native` and `/api/v1/import/native`. This is the most optimized approach for data migration between VictoriaMetrics instances. Both single-node and cluster instances are supported. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/787#issuecomment-700632551> . FEATURE: add `reducememusage` query option to `/api/v1/export` in order to reduce memory usage during data export / import. See for details. FEATURE: improve performance for `/api/v1/series` handler when it returns big number of time series. FEATURE: add `vmmergeneedfreedisk_space` metric, which can be used for estimating the number of deferred background data merges due to the lack of free disk space. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686> . FEATURE: add OpenBSD support. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/785> . BUGFIX: properly apply `-search.maxStalenessInterval` command-line flag value. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/784> . BUGFIX: fix displaying data in Grafana tables. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720> . BUGFIX: do not adjust the number of detected CPU cores found at `/sys/devices/system/cpu/online`. The adjustment was increasing the resulting GOMAXPROC by 1, which looked confusing to users. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685#issuecomment-698595309> . BUGFIX: vmagent: do not show `-remoteWrite.url` in initial logs if `-remoteWrite.showURL` isn't set. See <https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773> . BUGFIX: properly handle case when finds both a leaf and a node for the given `query=prefix.`. In this case only the node must be returned with stripped dot in the end of id as carbonapi