Repository: VictoriaMetrics/VictoriaMetrics
Pull request #297 (2020-01-27 15:35:30)

Add description for retention and how it works


Patch:
diff --git a/README.md b/README.md
index 5cdf4977d..a420c7c2b 100644
--- a/README.md
+++ b/README.md
@@ -88,7 +88,6 @@ Cluster version is available [here](https://github.com/VictoriaMetrics/VictoriaM
     - [Building docker images](#building-docker-images)
   - [Start with docker-compose](#start-with-docker-compose)
   - [Setting up service](#setting-up-service)
-  - [Third-party contributions](#third-party-contributions)
   - [How to work with snapshots?](#how-to-work-with-snapshots)
   - [How to delete time series?](#how-to-delete-time-series)
   - [How to export time series?](#how-to-export-time-series)
@@ -96,6 +95,7 @@ Cluster version is available [here](https://github.com/VictoriaMetrics/VictoriaM
   - [Federation](#federation)
   - [Capacity planning](#capacity-planning)
   - [High availability](#high-availability)
+  - [Retention](#retention)
   - [Multiple retentions](#multiple-retentions)
   - [Downsampling](#downsampling)
   - [Multi-tenancy](#multi-tenancy)
@@ -111,6 +111,7 @@ Cluster version is available [here](https://github.com/VictoriaMetrics/VictoriaM
 - [Roadmap](#roadmap)
 - [Contacts](#contacts)
 - [Community and contributions](#community-and-contributions)
+- [Third-party contributions](#third-party-contributions)
 - [Reporting bugs](#reporting-bugs)
 - [Victoria Metrics Logo](#victoria-metrics-logo)
   - [Logo Usage Guidelines](#logo-usage-guidelines)
@@ -502,12 +503,6 @@ More details may be found [here](https://github.com/VictoriaMetrics/VictoriaMetr
 Read [these instructions](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/43) on how to set up VictoriaMetrics as a service in your OS.
 
 
-### Third-party contributions
-
-* [Unofficial yum repository](https://copr.fedorainfracloud.org/coprs/antonpatsev/VictoriaMetrics/) ([source code](https://github.com/patsevanton/victoriametrics-rpm))
-* [Prometheus -> VictoriaMetrics exporter](https://github.com/ryotarai/prometheus-tsdb-dump)
-
-
 ### How to work with snapshots?
 
 VictoriaMetrics can create [instant snapshots](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282)
@@ -706,6 +701,12 @@ kill -HUP `pidof prometheus`
 If you have Prometheus HA pairs with replicas `r1` and `r2` in each pair, then configure each `r1`
 to write data to `victoriametrics-addr-1`, while each `r2` should write data to `victoriametrics-addr-2`.
 
+### Retention
+
+Metrics is stored in directories per month. Retention occurs on the first day of the new month.
+In order to keep data according to `retentionPeriod` max disk usage is going to be `retentionPeriod` + 1 month.
+For example if you have `retentionPeriod` set to 1, january data will be deleted at March 1st.
+
 
 ### Multiple retentions
 
@@ -913,6 +914,13 @@ We are open to third-party pull requests provided they follow [KISS design princ
 Adhering `KISS` principle simplifies the resulting code and architecture, so it can be reviewed, understood and verified by many people.
 
 
+### Third-party contributions
+
+* [Unofficial yum repository](https://copr.fedorainfracloud.org/coprs/antonpatsev/VictoriaMetrics/) ([source code](https://github.com/patsevanton/victoriametrics-rpm))
+* [Prometheus -> VictoriaMetrics exporter #1](https://github.com/ryotarai/prometheus-tsdb-dump)
+* [Prometheus -> VictoriaMetrics exporter #2](https://github.com/AnchorFree/tsdb-remote-write)
+
+
 ## Reporting bugs
 
 Report bugs and propose new features [here](https://github.com/VictoriaMetrics/VictoriaMetrics/issues).
diff --git a/app/vmselect/promql/transform.go b/app/vmselect/promql/transform.go
index 9801e757c..065b521f6 100644
--- a/app/vmselect/promql/transform.go
+++ b/app/vmselect/promql/transform.go
@@ -363,6 +363,7 @@ func vmrangeBucketsToLE(tss []*timeseries) []*timeseries {
 			ts := xs.ts
 			if isZeroTS(ts) {
 				// Skip time series with zeros. They are substituted by xssNew below.
+				xsPrev = xs
 				continue
 			}
 			if xs.start != xsPrev.end {
diff --git a/docs/Single-server-VictoriaMetrics.md b/docs/Single-server-VictoriaMetrics.md
index d40e6ddcf..a036bb312 100644
--- a/docs/Single-server-VictoriaMetrics.md
+++ b/docs/Single-server-VictoriaMetrics.md
@@ -78,7 +78,6 @@ Cluster version is available [here](https://github.com/VictoriaMetrics/VictoriaM
     - [Building docker images](#building-docker-images)
   - [Start with docker-compose](#start-with-docker-compose)
   - [Setting up service](#setting-up-service)
-  - [Third-party contributions](#third-party-contributions)
   - [How to work with snapshots?](#how-to-work-with-snapshots)
   - [How to delete time series?](#how-to-delete-time-series)
   - [How to export time series?](#how-to-export-time-series)
@@ -101,6 +100,7 @@ Cluster version is available [here](https://github.com/VictoriaMetrics/VictoriaM
 - [Roadmap](#roadmap)
 - [Contacts](#contacts)
 - [Community and contributions](#community-and-contributions)
+- [Third-party contributions](#third-party-contributions)
 - [Reporting bugs](#reporting-bugs)
 - [Victoria Metrics Logo](#victoria-metrics-logo)
   - [Logo Usage Guidelines](#logo-usage-guidelines)
@@ -492,12 +492,6 @@ More details may be found [here](https://github.com/VictoriaMetrics/VictoriaMetr
 Read [these instructions](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/43) on how to set up VictoriaMetrics as a service in your OS.
 
 
-### Third-party contributions
-
-* [Unofficial yum repository](https://copr.fedorainfracloud.org/coprs/antonpatsev/VictoriaMetrics/) ([source code](https://github.com/patsevanton/victoriametrics-rpm))
-* [Prometheus -> VictoriaMetrics exporter](https://github.com/ryotarai/prometheus-tsdb-dump)
-
-
 ### How to work with snapshots?
 
 VictoriaMetrics can create [instant snapshots](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282)
@@ -776,8 +770,11 @@ mkfs.ext4 ... -O 64bit,huge_file,extent -T huge
 
 ### Monitoring
 
-VictoriaMetrics exports internal metrics in Prometheus format on the `/metrics` page.
-Add this page to Prometheus' scrape config in order to collect VictoriaMetrics metrics.
+VictoriaMetrics exports internal metrics in Prometheus format at `/metrics` page.
+These metrics may be collected either via Prometheus by adding the corresponding scrape config to it.
+Alternatively they can be self-scraped by setting `-selfScrapeInterval` command-line flag to duration greater than 0.
+For example, `-scrapeInterval=10s` would enable self-scraping of `/metrics` page with 10 seconds interval.
+
 There are officials Grafana dashboards for [single-node VictoriaMetrics](https://grafana.com/dashboards/10229) and [clustered VictoriaMetrics](https://grafana.com/grafana/dashboards/11176).
 
 The most interesting metrics are:
@@ -900,6 +897,13 @@ We are open to third-party pull requests provided they follow [KISS design princ
 Adhering `KISS` principle simplifies the resulting code and architecture, so it can be reviewed, understood and verified by many people.
 
 
+### Third-party contributions
+
+* [Unofficial yum repository](https://copr.fedorainfracloud.org/coprs/antonpatsev/VictoriaMetrics/) ([source code](https://github.com/patsevanton/victoriametrics-rpm))
+* [Prometheus -> VictoriaMetrics exporter #1](https://github.com/ryotarai/prometheus-tsdb-dump)
+* [Prometheus -> VictoriaMetrics exporter #2](https://github.com/AnchorFree/tsdb-remote-write)
+
+
 ## Reporting bugs
 
 Report bugs and propose new features [here](https://github.com/VictoriaMetrics/VictoriaMetrics/issues).


Test patch:
diff --git a/app/vmselect/promql/exec_test.go b/app/vmselect/promql/exec_test.go
index 8717bb065..3461c93f4 100644
--- a/app/vmselect/promql/exec_test.go
+++ b/app/vmselect/promql/exec_test.go
@@ -2954,6 +2954,23 @@ func TestExecSuccess(t *testing.T) {
 		resultExpected := []netstorage.Result{r1, r2, r3, r4, r5, r6}
 		f(q, resultExpected)
 	})
+	t.Run(`prometheus_buckets(zero-vmrange-value)`, func(t *testing.T) {
+		t.Parallel()
+		q := `sort(prometheus_buckets(label_set(0, "vmrange", "0...0")))`
+		r1 := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{0, 0, 0, 0, 0, 0},
+			Timestamps: timestampsExpected,
+		}
+		r1.MetricName.Tags = []storage.Tag{
+			{
+				Key:   []byte("le"),
+				Value: []byte("+Inf"),
+			},
+		}
+		resultsExpected := []netstorage.Result{r1}
+		f(q, resultsExpected)
+	})
 	t.Run(`prometheus_buckets(valid)`, func(t *testing.T) {
 		t.Parallel()
 		q := `sort(prometheus_buckets((