Repository: VictoriaMetrics/VictoriaMetrics
Pull request #1115 (2021-03-08 04:04:55)

spelling fix: adjacent


Patch:
diff --git a/app/vmagent/README.md b/app/vmagent/README.md
index b8da8d52f..22aaa2360 100644
--- a/app/vmagent/README.md
+++ b/app/vmagent/README.md
@@ -326,14 +326,24 @@ It may be useful to perform `vmagent` rolling update without any scrape loss.
   the url may contain sensitive information such as auth tokens or passwords.
   Pass `-remoteWrite.showURL` command-line flag when starting `vmagent` in order to see all the valid urls.
 
-* If scrapes must be aligned in time (for instance, if they must be performed at the beginning of every hour), then set the `scrape_align_interval` option
-  in the corresponding scrape config. For example, the following config aligns hourly scrapes to the nearest 10 minutes:
+* By default `vmagent` evenly spreads scrape load in time. If a particular scrape target must be scraped at the beginning of some interval,
+  then `scrape_align_interval` option  must be used. For example, the following config aligns hourly scrapes to the beginning of hour:
 
   ```yml
   scrape_configs:
   - job_name: foo
     scrape_interval: 1h
-    scrape_align_interval: 10m
+    scrape_align_interval: 1h
+  ```
+
+* By default `vmagent` evenly spreads scrape load in time. If a particular scrape target must be scraped at specific offset, then `scrape_offset` option must be used.
+  For example, the following config instructs `vmagent` to scrape the target at 10 seconds of every minute:
+
+  ```yml
+  scrape_configs:
+  - job_name: foo
+    scrape_interval: 1m
+    scrape_offset: 10s
   ```
 
 * If you see `skipping duplicate scrape target with identical labels` errors when scraping Kubernetes pods, then it is likely these pods listen to multiple ports
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index a4e868465..b1823220d 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -7,6 +7,7 @@
   - `histogram_stdvar(buckets)` - returns standard variance for the given buckets.
   - `histogram_stddev(buckets)` - returns standard deviation for the given buckets.
 * FEATURE: vmagent: add ability to replicate scrape targets among `vmagent` instances in the cluster with `-promscrape.cluster.replicationFactor` command-line flag. See [these docs](https://victoriametrics.github.io/vmagent.html#scraping-big-number-of-targets).
+* FATURE: vmagent: accept `scrape_offset` option at `scrape_config`. This option may be useful when scrapes must start at the specified offset of every scrape interval. See [these docs](https://victoriametrics.github.io/vmagent.html#troubleshooting) for details.
 * FEATURE: vmauth: allow using regexp paths in `url_map`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1112) for details.
 
 * BUGFIX: vmagent: reduce memory usage when Kubernetes service discovery is used in big number of distinct jobs by sharing the cache. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1113
@@ -384,7 +385,7 @@ in front of VictoriaMetrics. [Contact us](mailto:sales@victoriametrics.com) if y
   See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/807
 
 * BUGFIX: properly handle `inf` values during [background merge of LSM parts](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
-  Previously `Inf` values could result in `NaN` values for adjancent samples in time series. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/805 .
+  Previously `Inf` values could result in `NaN` values for adjacent samples in time series. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/805 .
 * BUGFIX: fill gaps on graphs for `range_*` and `running_*` functions. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/806 .
 * BUGFIX: make a copy of label with new name during relabeling with `action: labelmap` in the same way as Prometheus does.
   Previously the original label name has been replaced. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/812 .
diff --git a/docs/MetricsQL.md b/docs/MetricsQL.md
index dbf8dfd4c..b67cb4d3c 100644
--- a/docs/MetricsQL.md
+++ b/docs/MetricsQL.md
@@ -143,8 +143,8 @@ This functionality can be tried at [an editable Grafana dashboard](http://play-g
 - `first_over_time(m[d])` - returns the first value for `m` on the time range `d`.
 - `outliersk(N, q) by (group)` - returns up to `N` outlier time series for `q` in every `group`. Outlier time series have the highest deviation from the `median(q)`.
   This aggregate function is useful to detect anomalies across groups of similar time series.
-- `ascent_over_time(m[d])` - returns the sum of positive deltas between adjancent data points in `m` over `d`. Useful for tracking height gains in GPS track.
-- `descent_over_time(m[d])` - returns the absolute sum of negative deltas between adjancent data points in `m` over `d`. Useful for tracking height loss in GPS track.
+- `ascent_over_time(m[d])` - returns the sum of positive deltas between adjacent data points in `m` over `d`. Useful for tracking height gains in GPS track.
+- `descent_over_time(m[d])` - returns the absolute sum of negative deltas between adjacent data points in `m` over `d`. Useful for tracking height loss in GPS track.
 - `mode_over_time(m[d])` - returns [mode](https://en.wikipedia.org/wiki/Mode_(statistics)) for `m` values over `d`. It is expected that `m` values are discrete.
 - `mode(q) by (x)` - returns [mode](https://en.wikipedia.org/wiki/Mode_(statistics)) for each point in `q` grouped by `x`. It is expected that `q` points are discrete.
 - `rate_over_sum(m[d])` - returns rate over the sum of `m` values over `d` duration.
diff --git a/docs/vmagent.md b/docs/vmagent.md
index b8da8d52f..22aaa2360 100644
--- a/docs/vmagent.md
+++ b/docs/vmagent.md
@@ -326,14 +326,24 @@ It may be useful to perform `vmagent` rolling update without any scrape loss.
   the url may contain sensitive information such as auth tokens or passwords.
   Pass `-remoteWrite.showURL` command-line flag when starting `vmagent` in order to see all the valid urls.
 
-* If scrapes must be aligned in time (for instance, if they must be performed at the beginning of every hour), then set the `scrape_align_interval` option
-  in the corresponding scrape config. For example, the following config aligns hourly scrapes to the nearest 10 minutes:
+* By default `vmagent` evenly spreads scrape load in time. If a particular scrape target must be scraped at the beginning of some interval,
+  then `scrape_align_interval` option  must be used. For example, the following config aligns hourly scrapes to the beginning of hour:
 
   ```yml
   scrape_configs:
   - job_name: foo
     scrape_interval: 1h
-    scrape_align_interval: 10m
+    scrape_align_interval: 1h
+  ```
+
+* By default `vmagent` evenly spreads scrape load in time. If a particular scrape target must be scraped at specific offset, then `scrape_offset` option must be used.
+  For example, the following config instructs `vmagent` to scrape the target at 10 seconds of every minute:
+
+  ```yml
+  scrape_configs:
+  - job_name: foo
+    scrape_interval: 1m
+    scrape_offset: 10s
   ```
 
 * If you see `skipping duplicate scrape target with identical labels` errors when scraping Kubernetes pods, then it is likely these pods listen to multiple ports
diff --git a/lib/promscrape/config.go b/lib/promscrape/config.go
index 118c5dcc5..8d558862f 100644
--- a/lib/promscrape/config.go
+++ b/lib/promscrape/config.go
@@ -114,6 +114,7 @@ type ScrapeConfig struct {
 	DisableKeepAlive    bool          `yaml:"disable_keepalive,omitempty"`
 	StreamParse         bool          `yaml:"stream_parse,omitempty"`
 	ScrapeAlignInterval time.Duration `yaml:"scrape_align_interval,omitempty"`
+	ScrapeOffset        time.Duration `yaml:"scrape_offset,omitempty"`
 
 	// This is set in loadConfig
 	swc *scrapeWorkConfig
@@ -569,6 +570,7 @@ func getScrapeWorkConfig(sc *ScrapeConfig, baseDir string, globalCfg *GlobalConf
 		disableKeepAlive:     sc.DisableKeepAlive,
 		streamParse:          sc.StreamParse,
 		scrapeAlignInterval:  sc.ScrapeAlignInterval,
+		scrapeOffset:         sc.ScrapeOffset,
 	}
 	return swc, nil
 }
@@ -592,6 +594,7 @@ type scrapeWorkConfig struct {
 	disableKeepAlive     bool
 	streamParse          bool
 	scrapeAlignInterval  time.Duration
+	scrapeOffset         time.Duration
 }
 
 type targetLabelsGetter interface {
@@ -853,6 +856,7 @@ func (swc *scrapeWorkConfig) getScrapeWork(target string, extraLabels, metaLabel
 		DisableKeepAlive:     swc.disableKeepAlive,
 		StreamParse:          swc.streamParse,
 		ScrapeAlignInterval:  swc.scrapeAlignInterval,
+		ScrapeOffset:         swc.scrapeOffset,
 
 		jobNameOriginal: swc.jobName,
 	}
diff --git a/lib/promscrape/scrapework.go b/lib/promscrape/scrapework.go
index 558f3e8b5..e5e28ff42 100644
--- a/lib/promscrape/scrapework.go
+++ b/lib/promscrape/scrapework.go
@@ -92,6 +92,9 @@ type ScrapeWork struct {
 	// The interval for aligning the first scrape.
 	ScrapeAlignInterval time.Duration
 
+	// The offset for the first scrape.
+	ScrapeOffset time.Duration
+
 	// The original 'job_name'
 	jobNameOriginal string
 }
@@ -102,9 +105,11 @@ type ScrapeWork struct {
 func (sw *ScrapeWork) key() string {
 	// Do not take into account OriginalLabels.
 	key := fmt.Sprintf("ScrapeURL=%s, ScrapeInterval=%s, ScrapeTimeout=%s, HonorLabels=%v, HonorTimestamps=%v, Labels=%s, "+
-		"AuthConfig=%s, MetricRelabelConfigs=%s, SampleLimit=%d, DisableCompression=%v, DisableKeepAlive=%v, StreamParse=%v, ScrapeAlignInterval=%s",
+		"AuthConfig=%s, MetricRelabelConfigs=%s, SampleLimit=%d, DisableCompression=%v, DisableKeepAlive=%v, StreamParse=%v, "+
+		"ScrapeAlignInterval=%s, ScrapeOffset=%s",
 		sw.ScrapeURL, sw.ScrapeInterval, sw.ScrapeTimeout, sw.HonorLabels, sw.HonorTimestamps, sw.LabelsString(),
-		sw.AuthConfig.String(), sw.MetricRelabelConfigs.String(), sw.SampleLimit, sw.DisableCompression, sw.DisableKeepAlive, sw.StreamParse, sw.ScrapeAlignInterval)
+		sw.AuthConfig.String(), sw.MetricRelabelConfigs.String(), sw.SampleLimit, sw.DisableCompression, sw.DisableKeepAlive, sw.StreamParse,
+		sw.ScrapeAlignInterval, sw.ScrapeOffset)
 	return key
 }
 
@@ -174,9 +179,14 @@ type scrapeWork struct {
 }
 
 func (sw *scrapeWork) run(stopCh <-chan struct{}) {
-	scrapeInterval := sw.Config.ScrapeInterval
 	var randSleep uint64
-	if sw.Config.ScrapeAlignInterval <= 0 {
+	scrapeInterval := sw.Config.ScrapeInterval
+	scrapeAlignInterval := sw.Config.ScrapeAlignInterval
+	scrapeOffset := sw.Config.ScrapeOffset
+	if scrapeOffset > 0 {
+		scrapeAlignInterval = scrapeInterval
+	}
+	if scrapeAlignInterval <= 0 {
 		// Calculate start time for the first scrape from ScrapeURL and labels.
 		// This should spread load when scraping many targets with different
 		// scrape urls and labels.
@@ -191,8 +201,11 @@ func (sw *scrapeWork) run(stopCh <-chan struct{}) {
 		}
 		randSleep -= sleepOffset
 	} else {
-		d := uint64(sw.Config.ScrapeAlignInterval)
+		d := uint64(scrapeAlignInterval)
 		randSleep = d - uint64(time.Now().UnixNano())%d
+		if scrapeOffset > 0 {
+			randSleep += uint64(scrapeOffset)
+		}
 		randSleep %= uint64(scrapeInterval)
 	}
 	timer := timerpool.Get(time.Duration(randSleep))
diff --git a/lib/storage/block_stream_writer.go b/lib/storage/block_stream_writer.go
index f80157b6b..5421a6656 100644
--- a/lib/storage/block_stream_writer.go
+++ b/lib/storage/block_stream_writer.go
@@ -42,7 +42,7 @@ type blockStreamWriter struct {
 
 	// prevTimestamps* is used as an optimization for reducing disk space usage
 	// when serially written blocks have identical timestamps.
-	// This is usually the case when adjancent blocks contain metrics scraped from the same target,
+	// This is usually the case when adjacent blocks contain metrics scraped from the same target,
 	// since such metrics have identical timestamps.
 	prevTimestampsData        []byte
 	prevTimestampsBlockOffset uint64
diff --git a/lib/storage/index_db.go b/lib/storage/index_db.go
index 21fbcc53d..99711f00e 100644
--- a/lib/storage/index_db.go
+++ b/lib/storage/index_db.go
@@ -2459,7 +2459,7 @@ func (is *indexSearch) getMetricIDsForTagFilterSlow(tf *tagFilter, filter *uint6
 		// Slow path: need tf.matchSuffix call.
 		ok, err := tf.matchSuffix(suffix)
 		// Assume that tf.matchSuffix call needs 10x more time than a single metric scan iteration.
-		loopsCount += 10*tf.matchCost
+		loopsCount += 10 * tf.matchCost
 		if err != nil {
 			return loopsCount, fmt.Errorf("error when matching %s against suffix %q: %w", tf, suffix, err)
 		}
@@ -3497,7 +3497,7 @@ func mergeTagToMetricIDsRowsInternal(data []byte, items []mergeset.Item, nsPrefi
 		if len(item) == 0 || item[0] != nsPrefix || i == 0 || i == len(items)-1 {
 			// Write rows not starting with nsPrefix as-is.
 			// Additionally write the first and the last row as-is in order to preserve
-			// sort order for adjancent blocks.
+			// sort order for adjacent blocks.
 			dstData, dstItems = tmm.flushPendingMetricIDs(dstData, dstItems, mpPrev)
 			dstData = append(dstData, item...)
 			dstItems = append(dstItems, mergeset.Item{
diff --git a/lib/storage/storage.go b/lib/storage/storage.go
index d1f8dd57c..6c0586357 100644
--- a/lib/storage/storage.go
+++ b/lib/storage/storage.go
@@ -1412,7 +1412,7 @@ func (s *Storage) add(rows []rawRow, mrs []MetricRow, precisionBits uint8) ([]ra
 	rows = rows[:rowsLen+len(mrs)]
 	j := 0
 	var (
-		// These vars are used for speeding up bulk imports of multiple adjancent rows for the same metricName.
+		// These vars are used for speeding up bulk imports of multiple adjacent rows for the same metricName.
 		prevTSID          TSID
 		prevMetricNameRaw []byte
 	)
@@ -1626,7 +1626,7 @@ func (s *Storage) updatePerDateData(rows []rawRow) error {
 	var hour uint64
 	var prevTimestamp int64
 	var (
-		// These vars are used for speeding up bulk imports when multiple adjancent rows
+		// These vars are used for speeding up bulk imports when multiple adjacent rows
 		// contain the same (metricID, date) pairs.
 		prevDate     uint64
 		prevMetricID uint64


Test patch:
diff --git a/lib/promscrape/config_test.go b/lib/promscrape/config_test.go
index d79bee3c8..f75dae834 100644
--- a/lib/promscrape/config_test.go
+++ b/lib/promscrape/config_test.go
@@ -1302,6 +1302,7 @@ scrape_configs:
     disable_compression: true
     stream_parse: true
     scrape_align_interval: 1s
+    scrape_offset: 0.5s
     static_configs:
       - targets:
         - 192.168.1.2  # SNMP device.
@@ -1356,6 +1357,7 @@ scrape_configs:
 			DisableCompression:  true,
 			StreamParse:         true,
 			ScrapeAlignInterval: time.Second,
+			ScrapeOffset:        500 * time.Millisecond,
 			jobNameOriginal:     "snmp",
 		},
 	})