Repository: VictoriaMetrics/VictoriaMetrics
Pull request #3737 (2023-02-01 11:09:22)

lib/storage: fixes finalDedup for backfilled data
previously historical data backfilling may trigger force merge for previous month every hour it consumes cpu, disk io and decrease cluster performance. Following commit fixes it by applying deduplication for InMemoryParts

codecov[bot]: # [Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3737?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) Report
Base: **57.50**% // Head: **57.46**% // Decreases project coverage by **`-0.04%`** :warning:
> Coverage data is based on head [(`66abb1d`)](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3737?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) compared to base [(`4cf37c5`)](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/commit/4cf37c5e7086e3415d9316f45ca58d4b8e98bc0e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).
> Patch coverage: 88.88% of modified lines in pull request are covered.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master    #3737      +/-   ##
==========================================
- Coverage   57.50%   57.46%   -0.04%     
==========================================
  Files         312      312              
  Lines       52348    52349       +1     
==========================================
- Hits        30103    30084      -19     
- Misses      20355    20371      +16     
- Partials     1890     1894       +4     
```


| [Impacted Files](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3737?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) | Coverage Δ | |
|---|---|---|
| [lib/storage/partition.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3737?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL3N0b3JhZ2UvcGFydGl0aW9uLmdv) | `61.39% <87.50%> (-0.38%)` | :arrow_down: |
| [lib/storage/partition\_search.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3737?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL3N0b3JhZ2UvcGFydGl0aW9uX3NlYXJjaC5nbw==) | `81.74% <100.00%> (ø)` | |
| [app/vmalert/remotewrite/remotewrite.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3737?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvcmVtb3Rld3JpdGUvcmVtb3Rld3JpdGUuZ28=) | `60.27% <0.00%> (-5.48%)` | :arrow_down: |
| [lib/fs/reader\_at.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3737?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL2ZzL3JlYWRlcl9hdC5nbw==) | `41.25% <0.00%> (-2.50%)` | :arrow_down: |
| [app/vmselect/promql/aggr\_incremental.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3737?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3Ztc2VsZWN0L3Byb21xbC9hZ2dyX2luY3JlbWVudGFsLmdv) | `84.13% <0.00%> (-2.40%)` | :arrow_down: |
| [lib/storage/storage.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3737?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL3N0b3JhZ2Uvc3RvcmFnZS5nbw==) | `71.15% <0.00%> (-0.12%)` | :arrow_down: |
| [app/vmselect/promql/transform.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3737?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3Ztc2VsZWN0L3Byb21xbC90cmFuc2Zvcm0uZ28=) | `88.20% <0.00%> (+0.15%)` | :arrow_up: |
| [lib/mergeset/table.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3737?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL21lcmdlc2V0L3RhYmxlLmdv) | `63.60% <0.00%> (+0.24%)` | :arrow_up: |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics)

</details>

[:umbrella: View full report at Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3737?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).


Patch:
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index 8a65a56b0..f183d6be7 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -36,6 +36,7 @@ The following tip changes can be tested by building VictoriaMetrics components f
 * BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth.html): allow re-entering authorization info in the web browser if the entered info was incorrect. Previously it was non-trivial to do via the web browser, since `vmauth` was returning `400 Bad Request` instead of `401 Unauthorized` http response code.
 * BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth.html): always log the client address and the requested URL on proxying errors. Previously some errors could miss this information.
 * BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup.html): fix snapshot not being deleted after backup completion. This issue could result in unnecessary snapshots being stored, it is required to delete unnecessary snapshots manually. See the [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3735).
+* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html): fix panic on top-level vmselect nodes of [multi-level setup](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multi-level-cluster-setup) when the `-replicationFactor` flag is set and request contains `trace` query parameter. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3734).
 
 ## [v1.86.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.86.2)
 
diff --git a/lib/storage/partition.go b/lib/storage/partition.go
index adb03daa8..cbcaf295d 100644
--- a/lib/storage/partition.go
+++ b/lib/storage/partition.go
@@ -736,12 +736,15 @@ func (pt *partition) HasTimestamp(timestamp int64) bool {
 // GetParts appends parts snapshot to dst and returns it.
 //
 // The appended parts must be released with PutParts.
-func (pt *partition) GetParts(dst []*partWrapper) []*partWrapper {
+func (pt *partition) GetParts(dst []*partWrapper, addInMemory bool) []*partWrapper {
 	pt.partsLock.Lock()
-	for _, pw := range pt.inmemoryParts {
-		pw.incRef()
+	if addInMemory {
+		for _, pw := range pt.inmemoryParts {
+			pw.incRef()
+		}
+		dst = append(dst, pt.inmemoryParts...)
 	}
-	dst = append(dst, pt.inmemoryParts...)
+
 	for _, pw := range pt.smallParts {
 		pw.incRef()
 	}
@@ -1227,7 +1230,7 @@ func (pt *partition) runFinalDedup() error {
 }
 
 func (pt *partition) getRequiredDedupInterval() (int64, int64) {
-	pws := pt.GetParts(nil)
+	pws := pt.GetParts(nil, false)
 	defer pt.PutParts(pws)
 	dedupInterval := GetDedupInterval()
 	minDedupInterval := getMinDedupInterval(pws)
@@ -1275,12 +1278,13 @@ func (pt *partition) mergeParts(pws []*partWrapper, stopCh <-chan struct{}, isFi
 		}()
 	}
 
-	if isFinal && len(pws) == 1 && pws[0].mp != nil {
+	if !isDedupEnabled() && isFinal && len(pws) == 1 && pws[0].mp != nil {
 		// Fast path: flush a single in-memory part to disk.
 		mp := pws[0].mp
 		if tmpPartPath == "" {
 			logger.Panicf("BUG: tmpPartPath must be non-empty")
 		}
+
 		if err := mp.StoreToDisk(tmpPartPath); err != nil {
 			return fmt.Errorf("cannot store in-memory part to %q: %w", tmpPartPath, err)
 		}
diff --git a/lib/storage/partition_search.go b/lib/storage/partition_search.go
index cb9125cbf..52e530887 100644
--- a/lib/storage/partition_search.go
+++ b/lib/storage/partition_search.go
@@ -80,7 +80,7 @@ func (pts *partitionSearch) Init(pt *partition, tsids []TSID, tr TimeRange) {
 		return
 	}
 
-	pts.pws = pt.GetParts(pts.pws[:0])
+	pts.pws = pt.GetParts(pts.pws[:0], true)
 
 	// Initialize psPool.
 	if n := len(pts.pws) - cap(pts.psPool); n > 0 {


Test patch:
diff --git a/lib/storage/storage_test.go b/lib/storage/storage_test.go
index d4b7ab29b..9ed26f67b 100644
--- a/lib/storage/storage_test.go
+++ b/lib/storage/storage_test.go
@@ -1055,7 +1055,7 @@ func testStorageAddRows(rng *rand.Rand, s *Storage) error {
 	}
 	ptws := s1.tb.GetPartitions(nil)
 	for _, ptw := range ptws {
-		pws := ptw.pt.GetParts(nil)
+		pws := ptw.pt.GetParts(nil, true)
 		numParts := len(pws)
 		ptw.pt.PutParts(pws)
 		if numParts != 1 {