Repository: VictoriaMetrics/VictoriaMetrics
Pull request #4448 (2023-06-14 07:31:08)

docs: mention errors processing for /api/v1/import API
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4446


Patch:
diff --git a/README.md b/README.md
index b22a0af64..387f6c4dc 100644
--- a/README.md
+++ b/README.md
@@ -1163,6 +1163,12 @@ Additionally, VictoriaMetrics can accept metrics via the following popular data
 
 ### How to import data in JSON line format
 
+`/api/v1/import` is an API optimized for performance and processes data in a streaming fashion. 
+The client can transfer unlimited amount of data through one open connection. 
+`/api/v1/import` API doesn't return parsing errors to the client, as it is expected for data stream
+to be not interrupted. Instead, look for parsing errors on server side (VictoriaMetrics single-node or vminsert) or
+check for changes in `vm_rows_invalid_total` (exported by server side) metric.
+
 Example for importing data obtained via [/api/v1/export](#how-to-export-data-in-json-line-format):
 
 ```console
diff --git a/app/vmctl/backoff/backoff.go b/app/vmctl/backoff/backoff.go
index 9dc9637e8..bedde6cdc 100644
--- a/app/vmctl/backoff/backoff.go
+++ b/app/vmctl/backoff/backoff.go
@@ -11,9 +11,9 @@ import (
 )
 
 const (
-	backoffRetries     = 5
-	backoffFactor      = 1.7
-	backoffMinDuration = time.Second
+	backoffRetries     = 10
+	backoffFactor      = 1.8
+	backoffMinDuration = time.Second * 2
 )
 
 // retryableFunc describes call back which will repeat on errors
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index 435495456..f2ea6ed6e 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -29,6 +29,7 @@ The following tip changes can be tested by building VictoriaMetrics components f
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): Adds `enable_http2` on scrape configuration level. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4283). Thanks to @Haleygo for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4295).
 * FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl.html): add verbose output for docker installations or when TTY isn't available. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4081).
 * FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl.html): interrupt backoff retries when import process is cancelled. The change makes vmctl more responsive in case of errors during the import. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4442).
+* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl.html): update backoff policy on retries to reduce probability of overloading for `source` or `destination` databases. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4442).
 * FEATURE: vmstorage: suppress "broken pipe" errors for search queries on vmstorage side. See [this commit](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4418/commits/a6a7795b9e1f210d614a2c5f9a3016b97ded4792).
 * FEATURE: [Official Grafana dashboards for VictoriaMetrics](https://grafana.com/orgs/victoriametrics): add panel for tracking rate of syscalls while writing or reading from disk via `process_io_(read|write)_syscalls_total` metrics.
 
diff --git a/docs/README.md b/docs/README.md
index dfb287298..1cdd98492 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -1166,6 +1166,12 @@ Additionally, VictoriaMetrics can accept metrics via the following popular data
 
 ### How to import data in JSON line format
 
+`/api/v1/import` is an API optimized for performance and processes data in a streaming fashion. 
+The client can transfer unlimited amount of data through one open connection. 
+`/api/v1/import` API doesn't return parsing errors to the client, as it is expected for data stream
+to be not interrupted. Instead, look for parsing errors on server side (VictoriaMetrics single-node or vminsert) or
+check for changes in `vm_rows_invalid_total` (exported by server side) metric.
+
 Example for importing data obtained via [/api/v1/export](#how-to-export-data-in-json-line-format):
 
 ```console
diff --git a/docs/Single-server-VictoriaMetrics.md b/docs/Single-server-VictoriaMetrics.md
index 22d51a1c6..d7accfbef 100644
--- a/docs/Single-server-VictoriaMetrics.md
+++ b/docs/Single-server-VictoriaMetrics.md
@@ -1174,6 +1174,12 @@ Additionally, VictoriaMetrics can accept metrics via the following popular data
 
 ### How to import data in JSON line format
 
+`/api/v1/import` is an API optimized for performance and processes data in a streaming fashion. 
+The client can transfer unlimited amount of data through one open connection. 
+`/api/v1/import` API doesn't return parsing errors to the client, as it is expected for data stream
+to be not interrupted. Instead, look for parsing errors on server side (VictoriaMetrics single-node or vminsert) or
+check for changes in `vm_rows_invalid_total` (exported by server side) metric.
+
 Example for importing data obtained via [/api/v1/export](#how-to-export-data-in-json-line-format):
 
 ```console
diff --git a/lib/mergeset/table.go b/lib/mergeset/table.go
index f9744ee31..f436307cf 100644
--- a/lib/mergeset/table.go
+++ b/lib/mergeset/table.go
@@ -274,6 +274,9 @@ type partWrapper struct {
 
 	refCount uint32
 
+	// mustBeDeleted marks partWrapper for deletion.
+	// This field should be updated only after partWrapper
+	// was removed from the list of active parts.
 	mustBeDeleted uint32
 
 	isInMerge bool


Test patch:
diff --git a/app/vmctl/backoff/backoff_test.go b/app/vmctl/backoff/backoff_test.go
index 46556d2d2..6d6d001a1 100644
--- a/app/vmctl/backoff/backoff_test.go
+++ b/app/vmctl/backoff/backoff_test.go
@@ -37,7 +37,7 @@ func TestRetry_Do(t *testing.T) {
 			},
 			ctx:     context.Background(),
 			want:    0,
-			wantErr: false,
+			wantErr: true,
 		},
 		{
 			name:               "only one retry test",
@@ -82,7 +82,7 @@ func TestRetry_Do(t *testing.T) {
 		{
 			name:               "cancel context",
 			backoffRetries:     5,
-			backoffFactor:      0.1,
+			backoffFactor:      1.7,
 			backoffMinDuration: time.Millisecond * 10,
 			retryableFunc: func() error {
 				t := time.NewTicker(time.Millisecond * 5)
@@ -93,14 +93,14 @@ func TestRetry_Do(t *testing.T) {
 				return nil
 			},
 			ctx:           context.Background(),
-			cancelTimeout: time.Second * 5,
+			cancelTimeout: time.Millisecond * 40,
 			want:          3,
 			wantErr:       true,
 		},
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			r := New()
+			r := &Backoff{retries: tt.backoffRetries, factor: tt.backoffFactor, minDuration: tt.backoffMinDuration}
 			if tt.cancelTimeout != 0 {
 				newCtx, cancelFn := context.WithTimeout(tt.ctx, tt.cancelTimeout)
 				tt.ctx = newCtx