Repository: VictoriaMetrics/VictoriaMetrics
Pull request #4442 (2023-06-13 07:19:29)

vmctl: finish retries if context canceled
Improved backoff retry policy for `vmctl`. 
`vmctl` should finish retries if the context is canceled.

Signed-off-by: dmitryk-dk [d.kozlov@victoriametrics.com](mailto:d.kozlov@victoriametrics.com)

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4442?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) Report
Patch coverage: **`72.72`**% and project coverage change: **`-0.03`** :warning:
> Comparison is base [(`40d12be`)](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/commit/40d12be607987a80bae5f0b03fd73f91038f9757?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) 61.27% compared to head [(`d4789a8`)](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4442?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) 61.25%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master    #4442      +/-   ##
==========================================
- Coverage   61.27%   61.25%   -0.03%     
==========================================
  Files         330      330              
  Lines       59066    59075       +9     
==========================================
- Hits        36194    36184      -10     
- Misses      20892    20907      +15     
- Partials     1980     1984       +4     
```


| [Impacted Files](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4442?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) | Coverage Δ | |
|---|---|---|
| [app/vmctl/backoff/backoff.go](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4442?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtY3RsL2JhY2tvZmYvYmFja29mZi5nbw==) | `90.90% <72.72%> (-9.10%)` | :arrow_down: |

... and [4 files with indirect coverage changes](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4442/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4442?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).

hagen1778: @dmitryk-dk would you mind adding changelog line?

Patch:
diff --git a/app/vmctl/backoff/backoff.go b/app/vmctl/backoff/backoff.go
index 431d6a0c4..9dc9637e8 100644
--- a/app/vmctl/backoff/backoff.go
+++ b/app/vmctl/backoff/backoff.go
@@ -42,7 +42,6 @@ func New() *Backoff {
 func (b *Backoff) Retry(ctx context.Context, cb retryableFunc) (uint64, error) {
 	var attempt uint64
 	for i := 0; i < b.retries; i++ {
-		// @TODO we should use context to cancel retries
 		err := cb()
 		if err == nil {
 			return attempt, nil
@@ -55,7 +54,19 @@ func (b *Backoff) Retry(ctx context.Context, cb retryableFunc) (uint64, error) {
 		backoff := float64(b.minDuration) * math.Pow(b.factor, float64(i))
 		dur := time.Duration(backoff)
 		logger.Errorf("got error: %s on attempt: %d; will retry in %v", err, attempt, dur)
-		time.Sleep(time.Duration(backoff))
+
+		t := time.NewTimer(dur)
+		select {
+		case <-t.C:
+			// duration elapsed, loop
+		case <-ctx.Done():
+			// context cancelled, kill the timer if it hasn't fired, and return
+			// the last error we got
+			if !t.Stop() {
+				<-t.C
+			}
+			return attempt, err
+		}
 	}
 	return attempt, fmt.Errorf("execution failed after %d retry attempts", b.retries)
 }
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index cc970b7c8..435495456 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -28,6 +28,7 @@ The following tip changes can be tested by building VictoriaMetrics components f
 
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): Adds `enable_http2` on scrape configuration level. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4283). Thanks to @Haleygo for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4295).
 * FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl.html): add verbose output for docker installations or when TTY isn't available. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4081).
+* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl.html): interrupt backoff retries when import process is cancelled. The change makes vmctl more responsive in case of errors during the import. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4442).
 * FEATURE: vmstorage: suppress "broken pipe" errors for search queries on vmstorage side. See [this commit](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4418/commits/a6a7795b9e1f210d614a2c5f9a3016b97ded4792).
 * FEATURE: [Official Grafana dashboards for VictoriaMetrics](https://grafana.com/orgs/victoriametrics): add panel for tracking rate of syscalls while writing or reading from disk via `process_io_(read|write)_syscalls_total` metrics.
 
diff --git a/docs/Release-Guide.md b/docs/Release-Guide.md
index 231887ef5..90b31ae10 100644
--- a/docs/Release-Guide.md
+++ b/docs/Release-Guide.md
@@ -22,12 +22,40 @@ git remote add enterprise <url>
 3. Make sure you have github token with at least `read:org, repo, write:packages` permissions exported under `GITHUB_TOKEN` env variable.
    You can create token [here](https://github.com/settings/tokens)
 
+### For MacOS users
+
+Make sure you have GNU version of utilities `zip`, `tar`, `sha256sum`. To install them run the following commands:
+```bash
+brew install coreutils
+brew install gnu-tar
+export PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"
+```
+
+Docker may need additional configuration changes:
+```bash 
+docker buildx create --use --name=qemu
+docker buildx inspect --bootstrap  
+```
+
+For ARM arch (M1/M2 processors) additionally configure docker with preferred platform:
+```
+export DOCKER_DEFAULT_PLATFORM=linux/amd64
+```
+
+By default, docker on MacOS has limited amount of resources (CPU, mem) to use. 
+Bumping the limits may significantly improve build speed.
+
 ## Release version and Docker images
 
 0. Make sure that the release commits have no security issues.
 1a. Document all the changes for new release in [CHANGELOG.md](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/CHANGELOG.md) and update version if needed in [SECURITY.md](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/SECURITY.md)
 1b. Add `(available starting from v1.xx.y)` line to feature docs introduced in the upcoming release.
-2. Create the following release tags:
+1c. Cut new version in [CHANGELOG.md](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/CHANGELOG.md)
+and make it merged. See example in this [commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/b771152039d23b5ccd637a23ea748bc44a9511a7).
+1d. Cherry-pick all changes from `master` to `cluster` branch, and to ENT branches `enterprise-single-node`, `enterprise-cluster`.
+1e. Cherry-pick bug fixes relevant for LTS releases.
+2. Make sure you get all changes fetched `git fetch --all`.
+3. Create the following release tags:
    * `git tag -s v1.xx.y` in `master` branch
    * `git tag -s v1.xx.y-cluster` in `cluster` branch
    * `git tag -s v1.xx.y-enterprise` in `enterprise` branch
@@ -62,8 +90,8 @@ git remote add enterprise <url>
         and all the needed assets are re-uploaded to it.
 6. Go to <https://github.com/VictoriaMetrics/VictoriaMetrics/releases> and verify that draft release with the name `TAG` has been created
    and this release contains all the needed binaries and checksums.
-7. Update the release description with the [CHANGELOG](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/CHANGELOG.md) for this release.
-8. Remove the `draft` checkbox for the `TAG` release and manually publish it.
+7. Update the release description with the content of [CHANGELOG](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/CHANGELOG.md) for this release.
+8. Publish release by pressing "Publish release" green button in GitHub's UI.
 9. Bump version of the VictoriaMetrics cluster in the [sandbox environment](https://github.com/VictoriaMetrics/ops/blob/main/gcp-test/sandbox/manifests/benchmark-vm/vmcluster.yaml)
    by [opening and merging PR](https://github.com/VictoriaMetrics/ops/pull/58).
 10. Bump VictoriaMetrics version at `deployment/docker/docker-compose.yml` and at `deployment/docker/docker-compose-cluster.yml`.
@@ -97,6 +125,11 @@ The helm chart repository [https://github.com/VictoriaMetrics/helm-charts/](http
 
 ### Bump the version of images
 
+Bump `tag` field in `values.yaml` with new release version.
+Bump `appVersion` field in `Chart.yaml` with new release version.
+Bump `version` field in `Chart.yaml` with incremental semver version.
+Do these updates to the following charts:
+
 1. Update `vmagent` chart version in [`values.yaml`](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-agent/values.yaml) and [`Chart.yaml`](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-agent/Chart.yaml) 
 2. Update `vmalert` chart version in [`values.yaml`](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-alert/values.yaml) and [`Chart.yaml`](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-alert/Chart.yaml)
 3. Update `vmauth` chart version in [`values.yaml`](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-auth/values.yaml) and [`Chart.yaml`](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-auth/Chart.yaml)
@@ -104,11 +137,14 @@ The helm chart repository [https://github.com/VictoriaMetrics/helm-charts/](http
 5. Update `k8s-stack` chart versions in [`values.yaml`](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-k8s-stack/values.yaml), bump version for `vmselect`, `vminsert`, `vmstorage`, `vmsingle`, `vmalert`, `vmagent` and [`Chart.yaml`](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-k8s-stack/Chart.yaml)
 6. Update `single-node` chart version in [`values.yaml`](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-single/values.yaml) and [`Chart.yaml`](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-single/Chart.yaml)
 7. Update `vmgateway` chart version in [`values.yaml`](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-gateway/values.yamll) and [`Chart.yaml`](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-gateway/Chart.yaml)
-8. Run `make gen-docs`
-9. Run `make package` that creates or updates zip file with the packed chart
-10. Run `make merge`. It creates or updates metadata for charts in index.yaml
-11. Push changes to master. `master` is a source of truth
-12. Push the changes to `gh-pages` branch
+
+Once updated, run the following commands:
+
+1. Run `make gen-docs`
+2. Run `make package` that creates or updates zip file with the packed chart
+3. Run `make merge`. It creates or updates metadata for charts in index.yaml
+4. Push changes to master. `master` is a source of truth
+5. Push the same changes to `gh-pages` branch
 
 ## Ansible Roles 
 
@@ -126,7 +162,7 @@ Repository [https://github.com/VictoriaMetrics/ansible-playbooks](https://github
 
 ## RPM packages
 
-### Bump the version of components
+### Bump the version of components (for LTS releases only)
 
 Repository [https://github.com/VictoriaMetrics/victoriametrics-lts-rpm](https://github.com/VictoriaMetrics/victoriametrics-lts-rpm)
 


Test patch:
diff --git a/app/vmctl/backoff/backoff_test.go b/app/vmctl/backoff/backoff_test.go
index e9972a27e..46556d2d2 100644
--- a/app/vmctl/backoff/backoff_test.go
+++ b/app/vmctl/backoff/backoff_test.go
@@ -16,7 +16,7 @@ func TestRetry_Do(t *testing.T) {
 		backoffMinDuration time.Duration
 		retryableFunc      retryableFunc
 		ctx                context.Context
-		withCancel         bool
+		cancelTimeout      time.Duration
 		want               uint64
 		wantErr            bool
 	}{
@@ -79,10 +79,33 @@ func TestRetry_Do(t *testing.T) {
 			want:    5,
 			wantErr: true,
 		},
+		{
+			name:               "cancel context",
+			backoffRetries:     5,
+			backoffFactor:      0.1,
+			backoffMinDuration: time.Millisecond * 10,
+			retryableFunc: func() error {
+				t := time.NewTicker(time.Millisecond * 5)
+				defer t.Stop()
+				for range t.C {
+					return fmt.Errorf("got some error")
+				}
+				return nil
+			},
+			ctx:           context.Background(),
+			cancelTimeout: time.Second * 5,
+			want:          3,
+			wantErr:       true,
+		},
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
 			r := New()
+			if tt.cancelTimeout != 0 {
+				newCtx, cancelFn := context.WithTimeout(tt.ctx, tt.cancelTimeout)
+				tt.ctx = newCtx
+				defer cancelFn()
+			}
 			got, err := r.Retry(tt.ctx, tt.retryableFunc)
 			if (err != nil) != tt.wantErr {
 				t.Errorf("Retry() error = %v, wantErr %v", err, tt.wantErr)