Repository: kubernetes-csi/external-snapshotter
Pull request #308 (2020-04-28 19:33:15)

Add ability to customize VolumeSnapshotContent workqueue
Two new timeout values ( retryIntervalStart & retryIntervalMax )
have been added to set the ratelimiter for volumesnapshotcontent queue.

Fixes:  https://github.com/kubernetes-csi/external-snapshotter/issues/463

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>


```release-note
 `retry-interval-start` and `retry-interval-max` arguments are added to csi-snapshotter sidecar which controls retry interval of failed volume snapshot creation and deletion. These values set the ratelimiter for volumesnapshotcontent queue.
```


k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-csi/external-snapshotter/pull/308#" title="Author self-approved">humblec</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **xing-yang**
You can assign the PR to them by writing `/assign @xing-yang` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-csi%2Fexternal-snapshotter).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-csi/external-snapshotter/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["xing-yang"]} -->
humblec: @xing-yang @yuxiangqian ptal . thanks
xing-yang: @humblec Thanks for adding this. Requeue currently does not work properly so adding these options won't really do anything.  I'm working on a fix here: https://github.com/kubernetes-csi/external-snapshotter/pull/230.  After that we can get your fix in.
humblec: > @humblec Thanks for adding this. Requeue currently does not work properly so adding these options won't really do anything. I'm working on a fix here: #230. After that we can get your fix in.

@xing-yang sure. Will wait for other PR to go in first. 
xing-yang: @humblec can you rebase?
k8s-ci-robot: @humblec: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
k8s-ci-robot: @humblec: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-kubernetes-csi-external-snapshotter-unit | ffae1d9417162ea708faea8a851033539a6fa27e | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-unit/1263742471406358534) | `/test pull-kubernetes-csi-external-snapshotter-unit`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-csi&repo=external-snapshotter&pr=308). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3humblec). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-csi/external-snapshotter/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
humblec: > @humblec can you rebase?

@xing-yang  Sure, it is done. :+1:  ptal. 
humblec: @xing-yang comments are addressed and also added release notes. Please let me know if any changes are required in any of it.
yuxiangqian: I am a little concerned for this change as it would change the behavior of the rateLimiter slightly.
1. how should this be effectively tested? I understand there are no existing tests to cover, but can we consider covering in this PR?
2. the original implementation uses the default controller rate limiter, which under the hood is just a combination of exponential failure rate limiter and a bucket rate limiter. For the exponential failure rate limiter, it uses 1 ms as the base. In this PR, the default base is 1s.
What are the main purposes of this change? is it just to allow users to specify the base and max-retry interval?

humblec: @yuxiangqian regarding the name of these parameters: While I agree that we could change or use some strings like `retry-interval-base` ..etc, we are actually deviating from the same parameters available in other sidecars and the parity between those. For ex: external provisioner/attacher (https://github.com/kubernetes-csi/external-attacher) too have these parameters.  Regarding the default value, we could change that to `1ms` or whatever preferred value , Is `1ms` good enough? 
humblec: > @yuxiangqian regarding the name of these parameters: While I agree that we could change or use some strings like `retry-interval-base` ..etc, we are actually deviating from the same parameters available in other sidecars and the parity between those. For ex: external provisioner/attacher (https://github.com/kubernetes-csi/external-attacher) too have these parameters. Regarding the default value, we could change that to `1ms` or whatever preferred value , Is `1ms` good enough?

@xing-yang can you share your thoughts.. I would like to get some consensus and hopefully finish this PR .
xing-yang: >> @yuxiangqian regarding the name of these parameters: While I agree that we could change or use some strings like retry-interval-base ..etc, we are actually deviating from the same parameters available in other sidecars and the parity between those. For ex: external provisioner/attacher (https://github.com/kubernetes-csi/external-attacher) too have these parameters. Regarding the default value, we could change that to 1ms or whatever preferred value , Is 1ms good enough?

> @xing-yang can you share your thoughts.. I would like to get some consensus and hopefully finish this PR .

I agree that the naming in externa-snapshotter should be consistent with that in other sidecars.  @yuxiangqian are you okay with this PR?
yuxiangqian: > @yuxiangqian regarding the name of these parameters: While I agree that we could change or use some strings like `retry-interval-base` ..etc, we are actually deviating from the same parameters available in other sidecars and the parity between those. For ex: external provisioner/attacher (https://github.com/kubernetes-csi/external-attacher) too have these parameters. Regarding the default value, we could change that to `1ms` or whatever preferred value , Is `1ms` good enough?
>
@humblec dropped the ball a bit, my apologies. If its consistent with others, I am good with  the name.
I still have concerns over the default value of 1s as users likely would just rely on the default. Snapshot is latency sensitive and that's different from attacher. I'd  suggest add some tests to that and default to the original 1ms starting. and in  framework_test.go, should we also use the same ratelimiter as in main.go.
k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-csi/external-snapshotter/pull/308#" title="Author self-approved">humblec</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **xing-yang**
You can assign the PR to them by writing `/assign @xing-yang` in a comment when ready.

The full list of commands accepted by this bot can be found [here](?repo=kubernetes-csi%2Fexternal-snapshotter).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-csi/external-snapshotter/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["xing-yang"]} -->
k8s-ci-robot: @humblec: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-kubernetes-csi-external-snapshotter-1-17-on-kubernetes-1-17 | 0bd41ab73679e711305a1002f4ea625b845b1b60 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-1-17-on-kubernetes-1-17/1277608447004643328) | `/test pull-kubernetes-csi-external-snapshotter-1-17-on-kubernetes-1-17`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-csi&repo=external-snapshotter&pr=308). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3humblec). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-csi/external-snapshotter/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
humblec: /retest
fejta-bot: Issues go stale after 90d of inactivity.
Mark the issue as fresh with `/remove-lifecycle stale`.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with `/close`.

Send feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta).
/lifecycle stale
k8s-ci-robot: @humblec: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
fejta-bot: Stale issues rot after 30d of inactivity.
Mark the issue as fresh with `/remove-lifecycle rotten`.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with `/close`.

Send feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta).
/lifecycle rotten
humblec: /remove-lifecycle rotten
fejta-bot: Issues go stale after 90d of inactivity.
Mark the issue as fresh with `/remove-lifecycle stale`.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with `/close`.

Send feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta).
/lifecycle stale
fejta-bot: Stale issues rot after 30d of inactivity.
Mark the issue as fresh with `/remove-lifecycle rotten`.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with `/close`.

Send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).
/lifecycle rotten
fejta-bot: Rotten issues close after 30d of inactivity.
Reopen the issue with `/reopen`.
Mark the issue as fresh with `/remove-lifecycle rotten`.

Send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).
/close
k8s-ci-robot: @fejta-bot: Closed this PR.

<details>

In response to [this](https://github.com/kubernetes-csi/external-snapshotter/pull/308#issuecomment-812564993):

>Rotten issues close after 30d of inactivity.
>Reopen the issue with `/reopen`.
>Mark the issue as fresh with `/remove-lifecycle rotten`.
>
>Send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).
>/close


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
k8s-ci-robot: @humblec: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-kubernetes-csi-external-snapshotter-1-18-on-kubernetes-1-18 | 0bd41ab73679e711305a1002f4ea625b845b1b60 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-1-18-on-kubernetes-1-18/1400444265191968768) | `/test pull-kubernetes-csi-external-snapshotter-1-18-on-kubernetes-1-18`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-csi&repo=external-snapshotter&pr=308). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Ahumblec). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-csi/external-snapshotter/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @humblec: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-kubernetes-csi-external-snapshotter-1-18-on-kubernetes-1-18 | 0bd41ab73679e711305a1002f4ea625b845b1b60 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-1-18-on-kubernetes-1-18/1400444265191968768) | `/test pull-kubernetes-csi-external-snapshotter-1-18-on-kubernetes-1-18`
pull-kubernetes-csi-external-snapshotter-1-19-on-kubernetes-1-19 | 0bd41ab73679e711305a1002f4ea625b845b1b60 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-1-19-on-kubernetes-1-19/1400444265225523200) | `/test pull-kubernetes-csi-external-snapshotter-1-19-on-kubernetes-1-19`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-csi&repo=external-snapshotter&pr=308). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Ahumblec). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-csi/external-snapshotter/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @humblec: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-kubernetes-csi-external-snapshotter-1-18-on-kubernetes-1-18 | 0bd41ab73679e711305a1002f4ea625b845b1b60 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-1-18-on-kubernetes-1-18/1400444265191968768) | `/test pull-kubernetes-csi-external-snapshotter-1-18-on-kubernetes-1-18`
pull-kubernetes-csi-external-snapshotter-1-19-on-kubernetes-1-19 | 0bd41ab73679e711305a1002f4ea625b845b1b60 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-1-19-on-kubernetes-1-19/1400444265225523200) | `/test pull-kubernetes-csi-external-snapshotter-1-19-on-kubernetes-1-19`
pull-kubernetes-csi-external-snapshotter-unit | 0bd41ab73679e711305a1002f4ea625b845b1b60 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-unit/1400444265330380800) | `/test pull-kubernetes-csi-external-snapshotter-unit`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-csi&repo=external-snapshotter&pr=308). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Ahumblec). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-csi/external-snapshotter/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-csi/external-snapshotter/pull/308#" title="Author self-approved">humblec</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **saad-ali** after the PR has been reviewed.
You can assign the PR to them by writing `/assign @saad-ali` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-csi%2Fexternal-snapshotter).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-csi/external-snapshotter/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["saad-ali"]} -->
k8s-ci-robot: @humblec: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-kubernetes-csi-external-snapshotter-1-18-on-kubernetes-1-18 | 0bd41ab73679e711305a1002f4ea625b845b1b60 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-1-18-on-kubernetes-1-18/1400444265191968768) | `/test pull-kubernetes-csi-external-snapshotter-1-18-on-kubernetes-1-18`
pull-kubernetes-csi-external-snapshotter-1-19-on-kubernetes-1-19 | 0bd41ab73679e711305a1002f4ea625b845b1b60 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-1-19-on-kubernetes-1-19/1400444265225523200) | `/test pull-kubernetes-csi-external-snapshotter-1-19-on-kubernetes-1-19`
pull-kubernetes-csi-external-snapshotter-unit | 0bd41ab73679e711305a1002f4ea625b845b1b60 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-unit/1400444265330380800) | `/test pull-kubernetes-csi-external-snapshotter-unit`
pull-kubernetes-csi-external-snapshotter-1-20-on-kubernetes-1-20 | 0bd41ab73679e711305a1002f4ea625b845b1b60 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-1-20-on-kubernetes-1-20/1400444265263271936) | `/test pull-kubernetes-csi-external-snapshotter-1-20-on-kubernetes-1-20`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-csi&repo=external-snapshotter&pr=308). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Ahumblec). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-csi/external-snapshotter/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @humblec: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-kubernetes-csi-external-snapshotter-1-18-on-kubernetes-1-18 | 0bd41ab73679e711305a1002f4ea625b845b1b60 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-1-18-on-kubernetes-1-18/1400444265191968768) | `/test pull-kubernetes-csi-external-snapshotter-1-18-on-kubernetes-1-18`
pull-kubernetes-csi-external-snapshotter-1-19-on-kubernetes-1-19 | 0bd41ab73679e711305a1002f4ea625b845b1b60 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-1-19-on-kubernetes-1-19/1400444265225523200) | `/test pull-kubernetes-csi-external-snapshotter-1-19-on-kubernetes-1-19`
pull-kubernetes-csi-external-snapshotter-unit | 0bd41ab73679e711305a1002f4ea625b845b1b60 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-unit/1400444265330380800) | `/test pull-kubernetes-csi-external-snapshotter-unit`
pull-kubernetes-csi-external-snapshotter-1-20-on-kubernetes-1-20 | 0bd41ab73679e711305a1002f4ea625b845b1b60 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-1-20-on-kubernetes-1-20/1400444265263271936) | `/test pull-kubernetes-csi-external-snapshotter-1-20-on-kubernetes-1-20`
pull-kubernetes-csi-external-snapshotter-1-21-on-kubernetes-1-21 | 0bd41ab73679e711305a1002f4ea625b845b1b60 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-1-21-on-kubernetes-1-21/1400444265292632064) | `/test pull-kubernetes-csi-external-snapshotter-1-21-on-kubernetes-1-21`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-csi&repo=external-snapshotter&pr=308). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Ahumblec). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-csi/external-snapshotter/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
xing-yang: @humblec can you reference this issue https://github.com/kubernetes-csi/external-snapshotter/issues/463 in the top section of this PR?
humblec: > @humblec can you reference this issue #463 in the top section of this PR?

Sure thing @xing-yang and thanks for reopening this PR, let me rebase and finish it off :+1: 
k8s-ci-robot: @humblec: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-kubernetes-csi-external-snapshotter-1-18-on-kubernetes-1-18 | 0bd41ab73679e711305a1002f4ea625b845b1b60 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-1-18-on-kubernetes-1-18/1400444265191968768) | `/test pull-kubernetes-csi-external-snapshotter-1-18-on-kubernetes-1-18`
pull-kubernetes-csi-external-snapshotter-1-19-on-kubernetes-1-19 | 0bd41ab73679e711305a1002f4ea625b845b1b60 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-1-19-on-kubernetes-1-19/1400444265225523200) | `/test pull-kubernetes-csi-external-snapshotter-1-19-on-kubernetes-1-19`
pull-kubernetes-csi-external-snapshotter-1-20-on-kubernetes-1-20 | 0bd41ab73679e711305a1002f4ea625b845b1b60 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-1-20-on-kubernetes-1-20/1400444265263271936) | `/test pull-kubernetes-csi-external-snapshotter-1-20-on-kubernetes-1-20`
pull-kubernetes-csi-external-snapshotter-1-21-on-kubernetes-1-21 | 0bd41ab73679e711305a1002f4ea625b845b1b60 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-1-21-on-kubernetes-1-21/1400444265292632064) | `/test pull-kubernetes-csi-external-snapshotter-1-21-on-kubernetes-1-21`
pull-kubernetes-csi-external-snapshotter-unit | 2fba1356bf5f12ec28c0ed547751b2226b6d5846 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-unit/1401886231482077184) | `/test pull-kubernetes-csi-external-snapshotter-unit`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-csi&repo=external-snapshotter&pr=308). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Ahumblec). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-csi/external-snapshotter/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
humblec: The default value of the ratelimiter has been set to `1ms`.  Apart from that, one thing to note here is,  framework testing does not make use of this new rate limiter yet ie it use the defaultRateLimiter.  If required, I can change framework, otherwise we are good with the change .
k8s-ci-robot: @humblec: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-kubernetes-csi-external-snapshotter-unit | 74a0b7a101e1a011ca5ea363a1f04a28af10f7db | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-unit/1401924697498587136) | `/test pull-kubernetes-csi-external-snapshotter-unit`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-csi&repo=external-snapshotter&pr=308). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Ahumblec). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-csi/external-snapshotter/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
xing-yang: /remove-lifecycle rotten
k8s-ci-robot: @humblec: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-kubernetes-csi-external-snapshotter-unit | 5afda69b9abb3a681313a1e30ba2976826769bbc | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_external-snapshotter/308/pull-kubernetes-csi-external-snapshotter-unit/1402869492396068864) | `/test pull-kubernetes-csi-external-snapshotter-unit`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-csi&repo=external-snapshotter&pr=308). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Ahumblec). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-csi/external-snapshotter/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
humblec: @xing-yang ptal.. Thanks !

Patch:
diff --git a/README.md b/README.md
index 6af01c3ba..a16d05c32 100644
--- a/README.md
+++ b/README.md
@@ -127,6 +127,9 @@ Read more about how to install the example webhook [here](deploy/kubernetes/webh
 
 * `--worker-threads`: Number of worker threads. Default value is 10.
 
+* `--retry-interval-start`: Initial retry interval of failed volume snapshot creation or deletion. It doubles with each failure, up to retry-interval-max. Default value is 1 second.
+
+*`--retry-interval-max`: Maximum retry interval of failed volume snapshot creation or deletion. Default value is 5 minutes.
 #### Other recognized arguments
 * `--kubeconfig <path>`: Path to Kubernetes client configuration that the snapshot controller uses to connect to Kubernetes API server. When omitted, default token provided by Kubernetes will be used. This option is useful only when the snapshot controller does not run as a Kubernetes pod, e.g. for debugging.
 
diff --git a/cmd/csi-snapshotter/main.go b/cmd/csi-snapshotter/main.go
index afdfcad27..a32875496 100644
--- a/cmd/csi-snapshotter/main.go
+++ b/cmd/csi-snapshotter/main.go
@@ -32,6 +32,7 @@ import (
 	"k8s.io/client-go/kubernetes/scheme"
 	"k8s.io/client-go/rest"
 	"k8s.io/client-go/tools/clientcmd"
+	"k8s.io/client-go/util/workqueue"
 	klog "k8s.io/klog/v2"
 
 	"github.com/container-storage-interface/spec/lib/go/csi"
@@ -57,7 +58,7 @@ const (
 var (
 	kubeconfig             = flag.String("kubeconfig", "", "Absolute path to the kubeconfig file. Required only when running out of cluster.")
 	csiAddress             = flag.String("csi-address", "/run/csi/socket", "Address of the CSI driver socket.")
-	resyncPeriod           = flag.Duration("resync-period", 15*time.Minute, "Resync interval of the controller.")
+	resyncPeriod           = flag.Duration("resync-period", 15*time.Minute, "Resync interval of the controller. Default is 15 minutes")
 	snapshotNamePrefix     = flag.String("snapshot-name-prefix", "snapshot", "Prefix to apply to the name of a created snapshot")
 	snapshotNameUUIDLength = flag.Int("snapshot-name-uuid-length", -1, "Length in characters for the generated uuid of a created snapshot. Defaults behavior is to NOT truncate.")
 	showVersion            = flag.Bool("version", false, "Show version.")
@@ -68,9 +69,11 @@ var (
 	leaderElection          = flag.Bool("leader-election", false, "Enables leader election.")
 	leaderElectionNamespace = flag.String("leader-election-namespace", "", "The namespace where the leader election resource exists. Defaults to the pod namespace if not set.")
 
-	metricsAddress = flag.String("metrics-address", "", "(deprecated) The TCP network address where the prometheus metrics endpoint will listen (example: `:8080`). The default is empty string, which means metrics endpoint is disabled. Only one of `--metrics-address` and `--http-endpoint` can be set.")
-	httpEndpoint   = flag.String("http-endpoint", "", "The TCP network address where the HTTP server for diagnostics, including metrics and leader election health check, will listen (example: `:8080`). The default is empty string, which means the server is disabled. Only one of `--metrics-address` and `--http-endpoint` can be set.")
-	metricsPath    = flag.String("metrics-path", "/metrics", "The HTTP path where prometheus metrics will be exposed. Default is `/metrics`.")
+	metricsAddress     = flag.String("metrics-address", "", "(deprecated) The TCP network address where the prometheus metrics endpoint will listen (example: `:8080`). The default is empty string, which means metrics endpoint is disabled. Only one of `--metrics-address` and `--http-endpoint` can be set.")
+	httpEndpoint       = flag.String("http-endpoint", "", "The TCP network address where the HTTP server for diagnostics, including metrics and leader election health check, will listen (example: `:8080`). The default is empty string, which means the server is disabled. Only one of `--metrics-address` and `--http-endpoint` can be set.")
+	metricsPath        = flag.String("metrics-path", "/metrics", "The HTTP path where prometheus metrics will be exposed. Default is `/metrics`.")
+	retryIntervalStart = flag.Duration("retry-interval-start", time.Second, "Initial retry interval of failed volume snapshot creation or deletion. It doubles with each failure, up to retry-interval-max. Default is 1 second")
+	retryIntervalMax   = flag.Duration("retry-interval-max", 5*time.Minute, "Maximum retry interval of failed volume snapshot creation or deletion. Default is 5 minutes")
 )
 
 var (
@@ -198,6 +201,7 @@ func main() {
 		*snapshotNamePrefix,
 		*snapshotNameUUIDLength,
 		*extraCreateMetadata,
+		workqueue.NewItemExponentialFailureRateLimiter(*retryIntervalStart, *retryIntervalMax),
 	)
 
 	run := func(context.Context) {
diff --git a/pkg/sidecar-controller/snapshot_controller_base.go b/pkg/sidecar-controller/snapshot_controller_base.go
index 56b9849ba..0d2030652 100644
--- a/pkg/sidecar-controller/snapshot_controller_base.go
+++ b/pkg/sidecar-controller/snapshot_controller_base.go
@@ -72,6 +72,7 @@ func NewCSISnapshotSideCarController(
 	snapshotNamePrefix string,
 	snapshotNameUUIDLength int,
 	extraCreateMetadata bool,
+	contentRateLimiter workqueue.RateLimiter,
 ) *csiSnapshotSideCarController {
 	broadcaster := record.NewBroadcaster()
 	broadcaster.StartLogging(klog.Infof)
@@ -87,7 +88,7 @@ func NewCSISnapshotSideCarController(
 		handler:             NewCSIHandler(snapshotter, timeout, snapshotNamePrefix, snapshotNameUUIDLength),
 		resyncPeriod:        resyncPeriod,
 		contentStore:        cache.NewStore(cache.DeletionHandlingMetaNamespaceKeyFunc),
-		contentQueue:        workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "csi-snapshotter-content"),
+		contentQueue:        workqueue.NewNamedRateLimitingQueue(contentRateLimiter, "csi-snapshotter-content"),
 		extraCreateMetadata: extraCreateMetadata,
 	}
 


Test patch:
diff --git a/pkg/sidecar-controller/framework_test.go b/pkg/sidecar-controller/framework_test.go
index 7a8f0f0e8..078cfd0a0 100644
--- a/pkg/sidecar-controller/framework_test.go
+++ b/pkg/sidecar-controller/framework_test.go
@@ -46,6 +46,7 @@ import (
 	core "k8s.io/client-go/testing"
 	"k8s.io/client-go/tools/cache"
 	"k8s.io/client-go/tools/record"
+	"k8s.io/client-go/util/workqueue"
 	klog "k8s.io/klog/v2"
 )
 
@@ -522,6 +523,7 @@ func newTestController(kubeClient kubernetes.Interface, clientset clientset.Inte
 		"snapshot",
 		-1,
 		true,
+		workqueue.NewItemExponentialFailureRateLimiter(1*time.Millisecond, 1*time.Minute),
 	)
 
 	ctrl.eventRecorder = record.NewFakeRecorder(1000)