Repository: kubernetes-sigs/gcp-compute-persistent-disk-csi-driver
Pull request #847 (2021-09-28 23:15:31)

Attach/Detach back off
**What type of PR is this?**
/kind feature

**What this PR does / why we need it**:
In GCE, Attach/Detach operations have a 32-deep queue on VM. Added a workqueue for Attach/Detach calls with a rate limiter.

**Special notes for your reviewer**:

**Does this PR introduce a user-facing change?**:
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".
-->
```release-note
NONE
```


k8s-ci-robot: Hi @lizhuqi. Thanks for your PR.

I'm waiting for a [kubernetes-sigs](https://github.com/orgs/kubernetes-sigs/people) member to verify that this patch is reasonable to test. If it is, they should reply with `/ok-to-test` on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should [join the org](https://git.k8s.io/community/community-membership.md#member) to skip this step.

Once the patch is verified, the new status will be reflected by the `ok-to-test` label.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fgcp-compute-persistent-disk-csi-driver).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>

k8s-ci-robot: Welcome @lizhuqi! <br><br>It looks like this is your first PR to <a href='https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver'>kubernetes-sigs/gcp-compute-persistent-disk-csi-driver</a> 🎉. Please refer to our [pull request process documentation](https://git.k8s.io/community/contributors/guide/pull-requests.md) to help your PR have a smooth ride to approval. <br><br>You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. [Here is the bot commands documentation](https://go.k8s.io/bot-commands). <br><br>You can also check if kubernetes-sigs/gcp-compute-persistent-disk-csi-driver has [its own contribution guidelines](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/tree/master/CONTRIBUTING.md). <br><br>You may want to refer to our [testing guide](https://git.k8s.io/community/contributors/devel/sig-testing/testing.md) if you run into trouble with your tests not passing. <br><br>If you are having difficulty getting your pull request seen, please follow the [recommended escalation practices](https://github.com/kubernetes/community/blob/master/contributors/guide/pull-requests.md#why-is-my-pull-request-not-getting-reviewed). Also, for tips and tricks in the contribution process you may want to read the [Kubernetes contributor cheat sheet](https://git.k8s.io/community/contributors/guide/contributor-cheatsheet/README.md). We want to make sure your contribution gets all the attention it needs! <br><br>Thank you, and welcome to Kubernetes. :smiley:
k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/pull/847#" title="Author self-approved">lizhuqi</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **saikat-royc** after the PR has been reviewed.
You can assign the PR to them by writing `/assign @saikat-royc` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fgcp-compute-persistent-disk-csi-driver).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["saikat-royc"]} -->
lizhuqi: /assign mattcary
jingxu97: Some scalability tests would be good to understand performance impact about this change
mattcary: /ok-to-test

Offline discussion: the problems we've been seeing in practice occur when a particular node starts to error (ie, the per-vm operation queue in GCE fills up causing attach requests to immediately return with an error until the queue drains). So Julie's looking into making the rate limiting be more fine-grained, at least to the node level and maybe only on error as well.

We don't want, eg, scale-ups for large clusters to be unnecessarily rate-limited when the requests per node are not high.
k8s-ci-robot: @lizhuqi: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-gcp-compute-persistent-disk-csi-driver-sanity | c388521cce50fc77a3f72defd91ec6d5d492e01d | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/847/pull-gcp-compute-persistent-disk-csi-driver-sanity/1443268880079261696/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-sanity`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=gcp-compute-persistent-disk-csi-driver&pr=847). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alizhuqi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lizhuqi: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-gcp-compute-persistent-disk-csi-driver-sanity | c388521cce50fc77a3f72defd91ec6d5d492e01d | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/847/pull-gcp-compute-persistent-disk-csi-driver-sanity/1443268880079261696/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-sanity`
pull-gcp-compute-persistent-disk-csi-driver-e2e | c388521cce50fc77a3f72defd91ec6d5d492e01d | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/847/pull-gcp-compute-persistent-disk-csi-driver-e2e/1443268880054095872/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-e2e`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=gcp-compute-persistent-disk-csi-driver&pr=847). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alizhuqi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/pull/847#" title="Author self-approved">lizhuqi</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please ask for approval from **mattcary** after the PR has been reviewed.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fgcp-compute-persistent-disk-csi-driver).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["mattcary"]} -->
lizhuqi: /retest
lizhuqi: /test pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration
k8s-ci-robot: @lizhuqi: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration | b98fe765faeee6354201e7de6faf9ff0cab050ad | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/847/pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration/1446008761494802432/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=gcp-compute-persistent-disk-csi-driver&pr=847). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alizhuqi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
lizhuqi: /cc @saikat-royc 
lizhuqi: /retest
k8s-ci-robot: @lizhuqi: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration | 732991aaa3651328b23c3913123d86505e0a6357 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/847/pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration/1450610847519870976/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration`
pull-gcp-compute-persistent-disk-csi-driver-verify | 732991aaa3651328b23c3913123d86505e0a6357 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/847/pull-gcp-compute-persistent-disk-csi-driver-verify/1456416748298309632/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-verify`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=gcp-compute-persistent-disk-csi-driver&pr=847). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alizhuqi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lizhuqi: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-gcp-compute-persistent-disk-csi-driver-verify | 732991aaa3651328b23c3913123d86505e0a6357 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/847/pull-gcp-compute-persistent-disk-csi-driver-verify/1456416748298309632/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-verify`
pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration | 732991aaa3651328b23c3913123d86505e0a6357 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/847/pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration/1456416748319281152/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=gcp-compute-persistent-disk-csi-driver&pr=847). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alizhuqi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lizhuqi: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration | 732991aaa3651328b23c3913123d86505e0a6357 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/847/pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration/1456416748319281152/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration`
pull-gcp-compute-persistent-disk-csi-driver-unit | 4e26a1ddcc40ea9ba59ee58386a0dc0a4970f04b | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/847/pull-gcp-compute-persistent-disk-csi-driver-unit/1457881411523448832/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-unit`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=gcp-compute-persistent-disk-csi-driver&pr=847). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alizhuqi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lizhuqi: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration | 732991aaa3651328b23c3913123d86505e0a6357 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/847/pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration/1456416748319281152/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration`
pull-gcp-compute-persistent-disk-csi-driver-verify | baa25dc6fc2be1b98795eafa4704222dae5ced72 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/847/pull-gcp-compute-persistent-disk-csi-driver-verify/1457891634921345024/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-verify`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=gcp-compute-persistent-disk-csi-driver&pr=847). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alizhuqi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lizhuqi: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration | 732991aaa3651328b23c3913123d86505e0a6357 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/847/pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration/1456416748319281152/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration`
pull-gcp-compute-persistent-disk-csi-driver-verify | baa25dc6fc2be1b98795eafa4704222dae5ced72 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/847/pull-gcp-compute-persistent-disk-csi-driver-verify/1457891634921345024/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-verify`
pull-gcp-compute-persistent-disk-csi-driver-unit | baa25dc6fc2be1b98795eafa4704222dae5ced72 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/847/pull-gcp-compute-persistent-disk-csi-driver-unit/1457891634879401984/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-unit`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=gcp-compute-persistent-disk-csi-driver&pr=847). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alizhuqi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lizhuqi: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration | 732991aaa3651328b23c3913123d86505e0a6357 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/847/pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration/1456416748319281152/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration`
pull-gcp-compute-persistent-disk-csi-driver-unit | baa25dc6fc2be1b98795eafa4704222dae5ced72 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/847/pull-gcp-compute-persistent-disk-csi-driver-unit/1457891634879401984/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-unit`
pull-gcp-compute-persistent-disk-csi-driver-verify | 82a96d9e02ab1d9d65628571974145f921343af0 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/847/pull-gcp-compute-persistent-disk-csi-driver-verify/1457892338352263168/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-verify`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=gcp-compute-persistent-disk-csi-driver&pr=847). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alizhuqi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lizhuqi: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-gcp-compute-persistent-disk-csi-driver-verify | 82a96d9e02ab1d9d65628571974145f921343af0 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/847/pull-gcp-compute-persistent-disk-csi-driver-verify/1457892338352263168/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-verify`
pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration | 82a96d9e02ab1d9d65628571974145f921343af0 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/847/pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration/1457892338381623296/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=gcp-compute-persistent-disk-csi-driver&pr=847). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alizhuqi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lizhuqi: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration | a5507168bd8c43ba0d9b1db0f44f9e813aa2786b | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/847/pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration/1458196203601661952/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=gcp-compute-persistent-disk-csi-driver&pr=847). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alizhuqi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lizhuqi: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration | 30e964c6969b9198f5c71e35a075e00e1a905cff | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/847/pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration/1458603927657254912/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=gcp-compute-persistent-disk-csi-driver&pr=847). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alizhuqi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lizhuqi: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration | b5b0e9b51ab8d07e43507ff3c60cb8675769a068 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/847/pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration/1460325275714719744/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-kubernetes-integration`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=gcp-compute-persistent-disk-csi-driver&pr=847). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alizhuqi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->

Patch:
diff --git a/CHANGELOG/CHANGELOG-1.0.md b/CHANGELOG/CHANGELOG-1.0.md
index ad4c8763..f846b701 100644
--- a/CHANGELOG/CHANGELOG-1.0.md
+++ b/CHANGELOG/CHANGELOG-1.0.md
@@ -1,28 +1,3 @@
-# v1.0.4 - Changelog since v1.0.2
-
-### Other
-
-- Fix build for various go versions, ([#832](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/pull/832), [@mattcary](https://github.com/mattcary))
-
-# v1.0.3 - **Bad tag, skipped**
-
-# v1.0.2 - Changelog since v1.0.1
-
-### Bug or Regression
-
-- Update base image to buster-v.1.9.0 ([#829](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/pull/829), [@mattcary](https://github.com/mattcary))
-
-## Dependencies
-
-### Added
-_Nothing has changed._
-
-### Changed
-_Nothing has changed._
-
-### Removed
-_Nothing has changed._
-
 # v1.0.1 - Changelog since v1.0.0
 
 ## Changes by Kind
diff --git a/CHANGELOG/CHANGELOG-1.2.md b/CHANGELOG/CHANGELOG-1.2.md
index b4c82010..74013d33 100644
--- a/CHANGELOG/CHANGELOG-1.2.md
+++ b/CHANGELOG/CHANGELOG-1.2.md
@@ -1,45 +1,3 @@
-# v1.2.4 - Changelog since v1.2.3
-
-## Changes by Kind
-
-### Bug or Regression
-
-- Update go builder to 1.17 ([#850](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/pull/850), [@mattcary](https://github.com/mattcary))
-
-## Dependencies
-
-### Added
-_Nothing has changed._
-
-### Changed
-_Nothing has changed._
-
-### Removed
-_Nothing has changed._
-
-# v1.2.3 - Changelog since v1.2.1
-
-## Changes by Kind
-
-### Bug or Regression
-
-- Update debian images to buster-v1.9.0 ([#839](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/pull/839), [@mattcary](https://github.com/mattcary))
-
-### Uncategorized
-
-- It is now possible to mount a volume with XFS filesystem and its restored snapshot. ([#838](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/pull/838), [@leiyiz](https://github.com/leiyiz))
-
-## Dependencies
-
-### Added
-_Nothing has changed._
-
-### Changed
-_Nothing has changed._
-
-### Removed
-_Nothing has changed._
-
 # v1.2.2 - Changelog since v1.2.1
 
 ## Changes by Kind
diff --git a/CHANGELOG/CHANGELOG-1.3.md b/CHANGELOG/CHANGELOG-1.3.md
index b0137eb7..c6de2f3a 100644
--- a/CHANGELOG/CHANGELOG-1.3.md
+++ b/CHANGELOG/CHANGELOG-1.3.md
@@ -1,41 +1,3 @@
-# v1.3.4 - Changelog since v1.3.3
-
-## Changes by Kind
-
-### Bug or Regression
-
-- Update go builder to 1.17 ([#850](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/pull/850), [@mattcary](https://github.com/mattcary))
-
-## Dependencies
-
-### Added
-_Nothing has changed._
-
-### Changed
-_Nothing has changed._
-
-### Removed
-_Nothing has changed._
-
-# v1.3.3 - Changelog since v1.3.1
-
-## Changes by Kind
-
-### Bug or Regression
-
-- Update debian image to buster-1.9.0. ([#841](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/pull/841), [@mattcary](https://github.com/mattcary))
-
-## Dependencies
-
-### Added
-_Nothing has changed._
-
-### Changed
-_Nothing has changed._
-
-### Removed
-_Nothing has changed._
-
 # v1.3.1 - Changelog since v1.3.0
 
 ### Issues
diff --git a/Dockerfile b/Dockerfile
index 1b39767d..29d3634c 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -23,44 +23,15 @@ WORKDIR /go/src/sigs.k8s.io/gcp-compute-persistent-disk-csi-driver
 ADD . .
 RUN GOARCH=$(echo $TARGETPLATFORM | cut -f2 -d '/') GCE_PD_CSI_STAGING_VERSION=$STAGINGVERSION make gce-pd-driver
 
-# Start from Kubernetes Debian base.
-FROM k8s.gcr.io/build-image/debian-base:buster-v1.9.0 as debian
+# MAD HACKS: Build a version first so we can take the scsi_id bin and put it somewhere else in our real build
+FROM k8s.gcr.io/build-image/debian-base:buster-v1.9.0 as mad-hack
+RUN clean-install udev
+
+# Start from Kubernetes Debian base
+FROM k8s.gcr.io/build-image/debian-base:buster-v1.9.0
+COPY --from=builder /go/src/sigs.k8s.io/gcp-compute-persistent-disk-csi-driver/bin/gce-pd-csi-driver /gce-pd-csi-driver
 # Install necessary dependencies
 RUN clean-install util-linux e2fsprogs mount ca-certificates udev xfsprogs
-# Since we're leveraging apt to pull in dependencies, we use `gcr.io/distroless/base` because it includes glibc.
-FROM gcr.io/distroless/base-debian11
-# Copy necessary dependencies into distroless base.
-COPY --from=builder /go/src/sigs.k8s.io/gcp-compute-persistent-disk-csi-driver/bin/gce-pd-csi-driver /gce-pd-csi-driver
-COPY --from=debian /etc/mke2fs.conf /etc/mke2fs.conf
-COPY --from=debian /lib/udev/scsi_id /lib/udev_containerized/scsi_id
-COPY --from=debian /bin/mount /bin/mount
-COPY --from=debian /bin/umount /bin/umount
-COPY --from=debian /sbin/blkid /sbin/blkid
-COPY --from=debian /sbin/blockdev /sbin/blockdev
-COPY --from=debian /sbin/dumpe2fs /sbin/dumpe2fs
-COPY --from=debian /sbin/e* /sbin/
-COPY --from=debian /sbin/e2fsck /sbin/e2fsck
-COPY --from=debian /sbin/fsck /sbin/fsck
-COPY --from=debian /sbin/fsck* /sbin/
-COPY --from=debian /sbin/fsck.xfs /sbin/fsck.xfs
-COPY --from=debian /sbin/mke2fs /sbin/mke2fs
-COPY --from=debian /sbin/mkfs* /sbin/
-COPY --from=debian /sbin/resize2fs /sbin/resize2fs
-COPY --from=debian /sbin/xfs_repair /sbin/xfs_repair
-COPY --from=debian /usr/include/xfs /usr/include/xfs
-COPY --from=debian /usr/lib/xfsprogs/xfs* /usr/lib/xfsprogs/
-COPY --from=debian /usr/sbin/xfs* /usr/sbin/
-
-# Copy x86 shared libraries into distroless base.
-COPY --from=debian /lib/x86_64-linux-gnu/libblkid.so.1 /lib/x86_64-linux-gnu/libblkid.so.1
-COPY --from=debian /lib/x86_64-linux-gnu/libcom_err.so.2 /lib/x86_64-linux-gnu/libcom_err.so.2
-COPY --from=debian /lib/x86_64-linux-gnu/libext2fs.so.2 /lib/x86_64-linux-gnu/libext2fs.so.2
-COPY --from=debian /lib/x86_64-linux-gnu/libe2p.so.2 /lib/x86_64-linux-gnu/libe2p.so.2
-COPY --from=debian /lib/x86_64-linux-gnu/libmount.so.1 /lib/x86_64-linux-gnu/libmount.so.1
-COPY --from=debian /lib/x86_64-linux-gnu/libpcre.so.3 /lib/x86_64-linux-gnu/libpcre.so.3
-COPY --from=debian /lib/x86_64-linux-gnu/libreadline.so.5 /lib/x86_64-linux-gnu/libreadline.so.5
-COPY --from=debian /lib/x86_64-linux-gnu/libselinux.so.1 /lib/x86_64-linux-gnu/libselinux.so.1
-COPY --from=debian /lib/x86_64-linux-gnu/libtinfo.so.6 /lib/x86_64-linux-gnu/libtinfo.so.6
-COPY --from=debian /lib/x86_64-linux-gnu/libuuid.so.1 /lib/x86_64-linux-gnu/libuuid.so.1
+COPY --from=mad-hack /lib/udev/scsi_id /lib/udev_containerized/scsi_id
 
-ENTRYPOINT ["/gce-pd-csi-driver"]
\ No newline at end of file
+ENTRYPOINT ["/gce-pd-csi-driver"]
diff --git a/Dockerfile.arm64 b/Dockerfile.arm64
deleted file mode 100644
index fe2b2604..00000000
--- a/Dockerfile.arm64
+++ /dev/null
@@ -1,34 +0,0 @@
-# Copyright 2021 The Kubernetes Authors.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-ARG BUILDPLATFORM
-
-FROM --platform=$BUILDPLATFORM golang:1.17.2 as builder
-
-ARG STAGINGVERSION
-ARG TARGETPLATFORM
-
-WORKDIR /go/src/sigs.k8s.io/gcp-compute-persistent-disk-csi-driver
-ADD . .
-RUN GOARCH=$(echo $TARGETPLATFORM | cut -f2 -d '/') GCE_PD_CSI_STAGING_VERSION=$STAGINGVERSION make gce-pd-driver
-
-# Start from Kubernetes Debian base
-# A distroless base is not used because it cannot be tested for ARM until GCE has ARM hardware
-FROM k8s.gcr.io/build-image/debian-base:buster-v1.9.0
-COPY --from=builder /go/src/sigs.k8s.io/gcp-compute-persistent-disk-csi-driver/bin/gce-pd-csi-driver /gce-pd-csi-driver
-# Install necessary dependencies
-RUN clean-install util-linux e2fsprogs mount ca-certificates udev xfsprogs
-COPY /lib/udev/scsi_id /lib/udev_containerized/scsi_id
-
-ENTRYPOINT ["/gce-pd-csi-driver"]
\ No newline at end of file
diff --git a/Makefile b/Makefile
index a9e69941..f4090e03 100644
--- a/Makefile
+++ b/Makefile
@@ -96,7 +96,7 @@ build-and-push-container-linux-amd64: require-GCE_PD_CSI_STAGING_IMAGE init-buil
 		--build-arg STAGINGVERSION=$(STAGINGVERSION) --push .
 
 build-and-push-container-linux-arm64: require-GCE_PD_CSI_STAGING_IMAGE init-buildx
-	$(DOCKER) buildx build --file=Dockerfile.arm64 --platform=linux/arm64 \
+	$(DOCKER) buildx build --platform=linux/arm64 \
 		-t $(STAGINGIMAGE):$(STAGINGVERSION)_linux_arm64 \
 		--build-arg BUILDPLATFORM=linux \
 		--build-arg STAGINGVERSION=$(STAGINGVERSION) --push .
diff --git a/README.md b/README.md
index d24cc9dc..18c52e7b 100644
--- a/README.md
+++ b/README.md
@@ -12,7 +12,7 @@ lifecycle of Google Compute Engine Persistent Disks.
 ## Project Status
 
 Status: GA
-Latest stable image: `k8s.gcr.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.3.4`
+Latest stable image: `k8s.gcr.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.3.1`
 
 ### Test Status
 
diff --git a/deploy/kubernetes/base/controller/cluster_setup.yaml b/deploy/kubernetes/base/controller/cluster_setup.yaml
index e22c46b5..70acd405 100644
--- a/deploy/kubernetes/base/controller/cluster_setup.yaml
+++ b/deploy/kubernetes/base/controller/cluster_setup.yaml
@@ -266,10 +266,10 @@ rules:
     verbs: ["get", "list", "watch"]
   - apiGroups: ["snapshot.storage.k8s.io"]
     resources: ["volumesnapshotcontents"]
-    verbs: ["create", "get", "list", "watch", "update", "delete", "patch"]
+    verbs: ["create", "get", "list", "watch", "update", "delete"]
   - apiGroups: ["snapshot.storage.k8s.io"]
     resources: ["volumesnapshotcontents/status"]
-    verbs: ["update", "patch"]
+    verbs: ["update"]
 ---
 
 kind: ClusterRoleBinding
diff --git a/deploy/kubernetes/base/node_linux/node.yaml b/deploy/kubernetes/base/node_linux/node.yaml
index a746683a..51b5c8b8 100644
--- a/deploy/kubernetes/base/node_linux/node.yaml
+++ b/deploy/kubernetes/base/node_linux/node.yaml
@@ -38,13 +38,6 @@ spec:
               mountPath: /csi
             - name: registration-dir
               mountPath: /registration
-          livenessProbe:
-            initialDelaySeconds: 3
-            exec:
-              command:
-                - /csi-node-driver-registrar
-                - --kubelet-registration-path=/var/lib/kubelet/plugins/pd.csi.storage.gke.io/csi.sock
-                - --mode=kubelet-registration-probe
         - name: gce-pd-driver
           # Don't change base image without changing pdImagePlaceholder in
           # test/k8s-integration/main.go
diff --git a/deploy/kubernetes/base/node_windows/node.yaml b/deploy/kubernetes/base/node_windows/node.yaml
index 65bdbeb2..63c01a79 100644
--- a/deploy/kubernetes/base/node_windows/node.yaml
+++ b/deploy/kubernetes/base/node_windows/node.yaml
@@ -39,13 +39,6 @@ spec:
               mountPath: /csi
             - name: registration-dir
               mountPath: /registration
-          livenessProbe:
-            initialDelaySeconds: 3
-            exec:
-              command:
-                - /csi-node-driver-registrar.exe
-                - --kubelet-registration-path=C:\\var\\lib\\kubelet\\plugins\\pd.csi.storage.gke.io\\csi.sock
-                - --mode=kubelet-registration-probe
         - name: gce-pd-driver
           # Don't change base image without changing pdImagePlaceholder in
           # test/k8s-integration/main.go
diff --git a/deploy/kubernetes/images/alpha/image.yaml b/deploy/kubernetes/images/alpha/image.yaml
new file mode 100644
index 00000000..84441ffb
--- /dev/null
+++ b/deploy/kubernetes/images/alpha/image.yaml
@@ -0,0 +1,16 @@
+apiVersion: builtin
+kind: ImageTagTransformer
+metadata:
+  name: imagetag-gcepd-driver-alpha-win
+imageTag:
+  name: gke.gcr.io/gcp-compute-persistent-disk-csi-driver
+  newTag: "v1.2.0-gke.8"
+---
+
+apiVersion: builtin
+kind: ImageTagTransformer
+metadata:
+  name: imagetag-node-registrar-win
+imageTag:
+  name: k8s.gcr.io/sig-storage/csi-node-driver-registrar
+  newTag: "v2.1.0"
diff --git a/deploy/kubernetes/images/alpha/kustomization.yaml b/deploy/kubernetes/images/alpha/kustomization.yaml
index 703ce473..2dfbdfd4 100644
--- a/deploy/kubernetes/images/alpha/kustomization.yaml
+++ b/deploy/kubernetes/images/alpha/kustomization.yaml
@@ -2,3 +2,4 @@ namespace:
   gce-pd-csi-driver
 resources:
 - ../stable-master/
+- image.yaml
diff --git a/deploy/kubernetes/images/prow-gke-release-staging-rc-master/image.yaml b/deploy/kubernetes/images/prow-gke-release-staging-rc-master/image.yaml
index 2f78e976..d7d325f6 100644
--- a/deploy/kubernetes/images/prow-gke-release-staging-rc-master/image.yaml
+++ b/deploy/kubernetes/images/prow-gke-release-staging-rc-master/image.yaml
@@ -43,7 +43,7 @@ metadata:
   name: imagetag-csi-node-registrar-prow-rc
 imageTag:
   name: k8s.gcr.io/sig-storage/csi-node-driver-registrar
-  newTag: "v2.3.0"
+  newTag: "v2.1.0"
 ---
 
 apiVersion: builtin
@@ -52,5 +52,6 @@ metadata:
   name: imagetag-csi-gce-driver-prow-rc
 imageTag:
   name: k8s.gcr.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver
-  newTag: "v1.3.4"
+  newName: gcr.io/k8s-staging-cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver
+  newTag: "v1.3.2-rc1"
 ---
diff --git a/deploy/kubernetes/images/stable-master/image.yaml b/deploy/kubernetes/images/stable-master/image.yaml
index ce5390b5..3c8804f9 100644
--- a/deploy/kubernetes/images/stable-master/image.yaml
+++ b/deploy/kubernetes/images/stable-master/image.yaml
@@ -52,5 +52,5 @@ imageTag:
   # Don't change stable image without changing pdImagePlaceholder in
   # test/k8s-integration/main.go
   newName: k8s.gcr.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver
-  newTag: "v1.3.4"
+  newTag: "v1.3.1"
 ---
diff --git a/deploy/kubernetes/overlays/dev/WARNING.md b/deploy/kubernetes/overlays/dev/WARNING.md
index 39cf7159..f3191fb3 100644
--- a/deploy/kubernetes/overlays/dev/WARNING.md
+++ b/deploy/kubernetes/overlays/dev/WARNING.md
@@ -6,3 +6,10 @@ This is the absolute cutting edge development Driver, it is intended for testing
 and development only and can have vast differences in
 functionality/behavior/configuration. Use only to try the newest features that
 are not guaranteed to work yet.
+
+APPROXIMATE CHANGELOG in latest:
+* Topology
+* RePD
+* Volume ID Format Changed
+* Node ID Format Changed
+* Parameter "zone" Removed
\ No newline at end of file
diff --git a/deploy/kubernetes/overlays/stable-1-17/kustomization.yaml b/deploy/kubernetes/overlays/stable-1-17/kustomization.yaml
index c33410d0..646021c7 100644
--- a/deploy/kubernetes/overlays/stable-1-17/kustomization.yaml
+++ b/deploy/kubernetes/overlays/stable-1-17/kustomization.yaml
@@ -9,6 +9,5 @@ resources:
 patchesStrategicMerge:
 - no_win_roles.yaml
 - no_v1_csidriver.yaml
-- no-node-driver-registrar-probe.yaml
 transformers:
 - ../../images/stable-1-17
diff --git a/deploy/kubernetes/overlays/stable-1-17/no-node-driver-registrar-probe.yaml b/deploy/kubernetes/overlays/stable-1-17/no-node-driver-registrar-probe.yaml
deleted file mode 100644
index bb5dfe96..00000000
--- a/deploy/kubernetes/overlays/stable-1-17/no-node-driver-registrar-probe.yaml
+++ /dev/null
@@ -1,25 +0,0 @@
-kind: DaemonSet
-apiVersion: apps/v1
-metadata:
-  name: csi-gce-pd-node-win
-spec:
-  template:
-    spec:
-      containers:
-        - name: csi-driver-registrar
-          # introduced in node-driver-registrar v2.3.0
-          livenessProbe:
-            $patch: delete
----
-kind: DaemonSet
-apiVersion: apps/v1
-metadata:
-  name: csi-gce-pd-node
-spec:
-  template:
-    spec:
-      containers:
-        - name: csi-driver-registrar
-          # introduced in node-driver-registrar v2.3.0
-          livenessProbe:
-            $patch: delete
diff --git a/deploy/kubernetes/overlays/stable-1-18/kustomization.yaml b/deploy/kubernetes/overlays/stable-1-18/kustomization.yaml
index 09543ee6..decf1f78 100644
--- a/deploy/kubernetes/overlays/stable-1-18/kustomization.yaml
+++ b/deploy/kubernetes/overlays/stable-1-18/kustomization.yaml
@@ -4,7 +4,5 @@ namespace:
   gce-pd-csi-driver
 resources:
 - ../../base
-patchesStrategicMerge:
-- no-node-driver-registrar-probe.yaml
 transformers:
 - ../../images/stable-1-18
diff --git a/deploy/kubernetes/overlays/stable-1-18/no-node-driver-registrar-probe.yaml b/deploy/kubernetes/overlays/stable-1-18/no-node-driver-registrar-probe.yaml
deleted file mode 100644
index bb5dfe96..00000000
--- a/deploy/kubernetes/overlays/stable-1-18/no-node-driver-registrar-probe.yaml
+++ /dev/null
@@ -1,25 +0,0 @@
-kind: DaemonSet
-apiVersion: apps/v1
-metadata:
-  name: csi-gce-pd-node-win
-spec:
-  template:
-    spec:
-      containers:
-        - name: csi-driver-registrar
-          # introduced in node-driver-registrar v2.3.0
-          livenessProbe:
-            $patch: delete
----
-kind: DaemonSet
-apiVersion: apps/v1
-metadata:
-  name: csi-gce-pd-node
-spec:
-  template:
-    spec:
-      containers:
-        - name: csi-driver-registrar
-          # introduced in node-driver-registrar v2.3.0
-          livenessProbe:
-            $patch: delete
diff --git a/deploy/kubernetes/overlays/stable-1-19/kustomization.yaml b/deploy/kubernetes/overlays/stable-1-19/kustomization.yaml
index 965394dc..b776204b 100644
--- a/deploy/kubernetes/overlays/stable-1-19/kustomization.yaml
+++ b/deploy/kubernetes/overlays/stable-1-19/kustomization.yaml
@@ -4,7 +4,5 @@ namespace:
   gce-pd-csi-driver
 resources:
 - ../../base/
-patchesStrategicMerge:
-- no-node-driver-registrar-probe.yaml
 transformers:
 - ../../images/stable-1-19
diff --git a/deploy/kubernetes/overlays/stable-1-19/no-node-driver-registrar-probe.yaml b/deploy/kubernetes/overlays/stable-1-19/no-node-driver-registrar-probe.yaml
deleted file mode 100644
index bb5dfe96..00000000
--- a/deploy/kubernetes/overlays/stable-1-19/no-node-driver-registrar-probe.yaml
+++ /dev/null
@@ -1,25 +0,0 @@
-kind: DaemonSet
-apiVersion: apps/v1
-metadata:
-  name: csi-gce-pd-node-win
-spec:
-  template:
-    spec:
-      containers:
-        - name: csi-driver-registrar
-          # introduced in node-driver-registrar v2.3.0
-          livenessProbe:
-            $patch: delete
----
-kind: DaemonSet
-apiVersion: apps/v1
-metadata:
-  name: csi-gce-pd-node
-spec:
-  template:
-    spec:
-      containers:
-        - name: csi-driver-registrar
-          # introduced in node-driver-registrar v2.3.0
-          livenessProbe:
-            $patch: delete
diff --git a/deploy/kubernetes/overlays/stable-master/kustomization.yaml b/deploy/kubernetes/overlays/stable-master/kustomization.yaml
index 96ed93d4..66a44c0e 100644
--- a/deploy/kubernetes/overlays/stable-master/kustomization.yaml
+++ b/deploy/kubernetes/overlays/stable-master/kustomization.yaml
@@ -4,7 +4,5 @@ namespace:
   gce-pd-csi-driver
 resources:
 - ../../base
-patchesStrategicMerge:
-- no-node-driver-registrar-probe.yaml
 transformers:
 - ../../images/stable-master
diff --git a/deploy/kubernetes/overlays/stable-master/no-node-driver-registrar-probe.yaml b/deploy/kubernetes/overlays/stable-master/no-node-driver-registrar-probe.yaml
deleted file mode 100644
index 516ad260..00000000
--- a/deploy/kubernetes/overlays/stable-master/no-node-driver-registrar-probe.yaml
+++ /dev/null
@@ -1,27 +0,0 @@
-# TODO(mauriciopoppe): remove this file once testgrid is green with
-# prow-gke-release-staging-rc-master
-kind: DaemonSet
-apiVersion: apps/v1
-metadata:
-  name: csi-gce-pd-node-win
-spec:
-  template:
-    spec:
-      containers:
-        - name: csi-driver-registrar
-          # introduced in node-driver-registrar v2.3.0
-          livenessProbe:
-            $patch: delete
----
-kind: DaemonSet
-apiVersion: apps/v1
-metadata:
-  name: csi-gce-pd-node
-spec:
-  template:
-    spec:
-      containers:
-        - name: csi-driver-registrar
-          # introduced in node-driver-registrar v2.3.0
-          livenessProbe:
-            $patch: delete
diff --git a/pkg/gce-cloud-provider/compute/cloud-disk.go b/pkg/gce-cloud-provider/compute/cloud-disk.go
index e6a4cd6b..86243d98 100644
--- a/pkg/gce-cloud-provider/compute/cloud-disk.go
+++ b/pkg/gce-cloud-provider/compute/cloud-disk.go
@@ -177,17 +177,6 @@ func (d *CloudDisk) GetSnapshotId() string {
 	}
 }
 
-func (d *CloudDisk) GetSourceDiskId() string {
-	switch {
-	case d.disk != nil:
-		return d.disk.SourceDiskId
-	case d.betaDisk != nil:
-		return d.betaDisk.SourceDiskId
-	default:
-		return ""
-	}
-}
-
 func (d *CloudDisk) GetKMSKeyName() string {
 	switch {
 	case d.disk != nil:
diff --git a/pkg/gce-cloud-provider/compute/fake-gce.go b/pkg/gce-cloud-provider/compute/fake-gce.go
index 6a7adeff..67b2d8af 100644
--- a/pkg/gce-cloud-provider/compute/fake-gce.go
+++ b/pkg/gce-cloud-provider/compute/fake-gce.go
@@ -220,7 +220,7 @@ func (cloud *FakeCloudProvider) ValidateExistingDisk(ctx context.Context, resp *
 	return ValidateDiskParameters(resp, params)
 }
 
-func (cloud *FakeCloudProvider) InsertDisk(ctx context.Context, project string, volKey *meta.Key, params common.DiskParameters, capBytes int64, capacityRange *csi.CapacityRange, replicaZones []string, snapshotID string, volumeContentSourceVolumeID string, multiWriter bool) error {
+func (cloud *FakeCloudProvider) InsertDisk(ctx context.Context, project string, volKey *meta.Key, params common.DiskParameters, capBytes int64, capacityRange *csi.CapacityRange, replicaZones []string, snapshotID string, multiWriter bool) error {
 	if disk, ok := cloud.disks[volKey.Name]; ok {
 		err := cloud.ValidateExistingDisk(ctx, disk, params,
 			int64(capacityRange.GetRequiredBytes()),
@@ -237,7 +237,6 @@ func (cloud *FakeCloudProvider) InsertDisk(ctx context.Context, project string,
 		Description:      "Disk created by GCE-PD CSI Driver",
 		Type:             cloud.GetDiskTypeURI(project, volKey, params.DiskType),
 		SourceSnapshotId: snapshotID,
-		SourceDiskId:     volumeContentSourceVolumeID,
 		Status:           cloud.mockDiskStatus,
 		Labels:           params.Labels,
 	}
diff --git a/pkg/gce-cloud-provider/compute/gce-compute.go b/pkg/gce-cloud-provider/compute/gce-compute.go
index 8e672baa..73787c67 100644
--- a/pkg/gce-cloud-provider/compute/gce-compute.go
+++ b/pkg/gce-cloud-provider/compute/gce-compute.go
@@ -56,7 +56,7 @@ type GCECompute interface {
 	GetDisk(ctx context.Context, project string, volumeKey *meta.Key, gceAPIVersion GCEAPIVersion) (*CloudDisk, error)
 	RepairUnderspecifiedVolumeKey(ctx context.Context, project string, volumeKey *meta.Key) (string, *meta.Key, error)
 	ValidateExistingDisk(ctx context.Context, disk *CloudDisk, params common.DiskParameters, reqBytes, limBytes int64, multiWriter bool) error
-	InsertDisk(ctx context.Context, project string, volKey *meta.Key, params common.DiskParameters, capBytes int64, capacityRange *csi.CapacityRange, replicaZones []string, snapshotID string, volumeContentSourceVolumeID string, multiWriter bool) error
+	InsertDisk(ctx context.Context, project string, volKey *meta.Key, params common.DiskParameters, capBytes int64, capacityRange *csi.CapacityRange, replicaZones []string, snapshotID string, multiWriter bool) error
 	DeleteDisk(ctx context.Context, project string, volumeKey *meta.Key) error
 	AttachDisk(ctx context.Context, project string, volKey *meta.Key, readWrite, diskType, instanceZone, instanceName string) error
 	DetachDisk(ctx context.Context, project, deviceName, instanceZone, instanceName string) error
@@ -315,7 +315,7 @@ func ValidateDiskParameters(disk *CloudDisk, params common.DiskParameters) error
 	return nil
 }
 
-func (cloud *CloudProvider) InsertDisk(ctx context.Context, project string, volKey *meta.Key, params common.DiskParameters, capBytes int64, capacityRange *csi.CapacityRange, replicaZones []string, snapshotID string, volumeContentSourceVolumeID string, multiWriter bool) error {
+func (cloud *CloudProvider) InsertDisk(ctx context.Context, project string, volKey *meta.Key, params common.DiskParameters, capBytes int64, capacityRange *csi.CapacityRange, replicaZones []string, snapshotID string, multiWriter bool) error {
 	klog.V(5).Infof("Inserting disk %v", volKey)
 
 	description, err := encodeDiskTags(params.Tags)
@@ -328,12 +328,12 @@ func (cloud *CloudProvider) InsertDisk(ctx context.Context, project string, volK
 		if description == "" {
 			description = "Disk created by GCE-PD CSI Driver"
 		}
-		return cloud.insertZonalDisk(ctx, project, volKey, params, capBytes, capacityRange, snapshotID, volumeContentSourceVolumeID, description, multiWriter)
+		return cloud.insertZonalDisk(ctx, project, volKey, params, capBytes, capacityRange, snapshotID, description, multiWriter)
 	case meta.Regional:
 		if description == "" {
 			description = "Regional disk created by GCE-PD CSI Driver"
 		}
-		return cloud.insertRegionalDisk(ctx, project, volKey, params, capBytes, capacityRange, replicaZones, snapshotID, volumeContentSourceVolumeID, description, multiWriter)
+		return cloud.insertRegionalDisk(ctx, project, volKey, params, capBytes, capacityRange, replicaZones, snapshotID, description, multiWriter)
 	default:
 		return fmt.Errorf("could not insert disk, key was neither zonal nor regional, instead got: %v", volKey.String())
 	}
@@ -377,7 +377,6 @@ func (cloud *CloudProvider) insertRegionalDisk(
 	capacityRange *csi.CapacityRange,
 	replicaZones []string,
 	snapshotID string,
-	volumeContentSourceVolumeID string,
 	description string,
 	multiWriter bool) error {
 	var (
@@ -400,9 +399,6 @@ func (cloud *CloudProvider) insertRegionalDisk(
 	if snapshotID != "" {
 		diskToCreate.SourceSnapshot = snapshotID
 	}
-	if volumeContentSourceVolumeID != "" {
-		diskToCreate.SourceDisk = volumeContentSourceVolumeID
-	}
 	if len(replicaZones) != 0 {
 		diskToCreate.ReplicaZones = replicaZones
 	}
@@ -476,7 +472,6 @@ func (cloud *CloudProvider) insertZonalDisk(
 	capBytes int64,
 	capacityRange *csi.CapacityRange,
 	snapshotID string,
-	volumeContentSourceVolumeID string,
 	description string,
 	multiWriter bool) error {
 	var (
@@ -500,9 +495,6 @@ func (cloud *CloudProvider) insertZonalDisk(
 	if snapshotID != "" {
 		diskToCreate.SourceSnapshot = snapshotID
 	}
-	if volumeContentSourceVolumeID != "" {
-		diskToCreate.SourceDisk = volumeContentSourceVolumeID
-	}
 
 	if params.DiskEncryptionKMSKey != "" {
 		diskToCreate.DiskEncryptionKey = &computev1.CustomerEncryptionKey{
diff --git a/pkg/gce-pd-csi-driver/controller.go b/pkg/gce-pd-csi-driver/controller.go
index 3bbc664c..e0a831ec 100644
--- a/pkg/gce-pd-csi-driver/controller.go
+++ b/pkg/gce-pd-csi-driver/controller.go
@@ -30,6 +30,8 @@ import (
 	"google.golang.org/grpc/status"
 	"k8s.io/apimachinery/pkg/util/sets"
 	"k8s.io/apimachinery/pkg/util/uuid"
+	"k8s.io/apimachinery/pkg/util/wait"
+	"k8s.io/client-go/util/workqueue"
 	"k8s.io/klog"
 
 	"sigs.k8s.io/gcp-compute-persistent-disk-csi-driver/pkg/common"
@@ -47,6 +49,22 @@ type GCEControllerServer struct {
 	// operations for that same volume (as defined by Volume Key) return an
 	// Aborted error
 	volumeLocks *common.VolumeLocks
+
+	// queue is a rate limited work queue for Controller Publish/Unpublish
+	// Volume calls
+	queue workqueue.RateLimitingInterface
+
+	// publishErrorsSeenOnNode is a list of nodes with attach/detach
+	// operation failures so those nodes shall be rate limited for all
+	// the attach/detach operations until there is an attach / detach
+	// operation succeeds
+	publishErrorsSeenOnNode map[string]bool
+}
+
+type workItem struct {
+	ctx          context.Context
+	publishReq   *csi.ControllerPublishVolumeRequest
+	unpublishReq *csi.ControllerUnpublishVolumeRequest
 }
 
 var _ csi.ControllerServer = &GCEControllerServer{}
@@ -179,8 +197,9 @@ func (gceCS *GCEControllerServer) CreateVolume(ctx context.Context, req *csi.Cre
 		if err != nil {
 			return nil, status.Error(codes.Internal, fmt.Sprintf("CreateVolume disk %v had error checking ready status: %v", volKey, err))
 		}
+
 		if !ready {
-			return nil, status.Error(codes.Internal, fmt.Sprintf("CreateVolume existing disk %v is not ready", volKey))
+			return nil, status.Error(codes.Internal, fmt.Sprintf("CreateVolume disk %v is not ready", volKey))
 		}
 
 		// If there is no validation error, immediately return success
@@ -189,10 +208,10 @@ func (gceCS *GCEControllerServer) CreateVolume(ctx context.Context, req *csi.Cre
 	}
 
 	snapshotID := ""
-	volumeContentSourceVolumeID := ""
 	content := req.GetVolumeContentSource()
 	if content != nil {
 		if content.GetSnapshot() != nil {
+			// TODO(#161): Add support for Volume Source (cloning) introduced in CSI v1.0.0
 			snapshotID = content.GetSnapshot().GetSnapshotId()
 
 			// Verify that snapshot exists
@@ -203,38 +222,8 @@ func (gceCS *GCEControllerServer) CreateVolume(ctx context.Context, req *csi.Cre
 				return nil, status.Errorf(codes.NotFound, "CreateVolume source snapshot %s does not exist", snapshotID)
 			}
 		}
-
-		if content.GetVolume() != nil {
-			volumeContentSourceVolumeID = content.GetVolume().GetVolumeId()
-			// Verify that the source VolumeID is in the correct format.
-			project, sourceVolKey, err := common.VolumeIDToKey(volumeContentSourceVolumeID)
-			if err != nil {
-				return nil, status.Error(codes.InvalidArgument, fmt.Sprintf("CreateVolume source volume id is invalid: %v", err))
-			}
-
-			// Verify that the volume in VolumeContentSource exists.
-			diskFromSourceVolume, err := gceCS.CloudProvider.GetDisk(ctx, project, sourceVolKey, gceAPIVersion)
-			if err != nil {
-				if gce.IsGCEError(err, "notFound") {
-					return nil, status.Errorf(codes.NotFound, "CreateVolume source volume %s does not exist", volumeContentSourceVolumeID)
-				} else {
-					return nil, status.Error(codes.Internal, fmt.Sprintf("CreateVolume unknown get disk error when validating: %v", err))
-				}
-			}
-			// Verify the zone, region, and disk type of the clone must be the same as that of the source disk.
-			if err := gce.ValidateDiskParameters(diskFromSourceVolume, params); err != nil {
-				return nil, status.Errorf(codes.InvalidArgument, `CreateVolume source volume parameters do not match CreateVolumeRequest Parameters: %v`, err)
-			}
-			// Verify the source disk is ready.
-			ready, err := isDiskReady(diskFromSourceVolume)
-			if err != nil {
-				return nil, status.Error(codes.Internal, fmt.Sprintf("CreateVolume disk from source volume %v had error checking ready status: %v", sourceVolKey, err))
-			}
-			if !ready {
-				return nil, status.Error(codes.Internal, fmt.Sprintf("CreateVolume disk from source volume %v is not ready", sourceVolKey))
-			}
-		}
 	}
+
 	// Create the disk
 	var disk *gce.CloudDisk
 	switch params.ReplicationType {
@@ -242,7 +231,7 @@ func (gceCS *GCEControllerServer) CreateVolume(ctx context.Context, req *csi.Cre
 		if len(zones) != 1 {
 			return nil, status.Error(codes.Internal, fmt.Sprintf("CreateVolume failed to get a single zone for creating zonal disk, instead got: %v", zones))
 		}
-		disk, err = createSingleZoneDisk(ctx, gceCS.CloudProvider, name, zones, params, capacityRange, capBytes, snapshotID, volumeContentSourceVolumeID, multiWriter)
+		disk, err = createSingleZoneDisk(ctx, gceCS.CloudProvider, name, zones, params, capacityRange, capBytes, snapshotID, multiWriter)
 		if err != nil {
 			return nil, status.Error(codes.Internal, fmt.Sprintf("CreateVolume failed to create single zonal disk %#v: %v", name, err))
 		}
@@ -250,7 +239,7 @@ func (gceCS *GCEControllerServer) CreateVolume(ctx context.Context, req *csi.Cre
 		if len(zones) != 2 {
 			return nil, status.Errorf(codes.Internal, fmt.Sprintf("CreateVolume failed to get a 2 zones for creating regional disk, instead got: %v", zones))
 		}
-		disk, err = createRegionalDisk(ctx, gceCS.CloudProvider, name, zones, params, capacityRange, capBytes, snapshotID, volumeContentSourceVolumeID, multiWriter)
+		disk, err = createRegionalDisk(ctx, gceCS.CloudProvider, name, zones, params, capacityRange, capBytes, snapshotID, multiWriter)
 		if err != nil {
 			return nil, status.Error(codes.Internal, fmt.Sprintf("CreateVolume failed to create regional disk %#v: %v", name, err))
 		}
@@ -309,25 +298,113 @@ func (gceCS *GCEControllerServer) DeleteVolume(ctx context.Context, req *csi.Del
 	return &csi.DeleteVolumeResponse{}, nil
 }
 
+// Run starts the GCEControllerServer.
+func (gceCS *GCEControllerServer) Run() {
+	go wait.Until(gceCS.worker, 1*time.Second, wait.NeverStop)
+}
+
+func (gceCS *GCEControllerServer) worker() {
+	// Runs until workqueue is shut down
+	for gceCS.processNextWorkItem() {
+	}
+}
+
+func (gceCS *GCEControllerServer) processNextWorkItem() bool {
+	item, quit := gceCS.queue.Get()
+	if quit {
+		return false
+	}
+	defer gceCS.queue.Done(item)
+
+	workItem, ok := item.(*workItem)
+	if !ok {
+		gceCS.queue.AddRateLimited(item)
+		return true
+	}
+
+	if workItem.publishReq != nil {
+		_, err := gceCS.executeControllerPublishVolume(workItem.ctx, workItem.publishReq)
+
+		if err != nil {
+			klog.Errorf("ControllerPublishVolume failed with error: %v", err)
+		}
+	}
+
+	if workItem.unpublishReq != nil {
+		_, err := gceCS.executeControllerUnpublishVolume(workItem.ctx, workItem.unpublishReq)
+
+		if err != nil {
+			klog.Errorf("ControllerUnpublishVolume failed with error: %v", err)
+		}
+	}
+
+	gceCS.queue.Forget(item)
+	return true
+}
+
 func (gceCS *GCEControllerServer) ControllerPublishVolume(ctx context.Context, req *csi.ControllerPublishVolumeRequest) (*csi.ControllerPublishVolumeResponse, error) {
+	// Only valid requests will be queued
+	_, _, err := gceCS.validateControllerPublishVolumeRequest(ctx, req)
+
+	if err != nil {
+		return nil, err
+	}
+
+	// If the node is not marked, proceed the request
+	if _, found := gceCS.publishErrorsSeenOnNode[req.NodeId]; !found {
+		return gceCS.executeControllerPublishVolume(ctx, req)
+	}
+
+	// Node is marked so queue up the request
+	gceCS.queue.AddRateLimited(&workItem{
+		ctx:        ctx,
+		publishReq: req,
+	})
+	return &csi.ControllerPublishVolumeResponse{}, nil
+}
+
+func (gceCS *GCEControllerServer) validateControllerPublishVolumeRequest(ctx context.Context, req *csi.ControllerPublishVolumeRequest) (string, *meta.Key, error) {
 	// Validate arguments
 	volumeID := req.GetVolumeId()
-	readOnly := req.GetReadonly()
 	nodeID := req.GetNodeId()
 	volumeCapability := req.GetVolumeCapability()
 	if len(volumeID) == 0 {
-		return nil, status.Error(codes.InvalidArgument, "ControllerPublishVolume Volume ID must be provided")
+		return "", nil, status.Error(codes.InvalidArgument, "ControllerPublishVolume Volume ID must be provided")
 	}
 	if len(nodeID) == 0 {
-		return nil, status.Error(codes.InvalidArgument, "ControllerPublishVolume Node ID must be provided")
+		return "", nil, status.Error(codes.InvalidArgument, "ControllerPublishVolume Node ID must be provided")
 	}
 	if volumeCapability == nil {
-		return nil, status.Error(codes.InvalidArgument, "ControllerPublishVolume Volume capability must be provided")
+		return "", nil, status.Error(codes.InvalidArgument, "ControllerPublishVolume Volume capability must be provided")
 	}
 
 	project, volKey, err := common.VolumeIDToKey(volumeID)
 	if err != nil {
-		return nil, status.Error(codes.InvalidArgument, fmt.Sprintf("ControllerPublishVolume volume ID is invalid: %v", err))
+		return "", nil, status.Error(codes.InvalidArgument, fmt.Sprintf("ControllerPublishVolume volume ID is invalid: %v", err))
+	}
+
+	// TODO(#253): Check volume capability matches for ALREADY_EXISTS
+	if err = validateVolumeCapability(volumeCapability); err != nil {
+		return "", nil, status.Error(codes.InvalidArgument, fmt.Sprintf("VolumeCapabilities is invalid: %v", err))
+	}
+
+	return project, volKey, nil
+}
+
+func (gceCS *GCEControllerServer) executeControllerPublishVolume(ctx context.Context, req *csi.ControllerPublishVolumeRequest) (*csi.ControllerPublishVolumeResponse, error) {
+	project, volKey, err := gceCS.validateControllerPublishVolumeRequest(ctx, req)
+
+	if err != nil {
+		return nil, err
+	}
+
+	volumeID := req.GetVolumeId()
+	readOnly := req.GetReadonly()
+	nodeID := req.GetNodeId()
+	volumeCapability := req.GetVolumeCapability()
+
+	pubVolResp := &csi.ControllerPublishVolumeResponse{
+		PublishContext: nil,
 	}
 
 	project, volKey, err = gceCS.CloudProvider.RepairUnderspecifiedVolumeKey(ctx, project, volKey)
@@ -346,15 +423,6 @@ func (gceCS *GCEControllerServer) ControllerPublishVolume(ctx context.Context, r
 	}
 	defer gceCS.volumeLocks.Release(lockingVolumeID)
 
-	// TODO(#253): Check volume capability matches for ALREADY_EXISTS
-	if err = validateVolumeCapability(volumeCapability); err != nil {
-		return nil, status.Error(codes.InvalidArgument, fmt.Sprintf("VolumeCapabilities is invalid: %v", err))
-	}
-
-	pubVolResp := &csi.ControllerPublishVolumeResponse{
-		PublishContext: nil,
-	}
-
 	_, err = gceCS.CloudProvider.GetDisk(ctx, project, volKey, gce.GCEAPIVersionV1)
 	if err != nil {
 		if gce.IsGCENotFoundError(err) {
@@ -404,29 +472,69 @@ func (gceCS *GCEControllerServer) ControllerPublishVolume(ctx context.Context, r
 
 	err = gceCS.CloudProvider.WaitForAttach(ctx, project, volKey, instanceZone, instanceName)
 	if err != nil {
+		// Mark the node and rate limit all the following attach/detach
+		// operations for this node
+		gceCS.publishErrorsSeenOnNode[nodeID] = true
 		return nil, status.Error(codes.Internal, fmt.Sprintf("unknown WaitForAttach error: %v", err))
 	}
 
+	// Attach succeeds so unmark the node
+	delete(gceCS.publishErrorsSeenOnNode, nodeID)
+
 	klog.V(4).Infof("ControllerPublishVolume succeeded for disk %v to instance %v", volKey, nodeID)
 	return pubVolResp, nil
 }
 
 func (gceCS *GCEControllerServer) ControllerUnpublishVolume(ctx context.Context, req *csi.ControllerUnpublishVolumeRequest) (*csi.ControllerUnpublishVolumeResponse, error) {
+	// Only valid requests will be queued
+	_, _, err := gceCS.validateControllerUnpublishVolumeRequest(ctx, req)
+
+	if err != nil {
+		return nil, err
+	}
+
+	// If the node is not marked, proceed the request
+	if _, found := gceCS.publishErrorsSeenOnNode[req.NodeId]; !found {
+		return gceCS.executeControllerUnpublishVolume(ctx, req)
+	}
+
+	// Node is marked so queue up the request
+	gceCS.queue.AddRateLimited(&workItem{
+		ctx:          ctx,
+		unpublishReq: req,
+	})
+
+	return &csi.ControllerUnpublishVolumeResponse{}, nil
+}
+
+func (gceCS *GCEControllerServer) validateControllerUnpublishVolumeRequest(ctx context.Context, req *csi.ControllerUnpublishVolumeRequest) (string, *meta.Key, error) {
 	// Validate arguments
 	volumeID := req.GetVolumeId()
 	nodeID := req.GetNodeId()
 	if len(volumeID) == 0 {
-		return nil, status.Error(codes.InvalidArgument, "ControllerUnpublishVolume Volume ID must be provided")
+		return "", nil, status.Error(codes.InvalidArgument, "ControllerUnpublishVolume Volume ID must be provided")
 	}
 	if len(nodeID) == 0 {
-		return nil, status.Error(codes.InvalidArgument, "ControllerUnpublishVolume Node ID must be provided")
+		return "", nil, status.Error(codes.InvalidArgument, "ControllerUnpublishVolume Node ID must be provided")
 	}
 
 	project, volKey, err := common.VolumeIDToKey(volumeID)
 	if err != nil {
-		return nil, status.Error(codes.InvalidArgument, fmt.Sprintf("ControllerUnpublishVolume Volume ID is invalid: %v", err))
+		return "", nil, status.Error(codes.InvalidArgument, fmt.Sprintf("ControllerUnpublishVolume Volume ID is invalid: %v", err))
+	}
+
+	return project, volKey, nil
+}
+
+func (gceCS *GCEControllerServer) executeControllerUnpublishVolume(ctx context.Context, req *csi.ControllerUnpublishVolumeRequest) (*csi.ControllerUnpublishVolumeResponse, error) {
+	project, volKey, err := gceCS.validateControllerUnpublishVolumeRequest(ctx, req)
+
+	if err != nil {
+		return nil, err
 	}
 
+	volumeID := req.GetVolumeId()
+	nodeID := req.GetNodeId()
 	project, volKey, err = gceCS.CloudProvider.RepairUnderspecifiedVolumeKey(ctx, project, volKey)
 	if err != nil {
 		if gce.IsGCENotFoundError(err) {
@@ -472,9 +580,15 @@ func (gceCS *GCEControllerServer) ControllerUnpublishVolume(ctx context.Context,
 
 	err = gceCS.CloudProvider.DetachDisk(ctx, project, deviceName, instanceZone, instanceName)
 	if err != nil {
+		// Mark the node and rate limit all the following attach/detach
+		// operations for this node
+		gceCS.publishErrorsSeenOnNode[nodeID] = true
 		return nil, status.Error(codes.Internal, fmt.Sprintf("unknown detach error: %v", err))
 	}
 
+	// Detach succeeds so unmark the node
+	delete(gceCS.publishErrorsSeenOnNode, nodeID)
+
 	klog.V(4).Infof("ControllerUnpublishVolume succeeded for disk %v from node %v", volKey, nodeID)
 	return &csi.ControllerUnpublishVolumeResponse{}, nil
 }
@@ -1082,28 +1196,15 @@ func generateCreateVolumeResponse(disk *gce.CloudDisk, zones []string) *csi.Crea
 		},
 	}
 	snapshotID := disk.GetSnapshotId()
-	diskID := disk.GetSourceDiskId()
-	if diskID != "" || snapshotID != "" {
-		contentSource := &csi.VolumeContentSource{}
-		if snapshotID != "" {
-			contentSource = &csi.VolumeContentSource{
-				Type: &csi.VolumeContentSource_Snapshot{
-					Snapshot: &csi.VolumeContentSource_SnapshotSource{
-						SnapshotId: snapshotID,
-					},
-				},
-			}
-		}
-		if diskID != "" {
-			contentSource = &csi.VolumeContentSource{
-				Type: &csi.VolumeContentSource_Volume{
-					Volume: &csi.VolumeContentSource_VolumeSource{
-						VolumeId: diskID,
-					},
+	if snapshotID != "" {
+		source := &csi.VolumeContentSource{
+			Type: &csi.VolumeContentSource_Snapshot{
+				Snapshot: &csi.VolumeContentSource_SnapshotSource{
+					SnapshotId: snapshotID,
 				},
-			}
+			},
 		}
-		createResp.Volume.ContentSource = contentSource
+		createResp.Volume.ContentSource = source
 	}
 	return createResp
 }
@@ -1114,7 +1215,7 @@ func cleanSelfLink(selfLink string) string {
 	return strings.TrimPrefix(temp, gce.GCEComputeAlphaAPIEndpoint)
 }
 
-func createRegionalDisk(ctx context.Context, cloudProvider gce.GCECompute, name string, zones []string, params common.DiskParameters, capacityRange *csi.CapacityRange, capBytes int64, snapshotID string, volumeContentSourceVolumeID string, multiWriter bool) (*gce.CloudDisk, error) {
+func createRegionalDisk(ctx context.Context, cloudProvider gce.GCECompute, name string, zones []string, params common.DiskParameters, capacityRange *csi.CapacityRange, capBytes int64, snapshotID string, multiWriter bool) (*gce.CloudDisk, error) {
 	project := cloudProvider.GetDefaultProject()
 	region, err := common.GetRegionFromZones(zones)
 	if err != nil {
@@ -1127,7 +1228,7 @@ func createRegionalDisk(ctx context.Context, cloudProvider gce.GCECompute, name
 			fullyQualifiedReplicaZones, cloudProvider.GetReplicaZoneURI(project, replicaZone))
 	}
 
-	err = cloudProvider.InsertDisk(ctx, project, meta.RegionalKey(name, region), params, capBytes, capacityRange, fullyQualifiedReplicaZones, snapshotID, volumeContentSourceVolumeID, multiWriter)
+	err = cloudProvider.InsertDisk(ctx, project, meta.RegionalKey(name, region), params, capBytes, capacityRange, fullyQualifiedReplicaZones, snapshotID, multiWriter)
 	if err != nil {
 		return nil, fmt.Errorf("failed to insert regional disk: %v", err)
 	}
@@ -1144,13 +1245,13 @@ func createRegionalDisk(ctx context.Context, cloudProvider gce.GCECompute, name
 	return disk, nil
 }
 
-func createSingleZoneDisk(ctx context.Context, cloudProvider gce.GCECompute, name string, zones []string, params common.DiskParameters, capacityRange *csi.CapacityRange, capBytes int64, snapshotID string, volumeContentSourceVolumeID string, multiWriter bool) (*gce.CloudDisk, error) {
+func createSingleZoneDisk(ctx context.Context, cloudProvider gce.GCECompute, name string, zones []string, params common.DiskParameters, capacityRange *csi.CapacityRange, capBytes int64, snapshotID string, multiWriter bool) (*gce.CloudDisk, error) {
 	project := cloudProvider.GetDefaultProject()
 	if len(zones) != 1 {
 		return nil, fmt.Errorf("got wrong number of zones for zonal create volume: %v", len(zones))
 	}
 	diskZone := zones[0]
-	err := cloudProvider.InsertDisk(ctx, project, meta.ZonalKey(name, diskZone), params, capBytes, capacityRange, nil, snapshotID, volumeContentSourceVolumeID, multiWriter)
+	err := cloudProvider.InsertDisk(ctx, project, meta.ZonalKey(name, diskZone), params, capBytes, capacityRange, nil, snapshotID, multiWriter)
 	if err != nil {
 		return nil, fmt.Errorf("failed to insert zonal disk: %v", err)
 	}
diff --git a/pkg/gce-pd-csi-driver/gce-pd-driver.go b/pkg/gce-pd-csi-driver/gce-pd-driver.go
index 4dcb3470..e4f48f65 100644
--- a/pkg/gce-pd-csi-driver/gce-pd-driver.go
+++ b/pkg/gce-pd-csi-driver/gce-pd-driver.go
@@ -20,6 +20,7 @@ import (
 	csi "github.com/container-storage-interface/spec/lib/go/csi"
 	"google.golang.org/grpc/codes"
 	"google.golang.org/grpc/status"
+	"k8s.io/client-go/util/workqueue"
 	"k8s.io/klog"
 	"k8s.io/mount-utils"
 	common "sigs.k8s.io/gcp-compute-persistent-disk-csi-driver/pkg/common"
@@ -67,7 +68,6 @@ func (gceDriver *GCEDriver) SetupGCEDriver(name, vendorVersion string, extraVolu
 		csi.ControllerServiceCapability_RPC_EXPAND_VOLUME,
 		csi.ControllerServiceCapability_RPC_LIST_VOLUMES,
 		csi.ControllerServiceCapability_RPC_LIST_VOLUMES_PUBLISHED_NODES,
-		csi.ControllerServiceCapability_RPC_CLONE_VOLUME,
 	}
 	gceDriver.AddControllerServiceCapabilities(csc)
 	ns := []csi.NodeServiceCapability_RPC_Type{
@@ -150,16 +150,22 @@ func NewNodeServer(gceDriver *GCEDriver, mounter *mount.SafeFormatAndMount, devi
 
 func NewControllerServer(gceDriver *GCEDriver, cloudProvider gce.GCECompute) *GCEControllerServer {
 	return &GCEControllerServer{
-		Driver:        gceDriver,
-		CloudProvider: cloudProvider,
-		seen:          map[string]int{},
-		volumeLocks:   common.NewVolumeLocks(),
+		Driver:                  gceDriver,
+		CloudProvider:           cloudProvider,
+		seen:                    map[string]int{},
+		volumeLocks:             common.NewVolumeLocks(),
+		queue:                   workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "controllerserver"),
+		publishErrorsSeenOnNode: map[string]bool{},
 	}
 }
 
 func (gceDriver *GCEDriver) Run(endpoint string) {
 	klog.V(4).Infof("Driver: %v", gceDriver.name)
 
+	if gceDriver.cs != nil {
+		gceDriver.cs.Run()
+	}
+
 	//Start the nonblocking GRPC
 	s := NewNonBlockingGRPCServer()
 	// TODO(#34): Only start specific servers based on a flag.
@@ -167,5 +173,6 @@ func (gceDriver *GCEDriver) Run(endpoint string) {
 	// The schema for that was in util. basically it was just s.start but with some nil servers.
 
 	s.Start(endpoint, gceDriver.ids, gceDriver.cs, gceDriver.ns)
+
 	s.Wait()
 }


Test patch:
diff --git a/pkg/gce-pd-csi-driver/controller_test.go b/pkg/gce-pd-csi-driver/controller_test.go
index fbba1058..eb8693db 100644
--- a/pkg/gce-pd-csi-driver/controller_test.go
+++ b/pkg/gce-pd-csi-driver/controller_test.go
@@ -31,6 +31,7 @@ import (
 	"google.golang.org/grpc/codes"
 	"google.golang.org/grpc/status"
 	"k8s.io/apimachinery/pkg/util/sets"
+	"k8s.io/apimachinery/pkg/util/wait"
 
 	csi "github.com/container-storage-interface/spec/lib/go/csi"
 	"sigs.k8s.io/gcp-compute-persistent-disk-csi-driver/pkg/common"
@@ -63,6 +64,7 @@ var (
 	region, _      = common.GetRegionFromZones([]string{zone})
 	testRegionalID = fmt.Sprintf("projects/%s/regions/%s/disks/%s", project, region, name)
 	testSnapshotID = fmt.Sprintf("projects/%s/global/snapshots/%s", project, name)
+	testNodeID     = fmt.Sprintf("projects/%s/zones/%s/instances/%s", project, zone, node)
 )
 
 func TestCreateSnapshotArguments(t *testing.T) {
@@ -824,7 +826,7 @@ func TestListVolumeArgs(t *testing.T) {
 	}
 }
 
-func TestCreateVolumeWithVolumeSourceFromSnapshot(t *testing.T) {
+func TestCreateVolumeWithVolumeSource(t *testing.T) {
 	// Define test cases
 	testCases := []struct {
 		name            string
@@ -900,162 +902,6 @@ func TestCreateVolumeWithVolumeSourceFromSnapshot(t *testing.T) {
 	}
 }
 
-func TestCreateVolumeWithVolumeSourceFromVolume(t *testing.T) {
-	testSourceVolumeName := "test-volume-source-name"
-	testZonalVolumeSourceID := fmt.Sprintf("projects/%s/zones/%s/disks/%s", project, zone, testSourceVolumeName)
-	testRegionalVolumeSourceID := fmt.Sprintf("projects/%s/regions/%s/disks/%s", project, region, testSourceVolumeName)
-	testVolumeSourceIDDifferentZone := fmt.Sprintf("projects/%s/zones/%s/disks/%s", project, "different-zone", testSourceVolumeName)
-	topology := &csi.TopologyRequirement{
-		Requisite: []*csi.Topology{
-			{
-				Segments: map[string]string{common.TopologyKeyZone: region + "-b"},
-			},
-			{
-				Segments: map[string]string{common.TopologyKeyZone: region + "-c"},
-			},
-		},
-	}
-	regionalParams := map[string]string{
-		common.ParameterKeyType: "test-type", common.ParameterKeyReplicationType: "regional-pd",
-	}
-	// Define test cases
-	testCases := []struct {
-		name                string
-		volumeOnCloud       bool
-		expErrCode          codes.Code
-		sourceVolumeID      string
-		reqParameters       map[string]string
-		sourceReqParameters map[string]string
-		topology            *csi.TopologyRequirement
-	}{
-		{
-			name:                "success with data source of zonal volume type",
-			volumeOnCloud:       true,
-			sourceVolumeID:      testZonalVolumeSourceID,
-			reqParameters:       stdParams,
-			sourceReqParameters: stdParams,
-		},
-		{
-			name:                "success with data source of regional volume type",
-			volumeOnCloud:       true,
-			sourceVolumeID:      testRegionalVolumeSourceID,
-			reqParameters:       regionalParams,
-			sourceReqParameters: regionalParams,
-			topology:            topology,
-		},
-		{
-			name:                "fail with with data source of replication-type different from CreateVolumeRequest",
-			volumeOnCloud:       true,
-			expErrCode:          codes.InvalidArgument,
-			sourceVolumeID:      testZonalVolumeSourceID,
-			reqParameters:       stdParams,
-			sourceReqParameters: regionalParams,
-			topology:            topology,
-		},
-		{
-			name:                "fail with data source of zonal volume type that doesn't exist",
-			volumeOnCloud:       false,
-			expErrCode:          codes.NotFound,
-			sourceVolumeID:      testZonalVolumeSourceID,
-			reqParameters:       stdParams,
-			sourceReqParameters: stdParams,
-		},
-		{
-			name:                "fail with data source of zonal volume type with invalid volume id format",
-			volumeOnCloud:       false,
-			expErrCode:          codes.InvalidArgument,
-			sourceVolumeID:      testZonalVolumeSourceID + "invalid/format",
-			reqParameters:       stdParams,
-			sourceReqParameters: stdParams,
-		},
-		{
-			name:           "fail with data source of zonal volume type with invalid disk parameters",
-			volumeOnCloud:  true,
-			expErrCode:     codes.InvalidArgument,
-			sourceVolumeID: testVolumeSourceIDDifferentZone,
-			reqParameters:  stdParams,
-			sourceReqParameters: map[string]string{
-				common.ParameterKeyType: "different-type",
-			},
-		},
-		{
-			name:                "fail with data source of zonal volume type with invalid replication type",
-			volumeOnCloud:       true,
-			expErrCode:          codes.InvalidArgument,
-			sourceVolumeID:      testZonalVolumeSourceID,
-			reqParameters:       regionalParams,
-			sourceReqParameters: stdParams,
-		},
-	}
-
-	for _, tc := range testCases {
-		t.Logf("test case: %s", tc.name)
-		gceDriver := initGCEDriver(t, nil)
-
-		req := &csi.CreateVolumeRequest{
-			Name:               name,
-			CapacityRange:      stdCapRange,
-			VolumeCapabilities: stdVolCaps,
-			Parameters:         tc.reqParameters,
-			VolumeContentSource: &csi.VolumeContentSource{
-				Type: &csi.VolumeContentSource_Volume{
-					Volume: &csi.VolumeContentSource_VolumeSource{
-						VolumeId: tc.sourceVolumeID,
-					},
-				},
-			},
-		}
-
-		sourceVolumeRequest := &csi.CreateVolumeRequest{
-			Name:               testSourceVolumeName,
-			CapacityRange:      stdCapRange,
-			VolumeCapabilities: stdVolCaps,
-			Parameters:         tc.sourceReqParameters,
-		}
-
-		if tc.topology != nil {
-			// req.AccessibilityRequirements = tc.topology
-			sourceVolumeRequest.AccessibilityRequirements = tc.topology
-		}
-
-		if tc.volumeOnCloud {
-			// Create the source volume.
-			sourceVolume, _ := gceDriver.cs.CreateVolume(context.Background(), sourceVolumeRequest)
-			req.VolumeContentSource = &csi.VolumeContentSource{
-				Type: &csi.VolumeContentSource_Volume{
-					Volume: &csi.VolumeContentSource_VolumeSource{
-						VolumeId: sourceVolume.GetVolume().VolumeId,
-					},
-				},
-			}
-		}
-
-		resp, err := gceDriver.cs.CreateVolume(context.Background(), req)
-		t.Logf("response: %v err: %v", resp, err)
-		if err != nil {
-			serverError, ok := status.FromError(err)
-			if !ok {
-				t.Fatalf("Could not get error status code from err: %v", serverError)
-			}
-			if serverError.Code() != tc.expErrCode {
-				t.Fatalf("Expected error code: %v, got: %v. err : %v", tc.expErrCode, serverError.Code(), err)
-			}
-			continue
-		}
-		if tc.expErrCode != codes.OK {
-			t.Fatalf("Expected error: %v, got no error", tc.expErrCode)
-		}
-
-		// Make sure the response has the source volume.
-		sourceVolume := resp.GetVolume()
-		t.Logf("response has source volume: %v ", sourceVolume)
-		if sourceVolume.ContentSource == nil || sourceVolume.ContentSource.Type == nil ||
-			sourceVolume.ContentSource.GetVolume() == nil || sourceVolume.ContentSource.GetVolume().VolumeId == "" {
-			t.Fatalf("Expected volume content source to have volume ID, got none")
-		}
-	}
-}
-
 func TestCreateVolumeRandomRequisiteTopology(t *testing.T) {
 	req := &csi.CreateVolumeRequest{
 		Name:               "test-name",
@@ -1872,3 +1718,185 @@ func TestCreateVolumeDiskReady(t *testing.T) {
 		})
 	}
 }
+
+func TestControllerPublishUnpublishVolume(t *testing.T) {
+	testCases := []struct {
+		name              string
+		seedDisks         []*gce.CloudDisk
+		pubReq            *csi.ControllerPublishVolumeRequest
+		unpubReq          *csi.ControllerUnpublishVolumeRequest
+		errorSeenOnNode   bool
+		fakeCloudProvider bool
+	}{
+		{
+			name: "queue up publish requests if node has publish error",
+			seedDisks: []*gce.CloudDisk{
+				createZonalCloudDisk(name),
+			},
+			pubReq: &csi.ControllerPublishVolumeRequest{
+				VolumeId: testVolumeID,
+				NodeId:   testNodeID,
+				VolumeCapability: &csi.VolumeCapability{
+					AccessType: &csi.VolumeCapability_Mount{
+						Mount: &csi.VolumeCapability_MountVolume{},
+					},
+					AccessMode: &csi.VolumeCapability_AccessMode{
+						Mode: csi.VolumeCapability_AccessMode_SINGLE_NODE_WRITER,
+					},
+				},
+			},
+			errorSeenOnNode:   true,
+			fakeCloudProvider: false,
+		},
+		{
+			name: "queue up and process publish requests if node has publish error",
+			seedDisks: []*gce.CloudDisk{
+				createZonalCloudDisk(name),
+			},
+			pubReq: &csi.ControllerPublishVolumeRequest{
+				VolumeId: testVolumeID,
+				NodeId:   testNodeID,
+				VolumeCapability: &csi.VolumeCapability{
+					AccessType: &csi.VolumeCapability_Mount{
+						Mount: &csi.VolumeCapability_MountVolume{},
+					},
+					AccessMode: &csi.VolumeCapability_AccessMode{
+						Mode: csi.VolumeCapability_AccessMode_SINGLE_NODE_WRITER,
+					},
+				},
+			},
+			errorSeenOnNode:   true,
+			fakeCloudProvider: true,
+		},
+		{
+			name: "do not queue up publish requests if node doesn't have publish error",
+			seedDisks: []*gce.CloudDisk{
+				createZonalCloudDisk(name),
+			},
+			pubReq: &csi.ControllerPublishVolumeRequest{
+				VolumeId: testVolumeID,
+				NodeId:   testNodeID,
+				VolumeCapability: &csi.VolumeCapability{
+					AccessType: &csi.VolumeCapability_Mount{
+						Mount: &csi.VolumeCapability_MountVolume{},
+					},
+					AccessMode: &csi.VolumeCapability_AccessMode{
+						Mode: csi.VolumeCapability_AccessMode_SINGLE_NODE_WRITER,
+					},
+				},
+			},
+			errorSeenOnNode:   false,
+			fakeCloudProvider: false,
+		},
+		{
+			name: "queue up unpublish requests if node has publish error",
+			seedDisks: []*gce.CloudDisk{
+				createZonalCloudDisk(name),
+			},
+			unpubReq: &csi.ControllerUnpublishVolumeRequest{
+				VolumeId: testVolumeID,
+				NodeId:   testNodeID,
+			},
+			errorSeenOnNode:   true,
+			fakeCloudProvider: false,
+		},
+		{
+			name: "queue up and process unpublish requests if node has publish error",
+			seedDisks: []*gce.CloudDisk{
+				createZonalCloudDisk(name),
+			},
+			unpubReq: &csi.ControllerUnpublishVolumeRequest{
+				VolumeId: testVolumeID,
+				NodeId:   testNodeID,
+			},
+			errorSeenOnNode:   true,
+			fakeCloudProvider: true,
+		},
+		{
+			name: "do not queue up unpublish requests if node doesn't have publish error",
+			seedDisks: []*gce.CloudDisk{
+				createZonalCloudDisk(name),
+			},
+			unpubReq: &csi.ControllerUnpublishVolumeRequest{
+				VolumeId: testVolumeID,
+				NodeId:   testNodeID,
+			},
+			errorSeenOnNode:   false,
+			fakeCloudProvider: false,
+		},
+	}
+	for _, tc := range testCases {
+		t.Logf("test case: %s", tc.name)
+
+		var gceDriver *GCEDriver
+
+		if tc.fakeCloudProvider {
+			fcp, err := gce.CreateFakeCloudProvider(project, zone, tc.seedDisks)
+			if err != nil {
+				t.Fatalf("Failed to create fake cloud provider: %v", err)
+			}
+
+			instance := &compute.Instance{
+				Name:  node,
+				Disks: []*compute.AttachedDisk{},
+			}
+			fcp.InsertInstance(instance, zone, node)
+
+			// Setup new driver each time so no interference
+			gceDriver = initGCEDriverWithCloudProvider(t, fcp)
+		} else {
+			gceDriver = initGCEDriver(t, tc.seedDisks)
+		}
+
+		// mark the node in the map
+		if tc.errorSeenOnNode {
+			gceDriver.cs.publishErrorsSeenOnNode[testNodeID] = true
+		}
+
+		requestCount := 50
+		for i := 0; i < requestCount; i++ {
+			if tc.pubReq != nil {
+				gceDriver.cs.ControllerPublishVolume(context.Background(), tc.pubReq)
+			}
+
+			if tc.unpubReq != nil {
+				gceDriver.cs.ControllerUnpublishVolume(context.Background(), tc.unpubReq)
+			}
+		}
+
+		queued := false
+
+		if tc.errorSeenOnNode {
+			if err := wait.Poll(10*time.Nanosecond, 1*time.Second, func() (bool, error) {
+				if gceDriver.cs.queue.Len() > 0 {
+					queued = true
+
+					if tc.fakeCloudProvider {
+						gceDriver.cs.Run()
+					}
+				}
+
+				// Items are queued up and eventually all processed
+				if tc.fakeCloudProvider {
+					return queued && gceDriver.cs.queue.Len() == 0, nil
+				}
+
+				return gceDriver.cs.queue.Len() == requestCount, nil
+			}); err != nil {
+				if tc.fakeCloudProvider {
+					t.Fatalf("%v requests not processed for node has seen error", gceDriver.cs.queue.Len())
+				} else {
+					t.Fatalf("Only %v requests queued up for node has seen error", gceDriver.cs.queue.Len())
+				}
+			}
+		}
+
+		if !tc.errorSeenOnNode {
+			if err := wait.Poll(10*time.Nanosecond, 10*time.Millisecond, func() (bool, error) {
+				return gceDriver.cs.queue.Len() != 0, nil
+			}); err == nil {
+				t.Fatalf("%v requests queued up for node hasn't seen error", gceDriver.cs.queue.Len())
+			}
+		}
+	}
+}
diff --git a/test/k8s-integration/driver-config.go b/test/k8s-integration/driver-config.go
index 97d7289e..c612ea91 100644
--- a/test/k8s-integration/driver-config.go
+++ b/test/k8s-integration/driver-config.go
@@ -76,6 +76,7 @@ func generateDriverConfigFile(testParams *testParameters, storageClassFile strin
 	}
 
 	/* Unsupported Capabilities:
+	   pvcDataSource
 	   RWX
 	   volumeLimits # PD Supports volume limits but test is very slow
 	   singleNodeVolume
@@ -116,7 +117,6 @@ func generateDriverConfigFile(testParams *testParameters, storageClassFile strin
 		absSnapshotClassFilePath = filepath.Join(testParams.pkgDir, testConfigDir, testParams.snapshotClassFile)
 	}
 
-	caps = append(caps, "pvcDataSource")
 	minimumVolumeSize := "5Gi"
 	numAllowedTopologies := 1
 	if storageClassFile == regionalPDStorageClass {
diff --git a/test/k8s-integration/main.go b/test/k8s-integration/main.go
index a456e155..c6646070 100644
--- a/test/k8s-integration/main.go
+++ b/test/k8s-integration/main.go
@@ -543,9 +543,6 @@ func generateGCETestSkip(testParams *testParameters) string {
 	if testParams.platform == "windows" {
 		skipString = skipString + "|\\[LinuxOnly\\]"
 	}
-	// Volume cloning has timeouts due to GCE disk cloning rate limits, and operation serialization
-	// race conditions where the cloning begins while the source disk is still being created.
-	skipString = skipString + "|provisioning\\sshould\\sprovision\\sstorage\\swith\\spvc\\sdata\\ssource[^|]*"
 	return skipString
 }
 
@@ -588,9 +585,7 @@ func generateGKETestSkip(testParams *testParameters) string {
 		(!testParams.useGKEManagedDriver && (*curVer).lessThan(mustParseVersion("1.17.0"))) {
 		skipString = skipString + "|VolumeSnapshotDataSource"
 	}
-	// Volume cloning has timeouts due to GCE disk cloning rate limits, and operation serialization
-	// race conditions where the cloning begins while the source disk is still being created.
-	skipString = skipString + "|provisioning\\sshould\\sprovision\\sstorage\\swith\\spvc\\sdata\\ssource[^|]*"
+
 	return skipString
 }
 
diff --git a/test/run-k8s-integration-ci.sh b/test/run-k8s-integration-ci.sh
index c10f7ca7..4f79f673 100755
--- a/test/run-k8s-integration-ci.sh
+++ b/test/run-k8s-integration-ci.sh
@@ -52,10 +52,11 @@ export GCE_PD_VERBOSITY=9
 make -C "${PKGDIR}" test-k8s-integration
 
 if [ "$use_kubetest2" = true ]; then
-    go install sigs.k8s.io/kubetest2@latest;
-    go install sigs.k8s.io/kubetest2/kubetest2-gce@latest;
-    go install sigs.k8s.io/kubetest2/kubetest2-gke@latest;
-    go install sigs.k8s.io/kubetest2/kubetest2-tester-ginkgo@latest;
+    export GO111MODULE=on;
+    go get sigs.k8s.io/kubetest2@latest;
+    go get sigs.k8s.io/kubetest2/kubetest2-gce@latest;
+    go get sigs.k8s.io/kubetest2/kubetest2-gke@latest;
+    go get sigs.k8s.io/kubetest2/kubetest2-tester-ginkgo@latest;
 fi
 
 base_cmd="${PKGDIR}/bin/k8s-integration-test \
diff --git a/test/run-k8s-integration-migration.sh b/test/run-k8s-integration-migration.sh
index ad2d2b61..203b6603 100755
--- a/test/run-k8s-integration-migration.sh
+++ b/test/run-k8s-integration-migration.sh
@@ -24,10 +24,11 @@ readonly use_kubetest2=${USE_KUBETEST2:-false}
 make -C "${PKGDIR}" test-k8s-integration
 
 if [ "$use_kubetest2" = true ]; then
-    go install sigs.k8s.io/kubetest2@latest;
-    go install sigs.k8s.io/kubetest2/kubetest2-gce@latest;
-    go install sigs.k8s.io/kubetest2/kubetest2-gke@latest;
-    go install sigs.k8s.io/kubetest2/kubetest2-tester-ginkgo@latest;
+    export GO111MODULE=on;
+    go get sigs.k8s.io/kubetest2@latest;
+    go get sigs.k8s.io/kubetest2/kubetest2-gce@latest;
+    go get sigs.k8s.io/kubetest2/kubetest2-gke@latest;
+    go get sigs.k8s.io/kubetest2/kubetest2-tester-ginkgo@latest;
 fi
 
 readonly GCE_PD_TEST_FOCUS="PersistentVolumes\sGCEPD|[V|v]olume\sexpand|\[sig-storage\]\sIn-tree\sVolumes\s\[Driver:\sgcepd\]|allowedTopologies|Pod\sDisks|PersistentVolumes\sDefault"
diff --git a/test/run-windows-k8s-integration.sh b/test/run-windows-k8s-integration.sh
index 08b204dd..6d902298 100755
--- a/test/run-windows-k8s-integration.sh
+++ b/test/run-windows-k8s-integration.sh
@@ -21,10 +21,11 @@ readonly use_kubetest2=${USE_KUBETEST2:-true}
 make -C "${PKGDIR}" test-k8s-integration
 
 if [ "$use_kubetest2" = true ]; then
-    go install sigs.k8s.io/kubetest2@latest;
-    go install sigs.k8s.io/kubetest2/kubetest2-gce@latest;
-    go install sigs.k8s.io/kubetest2/kubetest2-gke@latest;
-    go install sigs.k8s.io/kubetest2/kubetest2-tester-ginkgo@latest;
+    export GO111MODULE=on;
+    go get sigs.k8s.io/kubetest2@latest;
+    go get sigs.k8s.io/kubetest2/kubetest2-gce@latest;
+    go get sigs.k8s.io/kubetest2/kubetest2-gke@latest;
+    go get sigs.k8s.io/kubetest2/kubetest2-tester-ginkgo@latest;
 fi
 
 base_cmd="${PKGDIR}/bin/k8s-integration-test \