Repository: kubernetes-sigs/gcp-compute-persistent-disk-csi-driver
Pull request #1261 (2023-06-14 17:03:32)

Filter out GCE operation error codes caused by user errors
<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Follow the instructions for writing a release note: https://git.k8s.io/community/contributors/guide/release-notes.md
3. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

**What type of PR is this?**
> Uncomment only one ` /kind <>` line, hit enter to put that in a new line, and remove leading whitespaces from that line:
>
> /kind api-change
> /kind bug

 /kind cleanup

> /kind design
> /kind documentation
> /kind failing-test
> /kind feature
> /kind flake

**What this PR does / why we need it**:
Filter out GCE operation error codes caused by user errors. We are getting lots of errors counting against our SLO for GCE operation errors that are user errors. We have seen Internal errors returned from the following GCE operation error codes: `OPERATION_CANCELED_BY_USER`, `RESOURCE_NOT_FOUND`, `RESOURCE_IN_USE_BY_ANOTHER_RESOURCE`, `INVALID_USAGE`. I added a filtered out other error codes (quota error codes) which we haven't observed yet, but those are clearly user error codes. The errors that we have seen so far look like this:
```
ControllerPublishVolume returned with error: rpc error: code = Internal desc = unknown Attach error: failed when waiting for zonal op: operation operation-xxx failed (OPERATION_CANCELED_BY_USER): Operation was canceled by user ''.

ControllerPublishVolume returned with error: rpc error: code = Internal desc = unknown Attach error: failed when waiting for zonal op: operation operation-xxx failed (RESOURCE_NOT_FOUND): The resource 'projects/xxx/zones/xxx/instances/xxx' was not found

ControllerPublishVolume returned with error: rpc error: code = Internal desc = unknown Attach error: failed when waiting for zonal op: operation operation-xxx failed (RESOURCE_IN_USE_BY_ANOTHER_RESOURCE): The disk resource 'projects/xxx/zones/xxx/disks/xxx' is already being used by 'projects/xxx/zones/xxx/instances/xxx'

ControllerUnpublishVolume returned with error: rpc error: code = Internal desc = unknown detach error: operation operation-xxx failed (INVALID_USAGE): No attached disk found with device name 'pvc-xxx'
```
**Which issue(s) this PR fixes**:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes #

**Special notes for your reviewer**:

**Does this PR introduce a user-facing change?**:
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".
-->
```release-note
None
```


k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/pull/1261#" title="Author self-approved">amacaskill</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fgcp-compute-persistent-disk-csi-driver).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/blob/master/OWNERS)~~ [amacaskill]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
amacaskill: /assign @saikat-royc 
k8s-ci-robot: @amacaskill: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-gcp-compute-persistent-disk-csi-driver-verify | 2f66350966681e09576747047adefbdda23cf618 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/1261/pull-gcp-compute-persistent-disk-csi-driver-verify/1669027814386438144/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-verify`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=gcp-compute-persistent-disk-csi-driver&pr=1261). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Aamacaskill). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @amacaskill: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-gcp-compute-persistent-disk-csi-driver-verify | 2f66350966681e09576747047adefbdda23cf618 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/1261/pull-gcp-compute-persistent-disk-csi-driver-verify/1669027814386438144/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-verify`
pull-gcp-compute-persistent-disk-csi-driver-unit | 2f66350966681e09576747047adefbdda23cf618 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/1261/pull-gcp-compute-persistent-disk-csi-driver-unit/1669027814331912192/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-unit`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=gcp-compute-persistent-disk-csi-driver&pr=1261). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Aamacaskill). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @amacaskill: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-gcp-compute-persistent-disk-csi-driver-unit | dbc1c438a9d6f3f9e04f83283fe5c7442b6bbb6b | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-compute-persistent-disk-csi-driver/1261/pull-gcp-compute-persistent-disk-csi-driver-unit/1669040442198462464/) | true | `/test pull-gcp-compute-persistent-disk-csi-driver-unit`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=gcp-compute-persistent-disk-csi-driver&pr=1261). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Aamacaskill). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->

Patch:
diff --git a/pkg/gce-cloud-provider/compute/gce-compute.go b/pkg/gce-cloud-provider/compute/gce-compute.go
index 82f15431..d1f808ad 100644
--- a/pkg/gce-cloud-provider/compute/gce-compute.go
+++ b/pkg/gce-cloud-provider/compute/gce-compute.go
@@ -856,6 +856,10 @@ func (cloud *CloudProvider) WaitForAttach(ctx context.Context, project string, v
 }
 
 func wrapOpErr(name string, opErr *computev1.OperationErrorErrors) error {
+	if opErr == nil {
+		return nil
+	}
+
 	if opErr.Code == "UNSUPPORTED_OPERATION" {
 		if diskType := pdDiskTypeUnsupportedRegex.FindStringSubmatch(opErr.Message); diskType != nil {
 			return &UnsupportedDiskError{
@@ -863,7 +867,29 @@ func wrapOpErr(name string, opErr *computev1.OperationErrorErrors) error {
 			}
 		}
 	}
-	return fmt.Errorf("operation %v failed (%v): %v", name, opErr.Code, opErr.Message)
+	grpcErrCode := codeForGCEOpError(*opErr)
+	return status.Errorf(grpcErrCode, "operation %v failed (%v): %v", name, opErr.Code, opErr.Message)
+}
+
+// codeForGCEOpError return the grpc error code for the passed in
+// gce operation error.
+func codeForGCEOpError(err computev1.OperationErrorErrors) codes.Code {
+	userErrors := map[string]codes.Code{
+		"RESOURCE_NOT_FOUND":                        codes.NotFound,
+		"RESOURCE_ALREADY_EXISTS":                   codes.AlreadyExists,
+		"RESOURCE_IN_USE_BY_ANOTHER_RESOURCE":       codes.InvalidArgument,
+		"OPERATION_CANCELED_BY_USER":                codes.Aborted,
+		"QUOTA_EXCEEDED":                            codes.ResourceExhausted,
+		"ZONE_RESOURCE_POOL_EXHAUSTED":              codes.Unavailable,
+		"ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS": codes.Unavailable,
+		"REGION_QUOTA_EXCEEDED":                     codes.ResourceExhausted,
+		"RATE_LIMIT_EXCEEDED":                       codes.ResourceExhausted,
+		"INVALID_USAGE":                             codes.InvalidArgument,
+	}
+	if code, ok := userErrors[err.Code]; ok {
+		return code
+	}
+	return codes.Internal
 }
 
 func opIsDone(op *computev1.Operation) (bool, error) {


Test patch:
diff --git a/pkg/gce-cloud-provider/compute/gce-compute_test.go b/pkg/gce-cloud-provider/compute/gce-compute_test.go
index 1a9c9f6e..652b111a 100644
--- a/pkg/gce-cloud-provider/compute/gce-compute_test.go
+++ b/pkg/gce-cloud-provider/compute/gce-compute_test.go
@@ -18,6 +18,7 @@ import (
 	"testing"
 
 	computev1 "google.golang.org/api/compute/v1"
+	"google.golang.org/grpc/codes"
 	"sigs.k8s.io/gcp-compute-persistent-disk-csi-driver/pkg/common"
 )
 
@@ -101,3 +102,75 @@ func TestValidateDiskParameters(t *testing.T) {
 		}
 	}
 }
+
+func TestCodeForGCEOpError(t *testing.T) {
+	testCases := []struct {
+		name     string
+		inputErr computev1.OperationErrorErrors
+		expCode  codes.Code
+	}{
+		{
+			name:     "RESOURCE_NOT_FOUND error",
+			inputErr: computev1.OperationErrorErrors{Code: "RESOURCE_NOT_FOUND"},
+			expCode:  codes.NotFound,
+		},
+		{
+			name:     "RESOURCE_ALREADY_EXISTS error",
+			inputErr: computev1.OperationErrorErrors{Code: "RESOURCE_ALREADY_EXISTS"},
+			expCode:  codes.AlreadyExists,
+		},
+		{
+			name:     "OPERATION_CANCELED_BY_USER error",
+			inputErr: computev1.OperationErrorErrors{Code: "OPERATION_CANCELED_BY_USER"},
+			expCode:  codes.Aborted,
+		},
+		{
+			name:     "QUOTA_EXCEEDED error",
+			inputErr: computev1.OperationErrorErrors{Code: "QUOTA_EXCEEDED"},
+			expCode:  codes.ResourceExhausted,
+		},
+		{
+			name:     "ZONE_RESOURCE_POOL_EXHAUSTED error",
+			inputErr: computev1.OperationErrorErrors{Code: "ZONE_RESOURCE_POOL_EXHAUSTED"},
+			expCode:  codes.Unavailable,
+		},
+		{
+			name:     "ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS error",
+			inputErr: computev1.OperationErrorErrors{Code: "ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS"},
+			expCode:  codes.Unavailable,
+		},
+		{
+			name:     "REGION_QUOTA_EXCEEDED error",
+			inputErr: computev1.OperationErrorErrors{Code: "REGION_QUOTA_EXCEEDED"},
+			expCode:  codes.ResourceExhausted,
+		},
+		{
+			name:     "RATE_LIMIT_EXCEEDED error",
+			inputErr: computev1.OperationErrorErrors{Code: "RATE_LIMIT_EXCEEDED"},
+			expCode:  codes.ResourceExhausted,
+		},
+		{
+			name:     "INVALID_USAGE error",
+			inputErr: computev1.OperationErrorErrors{Code: "INVALID_USAGE"},
+			expCode:  codes.InvalidArgument,
+		},
+		{
+			name:     "RESOURCE_IN_USE_BY_ANOTHER_RESOURCE error",
+			inputErr: computev1.OperationErrorErrors{Code: "RESOURCE_IN_USE_BY_ANOTHER_RESOURCE"},
+			expCode:  codes.InvalidArgument,
+		},
+		{
+			name:     "UNSUPPORTED_OPERATION error",
+			inputErr: computev1.OperationErrorErrors{Code: "UNSUPPORTED_OPERATION"},
+			expCode:  codes.Internal,
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Logf("Running test: %v", tc.name)
+		errCode := codeForGCEOpError(tc.inputErr)
+		if errCode != tc.expCode {
+			t.Errorf("test %v failed: got %v, expected %v", tc.name, errCode, tc.expCode)
+		}
+	}
+}