Repository: kubernetes-sigs/gcp-filestore-csi-driver
Pull request #192 (2021-12-30 02:08:39)

enable dynamic provision with shared-vpc from service project
in order to do that, we need to pass appropriate `connect-mode`

<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Follow the instructions for writing a release note: https://git.k8s.io/community/contributors/guide/release-notes.md
3. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

**What type of PR is this?**

/kind feature


**What this PR does / why we need it**:
provision Filestore instances with shared-vpc from service project went into public preview and we need to support it in filestore csi driver

**Which issue(s) this PR fixes**:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes #

**Special notes for your reviewer**:

**Does this PR introduce a user-facing change?**:
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".
-->
```release-note
new parameter `connect-mode` needs to be set to `PRIVATE_SERVICE_ACCESS` if provisioning Filestore instances from service projects of shared-vpc. If a specific range needs to be specified, `reserved-ip-range` needs to be used instead of a CIDR range
```


k8s-ci-robot: Hi @leiyiz. Thanks for your PR.

I'm waiting for a [kubernetes-sigs](https://github.com/orgs/kubernetes-sigs/people) member to verify that this patch is reasonable to test. If it is, they should reply with `/ok-to-test` on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should [join the org](https://git.k8s.io/community/community-membership.md#member) to skip this step.

Once the patch is verified, the new status will be reflected by the `ok-to-test` label.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fgcp-filestore-csi-driver).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>

k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/gcp-filestore-csi-driver/pull/192#" title="Author self-approved">leiyiz</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **saad-ali** after the PR has been reviewed.
You can assign the PR to them by writing `/assign @saad-ali` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fgcp-filestore-csi-driver).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-sigs/gcp-filestore-csi-driver/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["saad-ali"]} -->
jingxu97: /approve
k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/gcp-filestore-csi-driver/pull/192#issuecomment-1003447495" title="Approved">jingxu97</a>*, *<a href="https://github.com/kubernetes-sigs/gcp-filestore-csi-driver/pull/192#" title="Author self-approved">leiyiz</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **saad-ali** after the PR has been reviewed.
You can assign the PR to them by writing `/assign @saad-ali` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fgcp-filestore-csi-driver).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-sigs/gcp-filestore-csi-driver/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["saad-ali"]} -->
saikat-royc: /ok-to-test
saikat-royc: /test pull-gcp-filestore-csi-driver-kubernetes-integration
k8s-ci-robot: @leiyiz: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-gcp-filestore-csi-driver-unit | 945d4598dae756a12078ae7c3c6edd3906dbe928 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-filestore-csi-driver/192/pull-gcp-filestore-csi-driver-unit/1478146391279669248/) | true | `/test pull-gcp-filestore-csi-driver-unit`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=gcp-filestore-csi-driver&pr=192). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Aleiyiz). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/gcp-filestore-csi-driver/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @leiyiz: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-gcp-filestore-csi-driver-e2e | 4b240e68e06699d2a83f0ff557d081e437ba0b9d | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_gcp-filestore-csi-driver/192/pull-gcp-filestore-csi-driver-e2e/1478503206576197632/) | true | `/test pull-gcp-filestore-csi-driver-e2e`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=gcp-filestore-csi-driver&pr=192). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Aleiyiz). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/gcp-filestore-csi-driver/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->

Patch:
diff --git a/pkg/cloud_provider/file/file.go b/pkg/cloud_provider/file/file.go
index a476d116..bf2ecf09 100644
--- a/pkg/cloud_provider/file/file.go
+++ b/pkg/cloud_provider/file/file.go
@@ -53,6 +53,7 @@ type Volume struct {
 
 type Network struct {
 	Name            string
+	ConnectMode     string
 	ReservedIpRange string
 	Ip              string
 }
@@ -124,18 +125,20 @@ func (manager *gcfsServiceManager) CreateInstance(ctx context.Context, obj *Serv
 				Network:         obj.Network.Name,
 				Modes:           []string{"MODE_IPV4"},
 				ReservedIpRange: obj.Network.ReservedIpRange,
+				ConnectMode:     obj.Network.ConnectMode,
 			},
 		},
 		Labels: obj.Labels,
 	}
 
-	glog.V(4).Infof("Creating instance %v: location %v, tier %v, capacity %v, network %v, ipRange %v, labels %v",
+	glog.V(4).Infof("Creating instance %v: location %v, tier %v, capacity %v, network %v, ipRange %v, connectMode %v, labels %v",
 		obj.Name,
 		obj.Location,
 		betaObj.Tier,
 		betaObj.FileShares[0].CapacityGb,
 		betaObj.Networks[0].Network,
 		betaObj.Networks[0].ReservedIpRange,
+		betaObj.Networks[0].ConnectMode,
 		betaObj.Labels)
 	op, err := manager.instancesService.Create(locationURI(obj.Project, obj.Location), betaObj).InstanceId(obj.Name).Context(ctx).Do()
 	if err != nil {
@@ -169,19 +172,21 @@ func (manager *gcfsServiceManager) CreateInstanceFromBackupSource(ctx context.Co
 				Network:         obj.Network.Name,
 				Modes:           []string{"MODE_IPV4"},
 				ReservedIpRange: obj.Network.ReservedIpRange,
+				ConnectMode:     obj.Network.ConnectMode,
 			},
 		},
 		Labels: obj.Labels,
 		State:  obj.State,
 	}
 
-	glog.V(4).Infof("Creating instance %v: location %v, tier %v, capacity %v, network %v, ipRange %v, labels %v backup source %v",
+	glog.V(4).Infof("Creating instance %v: location %v, tier %v, capacity %v, network %v, ipRange %v, connectMode %v, labels %v backup source %v",
 		obj.Name,
 		obj.Location,
 		instance.Tier,
 		instance.FileShares[0].CapacityGb,
 		instance.Networks[0].Network,
 		instance.Networks[0].ReservedIpRange,
+		instance.Networks[0].ConnectMode,
 		instance.Labels,
 		instance.FileShares[0].SourceBackup)
 	op, err := manager.instancesService.Create(locationURI(obj.Project, obj.Location), instance).InstanceId(obj.Name).Context(ctx).Do()
@@ -234,6 +239,7 @@ func cloudInstanceToServiceInstance(instance *filev1beta1.Instance) (*ServiceIns
 			Name:            instance.Networks[0].Network,
 			Ip:              instance.Networks[0].IpAddresses[0],
 			ReservedIpRange: instance.Networks[0].ReservedIpRange,
+			ConnectMode:     instance.Networks[0].ConnectMode,
 		},
 		Labels: instance.Labels,
 		State:  instance.State,
@@ -331,17 +337,20 @@ func (manager *gcfsServiceManager) ResizeInstance(ctx context.Context, obj *Serv
 				Network:         obj.Network.Name,
 				Modes:           []string{"MODE_IPV4"},
 				ReservedIpRange: obj.Network.ReservedIpRange,
+				ConnectMode:     obj.Network.ConnectMode,
 			},
 		},
 	}
 
-	glog.V(4).Infof("Patching instance %v: location %v, tier %v, capacity %v, network %v, ipRange %v",
+	glog.V(4).Infof("Patching instance %v: location %v, tier %v, capacity %v, network %v, ipRange %v, connectMode %v",
 		obj.Name,
 		obj.Location,
 		betaObj.Tier,
 		betaObj.FileShares[0].CapacityGb,
 		betaObj.Networks[0].Network,
-		betaObj.Networks[0].ReservedIpRange)
+		betaObj.Networks[0].ReservedIpRange,
+		betaObj.Networks[0].ConnectMode,
+	)
 	op, err := manager.instancesService.Patch(instanceuri, betaObj).UpdateMask(fileShareUpdateMask).Context(ctx).Do()
 	if err != nil {
 		return nil, fmt.Errorf("Patch operation failed: %v", err)
diff --git a/pkg/csi_driver/controller.go b/pkg/csi_driver/controller.go
index 37e61db4..b0e0842b 100644
--- a/pkg/csi_driver/controller.go
+++ b/pkg/csi_driver/controller.go
@@ -40,6 +40,9 @@ const (
 	enterpriseTier = "enterprise"
 	defaultNetwork = "default"
 
+	directPeering        = "DIRECT_PEERING"
+	privateServiceAccess = "PRIVATE_SERVICE_ACCESS"
+
 	// Keys for Topology.
 	TopologyKeyZone = "topology.gke.io/zone"
 )
@@ -56,6 +59,8 @@ const (
 	paramLocation         = "location"
 	paramNetwork          = "network"
 	paramReservedIPV4CIDR = "reserved-ipv4-cidr"
+	paramReservedIPRange  = "reserved-ip-range"
+	paramConnectMode      = "connect-mode"
 
 	// Keys for PV and PVC parameters as reported by external-provisioner
 	ParameterKeyPVCName      = "csi.storage.k8s.io/pvc/name"
@@ -171,9 +176,14 @@ func (s *controllerServer) CreateVolume(ctx context.Context, req *csi.CreateVolu
 			return nil, status.Error(codes.Internal, msg)
 		}
 	} else {
+		param := req.GetParameters()
 		// If we are creating a new instance, we need pick an unused CIDR range from reserved-ipv4-cidr
 		// If the param was not provided, we default reservedIPRange to "" and cloud provider takes care of the allocation
-		if reservedIPV4CIDR, ok := req.GetParameters()[paramReservedIPV4CIDR]; ok {
+		if newFiler.Network.ConnectMode == privateServiceAccess {
+			if reservedIPRange, ok := param[paramReservedIPRange]; ok {
+				newFiler.Network.ReservedIpRange = reservedIPRange
+			}
+		} else if reservedIPV4CIDR, ok := param[paramReservedIPV4CIDR]; ok {
 			reservedIPRange, err := s.reserveIPRange(ctx, newFiler, reservedIPV4CIDR)
 
 			// Possible cases are 1) CreateInstanceAborted, 2)CreateInstance running in background
@@ -189,7 +199,7 @@ func (s *controllerServer) CreateVolume(ctx context.Context, req *csi.CreateVolu
 		}
 
 		// Add labels.
-		labels, err := extractLabels(req.GetParameters(), s.config.driver.config.Name)
+		labels, err := extractLabels(param, s.config.driver.config.Name)
 		if err != nil {
 			return nil, err
 		}
@@ -378,6 +388,7 @@ func (s *controllerServer) generateNewFileInstance(name string, capBytes int64,
 	// Set default parameters
 	tier := defaultTier
 	network := defaultNetwork
+	connectMode := directPeering
 
 	// Validate parameters (case-insensitive).
 	for k, v := range params {
@@ -394,9 +405,15 @@ func (s *controllerServer) generateNewFileInstance(name string, capBytes int64,
 			}
 		case paramNetwork:
 			network = v
+		case paramConnectMode:
+			connectMode = v
+			if connectMode != directPeering && connectMode != privateServiceAccess {
+				return nil, fmt.Errorf("connect mode can only be one of %q or %q", directPeering, privateServiceAccess)
+			}
 		// Ignore the cidr flag as it is not passed to the cloud provider
 		// It will be used to get unreserved IP in the reserveIPV4Range function
-		case paramReservedIPV4CIDR:
+		// ignore IPRange flag as it will be handled at the same place as cidr
+		case paramReservedIPV4CIDR, paramReservedIPRange:
 			continue
 		case ParameterKeyLabels, ParameterKeyPVCName, ParameterKeyPVCNamespace, ParameterKeyPVName:
 		case "csiprovisionersecretname", "csiprovisionersecretnamespace":
@@ -410,7 +427,8 @@ func (s *controllerServer) generateNewFileInstance(name string, capBytes int64,
 		Location: location,
 		Tier:     tier,
 		Network: file.Network{
-			Name: network,
+			Name:        network,
+			ConnectMode: connectMode,
 		},
 		Volume: file.Volume{
 			Name:      newInstanceVolume,


Test patch:
diff --git a/pkg/csi_driver/controller_test.go b/pkg/csi_driver/controller_test.go
index 5e5a6262..a4d89fd2 100644
--- a/pkg/csi_driver/controller_test.go
+++ b/pkg/csi_driver/controller_test.go
@@ -344,7 +344,8 @@ func TestGenerateNewFileInstance(t *testing.T) {
 				Location: testLocation,
 				Tier:     defaultTier,
 				Network: file.Network{
-					Name: defaultNetwork,
+					Name:        defaultNetwork,
+					ConnectMode: directPeering,
 				},
 				Volume: file.Volume{
 					Name:      newInstanceVolume,
@@ -382,7 +383,8 @@ func TestGenerateNewFileInstance(t *testing.T) {
 				Location: "foo-location",
 				Tier:     "foo-tier",
 				Network: file.Network{
-					Name: "foo-network",
+					Name:        "foo-network",
+					ConnectMode: directPeering,
 				},
 				Volume: file.Volume{
 					Name:      newInstanceVolume,
@@ -425,7 +427,8 @@ func TestGenerateNewFileInstance(t *testing.T) {
 				Location: "bar-location",
 				Tier:     "foo-tier",
 				Network: file.Network{
-					Name: "foo-network",
+					Name:        "foo-network",
+					ConnectMode: directPeering,
 				},
 				Volume: file.Volume{
 					Name:      newInstanceVolume,
@@ -456,7 +459,48 @@ func TestGenerateNewFileInstance(t *testing.T) {
 				Location: "foo-location",
 				Tier:     "foo-tier",
 				Network: file.Network{
-					Name: "foo-network",
+					Name:        "foo-network",
+					ConnectMode: directPeering,
+				},
+				Volume: file.Volume{
+					Name:      newInstanceVolume,
+					SizeBytes: testBytes,
+				},
+			},
+		},
+		{
+			name: "custom params, private connect mode",
+			toporeq: &csi.TopologyRequirement{
+				Requisite: []*csi.Topology{
+					{
+						Segments: map[string]string{
+							TopologyKeyZone: "foo-location",
+						},
+					},
+				},
+				Preferred: []*csi.Topology{
+					{
+						Segments: map[string]string{
+							TopologyKeyZone: "foo-location",
+						},
+					},
+				},
+			},
+			params: map[string]string{
+				paramTier:                       "foo-tier",
+				paramNetwork:                    "foo-network",
+				paramConnectMode:                privateServiceAccess,
+				"csiProvisionerSecretName":      "foo-secret",
+				"csiProvisionerSecretNamespace": "foo-namespace",
+			},
+			instance: &file.ServiceInstance{
+				Project:  testProject,
+				Name:     testCSIVolume,
+				Location: "foo-location",
+				Tier:     "foo-tier",
+				Network: file.Network{
+					Name:        "foo-network",
+					ConnectMode: privateServiceAccess,
 				},
 				Volume: file.Volume{
 					Name:      newInstanceVolume,
@@ -471,6 +515,13 @@ func TestGenerateNewFileInstance(t *testing.T) {
 			},
 			expectErr: true,
 		},
+		{
+			name: "invalid connect mode",
+			params: map[string]string{
+				paramConnectMode: "CONNECT_MODE_UNSPECIFIED",
+			},
+			expectErr: true,
+		},
 	}
 
 	for _, test := range cases {