Repository: kubernetes-sigs/cloud-provider-azure
Pull request #1646 (2022-05-04 13:28:25)

feat: add DiskEncryptionType in CreateDisk
<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide/first-contribution.md#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?
/kind feature

#### What this PR does / why we need it:
feat: add DiskEncryptionType in CreateDisk

#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes #

#### Special notes for your reviewer:

#### Does this PR introduce a user-facing change?
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->
```release-note
feat: add DiskEncryptionType in CreateDisk
```

#### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

<!--
This section can be blank if this pull request does not require a release note.

When adding links which point to resources within git repositories, like
KEPs or supporting documentation, please reference a specific commit and avoid
linking directly to the master branch. This ensures that links reference a
specific point in time, rather than a document that may change over time.

See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files

Please use the following format for linking documentation:
- [KEP]: <link>
- [Usage]: <link>
- [Other doc]: <link>
-->
```docs
feat: add DiskEncryptionType in CreateDisk
```


k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/1646#" title="Author self-approved">andyzhangx</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/master/OWNERS)~~ [andyzhangx]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
k8s-ci-robot: @andyzhangx: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-check | 0c5eb100e1b61709e2d8775590c3ada0f40d8846 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/1646/pull-cloud-provider-azure-check/1521844201916993536) | true | `/test pull-cloud-provider-azure-check`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=1646). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Aandyzhangx). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
andyzhangx: chart config is broken, not related to this PR:
https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/1646/pull-cloud-provider-azure-check/1521844201916993536
```
+ tar -xf hack/../helm/repo/cloud-provider-azure-1.23.10.tgz -C chart_verify
+ diff -r hack/../chart_verify/cloud-provider-azure/ hack/../helm/cloud-provider-azure/ --exclude README.md
diff -r --exclude README.md hack/../chart_verify/cloud-provider-azure/Chart.yaml hack/../helm/cloud-provider-azure/Chart.yaml
16c16
< version: 1.23.10
---
> version: 1.23.11
diff -r --exclude README.md hack/../chart_verify/cloud-provider-azure/values.yaml hack/../helm/cloud-provider-azure/values.yaml
17c17
<   # imageTag: "canary"
---
>   imageTag: "v1.23.11"
45c45
<   # imageTag: "canary"
---
>   imageTag: "v1.23.11"
+ exit 1
make: *** [Makefile:334: test-helm] Error 1
+ EXIT_VALUE=2
+ set +o xtrace
```
coveralls: 
[![Coverage Status](https://coveralls.io/builds/48827366/badge)](https://coveralls.io/builds/48827366)

Coverage increased (+0.009%) to 80.475% when pulling **0c5eb100e1b61709e2d8775590c3ada0f40d8846 on andyzhangx:DiskEncryptionType** into **12be276cfe34e0fcad75902e42e2df5d6b9b06ee on kubernetes-sigs:master**.

k8s-ci-robot: @andyzhangx: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-check | 0c5eb100e1b61709e2d8775590c3ada0f40d8846 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/1646/pull-cloud-provider-azure-check/1521844201916993536) | true | `/test pull-cloud-provider-azure-check`
pull-cloud-provider-azure-e2e-ccm-vmss-basic-lb | 0c5eb100e1b61709e2d8775590c3ada0f40d8846 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/1646/pull-cloud-provider-azure-e2e-ccm-vmss-basic-lb/1521844201954742272) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-basic-lb`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=1646). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Aandyzhangx). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
jackfrancis: @andyzhangx I can't repro the non-zero exit code when running `hack/verify-helm-repo.sh` locally.

```
$ git status
On branch DiskEncryptionType
Your branch is up to date with 'andyzhangx/DiskEncryptionType'.

nothing to commit, working tree clean
$ hack/verify-helm-repo.sh 
++ dirname hack/verify-helm-repo.sh
+ REPO_ROOT=hack/..
+ for chart in '"cloud-provider-azure"'
++ ls -t hack/../helm/repo/cloud-provider-azure-1.23.10.tgz hack/../helm/repo/cloud-provider-azure-1.23.11.tgz hack/../helm/repo/cloud-provider-azure-1.23.8.tgz
++ head -n 1
+ LATEST_CHART=hack/../helm/repo/cloud-provider-azure-1.23.11.tgz
++ echo hack/../helm/repo/cloud-provider-azure-1.23.11.tgz
++ grep -Eoq '[0-9]+.[0-9]+.[0-9]+'
+ LATEST_VERSION=
+ MATCH_STRING='version: '
+ grep -q 'version: ' hack/../helm/cloud-provider-azure/Chart.yaml
+ rm -Rf hack/../chart_verify
+ mkdir hack/../chart_verify
+ tar -xf hack/../helm/repo/cloud-provider-azure-1.23.11.tgz -C chart_verify
+ diff -r hack/../chart_verify/cloud-provider-azure/ hack/../helm/cloud-provider-azure/ --exclude README.md
+ rm -Rf hack/../chart_verify
+ exit 0
```
andyzhangx: /retest
k8s-ci-robot: @andyzhangx: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-basic-lb | 0c5eb100e1b61709e2d8775590c3ada0f40d8846 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/1646/pull-cloud-provider-azure-e2e-ccm-vmss-basic-lb/1521844201954742272) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-basic-lb`
pull-cloud-provider-azure-check | 0c5eb100e1b61709e2d8775590c3ada0f40d8846 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/1646/pull-cloud-provider-azure-check/1522018448547254272) | true | `/test pull-cloud-provider-azure-check`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=1646). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Aandyzhangx). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
MartinForReal: please rebase this pr after https://github.com/kubernetes-sigs/cloud-provider-azure/pull/1640 is merged. Thanks!
k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/1646#" title="Author self-approved">andyzhangx</a>*, *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/1646#pullrequestreview-962808932" title="LGTM">ZeroMagic</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/master/OWNERS)~~ [andyzhangx]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
andyzhangx: /retest
k8s-ci-robot: @andyzhangx: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-basic-lb | 0c5eb100e1b61709e2d8775590c3ada0f40d8846 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/1646/pull-cloud-provider-azure-e2e-ccm-vmss-basic-lb/1521844201954742272) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-basic-lb`
pull-cloud-provider-azure-check | 0c5eb100e1b61709e2d8775590c3ada0f40d8846 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/1646/pull-cloud-provider-azure-check/1522043200397643776) | true | `/test pull-cloud-provider-azure-check`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=1646). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Aandyzhangx). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->

Patch:
diff --git a/pkg/provider/azure_managedDiskController.go b/pkg/provider/azure_managedDiskController.go
index 3ea931648..e53eac1b2 100644
--- a/pkg/provider/azure_managedDiskController.go
+++ b/pkg/provider/azure_managedDiskController.go
@@ -66,6 +66,8 @@ type ManagedDiskOptions struct {
 	SourceType string
 	// ResourceId of the disk encryption set to use for enabling encryption at rest.
 	DiskEncryptionSetID string
+	// DiskEncryption type, available values: EncryptionAtRestWithCustomerKey, EncryptionAtRestWithPlatformAndCustomerKeys
+	DiskEncryptionType string
 	// The size in GB.
 	SizeGB int
 	// The maximum number of VMs that can attach to the disk at the same time. Value greater than one indicates a disk that can be mounted on multiple VMs at the same time.
@@ -190,9 +192,18 @@ func (c *ManagedDiskController) CreateManagedDisk(ctx context.Context, options *
 		if strings.Index(strings.ToLower(options.DiskEncryptionSetID), "/subscriptions/") != 0 {
 			return "", fmt.Errorf("AzureDisk - format of DiskEncryptionSetID(%s) is incorrect, correct format: %s", options.DiskEncryptionSetID, consts.DiskEncryptionSetIDFormat)
 		}
+		encryptionType := compute.EncryptionTypeEncryptionAtRestWithCustomerKey
+		if options.DiskEncryptionType != "" {
+			encryptionType = compute.EncryptionType(options.DiskEncryptionType)
+			klog.V(4).Infof("azureDisk - DiskEncryptionType: %s, DiskEncryptionSetID: %s", options.DiskEncryptionType, options.DiskEncryptionSetID)
+		}
 		diskProperties.Encryption = &compute.Encryption{
 			DiskEncryptionSetID: &options.DiskEncryptionSetID,
-			Type:                compute.EncryptionTypeEncryptionAtRestWithCustomerKey,
+			Type:                encryptionType,
+		}
+	} else {
+		if options.DiskEncryptionType != "" {
+			return "", fmt.Errorf("AzureDisk - DiskEncryptionType(%s) should be empty when DiskEncryptionSetID is not set", options.DiskEncryptionType)
 		}
 	}
 


Test patch:
diff --git a/pkg/provider/azure_managedDiskController_test.go b/pkg/provider/azure_managedDiskController_test.go
index 302f0f0cd..b1947a19b 100644
--- a/pkg/provider/azure_managedDiskController_test.go
+++ b/pkg/provider/azure_managedDiskController_test.go
@@ -62,6 +62,7 @@ func TestCreateManagedDisk(t *testing.T) {
 		diskIOPSReadWrite   string
 		diskMBPSReadWrite   string
 		diskEncryptionSetID string
+		diskEncryptionType  string
 		subscriptionID      string
 		resouceGroup        string
 		networkAccessPolicy compute.NetworkAccessPolicy
@@ -134,6 +135,18 @@ func TestCreateManagedDisk(t *testing.T) {
 			expectedErr:         true,
 			expectedErrMsg:      fmt.Errorf("AzureDisk - format of DiskEncryptionSetID(%s) is incorrect, correct format: %s", badDiskEncryptionSetID, consts.DiskEncryptionSetIDFormat),
 		},
+		{
+			desc:                "DiskEncryptionType should be empty when DiskEncryptionSetID is not set",
+			diskID:              disk1ID,
+			diskName:            disk1Name,
+			storageAccountType:  compute.DiskStorageAccountTypesStandardLRS,
+			diskEncryptionSetID: "",
+			diskEncryptionType:  "EncryptionAtRestWithCustomerKey",
+			expectedDiskID:      "",
+			existedDisk:         compute.Disk{ID: to.StringPtr(disk1ID), Name: to.StringPtr(disk1Name), DiskProperties: &compute.DiskProperties{Encryption: &compute.Encryption{DiskEncryptionSetID: &goodDiskEncryptionSetID, Type: compute.EncryptionTypeEncryptionAtRestWithCustomerKey}, ProvisioningState: to.StringPtr("Succeeded")}, Tags: testTags},
+			expectedErr:         true,
+			expectedErrMsg:      fmt.Errorf("AzureDisk - DiskEncryptionType(EncryptionAtRestWithCustomerKey) should be empty when DiskEncryptionSetID is not set"),
+		},
 		{
 			desc:                "disk Id and no error shall be returned if everything is good with StandardLRS storage account with not empty diskIOPSReadWrite",
 			diskID:              disk1ID,
@@ -177,6 +190,7 @@ func TestCreateManagedDisk(t *testing.T) {
 			diskName:            disk1Name,
 			storageAccountType:  compute.DiskStorageAccountTypesStandardLRS,
 			diskEncryptionSetID: goodDiskEncryptionSetID,
+			diskEncryptionType:  "EncryptionAtRestWithCustomerKey",
 			networkAccessPolicy: compute.NetworkAccessPolicyAllowAll,
 			expectedDiskID:      disk1ID,
 			existedDisk:         compute.Disk{ID: to.StringPtr(disk1ID), Name: to.StringPtr(disk1Name), DiskProperties: &compute.DiskProperties{Encryption: &compute.Encryption{DiskEncryptionSetID: &goodDiskEncryptionSetID, Type: compute.EncryptionTypeEncryptionAtRestWithCustomerKey}, ProvisioningState: to.StringPtr("Succeeded")}, Tags: testTags},
@@ -233,6 +247,7 @@ func TestCreateManagedDisk(t *testing.T) {
 			DiskIOPSReadWrite:   test.diskIOPSReadWrite,
 			DiskMBpsReadWrite:   test.diskMBPSReadWrite,
 			DiskEncryptionSetID: test.diskEncryptionSetID,
+			DiskEncryptionType:  test.diskEncryptionType,
 			MaxShares:           maxShare,
 			NetworkAccessPolicy: test.networkAccessPolicy,
 			DiskAccessID:        test.diskAccessID,