Repository: kubernetes-sigs/cloud-provider-azure
Pull request #2275 (2022-09-07 00:43:30)

Invoke VMSS parallel update API when vmss tag exists
<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide/first-contribution.md#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?

<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature
/kind design

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->
/kind feature

#### What this PR does / why we need it:
It reads tag `aks-managed-coordination` from VMSS, and then only invoke parallel network profile update APIs when the tag exists (tag value could be omitted)
#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes #

#### Special notes for your reviewer:

#### Does this PR introduce a user-facing change?
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->
```release-note
NONE
```

#### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

<!--
This section can be blank if this pull request does not require a release note.

When adding links which point to resources within git repositories, like
KEPs or supporting documentation, please reference a specific commit and avoid
linking directly to the master branch. This ensures that links reference a
specific point in time, rather than a document that may change over time.

See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files

Please use the following format for linking documentation:
- [KEP]: <link>
- [Usage]: <link>
- [Other doc]: <link>
-->
```docs

```


netlify[bot]: ### <span aria-hidden="true">👷</span> Deploy Preview for *kubernetes-sigs-cloud-provide-azure* processing.


|  Name | Link |
|---------------------------------|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | a6520b96dc7e551f0625878be2765bced11c6766 |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/kubernetes-sigs-cloud-provide-azure/deploys/6317e93350e0860008e3c27a |
k8s-ci-robot: Hi @lodrem. Thanks for your PR.

I'm waiting for a [kubernetes-sigs](https://github.com/orgs/kubernetes-sigs/people) member to verify that this patch is reasonable to test. If it is, they should reply with `/ok-to-test` on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should [join the org](https://git.k8s.io/community/community-membership.md#member) to skip this step.

Once the patch is verified, the new status will be reflected by the `ok-to-test` label.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>

k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/2275#" title="Author self-approved">lodrem</a>*
**Once this PR has been reviewed and has the lgtm label**, please assign lzhecheng for approval by writing `/assign @lzhecheng` in a comment. For more information see:[The Kubernetes Code Review Process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process).

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["lzhecheng"]} -->
coveralls: 
[![Coverage Status](https://coveralls.io/builds/52260476/badge)](https://coveralls.io/builds/52260476)

Coverage increased (+0.006%) to 80.331% when pulling **a6520b96dc7e551f0625878be2765bced11c6766 on lodrem:feat-vmss/batch-operation** into **39bcb5e03023fbdea0abb64e0aba7a933373bc44 on kubernetes-sigs:master**.

nilo19: /ok-to-test
nilo19: Please don't forget to squash the commits.
k8s-ci-robot: @lodrem: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-capz | 6163eaf444d5290cdf354e2c574f8b638688b1fe | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2275/pull-cloud-provider-azure-e2e-ccm-capz/1567340845261131776) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=2275). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->

Patch:
diff --git a/pkg/consts/consts.go b/pkg/consts/consts.go
index 96de9febc..adf088217 100644
--- a/pkg/consts/consts.go
+++ b/pkg/consts/consts.go
@@ -469,3 +469,7 @@ const (
 	// Default number of IP configs for PLS
 	PLSDefaultNumOfIPConfig = 1
 )
+
+const (
+	VMSSTagForBatchOperation = "aks-managed-coordination"
+)
diff --git a/pkg/provider/azure_vmss.go b/pkg/provider/azure_vmss.go
index 9689f3183..72a8fdd44 100644
--- a/pkg/provider/azure_vmss.go
+++ b/pkg/provider/azure_vmss.go
@@ -1359,10 +1359,24 @@ func (ss *ScaleSet) EnsureHostsInPool(service *v1.Service, nodes []*v1.Node, bac
 		hostUpdates = append(hostUpdates, func() error {
 			ctx, cancel := getContextWithCancel()
 			defer cancel()
-			klog.V(2).Infof("EnsureHostInPool begins to UpdateVMs for VMSS(%s, %s) with new backendPoolID %s", meta.resourceGroup, meta.vmssName, backendPoolID)
-			rerr := ss.VirtualMachineScaleSetVMsClient.UpdateVMs(ctx, meta.resourceGroup, meta.vmssName, update, "network_update", ss.getPutVMSSVMBatchSize())
+
+			logFields := []interface{}{
+				"operation", "EnsureHostsInPool UpdateVMSSVMs",
+				"vmssName", meta.vmssName,
+				"resourceGroup", meta.resourceGroup,
+				"backendPoolID", backendPoolID,
+			}
+
+			batchSize, err := ss.VMSSBatchSize(meta.vmssName)
+			if err != nil {
+				klog.ErrorS(err, "Failed to get vmss batch size", logFields...)
+				return err
+			}
+
+			klog.V(2).InfoS("Begin to update VMs for VMSS with new backendPoolID", logFields...)
+			rerr := ss.VirtualMachineScaleSetVMsClient.UpdateVMs(ctx, meta.resourceGroup, meta.vmssName, update, "network_update", batchSize)
 			if rerr != nil {
-				klog.Errorf("EnsureHostInPool UpdateVMs for VMSS(%s, %s) failed with error %v", meta.resourceGroup, meta.vmssName, rerr.Error())
+				klog.ErrorS(err, "Failed to update VMs for VMSS", logFields...)
 				return rerr.Error()
 			}
 
@@ -1632,10 +1646,24 @@ func (ss *ScaleSet) EnsureBackendPoolDeleted(service *v1.Service, backendPoolID,
 		hostUpdates = append(hostUpdates, func() error {
 			ctx, cancel := getContextWithCancel()
 			defer cancel()
-			klog.V(2).Infof("EnsureBackendPoolDeleted begins to UpdateVMs for VMSS(%s, %s) with backendPoolID %s", meta.resourceGroup, meta.vmssName, backendPoolID)
-			rerr := ss.VirtualMachineScaleSetVMsClient.UpdateVMs(ctx, meta.resourceGroup, meta.vmssName, update, "network_update", ss.getPutVMSSVMBatchSize())
+
+			logFields := []interface{}{
+				"operation", "EnsureBackendPoolDeleted UpdateVMSSVMs",
+				"vmssName", meta.vmssName,
+				"resourceGroup", meta.resourceGroup,
+				"backendPoolID", backendPoolID,
+			}
+
+			batchSize, err := ss.VMSSBatchSize(meta.vmssName)
+			if err != nil {
+				klog.ErrorS(err, "Failed to get vmss batch size", logFields...)
+				return err
+			}
+
+			klog.V(2).InfoS("Begin to update VMs for VMSS with new backendPoolID", logFields...)
+			rerr := ss.VirtualMachineScaleSetVMsClient.UpdateVMs(ctx, meta.resourceGroup, meta.vmssName, update, "network_update", batchSize)
 			if rerr != nil {
-				klog.Errorf("EnsureBackendPoolDeleted UpdateVMs for VMSS(%s, %s) failed with error %v", meta.resourceGroup, meta.vmssName, rerr.Error())
+				klog.ErrorS(err, "Failed to update VMs for VMSS", logFields...)
 				return rerr.Error()
 			}
 
@@ -1825,3 +1853,17 @@ func (ss *ScaleSet) GetNodeVMSetName(node *v1.Node) (string, error) {
 	klog.V(4).Infof("ss.GetNodeVMSetName: found vmss name %s from node name %s", vmssName, node.Name)
 	return vmssName, nil
 }
+
+// VMSSBatchSize returns the batch size for VMSS operations.
+func (ss *ScaleSet) VMSSBatchSize(vmssName string) (int, error) {
+	batchSize := 0
+	vmss, err := ss.getVMSS(vmssName, azcache.CacheReadTypeDefault)
+	if err != nil {
+		return 0, fmt.Errorf("get vmss batch size: %w", err)
+	}
+	if _, ok := vmss.Tags[consts.VMSSTagForBatchOperation]; ok {
+		batchSize = ss.getPutVMSSVMBatchSize()
+	}
+	klog.V(2).InfoS("Fetch VMSS batch size", "vmss", vmssName, "size", batchSize)
+	return batchSize, nil
+}


Test patch:
diff --git a/pkg/provider/azure_vmss_test.go b/pkg/provider/azure_vmss_test.go
index d42081f4e..804c233ab 100644
--- a/pkg/provider/azure_vmss_test.go
+++ b/pkg/provider/azure_vmss_test.go
@@ -2797,3 +2797,69 @@ func TestGetNodeVMSetNameVMSS(t *testing.T) {
 	assert.NoError(t, err)
 	assert.Equal(t, "vmss", vmSetName)
 }
+
+func TestScaleSet_VMSSBatchSize(t *testing.T) {
+
+	const (
+		BatchSize = 500
+	)
+
+	t.Run("failed to get vmss", func(t *testing.T) {
+		ctrl := gomock.NewController(t)
+		defer ctrl.Finish()
+		ss, err := NewTestScaleSet(ctrl)
+		assert.NoError(t, err)
+
+		var (
+			vmssName   = "foo"
+			getVMSSErr = &retry.Error{RawError: fmt.Errorf("list vmss error")}
+		)
+		mockVMSSClient := ss.Cloud.VirtualMachineScaleSetsClient.(*mockvmssclient.MockInterface)
+		mockVMSSClient.EXPECT().List(gomock.Any(), gomock.Any()).
+			Return(nil, getVMSSErr)
+
+		_, err = ss.VMSSBatchSize(vmssName)
+		assert.Error(t, err)
+	})
+
+	t.Run("vmss contains batch operation tag", func(t *testing.T) {
+		ctrl := gomock.NewController(t)
+		defer ctrl.Finish()
+		ss, err := NewTestScaleSet(ctrl)
+		assert.NoError(t, err)
+		ss.Cloud.PutVMSSVMBatchSize = BatchSize
+
+		scaleSet := compute.VirtualMachineScaleSet{
+			Name: to.StringPtr("foo"),
+			Tags: map[string]*string{
+				consts.VMSSTagForBatchOperation: to.StringPtr(""),
+			},
+		}
+		mockVMSSClient := ss.Cloud.VirtualMachineScaleSetsClient.(*mockvmssclient.MockInterface)
+		mockVMSSClient.EXPECT().List(gomock.Any(), gomock.Any()).
+			Return([]compute.VirtualMachineScaleSet{scaleSet}, nil)
+
+		batchSize, err := ss.VMSSBatchSize(to.String(scaleSet.Name))
+		assert.NoError(t, err)
+		assert.Equal(t, BatchSize, batchSize)
+	})
+
+	t.Run("vmss doesn't contain batch operation tag", func(t *testing.T) {
+		ctrl := gomock.NewController(t)
+		defer ctrl.Finish()
+		ss, err := NewTestScaleSet(ctrl)
+		assert.NoError(t, err)
+		ss.Cloud.PutVMSSVMBatchSize = BatchSize
+
+		scaleSet := compute.VirtualMachineScaleSet{
+			Name: to.StringPtr("bar"),
+		}
+		mockVMSSClient := ss.Cloud.VirtualMachineScaleSetsClient.(*mockvmssclient.MockInterface)
+		mockVMSSClient.EXPECT().List(gomock.Any(), gomock.Any()).
+			Return([]compute.VirtualMachineScaleSet{scaleSet}, nil)
+
+		batchSize, err := ss.VMSSBatchSize(to.String(scaleSet.Name))
+		assert.NoError(t, err)
+		assert.Equal(t, 0, batchSize)
+	})
+}