Repository: kubernetes-sigs/cloud-provider-azure
Pull request #2816 (2022-11-23 13:22:59)

fix: reduce VMSS cache refresh in CSI driver
<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide/first-contribution.md#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?
/kind bug

#### What this PR does / why we need it:
fix: reduce VMSS cache refresh in CSI driver
CSI driver does not setInformer for node add/delete (informer setting is also not valid since when csi driver starts up, nodes are already there), so `ss.GetNodeNames()` is actually invalid in 
https://github.com/kubernetes-sigs/cloud-provider-azure/blob/c905fb10cb277916fb580fef75d276f328bc0d7c/pkg/provider/azure_vmss_cache.go#L420-L422

This PR would get nodeName from vmss uniform, avset, vmss flex cache directly, that could avoid unnecessary vmss refresh.

#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes #

#### Special notes for your reviewer:

#### Does this PR introduce a user-facing change?
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->
```release-note
fix: reduce VMSS cache refresh in CSI driver
```

#### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

<!--
This section can be blank if this pull request does not require a release note.

When adding links which point to resources within git repositories, like
KEPs or supporting documentation, please reference a specific commit and avoid
linking directly to the master branch. This ensures that links reference a
specific point in time, rather than a document that may change over time.

See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files

Please use the following format for linking documentation:
- [KEP]: <link>
- [Usage]: <link>
- [Other doc]: <link>
-->
```docs
fix: reduce VMSS cache refresh in CSI driver
```


netlify[bot]: ### <span aria-hidden="true">👷</span> Deploy Preview for *kubernetes-sigs-cloud-provide-azure* processing.


|  Name | Link |
|---------------------------------|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | 9e05efb1d31c6a6eca8b3882312c3097f7e177ce |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/kubernetes-sigs-cloud-provide-azure/deploys/637e1eb50f5d740008cd8bdd |
k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/2816#" title="Author self-approved">andyzhangx</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/master/OWNERS)~~ [andyzhangx]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
coveralls: 
[![Coverage Status](https://coveralls.io/builds/54501508/badge)](https://coveralls.io/builds/54501508)

Coverage increased (+0.0004%) to 79.629% when pulling **a7ab7c9b1598d3ddb2a1d57f2d1f8308d5923233 on andyzhangx:node-in-cache** into **c905fb10cb277916fb580fef75d276f328bc0d7c on kubernetes-sigs:master**.

k8s-ci-robot: @andyzhangx: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-check | a7ab7c9b1598d3ddb2a1d57f2d1f8308d5923233 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2816/pull-cloud-provider-azure-check/1595408381760245760) | true | `/test pull-cloud-provider-azure-check`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=2816). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Aandyzhangx). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->

Patch:
diff --git a/pkg/provider/azure_utils.go b/pkg/provider/azure_utils.go
index 945c121e2..910abb56f 100644
--- a/pkg/provider/azure_utils.go
+++ b/pkg/provider/azure_utils.go
@@ -31,6 +31,7 @@ import (
 	"k8s.io/klog/v2"
 	utilnet "k8s.io/utils/net"
 
+	azcache "sigs.k8s.io/cloud-provider-azure/pkg/cache"
 	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
 )
 
@@ -296,3 +297,37 @@ func removeDuplicatedSecurityRules(rules []network.SecurityRule) []network.Secur
 	}
 	return rules
 }
+
+// isNodeInVMSSVMCache check whether nodeName is in vmssVMCache
+func isNodeInVMSSVMCache(nodeName string, vmssVMCache *sync.Map) bool {
+	if vmssVMCache == nil {
+		return false
+	}
+	var isInCache bool
+	vmssVMCache.Range(func(_, value interface{}) bool {
+		if value != nil && value.(*azcache.TimedCache).Store != nil {
+			for _, v := range value.(*azcache.TimedCache).Store.List() {
+				if v != nil {
+					data := v.(*azcache.AzureCacheEntry).Data
+					if data != nil {
+						data.(*sync.Map).Range(func(vmName, _ interface{}) bool {
+							if vmName != nil && vmName.(string) == nodeName {
+								isInCache = true
+								return false
+							}
+							return true
+						})
+					}
+				}
+				if isInCache {
+					return false
+				}
+			}
+		}
+		if isInCache {
+			return false
+		}
+		return true
+	})
+	return isInCache
+}
diff --git a/pkg/provider/azure_vmss_cache.go b/pkg/provider/azure_vmss_cache.go
index 65ddf11f3..62ec948e9 100644
--- a/pkg/provider/azure_vmss_cache.go
+++ b/pkg/provider/azure_vmss_cache.go
@@ -348,7 +348,6 @@ func (ss *ScaleSet) updateCache(nodeName, resourceGroupName, vmssName, instanceI
 
 func (ss *ScaleSet) newNonVmssUniformNodesCache() (*azcache.TimedCache, error) {
 	getter := func(key string) (interface{}, error) {
-		klog.V(6).Infof("refresh the cache of NonVmssUniformNodesCache")
 		vmssFlexVMNodeNames := sets.NewString()
 		vmssFlexVMProviderIDs := sets.NewString()
 		avSetVMNodeNames := sets.NewString()
@@ -357,6 +356,7 @@ func (ss *ScaleSet) newNonVmssUniformNodesCache() (*azcache.TimedCache, error) {
 		if err != nil {
 			return nil, err
 		}
+		klog.V(2).Infof("refresh the cache of NonVmssUniformNodesCache in rg %v", resourceGroups)
 
 		for _, resourceGroup := range resourceGroups.List() {
 			vms, err := ss.Cloud.ListVirtualMachines(resourceGroup)
@@ -415,9 +415,20 @@ func (ss *ScaleSet) getVMManagementTypeByNodeName(nodeName string, crt azcache.A
 	}
 
 	cachedNodes := cached.(NonVmssUniformNodesEntry).ClusterNodeNames
-
 	// if the node is not in the cache, assume the node has joined after the last cache refresh and attempt to refresh the cache.
 	if !cachedNodes.Has(nodeName) {
+		if cached.(NonVmssUniformNodesEntry).AvSetVMNodeNames.Has(nodeName) {
+			return ManagedByAvSet, nil
+		}
+
+		if cached.(NonVmssUniformNodesEntry).VMSSFlexVMNodeNames.Has(nodeName) {
+			return ManagedByVmssFlex, nil
+		}
+
+		if isNodeInVMSSVMCache(nodeName, ss.vmssVMCache) {
+			return ManagedByVmssUniform, nil
+		}
+
 		klog.V(2).Infof("Node %s has joined the cluster since the last VM cache refresh in NonVmssUniformNodesEntry, refreshing the cache", nodeName)
 		cached, err = ss.nonVmssUniformNodesCache.Get(consts.NonVmssUniformNodesKey, azcache.CacheReadTypeForceRefresh)
 		if err != nil {
@@ -434,6 +445,7 @@ func (ss *ScaleSet) getVMManagementTypeByNodeName(nodeName string, crt azcache.A
 	if cachedVmssFlexVMs.Has(nodeName) {
 		return ManagedByVmssFlex, nil
 	}
+
 	return ManagedByVmssUniform, nil
 }
 


Test patch:
diff --git a/pkg/provider/azure_utils_test.go b/pkg/provider/azure_utils_test.go
index 48dc6e923..ff1a2c530 100644
--- a/pkg/provider/azure_utils_test.go
+++ b/pkg/provider/azure_utils_test.go
@@ -18,6 +18,7 @@ package provider
 
 import (
 	"fmt"
+	"sync"
 	"testing"
 	"time"
 
@@ -28,6 +29,7 @@ import (
 	v1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 
+	azcache "sigs.k8s.io/cloud-provider-azure/pkg/cache"
 	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
 )
 
@@ -314,3 +316,68 @@ func TestRemoveDuplicatedSecurityRules(t *testing.T) {
 		})
 	}
 }
+
+func TestIsNodeInVMSSVMCache(t *testing.T) {
+	emptyTimedCacheMap := &sync.Map{}
+	emptyTimedCacheMap.Store("key", &azcache.TimedCache{})
+
+	getter := func(key string) (interface{}, error) {
+		return nil, nil
+	}
+	emptyCacheEntryTimedCache, _ := azcache.NewTimedcache(fakeCacheTTL, getter)
+	emptyCacheEntryTimedCache.Set("key", nil)
+	emptyCacheEntryTimedCacheMap := &sync.Map{}
+	emptyCacheEntryTimedCacheMap.Store("key", emptyCacheEntryTimedCache)
+
+	cacheEntryTimedCache, _ := azcache.NewTimedcache(fakeCacheTTL, getter)
+	syncMap := &sync.Map{}
+	syncMap.Store("node", nil)
+	cacheEntryTimedCache.Set("key", syncMap)
+	cacheEntryTimedCacheMap := &sync.Map{}
+	cacheEntryTimedCacheMap.Store("key", cacheEntryTimedCache)
+
+	tests := []struct {
+		description    string
+		nodeName       string
+		vmssVMCache    *sync.Map
+		expectedResult bool
+	}{
+		{
+			description:    "nil cache",
+			vmssVMCache:    nil,
+			expectedResult: false,
+		},
+		{
+			description:    "empty cache",
+			vmssVMCache:    &sync.Map{},
+			expectedResult: false,
+		},
+		{
+			description:    "empty timed cache",
+			vmssVMCache:    emptyTimedCacheMap,
+			expectedResult: false,
+		},
+		{
+			description:    "empty CacheEntry timed cache",
+			vmssVMCache:    emptyCacheEntryTimedCacheMap,
+			expectedResult: false,
+		},
+		{
+			description:    "node name in the cache",
+			nodeName:       "node",
+			vmssVMCache:    cacheEntryTimedCacheMap,
+			expectedResult: true,
+		},
+		{
+			description:    "node name not in the cache",
+			nodeName:       "node2",
+			vmssVMCache:    cacheEntryTimedCacheMap,
+			expectedResult: false,
+		},
+	}
+
+	for _, test := range tests {
+		result := isNodeInVMSSVMCache(test.nodeName, test.vmssVMCache)
+		assert.Equal(t, test.expectedResult, result, test.description)
+	}
+}