Repository: kubernetes-sigs/cloud-provider-azure (go)

Issue:
Load Balancer Node membership not being updated for externalTrafficPolicy Local
#### What happened:

When creating a cluster and creating the first service with `externalTrafficPolicy: Local`, we are observing that the new nodes are not added to the load balancer in a timely manner as they should be. 

It takes one of three things to trigger the nodes to be added to the load balancer:
* Restarting the CCM
* Adding another node (this triggers old nodes to add, but not always the new node)
* Waiting for the full resync to occur (IIRC 10 hours?)

#### What you expected to happen:

Nodes should be added to the load balancer as soon as they are added to the cluster.

#### How to reproduce it (as minimally and precisely as possible):

Create a cluster with the standard load balancer, and create a single service of eTP: Local. This doesn't always happen, but, we are observing with a good frequency that the worker nodes are not being added to the load balancer when the load balancer is created, and the nodes are freshly added.

#### Anything else we need to know?:

I believe this to be an incompatibility between the predicates used in the cloud-provider core service controller, and the logic in the Azure provider that excludes nodes from being added to the load balancer when they are unready.

In https://github.com/kubernetes/kubernetes/pull/109706, the service controller was updated so that it monitors the nodes that it's passing into `UpdateLoadBalancer`. The PR aimed to reduce the number of times you would resync the load balancer membership.

The [UpdateLoadBalancer](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/ef1a8ffe65539e76f47c3c239bca77cc856b6355/pkg/provider/azure_loadbalancer.go#L212) function will only be called when the set of nodes changes, which means either a new node is added/removed or a node is excluded via label/tainted/has a transition of the provider ID.

The call to update the load balancers eventually calls down to [EnsureHostsInPool](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/ef1a8ffe65539e76f47c3c239bca77cc856b6355/pkg/provider/azure_standard.go#L1040C1-L1040C1), which then calls out to [ShouldNodeExcludedFromLoadBalancer](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/ef1a8ffe65539e76f47c3c239bca77cc856b6355/pkg/provider/azure.go#L1364). This determines whether the node should be included or not.

When a node is not ready, [it is excluded](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/ef1a8ffe65539e76f47c3c239bca77cc856b6355/pkg/provider/azure.go#L1236-L1242) which means that, when a node is first added, there is a race. If the CCM observes the update for the load balancers before the Node becomes ready, then the node is excluded from the load balancer. In this case, with the new reduction of calls to `UpdateLoadBalancer`, when the node flicks to being ready, the node no longer gets added to the load balancers, as nothing is calling `UpdateLoadBalancer`. According to the core CCM code, nothing has changed.

In my opinion, and that of [KEP-3458](https://github.com/kubernetes/enhancements/issues/3458), we need to remove the readiness check from the Azure CCM and allow the nodes to be added independent of their readiness.

To do this though, we must first fix the broken health checks (#3499) which has a PR up already (#3887). This is also related to #3500 which adds another reason not to manage load balancer membership based on whether a node is ready or not.

#### Environment:

- Kubernetes version (use `kubectl version`): 1.27
- Cloud provider or hardware configuration: Azure 1.27
- OS (e.g: `cat /etc/os-release`):
- Kernel (e.g. `uname -a`):
- Install tools:
- Network plugin and version (if this is a network-related bug):
- Others:


Pull request description:
Issue 4230: remove readiness check for cache exclusion
<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide/first-contribution.md#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?

<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature
/kind design

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

/kind bug


#### What this PR does / why we need it:

Issue https://github.com/kubernetes-sigs/cloud-provider-azure/issues/4230 explains things in greater depth, but here's the TL;DR

https://github.com/kubernetes/kubernetes/pull/109706 stopped syncing load balancers when changes are observed to the readiness state of the Node object. Load balancers are essentially almost only re-synced whenever a Node is: added, deleted, has the exclusion label added. The service controller expectation is that the cloud-providers just accept the list of nodes provided through the `UpdateLoadBalancerHost` call and try to configure the load balancers with the entire set of nodes provided. Given that Azure performs additional filtering of the nodes and excludes `NotReady` nodes, there will be situations where the node which just transitioned to `Ready` isn't added to the load balancer node set, because the service controller doesn't re-sync load balancers due to the readiness state change. 

This patch removes this additional filtering of the readiness state so that all `NotReady` nodes get added to the load balancer set. The health check probes used by the load balancer will then determine which nodes should be used for traffic load balancing. 

Please have a look at the referenced enhancement proposal which merged in 1.26: https://github.com/kubernetes/enhancements/issues/3458

/cc @JoelSpeed      

#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes #4230

#### Special notes for your reviewer:

#### Does this PR introduce a user-facing change?
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->
```release-note
Fixes issue 4230 and removes the additional filtering on `NotReady` nodes by the azure cloud provider code 
```

#### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

<!--
This section can be blank if this pull request does not require a release note.

When adding links which point to resources within git repositories, like
KEPs or supporting documentation, please reference a specific commit and avoid
linking directly to the master branch. This ensures that links reference a
specific point in time, rather than a document that may change over time.

See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files

Please use the following format for linking documentation:
- [KEP]: <link>
- [Usage]: <link>
- [Other doc]: <link>
-->
```docs
https://github.com/kubernetes/enhancements/issues/3458
```


Patch:
diff --git a/pkg/provider/azure.go b/pkg/provider/azure.go
index 455874f1c..e608da04d 100644
--- a/pkg/provider/azure.go
+++ b/pkg/provider/azure.go
@@ -76,7 +76,6 @@ import (
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/zoneclient"
 	azcache "sigs.k8s.io/cloud-provider-azure/pkg/cache"
 	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
-	nodemanager "sigs.k8s.io/cloud-provider-azure/pkg/nodemanager"
 	"sigs.k8s.io/cloud-provider-azure/pkg/retry"
 
 	"sigs.k8s.io/yaml"
@@ -1270,15 +1269,6 @@ func (az *Cloud) updateNodeCaches(prevNode, newNode *v1.Node) {
 			az.excludeLoadBalancerNodes.Insert(newNode.ObjectMeta.Name)
 			klog.V(6).Infof("excluding Node %q from LoadBalancer because it has exclude-from-external-load-balancers label", newNode.ObjectMeta.Name)
 
-		case !isNodeReady(newNode) && nodemanager.GetCloudTaint(newNode.Spec.Taints) == nil:
-			// If not in ready state and not a newly created node, add to excludeLoadBalancerNodes cache.
-			// New nodes (tainted with "node.cloudprovider.kubernetes.io/uninitialized") should not be
-			// excluded from load balancers regardless of their state, so as to reduce the number of
-			// VMSS API calls and not provoke VMScaleSetActiveModelsCountLimitReached.
-			// (https://github.com/kubernetes-sigs/cloud-provider-azure/issues/851)
-			az.excludeLoadBalancerNodes.Insert(newNode.ObjectMeta.Name)
-			klog.V(6).Infof("excluding Node %q from LoadBalancer because it is not in ready state or a newly created one", newNode.ObjectMeta.Name)
-
 		default:
 			// Nodes not falling into the three cases above are valid backends and
 			// should not appear in excludeLoadBalancerNodes cache.
@@ -1423,15 +1413,6 @@ func (az *Cloud) ShouldNodeExcludedFromLoadBalancer(nodeName string) (bool, erro
 	return az.excludeLoadBalancerNodes.Has(nodeName), nil
 }
 
-func isNodeReady(node *v1.Node) bool {
-	for _, cond := range node.Status.Conditions {
-		if cond.Type == v1.NodeReady && cond.Status == v1.ConditionTrue {
-			return true
-		}
-	}
-	return false
-}
-
 func (az *Cloud) getActiveNodesByLoadBalancerName(lbName string) sets.Set[string] {
 	az.multipleStandardLoadBalancersActiveNodesLock.Lock()
 	defer az.multipleStandardLoadBalancersActiveNodesLock.Unlock()


Test patch:
diff --git a/pkg/provider/azure_test.go b/pkg/provider/azure_test.go
index f23414e4e..1c1a2d496 100644
--- a/pkg/provider/azure_test.go
+++ b/pkg/provider/azure_test.go
@@ -3554,7 +3554,7 @@ func TestUpdateNodeCacheExcludeLoadBalancer(t *testing.T) {
 	az.nodeZones = map[string]sets.Set[string]{zone: nodesInZone}
 	az.nodeResourceGroups = map[string]string{"aNode": "rg"}
 
-	// a non-ready node should be excluded
+	// a non-ready node should be included
 	az.unmanagedNodes = sets.New[string]()
 	az.excludeLoadBalancerNodes = sets.New[string]()
 	az.nodeNames = sets.New[string]()
@@ -3575,9 +3575,9 @@ func TestUpdateNodeCacheExcludeLoadBalancer(t *testing.T) {
 		},
 	}
 	az.updateNodeCaches(nil, &nonReadyNode)
-	assert.Equal(t, 1, len(az.excludeLoadBalancerNodes))
+	assert.Equal(t, 0, len(az.excludeLoadBalancerNodes))
 
-	// node becomes ready, it should be removed from az.excludeLoadBalancerNodes
+	// node becomes ready => no impact on az.excludeLoadBalancerNodes
 	readyNode := v1.Node{
 		ObjectMeta: metav1.ObjectMeta{
 			Labels: map[string]string{