Change:
Doing a micro upgrade from one level of 1.23 to another level of 1.23, we noticed disruption to PDB based app went from about 5s to about 13s when compared to a 1.22.z to 1.22.z+1 update. I did not expect any disruption. This one is a bit difficult, we have thousands of runs to compare averages and distributions against. It's specifically azure, it's specifically the service load balancer, it's worst on 1.23 to 1.23 upgrades versus 1.22 to 1.23 upgrades. We tracked it down to one of these three commits: When we revert all three, we see improved behavior (our test was in ). We attempted to workaround by setting the exclude-from-lb label, but this filter prevents the change from being observed by the informer's cache: When we attempted to remove the filter entirely (attempt here ), we get from <details</detailsAzure <details</details<details</details<details</details<details</details
quoting from on an internal thread klog.V(2).Infof("reconcileLoadBalancer for service (%s)(%t): lb backendpool - found unwanted node %s, decouple it from the LB", serviceName, wantLb, nodeName)
/sig cloud-provider
ping
/cc
Adding more details to the issue... We analyzed how Azure Load Balancer backends are modified as cluster nodes become or stop being available, which in our tests can happen in two ways: by labeling nodes withby rebooting nodes With kubernetes prior to this PR (, solving ), namely kubernetes 1.22, shipped with openshift 4.9, we notice that: cloud service controller in receives event notifications on nodes and updates accordingly the list of available nodes for load balancers (): a node is excluded from a load balancer backend pool if it has the "exclude" label above, or if its status is not ; in receives the updated list of nodes from the cloud service controller this list differs from the current list of nodes in the backend pool for a given load balancer, it removes spurious nodes and adds any new nodes. particular, node deletion begins here: and takes place in the function. So far so good, if I label a node that I want to exclude from a load balancer, the following logs appear: Now, if we try to do the same with kubernetes 1.23 (planned to ship with openshift 4.10), we see that the above deletion NEVER takes place: steps 1 and 2 from above are the same: the cloud service controller realizes that a node has been labeled (or rebooted) and removes from the list of nodes it passes to the Azure The PR mentioned above, however, changes the logic of how nodes are removed and the result is that deletion never happens, at least for the two cases mentioned above (a node that gets the "exclude" label, a node that is rebooting). From what I understand in the current code: deletion now happens if this call returns true: this function does NOT make use of the node list received as input by the reconciler. It looks for the node name in , which is updated by , which is in turn triggered by separate node informers in for some reason, these node informers trigger a cache update only if the topology label () on them changes, so the list in is only correct at startup, and then becomes stale. As a consequence, in our tests, as nodes reboot or get labeled, the code always checks for inclusion against an empty list... The update logic with these node informers was not part of the PR mentioned above. Is this something that was done on purpose? In order to overcome this, I tried removing the condition on the topology label and triggering a cache update on every update event. This goes a little further, but deletion is skipped because of : Indeed, if I add debug lines I see that: - I am not familiar with these terms and what they entail. What's interesting, though, is that this code hasn't changed all that much since openshift 4.9, so I debug lines to see what the values for and were back then and it turned out that they were both empty strings! The check for equality was then successful and deletion took place. Now, why were they both empty? is an input parameter in and is set here, in the reconciler function: In one of my tests, held the same value as (), so in the code below we call : It turns out, returns an empty value in openshift 4.9: This code wasn't modified in the PR or since 1.22, so I do not see how now with 1.23 we finally get a proper value for ... So to recap, The node cache updates take place only when the topology label changes. Was this done on purpose? I don't see it mentioned in the issue linked to the original PReven if we trigger updates to the node cache systematically (disregarding the topology label check), we then hit another problem that prevents the delete operation to take place. This involves some subtleties in the Azure configuration (vmasName vs vmSet, and also Config.PrimaryAvailabilitySetName) and we definitely need someone from the upstream community to take a look at any code changes we might introduce in there...
Here I wrote down the steps needed to reproduce this issue, with the necessary files available in the same repo:
(I just updated my long comment above with more precise info on openshift 4.9)
Thanks, & We'll try to get the right folks looking at this.
Thanks , appreciate all your help and efforts on this one!
I am suspecting there is a conflict between: the original PR in the kubernetes-sigs repo ()the code that eventually got merged in kubernetes 1.23 () The update of the internal cache ( occurring only when the topology label changes might not have been in the intention of the original PR author... In the original pull request, the part of runs every time it gets executed, as I suggested in my comment above: In kubernetes 1.23, this got merged with the (pre-existing) checks on the topology labels that are not included in the kubernetes-sigs repo:
thanks for catching this, I will file a PR to fix it ASAP. By the way, is it possible to switch to out-of-tree cloud provider azure since it has been GA?
/assign
/triage accepted
Let me see if I got this right. In you say the following: Before was merged, a node was removed from the backend pool of a load balancer if: status was no longer Ready or was labeled with This decision happened in the cloud provider controller: ... and the result of that was passed over to the azure-specific code in that you quote in the PR: Now, with this PR merged, a node gets removed from the backend pool of a load balancer only if: is labeled with or is labeled with ( ) This means that nodes that are no longer Ready still appear in the cloud loadbalancer backend pools, whose contents no longer reflect available nodes. Is this really the desired behaviour? Were there good reasons behind this decision? Could we add a couple of lines in in that also lookup the status of a node? What are your thoughts on this? Thanks a lot!
The fix would be rather simple, something along the lines of What do you guys think? Thanks!

Discussion:
UpdateFunc: func(prev, obj interface{}) { prevNode := prev.(*v1.Node) newNode := obj.(*v1.Node) <del> if newNode.Labels[v1.LabelTopologyZone] == prevNode.Labels[v1.LabelTopologyZone] { return } </del> az.updateNodeCaches(prevNode, newNode) }, DeleteFunc: func(obj interface{}) {

assert.Equal(t, 1, len(az.nodeNames)) } <ins> func TestUpdateNodeCacheExcludeLoadBalancer(t *testing.T) { ctrl := gomock.NewController(t) defer ctrl.Finish() az := GetTestCloud(ctrl) zone := fmt.Sprintf("%s-0", az.Location) nodesInZone := sets.NewString("prevNode") az.nodeZones = map[string]sets.String{zone: nodesInZone} az.nodeResourceGroups = map[string]string{"prevNode": "rg"} az.unmanagedNodes = sets.NewString("prevNode") az.excludeLoadBalancerNodes = sets.NewString() az.nodeNames = sets.NewString("prevNode") // excluded node cache shall be updated even if the topology label does not change prevNode := v1.Node{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{ v1.LabelTopologyZone: zone, }, Name: "prevNode", }, } newNode := v1.Node{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{ v1.LabelTopologyZone:         zone, v1.LabelNodeExcludeBalancers: "true", }, Name: "newNode", }, } az.updateNodeCaches(&prevNode, &newNode) assert.Equal(t, 1, len(az.excludeLoadBalancerNodes)) } </ins> func TestGetActiveZones(t *testing.T) { ctrl := gomock.NewController(t) defer ctrl.Finish()