Change:
What happened: introduced new dependency on Availability Sets which is too restrictive for some existing installations: What you expected to happen: Azure cloud provider to successfully manage LBs, like it had been doing before. Instead, it died [with][1]: How to reproduce it (as minimally and precisely as possible): Fails every time in OpenShift CI. Can you point us to documentation around the infrastructure options that the Azure cloud provider expects for LB management? https://kubernetes- doesn't say anything about NIC names, and it doesn't have specifics around what things should look like without Availability Sets, or whether "without Availability Sets" is not supported (when previous iterations of the cloud provider had no problem with our lack of Availability Sets). Documenting cloud-provider assumptions about host hardware would have also helped avoid ; CC /sig cloud-provider /area provider/azure [1]:
/triage accepted
/assign
Notes on OpenShift's Azure infrastructure in ARM templates [here][1], in case that helps clarify the infra we use today. Those are OpenShift 4.6 docs, and 4.6 is based on Kubernetes 1.19, and things worked fine. The trouble we've been having when we took the same Azure-infra approach and tried to use it with Kubernetes 1.20, after openshift/kubernetes. [1]:
could you share an example of resourceID (please replace your subscription/resourceName with fake ones)?
Example CI run [here][1] from the NIC-renaming openshift/installer. Installer logs [here][2] include: I'm hoping that the subscription name is not sensitive, because we've been dumping them in public logs for a long time now . Should be IDs for any other resources you're interested in in those logs as well. [1]: [2]: https://gcsweb-
Looks like this is about the master node. Are you using standard LB or a basic one? For standard LB, the master node would be excluded from the LB by default.
[Standard][1] (and [here][2]). Is this documented somewhere? We've been using standard LBs [for years][3], with no problems until the bump to the Kube 1.20 Azure cloud-provider. [1]: [2]: [3]:
Are you referring to the built-in logic of Kube to never target master nodes with a service type LoadBalancer?
That used to be a problem (), but my understanding is that it has since been fixed.
Can we re-open this? kubernetes-sigs/cloud-provider-azure was [not sufficient][1] to get the cloud-provider happy on our infra, which again, contains no Availability Sets. There are some more notes about our infrastructure [here][2] (with discussion in [rhbz][3]), in case that helps. [1]: [2]: [3]:
who has a better grasp on this than me, also just opened . Sounds like fixing that broader issue might obsolete this ticket, because "the cloud-provider expects Availability Sets which OpenShift infra doesn't have in order to figure out which nodes to remove" doesn't matter if the cloud-provider isn't removing any OpenShift nodes in the first place.
can you please confirm the standalone VM scenario is supported? It's a critical use case for CAPZ (our control plane VMs in zone-enabled regions are not placed in availability sets)

Discussion:
return nil, err } primaryVMSetName := az.VMSet.GetPrimaryVMSetName() <del> if !strings.EqualFold(primaryVMSetName, vmSetName) { </del> <ins> if !strings.EqualFold(primaryVMSetName, vmSetName) && vmSetName != "" { </ins> klog.V(2).Infof("cleanBackendpoolForPrimarySLB: found unwanted vmSet %s, decouple it from the LB", vmSetName) // construct a backendPool that only contains the IP config of the node to be deleted interfaceIPConfigToBeDeleted := network.InterfaceIPConfiguration{

if err != nil { return nil, err } <ins> if nodeName == "" { // VM may under deletion continue } </ins> // If a node is not supposed to be included in the LB, it // would not be in the `nodes` slice. We need to check the nodes that // have been added to the LB's backendpool, find the unwanted ones and

}) existingVMForAS1 := buildDefaultTestVirtualMachine("/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Compute/availabilitySets/agentpool1-availabilitySet-00000000", []string{"/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool1-00000000-nic-1"}) existingVMForAS2 := buildDefaultTestVirtualMachine("/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Compute/availabilitySets/agentpool2-availabilitySet-00000000", []string{"/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool2-00000000-nic-1"}) <del> existingNIC := buildDefaultTestInterface(true, []string{"/subscriptions/sub/resourceGroups/gh/providers/Microsoft.Network/loadBalancers/testCluster/backendAddressPools/testCluster"}) </del> <ins> existingNICForAS1 := buildDefaultTestInterface(true, []string{"/subscriptions/sub/resourceGroups/gh/providers/Microsoft.Network/loadBalancers/testCluster/backendAddressPools/testCluster"}) existingNICForAS1.VirtualMachine = &network.SubResource{ ID: to.StringPtr("/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Compute/virtualMachines/k8s-agentpool1-00000000-1"), } existingNICForAS2 := buildDefaultTestInterface(true, []string{"/subscriptions/sub/resourceGroups/gh/providers/Microsoft.Network/loadBalancers/testCluster/backendAddressPools/testCluster"}) existingNICForAS2.VirtualMachine = &network.SubResource{ ID: to.StringPtr("/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Compute/virtualMachines/k8s-agentpool2-00000000-1"), } </ins> mockVMClient := mockvmclient.NewMockInterface(ctrl) mockVMClient.EXPECT().Get(gomock.Any(), cloud.ResourceGroup, "k8s-agentpool1-00000000-1", gomock.Any()).Return(existingVMForAS1, nil) mockVMClient.EXPECT().Get(gomock.Any(), cloud.ResourceGroup, "k8s-agentpool2-00000000-1", gomock.Any()).Return(existingVMForAS2, nil) cloud.VirtualMachinesClient = mockVMClient mockNICClient := mockinterfaceclient.NewMockInterface(ctrl) <del> mockNICClient.EXPECT().Get(gomock.Any(), "rg", "k8s-agentpool2-00000000-nic-1", gomock.Any()).Return(existingNIC, nil) </del> <ins> mockNICClient.EXPECT().Get(gomock.Any(), "rg", "k8s-agentpool1-00000000-nic-1", gomock.Any()).Return(existingNICForAS1, nil) mockNICClient.EXPECT().Get(gomock.Any(), "rg", "k8s-agentpool2-00000000-nic-1", gomock.Any()).Return(existingNICForAS2, nil).Times(3) </ins> mockNICClient.EXPECT().CreateOrUpdate(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(nil) cloud.InterfacesClient = mockNICClient cleanedLB, err := cloud.cleanBackendpoolForPrimarySLB(&lb, &service, clusterName)

providerIDRE       = regexp.MustCompile(`.*/subscriptions/(?:.*)/Microsoft.Compute/virtualMachines/(.+)$`) backendPoolIDRE    = regexp.MustCompile(`^/subscriptions/(?:.*)/resourceGroups/(?:.*)/providers/Microsoft.Network/loadBalancers/(.+)/backendAddressPools/(?:.*)`) nicResourceGroupRE = regexp.MustCompile(`.*/subscriptions/(?:.*)/resourceGroups/(.+)/providers/Microsoft.Network/networkInterfaces/(?:.*)`) <del> nicIDRE            = regexp.MustCompile(`/subscriptions/(?:.*)/resourceGroups/(?:.+)/providers/Microsoft.Network/networkInterfaces/(.+)-nic-(.+)/ipConfigurations/(?:.*)`) </del> <ins> nicIDRE            = regexp.MustCompile(`(?i)/subscriptions/(?:.*)/resourceGroups/(.+)/providers/Microsoft.Network/networkInterfaces/(.+)/ipConfigurations/(?:.*)`) vmIDRE             = regexp.MustCompile(`(?i)/subscriptions/(?:.*)/resourceGroups/(?:.*)/providers/Microsoft.Compute/virtualMachines/(.+)`) </ins> vmasIDRE           = regexp.MustCompile(`/subscriptions/(?:.*)/resourceGroups/(?:.*)/providers/Microsoft.Compute/availabilitySets/(.+)`) )

return "", "", fmt.Errorf("invalid ip config ID %s", ipConfigurationID) } <del> prefix := matches[1] suffix := matches[2] nodeName := fmt.Sprintf("%s-%s", prefix, suffix) </del> <ins> nicResourceGroup, nicName := matches[1], matches[2] if nicResourceGroup == "" || nicName == "" { return "", "", fmt.Errorf("invalid ip config ID %s", ipConfigurationID) } nic, rerr := as.InterfacesClient.Get(context.Background(), nicResourceGroup, nicName, "") if rerr != nil { return "", "", fmt.Errorf("GetNodeNameByIPConfigurationID(%s): failed to get interface of name %s: %s", ipConfigurationID, nicName, rerr.Error().Error()) } vmID := "" if nic.InterfacePropertiesFormat != nil && nic.VirtualMachine != nil { vmID = to.String(nic.VirtualMachine.ID) } if vmID == "" { klog.V(2).Infof("GetNodeNameByIPConfigurationID(%s): empty vmID", ipConfigurationID) return "", "", nil } </ins> <del> vm, err := as.getVirtualMachine(types.NodeName(nodeName), azcache.CacheReadTypeDefault) </del> <ins> matches = vmIDRE.FindStringSubmatch(vmID) if len(matches) != 2 { return "", "", fmt.Errorf("invalid virtual machine ID %s", vmID) } vmName := matches[1] vm, err := as.getVirtualMachine(types.NodeName(vmName), azcache.CacheReadTypeDefault) </ins> if err != nil { <del> return "", "", fmt.Errorf("cannot get the virtual machine by node name %s", nodeName) </del> <ins> return "", "", fmt.Errorf("cannot get the virtual machine by node name %s", vmName) </ins> } asID := "" if vm.VirtualMachineProperties != nil && vm.AvailabilitySet != nil { asID = to.String(vm.AvailabilitySet.ID) } if asID == "" { <del> return "", "", fmt.Errorf("cannot get the availability set ID from the virtual machine with node name %s", nodeName) </del> <ins> return vmName, "", nil </ins> } asName, err := getAvailabilitySetNameByID(asID) if err != nil { return "", "", fmt.Errorf("cannot get the availability set name by the availability set ID %s", asID) } <del> return nodeName, strings.ToLower(asName), nil </del> <ins> return vmName, strings.ToLower(asName), nil </ins> }

mockVMClient.EXPECT().Get(gomock.Any(), cloud.ResourceGroup, "k8s-agentpool1-00000000-1", gomock.Any()).Return(test.existingVM, nil) cloud.VirtualMachinesClient = mockVMClient mockNICClient := mockinterfaceclient.NewMockInterface(ctrl) <del> mockNICClient.EXPECT().Get(gomock.Any(), "rg", "k8s-agentpool1-00000000-nic-1", gomock.Any()).Return(test.existingNIC, nil) </del> <ins> test.existingNIC.VirtualMachine = &network.SubResource{ ID: to.StringPtr("/subscriptions/sub/resourceGroups/rg/providers/Microsoft.Compute/virtualMachines/k8s-agentpool1-00000000-1"), } mockNICClient.EXPECT().Get(gomock.Any(), "rg", "k8s-agentpool1-00000000-nic-1", gomock.Any()).Return(test.existingNIC, nil).Times(2) </ins> mockNICClient.EXPECT().CreateOrUpdate(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(nil) cloud.InterfacesClient = mockNICClient

defer ctrl.Finish() cloud := GetTestCloud(ctrl) expectedVM := buildDefaultTestVirtualMachine("/subscriptions/sub/resourceGroups/rg/providers/Microsoft.Compute/availabilitySets/AGENTPOOL1-AVAILABILITYSET-00000000", []string{}) <ins> expectedVM.Name = to.StringPtr("name") </ins> mockVMClient := cloud.VirtualMachinesClient.(*mockvmclient.MockInterface) mockVMClient.EXPECT().Get(gomock.Any(), "rg", "k8s-agentpool1-00000000-0", gomock.Any()).Return(expectedVM, nil) <ins> expectedNIC := buildDefaultTestInterface(true, []string{}) expectedNIC.VirtualMachine = &network.SubResource{ ID: to.StringPtr("/subscriptions/sub/resourceGroups/rg/providers/Microsoft.Compute/virtualMachines/k8s-agentpool1-00000000-0"), } mockNICClient := cloud.InterfacesClient.(*mockinterfaceclient.MockInterface) mockNICClient.EXPECT().Get(gomock.Any(), "rg", "k8s-agentpool1-00000000-nic-0", gomock.Any()).Return(expectedNIC, nil) </ins> ipConfigurationID := `/subscriptions/sub/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool1-00000000-nic-0/ipConfigurations/ipconfig1` nodeName, asName, err := cloud.VMSet.GetNodeNameByIPConfigurationID(ipConfigurationID) assert.NoError(t, err) <del> assert.Equal(t, nodeName, "k8s-agentpool1-00000000-0") assert.Equal(t, asName, "agentpool1-availabilityset-00000000") </del> <ins> assert.Equal(t, "k8s-agentpool1-00000000-0", nodeName) assert.Equal(t, "agentpool1-availabilityset-00000000", asName) </ins> }