Repository: kubernetes-sigs/cloud-provider-azure
Pull request #8796 (2025-04-02 05:59:05)

[release-1.29] feat: Support migration from basic to standard load balancer
<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide/first-contribution.md#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?

<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature
/kind design

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

/kind feature

#### What this PR does / why we need it:

Introducing a new feature to migrate from basic to standard sku load balancer. To trigger the migration, switch `loadBalancerSKU` in the cloud provider configuration from `basic` to `standard`. The basic load balancer will be removed automatically, and service workloads on it will be migrated to the newly created standard load balancer, with their ip addresses unchanged. This operation may cause downtime.

#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes #

#### Special notes for your reviewer:

#### Does this PR introduce a user-facing change?
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->
```release-note
feat: Support migration from basic to standard load balancer

Introducing a new feature to migrate from basic to standard sku load balancer. To trigger the migration, switch `loadBalancerSKU` in the cloud provider configuration from `basic` to `standard`. The basic load balancer will be removed automatically, and service workloads on it will be migrated to the newly created standard load balancer, with their ip addresses unchanged. This operation may cause downtime.
```

#### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

https://cloud-provider-azure.sigs.k8s.io/development/design-docs/blb-migration/

<!--
This section can be blank if this pull request does not require a release note.

When adding links which point to resources within git repositories, like
KEPs or supporting documentation, please reference a specific commit and avoid
linking directly to the master branch. This ensures that links reference a
specific point in time, rather than a document that may change over time.

See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files

Please use the following format for linking documentation:
- [KEP]: <link>
- [Usage]: <link>
- [Other doc]: <link>
-->
```docs

```


k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/8796#" title="Author self-approved">nilo19</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/release-1.29/OWNERS)~~ [nilo19]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
k8s-ci-robot: @nilo19: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-29 | 665a6b3d4905371a0fe31b22cf70640224edaa34 | [link](https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/8796/pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-29/1907312519521767424) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-29`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=8796). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Anilo19). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
nilo19: /retest

Patch:
diff --git a/pkg/provider/azure_loadbalancer.go b/pkg/provider/azure_loadbalancer.go
index 4f12f72e6..1fec269f3 100644
--- a/pkg/provider/azure_loadbalancer.go
+++ b/pkg/provider/azure_loadbalancer.go
@@ -665,13 +665,12 @@ func (az *Cloud) cleanOrphanedLoadBalancer(lb *network.LoadBalancer, existingLBs
 		klog.V(2).Infof("cleanOrphanedLoadBalancer(%s, %s, %s): deleting the LB since there are no remaining frontendIPConfigurations", lbName, serviceName, clusterName)
 
 		// Remove backend pools from vmSets. This is required for virtual machine scale sets before removing the LB.
-		vmSetName := az.mapLoadBalancerNameToVMSet(lbName, clusterName)
 		if _, ok := az.VMSet.(*availabilitySet); ok {
 			// do nothing for availability set
 			lb.BackendAddressPools = nil
 		}
 
-		if deleteErr := az.safeDeleteLoadBalancer(*lb, clusterName, vmSetName, service); deleteErr != nil {
+		if deleteErr := az.safeDeleteLoadBalancer(*lb, clusterName, service); deleteErr != nil {
 			klog.Warningf("cleanOrphanedLoadBalancer(%s, %s, %s): failed to DeleteLB: %v", lbName, serviceName, clusterName, deleteErr)
 
 			rgName, vmssName, parseErr := retry.GetVMSSMetadataByRawError(deleteErr)
@@ -711,8 +710,10 @@ func (az *Cloud) cleanOrphanedLoadBalancer(lb *network.LoadBalancer, existingLBs
 }
 
 // safeDeleteLoadBalancer deletes the load balancer after decoupling it from the vmSet
-func (az *Cloud) safeDeleteLoadBalancer(lb network.LoadBalancer, _, vmSetName string, service *v1.Service) *retry.Error {
+func (az *Cloud) safeDeleteLoadBalancer(lb network.LoadBalancer, clusterName string, service *v1.Service) *retry.Error {
 	lbBackendPoolIDsToDelete := []string{}
+	vmSetName := az.mapLoadBalancerNameToVMSet(ptr.Deref(lb.Name, ""), clusterName)
+
 	if lb.LoadBalancerPropertiesFormat != nil && lb.BackendAddressPools != nil {
 		for _, bp := range *lb.BackendAddressPools {
 			lbBackendPoolIDsToDelete = append(lbBackendPoolIDsToDelete, ptr.Deref(bp.ID, ""))
@@ -1131,6 +1132,18 @@ func (az *Cloud) ensurePublicIPExists(service *v1.Service, pipName string, domai
 			pip.Tags = make(map[string]*string)
 		}
 
+		if az.useStandardLoadBalancer() {
+			if pip.Sku == nil {
+				pip.Sku = &network.PublicIPAddressSku{
+					Name: network.PublicIPAddressSkuNameStandard,
+				}
+				changed = true
+			} else if !strings.EqualFold(string(pip.Sku.Name), string(network.PublicIPAddressSkuNameStandard)) {
+				pip.Sku.Name = network.PublicIPAddressSkuNameStandard
+				changed = true
+			}
+		}
+
 		// return if pip exist and dns label is the same
 		if strings.EqualFold(getDomainNameLabel(&pip), domainNameLabel) {
 			if existingServiceName := getServiceFromPIPDNSTags(pip.Tags); existingServiceName != "" && strings.EqualFold(existingServiceName, serviceName) {
@@ -1765,6 +1778,11 @@ func (az *Cloud) reconcileLoadBalancer(ctx context.Context, clusterName string,
 		return nil, fmt.Errorf("reconcileLoadBalancer: failed to list managed LB: %w", err)
 	}
 
+	if existingLBs, err = az.cleanupBasicLoadBalancer(clusterName, service, existingLBs); err != nil {
+		klog.ErrorS(err, "reconcileLoadBalancer: failed to check and remove outdated basic load balancers", "service", serviceName)
+		return nil, err
+	}
+
 	// Delete backend pools for local service if:
 	// 1. the cluster is migrating from multi-slb to single-slb,
 	// 2. the service is changed from local to cluster.
@@ -2589,7 +2607,11 @@ func (az *Cloud) reconcileFrontendIPConfigs(clusterName string,
 				} else if status != nil && len(status.Ingress) > 0 && ingressIPInSubnet(status.Ingress) {
 					klog.V(4).Infof("reconcileFrontendIPConfigs for service (%s): keep the original private IP %s", serviceName, privateIP)
 					configProperties.PrivateIPAllocationMethod = network.Static
-					configProperties.PrivateIPAddress = pointer.String(privateIP)
+					configProperties.PrivateIPAddress = ptr.To(privateIP)
+				} else if len(service.Status.LoadBalancer.Ingress) > 0 && ingressIPInSubnet(service.Status.LoadBalancer.Ingress) {
+					klog.V(4).Infof("reconcileFrontendIPConfigs for service (%s): keep the original private IP %s from service.status.loadbalacner.ingress", serviceName, privateIP)
+					configProperties.PrivateIPAllocationMethod = network.Static
+					configProperties.PrivateIPAddress = ptr.To(privateIP)
 				} else {
 					// We'll need to call GetLoadBalancer later to retrieve allocated IP.
 					klog.V(4).Infof("reconcileFrontendIPConfigs for service (%s): dynamically allocate the private IP", serviceName)
diff --git a/pkg/provider/azure_loadbalancer_repo.go b/pkg/provider/azure_loadbalancer_repo.go
index b9d0829fc..f448dba2f 100644
--- a/pkg/provider/azure_loadbalancer_repo.go
+++ b/pkg/provider/azure_loadbalancer_repo.go
@@ -409,3 +409,41 @@ func isNICPool(bp network.BackendAddressPool) bool {
 	}
 	return false
 }
+
+// cleanupBasicLoadBalancer removes outdated basic load balancers
+// when the loadBalancerSkus is `Standard`. It ensures the backend pool of the basic
+// load balancer is empty before deleting the load balancer.
+func (az *Cloud) cleanupBasicLoadBalancer(
+	clusterName string, service *v1.Service, existingLBs *[]network.LoadBalancer,
+) (*[]network.LoadBalancer, error) {
+	if !az.useStandardLoadBalancer() {
+		return existingLBs, nil
+	}
+
+	var elbRemoved bool
+	for i := len(*existingLBs) - 1; i >= 0; i-- {
+		lb := (*existingLBs)[i]
+		if lb.Sku != nil && lb.Sku.Name == network.LoadBalancerSkuNameBasic {
+			klog.V(2).Infof("cleanupBasicLoadBalancer: found basic load balancer %q, removing it", *lb.Name)
+			if err := az.safeDeleteLoadBalancer(lb, clusterName, service); err != nil {
+				klog.ErrorS(err.Error(), "cleanupBasicLoadBalancer: failed to delete outdated basic load balancer", "loadBalancerName", *lb.Name)
+				return nil, err.Error()
+			}
+			*existingLBs = append((*existingLBs)[:i], (*existingLBs)[i+1:]...)
+			if !strings.Contains(strings.ToLower(ptr.Deref(lb.Name, "")), strings.ToLower(consts.InternalLoadBalancerNameSuffix)) {
+				elbRemoved = true
+			}
+		}
+	}
+	// The lb refs in pip will be changed after the removal, so we need to
+	// reinitialize the cache to prevent etag mismatches.
+	if elbRemoved {
+		var err error
+		az.pipCache, err = az.newPIPCache()
+		if err != nil {
+			klog.ErrorS(err, "cleanupBasicLoadBalancer: failed to refresh pip cache")
+			return nil, err
+		}
+	}
+	return existingLBs, nil
+}
diff --git a/pkg/provider/azure_standard.go b/pkg/provider/azure_standard.go
index a4a3d8c25..f61cde02c 100644
--- a/pkg/provider/azure_standard.go
+++ b/pkg/provider/azure_standard.go
@@ -122,7 +122,7 @@ func (az *Cloud) getNetworkResourceSubscriptionID() string {
 	return az.SubscriptionID
 }
 
-func (az *Cloud) mapLoadBalancerNameToVMSet(lbName string, clusterName string) (vmSetName string) {
+func (az *Cloud) mapLoadBalancerNameToVMSet(lbName, clusterName string) (vmSetName string) {
 	vmSetName = trimSuffixIgnoreCase(lbName, consts.InternalLoadBalancerNameSuffix)
 	if strings.EqualFold(clusterName, vmSetName) {
 		vmSetName = az.VMSet.GetPrimaryVMSetName()


Test patch:
diff --git a/pkg/provider/azure_loadbalancer_repo_test.go b/pkg/provider/azure_loadbalancer_repo_test.go
index 7a6cc624a..b240fb3f5 100644
--- a/pkg/provider/azure_loadbalancer_repo_test.go
+++ b/pkg/provider/azure_loadbalancer_repo_test.go
@@ -525,3 +525,198 @@ func TestIsNICPool(t *testing.T) {
 		})
 	}
 }
+
+func TestCleanupBasicLoadBalancer(t *testing.T) {
+	ctrl := gomock.NewController(t)
+	defer ctrl.Finish()
+
+	testCases := []struct {
+		desc                 string
+		useStandardLB        bool
+		existingLBs          *[]network.LoadBalancer
+		expectedErr          bool
+		expectedDeleteCalled bool
+	}{
+		{
+			desc:                 "UseStandardLoadBalancer=false should skip deletion",
+			useStandardLB:        false,
+			existingLBs:          &[]network.LoadBalancer{},
+			expectedErr:          false,
+			expectedDeleteCalled: false,
+		},
+		{
+			desc:          "Basic LB should be deleted when UseStandardLoadBalancer=true",
+			useStandardLB: true,
+			existingLBs: &[]network.LoadBalancer{
+				{
+					Name: ptr.To("test-lb"),
+					Sku: &network.LoadBalancerSku{
+						Name: network.LoadBalancerSkuNameBasic,
+					},
+					LoadBalancerPropertiesFormat: &network.LoadBalancerPropertiesFormat{
+						BackendAddressPools: &[]network.BackendAddressPool{
+							{
+								ID: ptr.To("pool-id-1"),
+							},
+						},
+					},
+				},
+			},
+			expectedErr:          false,
+			expectedDeleteCalled: true,
+		},
+		{
+			desc:          "Internal basic LB should be deleted but not reinitialize pip cache",
+			useStandardLB: true,
+			existingLBs: &[]network.LoadBalancer{
+				{
+					Name: ptr.To("test-lb-" + consts.InternalLoadBalancerNameSuffix),
+					Sku: &network.LoadBalancerSku{
+						Name: network.LoadBalancerSkuNameBasic,
+					},
+					LoadBalancerPropertiesFormat: &network.LoadBalancerPropertiesFormat{
+						BackendAddressPools: &[]network.BackendAddressPool{
+							{
+								ID: ptr.To("pool-id-1"),
+							},
+						},
+					},
+				},
+			},
+			expectedErr:          false,
+			expectedDeleteCalled: true,
+		},
+		{
+			desc:          "Standard LB should not be deleted",
+			useStandardLB: true,
+			existingLBs: &[]network.LoadBalancer{
+				{
+					Name: ptr.To("test-lb"),
+					Sku: &network.LoadBalancerSku{
+						Name: network.LoadBalancerSkuNameStandard,
+					},
+					LoadBalancerPropertiesFormat: &network.LoadBalancerPropertiesFormat{},
+				},
+			},
+			expectedErr:          false,
+			expectedDeleteCalled: false,
+		},
+		{
+			desc:          "Mix of basic and standard LBs should only delete basic",
+			useStandardLB: true,
+			existingLBs: &[]network.LoadBalancer{
+				{
+					Name: ptr.To("test-lb-standard"),
+					Sku: &network.LoadBalancerSku{
+						Name: network.LoadBalancerSkuNameStandard,
+					},
+					LoadBalancerPropertiesFormat: &network.LoadBalancerPropertiesFormat{},
+				},
+				{
+					Name: ptr.To("test-lb-basic"),
+					Sku: &network.LoadBalancerSku{
+						Name: network.LoadBalancerSkuNameBasic,
+					},
+					LoadBalancerPropertiesFormat: &network.LoadBalancerPropertiesFormat{
+						BackendAddressPools: &[]network.BackendAddressPool{
+							{
+								ID: ptr.To("pool-id-1"),
+							},
+						},
+					},
+				},
+			},
+			expectedErr:          false,
+			expectedDeleteCalled: true,
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.desc, func(t *testing.T) {
+			// Print initial test case info
+			t.Logf("Test case: %s", tc.desc)
+			t.Logf("Initial LBs:")
+			for i, lb := range *tc.existingLBs {
+				t.Logf("  LB[%d]: Name=%s, SKU=%s", i, *lb.Name, lb.Sku.Name)
+			}
+
+			az := GetTestCloud(ctrl)
+			az.Config.LoadBalancerSku = consts.LoadBalancerSkuStandard
+
+			if !tc.useStandardLB {
+				az.Config.LoadBalancerSku = consts.LoadBalancerSkuBasic
+			}
+
+			service := &v1.Service{}
+			clusterName := "testCluster"
+
+			// Setup mocks
+			mockVMSet := NewMockVMSet(ctrl)
+			az.VMSet = mockVMSet
+			mockLBClient := az.LoadBalancerClient.(*mockloadbalancerclient.MockInterface)
+
+			// Set up mocks for safeDeleteLoadBalancer if we expect it to be called
+			if tc.expectedDeleteCalled {
+				for _, lb := range *tc.existingLBs {
+					if lb.Sku != nil && lb.Sku.Name == network.LoadBalancerSkuNameBasic {
+						// Mock the EnsureBackendPoolDeleted call
+						mockVMSet.EXPECT().EnsureBackendPoolDeleted(
+							gomock.Any(), // service
+							gomock.Any(), // backendPoolIDs
+							gomock.Any(), // vmSetName
+							gomock.Any(), // backendAddressPools
+							true,         // deleteEmptyPool
+						).Return(true, nil).AnyTimes()
+
+						// Mock the DeleteLB call
+						mockLBClient.EXPECT().Delete(
+							gomock.Any(), // context
+							az.ResourceGroup,
+							*lb.Name,
+						).Return(nil).AnyTimes()
+					}
+				}
+			}
+
+			// Call the function under test
+			result, err := az.cleanupBasicLoadBalancer(clusterName, service, tc.existingLBs)
+
+			// Debugging output
+			t.Logf("Original LBs: %d, Result LBs: %d", len(*tc.existingLBs), len(*result))
+			for i, lb := range *tc.existingLBs {
+				t.Logf("Original LB[%d]: Name=%s, SKU=%s", i, *lb.Name, lb.Sku.Name)
+			}
+			for i, lb := range *result {
+				t.Logf("Result LB[%d]: Name=%s, SKU=%s", i, *lb.Name, lb.Sku.Name)
+			}
+
+			// Check for errors if we're expecting them
+			if tc.expectedErr {
+				assert.Error(t, err)
+				return
+			}
+			assert.NoError(t, err)
+
+			// Verify the number of remaining LBs
+			expectedLBCount := 0
+			if !tc.expectedDeleteCalled {
+				expectedLBCount = len(*tc.existingLBs)
+			} else {
+				// Count standard LBs (which should not be deleted)
+				for _, lb := range *tc.existingLBs {
+					if lb.Sku != nil && lb.Sku.Name == network.LoadBalancerSkuNameStandard {
+						expectedLBCount++
+					}
+				}
+			}
+			assert.Equal(t, expectedLBCount, len(*result), "Expected %d load balancers after deletion, got %d", expectedLBCount, len(*result))
+
+			// If we expect LBs to remain, verify they are not basic
+			if len(*result) > 0 {
+				for _, lb := range *result {
+					assert.NotEqual(t, network.LoadBalancerSkuNameBasic, lb.Sku.Name, "Found a basic load balancer that should have been deleted")
+				}
+			}
+		})
+	}
+}
diff --git a/pkg/provider/azure_loadbalancer_test.go b/pkg/provider/azure_loadbalancer_test.go
index 9b1df569f..eb1a41f8e 100644
--- a/pkg/provider/azure_loadbalancer_test.go
+++ b/pkg/provider/azure_loadbalancer_test.go
@@ -5328,6 +5328,9 @@ func TestEnsurePublicIPExistsCommon(t *testing.T) {
 					PublicIPAllocationMethod: network.Static,
 				},
 				Tags: map[string]*string{},
+				Sku: &network.PublicIPAddressSku{
+					Name: network.PublicIPAddressSkuNameStandard,
+				},
 			},
 			shouldPutPIP: true,
 		},
@@ -6783,6 +6786,9 @@ func TestReconcileFrontendIPConfigs(t *testing.T) {
 						PublicIPAllocationMethod: network.Static,
 						IPAddress:                pointer.String("fe::1"),
 					},
+					Sku: &network.PublicIPAddressSku{
+						Name: network.PublicIPAddressSkuNameStandard,
+					},
 				},
 				{
 					Name: pointer.String("pipV4"),
@@ -6790,6 +6796,9 @@ func TestReconcileFrontendIPConfigs(t *testing.T) {
 						PublicIPAddressVersion: network.IPv4,
 						IPAddress:              pointer.String("1.2.3.4"),
 					},
+					Sku: &network.PublicIPAddressSku{
+						Name: network.PublicIPAddressSkuNameStandard,
+					},
 				},
 			},
 			status:        nil,
@@ -7128,7 +7137,7 @@ func TestSafeDeleteLoadBalancer(t *testing.T) {
 					BackendAddressPools: &[]network.BackendAddressPool{},
 				},
 			}
-			err := cloud.safeDeleteLoadBalancer(lb, "cluster", "vmss", &svc)
+			err := cloud.safeDeleteLoadBalancer(lb, "cluster", &svc)
 			assert.Equal(t, tc.expectedErr, err)
 			if len(tc.multiSLBConfigs) > 0 {
 				assert.Equal(t, tc.expectedMultiSLBConfigs, cloud.MultipleStandardLoadBalancerConfigurations)