Repository: kubernetes-sigs/cloud-provider-azure (go)

Issue:
Check the nil pointer of service.spec.ipfamilies
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!

If the matter is security related, please disclose it privately via https://kubernetes.io/security/
-->


#### What happened:
https://github.com/kubernetes-sigs/cloud-provider-azure/blame/release-1.29/pkg/provider/azure_utils.go#L443
In some cases, the service.spec.ipfamilies, introducing a crashing state for ccm.

#### What you expected to happen:
The ccm pod should not be crashed.

#### How to reproduce it (as minimally and precisely as possible):
Change a service type from load balancer to external names.

#### Anything else we need to know?:

#### Environment:

- Kubernetes version (use `kubectl version`):
- Cloud provider or hardware configuration:
- OS (e.g: `cat /etc/os-release`):
- Kernel (e.g. `uname -a`):
- Install tools:
- Network plugin and version (if this is a network-related bug):
- Others:


Pull request:
[release-1.28] fix: Support switching from loadbalancer to externalName for services
<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide/first-contribution.md#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?

<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature
/kind design

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

/kind bug

#### What this PR does / why we need it:

Fix two issues when changing service type from lb to externalName.
1. ExternalName service has no ipFamilies, we should not panic this.
2. We should not use `v4Enabled, v6Enabled := getIPFamiliesEnabled(service)` to check what backend pools should be deleted from the vmss.

#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes #7503

#### Special notes for your reviewer:

#### Does this PR introduce a user-facing change?
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->
```release-note
fix: Support switching from loadbalancer to externalName for services
```

#### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

<!--
This section can be blank if this pull request does not require a release note.

When adding links which point to resources within git repositories, like
KEPs or supporting documentation, please reference a specific commit and avoid
linking directly to the master branch. This ensures that links reference a
specific point in time, rather than a document that may change over time.

See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files

Please use the following format for linking documentation:
- [KEP]: <link>
- [Usage]: <link>
- [Other doc]: <link>
-->
```docs

```


Patch:
diff --git a/Dockerfile b/Dockerfile
index 9c34c4835..9e73ee082 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -14,7 +14,7 @@
 
 # syntax=docker/dockerfile:1
 
-FROM --platform=linux/amd64 golang:1.23-bullseye@sha256:1002103155eb5802f6876dceb8cb8bf965cc6d3deeb34f11e4774926aca6a1fd AS builder
+FROM --platform=linux/amd64 golang:1.23-bullseye@sha256:d4387d8b2ee5b78a199fae168a761d0fb0f33f172642d20856f29a67204af16b AS builder
 
 ARG ENABLE_GIT_COMMAND=true
 ARG ARCH=amd64
diff --git a/cloud-node-manager.Dockerfile b/cloud-node-manager.Dockerfile
index be185d9f5..c4fc398b7 100644
--- a/cloud-node-manager.Dockerfile
+++ b/cloud-node-manager.Dockerfile
@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM --platform=linux/amd64 golang:1.23-bullseye@sha256:1002103155eb5802f6876dceb8cb8bf965cc6d3deeb34f11e4774926aca6a1fd AS builder
+FROM --platform=linux/amd64 golang:1.23-bullseye@sha256:d4387d8b2ee5b78a199fae168a761d0fb0f33f172642d20856f29a67204af16b AS builder
 
 ARG ENABLE_GIT_COMMAND=true
 ARG ARCH=amd64
diff --git a/pkg/provider/azure_loadbalancer.go b/pkg/provider/azure_loadbalancer.go
index 932bb51c0..9cc560f88 100644
--- a/pkg/provider/azure_loadbalancer.go
+++ b/pkg/provider/azure_loadbalancer.go
@@ -41,6 +41,7 @@ import (
 	servicehelpers "k8s.io/cloud-provider/service/helpers"
 	"k8s.io/klog/v2"
 	"k8s.io/utils/pointer"
+	"k8s.io/utils/ptr"
 	"k8s.io/utils/strings/slices"
 
 	azcache "sigs.k8s.io/cloud-provider-azure/pkg/cache"
@@ -648,15 +649,12 @@ func (az *Cloud) cleanOrphanedLoadBalancer(lb *network.LoadBalancer, existingLBs
 }
 
 // safeDeleteLoadBalancer deletes the load balancer after decoupling it from the vmSet
-func (az *Cloud) safeDeleteLoadBalancer(lb network.LoadBalancer, clusterName, vmSetName string, service *v1.Service) *retry.Error {
-	lbBackendPoolIDs := az.getBackendPoolIDs(clusterName, pointer.StringDeref(lb.Name, ""))
+func (az *Cloud) safeDeleteLoadBalancer(lb network.LoadBalancer, _, vmSetName string, service *v1.Service) *retry.Error {
 	lbBackendPoolIDsToDelete := []string{}
-	v4Enabled, v6Enabled := getIPFamiliesEnabled(service)
-	if v4Enabled {
-		lbBackendPoolIDsToDelete = append(lbBackendPoolIDsToDelete, lbBackendPoolIDs[consts.IPVersionIPv4])
-	}
-	if v6Enabled {
-		lbBackendPoolIDsToDelete = append(lbBackendPoolIDsToDelete, lbBackendPoolIDs[consts.IPVersionIPv6])
+	if lb.LoadBalancerPropertiesFormat != nil && lb.BackendAddressPools != nil {
+		for _, bp := range *lb.BackendAddressPools {
+			lbBackendPoolIDsToDelete = append(lbBackendPoolIDsToDelete, ptr.Deref(bp.ID, ""))
+		}
 	}
 	if _, err := az.VMSet.EnsureBackendPoolDeleted(service, lbBackendPoolIDsToDelete, vmSetName, lb.BackendAddressPools, true); err != nil {
 		return retry.NewError(false, fmt.Errorf("safeDeleteLoadBalancer: failed to EnsureBackendPoolDeleted: %w", err))
@@ -1433,7 +1431,7 @@ func (az *Cloud) isFrontendIPChanged(
 	if fipIPVersion != "" {
 		isIPv6 = fipIPVersion == network.IPv6
 	} else {
-		if isIPv6, err = az.isFIPIPv6(service, pipRG, &config); err != nil {
+		if isIPv6, err = az.isFIPIPv6(service, &config); err != nil {
 			return false, err
 		}
 	}
@@ -1582,7 +1580,6 @@ func (az *Cloud) findFrontendIPConfigsOfService(
 	service *v1.Service,
 ) (map[bool]*network.FrontendIPConfiguration, error) {
 	fipsOfServiceMap := map[bool]*network.FrontendIPConfiguration{}
-	pipRG := az.getPublicIPAddressResourceGroup(service)
 	for _, config := range *fipConfigs {
 		config := config
 		owns, _, fipIPVersion := az.serviceOwnsFrontendIP(config, service)
@@ -1592,7 +1589,7 @@ func (az *Cloud) findFrontendIPConfigsOfService(
 			if fipIPVersion != "" {
 				fipIsIPv6 = fipIPVersion == network.IPv6
 			} else {
-				if fipIsIPv6, err = az.isFIPIPv6(service, pipRG, &config); err != nil {
+				if fipIsIPv6, err = az.isFIPIPv6(service, &config); err != nil {
 					return nil, err
 				}
 			}
@@ -1767,7 +1764,6 @@ func (az *Cloud) reconcileLoadBalancer(clusterName string, service *v1.Service,
 	}
 
 	// update probes/rules
-	pipRG := az.getPublicIPAddressResourceGroup(service)
 	for _, ownedFIPConfig := range ownedFIPConfigs {
 		if ownedFIPConfig == nil {
 			continue
@@ -1782,7 +1778,7 @@ func (az *Cloud) reconcileLoadBalancer(clusterName string, service *v1.Service,
 		if fipIPVersion != "" {
 			isIPv6 = fipIPVersion == network.IPv6
 		} else {
-			if isIPv6, err = az.isFIPIPv6(service, pipRG, ownedFIPConfig); err != nil {
+			if isIPv6, err = az.isFIPIPv6(service, ownedFIPConfig); err != nil {
 				return nil, err
 			}
 		}
@@ -2437,8 +2433,6 @@ func (az *Cloud) reconcileFrontendIPConfigs(clusterName string,
 			}
 		}
 
-		pipRG := az.getPublicIPAddressResourceGroup(service)
-
 		for i := len(newConfigs) - 1; i >= 0; i-- {
 			config := newConfigs[i]
 			isServiceOwnsFrontendIP, _, fipIPVersion := az.serviceOwnsFrontendIP(config, service)
@@ -2452,7 +2446,7 @@ func (az *Cloud) reconcileFrontendIPConfigs(clusterName string,
 			if fipIPVersion != "" {
 				isIPv6 = fipIPVersion == network.IPv6
 			} else {
-				if isIPv6, err = az.isFIPIPv6(service, pipRG, &config); err != nil {
+				if isIPv6, err = az.isFIPIPv6(service, &config); err != nil {
 					return nil, toDeleteConfigs, false, err
 				}
 			}
diff --git a/pkg/provider/azure_privatelinkservice.go b/pkg/provider/azure_privatelinkservice.go
index 80f28045b..909715263 100644
--- a/pkg/provider/azure_privatelinkservice.go
+++ b/pkg/provider/azure_privatelinkservice.go
@@ -42,7 +42,6 @@ func (az *Cloud) reconcilePrivateLinkService(
 	wantPLS bool,
 ) error {
 	isinternal := requiresInternalLoadBalancer(service)
-	pipRG := az.getPublicIPAddressResourceGroup(service)
 	_, _, fipIPVersion := az.serviceOwnsFrontendIP(*fipConfig, service)
 	serviceName := getServiceName(service)
 	var isIPv6 bool
@@ -50,7 +49,7 @@ func (az *Cloud) reconcilePrivateLinkService(
 	if fipIPVersion != "" {
 		isIPv6 = fipIPVersion == network.IPv6
 	} else {
-		if isIPv6, err = az.isFIPIPv6(service, pipRG, fipConfig); err != nil {
+		if isIPv6, err = az.isFIPIPv6(service, fipConfig); err != nil {
 			klog.Errorf("reconcilePrivateLinkService for service(%s): failed to get FIP IP family: %v", serviceName, err)
 			return err
 		}
diff --git a/pkg/provider/azure_utils.go b/pkg/provider/azure_utils.go
index 4c45e7ad6..4fdf0aaa4 100644
--- a/pkg/provider/azure_utils.go
+++ b/pkg/provider/azure_utils.go
@@ -473,9 +473,12 @@ func getResourceByIPFamily(resource string, isDualStack, isIPv6 bool) string {
 
 // isFIPIPv6 checks if the frontend IP configuration is of IPv6.
 // NOTICE: isFIPIPv6 assumes the FIP is owned by the Service and it is the primary Service.
-func (az *Cloud) isFIPIPv6(service *v1.Service, pipRG string, fip *network.FrontendIPConfiguration) (bool, error) {
+func (az *Cloud) isFIPIPv6(service *v1.Service, fip *network.FrontendIPConfiguration) (bool, error) {
 	isDualStack := isServiceDualStack(service)
 	if !isDualStack {
+		if len(service.Spec.IPFamilies) == 0 {
+			return false, nil
+		}
 		return service.Spec.IPFamilies[0] == v1.IPv6Protocol, nil
 	}
 	return managedResourceHasIPv6Suffix(pointer.StringDeref(fip.Name, "")), nil
diff --git a/pkg/provider/azure_vmss.go b/pkg/provider/azure_vmss.go
index e45572661..30c2e9bad 100644
--- a/pkg/provider/azure_vmss.go
+++ b/pkg/provider/azure_vmss.go
@@ -2009,7 +2009,9 @@ func (ss *ScaleSet) EnsureBackendPoolDeleted(service *v1.Service, backendPoolIDs
 
 	for _, backendPool := range *backendAddressPools {
 		for _, backendPoolID := range backendPoolIDs {
-			if strings.EqualFold(*backendPool.ID, backendPoolID) && backendPool.BackendIPConfigurations != nil {
+			if strings.EqualFold(*backendPool.ID, backendPoolID) &&
+				backendPool.BackendAddressPoolPropertiesFormat != nil &&
+				backendPool.BackendIPConfigurations != nil {
 				for _, ipConf := range *backendPool.BackendIPConfigurations {
 					if ipConf.ID == nil {
 						continue


Test patch:
diff --git a/e2e.Dockerfile b/e2e.Dockerfile
index 6e6631e48..2c221c5a5 100644
--- a/e2e.Dockerfile
+++ b/e2e.Dockerfile
@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM golang:1.23-bullseye@sha256:1002103155eb5802f6876dceb8cb8bf965cc6d3deeb34f11e4774926aca6a1fd
+FROM golang:1.23-bullseye@sha256:d4387d8b2ee5b78a199fae168a761d0fb0f33f172642d20856f29a67204af16b
 
 WORKDIR /go/src/sigs.k8s.io/cloud-provider-azure
 
diff --git a/pkg/provider/azure_loadbalancer_test.go b/pkg/provider/azure_loadbalancer_test.go
index e9d386b2d..449e13d16 100644
--- a/pkg/provider/azure_loadbalancer_test.go
+++ b/pkg/provider/azure_loadbalancer_test.go
@@ -7629,6 +7629,8 @@ func TestCleanOrphanedLoadBalancerLBInUseByVMSS(t *testing.T) {
 		expectedVMSS := buildTestVMSSWithLB(testVMSSName, "vmss-vm-", []string{testLBBackendpoolID0}, false)
 		mockVMSSClient := cloud.VirtualMachineScaleSetsClient.(*mockvmssclient.MockInterface)
 		mockVMSSClient.EXPECT().List(gomock.Any(), "rg").Return([]compute.VirtualMachineScaleSet{expectedVMSS}, nil)
+		mockVMSSClient.EXPECT().Get(gomock.Any(), "rg", testVMSSName).Return(expectedVMSS, nil)
+		mockVMSSClient.EXPECT().CreateOrUpdate(gomock.Any(), "rg", testVMSSName, gomock.Any()).Return(nil)
 
 		service := getTestService("test", v1.ProtocolTCP, nil, false, 80)
 		lb := getTestLoadBalancer(pointer.String("test"), pointer.String("rg"), pointer.String("test"), pointer.String("test"), service, consts.LoadBalancerSkuStandard)
diff --git a/pkg/provider/azure_utils_test.go b/pkg/provider/azure_utils_test.go
index c72930d01..ee64d43ff 100644
--- a/pkg/provider/azure_utils_test.go
+++ b/pkg/provider/azure_utils_test.go
@@ -985,12 +985,21 @@ func TestIsFIPIPv6(t *testing.T) {
 			},
 			expectedIsIPv6: true,
 		},
+		{
+			desc: "enpty ip families",
+			svc: v1.Service{
+				Spec: v1.ServiceSpec{
+					IPFamilies: []v1.IPFamily{},
+				},
+			},
+			expectedIsIPv6: false,
+		},
 	}
 	for _, tc := range testcases {
 		tc := tc
 		t.Run(tc.desc, func(t *testing.T) {
 			az := GetTestCloud(ctrl)
-			isIPv6, err := az.isFIPIPv6(&tc.svc, "rg", tc.fip)
+			isIPv6, err := az.isFIPIPv6(&tc.svc, tc.fip)
 			assert.Nil(t, err)
 			assert.Equal(t, tc.expectedIsIPv6, isIPv6)
 		})