Repository: kubernetes/kubernetes (language: go)

Issue:
Unable to create service with ipFamilyPolicy RequireDualStack when service cidr is bigger then /112
### What happened?

We run periodic conformance tests in github.com/kubernetes-sigs/cluster-api and they started to fail https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api#capi-e2e-latestk8s-main

These currently
* use https://dl.k8s.io/ci/latest-1.33.txt to build kind images from source with that version
* create a Cluster with that version by using Cluster API and cluster-api-provider-docker (and in this case with dualstack ip enabled)
* They set `--service-cluster-ip-range=10.128.0.0/12,fd00:100:64::/108` in kube-apiserver (and kube-controller-manager)
* And after finishing cluster creation run conformance tests, including the selector for the `IPv6DualStack` feature tests.

Several conformace tests for `IPv6DualStack` then failed with e.g. the following message: 
```
I0124 07:48:52.822155 16 utils.go:776] Unexpected error: Failed to create node-port-service service: Internal error occurred: failed to allocate a serviceIP: range is full: 
    <*errors.StatusError | 0xc0031bb9a0>: 
    Internal error occurred: failed to allocate a serviceIP: range is full
    {
        ErrStatus: 
            code: 500
            details:
              causes:
              - message: 'failed to allocate a serviceIP: range is full'
            message: 'Internal error occurred: failed to allocate a serviceIP: range is full'
            metadata: {}
            reason: InternalError
            status: Failure,
    }
[FAILED] Failure recorded during attempt 1:
```

Note: The kube-apiserver is able to create the IPAddress object, but the service does not get created.

### What did you expect to happen?

Tests to succeed.

### How can we reproduce it (as minimally and precisely as possible)?

Build kindest/node image for k/k commit 0798325ba13643

```
# checkout k/k on 0798325ba13643
kind build node-image --image "kindest/node:v1.33.0-alpha.0.614_0798325ba13643" $(pwd)
```

Create a kind cluster using the following configuration:

```yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  ipFamily: dual
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
        extraArgs:
          service-cluster-ip-range: "10.96.0.10/12,fd00:100:64::/108"
```

```sh
kind create cluster --name test --config kind-cluster.yaml --image kindest/node:v1.33.0-alpha.0.614_0798325ba13643
```

Try to create a service with `.spec.ipFamilyPolicy` set to `RequireDualStack` or `.spec.ipFamilies` set to `["IPv6"]`:

```sh
$ kubectl create svc clusterip --tcp=80:80 foo -o json --dry-run=client | jq '.spec.ipFamilies = ["IPv6"]' | k apply -f -
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed to allocate a serviceIP: range is full

$ kubectl create svc clusterip --tcp=80:80 foo -o json --dry-run=client | jq '.spec.ipFamilies = ["IPv6"]' | k apply -f -
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed to allocate a serviceIP: range is full
```

Note: there's a chance that the ip allocation in kube-apiserver picks an ipv6 address which is then part the first /112 part of and then works.

### Anything else we need to know?

Follow-up to https://github.com/kubernetes/kubernetes/issues/129753#issuecomment-2609425004

I also stepped into kube-apiserver:
* It fails to validate here that the ip is contained in the CIDR (example configuration: CIDR `fd00:100:64::/108`, IP `fd00:100:64::f:321c`
  * https://github.com/kubernetes/kubernetes/blob/0798325ba13643358aa3ebb7c6ddc3006ac26a7c/pkg/registry/core/service/ipallocator/bitmap.go#L173-L179
* The IP is part of that network, but the bitmap is capped to `/112`


Workarounds:
* Enable FeatureGate `DisableAllocatorDualWrite` in kube-apiserver
* Use a `/112` cidr for services


cc @aojea 

And Kudos for Antonio helping me to dig into it!

### Kubernetes version

<details>

```console
❯ kubectl version
Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.33.0-alpha.0.614+0798325ba13643
WARNING: version difference between client (1.29) and server (1.33) exceeds the supported minor version skew of +/-1
```

</details>


### Cloud provider

<details>
None
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>



Discussion / hints:

Fix patch:
diff --git a/pkg/registry/core/service/ipallocator/cidrallocator.go b/pkg/registry/core/service/ipallocator/cidrallocator.go
index 53d9bb2de668c..7ec7e9cebd914 100644
--- a/pkg/registry/core/service/ipallocator/cidrallocator.go
+++ b/pkg/registry/core/service/ipallocator/cidrallocator.go
@@ -361,6 +361,20 @@ func (c *MetaAllocator) Allocate(ip net.IP) error {
 }
 
 func (c *MetaAllocator) AllocateNextService(service *api.Service) (net.IP, error) {
+	// If the cluster is still using the old allocators use them first to try to
+	// get an IP address to keep backwards compatibility.
+	if !utilfeature.DefaultFeatureGate.Enabled(features.DisableAllocatorDualWrite) {
+		ip, err := c.bitmapAllocator.AllocateNext()
+		if err == nil {
+			allocator, err := c.getAllocator(ip, true)
+			if err != nil {
+				return nil, err
+			}
+			return ip, allocator.AllocateService(service, ip)
+		} else {
+			klog.Infof("no IP address available on the old ClusterIP allocator, trying to get a new address using the new allocators")
+		}
+	}
 	c.mu.Lock()
 	defer c.mu.Unlock()
 	// TODO(aojea) add strategy to return a random allocator but
@@ -376,15 +390,6 @@ func (c *MetaAllocator) AllocateNextService(service *api.Service) (net.IP, error
 		}
 		ip, err := item.allocator.AllocateNextService(service)
 		if err == nil {
-			if !utilfeature.DefaultFeatureGate.Enabled(features.DisableAllocatorDualWrite) {
-				cidr := c.bitmapAllocator.CIDR()
-				if cidr.Contains(ip) {
-					err := c.bitmapAllocator.Allocate(ip)
-					if err != nil {
-						continue
-					}
-				}
-			}
 			return ip, nil
 		}
 	}


Test patch:
diff --git a/pkg/registry/core/service/ipallocator/cidrallocator_test.go b/pkg/registry/core/service/ipallocator/cidrallocator_test.go
index 7bc8b279e8b29..bc72a56ffb1d2 100644
--- a/pkg/registry/core/service/ipallocator/cidrallocator_test.go
+++ b/pkg/registry/core/service/ipallocator/cidrallocator_test.go
@@ -484,9 +484,6 @@ func TestCIDRAllocateDualWrite(t *testing.T) {
 	if f := r.Free(); f != 0 {
 		t.Errorf("free: %d", f)
 	}
-	if _, err := r.AllocateNext(); err == nil {
-		t.Error(err)
-	}
 
 	cidr := newServiceCIDR("test", "192.168.0.0/28")
 	_, err = r.client.ServiceCIDRs().Create(context.Background(), cidr, metav1.CreateOptions{})
@@ -554,9 +551,6 @@ func TestCIDRAllocateDualWriteCollision(t *testing.T) {
 	if f := r.Free(); f != 0 {
 		t.Errorf("free: %d", f)
 	}
-	if _, err := r.AllocateNext(); err == nil {
-		t.Error(err)
-	}
 
 	cidr := newServiceCIDR("test", "192.168.0.0/28")
 	_, err = r.client.ServiceCIDRs().Create(context.Background(), cidr, metav1.CreateOptions{})
diff --git a/test/integration/dualstack/dualstack_test.go b/test/integration/dualstack/dualstack_test.go
index d6a1cc16727ff..da519748f0d32 100644
--- a/test/integration/dualstack/dualstack_test.go
+++ b/test/integration/dualstack/dualstack_test.go
@@ -298,7 +298,7 @@ func TestCreateServiceSingleStackIPv6(t *testing.T) {
 					apiServerOptions,
 					[]string{
 						fmt.Sprintf("--runtime-config=networking.k8s.io/v1beta1=%v", enableMultiServiceCIDR),
-						"--service-cluster-ip-range=2001:db8:1::/112",
+						"--service-cluster-ip-range=2001:db8:1::/108",
 						"--advertise-address=2001:db8::10",
 						"--disable-admission-plugins=ServiceAccount",
 						fmt.Sprintf("--feature-gates=%s=%v,%s=%v", features.MultiCIDRServiceAllocator, enableMultiServiceCIDR, features.DisableAllocatorDualWrite, disableAllocatorDualWrite),
@@ -529,7 +529,7 @@ func TestCreateServiceDualStackIPv4IPv6(t *testing.T) {
 					apiServerOptions,
 					[]string{
 						fmt.Sprintf("--runtime-config=networking.k8s.io/v1beta1=%v", enableMultiServiceCIDR),
-						"--service-cluster-ip-range=10.0.0.0/16,2001:db8:1::/112",
+						"--service-cluster-ip-range=10.0.0.0/16,2001:db8:1::/108",
 						"--advertise-address=10.0.0.1",
 						"--disable-admission-plugins=ServiceAccount",
 						fmt.Sprintf("--feature-gates=%s=%v,%s=%v", features.MultiCIDRServiceAllocator, enableMultiServiceCIDR, features.DisableAllocatorDualWrite, disableAllocatorDualWrite),
@@ -808,7 +808,7 @@ func TestCreateServiceDualStackIPv6IPv4(t *testing.T) {
 					apiServerOptions,
 					[]string{
 						fmt.Sprintf("--runtime-config=networking.k8s.io/v1beta1=%v", enableMultiServiceCIDR),
-						"--service-cluster-ip-range=2001:db8:1::/112,10.0.0.0/16",
+						"--service-cluster-ip-range=2001:db8:1::/108,10.0.0.0/16",
 						"--advertise-address=2001:db8::10",
 						"--disable-admission-plugins=ServiceAccount",
 						fmt.Sprintf("--feature-gates=%s=%v,%s=%v", features.MultiCIDRServiceAllocator, enableMultiServiceCIDR, features.DisableAllocatorDualWrite, disableAllocatorDualWrite),
@@ -1038,7 +1038,7 @@ func TestUpgradeDowngrade(t *testing.T) {
 	s := kubeapiservertesting.StartTestServerOrDie(t,
 		apiServerOptions,
 		[]string{
-			"--service-cluster-ip-range=10.0.0.0/16,2001:db8:1::/112",
+			"--service-cluster-ip-range=10.0.0.0/16,2001:db8:1::/108",
 			"--advertise-address=10.0.0.1",
 			"--disable-admission-plugins=ServiceAccount",
 		},
@@ -1149,7 +1149,7 @@ func TestConvertToFromExternalName(t *testing.T) {
 	s := kubeapiservertesting.StartTestServerOrDie(t,
 		apiServerOptions,
 		[]string{
-			"--service-cluster-ip-range=10.0.0.0/16,2001:db8:1::/112",
+			"--service-cluster-ip-range=10.0.0.0/16,2001:db8:1::/108",
 			"--advertise-address=10.0.0.1",
 			"--disable-admission-plugins=ServiceAccount",
 		},
@@ -1238,7 +1238,7 @@ func TestPreferDualStack(t *testing.T) {
 	s := kubeapiservertesting.StartTestServerOrDie(t,
 		apiServerOptions,
 		[]string{
-			"--service-cluster-ip-range=10.0.0.0/16,2001:db8:1::/112",
+			"--service-cluster-ip-range=10.0.0.0/16,2001:db8:1::/108",
 			"--advertise-address=10.0.0.1",
 			"--disable-admission-plugins=ServiceAccount",
 		},
@@ -1552,7 +1552,7 @@ func TestUpgradeServicePreferToDualStack(t *testing.T) {
 	s = kubeapiservertesting.StartTestServerOrDie(t,
 		apiServerOptions,
 		[]string{
-			"--service-cluster-ip-range=192.168.0.0/24,2001:db8:1::/112",
+			"--service-cluster-ip-range=192.168.0.0/24,2001:db8:1::/108",
 			"--disable-admission-plugins=ServiceAccount",
 		},
 		sharedEtcd)
@@ -1593,7 +1593,7 @@ func TestDowngradeServicePreferToDualStack(t *testing.T) {
 	s := kubeapiservertesting.StartTestServerOrDie(t,
 		apiServerOptions,
 		[]string{
-			"--service-cluster-ip-range=192.168.0.0/24,2001:db8:1::/112",
+			"--service-cluster-ip-range=192.168.0.0/24,2001:db8:1::/108",
 			"--advertise-address=10.0.0.1",
 			"--disable-admission-plugins=ServiceAccount",
 		},