Repository: kubernetes/kubernetes (language: go)

Issue:
Sidecar Containers for Services are not accessible
### What happened?

I'm using a helm chart to deploy an application, and I typically insert an specific proxy I need as a legacy sidecar (after init). However, this helm chart specifically does now allow specifying additional containers (again, legacy sidecar pattern), but it does allow specifying init containers using the newer [Sidecar Containers](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/) feature.

The sidecar config might look like this with the new sidecar pattern:
```yaml
      initContainers:
        - name: myproxy
          image: myproxy
          restartPolicy: Always
          ports:
            - containerPort: 8080
              name: proxy-port
              protocol: TCP
```

After deploying this:
- The `v1/Service` has endpoints that point to the target pods
- Manually, I can connect to `<pod IP>:8080`
- Attempting to connect to the service results in a connection refused.

### What did you expect to happen?

The connection to the sidecar should work.

### How can we reproduce it (as minimally and precisely as possible)?

- Create a pod with a named port in a container under `initContainers` with `restartPolicy: Always`
- Create a `v1/Service` that has a `selector` that targets the pod
- Attempt to lookup the SRV record

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
Client Version: v1.31.0
Kustomize Version: v5.4.2
Server Version: v1.30.5-gke.1443001
```

</details>


### Cloud provider

<details>
GKE
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>



Discussion / hints:
/sig network
The issue did not reproduce in my local kind cluster running Kubernetes 1.30.4 and 1.30.6.
At first, I thought it might be a bug fixed in https://github.com/kubernetes/kubernetes/pull/127976, but that doesn’t seem to be the case. Are you specifying the port name (`spec.ports[0].name` in the following example) defined in the Service? If not specified, the SRV record will not be created. Could you share the complete reproduction steps?

```console
❯ kind create cluster --image kindest/node:v1.30.4@sha256:976ea815844d5fa93be213437e3ff5754cd599b040946b5cca43ca45c2047114
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.30.4) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a nice day! 👋

❯ kubectl get nodes
NAME                 STATUS   ROLES           AGE   VERSION
kind-control-plane   Ready    control-plane   65s   v1.30.4

❯ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: http-server
  labels:
    app: http-server
spec:
  terminationGracePeriodSeconds: 1
  restartPolicy: Never
  initContainers:
  - name: el-sidecar1
    image: registry.k8s.io/e2e-test-images/agnhost:2.52
    args: ["netexec", "--http-port=8090", "--udp-port=-1"]
    restartPolicy: Always
    ports:
    - containerPort: 8090
      name: http-sidecar
      protocol: TCP
  containers:
  - name: el1
    image: registry.k8s.io/e2e-test-images/agnhost:2.52
    args: ["netexec", "--http-port=8080", "--udp-port=-1"]
    ports:
    - containerPort: 8080
      name: http
      protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: http-server
spec:
  ports:
  - name: http-sidecar
    protocol: TCP
    port: 8090
    targetPort: http-sidecar
  selector:
    app: http-server
  type: ClusterIP
EOF
pod/http-server created
service/http-server created

❯ kubectl exec -it http-server -c el1 -- dig SRV _http-sidecar._tcp.http-server.default.svc.cluster.local

; <<>> DiG 9.16.48 <<>> SRV _http-sidecar._tcp.http-server.default.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7488
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: f4bc48863118d01d (echoed)
;; QUESTION SECTION:
;_http-sidecar._tcp.http-server.default.svc.cluster.local. IN SRV

;; ANSWER SECTION:
_http-sidecar._tcp.http-server.default.svc.cluster.local. 27 IN	SRV 0 100 8090 http-server.default.svc.cluster.local.

;; ADDITIONAL SECTION:
http-server.default.svc.cluster.local. 27 IN A	10.96.98.56

;; Query time: 1 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Sat Nov 16 13:00:21 UTC 2024
;; MSG SIZE  rcvd: 263
```
I just want to clarify this means the `v1/Service` doesn't work as the ultimate result. I traced the cause back to a lack of SRV records for the port in question. 
Thanks @toVersus - I've taken your example and updated it. I've also updated the issue to be a bit more broad - since it is not a problem with SRV records.

Below is a an example with two services:
- `http-server` - points to the initContainer port
- `http-server-direct` - points to the main container port

```yaml
---
apiVersion: v1
kind: Pod
metadata:
  name: http-server
  labels:
    app: http-server
spec:
  terminationGracePeriodSeconds: 1
  restartPolicy: Never
  initContainers:
  - name: el-sidecar1
    image: registry.k8s.io/e2e-test-images/agnhost:2.52
    args: ["netexec", "--http-port=8090", "--udp-port=-1"]
    restartPolicy: Always
    ports:
    - containerPort: 8090
      name: http-sidecar
      protocol: TCP
  containers:
  - name: el1
    image: registry.k8s.io/e2e-test-images/agnhost:2.52
    args: ["netexec", "--http-port=8080", "--udp-port=-1"]
    ports:
    - containerPort: 8080
      name: http-direct
      protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: http-server
spec:
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: http-sidecar
  selector:
    app: http-server
  type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
  name: http-server-direct
spec:
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: http-direct
  selector:
    app: http-server
  type: ClusterIP
```

Here the `http-server-direct` service is accessible (this is from a pod in the cluster):
```
# curl -v http-server-direct.default.svc.cluster.local
*   Trying 10.43.119.252:80...
* Connected to http-server-direct.default.svc.cluster.local (10.43.119.252) port 80 (#0)
> GET / HTTP/1.1
> Host: http-server-direct.default.svc.cluster.local
> User-Agent: curl/7.88.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Mon, 18 Nov 2024 00:51:35 GMT
< Content-Length: 61
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host http-server-direct.default.svc.cluster.local left intact
NOW: 2024-11-18 00:51:35.820105082 +0000 UTC m=+492.084894194
```

However, the `http-server` service is NOT accessible:
```
# curl -v http-server.default.svc.cluster.local
*   Trying 10.43.62.224:80...
* connect to 10.43.62.224 port 80 failed: Connection refused
* Failed to connect to http-server.default.svc.cluster.local port 80 after 1 ms: Couldn't connect to server
* Closing connection 0
curl: (7) Failed to connect to http-server.default.svc.cluster.local port 80 after 1 ms: Couldn't connect to server
```

I'm testing with k3s here, but I originally experienced this on a GKE cluster:
```
% kubectl version
Client Version: v1.31.0
Kustomize Version: v5.4.2
Server Version: v1.30.5+k3s1
```

I think this may have been fixed in #127976 but its only in the 1.32 branch if GitHub's tag matching is correct for the commit.
It is a beta features and only happens with named ports, so it may not be backported,  if you use the port number it should work, right?

Fix patch:
diff --git a/staging/src/k8s.io/endpointslice/utils.go b/staging/src/k8s.io/endpointslice/utils.go
index 3aa35ec20c11b..2cd4e01831816 100644
--- a/staging/src/k8s.io/endpointslice/utils.go
+++ b/staging/src/k8s.io/endpointslice/utils.go
@@ -392,6 +392,17 @@ func findPort(pod *v1.Pod, svcPort *v1.ServicePort) (int, error) {
 				}
 			}
 		}
+		// also support sidecar container (initContainer with restartPolicy=Always)
+		for _, container := range pod.Spec.InitContainers {
+			if container.RestartPolicy == nil || *container.RestartPolicy != v1.ContainerRestartPolicyAlways {
+				continue
+			}
+			for _, port := range container.Ports {
+				if port.Name == name && port.Protocol == svcPort.Protocol {
+					return int(port.ContainerPort), nil
+				}
+			}
+		}
 	case intstr.Int:
 		return portName.IntValue(), nil
 	}


Test patch:
diff --git a/staging/src/k8s.io/endpointslice/utils_test.go b/staging/src/k8s.io/endpointslice/utils_test.go
index 43dfac317b41d..288c4e26fc314 100644
--- a/staging/src/k8s.io/endpointslice/utils_test.go
+++ b/staging/src/k8s.io/endpointslice/utils_test.go
@@ -515,6 +515,7 @@ func TestServiceControllerKey(t *testing.T) {
 
 func TestGetEndpointPorts(t *testing.T) {
 	protoTCP := v1.ProtocolTCP
+	restartPolicyAlways := v1.ContainerRestartPolicyAlways
 
 	testCases := map[string]struct {
 		service       *v1.Service
@@ -585,6 +586,88 @@ func TestGetEndpointPorts(t *testing.T) {
 				AppProtocol: pointer.String("https"),
 			}},
 		},
+		"service with named port for restartable init container": {
+			service: &v1.Service{
+				Spec: v1.ServiceSpec{
+					Ports: []v1.ServicePort{{
+						Name:       "http-sidecar",
+						Port:       8080,
+						TargetPort: intstr.FromInt32(8080),
+						Protocol:   protoTCP,
+					}, {
+						Name:       "http",
+						Port:       8090,
+						TargetPort: intstr.FromString("http"),
+						Protocol:   protoTCP,
+					}},
+				},
+			},
+			pod: &v1.Pod{
+				Spec: v1.PodSpec{
+					InitContainers: []v1.Container{{
+						Ports: []v1.ContainerPort{{
+							Name:          "http-sidecar",
+							ContainerPort: int32(8080),
+							Protocol:      protoTCP,
+						}},
+						RestartPolicy: &restartPolicyAlways,
+					}},
+					Containers: []v1.Container{{
+						Ports: []v1.ContainerPort{{
+							Name:          "http",
+							ContainerPort: int32(8090),
+							Protocol:      protoTCP,
+						}},
+					}},
+				},
+			},
+			expectedPorts: []*discovery.EndpointPort{{
+				Name:     pointer.String("http-sidecar"),
+				Port:     pointer.Int32(8080),
+				Protocol: &protoTCP,
+			}, {
+				Name:     pointer.String("http"),
+				Port:     pointer.Int32(8090),
+				Protocol: &protoTCP,
+			}},
+		},
+		"service with same named port for regular and restartable init container": {
+			service: &v1.Service{
+				Spec: v1.ServiceSpec{
+					Ports: []v1.ServicePort{
+						{
+							Name:       "http",
+							Port:       80,
+							TargetPort: intstr.FromString("http"),
+							Protocol:   protoTCP,
+						}},
+				},
+			},
+			pod: &v1.Pod{
+				Spec: v1.PodSpec{
+					InitContainers: []v1.Container{{
+						Ports: []v1.ContainerPort{{
+							Name:          "http",
+							ContainerPort: int32(8080),
+							Protocol:      protoTCP,
+						}},
+						RestartPolicy: &restartPolicyAlways,
+					}},
+					Containers: []v1.Container{{
+						Ports: []v1.ContainerPort{{
+							Name:          "http",
+							ContainerPort: int32(8090),
+							Protocol:      protoTCP,
+						}},
+					}},
+				},
+			},
+			expectedPorts: []*discovery.EndpointPort{{
+				Name:     pointer.String("http"),
+				Port:     pointer.Int32(8090),
+				Protocol: &protoTCP,
+			}},
+		},
 	}
 
 	for name, tc := range testCases {
diff --git a/test/e2e/network/service.go b/test/e2e/network/service.go
index e9fd2c35879d9..fc4610c2d75bd 100644
--- a/test/e2e/network/service.go
+++ b/test/e2e/network/service.go
@@ -4076,6 +4076,78 @@ var _ = common.SIGDescribe("Services", func() {
 		checkOneNodePort(hostExecPodNodeIP, true, v1.ServiceExternalTrafficPolicyLocal, deadline)
 		checkOneNodePort(thirdNodeIP, false, v1.ServiceExternalTrafficPolicyLocal, deadline)
 	})
+
+	ginkgo.It("should connect to the ports exposed by restartable init containers", func(ctx context.Context) {
+		serviceName := "sidecar-with-port"
+		ns := f.Namespace.Name
+
+		t := NewServerTest(cs, ns, serviceName)
+		defer func() {
+			defer ginkgo.GinkgoRecover()
+			errs := t.Cleanup()
+			if len(errs) != 0 {
+				framework.Failf("errors in cleanup: %v", errs)
+			}
+		}()
+
+		name := "sidecar-port"
+		port := int32(8080)
+		namedPort := "http-sidecar"
+
+		service := t.BuildServiceSpec()
+		service.Spec.Ports = []v1.ServicePort{
+			{
+				Name:       namedPort,
+				Port:       port,
+				TargetPort: intstr.FromInt(int(port)),
+			},
+		}
+		ports := []v1.ContainerPort{{Name: namedPort, ContainerPort: port, Protocol: v1.ProtocolTCP}}
+		args := []string{"netexec", fmt.Sprintf("--http-port=%d", port)}
+		createPodWithRestartableInitContainerOrFail(ctx, f, ns, name, t.Labels, ports, args...)
+
+		ginkgo.By(fmt.Sprintf("creating Service %v with selectors %v", service.Name, service.Spec.Selector))
+		service, err := t.CreateService(service)
+		framework.ExpectNoError(err)
+
+		checkServiceReachabilityFromExecPod(ctx, f.ClientSet, ns, service.Name, service.Spec.ClusterIP, port)
+	})
+
+	ginkgo.It("should connect to the named ports exposed by restartable init containers", func(ctx context.Context) {
+		serviceName := "sidecar-with-named-port"
+		ns := f.Namespace.Name
+
+		t := NewServerTest(cs, ns, serviceName)
+		defer func() {
+			defer ginkgo.GinkgoRecover()
+			errs := t.Cleanup()
+			if len(errs) != 0 {
+				framework.Failf("errors in cleanup: %v", errs)
+			}
+		}()
+
+		name := "sidecar-port"
+		port := int32(8080)
+		namedPort := "http-sidecar"
+
+		service := t.BuildServiceSpec()
+		service.Spec.Ports = []v1.ServicePort{
+			{
+				Name:       namedPort,
+				Port:       port,
+				TargetPort: intstr.FromString(namedPort),
+			},
+		}
+		ports := []v1.ContainerPort{{Name: namedPort, ContainerPort: port, Protocol: v1.ProtocolTCP}}
+		args := []string{"netexec", fmt.Sprintf("--http-port=%d", port)}
+		createPodWithRestartableInitContainerOrFail(ctx, f, ns, name, t.Labels, ports, args...)
+
+		ginkgo.By(fmt.Sprintf("creating Service %v with selectors %v", service.Name, service.Spec.Selector))
+		service, err := t.CreateService(service)
+		framework.ExpectNoError(err)
+
+		checkServiceReachabilityFromExecPod(ctx, f.ClientSet, ns, service.Name, service.Spec.ClusterIP, port)
+	})
 })
 
 // execAffinityTestForSessionAffinityTimeout is a helper function that wrap the logic of
@@ -4325,6 +4397,18 @@ func createPodOrFail(ctx context.Context, f *framework.Framework, ns, name strin
 	e2epod.NewPodClient(f).CreateSync(ctx, pod)
 }
 
+// createPodWithRestartableInitContainerOrFail creates a pod with restartable init containers using the specified containerPorts.
+func createPodWithRestartableInitContainerOrFail(ctx context.Context, f *framework.Framework, ns, name string, labels map[string]string, containerPorts []v1.ContainerPort, args ...string) {
+	ginkgo.By(fmt.Sprintf("Creating pod %s with restartable init containers in namespace %s", name, ns))
+	pod := e2epod.NewAgnhostPod(ns, name, nil, nil, nil, "pause")
+	pod.ObjectMeta.Labels = labels
+	restartPolicyAlways := v1.ContainerRestartPolicyAlways
+	init := e2epod.NewAgnhostContainer(name, nil, containerPorts, args...)
+	init.RestartPolicy = &restartPolicyAlways
+	pod.Spec.InitContainers = []v1.Container{init}
+	e2epod.NewPodClient(f).CreateSync(ctx, pod)
+}
+
 // launchHostExecPod launches a hostexec pod in the given namespace and waits
 // until it's Running. If avoidNode is non-nil, it will ensure that the pod doesn't
 // land on that node.
@@ -4375,6 +4459,30 @@ func checkReachabilityFromPod(ctx context.Context, expectToBeReachable bool, tim
 	framework.ExpectNoError(err)
 }
 
+// checkServiceReachabilityFromExecPod creates a dedicated client pod, executes into it,
+// and checks reachability to the specified target host and port.
+func checkServiceReachabilityFromExecPod(ctx context.Context, client clientset.Interface, namespace, name, clusterIP string, port int32) {
+	// We avoid relying on DNS lookup with the service name here because
+	// we only want to test whether the named port is accessible from the service.
+	serverHost := net.JoinHostPort(clusterIP, strconv.Itoa(int(port)))
+	ginkgo.By("creating a dedicated client to send request to the http server " + serverHost)
+	execPod := e2epod.CreateExecPodOrFail(ctx, client, namespace, "execpod-", nil)
+	execPodName := execPod.Name
+	cmd := fmt.Sprintf("curl -q -s --connect-timeout 2 http://%s/", serverHost)
+	var stdout string
+	if pollErr := wait.PollUntilContextTimeout(ctx, framework.Poll, e2eservice.KubeProxyLagTimeout, true, func(ctx context.Context) (bool, error) {
+		var err error
+		stdout, err = e2eoutput.RunHostCmd(namespace, execPodName, cmd)
+		if err != nil {
+			framework.Logf("error trying to connect to service %s: %v, ... retrying", name, err)
+			return false, nil
+		}
+		return true, nil
+	}); pollErr != nil {
+		framework.Failf("connection to the Service %v within %v should be succeeded, stdout: %v", name, e2eservice.KubeProxyLagTimeout, stdout)
+	}
+}
+
 func validatePorts(ep, expectedEndpoints portsByPodUID) error {
 	if len(ep) != len(expectedEndpoints) {
 		// should not happen because we check this condition before