Repository: kubernetes/kubernetes (language: go)

Issue:
Kubectl exec disconnects automatically after 5m post upgrading the k8s cluster to 1.30
### What happened?

We have several automated scripts that run kubectl commands to exec into the pods and execute some custom scripts scripts. We observed that on all clusters running version 1.30.x, the session automatically gets disconnected without any error message, which was not the case in versions lower than 1.30.

### What did you expect to happen?

Session should not terminate until we disconnect or exit from the pod 

### How can we reproduce it (as minimally and precisely as possible)?

Just run a script which does kubectl exec into the pod and you will see that it gets disconnected with in 5 min

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
# paste output here
```
Client Version: v1.30.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.3

</details>


### Cloud provider

<details>
Azure
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>



Discussion / hints:
/kind support
/sig cli

Hi @isaacdan, it looks interesting, could you share your scripts or how to reproduce this issue?
/remove-kind support
Do you have a proxy?. I'm thinking you have been affected by https://github.com/kubernetes/kubernetes/pull/126253 which was fixed in 1.30.4
/cc @seans3 
/triage accepted

Hi,

I don't have any proxy. The issue is if I simply connect to kubectl exec
-it and if the session is idle it kicks me out after a few min.

This was not the case for the versions below 1.30

Thanks,

On Fri, Oct 25, 2024 at 9:57 AM Arda Güçlü ***@***.***> wrote:

> /remove-kind support
> Do you have a proxy?. I'm thinking you have been affected by #126253
> <https://github.com/kubernetes/kubernetes/pull/126253> which was fixed in
> 1.30.4
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/kubernetes/kubernetes/issues/128314#issuecomment-2436799606>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AMSTN3KVWDAZ4VBFDAD44MTZ5HCETAVCNFSM6AAAAABQQ6WGTCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZWG44TSNRQGY>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>

Hi Ruyi,

If I connect to kubectl exec -it and keep the session idle it still kicks
me out after a few min.

Thanks,


On Fri, Oct 25, 2024 at 8:26 AM Ruyi Guo ***@***.***> wrote:

> Hi @isaacdan <https://github.com/isaacdan>, it looks interesting, could
> you share your scripts or how to reproduce this issue?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/kubernetes/kubernetes/issues/128314#issuecomment-2436707293>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AMSTN3PHMFMLDKOGJJ3OOTDZ5GXOZAVCNFSM6AAAAABQQ6WGTCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZWG4YDOMRZGM>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>

Can you try 1.30.4 to be sure?
Also, can you fill in more details like install tools? It's important to understand the network environment etc.
/kind support
/remove-triage accepted
I have encountered the same behavior. Downgrading client < 1.30.0  seems to fix this
After further experimentation setting `KUBECTL_REMOTE_COMMAND_WEBSOCKETS=false` fixes this without needing to downgrade the client
is the server version 1.30? can you use kubectl -v7 and try to get the output around the time it gets disconnected?
❯ kubectl version
```
Client Version: v1.30.5
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.5
```
Logs right before `kubectl -v7 exec -- bash` starts
```

                                                                               I0116 22:54:37.997277   21684 round_trippers.go:469] Request Headers:
                                                                                                                                                    I0116 22:54:37.997288   21684 round_trippers.go:473]     Sec-Websocket-Protocol: v5.channel.k8s.io
                                   I0116 22:54:37.997294   21684 round_trippers.go:473]     User-Agent: kubectl/v1.30.5 (darwin/arm64) kubernetes/74e84a9
                                                                                                                                                         I0116 22:54:38.497047   21684 round_trippers.go:574] Response Status: 101 Switching Protocols in 499 milliseconds
                                                       I0116 22:54:38.497120   21684 websocket.go:133] The subprotocol is v5.channel.k8s.io
                                                                                                                                           I0116 22:54:38.497568   21684 websocket.go:357] Write() on stream 4
                                                                                                    
```

Logs after idle timeout
```
I0116 22:59:39.647516   21684 websocket.go:395] Close() on stream 0
I0116 22:59:39.647587   21684 websocket.go:405] Close() done on stream 0
```
Interesting, the websockets send a ping periodically https://github.com/kubernetes/kubernetes/commit/b04d1177efb0cfc84b76732848eef8f0ea89b25b and I don't see any constant with 5m, the fact you mention 5m seems to point to some place that has that timeout.

 
I tried this with increased debug level and saw that the ping/pong is working.
```
I0116 23:05:43.928340   22710 websocket.go:497] Websocket Ping succeeeded
I0116 23:05:44.072934   22710 websocket.go:455] Pong message received ()--resetting read deadline
```
It could be cloud provider related, we are also using azure. One of the things I tried was increasing cluster Load balancer idle timeout to 10m, but it did not have any effect.

> 
> It could be cloud provider related, we are also using azure. One of the things I tried was increasing cluster Load balancer idle timeout to 10m, but it did not have any effect.

runtimes have an option, stream_idle_timeout is the name in containerd, that close idle sessions, however, the SPDY implementation has a bug and consider the ping frames part of the session, making that option useless

https://github.com/kubernetes/kubernetes/issues/102569
https://github.com/containerd/containerd/issues/5563

can this be related? with the change in SPDY we no longer send pings from the client, can you check the runtime config to see if you find this 5 mins magic value on that option?
>can this be related? with the change in SPDY we no longer send pings from the client, can you check the runtime config to see if you find this 5 mins magic value on that option?



I will try to check. During the investigation I only looked at `xstreaming-connection-idle-timeout` and it is set to `"4h0m0s"` 
We are also encountering this issue but only for our 1.30 AKS clusters . Could not reproduce on AKS 1.29, or our GKE & EKS 1.30 clusters. Initially suspected the AKS LoadBalancer idle timeout but it is set to 30 minutes.

kubectl version
```
Client Version: v1.31.1
Kustomize Version: v5.4.2
Server Version: v1.30.6
```

Used the following to reproduce
```
kubectl run nginx --image=nginx
```

idle exec exits after 5 minutes
```
time kubectl exec -it nginx -- bash
root@nginx:/# kubectl exec -it nginx -- bash
 0.15s user 0.06s system 0% cpu 5:02.78 total
```

Tried your workaround by setting the following and the idle exec commands were not killed after 5 minutes
```
KUBECTL_REMOTE_COMMAND_WEBSOCKETS=false
```

Eventually I intentionally exited the exec after 20 minutes
```
time kubectl exec -it nginx -- bash
root@nginx:/# exit
exit
kubectl exec -it nginx -- bash  0.27s user 0.17s system 0% cpu 20:14.73 total

```


Fix patch:
diff --git a/staging/src/k8s.io/apiserver/pkg/util/proxy/streamtranslator.go b/staging/src/k8s.io/apiserver/pkg/util/proxy/streamtranslator.go
index 6dabc1c7b4ab7..6593a3ed96676 100644
--- a/staging/src/k8s.io/apiserver/pkg/util/proxy/streamtranslator.go
+++ b/staging/src/k8s.io/apiserver/pkg/util/proxy/streamtranslator.go
@@ -21,6 +21,7 @@ import (
 	"net/http"
 	"net/url"
 	"strconv"
+	"time"
 
 	"github.com/mxk/go-flowrate/flowrate"
 	apierrors "k8s.io/apimachinery/pkg/api/errors"
@@ -70,7 +71,7 @@ func (h *StreamTranslatorHandler) ServeHTTP(w http.ResponseWriter, req *http.Req
 	defer websocketStreams.conn.Close()
 
 	// Creating SPDY executor, ensuring redirects are not followed.
-	spdyRoundTripper, err := spdy.NewRoundTripperWithConfig(spdy.RoundTripperConfig{UpgradeTransport: h.Transport})
+	spdyRoundTripper, err := spdy.NewRoundTripperWithConfig(spdy.RoundTripperConfig{UpgradeTransport: h.Transport, PingPeriod: 5 * time.Second})
 	if err != nil {
 		websocketStreams.writeStatus(apierrors.NewInternalError(err)) //nolint:errcheck
 		metrics.IncStreamTranslatorRequest(req.Context(), strconv.Itoa(http.StatusInternalServerError))


Test patch:
diff --git a/test/e2e/kubectl/kubectl.go b/test/e2e/kubectl/kubectl.go
index 6adb6fb219600..5592a5cd2b216 100644
--- a/test/e2e/kubectl/kubectl.go
+++ b/test/e2e/kubectl/kubectl.go
@@ -500,6 +500,16 @@ var _ = SIGDescribe("Kubectl client", func() {
 			}
 		})
 
+		// https://issues.k8s.io/128314
+		f.It(f.WithSlow(), "should support exec idle connections", func(ctx context.Context) {
+			ginkgo.By("executing a command in the container")
+
+			execOutput := e2ekubectl.RunKubectlOrDie(ns, "exec", podRunningTimeoutArg, simplePodName, "--", "/bin/sh", "-c", "sleep 320 && echo running in container")
+			if expected, got := "running in container", strings.TrimSpace(execOutput); expected != got {
+				framework.Failf("Unexpected kubectl exec output. Wanted %q, got %q", expected, got)
+			}
+		})
+
 		ginkgo.It("should support exec through kubectl proxy", func(ctx context.Context) {
 			_ = getTestContextHost()