Repository: traefik/traefik (language: go)

Issue:
TCP+TLS connection kill in v3.0.0
### Welcome!

- [X] Yes, I've searched similar issues on [GitHub](https://github.com/traefik/traefik/issues) and didn't find any.
- [X] Yes, I've searched similar issues on the [Traefik community forum](https://community.traefik.io) and didn't find any.

### What did you do?

I connect to PostgreSQL service via Traefik (terminates TLS connection to PostgreSQL service in k3s) and run
```
SELECT pg_sleep_for('3min');
```

### What did you see instead?

Traefik kills the connection to the service in about ~1min. None of the settings helped. Downgrading to 3.0.0-rc3 fixes the issue.

### What version of Traefik are you using?

3.0.0-rc5

### What is your environment & configuration?

```yaml
    image:
      tag: "v3.0.0-rc5"
    resources:
      requests:
        cpu: 15m
        memory: 64Mi
      limits:
        memory: 192Mi
    api:
      insecure: true
    logs:
      general:
        level: DEBUG
        format: json
      access:
        enabled: true
        format: json
    experimental:
      http3: true
    additionalArguments:
      - "--ping"
      - "--serversTransport.forwardingTimeouts.responseHeaderTimeout=10m"
      - "--serversTransport.forwardingTimeouts.dialTimeout=10m"
      - "--serversTransport.forwardingTimeouts.idleConnTimeout=30m"
      - "--serversTransport.maxIdleConnsPerHost=20"
    entryPoints:
      amqp:
        address: ":5672"
      psql:
        address: ":5432"
        transport:
          respondingTimeouts:
            idleTimeout: "30m"
      metrics:
        address: ":8082"
      traefik:
        address: ":9000"
      web:
        address: ":8000"
        http:
          redirections:
            entryPoint:
              to: websecure
              scheme: https
              permanent: true
      websecure:
        address: ":443"
        http3:
          enabled: true
          advertisedPort: 4443
    ingressClass:
      enabled: true
      isDefaultClass: true
    ingressRoute:
      dashboard:
        enabled: true
    deployment:
      kind: DaemonSet
      terminationGracePeriodSeconds: 200
      lifecycle:
        preStop:
          exec:
            command: [ "/bin/sh", "-c", "sleep 180" ]
    metrics:
      prometheus:
        addRoutersLabels: true
        addServicesLabels: true
        entryPoint: metrics
    persistence:
      enabled: false
    ports:
      amqp:
        expose:
          default: true
        exposedPort: 5672
        port: 5672
      psql:
        expose:
          default: true
        exposedPort: 5432
        port: 5432
      metrics:
        expose:
          default: true
        exposedPort: 8082
        port: 8082
      traefik:
        expose:
          default: false
        exposedPort: 9000
        port: 9000
        protocol: TCP
      web:
        expose:
          default: true
        redirectTo:
          port: websecure
      websecure:
        expose:
          default: true
        exposedPort: 443
        http3:
          advertisedPort: 4443
          enabled: true
        port: 443
    providers:
      kubernetesCRD:
        allowCrossNamespace: true
        allowEmptyServices: true
        allowExternalNameServices: true
        enabled: true
        namespaces: []
      kubernetesIngress:
        allowEmptyServices: true
        allowExternalNameServices: true
        enabled: true
        namespaces: []
        publishedService:
          enabled: false
    securityContext:
      capabilities:
        add:
          - NET_BIND_SERVICE
        drop:
          - ALL
      readOnlyRootFilesystem: true
      runAsGroup: 0
      runAsNonRoot: false
      runAsUser: 0
    service:
      annotations:
        service.beta.kubernetes.io/external-traffic: OnlyLocal
      spec:
        externalTrafficPolicy: Local
    tlsStore:
      default:
        defaultCertificate:
          secretName: my-tls

```

Add more configuration information here.


### If applicable, please paste the log output in DEBUG level

```
	2024-04-15 23:16:47.433	
{"level":"error","error":"readfrom tcp 10.42.4.56:59846->10.42.1.199:5432: read tcp 10.42.4.56:5432->10.65.250.102:37460: i/o timeout","time":"2024-04-15T20:16:47Z","caller":"github.com/traefik/traefik/v3/pkg/tcp/proxy.go:75","message":"Error while handling TCP connection"}
	
	
2024-04-15 23:16:47.433	
{"level":"debug","error":"set tcp 10.42.4.56:5432: use of closed network connection","time":"2024-04-15T20:16:47Z","caller":"github.com/traefik/traefik/v3/pkg/tcp/proxy.go:113","message":"Error while setting TCP connection deadline"}
```

```
$ time psql "sslmode=require host=mydb.internal.host dbname=mydb user=myuser keepalives=1 keepalives_idle=30 keepalives_interval=10 keepalives_count=10" -c "SELECT pg_sleep_for('5min');"
Password for user myuser: 
SSL connection has been closed unexpectedly
connection to server was lost

real  1m1.318s
```


Discussion / hints:
Hello,

it's because we have the default timeout: https://doc.traefik.io/traefik/migration/v2/#entrypointtransportrespondingtimeoutsreadtimeout

You can find community suggestions about the value to use in your context: https://github.com/traefik/traefik/wiki/respondingTimeouts-for-applications
> Hello,
> 
> it's because we have the default timeout: https://doc.traefik.io/traefik/migration/v2/#entrypointtransportrespondingtimeoutsreadtimeout
> 
> You can find community suggestions about the value to use in your context: https://github.com/traefik/traefik/wiki/respondingTimeouts-for-applications

nope, it still fails in the same ~1min with
```
SSL connection has been closed unexpectedly
The connection to the server was lost. Attempting reset: Succeeded.
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_128_GCM_SHA256, compression: off)
```

even with:
```
    entryPoints:
      psql:
        address: ":5432"
        transport:
          respondingTimeouts:
            readTimeout: "0"
```
This timeout issue is present already in v3.0.0-rc4 version.
Hello @kamikaze @jur3,

Depending on the routing configuration you have, this behavior could have been fixed by PR #10615.
This fix has not been released yet, but can you please try the experimental images `traefik/traefik:experimental-v3.0` in the mean time?
Unable to install it via Helm chart:

```
Helm install failed for release traefik-v3/traefik with chart traefik@27.0.2: template: traefik/templates/rbac/clusterrole.yaml:59:13: executing "traefik/templates/rbac/clusterrole.yaml" at <semverCompare "<3.0.0-0" (include "imageVersion" $)>: error calling semverCompare: Invalid Semantic Version
```
Hello @mloiseleur,

> Unable to install it via Helm chart:

It looks like a limitation, WDYT of adding support for our experimental images to the Helm chart?
@rtribotte Sure. Would you please let me know what you think of https://github.com/traefik/traefik-helm-chart/pull/1045 ?

BTW, wdyt of following semver for this _experimental_ version ? It would mean produce an image with tag `v3.0-experimental`.
@mloiseleur I agree, if there are no blockers, adopting Semver would allow supporting any experimental tag by default.
But `v3.0-experimental` it's not a [semver](https://semver.org/) version: the 3 numbers (`vX.Y.Z-xxx`) are required with [semver](https://semver.org/).

The `experimental-v3.0` is a floating tag, like `latest` or `v3.0`, there is no relation with a specific version or [semver](https://semver.org/).

I think we should not change the tag from `experimental-v3.0` to `v3.0-experimental`.
@ldez I agree with you: it's not a _complete_ semver version. Nevertheless, it's good enough to be supported by [semverCompare](https://helm.sh/docs/chart_template_guide/function_list/#semvercompare), see https://github.com/traefik/traefik-helm-chart/pull/1046
>  it's not a complete semver version

It's semver or it's not, there is no partial semver.

semverCompare uses a specific syntax `~` (from npm) to check only minor and major, but this doesn't mean that we should abuse this behavior to allow non-semver versions.

`experimental-v3.0` is a floating tag, so there is no need for comparison or being semver-ish.



Yes, there is no partial semver. 

Wondering why it's supported, I am discovering that this question has been heavily discussed : https://github.com/semver/semver/issues/237
They ended by not accepting version without patch as a valid semver *and* allows compare parsing, see https://github.com/semver/semver/issues/237#issuecomment-1630991559

Anyway, it's up to Traefik maintainers to decide how this tag should be managed. 

What I can say is that, from the Chart perspective, the v28 does not support anymore Traefik Proxy v2 specifics. This `semverCompare` call is useful to warn users who missed this major requirement.
Hello @kamikaze,

As https://github.com/traefik/traefik-helm-chart/pull/1045 is merged, did you test it? Is it fixed by https://github.com/traefik/traefik/pull/10615?
> Hello @kamikaze,
> 
> As [traefik/traefik-helm-chart#1045](https://github.com/traefik/traefik-helm-chart/pull/1045) is merged, did you test it? Is it fixed by #10615?

I have installed traefik v3.0.0 with 28.0.0-rc.1 helm chart.

```
SELECT pg_sleep_for('5min');
```

still fails with:

```
=> SELECT pg_sleep_for('5min');
SSL connection has been closed unexpectedly
The connection to the server was lost. Attempting reset: Succeeded.
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_128_GCM_SHA256, compression: off)
=>
```

```
	2024-04-30 09:04:56.373	
{"level":"error","error":"readfrom tcp 10.42.0.44:56596->10.12.0.71:5432: read tcp 10.42.0.44:5432->10.65.250.102:42692: i/o timeout","time":"2024-04-30T06:04:56Z","message":"Error while handling TCP connection"}
	
	
2024-04-30 09:05:56.442	
{"level":"error","error":"readfrom tcp 10.42.0.44:46888->10.12.0.71:5432: read tcp 10.42.0.44:5432->10.65.250.102:35248: i/o timeout","time":"2024-04-30T06:05:56Z","message":"Error while handling TCP connection"}
```
btw, traefik terminates TLS in my case, so I'm not sure it is related to "non-TLS" change in that PR

Fix patch:
diff --git a/pkg/server/router/tcp/router.go b/pkg/server/router/tcp/router.go
index b4b7384d9b..89096c7cac 100644
--- a/pkg/server/router/tcp/router.go
+++ b/pkg/server/router/tcp/router.go
@@ -123,6 +123,11 @@ func (r *Router) ServeTCP(conn tcp.WriteCloser) {
 	}
 
 	if postgres {
+		// Remove read/write deadline and delegate this to underlying TCP server.
+		if err := conn.SetDeadline(time.Time{}); err != nil {
+			log.Error().Err(err).Msg("Error while setting deadline")
+		}
+
 		r.servePostgres(r.GetConn(conn, getPeeked(br)))
 		return
 	}


Test patch: