Repository: traefik/traefik (language: go)

Issue:
Non-TLS content on HTTPS endpoint for AWS NLB with Proxy protocol v2
### Welcome!

- [X] Yes, I've searched similar issues on [GitHub](https://github.com/traefik/traefik/issues) and didn't find any.
- [X] Yes, I've searched similar issues on the [Traefik community forum](https://community.traefik.io) and didn't find any.

### What did you do?

Every time our Traefik instance reaches a higher number (1k>) of connections, it starts to return non-TLS content on the HTTPS endpoint.

Once Proxy protocol v2 is disabled, the problem disappears.

https://community.traefik.io/t/non-tls-content-on-https-endpoint/19330/2

### What did you see instead?

Body contains:

```
HTTP/1.1 400 Bad Request
Content-Type: text/plain; charset=utf-8
Connection: close
400 Bad Request
```

### What version of Traefik are you using?

 * Traefik docker image version: [docker.io/traefik:v2.10.1](http://docker.io/traefik:v2.10.1)
 * Helm chart version: v23.1.0
 
 I tried versions 20.8.0, 21.2.1, 22.0.0, and 22.3.0 charts with the same result.

Our setup is running as NodePort behind AWS NLB.

We use arm instances, but the issue also happens for the amd64 architecture.


### What is your environment & configuration?

We tried following arguments:

```yaml
 {
    name: 'service.type',
    value: 'NodePort',
  },
  {
    name: 'service.spec.externalTrafficPolicy',
    value: 'Local',
  },
  {
    name: 'service.spec.internalTrafficPolicy',
    value: 'Local',
  },
  {
    name: 'ingressClass.isDefaultClass',
    value: 'false',
  },
  {
    name: 'ports.web.nodePort',
    value: '30080',
  },
  {
    name: 'ports.web.proxyProtocol.trustedIPs',
    value: '10.100.0.0/16',
  },
  {
    name: 'ports.web.middlewares[0]',
    value: 'traefik-default@kubernetescrd',
  },
  {
    name: 'ports.websecure.nodePort',
    value: '30443',
  },
  {
    name: 'ports.websecure.proxyProtocol.trustedIPs',
    value: '10.100.0.0/16',
  },
  {
    name: 'ports.metrics.expose',
    value: 'true',
  },
  {
    name: 'ports.metrics.nodePort',
    value: '30100',
  },
  {
    name: 'ports.traefik.healthchecksPort',
    value: '8000',
  },
  {
    name: 'providers.kubernetesIngress.allowEmptyServices',
    value: 'true',
  },
  {
    name: 'providers.kubernetesIngress.allowExternalNameServices',
    value: 'true',
  },
  {
    name: 'additionalArguments[0]',
    value: '--providers.kubernetesingress.ingressendpoint.ip=127.0.0.1',
  },
  {
    name: 'additionalArguments[1]',
    value: '--serversTransport.insecureSkipVerify=true',
  },
  {
    name: 'additionalArguments[2]',
    value: '--serversTransport.maxIdleConnsPerHost=1000',
  },
  {
    name: 'additionalArguments[3]',
    value: '--entryPoints.web.transport.respondingTimeouts.readTimeout=300',
  },
  {
    name: 'additionalArguments[4]',
    value: '--entryPoints.web.transport.respondingTimeouts.writeTimeout=300',
  },
  {
    name: 'additionalArguments[5]',
    value: '--entryPoints.web.transport.respondingTimeouts.idleTimeout=300',
  },
  {
    name: 'additionalArguments[6]',
    value: '--entryPoints.web.transport.lifeCycle.graceTimeOut=60',
  },
  {
    name: 'additionalArguments[7]',
    value: '--entryPoints.web.http2.maxConcurrentStreams=2000',
  },
  {
    name: 'additionalArguments[8]',
    value: '--entryPoints.websecure.transport.respondingTimeouts.readTimeout=300',
  },
  {
    name: 'additionalArguments[9]',
    value: '--entryPoints.websecure.transport.respondingTimeouts.writeTimeout=300',
  },
  {
    name: 'additionalArguments[10]',
    value: '--entryPoints.websecure.transport.respondingTimeouts.idleTimeout=300',
  },
  {
    name: 'additionalArguments[11]',
    value: '--entryPoints.websecure.transport.lifeCycle.graceTimeOut=60',
  },
  {
    name: 'additionalArguments[12]',
    value: '--entryPoints.websecure.http2.maxConcurrentStreams=2000',
  },
```

To be sure there are no cross-wires, I disabled metrics and web endpoint, leaving only websecure entrypoint. The result was the same.

The issue happens regardless of the number of replicas.

### If applicable, please paste the log output in DEBUG level

There was nothing in the logs related to the issue.


Discussion / hints:
Hello @beranm14,

Thanks for reporting this!

Can you please share the full debug and access logs with us at the moment when the problem occurs?
We are experiencing the same issue. The problem is even on debug, there are no logs that indicate anything is wrong.  The only way I was able to determine there was an issue was to use Wireshark and packet capture the handshake. We also removed proxyprotocol v2 and the issue has stopped occurring. 

If you want I can provide a sanitized pcap where we see the handshake error. 
Hi @rtribotte, sry for the late reply

As @jmturwy wrote, there is nothing which would indicate any issue. I can send the log sample once anonymized.

Thanks
This looks like the same error we have in our setup that it's driving us crazy. We have exactly the same setup, and randomly when we run several maven build at the same time we get this:

```
[ERROR] Plugin net.alchim31.maven:scala-maven-plugin:4.4.0 or one of its dependencies could not be resolved: Failed to read artifact descriptor for net.alchim31.maven:scala-maven-plugin:jar:4.4.0: Could not transfer artifact net.alchim31.maven:scala-maven-plugin:pom:4.4.0 from/to xxxx (https://xxxxx/repository/mvn): transfer failed for https://xxxxxx/repository/mvn/net/alchim31/maven/scala-maven-plugin/4.4.0/scala-maven-plugin-4.4.0.pom: Unsupported or unrecognized SSL message -> [Help 1]
```

On python builds, we get something more descriptive like:
```
requests.exceptions.SSLError: HTTPSConnectionPool(host='xxxxxx', port=443): Max retries exceeded with url: /api/v1/bom (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:992)')))
```


We use letsencrypt to serve the certificates.

There is no error log in traefik nor nexus, just looks like it's sending http to https.
Hello,

@beranm14 I tried to reproduce the issue with no success. Can you gather everything to set up a minimal reproducible case and share it with us? (Traefik static configuration, dynamic configuration, service deployment, and the client to use to make requests).

@adriannieto-attechnest Thanks for reporting this. It is unclear wether you are experiencing the same issue as @beranm14. Can you also share with us as well more information and a minimal reproducible case?
Hello,

I have a minimal reproducible example.
prerequisite:
- an EKS cluster with [aws load balancer](https://kubernetes-sigs.github.io/aws-load-balancer-controller) installed

Manifest to deploy traefik + a minimal client : https://gist.githubusercontent.com/jb-abbadie/c2038d0ad9cd4918accfb943abca9e1e/raw/f420224c1f333a15f80ccde619808bc291f134b5/all.yaml

Once deploy you either need to change `podinfo.example.com` to another fqdn you manage or edit `/etc/hosts` in order to override dns to point to AWS NLB.

Once its deployed you can easily reproduce with [k6](https://k6.io/docs/) with a script like this one

```js
import http from 'k6/http';

import { sleep } from 'k6';


export default function () {
  http.get('https://podinfo.example.com/healthz');
  sleep(1);
}
```

and invoke k6 with this command 
```k6 run --insecure-skip-tls-verify --vus 2000 --duration 1s test.js```


Running this command will yield a lot of error like this one :
```WARN[0000] Request Failed                                error="Get \"https://podinfo.example.com/healthz\": http: server gave HTTP response to HTTPS client"
WARN[0000] Request Failed                                error="Get \"https://podinfo.example.com/healthz\": http: server gave HTTP response to HTTPS client"
WARN[0000] Request Failed                                error="Get \"https://podinfo.example.com/healthz\": http: server gave HTTP response to HTTPS client"
WARN[0000] Request Failed                                error="Get \"https://podinfo.example.com/healthz\": http: server gave HTTP response to HTTPS client"
WARN[0000] Request Failed                                error="Get \"https://podinfo.example.com/healthz\": http: server gave HTTP response to HTTPS client"
WARN[0000] Request Failed                                error="Get \"https://podinfo.example.com/healthz\": http: server gave HTTP response to HTTPS client"
WARN[0000] Request Failed                                error="Get \"https://podinfo.example.com/healthz\": http: server gave HTTP response to HTTPS client"
WARN[0000] Request Failed                                error="Get \"https://podinfo.example.com/healthz\": http: server gave HTTP response to HTTPS client"
WARN[0000] Request Failed                                error="Get \"https://podinfo.example.com/healthz\": http: server gave HTTP response to HTTPS client"
WARN[0000] Request Failed                                error="Get \"https://podinfo.example.com/healthz\": http: server gave HTTP response to HTTPS client"
WARN[0000] Request Failed                                error="Get \"https://podinfo.example.com/healthz\": http: server gave HTTP response to HTTPS client"
```


Thanks in advance for the investigation. If you need any other details in order to reproduce or collect additional information feel free to respond to this issue.
Hello,
We are facing exactly the same bug in the same configuration: AWS load balancer, proxy protocol v2, eks. 
For us, that happens with more than 400 https requests in one minute. 

Some details: 
`helm.sh/chart=traefik-24.0.0 `
`Image:       docker.io/traefik:v2.10.4`

Do you have any updates to share?

Regards
Same problems with `v2.10.7` and the `v3.0.0-beta5`

Fix patch:
diff --git a/pkg/server/server_entrypoint_tcp.go b/pkg/server/server_entrypoint_tcp.go
index e3e22998e4..26b4172932 100644
--- a/pkg/server/server_entrypoint_tcp.go
+++ b/pkg/server/server_entrypoint_tcp.go
@@ -418,7 +418,12 @@ func (ln tcpKeepAliveListener) Accept() (net.Conn, error) {
 }
 
 func buildProxyProtocolListener(ctx context.Context, entryPoint *static.EntryPoint, listener net.Listener) (net.Listener, error) {
-	proxyListener := &proxyproto.Listener{Listener: listener}
+	timeout := entryPoint.Transport.RespondingTimeouts.ReadTimeout
+	// proxyproto use 200ms if ReadHeaderTimeout is set to 0 and not no timeout
+	if timeout == 0 {
+		timeout = -1
+	}
+	proxyListener := &proxyproto.Listener{Listener: listener, ReadHeaderTimeout: time.Duration(timeout)}
 
 	if entryPoint.ProxyProtocol.Insecure {
 		log.FromContext(ctx).Infof("Enabling ProxyProtocol without trusted IPs: Insecure")


Test patch: