Repository: traefik/traefik (language: go)

Issue:
Error while Peeking first byte ; haproxy in front of Traefik with proxy-protocol
### Do you want to request a *feature* or report a *bug*?

Bug

### What did you do?

<!--

HOW TO WRITE A GOOD BUG REPORT?

- Respect the issue template as much as possible.
- The title should be short and descriptive.
- Explain the conditions which led you to report this issue: the context.
- The context should lead to something, an idea or a problem that you’re facing.
- Remain clear and concise.
- Format your messages to help the reader focus on what matters and understand the structure of your message, use Markdown syntax https://help.github.com/articles/github-flavored-markdown

-->

I have haproxy in front of **dockerized** Traefik in **swarm-mode** to load-balance requests to servers in a private network, with proxy-protocol to identify source IP addresses.
I perform tcp-checks for each entrypoint I use, but these result in the following errors in Traefik : `"Error while Peeking first byte: read tcp 192.168.224.3:110->10.1.1.1:52524: read: connection reset by peer"`

### What did you expect to see?

I do not expect to see an error due to haproxy tcp-checks.

### What did you see instead?

```bash
proxy_app.0.y75ghkvn81b7@m10    | time="2020-08-05T13:53:06Z" level=debug msg="Error while Peeking first byte: read tcp 192.168.224.3:110->10.1.1.1:52524: read: connection reset by peer"
```

### Output of `traefik version`: (_What version of Traefik are you using?_)

```
Version:      2.2.8
Codename:     chevrotin
Go version:   go1.14.6
Built:        2020-07-28T15:46:03Z
OS/Arch:      linux/amd64
```

### What is your environment & configuration (arguments, toml, provider, platform, ...)?

haproxy node is 10.1.1.1

Traefik node is 10.1.1.10

*Static conf*

```yml
entryPoints:
  ## POP3
  0110_pop3:
    address: ":110"
    proxyProtocol:
      trustedIPs:
      - "10.0.0.0/8"
    forwardedHeaders:
      trustedIPs:
      - "10.0.0.0/8"

providers:
  file:
    watch: true
    directory: /etc/traefik/dynanmic
  docker:
    endpoint: "tcp://socket:2375"
    swarmMode: true
    exposedByDefault: false
    network: public

api:
  dashboard: true
  insecure: false # default: false
  debug: false # default: false

certificatesResolvers:
  le:
    acme:
      email: xxx
      storage: /acme.json
      dnsChallenge:
        provider: xxx
        delayBeforeCheck: 10

log:
  level: DEBUG
```

Port 110 is exposed in host mode in the `docker-compose.yml` file:

```yml
## POP3
      - target: 110
        published: 110
        protocol: tcp
        mode: host
```

*haproxy conf*

```haproxy
global
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
	stats timeout 30s
	user haproxy
	group haproxy
	daemon

	# See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        # modern configuration
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tlsv12 no-tls-tickets

        ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tlsv12 no-tls-tickets

defaults
	log	global
	mode	http
	option	httplog

        timeout connect 1s
        timeout client 20s
        timeout server 20s
        timeout client-fin 20s
        timeout tunnel 1h

	errorfile 400 /etc/haproxy/errors/400.http
	errorfile 403 /etc/haproxy/errors/403.http
	errorfile 408 /etc/haproxy/errors/408.http
	errorfile 500 /etc/haproxy/errors/500.http
	errorfile 502 /etc/haproxy/errors/502.http
	errorfile 503 /etc/haproxy/errors/503.http
	errorfile 504 /etc/haproxy/errors/504.http

listen stats
        bind :32700
        stats enable
        stats uri /
        stats hide-version
        stats auth haproxy:password
        stats refresh 1s

## POP3
frontend 0110
        bind *:110
        mode tcp
        option tcplog
        default_backend 0110

backend 0110
        balance roundrobin
        mode tcp
        option redispatch
        option tcp-check
        tcp-check connect send-proxy

        server m10 10.1.1.10:110 check send-proxy-v2 fall 3 rise 2
```

### If applicable, please paste the log output in DEBUG level (`--log.level=DEBUG` switch)

```
proxy_app.0.y75ghkvn81b7@m10    | time="2020-08-05T13:53:06Z" level=debug msg="Trusted connection" entryPointName=0110_pop3
proxy_app.0.y75ghkvn81b7@m10    | time="2020-08-05T13:53:06Z" level=debug msg="Addresses len: 12" entryPointName=0110_pop3
proxy_app.0.y75ghkvn81b7@m10    | time="2020-08-05T13:53:06Z" level=debug msg="Address readed: 12" entryPointName=0110_pop3
proxy_app.0.y75ghkvn81b7@m10    | time="2020-08-05T13:53:06Z" level=debug msg="Use header remote addr" entryPointName=0110_pop3
proxy_app.0.y75ghkvn81b7@m10    | time="2020-08-05T13:53:06Z" level=debug msg="Header parsed &{10.1.1.1:52524 10.1.1.10:110}" entryPointName=0110_pop3
proxy_app.0.y75ghkvn81b7@m10    | time="2020-08-05T13:53:06Z" level=debug msg="Error while Peeking first byte: read tcp 192.168.224.3:110->10.1.1.1:52524: read: connection reset by peer"
```



Discussion / hints:
I have seen #5297 and #5652.
The [community answer](https://community.containo.us/t/error-while-peeking-first-byte-connection-reset-by-peer/1918) from the topic suggested in #5652 is not very useful and I believe I should use the tcp-check instead of the httpchk for TCP entrypoints, so the answer from #5297 doesn't apply.
I came here because I was faced with the same issue after upgrading from Traefik 3.1.4 to 3.1.5. In our case, it's the TCP health check from a load balancer in the Hetzner cloud every 15 seconds. Over time, this results in rather large, annoying logs. Is there a way to filter this useless logging messages or avoid them altogether?
Same issue here with HAProxy 3 LTS and Traefik 3.1.6
Our logs are flooded, too, in Traefik 3.1.6. Caused by our load balancer´s health checks...

`github.com/traefik/traefik/v3/pkg/server/router/tcp/postgres.go:32`

Error while Peeking first byte
Hello @Philluxx,

This error is not directly related to `github.com/traefik/traefik/v3/pkg/server/router/tcp/postgres.go:32`, before the support for Postgres StartTLS protocol, this error would have been raised here: https://github.com/traefik/traefik/blob/e8ff825ed272669604fc6c9ad2475f5019992f07/pkg/server/router/tcp/router.go#L367

Traefik expects to read the first bytes of the connection to determinate its nature, if an error occur within the timeout period, as for instance, the peer has closed the connection on its end, then the first bytes read operation fails with the error `connection reset by peer`.

I'm not sure Traefik is in cause in this issue, and how it should operated differently, but in any case, if any community member can help us find verified steps to reproduce and fix the issue if possible, we would love the help.
@rtribotte Load balancers will often check the liveness of a target by opening a connection and immediately closing it when the target connects. IMHO, Traefik should just ignore this special case and not log anything, or at the DEBUG level if anybody is interested.

That way, the false positive warnings which flood the logs would disappear. For now, this is a reason not to upgrade beyond 3.1.4 until that issue is fixed.
Hello @christian-schlichtherle,

Thanks for your feedback, I think you're right, I'll raise the issue at the next meeting with other Traefik maintainers and let you know if we're inclined to remove or change the level of these logs.

Fix patch:
diff --git a/pkg/server/router/tcp/router.go b/pkg/server/router/tcp/router.go
index 4790cd7516..f8d6572651 100644
--- a/pkg/server/router/tcp/router.go
+++ b/pkg/server/router/tcp/router.go
@@ -350,7 +350,7 @@ func clientHelloInfo(br *bufio.Reader) (*clientHello, error) {
 	if err != nil {
 		var opErr *net.OpError
 		if !errors.Is(err, io.EOF) && (!errors.As(err, &opErr) || !opErr.Timeout()) {
-			log.WithoutContext().Errorf("Error while Peeking first byte: %s", err)
+			log.WithoutContext().Debugf("Error while peeking first byte: %s", err)
 		}
 		return nil, err
 	}
@@ -376,7 +376,7 @@ func clientHelloInfo(br *bufio.Reader) (*clientHello, error) {
 	const recordHeaderLen = 5
 	hdr, err = br.Peek(recordHeaderLen)
 	if err != nil {
-		log.Errorf("Error while Peeking hello: %s", err)
+		log.WithoutContext().Errorf("Error while peeking client hello headers: %s", err)
 		return &clientHello{
 			peeked: getPeeked(br),
 		}, nil
@@ -390,7 +390,7 @@ func clientHelloInfo(br *bufio.Reader) (*clientHello, error) {
 
 	helloBytes, err := br.Peek(recordHeaderLen + recLen)
 	if err != nil {
-		log.Errorf("Error while Hello: %s", err)
+		log.WithoutContext().Errorf("Error while peeking client hello bytes: %s", err)
 		return &clientHello{
 			isTLS:  true,
 			peeked: getPeeked(br),
@@ -419,7 +419,7 @@ func clientHelloInfo(br *bufio.Reader) (*clientHello, error) {
 func getPeeked(br *bufio.Reader) string {
 	peeked, err := br.Peek(br.Buffered())
 	if err != nil {
-		log.Errorf("Could not get anything: %s", err)
+		log.WithoutContext().Errorf("Error while peeking bytes: %s", err)
 		return ""
 	}
 	return string(peeked)


Test patch: