Repository: traefik/traefik (language: go)

Issue:
TCP error finishing connection: Error while terminating connection
### Welcome!

- [X] Yes, I've searched similar issues on [GitHub](https://github.com/traefik/traefik/issues) and didn't find any.
- [X] Yes, I've searched similar issues on the [Traefik community forum](https://community.containo.us) and didn't find any.

### What did you do?

I migrated from voyager(HaProxy) to Traefik. When I used Voyager I didn't see errors like in Traefik.
Traefik installed to k8s via helm.
Сonfigured a TCP TLS routing to VMQ via IngressRouteTcps.

**UPD**:
The same situation with TCP without TLS.

**All connections work fine.** But I see a lot of  errors in logs.

### What did you see instead?

All connections work fine. But I see **a lot of  errors in logs**.
When MQTT connection finihing, Traefik send errors in logs. Whereas the Vernemq said `[debug] session normally stopped`.

Traefik (**exactly the Traefik pod**, not external client) continuously trying to connect to VMQ endpoint.

I can't find a root cause of this behavior. 
When I used Voyager I didn't see errors like in Traefik. 
I guess smth in the TCP settings ?

Very similar topics, but without solutions (or not working for me):

- [Connection errors for tcp endpoint via IngressRouteTCP](https://community.traefik.io/t/connection-errors-for-tcp-endpoint-via-ingressroutetcp/3499)
- [Traefik as Reverse-Proxy for internal Exchange](https://community.traefik.io/t/traefik-as-reverse-proxy-for-internal-exchange/10798)
- [Error while setting deadline && Error while terminating connection #8412](https://github.com/traefik/traefik/issues/8412)
- [Healthchecks for TCP forwarded connections](https://github.com/traefik/traefik/issues/5598)

### What version of Traefik are you using?

k8s GKE `v1.21.5-gke`
helm `traefik-10.6.2` (modified)
traefik:`2.5.6`

... and traefik:`2.6.0` also tested
```
Version:      2.6.0
Codename:     rocamadour
Go version:   go1.17.6
Built:        2022-01-24T17:08:39Z
OS/Arch:      linux/amd64
```



### What is your environment & configuration?

```yaml
Args:
--global.checknewversion=false
--global.sendanonymoususage=false
--entryPoints.ismpub.address=:4505/tcp
--entryPoints.ismreq.address=:4506/tcp
--entryPoints.metrics.address=:9100/tcp
--entryPoints.traefik.address=:9000/tcp
--entryPoints.vmq.address=:1883/tcp
--entryPoints.web.address=:8000/tcp
--entryPoints.websecure.address=:8443/tcp
--entryPoints.wss.address=:8080/tcp
--api.dashboard=true
--ping=true
--metrics.prometheus=true
--metrics.prometheus.entrypoint=metrics
--providers.kubernetescrd
--providers.kubernetescrd.labelSelector=ingress=traefik
--providers.kubernetesingress
--providers.kubernetesingress.ingressclass=traefik
--log.format=json
--log.level=DEBUG
--metrics.prometheus=true
--providers.file.watch=true
--providers.file.directory=/etc/traefik/config
--entrypoints.websecure.http.tls.domains[0].main=*.my-domain.com
```

IngressRouteTcps
```yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  labels:
    app.kubernetes.io/instance: traefik-routing-custom
    app.kubernetes.io/managed-by: Tiller
    app.kubernetes.io/name: traefik
    app.kubernetes.io/version: "1.0"
    helm.sh/chart: traefik
    ingress: traefik
  name: vernemq-tls
  namespace: custom
spec:
  entryPoints:
  - vmq
  routes:
  - match: HostSNI(`custom.my-domain.com`)
    services:
    - name: vernemq-custom
      port: 1883
      terminationDelay: 0
  tls:
    domains:
    - main: custom.my-domain.com
    passthrough: false
```


Vernemq, image: vernemq/vernemq:1.11.0-alpine
```
Environment: 
 DOCKER_VERNEMQ_DISCOVERY_KUBERNETES:                      1 
 DOCKER_VERNEMQ_LOG__CONSOLE__LEVEL:                       debug 
 MY_POD_NAME:                                               (v1:metadata.name) 
 DOCKER_VERNEMQ_KUBERNETES_NAMESPACE:                       (v1:metadata.namespace) 
 DOCKER_VERNEMQ_KUBERNETES_APP_LABEL:                      vernemq-sim-spb 
 DOCKER_VERNEMQ_KUBERNETES_LABEL_SELECTOR:                 app=vernemq-sim-spb 
 DOCKER_VERNEMQ_VMQ_PASSWD__PASSWORD_FILE:                 /etc/vmq-auth/vmq.passwd 
 DOCKER_VERNEMQ_VMQ_ACL__ACL_FILE:                         /etc/vmq-auth/vmq.acl 
 DOCKER_VERNEMQ_LISTENER__TCP__LOCALHOST:                  127.0.0.1:1883 
 DOCKER_VERNEMQ_ALLOW_REGISTER_DURING_NETSPLIT:            off 
 DOCKER_VERNEMQ_ALLOW_PUBLISH_DURING_NETSPLIT:             off 
 DOCKER_VERNEMQ_ALLOW_SUBSCRIBE_DURING_NETSPLIT:           off 
 DOCKER_VERNEMQ_ALLOW_UNSUBSCRIBE_DURING_NETSPLIT:         off 
 DOCKER_VERNEMQ_LISTENER__MAX_CONNECTIONS:                 1100 
 DOCKER_VERNEMQ_MAX_INFLIGHT_MESSAGES:                     500 
 DOCKER_VERNEMQ_MAX_ONLINE_MESSAGES:                       5000 
 DOCKER_VERNEMQ_MAX_OFFLINE_MESSAGES:                      5000 
 DOCKER_VERNEMQ_MAX_MESSAGE_SIZE:                          1048576 
 DOCKER_VERNEMQ_SYSTREE_ENABLED:                           off 
 DOCKER_VERNEMQ_LISTENER__TCP__ALLOWED_PROTOCOL_VERSIONS:  4 
 DOCKER_VERNEMQ_PERSISTENT_CLIENT_EXPIRATION:              1d 
 DOCKER_VERNEMQ_ACCEPT_EULA:                               yes
```

**UPD**:
- Also, I tried  to enable proxy protocol on the Traefik edge and VMQ endpoint, but there are no effects.
- Added a fresh traffic capture on the broken connection.

### If applicable, please paste the log output in DEBUG level

```
traefik-sim-spb-59f47fdd66-nktlc {"level":"debug","msg":"Handling connection from 10.156.0.65:2885","time":"2022-01-25T15:56:46Z"}  
traefik-sim-spb-59f47fdd66-nktlc {"level":"debug","msg":"Handling connection from 10.156.0.65:42720","time":"2022-01-25T15:56:46Z"} 
traefik-sim-spb-59f47fdd66-nktlc {"level":"error","msg":"Error during connection: read tcp 10.116.13.2:42436-\u003e10.116.4.22:1883: read: connection reset by peer","time":"2022-01-25T15:56:52Z"}
traefik-sim-spb-59f47fdd66-nktlc {"level":"debug","msg":"Error while terminating connection: close tcp 10.116.13.2:42436-\u003e10.116.4.22:1883: use of closed network connection","time":"2022-01-25T15:56:52Z"}
traefik-sim-spb-59f47fdd66-nktlc {"level":"error","msg":"Error during connection: read tcp 10.116.13.2:43270-\u003e10.116.5.33:1883: read: connection reset by peer","time":"2022-01-25T15:56:52Z"}
traefik-sim-spb-59f47fdd66-nktlc {"level":"debug","msg":"Error while terminating connection: close tcp 10.116.13.2:43270-\u003e10.116.5.33:1883: use of closed network connection","time":"2022-01-25T15:56:52Z"}
```

#### WireShark traffic capture, example on one broken connection
![image](https://user-images.githubusercontent.com/8701324/152894889-83c0f5ff-d893-46ad-adc9-16415e1d581f.png)

Plain text captured traffic
<details>
  <summary>Traefik->VMQ</summary>

  ```
No.     Time           Source                Destination           Protocol Length Info
      1 0.000000       10.68.0.198           10.68.67.144          TCP      80     55516 → 1883 [SYN] Seq=0 Win=42600 Len=0 MSS=1420 SACK_PERM=1 TSval=1334821873 TSecr=0 WS=128

Frame 1: 80 bytes on wire (640 bits), 80 bytes captured (640 bits) on interface -, id 0
    Interface id: 0 (-)
    Encapsulation type: Linux cooked-mode capture v2 (210)
    Arrival Time: Feb  8, 2022 02:55:42.348870000 MSK
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1644278142.348870000 seconds
    [Time delta from previous captured frame: 0.000000000 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 0.000000000 seconds]
    Frame Number: 1
    Frame Length: 80 bytes (640 bits)
    Capture Length: 80 bytes (640 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: sll:ethertype:ip:tcp]
    [Coloring Rule Name: TCP SYN/FIN]
    [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Linux cooked capture v2
Internet Protocol Version 4, Src: 10.68.0.198, Dst: 10.68.67.144
Transmission Control Protocol, Src Port: 55516, Dst Port: 1883, Seq: 0, Len: 0
    Source Port: 55516
    Destination Port: 1883
    [Stream index: 0]
    [TCP Segment Len: 0]
    Sequence Number: 0    (relative sequence number)
    Sequence Number (raw): 1710230021
    [Next Sequence Number: 1    (relative sequence number)]
    Acknowledgment Number: 0
    Acknowledgment number (raw): 0
    1010 .... = Header Length: 40 bytes (10)
    Flags: 0x002 (SYN)
    Window: 42600
    [Calculated window size: 42600]
    Checksum: 0x590c [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (20 bytes), Maximum segment size, SACK permitted, Timestamps, No-Operation (NOP), Window scale
    [Timestamps]

No.     Time           Source                Destination           Protocol Length Info
      2 0.000394       10.68.67.144          10.68.0.198           TCP      80     1883 → 55516 [SYN, ACK] Seq=0 Ack=1 Win=43648 Len=0 MSS=1420 SACK_PERM=1 TSval=1247120829 TSecr=1334821873 WS=128

Frame 2: 80 bytes on wire (640 bits), 80 bytes captured (640 bits) on interface -, id 0
    Interface id: 0 (-)
    Encapsulation type: Linux cooked-mode capture v2 (210)
    Arrival Time: Feb  8, 2022 02:55:42.349264000 MSK
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1644278142.349264000 seconds
    [Time delta from previous captured frame: 0.000394000 seconds]
    [Time delta from previous displayed frame: 0.000394000 seconds]
    [Time since reference or first frame: 0.000394000 seconds]
    Frame Number: 2
    Frame Length: 80 bytes (640 bits)
    Capture Length: 80 bytes (640 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: sll:ethertype:ip:tcp]
    [Coloring Rule Name: TCP SYN/FIN]
    [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Linux cooked capture v2
Internet Protocol Version 4, Src: 10.68.67.144, Dst: 10.68.0.198
Transmission Control Protocol, Src Port: 1883, Dst Port: 55516, Seq: 0, Ack: 1, Len: 0
    Source Port: 1883
    Destination Port: 55516
    [Stream index: 0]
    [TCP Segment Len: 0]
    Sequence Number: 0    (relative sequence number)
    Sequence Number (raw): 3548725987
    [Next Sequence Number: 1    (relative sequence number)]
    Acknowledgment Number: 1    (relative ack number)
    Acknowledgment number (raw): 1710230022
    1010 .... = Header Length: 40 bytes (10)
    Flags: 0x012 (SYN, ACK)
    Window: 43648
    [Calculated window size: 43648]
    Checksum: 0xf28e [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (20 bytes), Maximum segment size, SACK permitted, Timestamps, No-Operation (NOP), Window scale
    [SEQ/ACK analysis]
        [This is an ACK to the segment in frame: 1]
        [The RTT to ACK the segment was: 0.000394000 seconds]
        [iRTT: 0.000417000 seconds]
    [Timestamps]

No.     Time           Source                Destination           Protocol Length Info
      3 0.000417       10.68.0.198           10.68.67.144          TCP      72     55516 → 1883 [ACK] Seq=1 Ack=1 Win=42624 Len=0 TSval=1334821874 TSecr=1247120829

Frame 3: 72 bytes on wire (576 bits), 72 bytes captured (576 bits) on interface -, id 0
    Interface id: 0 (-)
    Encapsulation type: Linux cooked-mode capture v2 (210)
    Arrival Time: Feb  8, 2022 02:55:42.349287000 MSK
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1644278142.349287000 seconds
    [Time delta from previous captured frame: 0.000023000 seconds]
    [Time delta from previous displayed frame: 0.000023000 seconds]
    [Time since reference or first frame: 0.000417000 seconds]
    Frame Number: 3
    Frame Length: 72 bytes (576 bits)
    Capture Length: 72 bytes (576 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: sll:ethertype:ip:tcp]
    [Coloring Rule Name: TCP]
    [Coloring Rule String: tcp]
Linux cooked capture v2
Internet Protocol Version 4, Src: 10.68.0.198, Dst: 10.68.67.144
Transmission Control Protocol, Src Port: 55516, Dst Port: 1883, Seq: 1, Ack: 1, Len: 0
    Source Port: 55516
    Destination Port: 1883
    [Stream index: 0]
    [TCP Segment Len: 0]
    Sequence Number: 1    (relative sequence number)
    Sequence Number (raw): 1710230022
    [Next Sequence Number: 1    (relative sequence number)]
    Acknowledgment Number: 1    (relative ack number)
    Acknowledgment number (raw): 3548725988
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x010 (ACK)
    Window: 333
    [Calculated window size: 42624]
    [Window size scaling factor: 128]
    Checksum: 0x5904 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [SEQ/ACK analysis]
        [This is an ACK to the segment in frame: 2]
        [The RTT to ACK the segment was: 0.000023000 seconds]
        [iRTT: 0.000417000 seconds]
    [Timestamps]

No.     Time           Source                Destination           Protocol Length Info
      4 0.000482       10.68.0.198           10.68.67.144          MQTT     100    Reserved

Frame 4: 100 bytes on wire (800 bits), 100 bytes captured (800 bits) on interface -, id 0
    Interface id: 0 (-)
    Encapsulation type: Linux cooked-mode capture v2 (210)
    Arrival Time: Feb  8, 2022 02:55:42.349352000 MSK
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1644278142.349352000 seconds
    [Time delta from previous captured frame: 0.000065000 seconds]
    [Time delta from previous displayed frame: 0.000065000 seconds]
    [Time since reference or first frame: 0.000482000 seconds]
    Frame Number: 4
    Frame Length: 100 bytes (800 bits)
    Capture Length: 100 bytes (800 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: sll:ethertype:ip:tcp:mqtt]
    [Coloring Rule Name: TCP]
    [Coloring Rule String: tcp]
Linux cooked capture v2
Internet Protocol Version 4, Src: 10.68.0.198, Dst: 10.68.67.144
Transmission Control Protocol, Src Port: 55516, Dst Port: 1883, Seq: 1, Ack: 1, Len: 28
    Source Port: 55516
    Destination Port: 1883
    [Stream index: 0]
    [TCP Segment Len: 28]
    Sequence Number: 1    (relative sequence number)
    Sequence Number (raw): 1710230022
    [Next Sequence Number: 29    (relative sequence number)]
    Acknowledgment Number: 1    (relative ack number)
    Acknowledgment number (raw): 3548725988
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x018 (PSH, ACK)
    Window: 333
    [Calculated window size: 42624]
    [Window size scaling factor: 128]
    Checksum: 0x5920 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [SEQ/ACK analysis]
        [iRTT: 0.000417000 seconds]
        [Bytes in flight: 28]
        [Bytes sent since last PSH flag: 28]
    [Timestamps]
    TCP payload (28 bytes)
    [PDU Size: 12]
    TCP segment data (16 bytes)
MQ Telemetry Transport Protocol, Reserved
    [Expert Info (Note/Protocol): Unknown version (missing the CONNECT packet?)]
        [Unknown version (missing the CONNECT packet?)]
        [Severity level: Note]
        [Group: Protocol]
    Header Flags: 0x0d, Message Type: Reserved
        0000 .... = Message Type: Reserved (0)
        .... 1101 = Reserved: 13
    Msg Len: 10

No.     Time           Source                Destination           Protocol Length Info
      5 0.000724       10.68.67.144          10.68.0.198           TCP      72     1883 → 55516 [ACK] Seq=1 Ack=29 Win=43648 Len=0 TSval=1247120830 TSecr=1334821874

Frame 5: 72 bytes on wire (576 bits), 72 bytes captured (576 bits) on interface -, id 0
    Interface id: 0 (-)
    Encapsulation type: Linux cooked-mode capture v2 (210)
    Arrival Time: Feb  8, 2022 02:55:42.349594000 MSK
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1644278142.349594000 seconds
    [Time delta from previous captured frame: 0.000242000 seconds]
    [Time delta from previous displayed frame: 0.000242000 seconds]
    [Time since reference or first frame: 0.000724000 seconds]
    Frame Number: 5
    Frame Length: 72 bytes (576 bits)
    Capture Length: 72 bytes (576 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: sll:ethertype:ip:tcp]
    [Coloring Rule Name: TCP]
    [Coloring Rule String: tcp]
Linux cooked capture v2
Internet Protocol Version 4, Src: 10.68.67.144, Dst: 10.68.0.198
Transmission Control Protocol, Src Port: 1883, Dst Port: 55516, Seq: 1, Ack: 29, Len: 0
    Source Port: 1883
    Destination Port: 55516
    [Stream index: 0]
    [TCP Segment Len: 0]
    Sequence Number: 1    (relative sequence number)
    Sequence Number (raw): 3548725988
    [Next Sequence Number: 1    (relative sequence number)]
    Acknowledgment Number: 29    (relative ack number)
    Acknowledgment number (raw): 1710230050
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x010 (ACK)
    Window: 341
    [Calculated window size: 43648]
    [Window size scaling factor: 128]
    Checksum: 0xca40 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [SEQ/ACK analysis]
        [This is an ACK to the segment in frame: 4]
        [The RTT to ACK the segment was: 0.000242000 seconds]
        [iRTT: 0.000417000 seconds]
    [Timestamps]

No.     Time           Source                Destination           Protocol Length Info
      6 0.091252       10.68.0.198           10.68.67.144          MQTT     378    Connect Ack, Reserved, Reserved, Publish Release (id=28530), Publish Message[Malformed Packet]

Frame 6: 378 bytes on wire (3024 bits), 378 bytes captured (3024 bits) on interface -, id 0
    Interface id: 0 (-)
    Encapsulation type: Linux cooked-mode capture v2 (210)
    Arrival Time: Feb  8, 2022 02:55:42.440122000 MSK
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1644278142.440122000 seconds
    [Time delta from previous captured frame: 0.090528000 seconds]
    [Time delta from previous displayed frame: 0.090528000 seconds]
    [Time since reference or first frame: 0.091252000 seconds]
    Frame Number: 6
    Frame Length: 378 bytes (3024 bits)
    Capture Length: 378 bytes (3024 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: sll:ethertype:ip:tcp:mqtt:mqtt]
    [Coloring Rule Name: TCP]
    [Coloring Rule String: tcp]
Linux cooked capture v2
Internet Protocol Version 4, Src: 10.68.0.198, Dst: 10.68.67.144
Transmission Control Protocol, Src Port: 55516, Dst Port: 1883, Seq: 29, Ack: 1, Len: 306
    Source Port: 55516
    Destination Port: 1883
    [Stream index: 0]
    [TCP Segment Len: 306]
    Sequence Number: 29    (relative sequence number)
    Sequence Number (raw): 1710230050
    [Next Sequence Number: 335    (relative sequence number)]
    Acknowledgment Number: 1    (relative ack number)
    Acknowledgment number (raw): 3548725988
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x018 (PSH, ACK)
    Window: 333
    [Calculated window size: 42624]
    [Window size scaling factor: 128]
    Checksum: 0x5a36 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [SEQ/ACK analysis]
        [iRTT: 0.000417000 seconds]
        [Bytes in flight: 306]
        [Bytes sent since last PSH flag: 306]
    [Timestamps]
    TCP payload (306 bytes)
    [PDU Size: 19]
    TCP segment data (3 bytes)
    [PDU Size: 6]
    [PDU Size: 81]
    [PDU Size: 117]
    [PDU Size: 60]
    TCP segment data (39 bytes)
[2 Reassembled TCP Segments (19 bytes): #4(16), #6(3)]
MQ Telemetry Transport Protocol, Connect Ack
    [Expert Info (Note/Protocol): Unknown version (missing the CONNECT packet?)]
        [Unknown version (missing the CONNECT packet?)]
        [Severity level: Note]
        [Group: Protocol]
    Header Flags: 0x21, Message Type: Connect Ack
        0010 .... = Message Type: Connect Ack (2)
        .... 0001 = Reserved: 1
    Msg Len: 17
    Acknowledge Flags: 0x00
    Reason Code: Unknown (12)
MQ Telemetry Transport Protocol, Reserved
    [Expert Info (Note/Protocol): Unknown version (missing the CONNECT packet?)]
        [Unknown version (missing the CONNECT packet?)]
        [Severity level: Note]
        [Group: Protocol]
    Header Flags: 0x00, Message Type: Reserved
        0000 .... = Message Type: Reserved (0)
        .... 0000 = Reserved: 0
    Msg Len: 4
MQ Telemetry Transport Protocol, Reserved
    [Expert Info (Note/Protocol): Unknown version (missing the CONNECT packet?)]
        [Unknown version (missing the CONNECT packet?)]
        [Severity level: Note]
        [Group: Protocol]
    Header Flags: 0x04, Message Type: Reserved
        0000 .... = Message Type: Reserved (0)
        .... 0100 = Reserved: 4
    Msg Len: 78
MQ Telemetry Transport Protocol, Publish Release
    [Expert Info (Note/Protocol): Unknown version (missing the CONNECT packet?)]
        [Unknown version (missing the CONNECT packet?)]
        [Severity level: Note]
        [Group: Protocol]
    Header Flags: 0x6e, Message Type: Publish Release
        0110 .... = Message Type: Publish Release (6)
        .... 1110 = Reserved: 14
    Msg Len: 115
    Message Identifier: 28530
MQ Telemetry Transport Protocol, Publish Message
    [Expert Info (Note/Protocol): Unknown version (missing the CONNECT packet?)]
        [Unknown version (missing the CONNECT packet?)]
        [Severity level: Note]
        [Group: Protocol]
    Header Flags: 0x30, Message Type: Publish Message, QoS Level: At most once delivery (Fire and Forget)
        0011 .... = Message Type: Publish Message (3)
        .... 0... = DUP Flag: Not set
        .... .00. = QoS Level: At most once delivery (Fire and Forget) (0)
        .... ...0 = Retain: Not set
    Msg Len: 58
    Topic Length: 12344
[Malformed Packet: MQTT]
    [Expert Info (Error/Malformed): Malformed Packet (Exception occurred)]
        [Malformed Packet (Exception occurred)]
        [Severity level: Error]
        [Group: Malformed]

No.     Time           Source                Destination           Protocol Length Info
      7 0.091427       10.68.67.144          10.68.0.198           TCP      72     1883 → 55516 [ACK] Seq=1 Ack=335 Win=43392 Len=0 TSval=1247120920 TSecr=1334821965

Frame 7: 72 bytes on wire (576 bits), 72 bytes captured (576 bits) on interface -, id 0
    Interface id: 0 (-)
    Encapsulation type: Linux cooked-mode capture v2 (210)
    Arrival Time: Feb  8, 2022 02:55:42.440297000 MSK
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1644278142.440297000 seconds
    [Time delta from previous captured frame: 0.000175000 seconds]
    [Time delta from previous displayed frame: 0.000175000 seconds]
    [Time since reference or first frame: 0.091427000 seconds]
    Frame Number: 7
    Frame Length: 72 bytes (576 bits)
    Capture Length: 72 bytes (576 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: sll:ethertype:ip:tcp]
    [Coloring Rule Name: TCP]
    [Coloring Rule String: tcp]
Linux cooked capture v2
Internet Protocol Version 4, Src: 10.68.67.144, Dst: 10.68.0.198
Transmission Control Protocol, Src Port: 1883, Dst Port: 55516, Seq: 1, Ack: 335, Len: 0
    Source Port: 1883
    Destination Port: 55516
    [Stream index: 0]
    [TCP Segment Len: 0]
    Sequence Number: 1    (relative sequence number)
    Sequence Number (raw): 3548725988
    [Next Sequence Number: 1    (relative sequence number)]
    Acknowledgment Number: 335    (relative ack number)
    Acknowledgment number (raw): 1710230356
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x010 (ACK)
    Window: 339
    [Calculated window size: 43392]
    [Window size scaling factor: 128]
    Checksum: 0xc85b [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [SEQ/ACK analysis]
        [This is an ACK to the segment in frame: 6]
        [The RTT to ACK the segment was: 0.000175000 seconds]
        [iRTT: 0.000417000 seconds]
    [Timestamps]

No.     Time           Source                Destination           Protocol Length Info
      8 0.194385       10.68.67.144          10.68.0.198           MQTT     76     Connect Ack

Frame 8: 76 bytes on wire (608 bits), 76 bytes captured (608 bits) on interface -, id 0
    Interface id: 0 (-)
    Encapsulation type: Linux cooked-mode capture v2 (210)
    Arrival Time: Feb  8, 2022 02:55:42.543255000 MSK
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1644278142.543255000 seconds
    [Time delta from previous captured frame: 0.102958000 seconds]
    [Time delta from previous displayed frame: 0.102958000 seconds]
    [Time since reference or first frame: 0.194385000 seconds]
    Frame Number: 8
    Frame Length: 76 bytes (608 bits)
    Capture Length: 76 bytes (608 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: sll:ethertype:ip:tcp:mqtt]
    [Coloring Rule Name: TCP]
    [Coloring Rule String: tcp]
Linux cooked capture v2
Internet Protocol Version 4, Src: 10.68.67.144, Dst: 10.68.0.198
Transmission Control Protocol, Src Port: 1883, Dst Port: 55516, Seq: 1, Ack: 335, Len: 4
    Source Port: 1883
    Destination Port: 55516
    [Stream index: 0]
    [TCP Segment Len: 4]
    Sequence Number: 1    (relative sequence number)
    Sequence Number (raw): 3548725988
    [Next Sequence Number: 5    (relative sequence number)]
    Acknowledgment Number: 335    (relative ack number)
    Acknowledgment number (raw): 1710230356
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x018 (PSH, ACK)
    Window: 339
    [Calculated window size: 43392]
    [Window size scaling factor: 128]
    Checksum: 0xa7e6 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [SEQ/ACK analysis]
        [iRTT: 0.000417000 seconds]
        [Bytes in flight: 4]
        [Bytes sent since last PSH flag: 4]
    [Timestamps]
    TCP payload (4 bytes)
    [PDU Size: 4]
MQ Telemetry Transport Protocol, Connect Ack
    [Expert Info (Note/Protocol): Unknown version (missing the CONNECT packet?)]
        [Unknown version (missing the CONNECT packet?)]
        [Severity level: Note]
        [Group: Protocol]
    Header Flags: 0x20, Message Type: Connect Ack
        0010 .... = Message Type: Connect Ack (2)
        .... 0000 = Reserved: 0
    Msg Len: 2
    Acknowledge Flags: 0x00
    Reason Code: Success (0)

No.     Time           Source                Destination           Protocol Length Info
      9 0.194395       10.68.0.198           10.68.67.144          TCP      72     55516 → 1883 [ACK] Seq=335 Ack=5 Win=42624 Len=0 TSval=1334822068 TSecr=1247121023

Frame 9: 72 bytes on wire (576 bits), 72 bytes captured (576 bits) on interface -, id 0
    Interface id: 0 (-)
    Encapsulation type: Linux cooked-mode capture v2 (210)
    Arrival Time: Feb  8, 2022 02:55:42.543265000 MSK
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1644278142.543265000 seconds
    [Time delta from previous captured frame: 0.000010000 seconds]
    [Time delta from previous displayed frame: 0.000010000 seconds]
    [Time since reference or first frame: 0.194395000 seconds]
    Frame Number: 9
    Frame Length: 72 bytes (576 bits)
    Capture Length: 72 bytes (576 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: sll:ethertype:ip:tcp]
    [Coloring Rule Name: TCP]
    [Coloring Rule String: tcp]
Linux cooked capture v2
Internet Protocol Version 4, Src: 10.68.0.198, Dst: 10.68.67.144
Transmission Control Protocol, Src Port: 55516, Dst Port: 1883, Seq: 335, Ack: 5, Len: 0
    Source Port: 55516
    Destination Port: 1883
    [Stream index: 0]
    [TCP Segment Len: 0]
    Sequence Number: 335    (relative sequence number)
    Sequence Number (raw): 1710230356
    [Next Sequence Number: 335    (relative sequence number)]
    Acknowledgment Number: 5    (relative ack number)
    Acknowledgment number (raw): 3548725992
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x010 (ACK)
    Window: 333
    [Calculated window size: 42624]
    [Window size scaling factor: 128]
    Checksum: 0x5904 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [SEQ/ACK analysis]
        [This is an ACK to the segment in frame: 8]
        [The RTT to ACK the segment was: 0.000010000 seconds]
        [iRTT: 0.000417000 seconds]
    [Timestamps]

No.     Time           Source                Destination           Protocol Length Info
     10 0.194512       10.68.67.144          10.68.0.198           TCP      72     1883 → 55516 [RST, ACK] Seq=5 Ack=335 Win=43392 Len=0 TSval=1247121023 TSecr=1334821965

Frame 10: 72 bytes on wire (576 bits), 72 bytes captured (576 bits) on interface -, id 0
    Interface id: 0 (-)
    Encapsulation type: Linux cooked-mode capture v2 (210)
    Arrival Time: Feb  8, 2022 02:55:42.543382000 MSK
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1644278142.543382000 seconds
    [Time delta from previous captured frame: 0.000117000 seconds]
    [Time delta from previous displayed frame: 0.000117000 seconds]
    [Time since reference or first frame: 0.194512000 seconds]
    Frame Number: 10
    Frame Length: 72 bytes (576 bits)
    Capture Length: 72 bytes (576 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: sll:ethertype:ip:tcp]
    [Coloring Rule Name: TCP RST]
    [Coloring Rule String: tcp.flags.reset eq 1]
Linux cooked capture v2
Internet Protocol Version 4, Src: 10.68.67.144, Dst: 10.68.0.198
Transmission Control Protocol, Src Port: 1883, Dst Port: 55516, Seq: 5, Ack: 335, Len: 0
    Source Port: 1883
    Destination Port: 55516
    [Stream index: 0]
    [TCP Segment Len: 0]
    Sequence Number: 5    (relative sequence number)
    Sequence Number (raw): 3548725992
    [Next Sequence Number: 5    (relative sequence number)]
    Acknowledgment Number: 335    (relative ack number)
    Acknowledgment number (raw): 1710230356
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x014 (RST, ACK)
    Window: 339
    [Calculated window size: 43392]
    [Window size scaling factor: 128]
    Checksum: 0xc7ec [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [Timestamps]

No.     Time           Source                Destination           Protocol Length Info
     11 0.194528       10.68.67.144          10.68.0.198           TCP      60     1883 → 55516 [RST] Seq=5 Win=0 Len=0

Frame 11: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) on interface -, id 0
    Interface id: 0 (-)
    Encapsulation type: Linux cooked-mode capture v2 (210)
    Arrival Time: Feb  8, 2022 02:55:42.543398000 MSK
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1644278142.543398000 seconds
    [Time delta from previous captured frame: 0.000016000 seconds]
    [Time delta from previous displayed frame: 0.000016000 seconds]
    [Time since reference or first frame: 0.194528000 seconds]
    Frame Number: 11
    Frame Length: 60 bytes (480 bits)
    Capture Length: 60 bytes (480 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: sll:ethertype:ip:tcp]
    [Coloring Rule Name: TCP RST]
    [Coloring Rule String: tcp.flags.reset eq 1]
Linux cooked capture v2
Internet Protocol Version 4, Src: 10.68.67.144, Dst: 10.68.0.198
Transmission Control Protocol, Src Port: 1883, Dst Port: 55516, Seq: 5, Len: 0
    Source Port: 1883
    Destination Port: 55516
    [Stream index: 0]
    [TCP Segment Len: 0]
    Sequence Number: 5    (relative sequence number)
    Sequence Number (raw): 3548725992
    [Next Sequence Number: 5    (relative sequence number)]
    Acknowledgment Number: 0
    Acknowledgment number (raw): 0
    0101 .... = Header Length: 20 bytes (5)
    Flags: 0x004 (RST)
    Window: 0
    [Calculated window size: 0]
    [Window size scaling factor: 128]
    Checksum: 0x605d [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    [Timestamps]

  ```
</details>


WireShark traffic capture
![image](https://user-images.githubusercontent.com/8701324/151054383-94e6e5f6-fabb-4512-aa57-c8f8576e3e7a.png)



Discussion / hints:
 Let me know if there is anything you need from me.
 I look forward when this task will be moved  to work status!
Hello @kksudo,

Thanks for all the details you provided!

I'm not able to reproduce the issue. Precisely, I can, from time to time, observe connection error logs (while everything seems fine for the MQTT client and server), but only `DEBUG` level ones, like:
- `DEBU[2022-02-14T16:24:09+01:00] Error while setting deadline: set tcp 127.0.0.1:55429: use of closed network connection`
- `DEBU[2022-02-14T15:44:51+01:00] Error while terminating connection: close tcp 127.0.0.1:8001->127.0.0.1:54724: shutdown: socket is not connected`

What MQTT client are you using? (I used an MQTT go client to reproduce the issue)
@rtribotte 
Hello,
In our production case, we are using the "github.com/eclipse/paho.mqtt.golang" lib.

Also, we are observing the same errors with our test utility for traffic generator (test instance -> Traefik -> VMQ).

Besides 2 cases above, we can reproduce these errors with the [mqtt-explorer](http://mqtt-explorer.com/).

We  deployed the Traefik as the ingress controller, as the deployment entity in a k8s (more than 2 pods in a deployment).

When we are using more than 1000 connections from different sources, we are continuously observing the error flow in logs. Moreover, it's the logs "level":"error".

Second case: 
Same as above, we have more than 1000 connections.
We are continuously observing a lot of errors in logs.
When we try to redeploy the Traeffik release, or scale down pods count in deployment and scale up back. Then the Traefik continuously increasing connections(fake connections or already closed ?) from the Traeffik pod to the VMQ instance.


1. Did you check the traffic sniffer to analyze the errors what you found in logs?
2. Could you please share your test environment settings? Is it the k8s or smth else?
Hello @kksudo,

Thanks for your feedback!

I think the main difference between the reproduction setup with which I have tried to reproduce the issue and what you are describing is the number of concurrent connections.

I'll try again with those numbers and let you know.
Hello @kksudo,

Regarding my test environment settings, I used a simple client go client with the "github.com/eclipse/paho.mqtt.golang" lib,
I ran vernemq in a docker container, and ran Traefik locally with dynamic file configuration.

I tried again with a high number of concurrent connections (100 to 10000), but I did not succeed to reproduce the issue.

I experienced the `connection reset by peer` error but initiated by the client and not the server.

By looking around, I stumbled upon this issue https://github.com/vernemq/vernemq/issues/1433.
Which suggests that the behavior from the vernemq when closing TCP session could be dependent on some advanced configuration: https://docs.vernemq.com/misc/not-a-tuning-guide.

That made me wonder what is the version of vernemq you are using, and if you tried to setup custom configuration in that advanced configuration file.

Anyway, since the RST Packets are expected in some situations, and because you mentioned seeing no error logs when using HaProxy, I took a look at the HaProxy repository to find out wether those RST packets may be silently discarded. I could be wrong but it might be what is happening here: https://github.com/haproxy/haproxy/blob/06e66c84fcff0e4ab31a0b0f35f7a3ac63b9c29c/src/tcpcheck.c#L1581-L1590

In any case, I will open a pull request to make the `connection reset by peer` be logged at the  debug level (so that for you it will not show up).

Fix patch:
diff --git a/pkg/tcp/proxy.go b/pkg/tcp/proxy.go
index a9c5f1844c..8887bb91d4 100644
--- a/pkg/tcp/proxy.go
+++ b/pkg/tcp/proxy.go
@@ -1,9 +1,11 @@
 package tcp
 
 import (
+	"errors"
 	"fmt"
 	"io"
 	"net"
+	"syscall"
 	"time"
 
 	"github.com/pires/go-proxyproto"
@@ -74,7 +76,14 @@ func (p *Proxy) ServeTCP(conn WriteCloser) {
 
 	err = <-errChan
 	if err != nil {
-		log.WithoutContext().Errorf("Error during connection: %v", err)
+		// Treat connection reset error during a read operation with a lower log level.
+		// This allows to not report an RST packet sent by the peer as an error,
+		// as it is an abrupt but possible end for the TCP session
+		if isReadConnResetError(err) {
+			log.WithoutContext().Debugf("Error during connection: %v", err)
+		} else {
+			log.WithoutContext().Errorf("Error during connection: %v", err)
+		}
 	}
 
 	<-errChan
@@ -101,9 +110,18 @@ func (p Proxy) connCopy(dst, src WriteCloser, errCh chan error) {
 	_, err := io.Copy(dst, src)
 	errCh <- err
 
+	// Ends the connection with the dst connection peer.
+	// It corresponds to sending a FIN packet to gracefully end the TCP session.
 	errClose := dst.CloseWrite()
 	if errClose != nil {
-		log.WithoutContext().Debugf("Error while terminating connection: %v", errClose)
+		// Calling CloseWrite() on a connection which have a socket which is "not connected" is expected to fail.
+		// It happens notably when the dst connection has ended receiving an RST packet from the peer (within the other connCopy call).
+		// In that case, logging the error is superfluous,
+		// as in the first place we should not have needed to call CloseWrite.
+		if !isSocketNotConnectedError(errClose) {
+			log.WithoutContext().Debugf("Error while terminating connection: %v", errClose)
+		}
+
 		return
 	}
 
@@ -114,3 +132,9 @@ func (p Proxy) connCopy(dst, src WriteCloser, errCh chan error) {
 		}
 	}
 }
+
+// isSocketNotConnectedError reports whether err is a socket not connected error.
+func isSocketNotConnectedError(err error) bool {
+	var oerr *net.OpError
+	return errors.As(err, &oerr) && errors.Is(err, syscall.ENOTCONN)
+}
diff --git a/pkg/tcp/proxy_unix.go b/pkg/tcp/proxy_unix.go
new file mode 100644
index 0000000000..727074bccc
--- /dev/null
+++ b/pkg/tcp/proxy_unix.go
@@ -0,0 +1,16 @@
+//go:build !windows
+// +build !windows
+
+package tcp
+
+import (
+	"errors"
+	"net"
+	"syscall"
+)
+
+// isReadConnResetError reports whether err is a connection reset error during a read operation.
+func isReadConnResetError(err error) bool {
+	var oerr *net.OpError
+	return errors.As(err, &oerr) && oerr.Op == "read" && errors.Is(err, syscall.ECONNRESET)
+}
diff --git a/pkg/tcp/proxy_windows.go b/pkg/tcp/proxy_windows.go
new file mode 100644
index 0000000000..579d520634
--- /dev/null
+++ b/pkg/tcp/proxy_windows.go
@@ -0,0 +1,16 @@
+//go:build windows
+// +build windows
+
+package tcp
+
+import (
+	"errors"
+	"net"
+	"syscall"
+)
+
+// isReadConnResetError reports whether err is a connection reset error during a read operation.
+func isReadConnResetError(err error) bool {
+	var oerr *net.OpError
+	return errors.As(err, &oerr) && oerr.Op == "read" && errors.Is(err, syscall.WSAECONNRESET)
+}


Test patch: