Repository: traefik/traefik (language: go)

Issue:
[3.0] When any type of "catch all" route exists on tls enabled entrypoint, mtls (TLSOption/clientAuth) becomes unstable to extract certificate
### Welcome!

- [X] Yes, I've searched similar issues on [GitHub](https://github.com/traefik/traefik/issues) and didn't find any.
- [X] Yes, I've searched similar issues on the [Traefik community forum](https://community.traefik.io) and didn't find any.

### What did you do?

We are trying k8s deployed traefik 3.0, working through KubernetesCRD with traefik CRD's.
Part of our setup is mtls auth, through TLSOption/cluentAuth + Middleware/passTLSClientCert.
Additionally, we have "catch all" route for tls enabled entrypoint. This route is done as IngressRoute with "match: HostRegexp(`.*`)" or "match: 'PathPrefix(`/`)" (tried both, result is the same) attached to tls enabled entrypoint.
When this "catch all" route is present, any other route (catch all doesn't have mtls) which has mtls auth enabled, becomes unstable, and with some very high probability traefik will fail to extract client certificate from incoming connection, passing proper request, but without any client auth, to the route target.

### What did you see instead?

mtls is unstable, when some variant of "catch all" exists on tls enabled entrypoint.
when "catch all" route is removed, there is immediate effect with no issues with mtls anymore.

Additional Info: Despite any variant of "catch all" on tls throws some kind of warning, this setup work just fine on traefik 2.0, and especially the resulting effect is weird. I understand warnings about that it's not always possible to properly catch all in tls, but making client auth unstable as the result is unexpected.

### What version of Traefik are you using?

"traefik:v3.0" docker image - where issues is present
"traefik:v2.10" docker image where it is working fine

### What is your environment & configuration?

```yaml
    entryPoints:
      http:
        address: ':80'
        proxyProtocol:
          trustedIPs:
            - 10.117.16.0/21
      https:
        address: ':443'
        http:
          tls:
            options: default
        proxyProtocol:
          trustedIPs:
            - 10.117.16.0/21
      traefik:
        address: ':8080'
        proxyProtocol:
          trustedIPs:
            - 10.117.16.0/21
    ping: {}
    log:
      level: INFO
    api:
      dashboard: true
      insecure: true
    # log only 500's and long running requests to debug issues
    accessLog:
      filters:
        statusCodes:
          - "500-510"
        minDuration: "3s"
    providers:
      # process services of kind "Ingress" with "kubernetes.io/ingress.class" annotation set to "traefik"
      kubernetesIngress:
        ingressClass: traefik
      # process native traefik services of kind like "IngressRoute", "Middleware", etc.
      kubernetesCRD:
        # allow traefik IngressRoutes's reference resources in other namespace than theirs
        allowCrossNamespace: true
```

```yaml
apiVersion: traefik.io/v1alpha1
kind: TLSOption
metadata:
  name: default
  namespace: traefik
spec:
  minVersion: VersionTLS12
```

mtls option
```yaml
apiVersion: traefik.io/v1alpha1
kind: TLSOption
metadata:
  name: mtls
  namespace: traefik
spec:
  minVersion: VersionTLS12
  clientAuth:
    # the CA certificate is extracted from key `tls.ca` of the given secrets.
    secretNames:
      - defaultcertificate
    clientAuthType: VerifyClientCertIfGiven
```

mtls middleware
```yaml
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: pass-x-tls-client-cert-header
  namespace: traefik
spec:
  passTLSClientCert:
    info:
      subject:
        commonName: true
        serialNumber: true
      issuer:
        organization: true
        commonName: true
        serialNumber: true
```

Catch all route
```yaml
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: catch-all
  namespace: legends
spec:
  entryPoints:
    - https
  routes:
    - match: 'PathPrefix(`/`)'
      kind: Rule
      priority: 1
      services:
        - name: catch-all
          port: 8080
  tls: {}
```

example mtls enabled route
```yaml
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: metadata
  namespace: legends
spec:
  entryPoints:
    - https
  routes:
    - match: Host(`some...`)
      kind: Rule
      middlewares:
        - name: pass-x-tls-client-cert-header
          namespace: traefik
      services:
        - name: metadata
          port: 80
  tls:
    options:
      name: mtls
      namespace: traefik
    secretName: certificate-k8s-tl
```


### If applicable, please paste the log output in DEBUG level

when cahtch-all route is present, traefik gets elevated level of
2023-11-28T23:42:20Z WRN Tried to extract a certificate on a request without mutual TLS middlewareName=legends-pass-x-tls-client-cert-header@kubernetescrd middlewareType=PassClientTLSCert
messages


Discussion / hints:
Hello @Owersun,

Thanks for reporting this!

There seems to be a bug in the evaluation priority for TLS termination when having a "catchAll" router rule among all the routes and when not.

I'll mark this issue as a bug/possible while waiting for us to reproduce it.

Fix patch:
diff --git a/pkg/server/router/tcp/router.go b/pkg/server/router/tcp/router.go
index 8c2e8939bd..711a603138 100644
--- a/pkg/server/router/tcp/router.go
+++ b/pkg/server/router/tcp/router.go
@@ -266,9 +266,8 @@ func (r *Router) SetHTTPSForwarder(handler tcp.Handler) {
 			}
 		}
 
-		// muxerHTTPS only contains single HostSNI rules (and no other kind of rules),
-		// so there's no need for specifying a priority for them.
-		if err := r.muxerHTTPS.AddRoute("HostSNI(`"+sniHost+"`)", 0, tcpHandler); err != nil {
+		rule := "HostSNI(`" + sniHost + "`)"
+		if err := r.muxerHTTPS.AddRoute(rule, tcpmuxer.GetRulePriority(rule), tcpHandler); err != nil {
 			log.Error().Err(err).Msg("Error while adding route for host")
 		}
 	}


Test patch:
diff --git a/pkg/server/router/tcp/router_test.go b/pkg/server/router/tcp/router_test.go
index 5be801476c..29bb2c4182 100644
--- a/pkg/server/router/tcp/router_test.go
+++ b/pkg/server/router/tcp/router_test.go
@@ -494,6 +494,21 @@ func Test_Routing(t *testing.T) {
 				},
 			},
 		},
+		{
+			desc:    "HTTPS router && HTTPS CatchAll router",
+			routers: []applyRouter{routerHTTPS, routerHTTPSPathPrefix},
+			checks: []checkCase{
+				{
+					desc:          "HTTPS TLS 1.0 request should fail",
+					checkRouter:   checkHTTPSTLS10,
+					expectedError: "wrong TLS version",
+				},
+				{
+					desc:        "HTTPS TLS 1.2 request should be handled by HTTPS service",
+					checkRouter: checkHTTPSTLS12,
+				},
+			},
+		},
 		{
 			desc:    "All routers, all checks",
 			routers: []applyRouter{routerTCPCatchAll, routerHTTP, routerHTTPS, routerTCPTLS, routerTCPTLSCatchAll},