Repository: traefik/traefik (language: go)

Issue:
gRPC number of concurrent stream block at 250
### Welcome!

- [X] Yes, I've searched similar issues on [GitHub](https://github.com/traefik/traefik/issues) and didn't find any.
- [X] Yes, I've searched similar issues on the [Traefik community forum](https://community.containo.us) and didn't find any.

### What did you do?

I created a simple gRPC client and server. The server is deployed inside our k8s cluster (with k3s) and traefik v2.4.8.
There is only one replica of the server. I got an ingress using h2c protocol and everything works great.

This is my proto, a simple stream from the server to the client:
```proto
service TelemetryService {
  rpc GetTelemetry(TelemetryMeta) returns (stream Telemetry) {}
}
```

When I create a large number of stream from the client by calling GetTelemetry(), the number get blocked to exactly 250.
If I use a NodePort instead of the ingress, everything works as expected.

Moreover, when I close the client, I dont receive the disconnect message in the server. Again, if I use the NodePort, it works.

Some application we have can use up to 5000 concurrent steam.

### What did you see instead?

1) Limit of 250 of concurrent stream
2) No disconnect message in the server when the client is closed

### What version of Traefik are you using?

traefik v2.4.8 with helm chart traefik-9.18.2

### What is your environment & configuration?

```yaml
ports:                                                                                                                                                                                                            
  grpc:                                                                                                                                                                                                  
    port: 50051                                                                                                                                                                                                    
    protocol: TCP                                                                                                                                                                                                  
    expose: true                                                                                                                                                                                                   
    exposedPort: 50051
```

The rest is default from the helm chart.

We use kubernetes with k3s 1.21 deployed on ubuntu 20.04.


### If applicable, please paste the log output in DEBUG level

_No response_


Discussion / hints:

Fix patch:
diff --git a/docs/content/reference/static-configuration/cli-ref.md b/docs/content/reference/static-configuration/cli-ref.md
index 9345cded48..84296d2cca 100644
--- a/docs/content/reference/static-configuration/cli-ref.md
+++ b/docs/content/reference/static-configuration/cli-ref.md
@@ -147,8 +147,11 @@ Subject alternative names.
 `--entrypoints.<name>.http.tls.options`:  
 Default TLS options for the routers linked to the entry point.
 
+`--entrypoints.<name>.http2.maxconcurrentstreams`:  
+Specifies the number of concurrent streams per connection that each client is allowed to initiate. (Default: ```250```)
+
 `--entrypoints.<name>.http3`:  
-HTTP3 configuration. (Default: ```false```)
+HTTP/3 configuration. (Default: ```false```)
 
 `--entrypoints.<name>.http3.advertisedport`:  
 UDP port to advertise, on which HTTP/3 is available. (Default: ```0```)
diff --git a/docs/content/reference/static-configuration/env-ref.md b/docs/content/reference/static-configuration/env-ref.md
index 3409174002..bd3d96f981 100644
--- a/docs/content/reference/static-configuration/env-ref.md
+++ b/docs/content/reference/static-configuration/env-ref.md
@@ -114,8 +114,11 @@ Trust only forwarded headers from selected IPs.
 `TRAEFIK_ENTRYPOINTS_<NAME>_HTTP`:  
 HTTP configuration.
 
+`TRAEFIK_ENTRYPOINTS_<NAME>_HTTP2_MAXCONCURRENTSTREAMS`:  
+Specifies the number of concurrent streams per connection that each client is allowed to initiate. (Default: ```250```)
+
 `TRAEFIK_ENTRYPOINTS_<NAME>_HTTP3`:  
-HTTP3 configuration. (Default: ```false```)
+HTTP/3 configuration. (Default: ```false```)
 
 `TRAEFIK_ENTRYPOINTS_<NAME>_HTTP3_ADVERTISEDPORT`:  
 UDP port to advertise, on which HTTP/3 is available. (Default: ```0```)
diff --git a/docs/content/reference/static-configuration/file.toml b/docs/content/reference/static-configuration/file.toml
index f84a975d6c..96a70ee682 100644
--- a/docs/content/reference/static-configuration/file.toml
+++ b/docs/content/reference/static-configuration/file.toml
@@ -30,6 +30,8 @@
       trustedIPs = ["foobar", "foobar"]
     [entryPoints.EntryPoint0.udp]
       timeout = 42
+    [entryPoints.EntryPoint0.http2]
+      maxConcurrentStreams = 42
     [entryPoints.EntryPoint0.http3]
       advertisedPort = 42
     [entryPoints.EntryPoint0.http]
diff --git a/docs/content/reference/static-configuration/file.yaml b/docs/content/reference/static-configuration/file.yaml
index 260734fd02..a8d84e93e9 100644
--- a/docs/content/reference/static-configuration/file.yaml
+++ b/docs/content/reference/static-configuration/file.yaml
@@ -32,6 +32,8 @@ entryPoints:
       trustedIPs:
       - foobar
       - foobar
+    http2:
+      maxConcurrentStreams: 42
     http3:
       advertisedPort: 42
     udp:
diff --git a/docs/content/routing/entrypoints.md b/docs/content/routing/entrypoints.md
index 2aa6b6faf7..3913c4fb33 100644
--- a/docs/content/routing/entrypoints.md
+++ b/docs/content/routing/entrypoints.md
@@ -100,6 +100,8 @@ They can be defined by using a file (YAML or TOML) or CLI arguments.
     entryPoints:
       name:
         address: ":8888" # same as ":8888/tcp"
+        http2:
+          maxConcurrentStreams: 42
         http3:
           advertisedPort: 8888
         transport:
@@ -127,6 +129,8 @@ They can be defined by using a file (YAML or TOML) or CLI arguments.
     [entryPoints]
       [entryPoints.name]
         address = ":8888" # same as ":8888/tcp"
+        [entryPoints.name.http2]
+          maxConcurrentStreams = 42
         [entryPoints.name.http3]
           advertisedPort = 8888
         [entryPoints.name.transport]
@@ -148,6 +152,7 @@ They can be defined by using a file (YAML or TOML) or CLI arguments.
     ```bash tab="CLI"
     ## Static configuration
     --entryPoints.name.address=:8888 # same as :8888/tcp
+    --entryPoints.name.http2.maxConcurrentStreams=42
     --entryPoints.name.http3.advertisedport=8888
     --entryPoints.name.transport.lifeCycle.requestAcceptGraceTimeout=42
     --entryPoints.name.transport.lifeCycle.graceTimeOut=42
@@ -223,6 +228,32 @@ If both TCP and UDP are wanted for the same port, two entryPoints definitions ar
 
     Full details for how to specify `address` can be found in [net.Listen](https://golang.org/pkg/net/#Listen) (and [net.Dial](https://golang.org/pkg/net/#Dial)) of the doc for go.
 
+### HTTP/2
+
+#### `maxConcurrentStreams`
+
+_Optional, Default=250_
+
+`maxConcurrentStreams` specifies the number of concurrent streams per connection that each client is allowed to initiate.
+The `maxConcurrentStreams` value must be greater than zero.
+
+```yaml tab="File (YAML)"
+entryPoints:
+  foo:
+    http2:
+      maxConcurrentStreams: 250
+```
+
+```toml tab="File (TOML)"
+[entryPoints.foo]
+  [entryPoints.foo.http2]
+    maxConcurrentStreams = 250
+```
+
+```bash tab="CLI"
+--entryPoints.name.http2.maxConcurrentStreams=250
+```
+
 ### HTTP/3
 
 #### `http3`
diff --git a/pkg/config/static/entrypoints.go b/pkg/config/static/entrypoints.go
index 3e027ffb07..f97d78e223 100644
--- a/pkg/config/static/entrypoints.go
+++ b/pkg/config/static/entrypoints.go
@@ -16,7 +16,8 @@ type EntryPoint struct {
 	ProxyProtocol    *ProxyProtocol        `description:"Proxy-Protocol configuration." json:"proxyProtocol,omitempty" toml:"proxyProtocol,omitempty" yaml:"proxyProtocol,omitempty" label:"allowEmpty" file:"allowEmpty" export:"true"`
 	ForwardedHeaders *ForwardedHeaders     `description:"Trust client forwarding headers." json:"forwardedHeaders,omitempty" toml:"forwardedHeaders,omitempty" yaml:"forwardedHeaders,omitempty" export:"true"`
 	HTTP             HTTPConfig            `description:"HTTP configuration." json:"http,omitempty" toml:"http,omitempty" yaml:"http,omitempty" export:"true"`
-	HTTP3            *HTTP3Config          `description:"HTTP3 configuration." json:"http3,omitempty" toml:"http3,omitempty" yaml:"http3,omitempty" label:"allowEmpty" file:"allowEmpty" export:"true"`
+	HTTP2            *HTTP2Config          `description:"HTTP/2 configuration." json:"http2,omitempty" toml:"http2,omitempty" yaml:"http2,omitempty" export:"true"`
+	HTTP3            *HTTP3Config          `description:"HTTP/3 configuration." json:"http3,omitempty" toml:"http3,omitempty" yaml:"http3,omitempty" label:"allowEmpty" file:"allowEmpty" export:"true"`
 	UDP              *UDPConfig            `description:"UDP configuration." json:"udp,omitempty" toml:"udp,omitempty" yaml:"udp,omitempty"`
 }
 
@@ -50,6 +51,8 @@ func (ep *EntryPoint) SetDefaults() {
 	ep.ForwardedHeaders = &ForwardedHeaders{}
 	ep.UDP = &UDPConfig{}
 	ep.UDP.SetDefaults()
+	ep.HTTP2 = &HTTP2Config{}
+	ep.HTTP2.SetDefaults()
 }
 
 // HTTPConfig is the HTTP configuration of an entry point.
@@ -59,6 +62,16 @@ type HTTPConfig struct {
 	TLS          *TLSConfig    `description:"Default TLS configuration for the routers linked to the entry point." json:"tls,omitempty" toml:"tls,omitempty" yaml:"tls,omitempty" label:"allowEmpty" file:"allowEmpty" export:"true"`
 }
 
+// HTTP2Config is the HTTP2 configuration of an entry point.
+type HTTP2Config struct {
+	MaxConcurrentStreams int32 `description:"Specifies the number of concurrent streams per connection that each client is allowed to initiate." json:"maxConcurrentStreams,omitempty" toml:"maxConcurrentStreams,omitempty" yaml:"maxConcurrentStreams,omitempty" export:"true"`
+}
+
+// SetDefaults sets the default values.
+func (c *HTTP2Config) SetDefaults() {
+	c.MaxConcurrentStreams = 250 // https://cs.opensource.google/go/x/net/+/cd36cc07:http2/server.go;l=58
+}
+
 // HTTP3Config is the HTTP3 configuration of an entry point.
 type HTTP3Config struct {
 	AdvertisedPort int `description:"UDP port to advertise, on which HTTP/3 is available." json:"advertisedPort,omitempty" toml:"advertisedPort,omitempty" yaml:"advertisedPort,omitempty" export:"true"`
diff --git a/pkg/server/server_entrypoint_tcp.go b/pkg/server/server_entrypoint_tcp.go
index 41bd85fc1b..0d45dc564e 100644
--- a/pkg/server/server_entrypoint_tcp.go
+++ b/pkg/server/server_entrypoint_tcp.go
@@ -7,6 +7,8 @@ import (
 	stdlog "log"
 	"net"
 	"net/http"
+	"os"
+	"strings"
 	"sync"
 	"syscall"
 	"time"
@@ -507,6 +509,10 @@ type httpServer struct {
 }
 
 func createHTTPServer(ctx context.Context, ln net.Listener, configuration *static.EntryPoint, withH2c bool, reqDecorator *requestdecorator.RequestDecorator) (*httpServer, error) {
+	if configuration.HTTP2.MaxConcurrentStreams < 0 {
+		return nil, errors.New("max concurrent streams value must be greater than or equal to zero")
+	}
+
 	httpSwitcher := middlewares.NewHandlerSwitcher(router.BuildDefaultHTTPRouter())
 
 	next, err := alice.New(requestdecorator.WrapHandler(reqDecorator)).Then(httpSwitcher)
@@ -524,7 +530,9 @@ func createHTTPServer(ctx context.Context, ln net.Listener, configuration *stati
 	}
 
 	if withH2c {
-		handler = h2c.NewHandler(handler, &http2.Server{})
+		handler = h2c.NewHandler(handler, &http2.Server{
+			MaxConcurrentStreams: uint32(configuration.HTTP2.MaxConcurrentStreams),
+		})
 	}
 
 	serverHTTP := &http.Server{
@@ -535,6 +543,20 @@ func createHTTPServer(ctx context.Context, ln net.Listener, configuration *stati
 		IdleTimeout:  time.Duration(configuration.Transport.RespondingTimeouts.IdleTimeout),
 	}
 
+	// ConfigureServer configures HTTP/2 with the MaxConcurrentStreams option for the given server.
+	// Also keeping behavior the same as
+	// https://cs.opensource.google/go/go/+/refs/tags/go1.17.7:src/net/http/server.go;l=3262
+	if !strings.Contains(os.Getenv("GODEBUG"), "http2server=0") {
+		err = http2.ConfigureServer(serverHTTP, &http2.Server{
+			MaxConcurrentStreams: uint32(configuration.HTTP2.MaxConcurrentStreams),
+			NewWriteScheduler:    func() http2.WriteScheduler { return http2.NewPriorityWriteScheduler(nil) },
+		})
+
+		if err != nil {
+			return nil, fmt.Errorf("configure HTTP/2 server: %w", err)
+		}
+	}
+
 	listener := newHTTPForwarder(ln)
 	go func() {
 		err := serverHTTP.Serve(listener)


Test patch:
diff --git a/pkg/server/server_entrypoint_tcp_http3_test.go b/pkg/server/server_entrypoint_tcp_http3_test.go
index bf863653a6..6544dd2e04 100644
--- a/pkg/server/server_entrypoint_tcp_http3_test.go
+++ b/pkg/server/server_entrypoint_tcp_http3_test.go
@@ -88,6 +88,7 @@ func TestHTTP3AdvertisedPort(t *testing.T) {
 		Address:          "127.0.0.1:8090",
 		Transport:        epConfig,
 		ForwardedHeaders: &static.ForwardedHeaders{},
+		HTTP2:            &static.HTTP2Config{},
 		HTTP3: &static.HTTP3Config{
 			AdvertisedPort: 8080,
 		},
diff --git a/pkg/server/server_entrypoint_tcp_test.go b/pkg/server/server_entrypoint_tcp_test.go
index 1477629a7b..73cc9c28b4 100644
--- a/pkg/server/server_entrypoint_tcp_test.go
+++ b/pkg/server/server_entrypoint_tcp_test.go
@@ -83,6 +83,7 @@ func testShutdown(t *testing.T, router *tcprouter.Router) {
 		Address:          "127.0.0.1:0",
 		Transport:        epConfig,
 		ForwardedHeaders: &static.ForwardedHeaders{},
+		HTTP2:            &static.HTTP2Config{},
 	}, nil)
 	require.NoError(t, err)
 
@@ -166,6 +167,7 @@ func TestReadTimeoutWithoutFirstByte(t *testing.T) {
 		Address:          ":0",
 		Transport:        epConfig,
 		ForwardedHeaders: &static.ForwardedHeaders{},
+		HTTP2:            &static.HTTP2Config{},
 	}, nil)
 	require.NoError(t, err)
 
@@ -202,6 +204,7 @@ func TestReadTimeoutWithFirstByte(t *testing.T) {
 		Address:          ":0",
 		Transport:        epConfig,
 		ForwardedHeaders: &static.ForwardedHeaders{},
+		HTTP2:            &static.HTTP2Config{},
 	}, nil)
 	require.NoError(t, err)