Repository: traefik/traefik (language: go)

Issue:
Type conversion for Hijacker in custom plugins
### Welcome!

- [X] Yes, I've searched similar issues on [GitHub](https://github.com/traefik/traefik/issues) and didn't find any.
- [X] Yes, I've searched similar issues on the [Traefik community forum](https://community.containo.us) and didn't find any.

### What did you do?

I've implemented a custom middleware that simply wraps a `http.ResponseWriter`. It's main purpose is to investigate how we can handle websockets using middlewares.

The code is available [here](https://github.com/fdevillard/plugin). I've written a test which _validates_ that the `ResponseWriter` can indeed be hijacked (or, at least tries to validate it 😅). The test passes.

When used, I expect no changes in the requests/responses.

### What did you see instead?

The middleware panics with the following logs:

```
{"level":"error","module":"github.com/fdevillard/plugin","msg":"plugins-local/src/github.com/fdevillard/plugin/main.go:21:8: panic","plugin":"plugin-audit","time":"2022-01-06T15:42:54Z"}
{"level":"error","middlewareName":"traefik-internal-recovery","middlewareType":"Recovery","msg":"Recovered from panic in HTTP handler [172.20.0.27:36380 - /graphql]: interface conversion: stdlib._net_http_ResponseWriter is not http.Hijacker: missing method Hijack","time":"2022-01-06T15:42:54Z"}
```

I don't get why it fails to detect the `Hijack` function 🤔 Is it related to Yaegi?

Furthermore, from a frontend perspective, the websocket requests fail.

### What version of Traefik are you using?

Version:      2.5.6
Codename:     brie
Go version:   go1.17.5
Built:        2021-12-22T16:30:52Z
OS/Arch:      linux/amd64

### What is your environment & configuration?

Traefik runs on a kubernetes cluster and is setup with CRDs.

The plugin is shipped in the docker image and enabled using:
```
 --experimental.localPlugins.audit.moduleName=github.com/fdevillard/plugin                                                                                                                                            
```


### If applicable, please paste the log output in DEBUG level

_No response_


Discussion / hints:
Hi there,

Is there any further progress on this issue?
@kevinat this was confirmed as a bug since the current plugin mechanism lacks the Hijack interface exposure to allow plugins to use it. Unfortunately we're focused on other priorities at this moment and won't be able to handle it thus it was set as a P3. 

Of course if someone on the community is willing to contribute a fix that would be awesome.
I have dug into this a little bit to see where a patch would be needed. I can see lots of internal middlewares that have the `Hijack` interface, but I am having a hard time seeing how `http.Hijack()` is called / passed through to these middlewares.

Everything I have found about internal middlewares vs. plugins seems consistent (i.e. building up the `alice` chain, etc.). Is there something about the build process for plugins that would need to be patched? Or is there something else I am missing? Unfortunately the error is not super helpful for determining where this "missing method `Hijack`" message is coming from.

If anyone has any pointers on where to look, it would be most welcome 😄 

**EDIT:** Current understanding: middlewares have the `handler` [passed to them identically](https://github.com/traefik/traefik/blob/3c1d5e0393e91ce977ce29d1176c1303d7e6dbf7/pkg/server/middleware/middlewares.go#L358-L360) whether internal (and work with `Hijack()`) or a plugin, so it does not seem to be the case that the `responseWriter` passed by traefik in `middleware.ServeHTTP()` is missing the `Hijack()` method. 

Instead, my suspicion is that in the build / reflection process where the `plugin.New()` method is called by yaegi and then types are converted to `http.Handler()` that the method is lost or some wrapper injected into this process is not implementing `Hijack()` properly.

https://github.com/traefik/traefik/blob/3c1d5e0393e91ce977ce29d1176c1303d7e6dbf7/pkg/plugins/middlewares.go#L60-L78

Other worthwhile notes from the `net/http` source. Not sure if HTTP/2 is involved here at all:

> // The Hijacker interface is implemented by ResponseWriters that allow
> // an HTTP handler to take over the connection.
> //
> // The default ResponseWriter for HTTP/1.x connections supports
> // Hijacker, but HTTP/2 connections intentionally do not.
> // ResponseWriter wrappers may also not support Hijacker. Handlers
> // should always test for this ability at runtime.

Also, I was able to generate a stack trace for reference, which taught me about `pipelining` (where the `Hijack()` method lives). Unfortunately, my example is complicated enough that I have not done myself any favors.
<details>
<pre>
time="2022-06-30T03:34:40Z" level=debug msg="Error panic! net/http: abort Handler" plugin=plugin-rewriteHeaders module=github.com/colearendt/traefik-plugin-rewrite-headers
goroutine 486 [running]:
runtime/debug.Stack()
        /usr/local/golang/1.10.8/go/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
        /usr/local/golang/1.10.8/go/src/runtime/debug/stack.go:16 +0x19
reflect.Value.call({0x2cd29a0, 0x36d87f8, 0x4684d9}, {0x355d780, 0x4}, {0x5d26650, 0x0, 0x4af505})
        /usr/local/golang/1.10.8/go/src/reflect/value.go:543 +0x814
reflect.Value.Call({0x2cd29a0, 0x36d87f8, 0x4517f2}, {0x5d26650, 0x0, 0x0})
        /usr/local/golang/1.10.8/go/src/reflect/value.go:339 +0xc5
github.com/traefik/yaegi/interp.callBin.func1({0x2cd29a0, 0x36d87f8, 0x4af505}, {0x5d26650, 0xc000b2a5a0, 0xc000ef3308})
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:1472 +0x28
github.com/traefik/yaegi/interp.callBin.func10(0xc000f2c790)
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:1646 +0x158
github.com/traefik/yaegi/interp.runCfg(0xc00084eb40, 0xc000f2c790, 0x0, 0x0)
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:201 +0x2ac
github.com/traefik/yaegi/interp.genFunctionWrapper.func2.1({0x5d26650, 0x0, 0x0})
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:1023 +0x4a5
reflect.Value.call({0x2cd29a0, 0xc000af4a80, 0x0}, {0x355d780, 0x4}, {0xc000536a68, 0x0, 0xc000ef4178})
        /usr/local/golang/1.10.8/go/src/reflect/value.go:543 +0x814
reflect.Value.Call({0x2cd29a0, 0xc000af4a80, 0x5}, {0xc000536a68, 0x0, 0x0})
        /usr/local/golang/1.10.8/go/src/reflect/value.go:339 +0xc5
github.com/traefik/yaegi/interp.runCfg.func1()
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:184 +0x1a7
panic({0x2e8d960, 0xc000126a90})
        /usr/local/golang/1.10.8/go/src/runtime/panic.go:1038 +0x215
net/http/httputil.(*ReverseProxy).ServeHTTP(0xc0010aaaa0, {0x3b30fc0, 0xc000b81a60}, 0xc000e1fe00)
        /usr/local/golang/1.10.8/go/src/net/http/httputil/reverseproxy.go:348 +0xeba
github.com/traefik/traefik/v2/pkg/middlewares/pipelining.(*pipelining).ServeHTTP(0xc001171890, {0x7f727ab2d4f8, 0xc000536e88}, 0xc000e1fe00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/pipelining/pipelining.go:36 +0xc9
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.AddServiceFields({0x7f727ab2d4f8, 0xc000536e88}, 0xc000e1fe00, {0x3afe0c0, 0xc001171890}, 0xc0010afb80)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/accesslog/field_middleware.go:47 +0x110
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0xc0010c4300, {0x7f727ab2d4f8, 0xc000536e88}, 0xc000ed8da0)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/accesslog/field_middleware.go:36 +0xd8
github.com/traefik/traefik/v2/pkg/middlewares/metrics.(*metricsMiddleware).ServeHTTP(0xc0010c11f0, {0x7f727ab2bc60, 0xc000af51a0}, 0xc000e1fe00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/metrics/metrics.go:121 +0x722
github.com/vulcand/oxy/roundrobin.(*RoundRobin).ServeHTTP(0xc0010c1490, {0x7f727ab2bc60, 0xc000af51a0}, 0xc000e1fc00)
        /home/semaphore/go/pkg/mod/github.com/vulcand/oxy@v1.3.0/roundrobin/rr.go:147 +0x52c
github.com/traefik/traefik/v2/pkg/middlewares/emptybackendhandler.(*emptyBackend).ServeHTTP(0xc001171b90, {0x7f727ab2bc60, 0xc000af51a0}, 0x4684d9)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/emptybackendhandler/empty_backend_handler.go:25 +0xfd
github.com/traefik/traefik/v2/pkg/middlewares/tracing.(*forwarderMiddleware).ServeHTTP(0xc0006e8cc0, {0x7f727ab2bc60, 0xc000af51a0}, 0xc000e1fc00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/tracing/forwarder.go:38 +0x47d
reflect.Value.call({0x2f9cd60, 0xc0010c46d0, 0xc000aedc60}, {0x355d780, 0x4}, {0xc000af4b70, 0x2, 0xc00085a000})
        /usr/local/golang/1.10.8/go/src/reflect/value.go:543 +0x814
reflect.Value.Call({0x2f9cd60, 0xc0010c46d0, 0xc0010e85a0}, {0xc000af4b70, 0x2, 0x2})
        /usr/local/golang/1.10.8/go/src/reflect/value.go:339 +0xc5
github.com/traefik/yaegi/interp.callBin.func1({0x2f9cd60, 0xc0010c46d0, 0x4ac5c6}, {0xc000af4b70, 0xc0010c46c0, 0x35731e6})
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:1472 +0x28
github.com/traefik/yaegi/interp.callBin.func10(0xc000e09550)
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:1646 +0x158
github.com/traefik/yaegi/interp.runCfg(0xc00084e120, 0xc000e09550, 0x39, 0x3453a60)
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:201 +0x2ac
github.com/traefik/yaegi/interp.genFunctionWrapper.func2.1({0xc000af4a20, 0x2, 0x4})
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:1023 +0x4a5
github.com/traefik/yaegi/stdlib._net_http_Handler.ServeHTTP(...)
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/stdlib/go1_17_net_http.go:288
github.com/traefik/traefik/v2/pkg/middlewares/stripprefix.(*stripPrefix).serveRequest(0xc0010c4800, {0x3b30f90, 0xc00117f040}, 0xc000e1fc00, {0xc000afc17c, 0x4})
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/stripprefix/strip_prefix.go:61 +0x15c
github.com/traefik/traefik/v2/pkg/middlewares/stripprefix.(*stripPrefix).ServeHTTP(0xc0010c4800, {0x3b30f90, 0xc00117f040}, 0xc000e1fc00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/stripprefix/strip_prefix.go:51 +0x1b6
github.com/traefik/traefik/v2/pkg/middlewares/tracing.(*Wrapper).ServeHTTP(0xc000874fc0, {0x3b30f90, 0xc00117f040}, 0xc000e1fc00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/tracing/wrapper.go:57 +0x106
github.com/traefik/traefik/v2/pkg/middlewares/headers.(*Header).ServeHTTP(0xc0010c48c0, {0x3b3eb80, 0xc00117eff0}, 0xc000e1fc00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/headers/header.go:64 +0x15d
github.com/traefik/traefik/v2/pkg/middlewares/connectionheader.Remover.func1({0x3b3eb80, 0xc00117eff0}, 0xc000e1fc00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/connectionheader/connectionheader.go:34 +0x2a4
net/http.HandlerFunc.ServeHTTP(0x2928687, {0x3b3eb80, 0xc00117eff0}, 0x36d1798)
        /usr/local/golang/1.10.8/go/src/net/http/server.go:2047 +0x2f
github.com/traefik/traefik/v2/pkg/middlewares/headers.(*headers).ServeHTTP(0x3b64980, {0x3b3eb80, 0xc00117eff0}, 0x0)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/headers/headers.go:91 +0x2c
github.com/traefik/traefik/v2/pkg/middlewares/tracing.(*Wrapper).ServeHTTP(0xc000875980, {0x3b3eb80, 0xc00117eff0}, 0xc000e1fc00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/tracing/wrapper.go:57 +0x106
github.com/traefik/traefik/v2/pkg/middlewares/headers.(*Header).ServeHTTP(0xc0010c4980, {0x7f727aaf4e48, 0xc000aeda70}, 0xc000e1fc00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/headers/header.go:64 +0x15d
github.com/traefik/traefik/v2/pkg/middlewares/connectionheader.Remover.func1({0x7f727aaf4e48, 0xc000aeda70}, 0xc000e1fc00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/connectionheader/connectionheader.go:34 +0x2a4
net/http.HandlerFunc.ServeHTTP(0x2928687, {0x7f727aaf4e48, 0xc000aeda70}, 0x36d1798)
        /usr/local/golang/1.10.8/go/src/net/http/server.go:2047 +0x2f
github.com/traefik/traefik/v2/pkg/middlewares/headers.(*headers).ServeHTTP(0x3b64980, {0x7f727aaf4e48, 0xc000aeda70}, 0x4)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/headers/headers.go:91 +0x2c
github.com/traefik/traefik/v2/pkg/middlewares/tracing.(*Wrapper).ServeHTTP(0xc000cb23f0, {0x7f727aaf4e48, 0xc000aeda70}, 0xc000e1fc00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/tracing/wrapper.go:57 +0x106
reflect.Value.call({0x2f9cd60, 0xc0010c4c10, 0xc000b80a20}, {0x355d780, 0x4}, {0xc000af49f0, 0x2, 0x7f727abcd6a8})
        /usr/local/golang/1.10.8/go/src/reflect/value.go:543 +0x814
reflect.Value.Call({0x2f9cd60, 0xc0010c4c10, 0x4517f2}, {0xc000af49f0, 0x2, 0x2})
        /usr/local/golang/1.10.8/go/src/reflect/value.go:339 +0xc5
github.com/traefik/yaegi/interp.callBin.func1({0x2f9cd60, 0xc0010c4c10, 0x4ac5c6}, {0xc000af49f0, 0xc0010c4c00, 0x4ae4dc})
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:1472 +0x28
github.com/traefik/yaegi/interp.callBin.func10(0xc000e094a0)
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:1646 +0x158
github.com/traefik/yaegi/interp.runCfg(0xc00098b8c0, 0xc000e094a0, 0x39, 0x3453a60)
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:201 +0x2ac
github.com/traefik/yaegi/interp.genFunctionWrapper.func2.1({0xc000af4210, 0x2, 0x4})
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:1023 +0x4a5
github.com/traefik/yaegi/stdlib._net_http_Handler.ServeHTTP(...)
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/stdlib/go1_17_net_http.go:288
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0xc0010c4d00, {0x7f727aaf4e48, 0xc000aeda70}, 0x3489840)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/accesslog/field_middleware.go:38 +0xf6
github.com/gorilla/mux.(*Router).ServeHTTP(0xc00056f680, {0x7f727aaf4e48, 0xc000aeda70}, 0xc000e1fa00)
        /home/semaphore/go/pkg/mod/github.com/containous/mux@v0.0.0-20181024131434-c33f32e26898/mux.go:133 +0x270
github.com/traefik/traefik/v2/pkg/middlewares/recovery.(*recovery).ServeHTTP(0xc000e21db8, {0x7f727aaf4e48, 0xc000aeda70}, 0x2f935e0)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/recovery/recovery.go:32 +0x82
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.AddOriginFields({0x7f727aaf4e98, 0xc000aeda68}, 0x357d4a1, {0x3afe100, 0xc000f28d70}, 0xc0010afb80)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/accesslog/field_middleware.go:55 +0x1a2
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0xc0011022c0, {0x7f727aaf4e98, 0xc000aeda68}, 0x3ab5901)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/accesslog/field_middleware.go:36 +0xd8
github.com/traefik/traefik/v2/pkg/middlewares/requestdecorator.(*RequestDecorator).ServeHTTP(0xc00095db00, {0x7f727aaf4e98, 0xc000aeda68}, 0xc000e1f900, 0xc000ef74d0)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/requestdecorator/request_decorator.go:47 +0x57b
github.com/traefik/traefik/v2/pkg/middlewares/requestdecorator.WrapHandler.func1.1({0x7f727aaf4e98, 0xc000aeda68}, 0xc000aeda68)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/requestdecorator/request_decorator.go:84 +0x68
net/http.HandlerFunc.ServeHTTP(0xc0009f6630, {0x7f727aaf4e98, 0xc000aeda68}, 0xc0011979c0)
        /usr/local/golang/1.10.8/go/src/net/http/server.go:2047 +0x2f
github.com/traefik/traefik/v2/pkg/middlewares/metrics.(*metricsMiddleware).ServeHTTP(0xc001105260, {0x7f727aaf4e48, 0xc000aeda60}, 0xc000e1f900)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/metrics/metrics.go:121 +0x722
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.(*Handler).ServeHTTP(0xc00075a7e0, {0x3b3e070, 0xc0013988c0}, 0xc000e1f800, {0x3afe080, 0xc001105260})
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/accesslog/logger.go:227 +0xfa3
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.WrapHandler.func1.1({0x3b3e070, 0xc0013988c0}, 0xc000ef7800)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/accesslog/logger.go:71 +0x3b
net/http.HandlerFunc.ServeHTTP(0xc0009f9fa0, {0x3b3e070, 0xc0013988c0}, 0xc0008a3600)
        /usr/local/golang/1.10.8/go/src/net/http/server.go:2047 +0x2f
github.com/traefik/traefik/v2/pkg/middlewares.(*HTTPHandlerSwitcher).ServeHTTP(0xc00009ed70, {0x3b3e070, 0xc0013988c0}, 0x9)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/handler_switcher.go:23 +0x62
github.com/traefik/traefik/v2/pkg/middlewares/forwardedheaders.(*XForwarded).ServeHTTP(0xc00009ed70, {0x3b3e070, 0xc0013988c0}, 0xc000e1f800)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/forwardedheaders/forwarded_header.go:189 +0xca
golang.org/x/net/http2/h2c.h2cHandler.ServeHTTP({{0x3afdfc0, 0xc00009ed70}, 0xc000339480}, {0x3b3e070, 0xc0013988c0}, 0xc000e1f800)
        /home/semaphore/go/pkg/mod/golang.org/x/net@v0.0.0-20211209124913-491a49abca63/http2/h2c/h2c.go:104 +0x3e4
net/http.serverHandler.ServeHTTP({0xc0009db2f0}, {0x3b3e070, 0xc0013988c0}, 0xc000e1f800)
        /usr/local/golang/1.10.8/go/src/net/http/server.go:2879 +0x43b
net/http.(*conn).serve(0xc0005f94a0, {0x3b64980, 0xc00066e1b0})
        /usr/local/golang/1.10.8/go/src/net/http/server.go:1930 +0xb08
created by net/http.(*Server).Serve
        /usr/local/golang/1.10.8/go/src/net/http/server.go:3034 +0x4e8
</pre>
</details>

A better stack trace:
<details>
<pre>
goroutine 4968 [running]:
reflect.Value.call({0x2cd29a0, 0x36d87f8, 0x4684d9}, {0x355d780, 0x4}, {0x5d26650, 0x0, 0x4af505})
        /usr/local/golang/1.10.8/go/src/reflect/value.go:543 +0x814
reflect.Value.Call({0x2cd29a0, 0x36d87f8, 0x4517f2}, {0x5d26650, 0x0, 0x0})
        /usr/local/golang/1.10.8/go/src/reflect/value.go:339 +0xc5
github.com/traefik/yaegi/interp.callBin.func1({0x2cd29a0, 0x36d87f8, 0x4af505}, {0x5d26650, 0xc00129d1f0, 0xc001257068})
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:1472 +0x28
github.com/traefik/yaegi/interp.callBin.func10(0xc00136e840)
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:1646 +0x158
github.com/traefik/yaegi/interp.runCfg(0xc00084eb40, 0xc00136e840, 0x0, 0x0)
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:201 +0x2ac
github.com/traefik/yaegi/interp.genFunctionWrapper.func2.1({0x5d26650, 0x0, 0x0})
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:1023 +0x4a5
reflect.Value.call({0x2cd29a0, 0xc0014fc630, 0x0}, {0x355d780, 0x4}, {0xc00157d968, 0x0, 0xc001257ed8})
        /usr/local/golang/1.10.8/go/src/reflect/value.go:543 +0x814
reflect.Value.Call({0x2cd29a0, 0xc0014fc630, 0x5}, {0xc00157d968, 0x0, 0x0})
        /usr/local/golang/1.10.8/go/src/reflect/value.go:339 +0xc5
github.com/traefik/yaegi/interp.runCfg.func1()
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:184 +0x1a7
panic({0x2f79de0, 0xc0014fcf30})
        /usr/local/golang/1.10.8/go/src/runtime/panic.go:1038 +0x215
github.com/traefik/traefik/v2/pkg/middlewares/metrics.(*responseRecorder).Hijack(0xc0012581e8)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/metrics/recorder.go:55 +0x2c
github.com/traefik/traefik/v2/pkg/middlewares/pipelining.(*writerWithoutCloseNotify).Hijack(0x0)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/pipelining/pipelining.go:69 +0x37
net/http/httputil.(*ReverseProxy).handleUpgradeResponse(0xc000ec79f0, {0x3b30fc0, 0xc00129cff0}, 0xc001783200, 0xc000facf30)
        /usr/local/golang/1.10.8/go/src/net/http/httputil/reverseproxy.go:589 +0x595
net/http/httputil.(*ReverseProxy).ServeHTTP(0xc000ec79f0, {0x3b30fc0, 0xc00129cff0}, 0xc001783100)
        /usr/local/golang/1.10.8/go/src/net/http/httputil/reverseproxy.go:309 +0x9d6
github.com/traefik/traefik/v2/pkg/middlewares/pipelining.(*pipelining).ServeHTTP(0xc00168f4c0, {0x7f727ab2d4f8, 0xc00157de78}, 0xc001783100)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/pipelining/pipelining.go:36 +0xc9
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.AddServiceFields({0x7f727ab2d4f8, 0xc00157de78}, 0xc001783100, {0x3afe0c0, 0xc00168f4c0}, 0xc0016a9d00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/accesslog/field_middleware.go:47 +0x110
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0xc000096640, {0x7f727ab2d4f8, 0xc00157de78}, 0xc0011b11c0)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/accesslog/field_middleware.go:36 +0xd8
github.com/traefik/traefik/v2/pkg/middlewares/metrics.(*metricsMiddleware).ServeHTTP(0xc000df18f0, {0x7f727ab2bc60, 0xc0014fc900}, 0xc001783100)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/metrics/metrics.go:121 +0x722
github.com/vulcand/oxy/roundrobin.(*RoundRobin).ServeHTTP(0xc000df1b90, {0x7f727ab2bc60, 0xc0014fc900}, 0xc001782f00)
        /home/semaphore/go/pkg/mod/github.com/vulcand/oxy@v1.3.0/roundrobin/rr.go:147 +0x52c
github.com/traefik/traefik/v2/pkg/middlewares/emptybackendhandler.(*emptyBackend).ServeHTTP(0xc00168f7c0, {0x7f727ab2bc60, 0xc0014fc900}, 0x4684d9)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/emptybackendhandler/empty_backend_handler.go:25 +0xfd
github.com/traefik/traefik/v2/pkg/middlewares/tracing.(*forwarderMiddleware).ServeHTTP(0xc00167b3e0, {0x7f727ab2bc60, 0xc0014fc900}, 0xc001782f00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/tracing/forwarder.go:38 +0x47d
reflect.Value.call({0x2f9cd60, 0xc000096d90, 0xc000aed5b8}, {0x355d780, 0x4}, {0xc0014fc6f0, 0x2, 0xc00085a000})
        /usr/local/golang/1.10.8/go/src/reflect/value.go:543 +0x814
reflect.Value.Call({0x2f9cd60, 0xc000096d90, 0xc00109b440}, {0xc0014fc6f0, 0x2, 0x2})
        /usr/local/golang/1.10.8/go/src/reflect/value.go:339 +0xc5
github.com/traefik/yaegi/interp.callBin.func1({0x2f9cd60, 0xc000096d90, 0x4ac5c6}, {0xc0014fc6f0, 0xc000096d80, 0x35731e6})
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:1472 +0x28
github.com/traefik/yaegi/interp.callBin.func10(0xc00136e4d0)
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:1646 +0x158
github.com/traefik/yaegi/interp.runCfg(0xc00084e120, 0xc00136e4d0, 0x39, 0x3453a60)
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:201 +0x2ac
github.com/traefik/yaegi/interp.genFunctionWrapper.func2.1({0xc0014fc5d0, 0x2, 0x4})
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:1023 +0x4a5
github.com/traefik/yaegi/stdlib._net_http_Handler.ServeHTTP(...)
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/stdlib/go1_17_net_http.go:288
github.com/traefik/traefik/v2/pkg/middlewares/stripprefix.(*stripPrefix).serveRequest(0xc000096f00, {0x3b30f90, 0xc0010bad20}, 0xc001782f00, {0xc000afc17c, 0x4})
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/stripprefix/strip_prefix.go:61 +0x15c
github.com/traefik/traefik/v2/pkg/middlewares/stripprefix.(*stripPrefix).ServeHTTP(0xc000096f00, {0x3b30f90, 0xc0010bad20}, 0xc001782f00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/stripprefix/strip_prefix.go:51 +0x1b6
github.com/traefik/traefik/v2/pkg/middlewares/tracing.(*Wrapper).ServeHTTP(0xc000874b70, {0x3b30f90, 0xc0010bad20}, 0xc001782f00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/tracing/wrapper.go:57 +0x106
github.com/traefik/traefik/v2/pkg/middlewares/headers.(*Header).ServeHTTP(0xc000096fc0, {0x3b3eb80, 0xc0010bacd0}, 0xc001782f00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/headers/header.go:64 +0x15d
github.com/traefik/traefik/v2/pkg/middlewares/connectionheader.Remover.func1({0x3b3eb80, 0xc0010bacd0}, 0xc001782f00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/connectionheader/connectionheader.go:34 +0x2a4
net/http.HandlerFunc.ServeHTTP(0x2928687, {0x3b3eb80, 0xc0010bacd0}, 0x36d1798)
        /usr/local/golang/1.10.8/go/src/net/http/server.go:2047 +0x2f
github.com/traefik/traefik/v2/pkg/middlewares/headers.(*headers).ServeHTTP(0x3b64980, {0x3b3eb80, 0xc0010bacd0}, 0x0)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/headers/headers.go:91 +0x2c
github.com/traefik/traefik/v2/pkg/middlewares/tracing.(*Wrapper).ServeHTTP(0xc000875650, {0x3b3eb80, 0xc0010bacd0}, 0xc001782f00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/tracing/wrapper.go:57 +0x106
github.com/traefik/traefik/v2/pkg/middlewares/headers.(*Header).ServeHTTP(0xc000097600, {0x7f727aaf4e48, 0xc000aed358}, 0xc001782f00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/headers/header.go:64 +0x15d
github.com/traefik/traefik/v2/pkg/middlewares/connectionheader.Remover.func1({0x7f727aaf4e48, 0xc000aed358}, 0xc001782f00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/connectionheader/connectionheader.go:34 +0x2a4
net/http.HandlerFunc.ServeHTTP(0x2928687, {0x7f727aaf4e48, 0xc000aed358}, 0x36d1798)
        /usr/local/golang/1.10.8/go/src/net/http/server.go:2047 +0x2f
github.com/traefik/traefik/v2/pkg/middlewares/headers.(*headers).ServeHTTP(0x3b64980, {0x7f727aaf4e48, 0xc000aed358}, 0x4)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/headers/headers.go:91 +0x2c
github.com/traefik/traefik/v2/pkg/middlewares/tracing.(*Wrapper).ServeHTTP(0xc0011de1b0, {0x7f727aaf4e48, 0xc000aed358}, 0xc001782f00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/tracing/wrapper.go:57 +0x106
reflect.Value.call({0x2f9cd60, 0xc00118c050, 0xc00129c4b0}, {0x355d780, 0x4}, {0xc0014fc5a0, 0x2, 0x7f727ab82068})
        /usr/local/golang/1.10.8/go/src/reflect/value.go:543 +0x814
reflect.Value.Call({0x2f9cd60, 0xc00118c050, 0x4517f2}, {0xc0014fc5a0, 0x2, 0x2})
        /usr/local/golang/1.10.8/go/src/reflect/value.go:339 +0xc5
github.com/traefik/yaegi/interp.callBin.func1({0x2f9cd60, 0xc00118c050, 0x4ac5c6}, {0xc0014fc5a0, 0xc00118c040, 0x4ae4dc})
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:1472 +0x28
github.com/traefik/yaegi/interp.callBin.func10(0xc00136e420)
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:1646 +0x158
github.com/traefik/yaegi/interp.runCfg(0xc00098b8c0, 0xc00136e420, 0x39, 0x3453a60)
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:201 +0x2ac
github.com/traefik/yaegi/interp.genFunctionWrapper.func2.1({0xc00120fe30, 0x2, 0x4})
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:1023 +0x4a5
github.com/traefik/yaegi/stdlib._net_http_Handler.ServeHTTP(...)
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/stdlib/go1_17_net_http.go:288
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0xc00118c4c0, {0x7f727aaf4e48, 0xc000aed358}, 0x3489840)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/accesslog/field_middleware.go:38 +0xf6
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000ef25a0, {0x7f727aaf4e48, 0xc000aed358}, 0xc001782c00)
        /home/semaphore/go/pkg/mod/github.com/containous/mux@v0.0.0-20181024131434-c33f32e26898/mux.go:133 +0x270
github.com/traefik/traefik/v2/pkg/middlewares/recovery.(*recovery).ServeHTTP(0xc000fbd4c8, {0x7f727aaf4e48, 0xc000aed358}, 0x2f935e0)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/recovery/recovery.go:32 +0x82
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.AddOriginFields({0x7f727aaf4e98, 0xc000aed350}, 0x357d4a1, {0x3afe100, 0xc000f90e90}, 0xc0016a9d00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/accesslog/field_middleware.go:55 +0x1a2
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0xc000f8d640, {0x7f727aaf4e98, 0xc000aed350}, 0x3ab5901)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/accesslog/field_middleware.go:36 +0xd8
github.com/traefik/traefik/v2/pkg/middlewares/requestdecorator.(*RequestDecorator).ServeHTTP(0xc00095db00, {0x7f727aaf4e98, 0xc000aed350}, 0xc001782b00, 0xc000c7d4d0)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/requestdecorator/request_decorator.go:47 +0x57b
github.com/traefik/traefik/v2/pkg/middlewares/requestdecorator.WrapHandler.func1.1({0x7f727aaf4e98, 0xc000aed350}, 0xc000aed350)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/requestdecorator/request_decorator.go:84 +0x68
net/http.HandlerFunc.ServeHTTP(0xc0009f6630, {0x7f727aaf4e98, 0xc000aed350}, 0xc000af9860)
        /usr/local/golang/1.10.8/go/src/net/http/server.go:2047 +0x2f
github.com/traefik/traefik/v2/pkg/middlewares/metrics.(*metricsMiddleware).ServeHTTP(0xc000fbacb0, {0x7f727aaf4e48, 0xc000aed348}, 0xc001782b00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/metrics/metrics.go:121 +0x722
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.(*Handler).ServeHTTP(0xc00075a7e0, {0x3b3e070, 0xc000c93180}, 0xc001782a00, {0x3afe080, 0xc000fbacb0})
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/accesslog/logger.go:227 +0xfa3
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.WrapHandler.func1.1({0x3b3e070, 0xc000c93180}, 0xc000dd7800)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/accesslog/logger.go:71 +0x3b
net/http.HandlerFunc.ServeHTTP(0xc000dd7940, {0x3b3e070, 0xc000c93180}, 0xc0008a8da0)
        /usr/local/golang/1.10.8/go/src/net/http/server.go:2047 +0x2f
github.com/traefik/traefik/v2/pkg/middlewares.(*HTTPHandlerSwitcher).ServeHTTP(0xc00009ed70, {0x3b3e070, 0xc000c93180}, 0x9)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/handler_switcher.go:23 +0x62
github.com/traefik/traefik/v2/pkg/middlewares/forwardedheaders.(*XForwarded).ServeHTTP(0xc00009ed70, {0x3b3e070, 0xc000c93180}, 0xc001782a00)
        /home/semaphore/go/src/github.com/traefik/traefik/pkg/middlewares/forwardedheaders/forwarded_header.go:189 +0xca
golang.org/x/net/http2/h2c.h2cHandler.ServeHTTP({{0x3afdfc0, 0xc00009ed70}, 0xc000339480}, {0x3b3e070, 0xc000c93180}, 0xc001782a00)
        /home/semaphore/go/pkg/mod/golang.org/x/net@v0.0.0-20211209124913-491a49abca63/http2/h2c/h2c.go:104 +0x3e4
net/http.serverHandler.ServeHTTP({0xc00120faa0}, {0x3b3e070, 0xc000c93180}, 0xc001782a00)
        /usr/local/golang/1.10.8/go/src/net/http/server.go:2879 +0x43b
net/http.(*conn).serve(0xc001201180, {0x3b64980, 0xc00066e1b0})
        /usr/local/golang/1.10.8/go/src/net/http/server.go:1930 +0xb08
created by net/http.(*Server).Serve
        /usr/local/golang/1.10.8/go/src/net/http/server.go:3034 +0x4e8

</pre>
</details>
The better stack trace gave me a lead (panic in the `metrics` middleware) that I have patched, built, and tested with. Now the error messages are cleaner, although harder to tell where they are coming from:
```
time="2022-06-30T08:08:49Z" level=debug msg="'500 Internal Server Error' caused by: Hijack failed on protocol switch: stdlib._net_http_ResponseWriter is not a http.Hijacker"
```

Ok, so I am fairly confident that that message is coming from the `metrics` middleware, so however it is getting initialized is not handling `Hijack()` properly (hence the earlier panic). There are still a handful of `httputil.ReverseProxy` panics that I cannot tell whether they are related or not. However, I disabled metrics and have landed on another stack trace:

<details>
<pre>
time="2022-06-30T08:59:07Z" level=debug msg="Error panic! interface conversion: stdlib._net_http_ResponseWriter is not http.Hijacker: missing method Hijack" plugin=plugin-rewriteHeaders module=github.com/colearendt/traefik-plugin-rewrite-headers
goroutine 426 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
        /usr/local/go/src/runtime/debug/stack.go:16 +0x19
reflect.Value.call({0x2fb6000, 0x3e119d0, 0x468839}, {0x390d2f9, 0x4}, {0x63097b8, 0x0, 0x4af9a5})
        /usr/local/go/src/reflect/value.go:556 +0x845
reflect.Value.Call({0x2fb6000, 0x3e119d0, 0x451b32}, {0x63097b8, 0x0, 0x0})
        /usr/local/go/src/reflect/value.go:339 +0xc5
github.com/traefik/yaegi/interp.callBin.func2({0x2fb6000, 0x3e119d0, 0xc001478228}, {0x63097b8, 0xc00140dfb0, 0xc00140def0})
        /go/pkg/mod/github.com/traefik/yaegi@v0.13.0/interp/run.go:1497 +0x28
github.com/traefik/yaegi/interp.callBin.func11(0xc001454b00)
        /go/pkg/mod/github.com/traefik/yaegi@v0.13.0/interp/run.go:1677 +0x15f
github.com/traefik/yaegi/interp.runCfg(0xc0008386c0, 0xc001454b00, 0x0, 0x0)
        /go/pkg/mod/github.com/traefik/yaegi@v0.13.0/interp/run.go:200 +0x2ac
github.com/traefik/yaegi/interp.genFunctionWrapper.func2.1({0x63097b8, 0x0, 0x0})
        /go/pkg/mod/github.com/traefik/yaegi@v0.13.0/interp/run.go:1022 +0x4a5
reflect.Value.call({0x2fb6000, 0xc001273e30, 0xc001464310}, {0x390d2f9, 0x4}, {0xc00127d380, 0x0, 0xc001479068})
        /usr/local/go/src/reflect/value.go:556 +0x845
reflect.Value.Call({0x2fb6000, 0xc001273e30, 0x5}, {0xc00127d380, 0x0, 0x0})
        /usr/local/go/src/reflect/value.go:339 +0xc5
github.com/traefik/yaegi/interp.runCfg.func1()
        /go/pkg/mod/github.com/traefik/yaegi@v0.13.0/interp/run.go:183 +0x1a7
panic({0x3291020, 0xc001407da0})
        /usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/traefik/traefik/v2/pkg/middlewares/pipelining.(*writerWithoutCloseNotify).Hijack(0x0)
        /go/src/github.com/traefik/traefik/pkg/middlewares/pipelining/pipelining.go:69 +0x2c
net/http/httputil.(*ReverseProxy).handleUpgradeResponse(0xc001030ff0, {0x42a9f40, 0xc0012b1030}, 0xc000015600, 0xc001436ea0)
        /usr/local/go/src/net/http/httputil/reverseproxy.go:589 +0x595
net/http/httputil.(*ReverseProxy).ServeHTTP(0xc001030ff0, {0x42a9f40, 0xc0012b1030}, 0xc000015500)
        /usr/local/go/src/net/http/httputil/reverseproxy.go:309 +0x9d6
github.com/traefik/traefik/v2/pkg/middlewares/pipelining.(*pipelining).ServeHTTP(0xc0009bdbf0, {0x7f9d2f8a63b8, 0xc0012ee0c0}, 0xc000015500)
        /go/src/github.com/traefik/traefik/pkg/middlewares/pipelining/pipelining.go:36 +0xc9
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.AddServiceFields({0x7f9d2f8a63b8, 0xc0012ee0c0}, 0xc000015500, {0x4275d20, 0xc0009bdbf0}, 0xc0012a6400)
        /go/src/github.com/traefik/traefik/pkg/middlewares/accesslog/field_middleware.go:47 +0x110
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0xc0003b2580, {0x7f9d2f8a63b8, 0xc0012ee0c0}, 0xa)
        /go/src/github.com/traefik/traefik/pkg/middlewares/accesslog/field_middleware.go:36 +0xd8
github.com/traefik/traefik/v2/pkg/middlewares/emptybackendhandler.(*emptyBackend).ServeHTTP(0xc000e98000, {0x7f9d2f8a63b8, 0xc0012ee0c0}, 0x468839)
        /go/src/github.com/traefik/traefik/pkg/middlewares/emptybackendhandler/empty_backend_handler.go:25 +0xfd
github.com/traefik/traefik/v2/pkg/middlewares/tracing.(*forwarderMiddleware).ServeHTTP(0xc000fc1560, {0x7f9d2f8a63b8, 0xc0012ee0c0}, 0xc000015400)
        /go/src/github.com/traefik/traefik/pkg/middlewares/tracing/forwarder.go:38 +0x47d
reflect.Value.call({0x32bc3e0, 0xc0003b2890, 0x366cfe0}, {0x390d2f9, 0x4}, {0xc001273e90, 0x2, 0x7f9d2f912590})
        /usr/local/go/src/reflect/value.go:556 +0x845
reflect.Value.Call({0x32bc3e0, 0xc0003b2890, 0x451b32}, {0xc001273e90, 0x2, 0x2})
        /usr/local/go/src/reflect/value.go:339 +0xc5
github.com/traefik/yaegi/interp.callBin.func2({0x32bc3e0, 0xc0003b2890, 0xc0011984c8}, {0xc001273e90, 0xc00076e300, 0xc0003b2880})
        /go/pkg/mod/github.com/traefik/yaegi@v0.13.0/interp/run.go:1497 +0x28
github.com/traefik/yaegi/interp.callBin.func11(0xc001251970)
        /go/pkg/mod/github.com/traefik/yaegi@v0.13.0/interp/run.go:1677 +0x15f
github.com/traefik/yaegi/interp.runCfg(0xc000837c20, 0xc001251970, 0x39, 0x37f01a0)
        /go/pkg/mod/github.com/traefik/yaegi@v0.13.0/interp/run.go:200 +0x2ac
github.com/traefik/yaegi/interp.genFunctionWrapper.func2.1({0xc001273dd0, 0x2, 0x4})
        /go/pkg/mod/github.com/traefik/yaegi@v0.13.0/interp/run.go:1022 +0x4a5
github.com/traefik/yaegi/stdlib._net_http_Handler.ServeHTTP(...)
        /go/pkg/mod/github.com/traefik/yaegi@v0.13.0/stdlib/go1_17_net_http.go:288
github.com/traefik/traefik/v2/pkg/middlewares/stripprefix.(*stripPrefix).serveRequest(0xc0003b2b00, {0x42a9f10, 0xc0012ec000}, 0xc000015400, {0xc001164d2c, 0x4})
        /go/src/github.com/traefik/traefik/pkg/middlewares/stripprefix/strip_prefix.go:61 +0x15c
github.com/traefik/traefik/v2/pkg/middlewares/stripprefix.(*stripPrefix).ServeHTTP(0xc0003b2b00, {0x42a9f10, 0xc0012ec000}, 0xc000015400)
        /go/src/github.com/traefik/traefik/pkg/middlewares/stripprefix/strip_prefix.go:51 +0x1b6
github.com/traefik/traefik/v2/pkg/middlewares/tracing.(*Wrapper).ServeHTTP(0xc0010e0c60, {0x42a9f10, 0xc0012ec000}, 0xc000015400)
        /go/src/github.com/traefik/traefik/pkg/middlewares/tracing/wrapper.go:57 +0x106
github.com/traefik/traefik/v2/pkg/middlewares/headers.(*Header).ServeHTTP(0xc00107a000, {0x42b7ec0, 0xc000f9df90}, 0xc000015400)
        /go/src/github.com/traefik/traefik/pkg/middlewares/headers/header.go:64 +0x15d
github.com/traefik/traefik/v2/pkg/middlewares/connectionheader.Remover.func1({0x42b7ec0, 0xc000f9df90}, 0xc000015400)
        /go/src/github.com/traefik/traefik/pkg/middlewares/connectionheader/connectionheader.go:34 +0x2a4
net/http.HandlerFunc.ServeHTTP(0x2bd1a27, {0x42b7ec0, 0xc000f9df90}, 0x3e0a858)
        /usr/local/go/src/net/http/server.go:2047 +0x2f
github.com/traefik/traefik/v2/pkg/middlewares/headers.(*headers).ServeHTTP(0x42e0560, {0x42b7ec0, 0xc000f9df90}, 0x0)
        /go/src/github.com/traefik/traefik/pkg/middlewares/headers/headers.go:91 +0x2c
github.com/traefik/traefik/v2/pkg/middlewares/tracing.(*Wrapper).ServeHTTP(0xc0010e1590, {0x42b7ec0, 0xc000f9df90}, 0xc000015400)
        /go/src/github.com/traefik/traefik/pkg/middlewares/tracing/wrapper.go:57 +0x106
github.com/traefik/traefik/v2/pkg/middlewares/headers.(*Header).ServeHTTP(0xc00107a200, {0x7f9d2f92dfd8, 0xc000b21d28}, 0xc000015400)
        /go/src/github.com/traefik/traefik/pkg/middlewares/headers/header.go:64 +0x15d
github.com/traefik/traefik/v2/pkg/middlewares/connectionheader.Remover.func1({0x7f9d2f92dfd8, 0xc000b21d28}, 0xc000015400)
        /go/src/github.com/traefik/traefik/pkg/middlewares/connectionheader/connectionheader.go:34 +0x2a4
net/http.HandlerFunc.ServeHTTP(0x2bd1a27, {0x7f9d2f92dfd8, 0xc000b21d28}, 0x3e0a858)
        /usr/local/go/src/net/http/server.go:2047 +0x2f
github.com/traefik/traefik/v2/pkg/middlewares/headers.(*headers).ServeHTTP(0x42e0560, {0x7f9d2f92dfd8, 0xc000b21d28}, 0x0)
        /go/src/github.com/traefik/traefik/pkg/middlewares/headers/headers.go:91 +0x2c
github.com/traefik/traefik/v2/pkg/middlewares/tracing.(*Wrapper).ServeHTTP(0xc0010e1a10, {0x7f9d2f92dfd8, 0xc000b21d28}, 0xc000015400)
        /go/src/github.com/traefik/traefik/pkg/middlewares/tracing/wrapper.go:57 +0x106
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0xc00107a700, {0x7f9d2f92dfd8, 0xc000b21d28}, 0x31e8f20)
        /go/src/github.com/traefik/traefik/pkg/middlewares/accesslog/field_middleware.go:38 +0xf6
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000b6c0c0, {0x7f9d2f92dfd8, 0xc000b21d28}, 0xc000015300)
        /go/pkg/mod/github.com/containous/mux@v0.0.0-20220627093034-b2dd784e613f/mux.go:141 +0x24c
github.com/traefik/traefik/v2/pkg/middlewares/recovery.(*recovery).ServeHTTP(0xc00129def8, {0x7f9d2f92dfd8, 0xc000b21d28}, 0x32b2be0)
        /go/src/github.com/traefik/traefik/pkg/middlewares/recovery/recovery.go:32 +0x82
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.AddOriginFields({0x7f9d2f92dfd8, 0xc000b21d20}, 0x392e3de, {0x4275d60, 0xc0010de6d0}, 0xc0012a6400)
        /go/src/github.com/traefik/traefik/pkg/middlewares/accesslog/field_middleware.go:55 +0x1a2
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0xc000de6d80, {0x7f9d2f92dfd8, 0xc000b21d20}, 0xa)
        /go/src/github.com/traefik/traefik/pkg/middlewares/accesslog/field_middleware.go:36 +0xd8
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.(*Handler).ServeHTTP(0xc0007015c0, {0x42b7140, 0xc000bff420}, 0xc000015200, {0x4275ae0, 0xc000de6d80})
        /go/src/github.com/traefik/traefik/pkg/middlewares/accesslog/logger.go:227 +0xfa3
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.WrapHandler.func1.1({0x42b7140, 0xc000bff420}, 0xc0010ef600)
        /go/src/github.com/traefik/traefik/pkg/middlewares/accesslog/logger.go:71 +0x3b
net/http.HandlerFunc.ServeHTTP(0x90, {0x42b7140, 0xc000bff420}, 0xc000099800)
        /usr/local/go/src/net/http/server.go:2047 +0x2f
github.com/traefik/traefik/v2/pkg/middlewares.(*HTTPHandlerSwitcher).ServeHTTP(0x4101a7, {0x42b7140, 0xc000bff420}, 0x4226301)
        /go/src/github.com/traefik/traefik/pkg/middlewares/handler_switcher.go:23 +0x62
github.com/traefik/traefik/v2/pkg/middlewares/requestdecorator.(*RequestDecorator).ServeHTTP(0xc0000c8ca0, {0x42b7140, 0xc000bff420}, 0xc000015100, 0xc0010ef868)
        /go/src/github.com/traefik/traefik/pkg/middlewares/requestdecorator/request_decorator.go:47 +0x57b
github.com/traefik/traefik/v2/pkg/middlewares/requestdecorator.WrapHandler.func1.1({0x42b7140, 0xc000bff420}, 0xc001273c20)
        /go/src/github.com/traefik/traefik/pkg/middlewares/requestdecorator/request_decorator.go:84 +0x68
net/http.HandlerFunc.ServeHTTP(0xc0000b25f0, {0x42b7140, 0xc000bff420}, 0x9)
        /usr/local/go/src/net/http/server.go:2047 +0x2f
github.com/traefik/traefik/v2/pkg/middlewares/forwardedheaders.(*XForwarded).ServeHTTP(0xc0000b25f0, {0x42b7140, 0xc000bff420}, 0xc000015100)
        /go/src/github.com/traefik/traefik/pkg/middlewares/forwardedheaders/forwarded_header.go:189 +0xca
net/http.AllowQuerySemicolons.func1({0x42b7140, 0xc000bff420}, 0xc000015100)
        /usr/local/go/src/net/http/server.go:2906 +0x223
net/http.HandlerFunc.ServeHTTP(0x42b7140, {0x42b7140, 0xc000bff420}, 0x7f9d5696c108)
        /usr/local/go/src/net/http/server.go:2047 +0x2f
golang.org/x/net/http2/h2c.h2cHandler.ServeHTTP({{0x427d240, 0xc0007248e8}, 0xc0003b3580}, {0x42b7140, 0xc000bff420}, 0xc000015100)
        /go/pkg/mod/golang.org/x/net@v0.0.0-20220425223048-2871e0cb64e4/http2/h2c/h2c.go:104 +0x3e4
net/http.serverHandler.ServeHTTP({0xc001273bf0}, {0x42b7140, 0xc000bff420}, 0xc000015100)
        /usr/local/go/src/net/http/server.go:2879 +0x43b
net/http.(*conn).serve(0xc0016dce60, {0x42e0560, 0xc0005e80f0})
        /usr/local/go/src/net/http/server.go:1930 +0xb08
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:3034 +0x4e8
</pre>
</details>

The interesting thing about this process is that when `Hijack()` fails, it almost seems like the "previous" HTTP handler may be to blame. In this case, it seems like the Access Log handler... Part of this doesn't make sense though, because metrics + access logs work without plugins. I may be following red herrings... maybe it is _anything_ higher up the stack?

Ohh yes! That's it! `yaegi` is where the `stdlib._net_http_` message is coming from. Perhaps it is the place where the `Hijack()` chain is getting broken
```
github.com/traefik/yaegi/stdlib._net_http_Handler.ServeHTTP(...)
        /go/pkg/mod/github.com/traefik/yaegi@v0.13.0/stdlib/go1_17_net_http.go:288
```

Yep! `http.Hijack()` method missing here. Now I know - underscores ~ yaegi 😄 :
https://github.com/traefik/yaegi/blob/f76db27c770599da3cffc0a7553f8e1403bb23f8/stdlib/go1_17_net_http.go#L312-L317
@colearendt 

> Ohh yes! That's it! yaegi is where the stdlib._net_http_ message is coming from. Perhaps it is the place where the Hijack() chain is getting broken
> 
> github.com/traefik/yaegi/stdlib._net_http_Handler.ServeHTTP(...)
>         /go/pkg/mod/github.com/traefik/yaegi@v0.13.0/stdlib/go1_17_net_http.go:288
> Yep! http.Hijack() method missing here. Now I know - underscores ~ yaegi 😄 :
> https://github.com/traefik/yaegi/blob/f76db27c770599da3cffc0a7553f8e1403bb23f8/stdlib/go1_17_net_http.go#L312-L317

I'm not saying there isn't an issue in yaegi (I'm looking into it), but I don't understand the point you're trying to make here. You're pointing at the representation of the ResponseWriter interface, which indeed is not supposed to include the Hijacker interface (https://pkg.go.dev/net/http#ResponseWriter), so it's completely normal that you didn't find any Hijack method there.

@mpl Yes, you're right. I think what you're seeing is ignorance slowly being rolled back 😅 

It wasn't clear to me where the `_net` messages were coming from, so recognizing it was syntax from yaegi was helpful. I'm suspicious that the `ResponseWriter` is being type asserted to a `ResponseWriter` and is losing the `Hijacker` interface when getting passed either to or from `yaegi`. However, the `ResponseWriter` is a bit tricky to track since it is an argument passed to the Handler's `ServeHTTP` method (the Handler is [definitely type asserted here](https://github.com/traefik/traefik/blob/3c1d5e0393e91ce977ce29d1176c1303d7e6dbf7/pkg/plugins/middlewares.go#L72))
Actually, would this do it? This is where it looks like the type assertion to the `arg` type (in this case a `ResponseWriter`) seems to happen. It may be on passing it back to the main thread though...? It's not clear to me how the "next" call gets back out of Yaegi

https://github.com/traefik/yaegi/blob/fbee2baf9ddfad88f1d9b8c32fc3ca492e5c1ea5/interp/run.go#L1004-L1023

More thorough writeup below:

My current understanding is:
- At the start of the server, all of the plugins are built / compiled / have their `.New` functions called
- As a part of this process, there is an `alice` chain of middlewares built that call one another at the end of their `ServeHTTP` method
- As an aside, there is an interface here that represents what is "needed" for this type of workflow
https://github.com/traefik/traefik/blob/c51e59059195257adba402ece7583083e91bd350/pkg/middlewares/stateful.go#L5-L12
    - Example usage here:
https://github.com/traefik/traefik/blob/c51e59059195257adba402ece7583083e91bd350/pkg/middlewares/customerrors/custom_errors.go#L22-L27
- When a request comes in, the server's `ServeHTTP` method (depending on the entrypoints and routers) ends up calling the alice chain
https://github.com/traefik/traefik/blob/c51e59059195257adba402ece7583083e91bd350/pkg/server/router/router.go#L129-L136
- If the chain has only "proper middlewares," things work well. They are called something like this:
https://github.com/traefik/traefik/blob/c51e59059195257adba402ece7583083e91bd350/pkg/server/middleware/middlewares.go#L191-L199
- If the chain has "plugins," they go through a different initialization process:
https://github.com/traefik/traefik/blob/c51e59059195257adba402ece7583083e91bd350/pkg/server/middleware/middlewares.go#L353-L361
- In particular, the `plug` handler represents the output of the arguments getting passed to Yaegi for the Plugin's "New" function to be called
https://github.com/traefik/traefik/blob/c51e59059195257adba402ece7583083e91bd350/pkg/plugins/middlewares.go#L62-L77
- When this handler is called through `run.go` inside of Yaegi, we see the args type cast into the "defined" type (i.e. `ResponseWriter`). (Although I must admit to getting a bit lost in the reflection)
https://github.com/traefik/yaegi/blob/fbee2baf9ddfad88f1d9b8c32fc3ca492e5c1ea5/interp/run.go#L1004-L1023
- This stream can also be seen inside of the stack trace above. I have somehow convinced myself that this is an orthogonal stack to the `New` function call I reference above:
```
reflect.Value.call({0x2f9cd60, 0xc00118c050, 0xc00129c4b0}, {0x355d780, 0x4}, {0xc0014fc5a0, 0x2, 0x7f727ab82068})
        /usr/local/golang/1.10.8/go/src/reflect/value.go:543 +0x814
reflect.Value.Call({0x2f9cd60, 0xc00118c050, 0x4517f2}, {0xc0014fc5a0, 0x2, 0x2})
        /usr/local/golang/1.10.8/go/src/reflect/value.go:339 +0xc5
github.com/traefik/yaegi/interp.callBin.func1({0x2f9cd60, 0xc00118c050, 0x4ac5c6}, {0xc0014fc5a0, 0xc00118c040, 0x4ae4dc})
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:1472 +0x28
github.com/traefik/yaegi/interp.callBin.func10(0xc00136e420)
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:1646 +0x158
github.com/traefik/yaegi/interp.runCfg(0xc00098b8c0, 0xc00136e420, 0x39, 0x3453a60)
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:201 +0x2ac
github.com/traefik/yaegi/interp.genFunctionWrapper.func2.1({0xc00120fe30, 0x2, 0x4})
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/interp/run.go:1023 +0x4a5
github.com/traefik/yaegi/stdlib._net_http_Handler.ServeHTTP(...)
        /home/semaphore/go/pkg/mod/github.com/traefik/yaegi@v0.11.2/stdlib/go1_17_net_http.go:288
```

This seems like the expected behavior from Yaegi. However, it does break our chain because the `Hijacker` interface would be lost on the type assertion to `ResponseWriter`. This is really difficult though, because `Handler` is not what we need to fix - we must fix `ResponseWriter`, which is passed when `ServeHTTP` is called by the "previous middleware in the chain." i.e.

https://github.com/traefik/traefik/blob/c51e59059195257adba402ece7583083e91bd350/pkg/middlewares/metrics/metrics.go#L121

Maybe having the plugin define the type of its `ServeHTTP` method to be an interface that extends `ResponseWriter` (i.e. `Stateful`) would solve the problem / test this theory? However, we will need such an interface inside of Yaegi - could I define it in the plugin, or would it need to be inside of Yaegi's source? If does need to be in Yaegi's source, how do we reference it in the plugin so that Yaegi picks it up properly? It seems (from docs) that Yaegi does not support modules, so does that mean that the base library needs to be patched with such an interface? Does Yaegi have access to traefik's source / interfaces?

Does this sound like a reasonable interpretation of the lay of the land?
It does seem like it is related to yaegi.
I opened the corresponding issue with a simpler reproduction: https://github.com/traefik/yaegi/issues/1425
So much simpler 😅 That’s impressive, thanks!

Fix patch:
diff --git a/go.mod b/go.mod
index 8e4cac64df..4612f025ff 100644
--- a/go.mod
+++ b/go.mod
@@ -56,7 +56,7 @@ require (
 	github.com/stretchr/testify v1.8.0
 	github.com/stvp/go-udp-testing v0.0.0-20191102171040-06b61409b154
 	github.com/traefik/paerser v0.1.9
-	github.com/traefik/yaegi v0.14.1
+	github.com/traefik/yaegi v0.14.2
 	github.com/uber/jaeger-client-go v2.30.0+incompatible
 	github.com/uber/jaeger-lib v2.2.0+incompatible
 	github.com/unrolled/render v1.0.2
diff --git a/go.sum b/go.sum
index 70b1a54e67..23e823d6db 100644
--- a/go.sum
+++ b/go.sum
@@ -1905,8 +1905,8 @@ github.com/tonistiigi/vt100 v0.0.0-20190402012908-ad4c4a574305 h1:y/1cL5AL2oRcfz
 github.com/tonistiigi/vt100 v0.0.0-20190402012908-ad4c4a574305/go.mod h1:gXOLibKqQTRAVuVZ9gX7G9Ykky8ll8yb4slxsEMoY0c=
 github.com/traefik/paerser v0.1.9 h1:x5hZafOt/yogLvr6upoSOYIAn2nh2GsnLb236MOzd4I=
 github.com/traefik/paerser v0.1.9/go.mod h1:Dk3Bfz6Zyj13/S8pJyRdx/FNvXlsVRVbtp0UK4ZSiA0=
-github.com/traefik/yaegi v0.14.1 h1:t0ssyzeZCWTFGd/JnVuDxH/slMQfYg+2CDD4dLW/rU0=
-github.com/traefik/yaegi v0.14.1/go.mod h1:AVRxhaI2G+nUsaM1zyktzwXn69G3t/AuTDrCiTds9p0=
+github.com/traefik/yaegi v0.14.2 h1:9t9xepIfar6BrYdwJHGc+XRKo6qFoJCl6Z46N3hUtUw=
+github.com/traefik/yaegi v0.14.2/go.mod h1:AVRxhaI2G+nUsaM1zyktzwXn69G3t/AuTDrCiTds9p0=
 github.com/transip/gotransip/v6 v6.6.1 h1:nsCU1ErZS5G0FeOpgGXc4FsWvBff9GPswSMggsC4564=
 github.com/transip/gotransip/v6 v6.6.1/go.mod h1:pQZ36hWWRahCUXkFWlx9Hs711gLd8J4qdgLdRzmtY+g=
 github.com/tv42/httpunix v0.0.0-20150427012821-b75d8614f926 h1:G3dpKMzFDjgEh2q1Z7zUUtKa8ViPtH+ocF0bE0g00O8=


Test patch: