Repository: traefik/traefik (language: go)

Issue:
retry middleware is logging http.ErrAbortHandler as ERROR
### Welcome!

- [X] Yes, I've searched similar issues on [GitHub](https://github.com/traefik/traefik/issues) and didn't find any.
- [X] Yes, I've searched similar issues on the [Traefik community forum](https://community.traefik.io) and didn't find any.

### What did you do?

I came into a situation where `reverseproxy` (executed by Traefik) panicked with http.ErrAbortHandler.
From this panic documentation, it should not be considered an error.
Accordingly, the `recovery` middleware does log this panic as DEBUG.


### What did you see instead?

This panic is logged at ERROR level by the `retry` middleware:
```
2022-07-13T13:40:18.563Z - ERROR - Error in Go routine: net/http: abort Handler
2022-07-13T13:40:18.563Z - ERROR - Stack: goroutine 5576 [running]:
runtime/debug.Stack()
	/root/.gvm/gos/go1.18.2/src/runtime/debug/stack.go:24 +0x68
github.com/traefik/traefik/v2/pkg/safe.defaultRecoverGoroutine({0x213f040?, 0x4000104ae0})
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/traefik/traefik/v2@v2.6.3/pkg/safe/routine.go:66 +0xb0
github.com/traefik/traefik/v2/pkg/safe.OperationWithRecover.func1.1()
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/traefik/traefik/v2@v2.6.3/pkg/safe/routine.go:74 +0x38
panic({0x213f040, 0x4000104ae0})
	/root/.gvm/gos/go1.18.2/src/runtime/panic.go:838 +0x20c
net/http/httputil.(*ReverseProxy).ServeHTTP(0x4000e11860, {0xffff6b423f08?, 0x40006f8b58}, 0x4001b14f00)
	/root/.gvm/gos/go1.18.2/src/net/http/httputil/reverseproxy.go:349 +0xb8c
github.com/traefik/traefik/v2/pkg/middlewares/pipelining.(*pipelining).ServeHTTP(0x40009ad300?, {0xffff6b423f08?, 0x40006f8b58?}, 0x4001b14f00?)
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/traefik/traefik/v2@v2.6.3/pkg/middlewares/pipelining/pipelining.go:34 +0x8c
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.AddServiceFields({0xffff6b423f08, 0x40006f8b58}, 0x4001b14f00, {0x2ba1608, 0x40009ad300}, 0x4000f5df00)
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/traefik/traefik/v2@v2.6.3/pkg/middlewares/accesslog/field_middleware.go:47 +0x118
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0x4000efc180, {0xffff6b423f08, 0x40006f8b58}, 0x36dee4?)
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/traefik/traefik/v2@v2.6.3/pkg/middlewares/accesslog/field_middleware.go:36 +0xd8
github.com/vulcand/oxy/roundrobin.(*RoundRobin).ServeHTTP(0x4000587810, {0xffff6b423f08, 0x40006f8b58}, 0x4001b14d00)
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/vulcand/oxy@v1.3.0/roundrobin/rr.go:147 +0x460
github.com/traefik/traefik/v2/pkg/middlewares/emptybackendhandler.(*emptyBackend).ServeHTTP(0x40009ad7b0, {0xffff6b423f08, 0x40006f8b58}, 0x205a4a0?)
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/traefik/traefik/v2@v2.6.3/pkg/middlewares/emptybackendhandler/empty_backend_handler.go:25 +0x10c
github.com/traefik/traefik/v2/pkg/middlewares/tracing.(*forwarderMiddleware).ServeHTTP(0x4000eb1e90, {0xffff6b36fe40?, 0x40006f8b50}, 0x4001b14c00)
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/traefik/traefik/v2@v2.6.3/pkg/middlewares/tracing/forwarder.go:56 +0x354
github.com/traefik/traefik/v2/pkg/middlewares/retry.(*retry).ServeHTTP.func1()
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/traefik/traefik/v2@v2.6.3/pkg/middlewares/retry/retry.go:101 +0x454
github.com/traefik/traefik/v2/pkg/safe.OperationWithRecover.func1()
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/traefik/traefik/v2@v2.6.3/pkg/safe/routine.go:78 +0x5c
github.com/cenkalti/backoff/v4.RetryNotifyWithTimer(0x4000f2d650, {0x2baa9c0, 0x4000eff500}, 0x4000d21320, {0x0?, 0x0?})
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.2/retry.go:55 +0xc8
github.com/cenkalti/backoff/v4.RetryNotify(...)
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.2/retry.go:34
github.com/traefik/traefik/v2/pkg/middlewares/retry.(*retry).ServeHTTP(0x4000efc200, {0xffff6b423e98?, 0x40006f8b30}, 0x4001b14b00)
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/traefik/traefik/v2@v2.6.3/pkg/middlewares/retry/retry.go:121 +0x344
github.com/traefik/traefik/v2/pkg/middlewares/tracing.(*Wrapper).ServeHTTP(0x4000d82210, {0xffff6b423e98, 0x40006f8b30}, 0x4001b14a00)
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/traefik/traefik/v2@v2.6.3/pkg/middlewares/tracing/wrapper.go:66 +0xc8
[...]
net/http.HandlerFunc.ServeHTTP(0x21f8820?, {0xffff6b423e98?, 0x40006f8b30?}, 0xa?)
	/root/.gvm/gos/go1.18.2/src/net/http/server.go:2084 +0x3c
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0x4000efc240, {0xffff6b423e98, 0x40006f8b30}, 0x25b5520?)
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/traefik/traefik/v2@v2.6.3/pkg/middlewares/accesslog/field_middleware.go:38 +0xf8
github.com/gorilla/mux.(*Router).ServeHTTP(0x40008b5140, {0xffff6b423e98, 0x40006f8b30}, 0x4001b14800)
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/containous/mux@v0.0.0-20181024131434-c33f32e26898/mux.go:133 +0x214
github.com/traefik/traefik/v2/pkg/middlewares/recovery.(*recovery).ServeHTTP(0x4000b04fc8?, {0xffff6b423e98?, 0x40006f8b30?}, 0x4000807200?)
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/traefik/traefik/v2@v2.6.3/pkg/middlewares/recovery/recovery.go:32 +0x70
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.AddOriginFields({0xffff6b423f08?, 0x40006f8b18}, 0x2636420?, {0x2ba1648, 0x4000a1ddc0}, 0x4000f5df00)
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/traefik/traefik/v2@v2.6.3/pkg/middlewares/accesslog/field_middleware.go:55 +0x174
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0x4000a1e0c0, {0xffff6b423f08, 0x40006f8b18}, 0x2b97330?)
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/traefik/traefik/v2@v2.6.3/pkg/middlewares/accesslog/field_middleware.go:36 +0xd8
github.com/traefik/traefik/v2/pkg/middlewares/tracing.(*entryPointMiddleware).ServeHTTP(0x4001106c00, {0xffff6b423e98?, 0x40006f8b08}, 0x4001b14600)
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/traefik/traefik/v2@v2.6.3/pkg/middlewares/tracing/entrypoint.go:52 +0x5e0
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.(*Handler).ServeHTTP(0x400073ad20, {0x2bbab00?, 0x40006f84f8}, 0x4001b14500, {0x2ba17a8, 0x4001106c00})
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/traefik/traefik/v2@v2.6.3/pkg/middlewares/accesslog/logger.go:227 +0xe60
github.com/traefik/traefik/v2/pkg/middlewares/accesslog.WrapHandler.func1.1({0x2bbab00?, 0x40006f84f8?}, 0x4000a4e200?)
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/traefik/traefik/v2@v2.6.3/pkg/middlewares/accesslog/logger.go:71 +0x44
net/http.HandlerFunc.ServeHTTP(0xffff92439108?, {0x2bbab00?, 0x40006f84f8?}, 0x4000b2c1b0?)
	/root/.gvm/gos/go1.18.2/src/net/http/server.go:2084 +0x3c
github.com/traefik/traefik/v2/pkg/middlewares.(*HTTPHandlerSwitcher).ServeHTTP(0x90?, {0x2bbab00, 0x40006f84f8}, 0x1046a38?)
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/traefik/traefik/v2@v2.6.3/pkg/middlewares/handler_switcher.go:23 +0x64
github.com/traefik/traefik/v2/pkg/middlewares/requestdecorator.(*RequestDecorator).ServeHTTP(0x40006f9980, {0x2bbab00, 0x40006f84f8}, 0x40017ed700, 0x4000edfe70)
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/traefik/traefik/v2@v2.6.3/pkg/middlewares/requestdecorator/request_decorator.go:47 +0x520
github.com/traefik/traefik/v2/pkg/middlewares/requestdecorator.WrapHandler.func1.1({0x2bbab00?, 0x40006f84f8?}, 0x4000f2d1e0?)
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/traefik/traefik/v2@v2.6.3/pkg/middlewares/requestdecorator/request_decorator.go:84 +0x68
net/http.HandlerFunc.ServeHTTP(0x4000a11cc0?, {0x2bbab00?, 0x40006f84f8?}, 0x611df0?)
	/root/.gvm/gos/go1.18.2/src/net/http/server.go:2084 +0x3c
github.com/traefik/traefik/v2/pkg/middlewares/forwardedheaders.(*XForwarded).ServeHTTP(0x4000a11cc0, {0x2bbab00, 0x40006f84f8}, 0x40017ed700)
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/github.com/traefik/traefik/v2@v2.6.3/pkg/middlewares/forwardedheaders/forwarded_header.go:189 +0xe0
golang.org/x/net/http2.(*serverConn).runHandler(0x40009a5920?, 0x0?, 0x4000f6a780?, 0x0?)
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/golang.org/x/net@v0.0.0-20211216030914-fe4d6282115f/http2/server.go:2184 +0x74
created by golang.org/x/net/http2.(*serverConn).processHeaders
	/root/.gvm/pkgsets/go1.18.2/global/pkg/mod/golang.org/x/net@v0.0.0-20211216030914-fe4d6282115f/http2/server.go:1914 +0x50c
```

### What version of Traefik are you using?

mod/github.com/traefik/traefik/v2@v2.6.3

### What is your environment & configuration?

```yaml
[http.middlewares]
[http.middlewares.retry.retry]
attempts = 2
```


### If applicable, please paste the log output in DEBUG level

_No response_
Fix logging aborts as error in retry middleware
### What does this PR do?
This PR logs `http.ErrAbortHandler` in retry middleware as 'debug' instead of 'error'.


### Motivation
Closes #9188 


### More

- [N/A] Added/updated tests
- [N/A] Added/updated documentation




Discussion / hints:
IMO, the `retry` middleware should handle `http. ErrAbortHandler` in a specific manner and log it as DEBUG instead of ERROR, just like the `recovery` middleware does.
Hello @kianelbo,
 
Thanks for your interest in Traefik,

This PR is labeled as `needs-design-review`, 
as you can see in our [contribution guide](https://github.com/traefik/contributors-guide/blob/master/review_guidelines.md), this means that we have to take a longer look to evaluate how it would interact with the other parts of Traefik.

We will come back to you once the design review iteration is done.
@kianelbo you couldn't know about this, but this approach (capturing the panic in the retry middleware) isn't ideal because it makes the recovery middleware useless if/when the retry middleware is used. Consider this simplified view of how traefik works:

```go
package main

import (
	"fmt"
)

func recoverFoo() {
	if err := recover(); err != nil {
		fmt.Printf("RECOVERED FROM FOO: %v\n", err)
	}
}

func retryMiddleware(work func()) {
	defer recoverFoo()
	work()
}

func recoverBar() {
	if err := recover(); err != nil {
		fmt.Printf("RECOVERED FROM BAR: %v\n", err)
	}
}

func recoverMiddleware(work func()) {
	defer recoverBar()
	work()
}

func routerHandler() {
	panic("IN WORK")
}

func main() {
	// retryMiddleware is the retry middleware, chained directly to the router handler.
	// recoverMiddleware is the recovery middleware, chained to the muxer of all routers.
	// -> if retryMiddleware recovers from the panic in routerHandler,
	// recoverMiddleware never gets the panic -> recovery becomes useless.
	recoverMiddleware(func() {retryMiddleware(routerHandler)})
}
```

I guess one could argue that the retry middleware could replicate exactly everything that the recovery middleware does, which overall would maybe keep things behaving as they should, but it sounds like a bad design to have the recovery done in two different places imho.

So I would suggest trying to fix the problem directly in the recovery middleware instead. But I haven't tried myself.
@mpl I totally agree with you. However in the issue it was said that the retry middleware should log aborts with debug level (just like recovery) and that's what I did.

> So I would suggest trying to fix the problem directly in the recovery middleware instead. But I haven't tried myself.

Handling panic in retry was an existing behavior. So what do you suggest? Adding retry to recovery middleware or ...? :)
I suggest not retrying on this error and just catching the error inside the `recovery` middleware.
> I suggest not retrying on this error and just catching the error inside the `recovery` middleware.

@ldez so you mean removing the recovery routine inside retry middleware (because it should be handled with recovery middleware) and also stop retrying on aborts?
If so how is it possible to detect the abort in retry without a recover()?

Fix patch:
diff --git a/pkg/middlewares/retry/retry.go b/pkg/middlewares/retry/retry.go
index 0a21e160aa..37dd4abef3 100644
--- a/pkg/middlewares/retry/retry.go
+++ b/pkg/middlewares/retry/retry.go
@@ -16,7 +16,6 @@ import (
 	"github.com/traefik/traefik/v2/pkg/config/dynamic"
 	"github.com/traefik/traefik/v2/pkg/log"
 	"github.com/traefik/traefik/v2/pkg/middlewares"
-	"github.com/traefik/traefik/v2/pkg/safe"
 	"github.com/traefik/traefik/v2/pkg/tracing"
 )
 
@@ -118,7 +117,7 @@ func (r *retry) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
 		r.listener.Retried(req, attempts)
 	}
 
-	err := backoff.RetryNotify(safe.OperationWithRecover(operation), backOff, notify)
+	err := backoff.RetryNotify(operation, backOff, notify)
 	if err != nil {
 		log.FromContext(middlewares.GetLoggerCtx(req.Context(), r.name, typeName)).
 			Debugf("Final retry attempt failed: %v", err.Error())


Test patch: