Repository: litestar-org/litestar

Issue:
Bug: using Any return type on before_request handler throws an error on schema
### Description

Using `Any` as a return type for a `before_request` handler throws an error when accessing the schema pages/openapi.json endpoints. Looking at the documentation, it should return some value or `None`, and using one or both of those as a return type is successful. The API itself runs just fine with an invalid return type, and this only seems to affect the schema routes.

It seems like maybe this is expected behavior so that you are forced to set a proper return type, but maybe a more extensive error message pointing to the location of the issue could be added? The current error I get is `litestar.exceptions.http_exceptions.ImproperlyConfiguredException: 500: Unable to serialize response content`, which took a while to figure out what caused it.

### URL to code causing the issue

https://github.com/rseeley/litestar-before-request-error-mvce/blob/main/app.py

### MCVE

_No response_

### Steps to reproduce

```bash
1. Follow the steps in the README section titled "Install litestar"
2. Access the schema at http://127.0.0.1:8000/schema or http://127.0.0.1:8000/schema/openapi.json
```


### Screenshots

_No response_

### Logs

_No response_

### Litestar Version

2.1.1

### Platform

- [ ] Linux
- [X] Mac
- [ ] Windows
- [ ] Other (Please specify in the description above)

<!-- POLAR PLEDGE BADGE START -->
---

## Funding
* If you would like to see an issue prioritized, make a pledge towards it!
* We receive the pledge once the issue is completed & verified

<a href="https://polar.sh/litestar-org/litestar/issues/2368">
<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://polar.sh/api/github/litestar-org/litestar/issues/2368/pledge.svg?darkmode=1">
  <img alt="Fund with Polar" src="https://polar.sh/api/github/litestar-org/litestar/issues/2368/pledge.svg">
</picture>
</a>
<!-- POLAR PLEDGE BADGE END -->



Discussion:
Thanks for the great MCVE!

What's happening here is Litestar alters the return type of the handler based on the return type of the hook if it's not `None` or ` | None`. Since the hook returns an `ASGIResponse` if you don't return anything, and we default to encode with JSON, Litestar will then attempt to encode the `ASGIResponse` as JSON.

I'm not entirely sure *why* we alter the return type like this, @peterschutt do you have any insight there? I removed the code in question and it didn't seem to break anything, so I'm not even sure this is actually intentional.

In any case, I don't think we *should* be doing this, since it's extremely unexpected and I can't think of a good reason why this would be the desired behaviour. The return type of the handler should be the source of truth in any case, and it's the user's responsibility to return an appropriate type from any hooks that might be in the pipeline.

Resolving patch:
diff --git a/litestar/handlers/http_handlers/base.py b/litestar/handlers/http_handlers/base.py
--- a/litestar/handlers/http_handlers/base.py
+++ b/litestar/handlers/http_handlers/base.py
@@ -435,11 +435,6 @@ def get_response_handler(self, is_response_type_data: bool = False) -> Callable[
             return_type = self.parsed_fn_signature.return_type
             return_annotation = return_type.annotation
 
-            if before_request_handler := self.resolve_before_request():
-                handler_return_type = before_request_handler.parsed_signature.return_type
-                if not handler_return_type.is_subclass_of((Empty, NoneType)):
-                    return_annotation = handler_return_type.annotation
-
             self._response_handler_mapping["response_type_handler"] = response_type_handler = create_response_handler(
                 after_request=after_request,
                 background=self.background,
@@ -494,9 +489,6 @@ async def to_response(self, app: Litestar, data: Any, request: Request) -> ASGIA
         return await response_handler(app=app, data=data, request=request)  # type: ignore
 
     def on_registration(self, app: Litestar) -> None:
-        if before_request := self.resolve_before_request():
-            before_request.set_parsed_signature(self.resolve_signature_namespace())
-
         super().on_registration(app)
         self.resolve_after_response()
         self.resolve_include_in_schema()
@@ -527,13 +519,6 @@ def _validate_handler_function(self) -> None:
                 "If the function should return a value, change the route handler status code to an appropriate value.",
             )
 
-        if (
-            (before_request := self.resolve_before_request())
-            and not before_request.parsed_signature.return_type.is_subclass_of(NoneType)
-            and not before_request.parsed_signature.return_type.is_optional
-        ):
-            return_type = before_request.parsed_signature.return_type
-
         if not self.media_type:
             if return_type.is_subclass_of((str, bytes)) or return_type.annotation is AnyStr:
                 self.media_type = MediaType.TEXT


Test patch: