Repository: encode/starlette

Issue:
Question about Request.Body()
Hi dears developers, how can I retrieve the request.body inside an exception_handler? 

My scenario:

I have an exception_handler hook that receives a request and an exception, I'm also using FastApi which tries to parse the body to a JSON, if it does not work it raises an HTTPException. So I'm trying to build a hook to log invalid JSON requests, this exception is being correctly captured by my exception_handler but when I'm trying to retrieve the body I can't it hangs, it simple doesn't work. It does make sense as after reading for the first time it is being cached inside the Request object but the Request object is being recreated before calling the exception_handler so I'm losing my cache and I'm able to ready from the stream again.

https://github.com/encode/starlette/blob/d23bfd0d8ff68d535d0283aa4099e5055da88bb9/starlette/exceptions.py#L79

Not sure if it is supposed to be like this by design, but I think my scenario is not that fancy, it should be something possible, what do you guys think? Any workaround?

My code:

```@app.exception_handler(HTTPException)  
async def httpexception_logger(request: Request, exc: HTTPException):
    body = await request.body()
    text: str = bytes.decode(body)
    logger.error("Exception at %s, status code: %s, body: %s: exception %s", request.url, exception.status_code, text, exc)
    ... other stuff...
```



Discussion:
What’s the simplest possible code sample someone could use to replicate the issue? (Ideally just using Starlette itself if possible)
I expect there’s an issue with the request body already having been streamed, but we ought to be able to handle things more gracefully.
Hi @tomchristie , I was able to reproduce with the code below:

```python
from starlette.applications import Starlette
from starlette.responses import JSONResponse
from starlette.exceptions import HTTPException
from json.decoder import JSONDecodeError
import uvicorn

# Force an exception sending an invalid json to "/"
# it will trigger an exception and it will be handled
# by log_exceptions and the assert will trigger an
# exception because request.body() cannot be read
# anymore (stream already consumed)

app = Starlette(debug=True)

@app.exception_handler(JSONDecodeError)
async def log_exceptions(request, exc):
    body: bytes = await request.body() # hangs forever
    body_text: str = bytes.decode(body)
    assert body_text, "I could not retrieve the body"
    return JSONResponse({ "error": "Unexpected error happened!" })


@app.route('/', methods=["POST"])
async def homepage(request):
    body_bytes = await request.body()
    if body_bytes:
        json = await request.json()
        return JSONResponse(json)
    return JSONResponse({ "error": "empty body, please send something" }) 


if __name__ == '__main__':
    uvicorn.run(app, host='0.0.0.0', port=8000)
```

to test:

```
curl -d "{\"key1\":\"value1\", \"key2\": invalid }" -H "Content-Type: application/json" -X POST http://localhost:8000
```
I have a very similar case running into the same problem: 
Using FastAPI I would like to log requests and responses for requests that have triggered a HTTPException. My idea was to wrap FastAPIs existing http exception handler like so:

```python
from fastapi.applications import http_exception
from starlette.exceptions import HTTPException
from logging import getLogger

logger = getLogger("uvicorn")

app = FastAPI(title='x')

@app.exception_handler(HTTPException)
async def new_http_exception(request, exc):
    request_json = await request.json()
    response = await http_exception(request, exc)
    log_prefix = f"('{request.client.host}', {request.client.port}) - \"{request.method} {request._scope['path']} {request._scope['type'].upper()}/{request._scope['http_version']}\" {exc}"
    logger.info(f"{log_prefix} request: {request_json}")
    logger.info(f"{log_prefix} response: {response.body.decode()}")
    return response

```

This hangs forever on the first line.
#495 has this similar issue as well.
@tomchristie , I was reading your comments related to @blueyed changes to store the Request in the Scope and I completely agree that Request should reflect scope, and as scope is used to construct a request perhaps a small change to store/cache the body inside the scope instead of a local var, do you think this could solve this issue?

I did this changes locally just to test and it seems to work but I have not enough knowledge about the impact it may cause:

requests.py@140...171

```python

    async def stream(self) -> typing.AsyncGenerator[bytes, None]:
        if "body" in self._scope:
            yield self._scope["body"]
            yield b""
            return

        if self._stream_consumed:
            raise RuntimeError("Stream consumed")

        self._stream_consumed = True
        while True:
            message = await self._receive()
            if message["type"] == "http.request":
                body = message.get("body", b"")
                if body:
                    yield body
                if not message.get("more_body", False):
                    break
            elif message["type"] == "http.disconnect":
                self._is_disconnected = True
                raise ClientDisconnect()
        yield b""

    async def body(self) -> bytes:
        if not "body" in self._scope:
            body = b""
            async for chunk in self.stream():
                body += chunk
            self._scope["body"] = body
        return self._scope["body"]

```
Any movements? Without this functionality, we don't have access to origin request on exception handler.
This is really strange.

Resolving patch:
diff --git a/starlette/_exception_handler.py b/starlette/_exception_handler.py
new file mode 100644
--- /dev/null
+++ b/starlette/_exception_handler.py
@@ -0,0 +1,76 @@
+import typing
+
+from starlette._utils import is_async_callable
+from starlette.concurrency import run_in_threadpool
+from starlette.exceptions import HTTPException
+from starlette.requests import Request
+from starlette.responses import Response
+from starlette.types import ASGIApp, Message, Receive, Scope, Send
+from starlette.websockets import WebSocket
+
+Handler = typing.Callable[..., typing.Any]
+ExceptionHandlers = typing.Dict[typing.Any, Handler]
+StatusHandlers = typing.Dict[int, Handler]
+
+
+def _lookup_exception_handler(
+    exc_handlers: ExceptionHandlers, exc: Exception
+) -> typing.Optional[Handler]:
+    for cls in type(exc).__mro__:
+        if cls in exc_handlers:
+            return exc_handlers[cls]
+    return None
+
+
+def wrap_app_handling_exceptions(
+    app: ASGIApp, conn: typing.Union[Request, WebSocket]
+) -> ASGIApp:
+    exception_handlers: ExceptionHandlers
+    status_handlers: StatusHandlers
+    try:
+        exception_handlers, status_handlers = conn.scope["starlette.exception_handlers"]
+    except KeyError:
+        exception_handlers, status_handlers = {}, {}
+
+    async def wrapped_app(scope: Scope, receive: Receive, send: Send) -> None:
+        response_started = False
+
+        async def sender(message: Message) -> None:
+            nonlocal response_started
+
+            if message["type"] == "http.response.start":
+                response_started = True
+            await send(message)
+
+        try:
+            await app(scope, receive, sender)
+        except Exception as exc:
+            handler = None
+
+            if isinstance(exc, HTTPException):
+                handler = status_handlers.get(exc.status_code)
+
+            if handler is None:
+                handler = _lookup_exception_handler(exception_handlers, exc)
+
+            if handler is None:
+                raise exc
+
+            if response_started:
+                msg = "Caught handled exception, but response already started."
+                raise RuntimeError(msg) from exc
+
+            if scope["type"] == "http":
+                response: Response
+                if is_async_callable(handler):
+                    response = await handler(conn, exc)
+                else:
+                    response = await run_in_threadpool(handler, conn, exc)
+                await response(scope, receive, sender)
+            elif scope["type"] == "websocket":
+                if is_async_callable(handler):
+                    await handler(conn, exc)
+                else:
+                    await run_in_threadpool(handler, conn, exc)
+
+    return wrapped_app
diff --git a/starlette/middleware/exceptions.py b/starlette/middleware/exceptions.py
--- a/starlette/middleware/exceptions.py
+++ b/starlette/middleware/exceptions.py
@@ -1,11 +1,14 @@
 import typing
 
-from starlette._utils import is_async_callable
-from starlette.concurrency import run_in_threadpool
+from starlette._exception_handler import (
+    ExceptionHandlers,
+    StatusHandlers,
+    wrap_app_handling_exceptions,
+)
 from starlette.exceptions import HTTPException, WebSocketException
 from starlette.requests import Request
 from starlette.responses import PlainTextResponse, Response
-from starlette.types import ASGIApp, Message, Receive, Scope, Send
+from starlette.types import ASGIApp, Receive, Scope, Send
 from starlette.websockets import WebSocket
 
 
@@ -20,12 +23,10 @@ def __init__(
     ) -> None:
         self.app = app
         self.debug = debug  # TODO: We ought to handle 404 cases if debug is set.
-        self._status_handlers: typing.Dict[int, typing.Callable] = {}
-        self._exception_handlers: typing.Dict[
-            typing.Type[Exception], typing.Callable
-        ] = {
+        self._status_handlers: StatusHandlers = {}
+        self._exception_handlers: ExceptionHandlers = {
             HTTPException: self.http_exception,
-            WebSocketException: self.websocket_exception,
+            WebSocketException: self.websocket_exception,  # type: ignore[dict-item]
         }
         if handlers is not None:
             for key, value in handlers.items():
@@ -42,68 +43,32 @@ def add_exception_handler(
             assert issubclass(exc_class_or_status_code, Exception)
             self._exception_handlers[exc_class_or_status_code] = handler
 
-    def _lookup_exception_handler(
-        self, exc: Exception
-    ) -> typing.Optional[typing.Callable]:
-        for cls in type(exc).__mro__:
-            if cls in self._exception_handlers:
-                return self._exception_handlers[cls]
-        return None
-
     async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
         if scope["type"] not in ("http", "websocket"):
             await self.app(scope, receive, send)
             return
 
-        response_started = False
-
-        async def sender(message: Message) -> None:
-            nonlocal response_started
-
-            if message["type"] == "http.response.start":
-                response_started = True
-            await send(message)
-
-        try:
-            await self.app(scope, receive, sender)
-        except Exception as exc:
-            handler = None
-
-            if isinstance(exc, HTTPException):
-                handler = self._status_handlers.get(exc.status_code)
-
-            if handler is None:
-                handler = self._lookup_exception_handler(exc)
-
-            if handler is None:
-                raise exc
+        scope["starlette.exception_handlers"] = (
+            self._exception_handlers,
+            self._status_handlers,
+        )
 
-            if response_started:
-                msg = "Caught handled exception, but response already started."
-                raise RuntimeError(msg) from exc
+        conn: typing.Union[Request, WebSocket]
+        if scope["type"] == "http":
+            conn = Request(scope, receive, send)
+        else:
+            conn = WebSocket(scope, receive, send)
 
-            if scope["type"] == "http":
-                request = Request(scope, receive=receive)
-                if is_async_callable(handler):
-                    response = await handler(request, exc)
-                else:
-                    response = await run_in_threadpool(handler, request, exc)
-                await response(scope, receive, sender)
-            elif scope["type"] == "websocket":
-                websocket = WebSocket(scope, receive=receive, send=send)
-                if is_async_callable(handler):
-                    await handler(websocket, exc)
-                else:
-                    await run_in_threadpool(handler, websocket, exc)
+        await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
 
-    def http_exception(self, request: Request, exc: HTTPException) -> Response:
+    def http_exception(self, request: Request, exc: Exception) -> Response:
+        assert isinstance(exc, HTTPException)
         if exc.status_code in {204, 304}:
             return Response(status_code=exc.status_code, headers=exc.headers)
         return PlainTextResponse(
             exc.detail, status_code=exc.status_code, headers=exc.headers
         )
 
-    async def websocket_exception(
-        self, websocket: WebSocket, exc: WebSocketException
-    ) -> None:
-        await websocket.close(code=exc.code, reason=exc.reason)
+    async def websocket_exception(self, websocket: WebSocket, exc: Exception) -> None:
+        assert isinstance(exc, WebSocketException)
+        await websocket.close(code=exc.code, reason=exc.reason)  # pragma: no cover
diff --git a/starlette/routing.py b/starlette/routing.py
--- a/starlette/routing.py
+++ b/starlette/routing.py
@@ -9,6 +9,7 @@
 from contextlib import asynccontextmanager
 from enum import Enum
 
+from starlette._exception_handler import wrap_app_handling_exceptions
 from starlette._utils import is_async_callable
 from starlette.concurrency import run_in_threadpool
 from starlette.convertors import CONVERTOR_TYPES, Convertor
@@ -61,12 +62,16 @@ def request_response(func: typing.Callable) -> ASGIApp:
     is_coroutine = is_async_callable(func)
 
     async def app(scope: Scope, receive: Receive, send: Send) -> None:
-        request = Request(scope, receive=receive, send=send)
-        if is_coroutine:
-            response = await func(request)
-        else:
-            response = await run_in_threadpool(func, request)
-        await response(scope, receive, send)
+        request = Request(scope, receive, send)
+
+        async def app(scope: Scope, receive: Receive, send: Send) -> None:
+            if is_coroutine:
+                response = await func(request)
+            else:
+                response = await run_in_threadpool(func, request)
+            await response(scope, receive, send)
+
+        await wrap_app_handling_exceptions(app, request)(scope, receive, send)
 
     return app
 
@@ -79,7 +84,11 @@ def websocket_session(func: typing.Callable) -> ASGIApp:
 
     async def app(scope: Scope, receive: Receive, send: Send) -> None:
         session = WebSocket(scope, receive=receive, send=send)
-        await func(session)
+
+        async def app(scope: Scope, receive: Receive, send: Send) -> None:
+            await func(session)
+
+        await wrap_app_handling_exceptions(app, session)(scope, receive, send)
 
     return app
 


Test patch:
diff --git a/tests/test_exceptions.py b/tests/test_exceptions.py
--- a/tests/test_exceptions.py
+++ b/tests/test_exceptions.py
@@ -4,7 +4,8 @@
 
 from starlette.exceptions import HTTPException, WebSocketException
 from starlette.middleware.exceptions import ExceptionMiddleware
-from starlette.responses import PlainTextResponse
+from starlette.requests import Request
+from starlette.responses import JSONResponse, PlainTextResponse
 from starlette.routing import Route, Router, WebSocketRoute
 
 
@@ -28,6 +29,22 @@ def with_headers(request):
     raise HTTPException(status_code=200, headers={"x-potato": "always"})
 
 
+class BadBodyException(HTTPException):
+    pass
+
+
+async def read_body_and_raise_exc(request: Request):
+    await request.body()
+    raise BadBodyException(422)
+
+
+async def handler_that_reads_body(
+    request: Request, exc: BadBodyException
+) -> JSONResponse:
+    body = await request.body()
+    return JSONResponse(status_code=422, content={"body": body.decode()})
+
+
 class HandledExcAfterResponse:
     async def __call__(self, scope, receive, send):
         response = PlainTextResponse("OK", status_code=200)
@@ -44,11 +61,19 @@ async def __call__(self, scope, receive, send):
         Route("/with_headers", endpoint=with_headers),
         Route("/handled_exc_after_response", endpoint=HandledExcAfterResponse()),
         WebSocketRoute("/runtime_error", endpoint=raise_runtime_error),
+        Route(
+            "/consume_body_in_endpoint_and_handler",
+            endpoint=read_body_and_raise_exc,
+            methods=["POST"],
+        ),
     ]
 )
 
 
-app = ExceptionMiddleware(router)
+app = ExceptionMiddleware(
+    router,
+    handlers={BadBodyException: handler_that_reads_body},  # type: ignore[dict-item]
+)
 
 
 @pytest.fixture
@@ -160,3 +185,9 @@ def test_exception_middleware_deprecation() -> None:
 
     with pytest.warns(DeprecationWarning):
         starlette.exceptions.ExceptionMiddleware
+
+
+def test_request_in_app_and_handler_is_the_same_object(client) -> None:
+    response = client.post("/consume_body_in_endpoint_and_handler", content=b"Hello!")
+    assert response.status_code == 422
+    assert response.json() == {"body": "Hello!"}
diff --git a/tests/test_routing.py b/tests/test_routing.py
--- a/tests/test_routing.py
+++ b/tests/test_routing.py
@@ -1033,13 +1033,9 @@ async def modified_send(msg: Message) -> None:
     assert resp.status_code == 200, resp.content
     assert "X-Mounted" in resp.headers
 
-    # this is the "surprising" behavior bit
-    # the middleware on the mount never runs because there
-    # is nothing to catch the HTTPException
-    # since Mount middlweare is not wrapped by ExceptionMiddleware
     resp = client.get("/mount/err")
     assert resp.status_code == 403, resp.content
-    assert "X-Mounted" not in resp.headers
+    assert "X-Mounted" in resp.headers
 
 
 def test_route_repr() -> None: