Repository: litestar-org/litestar

Issue:
Bug: Logging middleware with structlog causes application to return HTTP 500 when request body is malformed
### Description

It looks to me that the logging middleware is attempting to parse the body of the incoming requests and if the parsing fails this results in an HTTP 500 Internal Error instead of an HTTP 400 Bad Request. If I don't use the logging middleware, the behavior is what I would normally expect to happen.

I've tested this with standard logging and structlog (structlog being what I use in my project) and the behavior is the same.

### URL to code causing the issue

_No response_

### MCVE

```python
from typing import Any

from litestar import Litestar, post
from litestar.logging.config import LoggingConfig, StructLoggingConfig
from litestar.middleware.logging import LoggingMiddlewareConfig


@post("/")
async def hello_world(data: dict[str, Any]) -> dict[str, Any]:
    """Route Handler that outputs hello world."""
    return data


app = Litestar(
    route_handlers=[hello_world],
    # logging_config=LoggingConfig(log_exceptions="always"),  # Also happens with standard logging
    logging_config=StructLoggingConfig(log_exceptions="always"),
    middleware=[LoggingMiddlewareConfig().middleware],
)
```


### Steps to reproduce

```bash
1. Run example, e.g.: `litestar --app bug:app run`
2. Post invalid json: `curl --header "Content-Type: application/json" --request POST --data '{"a": "b",}' http://localhost:8000`
3. Examine the logs.
```


### Screenshots

_No response_

### Logs

```bash
!!! I'm using the standard logging version output as it formats the exception in a readable manner
!!! but the outcome is essentially the same regardless if using structlog or standard logging.

┌─┤10:26:41│mihai@mobilews:~/bug
└──────────╼ litestar --app bug:app run
Using Litestar app from env: 'bug:app'
Starting server process ───────────────────────────────────────────────────────────────────────────────────────────────────────
┌──────────────────────────────┬──────────────────────┐
│ Litestar version             │ 2.5.4                │
│ Debug mode                   │ Disabled             │
│ Python Debugger on exception │ Disabled             │
│ CORS                         │ Disabled             │
│ CSRF                         │ Disabled             │
│ OpenAPI                      │ Enabled path=/schema │
│ Compression                  │ Disabled             │
│ Middlewares                  │ LoggingMiddleware    │
└──────────────────────────────┴──────────────────────┘
INFO:     Started server process [110144]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     127.0.0.1:35020 - "POST / HTTP/1.1" 500 Internal Server Error
ERROR - 2024-02-02 10:26:45,152 - litestar - config - exception raised on http connection to route /

Traceback (most recent call last):
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/serialization/msgspec_hooks.py", line 186, in decode_json
    return _msgspec_json_decoder.decode(value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
msgspec.DecodeError: JSON is malformed: trailing comma in object (byte 10)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/middleware/exceptions/middleware.py", line 192, in __call__
    await self.app(scope, receive, send)
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/middleware/base.py", line 129, in wrapped_call
    await original__call__(self, scope, receive, send)  # pyright: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/middleware/logging.py", line 112, in __call__
    await self.log_request(scope=scope, receive=receive)
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/middleware/logging.py", line 126, in log_request
    extracted_data = await self.extract_request_data(request=scope["app"].request_class(scope, receive))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/middleware/logging.py", line 179, in extract_request_data
    value = await value
            ^^^^^^^^^^^
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/data_extractors.py", line 277, in extract_body
    return await request.json()
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/connection/request.py", line 134, in json
    self._json = self._connection_state.json = decode_json(
                                               ^^^^^^^^^^^^
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/serialization/msgspec_hooks.py", line 191, in decode_json
    raise SerializationException(str(msgspec_error)) from msgspec_error
litestar.exceptions.base_exceptions.SerializationException: JSON is malformed: trailing comma in object (byte 10)
Traceback (most recent call last):
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/serialization/msgspec_hooks.py", line 186, in decode_json
    return _msgspec_json_decoder.decode(value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
msgspec.DecodeError: JSON is malformed: trailing comma in object (byte 10)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/middleware/exceptions/middleware.py", line 192, in __call__
    await self.app(scope, receive, send)
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/middleware/base.py", line 129, in wrapped_call
    await original__call__(self, scope, receive, send)  # pyright: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/middleware/logging.py", line 112, in __call__
    await self.log_request(scope=scope, receive=receive)
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/middleware/logging.py", line 126, in log_request
    extracted_data = await self.extract_request_data(request=scope["app"].request_class(scope, receive))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/middleware/logging.py", line 179, in extract_request_data
    value = await value
            ^^^^^^^^^^^
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/data_extractors.py", line 277, in extract_body
    return await request.json()
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/connection/request.py", line 134, in json
    self._json = self._connection_state.json = decode_json(
                                               ^^^^^^^^^^^^
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/serialization/msgspec_hooks.py", line 191, in decode_json
    raise SerializationException(str(msgspec_error)) from msgspec_error
litestar.exceptions.base_exceptions.SerializationException: JSON is malformed: trailing comma in object (byte 10)
```


### Litestar Version

litestar[standard]==2.5.4
structlog==24.1.0

### Platform

- [x] Linux
- [ ] Mac
- [ ] Windows
- [ ] Other (Please specify in the description above)

<!-- POLAR PLEDGE BADGE START -->
---
> [!NOTE]  
> While we are open for sponsoring on [GitHub Sponsors](https://github.com/sponsors/litestar-org/) and 
> [OpenCollective](https://opencollective.com/litestar), we also utilize [Polar.sh](https://polar.sh/) to engage in pledge-based sponsorship.
>
> Check out all issues funded or available for funding [on our Polar.sh dashboard](https://polar.sh/litestar-org)
> * If you would like to see an issue prioritized, make a pledge towards it!
> * We receive the pledge once the issue is completed & verified
> * This, along with engagement in the community, helps us know which features are a priority to our users.

<a href="https://polar.sh/litestar-org/litestar/issues/3063">
<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://polar.sh/api/github/litestar-org/litestar/issues/3063/pledge.svg?darkmode=1">
  <img alt="Fund with Polar" src="https://polar.sh/api/github/litestar-org/litestar/issues/3063/pledge.svg">
</picture>
</a>
<!-- POLAR PLEDGE BADGE END -->



Discussion:
I think this has been resolved in the development branch.  @zoopp are you able to give that branch a try to confirm if the problem remains?
@cofin I gave it a try now but the issue is still reproducible with the MCVE above:

```
!!! I'm using the standard logging version output as it formats the exception in a readable manner
!!! but the outcome is essentially the same regardless if using structlog or standard logging.

(litestar)
┌─┤10:11:07│mihai@mobilews:~/bug
└──────────╼ litestar --app bug:app run --reload
Using Litestar app from env: 'bug:app'
Starting server process ───────────────────────────────────────────────────────────────────────────────────────────────────────
┌──────────────────────────────┬──────────────────────┐
│ Litestar version             │ 2.6.0                │
│ Debug mode                   │ Disabled             │
│ Python Debugger on exception │ Disabled             │
│ CORS                         │ Disabled             │
│ CSRF                         │ Disabled             │
│ OpenAPI                      │ Enabled path=/schema │
│ Compression                  │ Disabled             │
│ Middlewares                  │ LoggingMiddleware    │
└──────────────────────────────┴──────────────────────┘
INFO:     Will watch for changes in these directories: ['/home/mihai/bug']
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [150319] using WatchFiles
INFO:     Started server process [150321]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     127.0.0.1:58246 - "POST / HTTP/1.1" 500 Internal Server Error
ERROR - 2024-02-05 10:11:12,310 - litestar - config - exception raised on http connection to route /

Traceback (most recent call last):
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/serialization/msgspec_hooks.py", line 186, in decode_json
    return _msgspec_json_decoder.decode(value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
msgspec.DecodeError: JSON is malformed: trailing comma in object (byte 10)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/middleware/exceptions/middleware.py", line 192, in __call__
    await self.app(scope, receive, send)
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/middleware/base.py", line 129, in wrapped_call
    await original__call__(self, scope, receive, send)  # pyright: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/middleware/logging.py", line 112, in __call__
    await self.log_request(scope=scope, receive=receive)
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/middleware/logging.py", line 126, in log_request
    extracted_data = await self.extract_request_data(request=scope["app"].request_class(scope, receive))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/middleware/logging.py", line 179, in extract_request_data
    value = await value
            ^^^^^^^^^^^
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/data_extractors.py", line 277, in extract_body
    return await request.json()
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/connection/request.py", line 134, in json
    self._json = self._connection_state.json = decode_json(
                                               ^^^^^^^^^^^^
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/serialization/msgspec_hooks.py", line 191, in decode_json
    raise SerializationException(str(msgspec_error)) from msgspec_error
litestar.exceptions.base_exceptions.SerializationException: JSON is malformed: trailing comma in object (byte 10)
Traceback (most recent call last):
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/serialization/msgspec_hooks.py", line 186, in decode_json
    return _msgspec_json_decoder.decode(value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
msgspec.DecodeError: JSON is malformed: trailing comma in object (byte 10)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/middleware/exceptions/middleware.py", line 192, in __call__
    await self.app(scope, receive, send)
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/middleware/base.py", line 129, in wrapped_call
    await original__call__(self, scope, receive, send)  # pyright: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/middleware/logging.py", line 112, in __call__
    await self.log_request(scope=scope, receive=receive)
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/middleware/logging.py", line 126, in log_request
    extracted_data = await self.extract_request_data(request=scope["app"].request_class(scope, receive))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/middleware/logging.py", line 179, in extract_request_data
    value = await value
            ^^^^^^^^^^^
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/data_extractors.py", line 277, in extract_body
    return await request.json()
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/connection/request.py", line 134, in json
    self._json = self._connection_state.json = decode_json(
                                               ^^^^^^^^^^^^
  File "/home/mihai/.virtualenvs/litestar/lib/python3.11/site-packages/litestar/serialization/msgspec_hooks.py", line 191, in decode_json
    raise SerializationException(str(msgspec_error)) from msgspec_error
litestar.exceptions.base_exceptions.SerializationException: JSON is malformed: trailing comma in object (byte 10)
```

Resolving patch:
diff --git a/litestar/data_extractors.py b/litestar/data_extractors.py
--- a/litestar/data_extractors.py
+++ b/litestar/data_extractors.py
@@ -1,6 +1,7 @@
 from __future__ import annotations
 
-from typing import TYPE_CHECKING, Any, Callable, Coroutine, Literal, TypedDict, cast
+import inspect
+from typing import TYPE_CHECKING, Any, Callable, Coroutine, Iterable, Literal, TypedDict, cast
 
 from litestar._parsers import parse_cookie_string
 from litestar.connection.request import Request
@@ -70,6 +71,7 @@ class ConnectionDataExtractor:
         "parse_query",
         "obfuscate_headers",
         "obfuscate_cookies",
+        "skip_parse_malformed_body",
     )
 
     def __init__(
@@ -88,6 +90,7 @@ def __init__(
         obfuscate_headers: set[str] | None = None,
         parse_body: bool = False,
         parse_query: bool = False,
+        skip_parse_malformed_body: bool = False,
     ) -> None:
         """Initialize ``ConnectionDataExtractor``
 
@@ -106,9 +109,11 @@ def __init__(
             obfuscate_cookies: cookie keys to obfuscate. Obfuscated values are replaced with '*****'.
             parse_body: Whether to parse the body value or return the raw byte string, (for requests only).
             parse_query: Whether to parse query parameters or return the raw byte string.
+            skip_parse_malformed_body: Whether to skip parsing the body if it is malformed
         """
         self.parse_body = parse_body
         self.parse_query = parse_query
+        self.skip_parse_malformed_body = skip_parse_malformed_body
         self.obfuscate_headers = {h.lower() for h in (obfuscate_headers or set())}
         self.obfuscate_cookies = {c.lower() for c in (obfuscate_cookies or set())}
         self.connection_extractors: dict[str, Callable[[ASGIConnection[Any, Any, Any, Any]], Any]] = {}
@@ -153,6 +158,25 @@ def __call__(self, connection: ASGIConnection[Any, Any, Any, Any]) -> ExtractedR
         )
         return cast("ExtractedRequestData", {key: extractor(connection) for key, extractor in extractors.items()})
 
+    async def extract(
+        self, connection: ASGIConnection[Any, Any, Any, Any], fields: Iterable[str]
+    ) -> ExtractedRequestData:
+        extractors = (
+            {**self.connection_extractors, **self.request_extractors}  # type: ignore
+            if isinstance(connection, Request)
+            else self.connection_extractors
+        )
+        data = {}
+        for key, extractor in extractors.items():
+            if key not in fields:
+                continue
+            if inspect.iscoroutinefunction(extractor):
+                value = await extractor(connection)
+            else:
+                value = extractor(connection)
+            data[key] = value
+        return cast("ExtractedRequestData", data)
+
     @staticmethod
     def extract_scheme(connection: ASGIConnection[Any, Any, Any, Any]) -> str:
         """Extract the scheme from an ``ASGIConnection``
@@ -272,13 +296,20 @@ async def extract_body(self, request: Request[Any, Any, Any]) -> Any:
             return None
         if not self.parse_body:
             return await request.body()
-        request_encoding_type = request.content_type[0]
-        if request_encoding_type == RequestEncodingType.JSON:
-            return await request.json()
-        form_data = await request.form()
-        if request_encoding_type == RequestEncodingType.URL_ENCODED:
-            return dict(form_data)
-        return {key: repr(value) if isinstance(value, UploadFile) else value for key, value in form_data.multi_items()}
+        try:
+            request_encoding_type = request.content_type[0]
+            if request_encoding_type == RequestEncodingType.JSON:
+                return await request.json()
+            form_data = await request.form()
+            if request_encoding_type == RequestEncodingType.URL_ENCODED:
+                return dict(form_data)
+            return {
+                key: repr(value) if isinstance(value, UploadFile) else value for key, value in form_data.multi_items()
+            }
+        except Exception as exc:
+            if self.skip_parse_malformed_body:
+                return await request.body()
+            raise exc
 
 
 class ExtractedResponseData(TypedDict, total=False):
diff --git a/litestar/middleware/logging.py b/litestar/middleware/logging.py
--- a/litestar/middleware/logging.py
+++ b/litestar/middleware/logging.py
@@ -1,7 +1,6 @@
 from __future__ import annotations
 
 from dataclasses import dataclass, field
-from inspect import isawaitable
 from typing import TYPE_CHECKING, Any, Iterable
 
 from litestar.constants import (
@@ -81,6 +80,7 @@ def __init__(self, app: ASGIApp, config: LoggingMiddlewareConfig) -> None:
             obfuscate_headers=self.config.request_headers_to_obfuscate,
             parse_body=self.is_struct_logger,
             parse_query=self.is_struct_logger,
+            skip_parse_malformed_body=True,
         )
         self.response_extractor = ResponseDataExtractor(
             extract_body="body" in self.config.response_log_fields,
@@ -172,12 +172,11 @@ async def extract_request_data(self, request: Request) -> dict[str, Any]:
 
         data: dict[str, Any] = {"message": self.config.request_log_message}
         serializer = get_serializer_from_scope(request.scope)
-        extracted_data = self.request_extractor(connection=request)
+
+        extracted_data = await self.request_extractor.extract(connection=request, fields=self.config.request_log_fields)
+
         for key in self.config.request_log_fields:
-            value = extracted_data.get(key)
-            if isawaitable(value):
-                value = await value
-            data[key] = self._serialize_value(serializer, value)
+            data[key] = self._serialize_value(serializer, extracted_data.get(key))
         return data
 
     def extract_response_data(self, scope: Scope) -> dict[str, Any]:


Test patch:
diff --git a/tests/unit/test_data_extractors.py b/tests/unit/test_data_extractors.py
--- a/tests/unit/test_data_extractors.py
+++ b/tests/unit/test_data_extractors.py
@@ -1,6 +1,8 @@
 from typing import Any, List
+from unittest.mock import AsyncMock
 
 import pytest
+from pytest_mock import MockFixture
 
 from litestar import Request
 from litestar.connection.base import empty_receive
@@ -108,3 +110,18 @@ async def send(message: "Any") -> None:
     assert extracted_data.get("body") == b'{"hello":"world"}'
     assert extracted_data.get("headers") == {**headers, "content-length": "17"}
     assert extracted_data.get("cookies") == {"Path": "/", "SameSite": "lax", "auth": "", "regular": ""}
+
+
+async def test_request_data_extractor_skip_keys() -> None:
+    req = factory.get()
+    extractor = ConnectionDataExtractor()
+    assert (await extractor.extract(req, {"body"})).keys() == {"body"}
+
+
+async def test_skip_parse_malformed_body_false_raises(mocker: MockFixture) -> None:
+    mocker.patch("litestar.testing.request_factory.Request.json", new=AsyncMock(side_effect=ValueError()))
+    req = factory.post(headers={"Content-Type": "application/json"})
+    extractor = ConnectionDataExtractor(parse_body=True, skip_parse_malformed_body=False)
+
+    with pytest.raises(ValueError):
+        await extractor.extract(req, {"body"})
diff --git a/tests/unit/test_middleware/test_logging_middleware.py b/tests/unit/test_middleware/test_logging_middleware.py
--- a/tests/unit/test_middleware/test_logging_middleware.py
+++ b/tests/unit/test_middleware/test_logging_middleware.py
@@ -1,5 +1,5 @@
 from logging import INFO
-from typing import TYPE_CHECKING, Dict
+from typing import TYPE_CHECKING, Any, Dict
 
 import pytest
 from structlog.testing import capture_logs
@@ -286,3 +286,17 @@ async def get_session() -> None:
         assert response.status_code == HTTP_200_OK
         assert "session" in client.cookies
         assert client.cookies["session"] == session_id
+
+
+def test_structlog_invalid_request_body_handled() -> None:
+    # https://github.com/litestar-org/litestar/issues/3063
+    @post("/")
+    async def hello_world(data: Dict[str, Any]) -> Dict[str, Any]:
+        return data
+
+    with create_test_client(
+        route_handlers=[hello_world],
+        logging_config=StructLoggingConfig(log_exceptions="always"),
+        middleware=[LoggingMiddlewareConfig().middleware],
+    ) as client:
+        assert client.post("/", headers={"Content-Type": "application/json"}, content=b'{"a": "b",}').status_code == 400