Repository: elastic/apm-agent-python

Issue:
capture_body makes requests hangs / validation error
**Describe the bug**: 
When using CAPTURE_BODY:
1) "all" - you get "validation error: error: context: request: 'body': validation rule 'types(string;map[string]interface)' violated "
2) "errors" - sending httpx request to unreachable address, the client gets stuck and no repsonse is returned to the client


**To Reproduce**

1. Create FastAPI.
2. do any request using  httpx.AsyncClient to unknown request (ex: localhost: 5050)
3. if your CAPTURE_BODY is set to all, you will get validation error
4. if your CAPTURE_BODY is set to errors, the requests will hang forever

**Environment (please complete the following information)**
- OS: [e.g. Linux]: Docker python:3.8.2-slim-buster
- Python version: 3.8.2
- Framework and version [e.g. Django 2.1]: FastAPI 63.0
- APM Server version: 7.10.2
- Agent version: 5.10.1


**Additional context**

Add any other context about the problem here.

- Agent config options <!-- be careful not to post sensitive information -->
  <details>
    <summary>Click to expand</summary>

    ```
    apm_config = {
        "SERVICE_NAME": "map",
        "ENABLED": True,
        "SERVER_URL": "http://**********:8200",
        "CAPTURE_HEADERS": True,
        "CAPTURE_BODY": "all",
    }
    ```
  </details>
- `requirements.txt`:
  <details>
    <summary>Click to expand</summary>

    ```
    aiofiles
    aredis
    async-timeout
    asyncpg
    authlib
    certifi
    cffi
    chardet
    click
    cryptography
    debugpy
    python-dateutil
    ecdsa<0.15
    elastic-apm
    fastapi
    gunicorn
    h11
    h2
    hiredis
    hpack
    hstspreload
    httpcore
    httptools
    httpx
    hyperframe
    idna
    loguru
    orjson
    pyasn1
    pycparser
    pydantic
    python-jose
    python-multipart
    rfc3986
    rsa
    six
    sniffio
    starlette==0.13.6
    ujson
    uvicorn
    uvloop
    websockets
    # yappi
    ```
  </details>



Discussion:
After debugging a little, it's related to the body payload, it seems flat body like: [10,20] will make it not work
@Syniex To clarify, using a flat body like that will make the request hang? Does it hang even without the agent present? If the agent makes the application hang for any reason, we consider that a bug, even if a request body is malformed.
@basepi  without the agent the application doesn't hangs.
I will provide code tomorrow, and dig deeper to it.

Thank you. This sounds like something we would like to fix.
```python
from typing import List

import httpx
import uvicorn
from elasticapm.contrib.starlette import ElasticAPM, make_apm_client
from fastapi import FastAPI
from pydantic import BaseModel


class Point(BaseModel):
    coordinates: List[float]


app = FastAPI()
apm_config = {
    "SERVICE_NAME": "testing",
    "ENABLED": True,
    "SERVER_URL": "http://localhost:8200",
    "CAPTURE_HEADERS": True,
    "CAPTURE_BODY": "error",
    # "CAPTURE_BODY": "all",
}
apm = make_apm_client(apm_config)
app.add_middleware(ElasticAPM, client=apm)

@app.post("/working")
async def working_fine():
    #Example of working endpoint when no body is provided
    async with httpx.AsyncClient() as client:
        response = await client.get("http://notexisthost/")
        return response.json()


@app.post("/not/working/1")
async def not_working(point: Point):
    # This won't work when using "CAPTURE_BODY": "errors" and will hang the request forever
    async with httpx.AsyncClient() as client:
        response = await client.get("http://notexisthost/")
        return response.json()


@app.post("/not/working/2")
async def not_working_2(coordinates: List[float]):
    # "CAPTURE_BODY": "all" will make elasticsearch says "body": validation error
    async with httpx.AsyncClient() as client:
        response = await client.get("http://notexisthost/")
        return response.json()


if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0")
```

@basepi  Please take a look at this example

You can also try to disable the apm middleware and see all endspoints will work as expected.

Resolving patch:
diff --git a/elasticapm/contrib/starlette/__init__.py b/elasticapm/contrib/starlette/__init__.py
--- a/elasticapm/contrib/starlette/__init__.py
+++ b/elasticapm/contrib/starlette/__init__.py
@@ -43,7 +43,7 @@
 from elasticapm.base import Client
 from elasticapm.conf import constants
 from elasticapm.contrib.asyncio.traces import set_context
-from elasticapm.contrib.starlette.utils import get_data_from_request, get_data_from_response
+from elasticapm.contrib.starlette.utils import get_body, get_data_from_request, get_data_from_response
 from elasticapm.utils.disttracing import TraceParent
 from elasticapm.utils.logging import get_logger
 
@@ -175,6 +175,14 @@ async def _request_started(self, request: Request):
         Args:
             request (Request)
         """
+        # When we consume the body, we replace the streaming mechanism with
+        # a mocked version -- this workaround came from
+        # https://github.com/encode/starlette/issues/495#issuecomment-513138055
+        # and we call the workaround here to make sure that regardless of
+        # `capture_body` settings, we will have access to the body if we need it.
+        if self.client.config.capture_body != "off":
+            await get_body(request)
+
         if not self.client.should_ignore_url(request.url.path):
             trace_parent = TraceParent.from_headers(dict(request.headers))
             self.client.begin_transaction("request", trace_parent=trace_parent)
diff --git a/elasticapm/contrib/starlette/utils.py b/elasticapm/contrib/starlette/utils.py
--- a/elasticapm/contrib/starlette/utils.py
+++ b/elasticapm/contrib/starlette/utils.py
@@ -121,7 +121,9 @@ async def receive() -> Message:
 async def get_body(request: Request) -> str:
     """Gets body from the request.
 
-    todo: This is not very pretty however it is not usual to get request body out of the target method (business logic).
+    When we consume the body, we replace the streaming mechanism with
+    a mocked version -- this workaround came from
+    https://github.com/encode/starlette/issues/495#issuecomment-513138055
 
     Args:
         request (Request)


Test patch:
diff --git a/tests/contrib/asyncio/starlette_tests.py b/tests/contrib/asyncio/starlette_tests.py
--- a/tests/contrib/asyncio/starlette_tests.py
+++ b/tests/contrib/asyncio/starlette_tests.py
@@ -75,6 +75,7 @@ async def hello(request):
 
     @app.route("/raise-exception", methods=["GET", "POST"])
     async def raise_exception(request):
+        await request.body()
         raise ValueError()
 
     @app.route("/hi/{name}/with/slash/", methods=["GET", "POST"])
@@ -291,3 +292,21 @@ def test_enabled_instrumentation(app, elasticapm_client):
     client = TestClient(app)
 
     assert not isinstance(urllib3.connectionpool.HTTPConnectionPool.urlopen, wrapt.BoundFunctionWrapper)
+
+
+@pytest.mark.parametrize(
+    "elasticapm_client",
+    [
+        {"capture_body": "error"},
+    ],
+    indirect=True,
+)
+def test_capture_body_error(app, elasticapm_client):
+    """
+    Context: https://github.com/elastic/apm-agent-python/issues/1032
+
+    Before the above issue was fixed, this test would hang
+    """
+    client = TestClient(app)
+    with pytest.raises(ValueError):
+        response = client.post("/raise-exception", data="[0, 1]")