Repository: elastic/apm-agent-python

Issue:
Consider using route information instead of URL for Starlette transactions
We're slowly started using Starlette for some smaller applications due to its simplicity and async capabilities. We also happen to use Elastic APM for our monitoring, but we've found that the agent works quite different from other agents, such as Django or Flask.

The default for Django, is "HTTP Method + path.to.view". This can be changed for Django 2.2+  using the `django-transaction-name-from-route` setting, to instead use the route that was matched against. For example `GET /users/<int:user_id>/`.

For Flask, the default seems to be the that of method + route, e.g.: `GET /users/<int:user_id/`.

For Starlette however, information about the matched route is not actually available on the request, and for that reason (I assume), the full URL is instead used as the transaction name.
In the example above, it would mean that every time a user with a different URL was hit, it would generate two separate transaction entries. For some of our views, that would mean 1+ milllion transaction names for the same view.

However, it is somewhat possible to extract the route by mimicking what Starlette itself does. On the request, the Application itself is available, and from that route router + routes. This means it's possible to loop over them, and match their regular expression against the current request.

1) I realise this isn't pretty in any way, shape or form - but it would make instrumentation of Starlette applications a lot more useable
2) I also realise it's backwards incompatible - but perhaps this could be solved in the same way that it's done with Django?

What would your thought be on such a change?

For us at our workplace, we've actually have one major of our main applications ported to Starlette, but once we saw what was happening to the transaction names, we had to revert the deployment. We'd love to have it running, so we'd probably be interested in providing a patch for it as well.

**EDIT**
Just wanted to add this here, since it's a discussion of making this type of information available in the middleware system: https://github.com/encode/starlette/issues/685
It also links another repo for timing of ASGI that basically does what I describe here, by looping over all routes and comparing them to the current URL.


Discussion:
The Starlette support is new enough I'd be willing to entertain a change like this. I would much rather have route information than the full path, but like you mentioned, it's unavailable without manually calculating the route. My only concern with that solution is that it adds some processing overhead. But I do think it would give us more useful transaction names, and hopefully they'll add an official way to get that information in middleware.

@HenrikOssipoff is this a change for which you'd be willing to open a PR?
@basepi Definitely, I'll give it a go in the near future. Would enable us to use Starlette much more than we currently do :) 
@HenrikOssipoff Is this a change you're still interested in contributing?
> @HenrikOssipoff Is this a change you're still interested in contributing?

@basepi  There is a solution mentioned here https://github.com/tiangolo/fastapi/issues/486#issuecomment-526892238, but it needs to iterate through all the routes for each request.

```python
path = [route for route in request.scope['router'].routes if route.endpoint == request.scope['endpoint']][0].path
```
Thanks. I think that's probably worthwhile overhead, the current implementation isn't super great.
> Thanks. I think that's probably worthwhile overhead, the current implementation isn't super great.

Can't wait for the new version. Maybe we can make it an optional choice via an env variable.
I'm honestly still leaning towards just making the change without backwards compatibility considerations. The starlette support is very new and the current behavior could probably be considered a bug. Thoughts?
> I'm honestly still leaning towards just making the change without backwards compatibility considerations. The starlette support is very new and the current behavior could probably be considered a bug. Thoughts?

In my use case, I am happy to fix this one a bug. I don't need old behavior. 
> @HenrikOssipoff Is this a change you're still interested in contributing?

Sorry very late to the game - I can see @beniwohli has assigned himself, which is more than fine by me. Work's not leaving me much free time unfortunately.

Looking forward to the "fix" though - sorry for being slow here.

Personally I would say you shouldn't think about backwards compatibility for this one, given most likely very few people even use this particular integration, but that's my opinion alone.

Resolving patch:
diff --git a/elasticapm/contrib/starlette/__init__.py b/elasticapm/contrib/starlette/__init__.py
--- a/elasticapm/contrib/starlette/__init__.py
+++ b/elasticapm/contrib/starlette/__init__.py
@@ -35,6 +35,7 @@
 from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
 from starlette.requests import Request
 from starlette.responses import Response
+from starlette.routing import Match
 from starlette.types import ASGIApp
 
 import elasticapm
@@ -181,7 +182,8 @@ async def _request_started(self, request: Request):
             await set_context(
                 lambda: get_data_from_request(request, self.client.config, constants.TRANSACTION), "request"
             )
-            elasticapm.set_transaction_name("{} {}".format(request.method, request.url.path), override=False)
+            transaction_name = self.get_route_name(request) or request.url.path
+            elasticapm.set_transaction_name("{} {}".format(request.method, transaction_name), override=False)
 
     async def _request_finished(self, response: Response):
         """Captures the end of the request processing to APM.
@@ -195,3 +197,34 @@ async def _request_finished(self, response: Response):
 
         result = "HTTP {}xx".format(response.status_code // 100)
         elasticapm.set_transaction_result(result, override=False)
+
+    def get_route_name(self, request: Request) -> str:
+        route_name = None
+        app = request.app
+        scope = request.scope
+        routes = app.routes
+
+        for route in routes:
+            match, _ = route.matches(scope)
+            if match == Match.FULL:
+                route_name = route.path
+                break
+            elif match == Match.PARTIAL and route_name is None:
+                route_name = route.path
+        # Starlette magically redirects requests if the path matches a route name with a trailing slash
+        # appended or removed. To not spam the transaction names list, we do the same here and put these
+        # redirects all in the same "redirect trailing slashes" transaction name
+        if not route_name and app.router.redirect_slashes and scope["path"] != "/":
+            redirect_scope = dict(scope)
+            if scope["path"].endswith("/"):
+                redirect_scope["path"] = scope["path"][:-1]
+                trim = True
+            else:
+                redirect_scope["path"] = scope["path"] + "/"
+                trim = False
+            for route in routes:
+                match, _ = route.matches(redirect_scope)
+                if match != Match.NONE:
+                    route_name = route.path + "/" if trim else route.path[:-1]
+                    break
+        return route_name


Test patch:
diff --git a/tests/contrib/asyncio/starlette_tests.py b/tests/contrib/asyncio/starlette_tests.py
--- a/tests/contrib/asyncio/starlette_tests.py
+++ b/tests/contrib/asyncio/starlette_tests.py
@@ -55,6 +55,11 @@ async def hi(request):
             pass
         return PlainTextResponse("ok")
 
+    @app.route("/hi/{name}", methods=["GET"])
+    async def hi_name(request):
+        name = request.path_params["name"]
+        return PlainTextResponse("Hi {}".format(name))
+
     @app.route("/hello", methods=["GET", "POST"])
     async def hello(request):
         with async_capture_span("test"):
@@ -65,6 +70,14 @@ async def hello(request):
     async def raise_exception(request):
         raise ValueError()
 
+    @app.route("/hi/{name}/with/slash/", methods=["GET", "POST"])
+    async def with_slash(request):
+        return PlainTextResponse("Hi {}".format(request.path_params["name"]))
+
+    @app.route("/hi/{name}/without/slash", methods=["GET", "POST"])
+    async def without_slash(request):
+        return PlainTextResponse("Hi {}".format(request.path_params["name"]))
+
     app.add_middleware(ElasticAPM, client=elasticapm_client)
 
     return app
@@ -226,3 +239,32 @@ def test_starlette_transaction_ignore_urls(app, elasticapm_client):
     elasticapm_client.config.update(1, transaction_ignore_urls="/*ello,/world")
     response = client.get("/hello")
     assert len(elasticapm_client.events[constants.TRANSACTION]) == 1
+
+
+def test_transaction_name_is_route(app, elasticapm_client):
+    client = TestClient(app)
+
+    response = client.get("/hi/shay")
+
+    assert response.status_code == 200
+
+    assert len(elasticapm_client.events[constants.TRANSACTION]) == 1
+    transaction = elasticapm_client.events[constants.TRANSACTION][0]
+    assert transaction["name"] == "GET /hi/{name}"
+    assert transaction["context"]["request"]["url"]["pathname"] == "/hi/shay"
+
+
+@pytest.mark.parametrize(
+    "url,expected",
+    (
+        ("/hi/shay/with/slash", "GET /hi/{name}/with/slash"),
+        ("/hi/shay/without/slash/", "GET /hi/{name}/without/slash/"),
+    ),
+)
+def test_trailing_slash_redirect_detection(app, elasticapm_client, url, expected):
+    client = TestClient(app)
+    response = client.get(url, allow_redirects=False)
+    assert response.status_code == 307
+    assert len(elasticapm_client.events[constants.TRANSACTION]) == 1
+    for transaction in elasticapm_client.events[constants.TRANSACTION]:
+        assert transaction["name"] == expected