Repository: getsentry/sentry
Pull request: analytics(toolbar): record api requests in custom middleware
Closes https://github.com/getsentry/sentry-toolbar/issues/33

We're ready to release a developer toolbar for installation on 3rd party sites. Once installed, toolbars will query sentry backend for data - see the [repo](https://github.com/getsentry/sentry-toolbar) or /devtoolbar folder of sentry for examples. This helps us get analytics on every endpoint, whenever the `querryReferrer=devtoolbar` header is included.

Problem statement:
[DevToolbar] Analytics for which orgs have the toolbar enabled
I would like to have metrics/analytics on the server side to show what endpoints the toolbar is hitting. This should be possible by looking at incoming requests to the api where the `?queryReferrer=devtoolbar` url query param is set. 

The purpose of these metrics/analytics would be to:
- Detect when an organization and/or project has setup the toolbar
    - "has setup the toolbar" also depends on having the project setting configured, which is another metric we should have visibility into
- Log the volume of api requests generated by people using the toolbar. it shouldn't ever generate so much load as to be an issue, but we should have visibility into it
- See which endpoints are hit most often from the toolbar, so we can infer usage of different panels
- See which endpoints ar ehit most often from the toolabr, so we can think about optimizations to better suit use-cases

With this kind of backend logging we'd be able to make a trade-off: no user-specific logging, but it's also not something users can opt-out of. We want to observe the infra and load.

So we should aim to create some charts like:
- Total number of orgs using (sending requests with the query param) the toolbar
- Total number of projects using (sending requests to a specific endpoint with the query param) the toolbar
- Total unique viewers/users of the toolbar
- unique users per org
- Volume of requests per endpoint, where the query param is set.



It would also be cool to have stats about the orgs that are using the toolbar. So like a table with columns: "org_slug, # of users total, # toolbar users, # issues, # issues resvoled" So that we can start to learn about how the toolbar can be more helpful. This might not need to be on a dashboard, more like a one-off (or something we query every month) kind of query.

Hints from issue discussion:
@ryan953 could you link the sample dashboards/charts we were looking at?

Update: nvm, I remembered they're Hydration Errors, Rage Clicks, and Replays Search in redash. If there's other good examples lmk.

Commit message: Rm mixin from base classes

Patch to review:
diff --git a/src/sentry/api/analytics.py b/src/sentry/api/analytics.py
index 339c9bc5bc458d..27e3e71928eb27 100644
--- a/src/sentry/api/analytics.py
+++ b/src/sentry/api/analytics.py
@@ -33,6 +33,23 @@ class GroupSimilarIssuesEmbeddingsCountEvent(analytics.Event):
     )
 
 
+class DevToolbarRequestEvent(analytics.Event):
+    type = "devtoolbar.request"
+
+    attributes = (
+        analytics.Attribute("path"),  # path to endpoint
+        analytics.Attribute("query"),  # string or dict?
+        analytics.Attribute("origin"),
+        analytics.Attribute("organization_id", required=False),
+        analytics.Attribute("organization_slug", required=False),
+        analytics.Attribute("project_id", required=False),
+        analytics.Attribute("project_slug", required=False),
+        analytics.Attribute("issue_id", required=False),
+        analytics.Attribute("user_id"),  # needed to aggregate/send to amplitude(?)
+    )
+
+
 analytics.register(OrganizationSavedSearchCreatedEvent)
 analytics.register(OrganizationSavedSearchDeletedEvent)
 analytics.register(GroupSimilarIssuesEmbeddingsCountEvent)
+analytics.register(DevToolbarRequestEvent)
diff --git a/src/sentry/conf/server.py b/src/sentry/conf/server.py
index c8842590162126..f0abc1be623bf5 100644
--- a/src/sentry/conf/server.py
+++ b/src/sentry/conf/server.py
@@ -346,6 +346,7 @@ def env(
     "sentry.middleware.locale.SentryLocaleMiddleware",
     "sentry.middleware.ratelimit.RatelimitMiddleware",
     "django.contrib.messages.middleware.MessageMiddleware",
+    "sentry.middleware.devtoolbar.AnalyticsMiddleware",
 )
 
 ROOT_URLCONF = "sentry.conf.urls"
diff --git a/src/sentry/middleware/devtoolbar.py b/src/sentry/middleware/devtoolbar.py
new file mode 100644
index 00000000000000..75a238ad4d0c0a
--- /dev/null
+++ b/src/sentry/middleware/devtoolbar.py
@@ -0,0 +1,64 @@
+import logging
+
+from sentry import analytics
+from sentry.utils.http import (
+    get_api_path_from_request,
+    origin_from_request,
+    parse_id_or_slug_param,
+    query_string,
+)
+
+logger = logging.getLogger(__name__)
+
+
+class AnalyticsMiddleware:
+    def __init__(self, get_response):
+        self.get_response = get_response
+
+    def __call__(self, request):
+        return self.get_response(request)
+
+    def process_view(self, request, _view_func, _view_args, view_kwargs) -> None:
+        # Intercepts requests right before they're passed to the Endpoint/View.
+        org_id, org_slug, project_id, project_slug = None, None, None, None
+        try:
+            if request.headers.get("queryReferrer") == "devtoolbar":
+                org_id_or_slug = view_kwargs.get(
+                    "organization_id_or_slug", view_kwargs.get("organization_slug")
+                )
+                org_id, org_slug = parse_id_or_slug_param(org_id_or_slug)
+
+                project_id_or_slug = view_kwargs.get("project_id_or_slug")
+                project_id, project_slug = parse_id_or_slug_param(project_id_or_slug)
+
+                issue_id = view_kwargs.get("issue_id")
+                scope = (
+                    "group"
+                    if issue_id
+                    else "project"
+                    if project_id_or_slug
+                    else "organization"
+                    if org_id_or_slug
+                    else None
+                )
+
+                origin = origin_from_request(request)
+                query = query_string(request)  # starts with '?'
+                path = get_api_path_from_request(request, scope)
+                analytics.record(
+                    "devtoolbar.request",
+                    path=path,
+                    query=query,
+                    origin=origin,
+                    organization_id=str(org_id) if org_id else None,
+                    organization_slug=org_slug,
+                    project_id=str(project_id) if project_id else None,
+                    project_slug=project_slug,
+                    issue_id=issue_id,
+                    user_id=str(request.user.id) if request.user else None,
+                )
+        except Exception:
+            logger.exception(
+                "devtoolbar: failed to record api analytics event.",
+                extra={"org_slug": org_slug, "project_slug": project_slug},
+            )
diff --git a/src/sentry/utils/http.py b/src/sentry/utils/http.py
index b0d8e293dc23d4..0b84a139a80bb1 100644
--- a/src/sentry/utils/http.py
+++ b/src/sentry/utils/http.py
@@ -226,3 +226,32 @@ class _HttpRequestWithSubdomain(HttpRequest):
 
 def is_using_customer_domain(request: HttpRequest) -> TypeGuard[_HttpRequestWithSubdomain]:
     return bool(hasattr(request, "subdomain") and request.subdomain)
+
+
+def get_api_path_from_request(request: HttpRequest, endpoint_scope: str | None) -> str:
+    """
+    Returns request.path without the specific org, project, or group identifiers, depending on `scope`.
+
+    @param request        - the request object.
+    @param endpoint_scope - "organization", "project", "group", or None.
+    """
+    segments = request.path.split("/")
+    if endpoint_scope is not None:
+        segments[2] = "<organization_id_or_slug>"
+
+    if endpoint_scope == "project":
+        segments[3] = "<project_id_or_slug>"
+    elif endpoint_scope == "group":
+        # segments[3] is either "issues" or "groups"
+        segments[4] = "<issue_id>"
+
+    return "/" + "/".join(segments) + "/"
+
+
+def parse_id_or_slug_param(id_or_slug: str | None) -> tuple[int | None, str | None]:
+    if not id_or_slug:
+        return None, None
+
+    if id_or_slug.isnumeric():
+        return int(id_or_slug), None
+    return None, id_or_slug


Code review comments:
diff_hunk: @@ -0,0 +1,64 @@
+import logging
+
+from sentry import analytics
+from sentry.utils.http import (
+    get_api_path_from_request,
+    origin_from_request,
+    parse_id_or_slug_param,
+    query_string,
+)
+
+logger = logging.getLogger(__name__)
+
+
+class AnalyticsMiddleware:
path: src/sentry/middleware/devtoolbar.py
text: @user1:
can we call this `DevToolBarAnalytics`?

@author:
sure, can we keep the `Middleware` suffix?

@user1:
yeah

diff_hunk: @@ -0,0 +1,64 @@
+import logging
+
+from sentry import analytics
+from sentry.utils.http import (
+    get_api_path_from_request,
+    origin_from_request,
+    parse_id_or_slug_param,
+    query_string,
+)
+
+logger = logging.getLogger(__name__)
+
+
+class AnalyticsMiddleware:
+    def __init__(self, get_response):
+        self.get_response = get_response
+
+    def __call__(self, request):
+        return self.get_response(request)
+
+    def process_view(self, request, _view_func, _view_args, view_kwargs) -> None:
+        # Intercepts requests right before they're passed to the Endpoint/View.
+        org_id, org_slug, project_id, project_slug = None, None, None, None
+        try:
+            if request.headers.get("queryReferrer") == "devtoolbar":
path: src/sentry/middleware/devtoolbar.py
text: @user1:
prefer return early for readability https://refactoring.com/catalog/replaceNestedConditionalWithGuardClauses.html

diff_hunk: @@ -33,6 +33,23 @@ class GroupSimilarIssuesEmbeddingsCountEvent(analytics.Event):
     )
 
 
+class DevToolbarRequestEvent(analytics.Event):
+    type = "devtoolbar.request"
+
+    attributes = (
+        analytics.Attribute("path"),  # path to endpoint
+        analytics.Attribute("query"),  # string or dict?
+        analytics.Attribute("origin"),
+        analytics.Attribute("organization_id", required=False),
+        analytics.Attribute("organization_slug", required=False),
+        analytics.Attribute("project_id", required=False),
+        analytics.Attribute("project_slug", required=False),
+        analytics.Attribute("issue_id", required=False),
+        analytics.Attribute("user_id"),  # needed to aggregate/send to amplitude(?)
path: src/sentry/api/analytics.py
text: @user1:
best not to mention any specific analytics provider in our code