Repository: DataDog/dd-trace-py
Pull request #12644: chore(llmobs): add telemetry on span event size/truncation
Author: login: Yun-Kim
Labels: name: changelog/no-changelog
description: A changelog entry is not required for this PR.
Created: 2025-03-05T21:01:42Z | Merged: 2025-03-10T22:22:20Z

[MLOB-2355]

This PR adds the following telemetry metrics `dd.instrumentation_telemetry_data.span.raw_size/size` to report encoded span sizes (prior and post truncation). Each metric is tagged with `span_kind, autoinstrumented, integration, error, truncated`, where `truncated` indicates if the span was truncated before sending due to size limitations.

This PR also performs some clenaup and minor type fixes to the other telemetry metrics for llmobs.

## Checklist
- [x] PR author has checked that all the criteria below are met
- The PR description includes an overview of the change
- The PR description articulates the motivation for the change
- The change includes tests OR the PR description describes a testing strategy
- The PR description notes risks associated with the change, if any
- Newly-added code is easy to change
- The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
- The change includes or references documentation updates if necessary
- Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))

## Reviewer Checklist
- [x] Reviewer has checked that all the criteria below are met 
- Title is accurate
- All changes are related to the pull request's stated goal
- Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes
- Testing strategy adequately addresses listed risks
- Newly-added code is easy to change
- Release note makes sense to a user of the library
- If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
- Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)


[MLOB-2355]: https://datadoghq.atlassian.net/browse/MLOB-2355?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ


Diff:
diff --git a/ddtrace/llmobs/_telemetry.py b/ddtrace/llmobs/_telemetry.py
index ad03c334eb4..cc25fda96a0 100644
--- a/ddtrace/llmobs/_telemetry.py
+++ b/ddtrace/llmobs/_telemetry.py
@@ -9,9 +9,39 @@
 from ddtrace.llmobs._constants import ROOT_PARENT_ID
 from ddtrace.llmobs._constants import SESSION_ID
 from ddtrace.llmobs._constants import SPAN_KIND
+from ddtrace.llmobs._writer import LLMObsSpanEvent
 from ddtrace.trace import Span
 
 
+class LLMObsTelemetryMetrics:
+    INIT_TIME = "init_time"
+    ENABLED = "product_enabled"
+    RAW_SPAN_SIZE = "span.raw_size"
+    SPAN_SIZE = "span.size"
+    SPAN_STARTED = "span.start"
+    SPAN_FINISHED = "span.finished"
+
+
+def _find_integration_from_tags(tags):
+    integration_tag = next((tag for tag in tags if tag.startswith("integration:")), None)
+    if not integration_tag:
+        return None
+    return integration_tag.split("integration:")[-1]
+
+
+def _get_tags_from_span_event(event: LLMObsSpanEvent):
+    span_kind = event.get("meta", {}).get("span.kind", "")
+    integration = _find_integration_from_tags(event.get("tags", []))
+    autoinstrumented = integration is not None
+    error = event.get("status") == "error"
+    return [
+        ("span_kind", span_kind),
+        ("autoinstrumented", str(int(autoinstrumented))),
+        ("error", str(int(error))),
+        ("integration", integration if integration else "N/A"),
+    ]
+
+
 def record_llmobs_enabled(error: Optional[str], agentless_enabled: bool, site: str, start_ns: int, auto: bool):
     tags = [
         ("agentless", str(int(agentless_enabled))),
@@ -23,15 +53,17 @@ def record_llmobs_enabled(error: Optional[str], agentless_enabled: bool, site: s
         tags.append(("error_type", error))
     init_time_ms = (time.time_ns() - start_ns) / 1e6
     telemetry_writer.add_distribution_metric(
-        namespace=TELEMETRY_NAMESPACE.MLOBS, name="init_time", value=init_time_ms, tags=tuple(tags)
+        namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.INIT_TIME, value=init_time_ms, tags=tuple(tags)
     )
     telemetry_writer.add_count_metric(
-        namespace=TELEMETRY_NAMESPACE.MLOBS, name="product_enabled", value=1, tags=tuple(tags)
+        namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.ENABLED, value=1, tags=tuple(tags)
     )
 
 
 def record_span_started():
-    telemetry_writer.add_count_metric(namespace=TELEMETRY_NAMESPACE.MLOBS, name="span.start", value=1)
+    telemetry_writer.add_count_metric(
+        namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.SPAN_STARTED, value=1
+    )
 
 
 def record_span_created(span: Span):
@@ -48,14 +80,30 @@ def record_span_created(span: Span):
         ("has_session_id", str(int(has_session_id))),
         ("is_root_span", str(int(is_root_span))),
         ("span_kind", span_kind or "N/A"),
+        ("integration", integration or "N/A"),
         ("error", str(span.error)),
     ]
-    if autoinstrumented:
-        tags.append(("integration", integration))
-    else:
+    if not autoinstrumented:
         tags.append(("decorator", str(int(decorator))))
     if model_provider:
         tags.append(("model_provider", model_provider))
     telemetry_writer.add_count_metric(
-        namespace=TELEMETRY_NAMESPACE.MLOBS, name="span.finished", value=1, tags=tuple(tags)
+        namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.SPAN_FINISHED, value=1, tags=tuple(tags)
+    )
+
+
+def record_span_event_raw_size(event: LLMObsSpanEvent, raw_event_size: int):
+    telemetry_writer.add_distribution_metric(
+        namespace=TELEMETRY_NAMESPACE.MLOBS,
+        name=LLMObsTelemetryMetrics.RAW_SPAN_SIZE,
+        value=raw_event_size,
+        tags=tuple(_get_tags_from_span_event(event)),
+    )
+
+
+def record_span_event_size(event: LLMObsSpanEvent, event_size: int, truncated: bool):
+    tags = _get_tags_from_span_event(event)
+    tags.append(("truncated", str(int(truncated))))
+    telemetry_writer.add_distribution_metric(
+        namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.SPAN_SIZE, value=event_size, tags=tuple(tags)
     )
diff --git a/ddtrace/llmobs/_writer.py b/ddtrace/llmobs/_writer.py
index 7c8e5dae9a9..b6673d6c27d 100644
--- a/ddtrace/llmobs/_writer.py
+++ b/ddtrace/llmobs/_writer.py
@@ -22,6 +22,7 @@
 from ddtrace.internal.periodic import PeriodicService
 from ddtrace.internal.writer import HTTPWriter
 from ddtrace.internal.writer import WriterClientBase
+from ddtrace.llmobs import _telemetry as telemetry
 from ddtrace.llmobs._constants import AGENTLESS_ENDPOINT
 from ddtrace.llmobs._constants import DROPPED_IO_COLLECTION_ERROR
 from ddtrace.llmobs._constants import DROPPED_VALUE_TEXT
@@ -282,21 +283,25 @@ def stop(self, timeout=None):
             super(LLMObsSpanWriter, self).stop(timeout=timeout)
 
     def enqueue(self, event: LLMObsSpanEvent) -> None:
-        event_size = len(safe_json(event))
+        raw_event_size = len(safe_json(event))
+        should_truncate = raw_event_size >= EVP_EVENT_SIZE_LIMIT
 
-        if event_size >= EVP_EVENT_SIZE_LIMIT:
+        if should_truncate:
             logger.warning(
                 "dropping event input/output because its size (%d) exceeds the event size limit (1MB)",
-                event_size,
+                raw_event_size,
             )
             event = _truncate_span_event(event)
 
         for client in self._clients:
             if isinstance(client, LLMObsEventClient) and isinstance(client.encoder, LLMObsSpanEncoder):
                 with client.encoder._lock:
-                    if (client.encoder.buffer_size + event_size) > EVP_PAYLOAD_SIZE_LIMIT:
+                    if (client.encoder.buffer_size + raw_event_size) > EVP_PAYLOAD_SIZE_LIMIT:
                         logger.debug("flushing queue because queuing next event will exceed EVP payload limit")
                         self._flush_queue_with_client(client)
+
+        telemetry.record_span_event_raw_size(event, raw_event_size)
+        telemetry.record_span_event_size(event, len(event), should_truncate)
         self.write([event])
 
     # Noop to make it compatible with HTTPWriter interface
diff --git a/tests/llmobs/_utils.py b/tests/llmobs/_utils.py
index c379d184f09..26447830b5a 100644
--- a/tests/llmobs/_utils.py
+++ b/tests/llmobs/_utils.py
@@ -268,7 +268,7 @@ def _completion_event():
         "tags": ["version:", "env:", "service:tests.llmobs", "source:integration"],
         "start_ns": 1707763310981223236,
         "duration": 12345678900,
-        "error": 0,
+        "status": "ok",
         "meta": {
             "span.kind": "llm",
             "model_name": "ada",
@@ -299,7 +299,7 @@ def _chat_completion_event():
         "tags": ["version:", "env:", "service:tests.llmobs", "source:integration"],
         "start_ns": 1707763310981223936,
         "duration": 12345678900,
-        "error": 0,
+        "status": "ok",
         "meta": {
             "span.kind": "llm",
             "model_name": "gpt-3.5-turbo",
@@ -337,7 +337,7 @@ def _chat_completion_event_with_unserializable_field():
         "tags": ["version:", "env:", "service:tests.llmobs", "source:integration"],
         "start_ns": 1707763310981223936,
         "duration": 12345678900,
-        "error": 0,
+        "status": "ok",
         "meta": {
             "span.kind": "llm",
             "model_name": "gpt-3.5-turbo",
@@ -376,7 +376,7 @@ def _large_event():
         "tags": ["version:", "env:", "service:tests.llmobs", "source:integration"],
         "start_ns": 1707763310981223936,
         "duration": 12345678900,
-        "error": 0,
+        "status": "ok",
         "meta": {
             "span.kind": "llm",
             "model_name": "gpt-3.5-turbo",
@@ -414,7 +414,7 @@ def _oversized_llm_event():
         "tags": ["version:", "env:", "service:tests.llmobs", "source:integration"],
         "start_ns": 1707763310981223936,
         "duration": 12345678900,
-        "error": 0,
+        "status": "ok",
         "meta": {
             "span.kind": "llm",
             "model_name": "gpt-3.5-turbo",
@@ -452,7 +452,7 @@ def _oversized_workflow_event():
         "tags": ["version:", "env:", "service:tests.llmobs", "source:integration"],
         "start_ns": 1707763310981223936,
         "duration": 12345678900,
-        "error": 0,
+        "status": "ok",
         "meta": {
             "span.kind": "workflow",
             "input": {"value": "A" * 700_000},
@@ -472,7 +472,7 @@ def _oversized_retrieval_event():
         "tags": ["version:", "env:", "service:tests.llmobs", "source:integration"],
         "start_ns": 1707763310981223936,
         "duration": 12345678900,
-        "error": 0,
+        "status": "ok",
         "meta": {
             "span.kind": "retrieval",
             "input": {"documents": {"content": "A" * 700_000}},
diff --git a/tests/llmobs/test_llmobs_span_agent_writer.py b/tests/llmobs/test_llmobs_span_agent_writer.py
index 55f0a56e4d5..a14a7e626ad 100644
--- a/tests/llmobs/test_llmobs_span_agent_writer.py
+++ b/tests/llmobs/test_llmobs_span_agent_writer.py
@@ -56,9 +56,9 @@ def test_truncating_oversized_events(mock_writer_logs, mock_http_writer_send_pay
     llmobs_span_writer.enqueue(_oversized_workflow_event())
     mock_writer_logs.warning.assert_has_calls(
         [
-            mock.call("dropping event input/output because its size (%d) exceeds the event size limit (1MB)", 1400720),
-            mock.call("dropping event input/output because its size (%d) exceeds the event size limit (1MB)", 1400460),
-            mock.call("dropping event input/output because its size (%d) exceeds the event size limit (1MB)", 1400441),
+            mock.call("dropping event input/output because its size (%d) exceeds the event size limit (1MB)", 1400724),
+            mock.call("dropping event input/output because its size (%d) exceeds the event size limit (1MB)", 1400464),
+            mock.call("dropping event input/output because its size (%d) exceeds the event size limit (1MB)", 1400445),
         ]
     )
 
diff --git a/tests/llmobs/test_llmobs_span_agentless_writer.py b/tests/llmobs/test_llmobs_span_agentless_writer.py
index a4fc7b9b162..4224587a8c3 100644
--- a/tests/llmobs/test_llmobs_span_agentless_writer.py
+++ b/tests/llmobs/test_llmobs_span_agentless_writer.py
@@ -64,13 +64,13 @@ def test_truncating_oversized_events(mock_writer_logs, mock_http_writer_send_pay
         mock_writer_logs.warning.assert_has_calls(
             [
                 mock.call(
-                    "dropping event input/output because its size (%d) exceeds the event size limit (1MB)", 1400720
+                    "dropping event input/output because its size (%d) exceeds the event size limit (1MB)", 1400724
                 ),
                 mock.call(
-                    "dropping event input/output because its size (%d) exceeds the event size limit (1MB)", 1400460
+                    "dropping event input/output because its size (%d) exceeds the event size limit (1MB)", 1400464
                 ),
                 mock.call(
-                    "dropping event input/output because its size (%d) exceeds the event size limit (1MB)", 1400441
+                    "dropping event input/output because its size (%d) exceeds the event size limit (1MB)", 1400445
                 ),
             ]
         )


Reviews:
user: login: lievan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T19:12:10Z
user: login: Kyle-Verhoog
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T19:13:00Z
user: login: Kyle-Verhoog
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T19:22:13Z
user: login: Yun-Kim
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T20:31:05Z
user: login: Yun-Kim
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T20:39:46Z
user: login: Yun-Kim
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T21:11:22Z
user: login: lievan
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T20:13:35Z

Review comments:
diff_hunk: @@ -268,7 +268,7 @@ def _completion_event():
         "tags": ["version:", "env:", "service:tests.llmobs", "source:integration"],
         "start_ns": 1707763310981223236,
         "duration": 12345678900,
-        "error": 0,
+        "status": "ok",
path: tests/llmobs/_utils.py
user: login: lievan
body: looks like we didn't have any tests that relied on asserting equality on these events, so this change is just a cleanup?
created_at: 2025-03-07T19:08:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -268,7 +268,7 @@ def _completion_event():
         "tags": ["version:", "env:", "service:tests.llmobs", "source:integration"],
         "start_ns": 1707763310981223236,
         "duration": 12345678900,
-        "error": 0,
path: tests/llmobs/_utils.py
user: login: Kyle-Verhoog
body: 🤔 why did this change?
created_at: 2025-03-07T19:11:33Z
author_association: MEMBER
diff_hunk: @@ -10,8 +25,27 @@ def record_llmobs_enabled(error, agentless_enabled, site, start_ns):
         tags.extend([("error", "1"), ("error_type", error)])
     init_time_ms = (time.time_ns() - start_ns) / 1e6
     telemetry_writer.add_distribution_metric(
-        namespace=TELEMETRY_NAMESPACE.MLOBS, name="llmobs.init_time", value=init_time_ms, tags=tuple(tags)
+        namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.INIT_TIME, value=init_time_ms, tags=tuple(tags)
     )
     telemetry_writer.add_count_metric(
-        namespace=TELEMETRY_NAMESPACE.MLOBS, name="llmobs.enabled", value=1, tags=tuple(tags)
+        namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.ENABLED, value=1, tags=tuple(tags)
+    )
+
+
+def record_span_event_size(event: LLMObsSpanEvent, event_size: int):
+    span_kind = event.get("meta", {}).get("span.kind", "")
+    integration = _find_integration_from_tags(event.get("tags", []))
path: ddtrace/llmobs/_telemetry.py
user: login: lievan
body: do we record the `integration:..` tag? i cant seem to find it in our code base, nor do i see it on spans im sending now

or is the tagging being done in another pr?
created_at: 2025-03-07T19:12:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -10,8 +25,27 @@ def record_llmobs_enabled(error, agentless_enabled, site, start_ns):
         tags.extend([("error", "1"), ("error_type", error)])
     init_time_ms = (time.time_ns() - start_ns) / 1e6
     telemetry_writer.add_distribution_metric(
-        namespace=TELEMETRY_NAMESPACE.MLOBS, name="llmobs.init_time", value=init_time_ms, tags=tuple(tags)
+        namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.INIT_TIME, value=init_time_ms, tags=tuple(tags)
     )
     telemetry_writer.add_count_metric(
-        namespace=TELEMETRY_NAMESPACE.MLOBS, name="llmobs.enabled", value=1, tags=tuple(tags)
+        namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.ENABLED, value=1, tags=tuple(tags)
+    )
+
+
+def record_span_event_size(event: LLMObsSpanEvent, event_size: int):
+    span_kind = event.get("meta", {}).get("span.kind", "")
+    integration = _find_integration_from_tags(event.get("tags", []))
+    autoinstrumented = 1 if integration is not None else 0
+    error = 1 if event.get("status") == "error" else 0
+    tags = [
+        ("span_kind", span_kind),
+        ("autoinstrumented", str(autoinstrumented)),
+        ("error", str(error)),
+    ]
+    if integration:
+        tags.append(("integration", integration))
+    if event_size > EVP_EVENT_SIZE_LIMIT:
path: ddtrace/llmobs/_telemetry.py
user: login: Kyle-Verhoog
body: instead of reimplementing the truncation logic i'd prefer to emit the metric in the same control flow that we do in the outer function (just to ensure future changes to truncation will be tracked)
created_at: 2025-03-07T19:12:48Z
author_association: MEMBER
diff_hunk: @@ -10,8 +25,27 @@ def record_llmobs_enabled(error, agentless_enabled, site, start_ns):
         tags.extend([("error", "1"), ("error_type", error)])
     init_time_ms = (time.time_ns() - start_ns) / 1e6
     telemetry_writer.add_distribution_metric(
-        namespace=TELEMETRY_NAMESPACE.MLOBS, name="llmobs.init_time", value=init_time_ms, tags=tuple(tags)
+        namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.INIT_TIME, value=init_time_ms, tags=tuple(tags)
     )
     telemetry_writer.add_count_metric(
-        namespace=TELEMETRY_NAMESPACE.MLOBS, name="llmobs.enabled", value=1, tags=tuple(tags)
+        namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.ENABLED, value=1, tags=tuple(tags)
+    )
+
+
+def record_span_event_size(event: LLMObsSpanEvent, event_size: int):
+    span_kind = event.get("meta", {}).get("span.kind", "")
+    integration = _find_integration_from_tags(event.get("tags", []))
+    autoinstrumented = 1 if integration is not None else 0
+    error = 1 if event.get("status") == "error" else 0
+    tags = [
+        ("span_kind", span_kind),
+        ("autoinstrumented", str(autoinstrumented)),
+        ("error", str(error)),
+    ]
+    if integration:
+        tags.append(("integration", integration))
+    if event_size > EVP_EVENT_SIZE_LIMIT:
+        tags.append(("truncated", "1"))
+    telemetry_writer.add_distribution_metric(
+        namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.SPAN_SIZE, value=event_size, tags=tuple(tags)
path: ddtrace/llmobs/_telemetry.py
user: login: Kyle-Verhoog
body: this metric is a little tricky - does it cover the raw size or the size post truncation? I think we'd find value in having both: the raw event size; and then the actual size that we submit to the intake.

I'd rather we report these separately with the actual size being tagged with whether truncation has occurred.
created_at: 2025-03-07T19:22:13Z
author_association: MEMBER
diff_hunk: @@ -268,7 +268,7 @@ def _completion_event():
         "tags": ["version:", "env:", "service:tests.llmobs", "source:integration"],
         "start_ns": 1707763310981223236,
         "duration": 12345678900,
-        "error": 0,
+        "status": "ok",
path: tests/llmobs/_utils.py
user: login: Yun-Kim
body: This is actually a required change. Our test fixtures for expected span events actually continued to use old `error` span event fields instead of `status`, and our telemetry metric tags rely on this `status` field.
created_at: 2025-03-07T20:31:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -10,8 +25,27 @@ def record_llmobs_enabled(error, agentless_enabled, site, start_ns):
         tags.extend([("error", "1"), ("error_type", error)])
     init_time_ms = (time.time_ns() - start_ns) / 1e6
     telemetry_writer.add_distribution_metric(
-        namespace=TELEMETRY_NAMESPACE.MLOBS, name="llmobs.init_time", value=init_time_ms, tags=tuple(tags)
+        namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.INIT_TIME, value=init_time_ms, tags=tuple(tags)
     )
     telemetry_writer.add_count_metric(
-        namespace=TELEMETRY_NAMESPACE.MLOBS, name="llmobs.enabled", value=1, tags=tuple(tags)
+        namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.ENABLED, value=1, tags=tuple(tags)
+    )
+
+
+def record_span_event_size(event: LLMObsSpanEvent, event_size: int):
+    span_kind = event.get("meta", {}).get("span.kind", "")
+    integration = _find_integration_from_tags(event.get("tags", []))
path: ddtrace/llmobs/_telemetry.py
user: login: Yun-Kim
body: Good catch - this should build on top of #12614. Gonna change the branch accordingly
created_at: 2025-03-07T20:39:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -268,7 +268,7 @@ def _completion_event():
         "tags": ["version:", "env:", "service:tests.llmobs", "source:integration"],
         "start_ns": 1707763310981223236,
         "duration": 12345678900,
-        "error": 0,
path: tests/llmobs/_utils.py
user: login: Yun-Kim
body: This was an outdated fixture ([context](https://github.com/DataDog/dd-trace-py/pull/12644#discussion_r1985665120): we switched from `error` to `status` as the primary span event field, but this PR relies on the `status` field on the span event itself) 
created_at: 2025-03-07T21:11:22Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-03-05T21:02:13Z
body: `CODEOWNERS` have been resolved as:

```
ddtrace/llmobs/_telemetry.py                                            @DataDog/ml-observability
ddtrace/llmobs/_writer.py                                               @DataDog/ml-observability
tests/llmobs/_utils.py                                                  @DataDog/ml-observability
tests/llmobs/test_llmobs_span_agent_writer.py                           @DataDog/ml-observability
tests/llmobs/test_llmobs_span_agentless_writer.py                       @DataDog/ml-observability
```
<!-- thollander/actions-comment-pull-request "codeowners_resolved" -->
author_association: CONTRIBUTOR
user: login: pr-commenter[bot]
created_at: 2025-03-05T21:44:52Z
body: ## Benchmarks

Benchmark execution time: 2025-03-10 17:37:54

Comparing candidate commit ed00d0ae56117da6f91d17a8a06d990425c2ddb2 in PR branch `yunkim/llmobs-telemetry-payloads` with baseline commit bcab4496e3865596bbaedd839bcc63b77b062160 in branch `main`.

Found 0 performance improvements and 3 performance regressions! Performance is the same for 465 metrics, 2 unstable metrics.

#### scenario:iast_aspects-ospathbasename_aspect

- 🟥 `execution_time` [+329.840ns; +391.033ns] or [+9.781%; +11.596%]

#### scenario:iast_aspects-ospathjoin_aspect

- 🟥 `execution_time` [+463.221ns; +525.331ns] or [+9.144%; +10.370%]

#### scenario:iast_aspects-replace_aspect

- 🟥 `execution_time` [+565.842ns; +664.068ns] or [+8.634%; +10.133%]




author_association: NONE