Repository: DataDog/dd-trace-py
Pull request #12689: chore(llmobs): telemetry metrics for LLMObs API usage
Author: login: Yun-Kim
Labels: name: changelog/no-changelog
description: A changelog entry is not required for this PR.
Created: 2025-03-10T18:58:49Z | Merged: 2025-03-11T19:11:43Z

[MLOB-2390]

This PR adds telemetry metrics for LLMObs API usage to see how many times users manually call the following API methods:
- `LLMObs.annotate()` --> `dd.instrumentation_telemetry_data.mlobs.annotations`
- `LLMObs.submit_evaluation/submit_evaluation_for()` --> `dd.instrumentation_telemetry_data.mlobs.evals_submitted`
- `LLMObs.export_span()` --> `dd.instrumentation_telemetry_data.mlobs.spans_exported`
- `LLMObs.flush()` --> `dd.instrumentation_telemetry_data.mlobs.user_flushes`
- `LLMObs.inject_distributed_headers()` --> `dd.instrumentation_telemetry_data.mlobs.inject_distributed_headers`
- `LLMObs.activate_distributed_headers()` --> `dd.instrumentation_telemetry_data.mlobs.activate_distributed_headers`

These telemetry metrics are tagged with `error: "1"/"0"`, `error_type: <ERROR_TYPE>`, `span_kind: <SPAN_KIND or "N/A">`, `is_root_span: "1"/"0"`. The `evals_submitted` metric is additionally tagged with `metric_type: "categorical"/"score"/"N/A"`, `custom_joining_key: "1"/"0"`.

The goal of these metrics is to see how many times the public API is being invoked by users, as well as to get a sense of how often manual instrumentation (which these methods provide support for) is being used, as well as some more details regarding instrumentation (errors, metric types, custom joining key vs span contexts, etc).

## Checklist
- [x] PR author has checked that all the criteria below are met
- The PR description includes an overview of the change
- The PR description articulates the motivation for the change
- The change includes tests OR the PR description describes a testing strategy
- The PR description notes risks associated with the change, if any
- Newly-added code is easy to change
- The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
- The change includes or references documentation updates if necessary
- Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))

## Reviewer Checklist
- [x] Reviewer has checked that all the criteria below are met 
- Title is accurate
- All changes are related to the pull request's stated goal
- Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes
- Testing strategy adequately addresses listed risks
- Newly-added code is easy to change
- Release note makes sense to a user of the library
- If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
- Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)


[MLOB-2390]: https://datadoghq.atlassian.net/browse/MLOB-2390?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ


Diff:
diff --git a/ddtrace/llmobs/_llmobs.py b/ddtrace/llmobs/_llmobs.py
index 17352e77b6d..61ead184924 100644
--- a/ddtrace/llmobs/_llmobs.py
+++ b/ddtrace/llmobs/_llmobs.py
@@ -530,17 +530,22 @@ def flush(cls) -> None:
             log.warning("flushing when LLMObs is disabled. No spans or evaluation metrics will be sent.")
             return
 
+        error = None
         try:
             cls._instance._evaluator_runner.periodic()
         except Exception:
+            error = "evaluator_flush_error"
             log.warning("Failed to run evaluator runner.", exc_info=True)
 
         try:
             cls._instance._llmobs_span_writer.periodic()
             cls._instance._llmobs_eval_metric_writer.periodic()
         except Exception:
+            error = "writer_flush_error"
             log.warning("Failed to flush LLMObs spans and evaluation metrics.", exc_info=True)
 
+        telemetry.record_user_flush(error)
+
     @staticmethod
     def _patch_integrations() -> None:
         """
@@ -570,16 +575,22 @@ def export_span(cls, span: Optional[Span] = None) -> Optional[ExportedLLMObsSpan
         if span is None:
             span = cls._instance._current_span()
             if span is None:
+                telemetry.record_span_exported(span, "no_active_span")
                 log.warning("No span provided and no active LLMObs-generated span found.")
                 return None
+        error = None
         try:
             if span.span_type != SpanTypes.LLM:
+                error = "invalid_span"
                 log.warning("Span must be an LLMObs-generated span.")
                 return None
             return ExportedLLMObsSpan(span_id=str(span.span_id), trace_id=format_trace_id(span.trace_id))
         except (TypeError, AttributeError):
+            error = "invalid_span"
             log.warning("Failed to export span. Span must be a valid Span object.")
             return None
+        finally:
+            telemetry.record_span_exported(span, error)
 
     def _current_span(self) -> Optional[Span]:
         """Returns the currently active LLMObs-generated span.
@@ -875,56 +886,67 @@ def annotate(
         :param metrics: Dictionary of JSON serializable key-value metric pairs,
                         such as `{prompt,completion,total}_tokens`.
         """
-        if span is None:
-            span = cls._instance._current_span()
+        error = None
+        try:
             if span is None:
-                log.warning("No span provided and no active LLMObs-generated span found.")
+                span = cls._instance._current_span()
+                if span is None:
+                    error = "invalid_span_no_active_spans"
+                    log.warning("No span provided and no active LLMObs-generated span found.")
+                    return
+            if span.span_type != SpanTypes.LLM:
+                error = "invalid_span_type"
+                log.warning("Span must be an LLMObs-generated span.")
                 return
-        if span.span_type != SpanTypes.LLM:
-            log.warning("Span must be an LLMObs-generated span.")
-            return
-        if span.finished:
-            log.warning("Cannot annotate a finished span.")
-            return
-        if metadata is not None:
-            if not isinstance(metadata, dict):
-                log.warning("metadata must be a dictionary")
-            else:
-                cls._set_dict_attribute(span, METADATA, metadata)
-        if metrics is not None:
-            if not isinstance(metrics, dict):
-                log.warning("metrics must be a dictionary of string key - numeric value pairs.")
-            else:
-                cls._set_dict_attribute(span, METRICS, metrics)
-        if tags is not None:
-            if not isinstance(tags, dict):
-                log.warning("span tags must be a dictionary of string key - primitive value pairs.")
-            else:
-                cls._set_dict_attribute(span, TAGS, tags)
-        span_kind = span._get_ctx_item(SPAN_KIND)
-        if _name is not None:
-            span.name = _name
-        if prompt is not None:
-            try:
-                validated_prompt = validate_prompt(prompt)
-                cls._set_dict_attribute(span, INPUT_PROMPT, validated_prompt)
-            except TypeError:
-                log.warning("Failed to validate prompt with error: ", exc_info=True)
-        if not span_kind:
-            log.debug("Span kind not specified, skipping annotation for input/output data")
-            return
-        if input_data is not None or output_data is not None:
-            if span_kind == "llm":
-                cls._tag_llm_io(span, input_messages=input_data, output_messages=output_data)
-            elif span_kind == "embedding":
-                cls._tag_embedding_io(span, input_documents=input_data, output_text=output_data)
-            elif span_kind == "retrieval":
-                cls._tag_retrieval_io(span, input_text=input_data, output_documents=output_data)
-            else:
-                cls._tag_text_io(span, input_value=input_data, output_value=output_data)
+            if span.finished:
+                error = "invalid_finished_span"
+                log.warning("Cannot annotate a finished span.")
+                return
+            if metadata is not None:
+                if not isinstance(metadata, dict):
+                    error = "invalid_metadata"
+                    log.warning("metadata must be a dictionary")
+                else:
+                    cls._set_dict_attribute(span, METADATA, metadata)
+            if metrics is not None:
+                if not isinstance(metrics, dict):
+                    error = "invalid_metrics"
+                    log.warning("metrics must be a dictionary of string key - numeric value pairs.")
+                else:
+                    cls._set_dict_attribute(span, METRICS, metrics)
+            if tags is not None:
+                if not isinstance(tags, dict):
+                    error = "invalid_tags"
+                    log.warning("span tags must be a dictionary of string key - primitive value pairs.")
+                else:
+                    cls._set_dict_attribute(span, TAGS, tags)
+            span_kind = span._get_ctx_item(SPAN_KIND)
+            if _name is not None:
+                span.name = _name
+            if prompt is not None:
+                try:
+                    validated_prompt = validate_prompt(prompt)
+                    cls._set_dict_attribute(span, INPUT_PROMPT, validated_prompt)
+                except TypeError:
+                    error = "invalid_prompt"
+                    log.warning("Failed to validate prompt with error: ", exc_info=True)
+            if not span_kind:
+                log.debug("Span kind not specified, skipping annotation for input/output data")
+                return
+            if input_data is not None or output_data is not None:
+                if span_kind == "llm":
+                    error = cls._tag_llm_io(span, input_messages=input_data, output_messages=output_data)
+                elif span_kind == "embedding":
+                    error = cls._tag_embedding_io(span, input_documents=input_data, output_text=output_data)
+                elif span_kind == "retrieval":
+                    error = cls._tag_retrieval_io(span, input_text=input_data, output_documents=output_data)
+                else:
+                    cls._tag_text_io(span, input_value=input_data, output_value=output_data)
+        finally:
+            telemetry.record_llmobs_annotate(span, error)
 
     @classmethod
-    def _tag_llm_io(cls, span, input_messages=None, output_messages=None):
+    def _tag_llm_io(cls, span, input_messages=None, output_messages=None) -> Optional[str]:
         """Tags input/output messages for LLM-kind spans.
         Will be mapped to span's `meta.{input,output}.messages` fields.
         """
@@ -936,19 +958,22 @@ def _tag_llm_io(cls, span, input_messages=None, output_messages=None):
                     span._set_ctx_item(INPUT_MESSAGES, input_messages.messages)
             except TypeError:
                 log.warning("Failed to parse input messages.", exc_info=True)
+                return "invalid_io_messages"
         if output_messages is None:
-            return
+            return None
         try:
             if not isinstance(output_messages, Messages):
                 output_messages = Messages(output_messages)
             if not output_messages.messages:
-                return
+                return None
             span._set_ctx_item(OUTPUT_MESSAGES, output_messages.messages)
         except TypeError:
             log.warning("Failed to parse output messages.", exc_info=True)
+            return "invalid_io_messages"
+        return None
 
     @classmethod
-    def _tag_embedding_io(cls, span, input_documents=None, output_text=None):
+    def _tag_embedding_io(cls, span, input_documents=None, output_text=None) -> Optional[str]:
         """Tags input documents and output text for embedding-kind spans.
         Will be mapped to span's `meta.{input,output}.text` fields.
         """
@@ -960,27 +985,31 @@ def _tag_embedding_io(cls, span, input_documents=None, output_text=None):
                     span._set_ctx_item(INPUT_DOCUMENTS, input_documents.documents)
             except TypeError:
                 log.warning("Failed to parse input documents.", exc_info=True)
+                return "invalid_embedding_io"
         if output_text is None:
-            return
+            return None
         span._set_ctx_item(OUTPUT_VALUE, str(output_text))
+        return None
 
     @classmethod
-    def _tag_retrieval_io(cls, span, input_text=None, output_documents=None):
+    def _tag_retrieval_io(cls, span, input_text=None, output_documents=None) -> Optional[str]:
         """Tags input text and output documents for retrieval-kind spans.
         Will be mapped to span's `meta.{input,output}.text` fields.
         """
         if input_text is not None:
             span._set_ctx_item(INPUT_VALUE, safe_json(input_text))
         if output_documents is None:
-            return
+            return None
         try:
             if not isinstance(output_documents, Documents):
                 output_documents = Documents(output_documents)
             if not output_documents.documents:
-                return
+                return None
             span._set_ctx_item(OUTPUT_DOCUMENTS, output_documents.documents)
         except TypeError:
             log.warning("Failed to parse output documents.", exc_info=True)
+            return "invalid_retrieval_io"
+        return None
 
     @classmethod
     def _tag_text_io(cls, span, input_value=None, output_value=None):
@@ -1040,99 +1069,115 @@ def submit_evaluation_for(
             )
             return
 
-        has_exactly_one_joining_key = (span is not None) ^ (span_with_tag_value is not None)
-
-        if not has_exactly_one_joining_key:
-            raise ValueError(
-                "Exactly one of `span` or `span_with_tag_value` must be specified to submit an evaluation metric."
-            )
-
+        error = None
         join_on = {}
-        if span is not None:
-            if (
-                not isinstance(span, dict)
-                or not isinstance(span.get("span_id"), str)
-                or not isinstance(span.get("trace_id"), str)
-            ):
-                raise TypeError(
-                    "`span` must be a dictionary containing both span_id and trace_id keys. "
-                    "LLMObs.export_span() can be used to generate this dictionary from a given span."
-                )
-            join_on["span"] = span
-        elif span_with_tag_value is not None:
-            if (
-                not isinstance(span_with_tag_value, dict)
-                or not isinstance(span_with_tag_value.get("tag_key"), str)
-                or not isinstance(span_with_tag_value.get("tag_value"), str)
-            ):
-                raise TypeError(
-                    "`span_with_tag_value` must be a dict with keys 'tag_key' and 'tag_value' containing string values"
+        try:
+            has_exactly_one_joining_key = (span is not None) ^ (span_with_tag_value is not None)
+
+            if not has_exactly_one_joining_key:
+                error = "provided_both_span_and_tag_joining_key"
+                raise ValueError(
+                    "Exactly one of `span` or `span_with_tag_value` must be specified to submit an evaluation metric."
                 )
-            join_on["tag"] = {
-                "key": span_with_tag_value.get("tag_key"),
-                "value": span_with_tag_value.get("tag_value"),
-            }
 
-        timestamp_ms = timestamp_ms if timestamp_ms else int(time.time() * 1000)
+            if span is not None:
+                if (
+                    not isinstance(span, dict)
+                    or not isinstance(span.get("span_id"), str)
+                    or not isinstance(span.get("trace_id"), str)
+                ):
+                    error = "invalid_span"
+                    raise TypeError(
+                        "`span` must be a dictionary containing both span_id and trace_id keys. "
+                        "LLMObs.export_span() can be used to generate this dictionary from a given span."
+                    )
+                join_on["span"] = span
+            elif span_with_tag_value is not None:
+                if (
+                    not isinstance(span_with_tag_value, dict)
+                    or not isinstance(span_with_tag_value.get("tag_key"), str)
+                    or not isinstance(span_with_tag_value.get("tag_value"), str)
+                ):
+                    error = "invalid_joining_key"
+                    raise TypeError(
+                        "`span_with_tag_value` must be a dict with keys 'tag_key' and 'tag_value' "
+                        "containing string values"
+                    )
+                join_on["tag"] = {
+                    "key": span_with_tag_value.get("tag_key"),
+                    "value": span_with_tag_value.get("tag_value"),
+                }
 
-        if not isinstance(timestamp_ms, int) or timestamp_ms < 0:
-            raise ValueError("timestamp_ms must be a non-negative integer. Evaluation metric data will not be sent")
+            timestamp_ms = timestamp_ms if timestamp_ms else int(time.time() * 1000)
 
-        if not label:
-            raise ValueError("label must be the specified name of the evaluation metric.")
+            if not isinstance(timestamp_ms, int) or timestamp_ms < 0:
+                error = "invalid_timestamp"
+                raise ValueError("timestamp_ms must be a non-negative integer. Evaluation metric data will not be sent")
 
-        metric_type = metric_type.lower()
-        if metric_type not in ("categorical", "score"):
-            raise ValueError("metric_type must be one of 'categorical' or 'score'.")
+            if not label:
+                error = "invalid_metric_label"
+                raise ValueError("label must be the specified name of the evaluation metric.")
 
-        if metric_type == "categorical" and not isinstance(value, str):
-            raise TypeError("value must be a string for a categorical metric.")
-        if metric_type == "score" and not isinstance(value, (int, float)):
-            raise TypeError("value must be an integer or float for a score metric.")
+            metric_type = metric_type.lower()
+            if metric_type not in ("categorical", "score"):
+                error = "invalid_metric_type"
+                raise ValueError("metric_type must be one of 'categorical' or 'score'.")
 
-        if tags is not None and not isinstance(tags, dict):
-            log.warning("tags must be a dictionary of string key-value pairs.")
-            tags = {}
+            if metric_type == "categorical" and not isinstance(value, str):
+                error = "invalid_metric_value"
+                raise TypeError("value must be a string for a categorical metric.")
+            if metric_type == "score" and not isinstance(value, (int, float)):
+                error = "invalid_metric_value"
+                raise TypeError("value must be an integer or float for a score metric.")
 
-        evaluation_tags = {
-            "ddtrace.version": ddtrace.__version__,
-            "ml_app": ml_app,
-        }
+            if tags is not None and not isinstance(tags, dict):
+                log.warning("tags must be a dictionary of string key-value pairs.")
+                tags = {}
 
-        if tags:
-            for k, v in tags.items():
-                try:
-                    evaluation_tags[ensure_text(k)] = ensure_text(v)
-                except TypeError:
-                    log.warning("Failed to parse tags. Tags for evaluation metrics must be strings.")
+            evaluation_tags = {
+                "ddtrace.version": ddtrace.__version__,
+                "ml_app": ml_app,
+            }
 
-        ml_app = ml_app if ml_app else config._llmobs_ml_app
-        if not ml_app:
-            log.warning(
-                "ML App name is required for sending evaluation metrics. Evaluation metric data will not be sent. "
-                "Ensure this configuration is set before running your application."
-            )
-            return
+            if tags:
+                for k, v in tags.items():
+                    try:
+                        evaluation_tags[ensure_text(k)] = ensure_text(v)
+                    except TypeError:
+                        error = "invalid_tags"
+                        log.warning("Failed to parse tags. Tags for evaluation metrics must be strings.")
 
-        evaluation_metric = {
-            "join_on": join_on,
-            "label": str(label),
-            "metric_type": metric_type,
-            "timestamp_ms": timestamp_ms,
-            "{}_value".format(metric_type): value,
-            "ml_app": ml_app,
-            "tags": ["{}:{}".format(k, v) for k, v in evaluation_tags.items()],
-        }
+            ml_app = ml_app if ml_app else config._llmobs_ml_app
+            if not ml_app:
+                error = "missing_ml_app"
+                log.warning(
+                    "ML App name is required for sending evaluation metrics. Evaluation metric data will not be sent. "
+                    "Ensure this configuration is set before running your application."
+                )
+                return
 
-        if metadata:
-            if not isinstance(metadata, dict):
-                log.warning("metadata must be json serializable dictionary.")
-            else:
-                metadata = safe_json(metadata)
-                if metadata and isinstance(metadata, str):
-                    evaluation_metric["metadata"] = json.loads(metadata)
+            evaluation_metric = {
+                "join_on": join_on,
+                "label": str(label),
+                "metric_type": metric_type,
+                "timestamp_ms": timestamp_ms,
+                "{}_value".format(metric_type): value,
+                "ml_app": ml_app,
+                "tags": ["{}:{}".format(k, v) for k, v in evaluation_tags.items()],
+            }
 
-        cls._instance._llmobs_eval_metric_writer.enqueue(evaluation_metric)
+            if metadata:
+                if not isinstance(metadata, dict):
+                    error = "invalid_metadata"
+                    log.warning("metadata must be json serializable dictionary.")
+                else:
+                    metadata = safe_json(metadata)
+                    if metadata and isinstance(metadata, str):
+                        evaluation_metric["metadata"] = json.loads(metadata)
+
+            cls._instance._llmobs_eval_metric_writer.enqueue(evaluation_metric)
+        finally:
+            telemetry.record_llmobs_submit_evaluation(join_on, metric_type, error)
 
     @classmethod
     def submit_evaluation(
@@ -1165,96 +1210,115 @@ def submit_evaluation(
                 "LLMObs.submit_evaluation() called when LLMObs is not enabled. Evaluation metric data will not be sent."
             )
             return
-        if not config._dd_api_key:
-            log.warning(
-                "DD_API_KEY is required for sending evaluation metrics. Evaluation metric data will not be sent. "
-                "Ensure this configuration is set before running your application."
-            )
-            return
-        if not isinstance(span_context, dict):
-            log.warning(
-                "span_context must be a dictionary containing both span_id and trace_id keys. "
-                "LLMObs.export_span() can be used to generate this dictionary from a given span."
-            )
-            return
+        error = None
+        try:
+            if not config._dd_api_key:
+                error = "missing_api_key"
+                log.warning(
+                    "DD_API_KEY is required for sending evaluation metrics. Evaluation metric data will not be sent. "
+                    "Ensure this configuration is set before running your application."
+                )
+                return
+            if not isinstance(span_context, dict):
+                error = "invalid_span"
+                log.warning(
+                    "span_context must be a dictionary containing both span_id and trace_id keys. "
+                    "LLMObs.export_span() can be used to generate this dictionary from a given span."
+                )
+                return
 
-        ml_app = ml_app if ml_app else config._llmobs_ml_app
-        if not ml_app:
-            log.warning(
-                "ML App name is required for sending evaluation metrics. Evaluation metric data will not be sent. "
-                "Ensure this configuration is set before running your application."
-            )
-            return
+            ml_app = ml_app if ml_app else config._llmobs_ml_app
+            if not ml_app:
+                error = "missing_ml_app"
+                log.warning(
+                    "ML App name is required for sending evaluation metrics. Evaluation metric data will not be sent. "
+                    "Ensure this configuration is set before running your application."
+                )
+                return
 
-        timestamp_ms = timestamp_ms if timestamp_ms else int(time.time() * 1000)
+            timestamp_ms = timestamp_ms if timestamp_ms else int(time.time() * 1000)
 
-        if not isinstance(timestamp_ms, int) or timestamp_ms < 0:
-            log.warning("timestamp_ms must be a non-negative integer. Evaluation metric data will not be sent")
-            return
+            if not isinstance(timestamp_ms, int) or timestamp_ms < 0:
+                error = "invalid_timestamp"
+                log.warning("timestamp_ms must be a non-negative integer. Evaluation metric data will not be sent")
+                return
 
-        span_id = span_context.get("span_id")
-        trace_id = span_context.get("trace_id")
-        if not (span_id and trace_id):
-            log.warning("span_id and trace_id must both be specified for the given evaluation metric to be submitted.")
-            return
-        if not label:
-            log.warning("label must be the specified name of the evaluation metric.")
-            return
+            span_id = span_context.get("span_id")
+            trace_id = span_context.get("trace_id")
+            if not (span_id and trace_id):
+                error = "invalid_span"
+                log.warning(
+                    "span_id and trace_id must both be specified for the given evaluation metric to be submitted."
+                )
+                return
+            if not label:
+                error = "invalid_metric_label"
+                log.warning("label must be the specified name of the evaluation metric.")
+                return
 
-        if not metric_type or metric_type.lower() not in ("categorical", "numerical", "score"):
-            log.warning("metric_type must be one of 'categorical' or 'score'.")
-            return
+            if not metric_type or metric_type.lower() not in ("categorical", "numerical", "score"):
+                error = "invalid_metric_type"
+                log.warning("metric_type must be one of 'categorical' or 'score'.")
+                return
 
-        metric_type = metric_type.lower()
-        if metric_type == "numerical":
-            log.warning(
-                "The evaluation metric type 'numerical' is unsupported. Use 'score' instead. "
-                "Converting `numerical` metric to `score` type."
-            )
-            metric_type = "score"
+            metric_type = metric_type.lower()
+            if metric_type == "numerical":
+                error = "invalid_metric_type"
+                log.warning(
+                    "The evaluation metric type 'numerical' is unsupported. Use 'score' instead. "
+                    "Converting `numerical` metric to `score` type."
+                )
+                metric_type = "score"
 
-        if metric_type == "categorical" and not isinstance(value, str):
-            log.warning("value must be a string for a categorical metric.")
-            return
-        if metric_type == "score" and not isinstance(value, (int, float)):
-            log.warning("value must be an integer or float for a score metric.")
-            return
-        if tags is not None and not isinstance(tags, dict):
-            log.warning("tags must be a dictionary of string key-value pairs.")
-            return
+            if metric_type == "categorical" and not isinstance(value, str):
+                error = "invalid_metric_value"
+                log.warning("value must be a string for a categorical metric.")
+                return
+            if metric_type == "score" and not isinstance(value, (int, float)):
+                error = "invalid_metric_value"
+                log.warning("value must be an integer or float for a score metric.")
+                return
+            if tags is not None and not isinstance(tags, dict):
+                error = "invalid_tags"
+                log.warning("tags must be a dictionary of string key-value pairs.")
+                return
 
-        # initialize tags with default values that will be overridden by user-provided tags
-        evaluation_tags = {
-            "ddtrace.version": ddtrace.__version__,
-            "ml_app": ml_app,
-        }
+            # initialize tags with default values that will be overridden by user-provided tags
+            evaluation_tags = {
+                "ddtrace.version": ddtrace.__version__,
+                "ml_app": ml_app,
+            }
 
-        if tags:
-            for k, v in tags.items():
-                try:
-                    evaluation_tags[ensure_text(k)] = ensure_text(v)
-                except TypeError:
-                    log.warning("Failed to parse tags. Tags for evaluation metrics must be strings.")
-
-        evaluation_metric = {
-            "join_on": {"span": {"span_id": span_id, "trace_id": trace_id}},
-            "label": str(label),
-            "metric_type": metric_type.lower(),
-            "timestamp_ms": timestamp_ms,
-            "{}_value".format(metric_type): value,
-            "ml_app": ml_app,
-            "tags": ["{}:{}".format(k, v) for k, v in evaluation_tags.items()],
-        }
+            if tags:
+                for k, v in tags.items():
+                    try:
+                        evaluation_tags[ensure_text(k)] = ensure_text(v)
+                    except TypeError:
+                        error = "invalid_tags"
+                        log.warning("Failed to parse tags. Tags for evaluation metrics must be strings.")
+
+            evaluation_metric = {
+                "join_on": {"span": {"span_id": span_id, "trace_id": trace_id}},
+                "label": str(label),
+                "metric_type": metric_type.lower(),
+                "timestamp_ms": timestamp_ms,
+                "{}_value".format(metric_type): value,
+                "ml_app": ml_app,
+                "tags": ["{}:{}".format(k, v) for k, v in evaluation_tags.items()],
+            }
 
-        if metadata:
-            if not isinstance(metadata, dict):
-                log.warning("metadata must be json serializable dictionary.")
-            else:
-                metadata = safe_json(metadata)
-                if metadata and isinstance(metadata, str):
-                    evaluation_metric["metadata"] = json.loads(metadata)
+            if metadata:
+                if not isinstance(metadata, dict):
+                    error = "invalid_metadata"
+                    log.warning("metadata must be json serializable dictionary.")
+                else:
+                    metadata = safe_json(metadata)
+                    if metadata and isinstance(metadata, str):
+                        evaluation_metric["metadata"] = json.loads(metadata)
 
-        cls._instance._llmobs_eval_metric_writer.enqueue(evaluation_metric)
+            cls._instance._llmobs_eval_metric_writer.enqueue(evaluation_metric)
+        finally:
+            telemetry.record_llmobs_submit_evaluation({"span": span_context}, metric_type, error)
 
     @classmethod
     def _inject_llmobs_context(cls, span_context: Context, request_headers: Dict[str, str]) -> None:
@@ -1276,38 +1340,46 @@ def inject_distributed_headers(cls, request_headers: Dict[str, str], span: Optio
                 "Distributed context will not be injected."
             )
             return request_headers
-        if not isinstance(request_headers, dict):
-            log.warning("request_headers must be a dictionary of string key-value pairs.")
-            return request_headers
-        if span is None:
-            span = cls._instance.tracer.current_span()
-        if span is None:
-            log.warning("No span provided and no currently active span found.")
-            return request_headers
-        if not isinstance(span, Span):
-            log.warning("span must be a valid Span object. Distributed context will not be injected.")
+        error = None
+        try:
+            if not isinstance(request_headers, dict):
+                error = "invalid_request_headers"
+                log.warning("request_headers must be a dictionary of string key-value pairs.")
+                return request_headers
+            if span is None:
+                span = cls._instance.tracer.current_span()
+            if span is None:
+                error = "no_active_span"
+                log.warning("No span provided and no currently active span found.")
+                return request_headers
+            if not isinstance(span, Span):
+                error = "invalid_span"
+                log.warning("span must be a valid Span object. Distributed context will not be injected.")
+                return request_headers
+            HTTPPropagator.inject(span.context, request_headers)
             return request_headers
-        HTTPPropagator.inject(span.context, request_headers)
-        return request_headers
+        finally:
+            telemetry.record_inject_distributed_headers(error)
 
     @classmethod
-    def _activate_llmobs_distributed_context(cls, request_headers: Dict[str, str], context: Context) -> None:
+    def _activate_llmobs_distributed_context(cls, request_headers: Dict[str, str], context: Context) -> Optional[str]:
         if cls.enabled is False:
-            return
+            return None
         if not context.trace_id or not context.span_id:
             log.warning("Failed to extract trace/span ID from request headers.")
-            return
+            return "missing_context"
         _parent_id = context._meta.get(PROPAGATED_PARENT_ID_KEY)
         if _parent_id is None:
             log.warning("Failed to extract LLMObs parent ID from request headers.")
-            return
+            return "missing_parent_id"
         try:
             parent_id = int(_parent_id)
         except ValueError:
             log.warning("Failed to parse LLMObs parent ID from request headers.")
-            return
+            return "invalid_parent_id"
         llmobs_context = Context(trace_id=context.trace_id, span_id=parent_id)
         cls._instance._llmobs_context_provider.activate(llmobs_context)
+        return None
 
     @classmethod
     def activate_distributed_headers(cls, request_headers: Dict[str, str]) -> None:
@@ -1324,7 +1396,8 @@ def activate_distributed_headers(cls, request_headers: Dict[str, str]) -> None:
             return
         context = HTTPPropagator.extract(request_headers)
         cls._instance.tracer.context_provider.activate(context)
-        cls._instance._activate_llmobs_distributed_context(request_headers, context)
+        error = cls._instance._activate_llmobs_distributed_context(request_headers, context)
+        telemetry.record_activate_distributed_headers(error)
 
 
 # initialize the default llmobs instance
diff --git a/ddtrace/llmobs/_telemetry.py b/ddtrace/llmobs/_telemetry.py
index cc25fda96a0..690fbb49184 100644
--- a/ddtrace/llmobs/_telemetry.py
+++ b/ddtrace/llmobs/_telemetry.py
@@ -1,4 +1,6 @@
 import time
+from typing import Any
+from typing import Dict
 from typing import Optional
 
 from ddtrace.internal.telemetry import telemetry_writer
@@ -20,6 +22,12 @@ class LLMObsTelemetryMetrics:
     SPAN_SIZE = "span.size"
     SPAN_STARTED = "span.start"
     SPAN_FINISHED = "span.finished"
+    ANNOTATIONS = "annotations"
+    EVALS_SUBMITTED = "evals_submitted"
+    SPANS_EXPORTED = "spans_exported"
+    USER_FLUSHES = "user_flushes"
+    INJECT_HEADERS = "inject_distributed_headers"
+    ACTIVATE_HEADERS = "activate_distributed_headers"
 
 
 def _find_integration_from_tags(tags):
@@ -42,15 +50,16 @@ def _get_tags_from_span_event(event: LLMObsSpanEvent):
     ]
 
 
-def record_llmobs_enabled(error: Optional[str], agentless_enabled: bool, site: str, start_ns: int, auto: bool):
-    tags = [
-        ("agentless", str(int(agentless_enabled))),
-        ("site", site),
-        ("auto", str(int(auto))),
-        ("error", "1" if error else "0"),
-    ]
+def _base_tags(error: Optional[str]):
+    tags = [("error", "1" if error else "0")]
     if error:
         tags.append(("error_type", error))
+    return tags
+
+
+def record_llmobs_enabled(error: Optional[str], agentless_enabled: bool, site: str, start_ns: int, auto: bool):
+    tags = _base_tags(error)
+    tags.extend([("agentless", str(int(agentless_enabled))), ("site", site), ("auto", str(int(auto)))])
     init_time_ms = (time.time_ns() - start_ns) / 1e6
     telemetry_writer.add_distribution_metric(
         namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.INIT_TIME, value=init_time_ms, tags=tuple(tags)
@@ -107,3 +116,60 @@ def record_span_event_size(event: LLMObsSpanEvent, event_size: int, truncated: b
     telemetry_writer.add_distribution_metric(
         namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.SPAN_SIZE, value=event_size, tags=tuple(tags)
     )
+
+
+def record_llmobs_annotate(span: Optional[Span], error: Optional[str]):
+    tags = _base_tags(error)
+    span_kind = "N/A"
+    is_root_span = "0"
+    if span and isinstance(span, Span):
+        span_kind = span._get_ctx_item(SPAN_KIND) or "N/A"
+        is_root_span = str(int(span._get_ctx_item(PARENT_ID_KEY) != ROOT_PARENT_ID))
+    tags.extend([("span_kind", span_kind), ("is_root_span", is_root_span)])
+    telemetry_writer.add_count_metric(
+        namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.ANNOTATIONS, value=1, tags=tuple(tags)
+    )
+
+
+def record_llmobs_submit_evaluation(join_on: Dict[str, Any], metric_type: str, error: Optional[str]):
+    _metric_type = metric_type if metric_type in ("categorical", "score") else "other"
+    custom_joining_key = str(int(join_on.get("tag") is not None))
+    tags = _base_tags(error)
+    tags.extend([("metric_type", _metric_type), ("custom_joining_key", custom_joining_key)])
+    telemetry_writer.add_count_metric(
+        namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.EVALS_SUBMITTED, value=1, tags=tuple(tags)
+    )
+
+
+def record_span_exported(span: Optional[Span], error: Optional[str]):
+    tags = _base_tags(error)
+    span_kind = "N/A"
+    is_root_span = "0"
+    if span and isinstance(span, Span):
+        span_kind = span._get_ctx_item(SPAN_KIND) or "N/A"
+        is_root_span = str(int(span._get_ctx_item(PARENT_ID_KEY) != ROOT_PARENT_ID))
+    tags.extend([("span_kind", span_kind), ("is_root_span", is_root_span)])
+    telemetry_writer.add_count_metric(
+        namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.SPANS_EXPORTED, value=1, tags=tuple(tags)
+    )
+
+
+def record_user_flush(error: Optional[str]):
+    tags = _base_tags(error)
+    telemetry_writer.add_count_metric(
+        namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.USER_FLUSHES, value=1, tags=tuple(tags)
+    )
+
+
+def record_inject_distributed_headers(error: Optional[str]):
+    tags = _base_tags(error)
+    telemetry_writer.add_count_metric(
+        namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.INJECT_HEADERS, value=1, tags=tuple(tags)
+    )
+
+
+def record_activate_distributed_headers(error: Optional[str]):
+    tags = _base_tags(error)
+    telemetry_writer.add_count_metric(
+        namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.ACTIVATE_HEADERS, value=1, tags=tuple(tags)
+    )


Reviews:
user: login: datadog-datadog-prod-us1[bot]
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T18:59:25Z
user: login: lievan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T17:09:52Z
user: login: Yun-Kim
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T17:57:46Z
user: login: lievan
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T18:17:20Z

Review comments:
diff_hunk: @@ -107,3 +116,55 @@ def record_span_event_size(event: LLMObsSpanEvent, event_size: int, truncated: b
     telemetry_writer.add_distribution_metric(
         namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.SPAN_SIZE, value=event_size, tags=tuple(tags)
     )
+
+
+def record_llmobs_annotate(span: Span, error: Optional[str]):
+    span_kind = span._get_ctx_item(SPAN_KIND)
+    is_root_span = span._get_ctx_item(PARENT_ID_KEY) != ROOT_PARENT_ID
+    tags = _base_tags(error)
+    tags.extend([("span_kind", span_kind or "N/A"), ("is_root_span", str(int(is_root_span)))])
+    telemetry_writer.add_count_metric(
+        namespace=TELEMETRY_NAMESPACE.MLOBS, name=LLMObsTelemetryMetrics.ANNOTATIONS, value=1, tags=tuple(tags)
+    )
+
+
+def record_llmobs_submit_evaluation(join_on: Dict[str, Any], metric_type: str, error: Optional[str]):
+    metric_type = metric_type if metric_type in ("categorical", "score") else "other"
path: ddtrace/llmobs/_telemetry.py
user: login: datadog-datadog-prod-us1[bot]
body: <!-- python-best-practices/function-variable-argument-name -->

<h3><span title="Warning"> :orange_circle: </span>Code Quality Violation</h3>



<details>
  <summary>
    variable name is the same as a function parameter <i>(...read more)</i>
  </summary>

<br>

A function parameter should only be read and not be modified. If your intent is to modify the value of the parameter, return the value in the function and handle the new value in the caller of the function.
</details>

<p align="right"><a href="https://app.datadoghq.com/ci/code-analysis/github.com%2Fdatadog%2Fdd-trace-py/yunkim%2Fllmobs-telemetry-api-usage/710fd6a6a8cc08b9a501b1972a1b2f5f6c81aec7/code-quality?index=cicodescan&query=%40git.repository.id%3A%22github.com%2Fdatadog%2Fdd-trace-py%22+%40git.branch%3A%22yunkim%2Fllmobs-telemetry-api-usage%22+%40git.commit.sha%3A%22710fd6a6a8cc08b9a501b1972a1b2f5f6c81aec7%22+-%40static_analysis.result.category%3Asecurity+-%40static_analysis.result.item_type%3Asecret&staticAnalysisId=AgAAAZWBa8GV-xQVVwAAAAAAAAAYAAAAAEFaV0JhN2RFQUFETmNDdGxFakR3QUFBQQAAACQAAAAAMDE5NTgxNmItYzI2Ni00M2JiLTlmZjItMzU1MzA5ZmVmN2Uw&trace=AgAAAZWBa8GV-xQVVwAAAAAAAAAYAAAAAEFaV0JhN2RFQUFETmNDdGxFakR3QUFBQQAAACQAAAAAMDE5NTgxNmItYzI2Ni00M2JiLTlmZjItMzU1MzA5ZmVmN2Uw" title="View in Datadog"><img src="https://app.datadoghq.com/api/ui/asset/images/pr_comments/software_delivery/datadog_icon.svg" alt="View in Datadog" align="center" /></a>&nbsp;&nbsp;<a href="https://dd.enterprise.slack.com/archives/C03NGS0N3GA" title="Leave us feedback"><picture><source media="(prefers-color-scheme: dark)" srcset="https://app.datadoghq.com/api/ui/asset/images/pr_comments/software_delivery/comment_dark.svg"/><img src="https://app.datadoghq.com/api/ui/asset/images/pr_comments/software_delivery/comment_light.svg" alt="Leave us feedback" align="center" /></picture></a>&nbsp;&nbsp;<a href="https://docs.datadoghq.com/static_analysis/rules/python-best-practices/function-variable-argument-name" title="Documentation"><picture><source media="(prefers-color-scheme: dark)" srcset="https://app.datadoghq.com/api/ui/asset/images/pr_comments/software_delivery/books_dark.svg"/><img src="https://app.datadoghq.com/api/ui/asset/images/pr_comments/software_delivery/books_light.svg" alt="Documentation" align="center" /></picture></a></p>

created_at: 2025-03-10T18:59:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -875,56 +886,68 @@ def annotate(
         :param metrics: Dictionary of JSON serializable key-value metric pairs,
                         such as `{prompt,completion,total}_tokens`.
         """
-        if span is None:
-            span = cls._instance._current_span()
+        error = None
+        try:
             if span is None:
-                log.warning("No span provided and no active LLMObs-generated span found.")
+                span = cls._instance._current_span()
+                if span is None:
+                    error = "invalid_span_no_active_spans"
+                    log.warning("No span provided and no active LLMObs-generated span found.")
+                    return
+            if span.span_type != SpanTypes.LLM:
+                error = "invalid_span_type"
+                log.warning("Span must be an LLMObs-generated span.")
                 return
-        if span.span_type != SpanTypes.LLM:
-            log.warning("Span must be an LLMObs-generated span.")
-            return
-        if span.finished:
-            log.warning("Cannot annotate a finished span.")
-            return
-        if metadata is not None:
-            if not isinstance(metadata, dict):
-                log.warning("metadata must be a dictionary")
-            else:
-                cls._set_dict_attribute(span, METADATA, metadata)
-        if metrics is not None:
-            if not isinstance(metrics, dict):
-                log.warning("metrics must be a dictionary of string key - numeric value pairs.")
-            else:
-                cls._set_dict_attribute(span, METRICS, metrics)
-        if tags is not None:
-            if not isinstance(tags, dict):
-                log.warning("span tags must be a dictionary of string key - primitive value pairs.")
-            else:
-                cls._set_dict_attribute(span, TAGS, tags)
-        span_kind = span._get_ctx_item(SPAN_KIND)
-        if _name is not None:
-            span.name = _name
-        if prompt is not None:
-            try:
-                validated_prompt = validate_prompt(prompt)
-                cls._set_dict_attribute(span, INPUT_PROMPT, validated_prompt)
-            except TypeError:
-                log.warning("Failed to validate prompt with error: ", exc_info=True)
-        if not span_kind:
-            log.debug("Span kind not specified, skipping annotation for input/output data")
-            return
-        if input_data is not None or output_data is not None:
-            if span_kind == "llm":
-                cls._tag_llm_io(span, input_messages=input_data, output_messages=output_data)
-            elif span_kind == "embedding":
-                cls._tag_embedding_io(span, input_documents=input_data, output_text=output_data)
-            elif span_kind == "retrieval":
-                cls._tag_retrieval_io(span, input_text=input_data, output_documents=output_data)
-            else:
-                cls._tag_text_io(span, input_value=input_data, output_value=output_data)
+            if span.finished:
+                error = "invalid_finished_span"
+                log.warning("Cannot annotate a finished span.")
+                return
+            if metadata is not None:
+                if not isinstance(metadata, dict):
+                    error = "invalid_metadata"
+                    log.warning("metadata must be a dictionary")
+                else:
+                    cls._set_dict_attribute(span, METADATA, metadata)
+            if metrics is not None:
+                if not isinstance(metrics, dict):
+                    error = "invalid_metrics"
+                    log.warning("metrics must be a dictionary of string key - numeric value pairs.")
+                else:
+                    cls._set_dict_attribute(span, METRICS, metrics)
+            if tags is not None:
+                if not isinstance(tags, dict):
+                    error = "invalid_tags"
+                    log.warning("span tags must be a dictionary of string key - primitive value pairs.")
+                else:
+                    cls._set_dict_attribute(span, TAGS, tags)
+            span_kind = span._get_ctx_item(SPAN_KIND)
+            if _name is not None:
+                span.name = _name
+            if prompt is not None:
+                try:
+                    validated_prompt = validate_prompt(prompt)
+                    cls._set_dict_attribute(span, INPUT_PROMPT, validated_prompt)
+                except TypeError:
+                    error = "invalid_prompt"
+                    log.warning("Failed to validate prompt with error: ", exc_info=True)
+            if not span_kind:
path: ddtrace/llmobs/_llmobs.py
user: login: lievan
body: span kind would be missing at annotation time for anyone using `annotation_context` on autoinstrumented spans. In this case, its not necessarily an error. we can either:
- pass in a flag to `annotate` to its being used by an annotation context
- refactor so annotation context no longer calls annotate; instead it only relies on its own logic to set span attributes
im leaning toward latter 
created_at: 2025-03-11T17:04:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1040,99 +1070,115 @@ def submit_evaluation_for(
             )
             return
 
-        has_exactly_one_joining_key = (span is not None) ^ (span_with_tag_value is not None)
-
-        if not has_exactly_one_joining_key:
-            raise ValueError(
-                "Exactly one of `span` or `span_with_tag_value` must be specified to submit an evaluation metric."
-            )
-
+        error = None
         join_on = {}
-        if span is not None:
-            if (
-                not isinstance(span, dict)
-                or not isinstance(span.get("span_id"), str)
-                or not isinstance(span.get("trace_id"), str)
-            ):
-                raise TypeError(
-                    "`span` must be a dictionary containing both span_id and trace_id keys. "
-                    "LLMObs.export_span() can be used to generate this dictionary from a given span."
-                )
-            join_on["span"] = span
-        elif span_with_tag_value is not None:
-            if (
-                not isinstance(span_with_tag_value, dict)
-                or not isinstance(span_with_tag_value.get("tag_key"), str)
-                or not isinstance(span_with_tag_value.get("tag_value"), str)
-            ):
-                raise TypeError(
-                    "`span_with_tag_value` must be a dict with keys 'tag_key' and 'tag_value' containing string values"
+        try:
+            has_exactly_one_joining_key = (span is not None) ^ (span_with_tag_value is not None)
+
+            if not has_exactly_one_joining_key:
+                error = "provided_both_span_and_joining_key"
path: ddtrace/llmobs/_llmobs.py
user: login: lievan
body: ```suggestion
                error = "provided_both_span_and_tag_joining_key"
```
nit, since span/trace and tag are both joining keys
created_at: 2025-03-11T17:05:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -875,56 +886,68 @@ def annotate(
         :param metrics: Dictionary of JSON serializable key-value metric pairs,
                         such as `{prompt,completion,total}_tokens`.
         """
-        if span is None:
-            span = cls._instance._current_span()
+        error = None
+        try:
             if span is None:
-                log.warning("No span provided and no active LLMObs-generated span found.")
+                span = cls._instance._current_span()
+                if span is None:
+                    error = "invalid_span_no_active_spans"
+                    log.warning("No span provided and no active LLMObs-generated span found.")
+                    return
+            if span.span_type != SpanTypes.LLM:
+                error = "invalid_span_type"
+                log.warning("Span must be an LLMObs-generated span.")
                 return
-        if span.span_type != SpanTypes.LLM:
-            log.warning("Span must be an LLMObs-generated span.")
-            return
-        if span.finished:
-            log.warning("Cannot annotate a finished span.")
-            return
-        if metadata is not None:
-            if not isinstance(metadata, dict):
-                log.warning("metadata must be a dictionary")
-            else:
-                cls._set_dict_attribute(span, METADATA, metadata)
-        if metrics is not None:
-            if not isinstance(metrics, dict):
-                log.warning("metrics must be a dictionary of string key - numeric value pairs.")
-            else:
-                cls._set_dict_attribute(span, METRICS, metrics)
-        if tags is not None:
-            if not isinstance(tags, dict):
-                log.warning("span tags must be a dictionary of string key - primitive value pairs.")
-            else:
-                cls._set_dict_attribute(span, TAGS, tags)
-        span_kind = span._get_ctx_item(SPAN_KIND)
-        if _name is not None:
-            span.name = _name
-        if prompt is not None:
-            try:
-                validated_prompt = validate_prompt(prompt)
-                cls._set_dict_attribute(span, INPUT_PROMPT, validated_prompt)
-            except TypeError:
-                log.warning("Failed to validate prompt with error: ", exc_info=True)
-        if not span_kind:
-            log.debug("Span kind not specified, skipping annotation for input/output data")
-            return
-        if input_data is not None or output_data is not None:
-            if span_kind == "llm":
-                cls._tag_llm_io(span, input_messages=input_data, output_messages=output_data)
-            elif span_kind == "embedding":
-                cls._tag_embedding_io(span, input_documents=input_data, output_text=output_data)
-            elif span_kind == "retrieval":
-                cls._tag_retrieval_io(span, input_text=input_data, output_documents=output_data)
-            else:
-                cls._tag_text_io(span, input_value=input_data, output_value=output_data)
+            if span.finished:
+                error = "invalid_finished_span"
+                log.warning("Cannot annotate a finished span.")
+                return
+            if metadata is not None:
+                if not isinstance(metadata, dict):
+                    error = "invalid_metadata"
+                    log.warning("metadata must be a dictionary")
+                else:
+                    cls._set_dict_attribute(span, METADATA, metadata)
+            if metrics is not None:
+                if not isinstance(metrics, dict):
+                    error = "invalid_metrics"
+                    log.warning("metrics must be a dictionary of string key - numeric value pairs.")
+                else:
+                    cls._set_dict_attribute(span, METRICS, metrics)
+            if tags is not None:
+                if not isinstance(tags, dict):
+                    error = "invalid_tags"
+                    log.warning("span tags must be a dictionary of string key - primitive value pairs.")
+                else:
+                    cls._set_dict_attribute(span, TAGS, tags)
+            span_kind = span._get_ctx_item(SPAN_KIND)
+            if _name is not None:
+                span.name = _name
+            if prompt is not None:
+                try:
+                    validated_prompt = validate_prompt(prompt)
+                    cls._set_dict_attribute(span, INPUT_PROMPT, validated_prompt)
+                except TypeError:
+                    error = "invalid_prompt"
+                    log.warning("Failed to validate prompt with error: ", exc_info=True)
+            if not span_kind:
path: ddtrace/llmobs/_llmobs.py
user: login: Yun-Kim
body: If this isn't necessarily an error (manual spans will always have a span kind) then I'll just get rid of this error tag.
created_at: 2025-03-11T17:57:46Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-03-10T18:59:25Z
body: `CODEOWNERS` have been resolved as:

```
ddtrace/llmobs/_llmobs.py                                               @DataDog/ml-observability
ddtrace/llmobs/_telemetry.py                                            @DataDog/ml-observability
```
<!-- thollander/actions-comment-pull-request "codeowners_resolved" -->
author_association: CONTRIBUTOR
user: login: pr-commenter[bot]
created_at: 2025-03-10T19:51:24Z
body: ## Benchmarks

Benchmark execution time: 2025-03-11 18:47:41

Comparing candidate commit 71d012356f4a4574cfffdb4660b478fab4ce2e8b in PR branch `yunkim/llmobs-telemetry-api-usage` with baseline commit b47b06140dfcfc28a4794086a23bf4288202fb1b in branch `main`.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 468 metrics, 2 unstable metrics.


author_association: NONE