Repository: mlflow/mlflow
Pull request: Reuse a single empty OTEL Resource in Span.from_dict to cut search_tr…
<details><summary>&#x1F6E0 DevTools &#x1F6E0</summary>
<p>

[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/joelrobin18/mlflow/pull/15637?quickstart=1)

#### Install mlflow from this PR

```
# mlflow
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/15637/merge
# mlflow-skinny
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/15637/merge#subdirectory=skinny
```

For Databricks, use the following command:

```
%sh curl -LsSf https://raw.githubusercontent.com/mlflow/mlflow/HEAD/dev/install-skinny.sh | sh -s pull/15637/merge
```

</p>
</details>

…aces deserialization time by more than 70%

### Related Issues/PRs

Fix #15625 

### What changes are proposed in this pull request?

Every call to Span.from_dict() built a fresh opentelemetry.sdk.resources.Resource, adding significant time for each span when searching traces. 

In this fix, we introduce a module-level `_EMPTY_RESOURCE` singleton and let Span.from_dict(..., _reuse_resource=True) pass that object into ReadableSpan. We avoid rebuilding the same empty Resource object for every span by creating it just once at the module level. Then, when converting span data back into Span instances during a bulk download, we reuse that single Resource instead of making a new one each time, cutting out the repeated cost per span.

After introducing this fix, we can see a decrease in time by more than 70%.

Before fix:
<img width="928" alt="Screenshot 2025-05-09 at 6 34 53 AM" src="https://github.com/user-attachments/assets/11f416b6-d9be-4428-8aba-60baf33fd5a6" />


After fix:
<img width="913" alt="Screenshot 2025-05-09 at 6 37 27 AM" src="https://github.com/user-attachments/assets/36a7b703-37b0-4f50-8b63-ad6b5296d053" />

After the fix, we can see that there is a significant decrease in the time(from 70sec to 10 sec -> More than 70% decrease in the time) for searching traces.

### How is this PR tested?

- [x] Existing unit/integration tests
- [ ] New unit/integration tests
- [x] Manual tests


### Does this PR require documentation update?

- [x] No. You can skip the rest of this section.
- [ ] Yes. I've updated:
  - [ ] Examples
  - [ ] API references
  - [ ] Instructions

### Release Notes

#### Is this a user-facing change?

- [x] No. You can skip the rest of this section.
- [ ] Yes. Give a description of this change to be included in the release notes for MLflow users.


#### What component(s), interfaces, languages, and integrations does this PR affect?

Components

- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/deployments`: MLflow Deployments client APIs, server, and third-party Deployments integrations
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [x] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interface

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

Language

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

Integrations

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations

#### How should the PR be classified in the release notes? Choose one:

- [x] `rn/none` - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
- [ ] `rn/breaking-change` - The PR will be mentioned in the "Breaking Changes" section
- [ ] `rn/feature` - A new user-facing feature worth mentioning in the release notes
- [ ] `rn/bug-fix` - A user-facing bug fix worth mentioning in the release notes
- [ ] `rn/documentation` - A user-facing documentation change worth mentioning in the release notes

#### Should this PR be included in the next patch release?

`Yes` should be selected for bug fixes, documentation updates, and other small changes. `No` should be selected for new features and larger changes. If you're unsure about the release classification of this PR, leave this unchecked to let the maintainers decide.

<details>
<summary>What is a minor/patch release?</summary>

- Minor release: a release that increments the second part of the version number (e.g., 1.2.0 -> 1.3.0).
  Bug fixes, doc updates and new features usually go into minor releases.
- Patch release: a release that increments the third part of the version number (e.g., 1.2.0 -> 1.2.1).
  Bug fixes and doc updates usually go into patch releases.

</details>

<!-- patch -->

- [x] Yes (this PR will be cherry-picked and included in the next patch release)
- [ ] No (this PR will be included in the next minor release)


Problem statement:
[FR] Improve Download Trace Data Speed by Avoiding Resource Creation in span.from_dict()
### Willingness to contribute

No. I cannot contribute this feature at this time.

### Proposal Summary

The process of downloading trace data in mlflow.search_traces() involves creating a Resource object within the span.from_dict() method, which can be time-consuming and inefficient.

This feature request proposes optimizing the download trace data functionality by bypassing unnecessary resource creation, thereby improving overall performance. 

The expected result is a faster and more efficient trace data downloading process, enhancing system responsiveness and user experience.

### Motivation

> #### What is the use case for this feature?
Users who frequently download large volumes of trace data will benefit from reduced wait times and improved responsiveness.
> #### Why is this use case valuable to support for MLflow users in general?
For MLflow users, optimizing the download trace data process is valuable because it enhances the efficiency of tracking and managing machine learning experiments.
> #### Why is this use case valuable to support for your project(s) or organization?
This use case is valuable for our organization because we regularly need to search and analyze large volumes of trace data. 
> #### Why is it currently difficult to achieve this use case?
Achieving this use case is difficult because the existing process of  mlflow.search_traces()  involves creating unnecessary Resource objects within the span.from_dict() method, which slows down the download of trace data. 

### Details

Avoid creating unnecessary resource objects within span.from_dict() during downloading trace data.

### What component(s) does this bug affect?

- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/deployments`: MLflow Deployments client APIs, server, and third-party Deployments integrations
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [x] `area/tracking`: Tracking Service, tracking client APIs, autologging

### What interface(s) does this bug affect?

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

### What language(s) does this bug affect?

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations

Hints from issue discussion:
Downloading trace data takes over 7 seconds, with most of the time spent on creating resources within span.from_dict().
![Image](https://github.com/user-attachments/assets/aa90cd92-ee6c-4966-b474-10b1534122ef)

Commit message: Fix

Signed-off-by: joelrobin18 <joelrobin1818@gmail.com>

Patch to review:
diff --git a/mlflow/entities/span.py b/mlflow/entities/span.py
index c3d33c8369ad0..dc3bba30862b6 100644
--- a/mlflow/entities/span.py
+++ b/mlflow/entities/span.py
@@ -6,6 +6,7 @@
 
 from google.protobuf.json_format import MessageToDict, ParseDict
 from google.protobuf.struct_pb2 import Value
+from opentelemetry.sdk.resources import Resource as _OTelResource
 from opentelemetry.sdk.trace import Event as OTelEvent
 from opentelemetry.sdk.trace import ReadableSpan as OTelReadableSpan
 from opentelemetry.trace import NonRecordingSpan, SpanContext, TraceFlags
@@ -222,7 +223,10 @@ def to_dict(self) -> dict[str, Any]:
         return d
 
     @classmethod
-    def from_dict(cls, data: dict[str, Any]) -> "Span":
+    def from_dict(
+        cls,
+        data: dict[str, Any],
+    ) -> "Span":
         """Create a Span object from the given dictionary."""
         try:
             # Try to deserialize the span using the v3 schema
@@ -257,6 +261,7 @@ def from_dict(cls, data: dict[str, Any]) -> "Span":
                     status_code=SpanStatusCode.from_proto_status_code(data["status"]["code"]),
                     description=data["status"].get("message"),
                 ).to_otel_status(),
+                resource=_OTelResource.get_empty(),
                 events=[
                     OTelEvent(
                         name=event["name"],


Code review comments:
diff_hunk: @@ -257,6 +261,7 @@ def from_dict(cls, data: dict[str, Any]) -> "Span":
                     status_code=SpanStatusCode.from_proto_status_code(data["status"]["code"]),
                     description=data["status"].get("message"),
                 ).to_otel_status(),
path: mlflow/entities/span.py
text: @user1:
```suggestion
                ).to_otel_status(),
                # Setting an empty resource explicitly. Otherwise OTel create a new Resource by
                # Resource.create(), which introduces a significant overhead in some environments.
                # https://github.com/mlflow/mlflow/issues/15625
```