Repository: comet-ml/opik
Pull request: [OPIK-1511]: [SDK] Long running jobs traces start and end batch creation with feature flag
## Details
The Python SDK needs to support the ingestion of long running jobs, for instance taking 500 seconds. 
For that: 

1. It should create traces when they start by taking benefit of the relaxation of the required input fields. 
Then, it should flush the complete details of the traces at finalization. 
2. For these two operations, It will use the batch `createTraces` endpoint as an upsert endpoint, by providing a proper timestamp in the `last_updated_at` field for all traces on each of those calls. 


## Issues

Resolves #1610 

## Testing
Updated all related tests.


Problem statement:
[FR]: Flush partially completed traces - flush as they come
### Proposal summary

It would be great if traces could flush as they arrive for partial elements, instead of awaiting the top level function/trace to finish. 
If the code raises an exception I don't get any data, which sucks for a case with 500s execution.

Additionally, I don't want to wait 500s to get any data to look at, and want to be able to abort runs early from monitoring this data.

`OPIK_DEFAULT_FLUSH_TIMEOUT` did not have an effect

### Motivation

- **What problem are you trying to solve?**: loss of failed run data & convenience. Don't want to wait for a the top level function/trace
- **How are you currently solving this problem?**: monitoring our internal log system
- **What are the benefits of this feature?**: Better understanding of functionality even in failed and aborted runs

Hints from issue discussion:
Thanks for reporting @gustavhartz. We will look into the best way to support this.
Can you share a bit about your use case? Are these spans coming from a production workflow or this is in offline experimentation? 
For a long running trace with a lot of sub traces and LLM calls, we cannot see the trace showing in the dashboard until the entire job is completed. 

Ideally, we would want to see the trace & data being added as it is logged into the server.
There are two cases
- Avoid loss of logs in case of errors
- Faster monitoring of results 


Case one is only really intended for dev/test env as we don't plan on bugs in prod and have opik disabled on prod

Case two is for long running jobs, it's inefficient to wait 500s plus for the logs to come in. Also in case we want to abort runs

I'm looking into implementing it to an existing large code base, thus we are a little limited on implementation options 

Let me know if you need anything g else 
Thanks @gustavhartz , ill take this with our team and see how we can better improve this use case
Hi @gustavhartz, Thanks again for the suggestion!

We've synced internally and identified a technical solution that will allow trace and span visibility upon submission, without needing to wait for task completion.

Since this requires several changes across our data ingestion workflow (both on the SDK and backend), we're planning to start work on it soon. Just to set expectations, this will take some time to complete (likely over the next few weeks)

We'll keep you posted as we make progress
sounds good 👍
Hi @gustavhartz 

Thank you for raising this issue! I'm Andrés, a Principal Engineer at Comet, and I wanted to let you know that I’ll be taking this forward. My initial focus will be on creating a proof of concept to address the problem you've outlined.

I'll share updates here as I make progress. Please feel free to provide any additional context that you think would be helpful.

Looking forward to working on this!

Best,
Andrés
Sounds good @andrescrz! I think all is captured here
Hi @gustavhartz,

The first change at the trace level has been successfully implemented in the backend. The next step will be to propagate these changes to the spans.

Once that is completed, we’ll proceed to schedule the SDK changes.

Best regards,
Andrés
Hi @gustavhartz,

Thank you for your patience. I wanted to let you know that support for this feature has been fully implemented in the backend. We are planning to schedule the SDK implementation for the next sprint.

We will keep you updated on our progress.

Best regards,
Andrés

Commit message: Merge branch 'main' into OPIK-1511-long-running-traces

Patch to review:
diff --git a/sdks/python/src/opik/api_objects/opik_client.py b/sdks/python/src/opik/api_objects/opik_client.py
index 2e3ca362fd..ae421040c4 100644
--- a/sdks/python/src/opik/api_objects/opik_client.py
+++ b/sdks/python/src/opik/api_objects/opik_client.py
@@ -208,8 +208,9 @@ def trace(
         error_info: Optional[ErrorInfoDict] = None,
         thread_id: Optional[str] = None,
         attachments: Optional[List[Attachment]] = None,
+        _trace_start: bool = False,
         **ignored_kwargs: Any,
-    ) -> trace.Trace:
+    ) -> Optional[trace.Trace]:
         """
         Create and log a new trace.
 
@@ -229,14 +230,20 @@ def trace(
             thread_id: Used to group multiple traces into a thread.
                 The identifier is user-defined and has to be unique per project.
             attachments: The list of attachments to be uploaded to the trace.
+            _trace_start: Internal use only.
 
         Returns:
             trace.Trace: The created trace object.
         """
+        if _trace_start is True and self._config.log_start_trace is False:
+            # Do not log start traces if not configured
+            return None
+
         id = id if id is not None else id_helpers.generate_id()
         start_time = (
             start_time if start_time is not None else datetime_helpers.local_timestamp()
         )
+        last_updated_at = datetime_helpers.local_timestamp()
 
         if project_name is None:
             project_name = self._project_name
@@ -253,6 +260,7 @@ def trace(
             tags=tags,
             error_info=error_info,
             thread_id=thread_id,
+            last_updated_at=last_updated_at,
         )
         self._streamer.put(create_trace_message)
         self._display_trace_url(trace_id=id, project_name=project_name)
@@ -408,6 +416,7 @@ def span(
         start_time = (
             start_time if start_time is not None else datetime_helpers.local_timestamp()
         )
+        last_updated_at = datetime_helpers.local_timestamp()
 
         backend_compatible_usage = validation_helpers.validate_and_parse_usage(
             usage=usage,
@@ -437,6 +446,7 @@ def span(
                 tags=tags,
                 error_info=error_info,
                 thread_id=None,
+                last_updated_at=last_updated_at,
             )
             self._streamer.put(create_trace_message)
 
diff --git a/sdks/python/src/opik/api_objects/trace/trace_data.py b/sdks/python/src/opik/api_objects/trace/trace_data.py
index 2b03969320..20cfcfbcbf 100644
--- a/sdks/python/src/opik/api_objects/trace/trace_data.py
+++ b/sdks/python/src/opik/api_objects/trace/trace_data.py
@@ -101,3 +101,31 @@ def _update_attachments(self, attachments: List[Attachment]) -> None:
             self.attachments = attachments
         else:
             self.attachments.extend(attachments)
+
+    @property
+    def trace_start_parameters(self) -> Dict[str, Any]:
+        return {
+            "id": self.id,
+            "start_time": self.start_time,
+            "project_name": self.project_name,
+            "_trace_start": True,
+        }
+
+    @property
+    def trace_end_parameters(self) -> Dict[str, Any]:
+        return {
+            "id": self.id,
+            "name": self.name,
+            "start_time": self.start_time,
+            "end_time": self.end_time,
+            "metadata": self.metadata,
+            "input": self.input,
+            "output": self.output,
+            "tags": self.tags,
+            "feedback_scores": self.feedback_scores,
+            "project_name": self.project_name,
+            "error_info": self.error_info,
+            "thread_id": self.thread_id,
+            "attachments": self.attachments,
+            "_trace_start": False,
+        }
diff --git a/sdks/python/src/opik/config.py b/sdks/python/src/opik/config.py
index cb32f47c85..eadd1e84ff 100644
--- a/sdks/python/src/opik/config.py
+++ b/sdks/python/src/opik/config.py
@@ -210,6 +210,12 @@ def settings_customise_sources(
     Defines the factor applied to the `maximal_queue_size` to reduce the maximal message queue size when batching is enabled.
     """
 
+    log_start_trace: bool = False
+    """
+    If set to True, both the start and end of the trace will be logged. This is useful for traces that span long durations.
+    For shorter traces, it is recommended to keep this setting disabled to minimize data logging overhead.
+    """
+
     @property
     def config_file_fullpath(self) -> pathlib.Path:
         config_file_path = os.getenv("OPIK_CONFIG_PATH", CONFIG_FILE_PATH_DEFAULT)
diff --git a/sdks/python/src/opik/decorator/base_track_decorator.py b/sdks/python/src/opik/decorator/base_track_decorator.py
index a570f85b5e..b742e97ada 100644
--- a/sdks/python/src/opik/decorator/base_track_decorator.py
+++ b/sdks/python/src/opik/decorator/base_track_decorator.py
@@ -418,6 +418,9 @@ def _before_call(
             context_storage.set_trace_data(created_trace_data)
             TRACES_CREATED_BY_DECORATOR.add(created_trace_data.id)
 
+            client = opik_client.get_client_cached()
+            client.trace(**created_trace_data.trace_start_parameters)
+
         context_storage.add_span_data(created_span_data)
 
     def _after_call(
@@ -479,7 +482,7 @@ def _after_call(
                     ),
                 )
 
-                client.trace(**trace_data_to_end.__dict__)
+                client.trace(**trace_data_to_end.trace_end_parameters)
 
             if flush:
                 client.flush()
diff --git a/sdks/python/src/opik/evaluation/engine/helpers.py b/sdks/python/src/opik/evaluation/engine/helpers.py
index 43d6e8926c..c6fec27a86 100644
--- a/sdks/python/src/opik/evaluation/engine/helpers.py
+++ b/sdks/python/src/opik/evaluation/engine/helpers.py
@@ -16,8 +16,8 @@ def evaluate_llm_task_context(
     trace_data: trace.TraceData,
     client: opik_client.Opik,
 ) -> Iterator[None]:
+    error_info: Optional[ErrorInfoDict] = None
     try:
-        error_info: Optional[ErrorInfoDict] = None
         context_storage.set_trace_data(trace_data)
         yield
     except Exception as exception:
@@ -34,7 +34,7 @@ def evaluate_llm_task_context(
         trace_data.init_end_time()
 
         client = client if client is not None else opik_client.get_client_cached()
-        client.trace(**trace_data.__dict__)
+        client.trace(**trace_data.trace_end_parameters)
 
         experiment_item_ = experiment_item.ExperimentItemReferences(
             dataset_item_id=dataset_item_id,
diff --git a/sdks/python/src/opik/integrations/adk/opik_tracer.py b/sdks/python/src/opik/integrations/adk/opik_tracer.py
index 79923452a8..f38de84e02 100644
--- a/sdks/python/src/opik/integrations/adk/opik_tracer.py
+++ b/sdks/python/src/opik/integrations/adk/opik_tracer.py
@@ -128,10 +128,12 @@ def _start_trace(
     ) -> None:
         self._set_current_context_data(new_trace_data)
 
+        self._opik_client.trace(**new_trace_data.trace_start_parameters)
+
     def _end_current_trace(self) -> None:
         if (trace_data := self._context_storage.get_trace_data()) is not None:
             trace_data.init_end_time()
-            self._opik_client.trace(**trace_data.__dict__)
+            self._opik_client.trace(**trace_data.trace_end_parameters)
 
             self._context_storage.set_trace_data(None)
 
diff --git a/sdks/python/src/opik/integrations/dspy/callback.py b/sdks/python/src/opik/integrations/dspy/callback.py
index 7d2bebc2c6..725cecd59c 100644
--- a/sdks/python/src/opik/integrations/dspy/callback.py
+++ b/sdks/python/src/opik/integrations/dspy/callback.py
@@ -151,6 +151,8 @@ def _start_trace(
         self._map_call_id_to_trace_data[call_id] = trace_data
         self._set_current_context_data(trace_data)
 
+        self._opik_client.trace(**trace_data.trace_start_parameters)
+
     def on_module_end(
         self,
         call_id: str,
@@ -167,7 +169,7 @@ def on_module_end(
     def _end_trace(self, call_id: str) -> None:
         if trace_data := self._map_call_id_to_trace_data.pop(call_id, None):
             trace_data.init_end_time()
-            self._opik_client.trace(**trace_data.__dict__)
+            self._opik_client.trace(**trace_data.trace_end_parameters)
 
             if self._context_storage.get_trace_data() == trace_data:
                 self._context_storage.set_trace_data(None)
diff --git a/sdks/python/src/opik/integrations/haystack/opik_tracer.py b/sdks/python/src/opik/integrations/haystack/opik_tracer.py
index 0cca1906c2..6c711972ba 100644
--- a/sdks/python/src/opik/integrations/haystack/opik_tracer.py
+++ b/sdks/python/src/opik/integrations/haystack/opik_tracer.py
@@ -179,6 +179,7 @@ def trace(
                 name=self._name,
                 tags=context.get("tags"),
             )
+            assert trace is not None
             span = OpikSpanBridge(trace)
         elif tags.get(_COMPONENT_TYPE_KEY) in _ALL_SUPPORTED_GENERATORS:
             span = OpikSpanBridge(
diff --git a/sdks/python/src/opik/integrations/langchain/opik_tracer.py b/sdks/python/src/opik/integrations/langchain/opik_tracer.py
index 4c0ba23c38..fb0f374a17 100644
--- a/sdks/python/src/opik/integrations/langchain/opik_tracer.py
+++ b/sdks/python/src/opik/integrations/langchain/opik_tracer.py
@@ -139,7 +139,9 @@ def _persist_run(self, run: "Run") -> None:
                 trace_data.input = run_dict["inputs"]
 
             trace_data.init_end_time().update(output=output, error_info=error_info)
-            trace_ = self._opik_client.trace(**trace_data.__dict__)
+            trace_ = self._opik_client.trace(**trace_data.trace_end_parameters)
+
+            assert trace_ is not None
             self._created_traces.append(trace_)
             self._opik_context_storage.pop_trace_data(ensure_id=trace_data.id)
 
diff --git a/sdks/python/src/opik/integrations/llama_index/callback.py b/sdks/python/src/opik/integrations/llama_index/callback.py
index 41b8909575..2565a288b3 100644
--- a/sdks/python/src/opik/integrations/llama_index/callback.py
+++ b/sdks/python/src/opik/integrations/llama_index/callback.py
@@ -74,6 +74,8 @@ def start_trace(self, trace_id: Optional[str] = None) -> None:
         else:
             self._opik_trace_data = self._create_trace_data(trace_name=trace_id)
 
+        self._opik_client.trace(**self._opik_trace_data.trace_start_parameters)
+
     def _get_last_event(self, trace_map: Dict[str, List[str]]) -> str:
         def dfs(key: str) -> str:
             if key not in trace_map or not trace_map[key]:
@@ -101,7 +103,7 @@ def end_trace(
             or self._opik_trace_data.name != "index_construction"
         ):
             self._opik_trace_data.init_end_time().update(output=last_event_output)
-            self._opik_client.trace(**self._opik_trace_data.__dict__)
+            self._opik_client.trace(**self._opik_trace_data.trace_end_parameters)
             self._opik_trace_data = None
 
         # Do not clean _map_event_id_to_span_data as streaming LLM events can
diff --git a/sdks/python/src/opik/integrations/openai/agents/opik_tracing_processor.py b/sdks/python/src/opik/integrations/openai/agents/opik_tracing_processor.py
index 57c95b1d75..78f5fd5872 100644
--- a/sdks/python/src/opik/integrations/openai/agents/opik_tracing_processor.py
+++ b/sdks/python/src/opik/integrations/openai/agents/opik_tracing_processor.py
@@ -44,6 +44,7 @@ def on_trace_start(self, trace: tracing.Trace) -> None:
                 metadata=trace_metadata,
             )
             self._created_traces_data_map[trace.trace_id] = opik_trace_data
+            self._opik_client.trace(**opik_trace_data.trace_start_parameters)
         except Exception:
             LOGGER.debug("on_trace_start failed", exc_info=True)
 
@@ -53,7 +54,7 @@ def on_trace_end(self, trace: tracing.Trace) -> None:
             opik_trace_data.init_end_time()
 
             self._collect_trace_input_and_output_from_spans(opik_trace_data)
-            self._opik_client.trace(**opik_trace_data.__dict__)
+            self._opik_client.trace(**opik_trace_data.trace_end_parameters)
         except Exception:
             LOGGER.debug("on_trace_end failed", exc_info=True)
 
diff --git a/sdks/python/src/opik/message_processing/messages.py b/sdks/python/src/opik/message_processing/messages.py
index 5531742585..d81541320c 100644
--- a/sdks/python/src/opik/message_processing/messages.py
+++ b/sdks/python/src/opik/message_processing/messages.py
@@ -34,6 +34,7 @@ class CreateTraceMessage(BaseMessage):
     tags: Optional[List[str]]
     error_info: Optional[ErrorInfoDict]
     thread_id: Optional[str]
+    last_updated_at: Optional[datetime.datetime]
 
     def __post_init__(self) -> None:
         if self.input is not None:
diff --git a/sdks/python/tests/conftest.py b/sdks/python/tests/conftest.py
index 253960ec8c..98449a22e6 100644
--- a/sdks/python/tests/conftest.py
+++ b/sdks/python/tests/conftest.py
@@ -128,6 +128,31 @@ def fake_backend_without_batching(patch_streamer_without_batching):
         yield fake_message_processor_
 
 
+@pytest.fixture
+def fake_backend_with_patched_environment(request, patch_streamer):
+    """
+    Allows patching environment variables for the duration of the test.
+    Creates a fake backend as in the `fake_backend` fixture.
+    """
+    with testlib.patch_environ(add_keys=request.param):
+        streamer, fake_message_processor_ = patch_streamer
+
+        fake_message_processor_ = cast(
+            backend_emulator_message_processor.BackendEmulatorMessageProcessor,
+            fake_message_processor_,
+        )
+
+        mock_construct_online_streamer = mock.Mock()
+        mock_construct_online_streamer.return_value = streamer
+
+        with mock.patch.object(
+            streamer_constructors,
+            "construct_online_streamer",
+            mock_construct_online_streamer,
+        ):
+            yield fake_message_processor_
+
+
 def random_chars(n: int = 6) -> str:
     return "".join(random.choice(string.ascii_letters) for _ in range(n))
 
diff --git a/sdks/python/tests/library_integration/aisuite/test_aisuite.py b/sdks/python/tests/library_integration/aisuite/test_aisuite.py
index ccdfc3bd84..69149e12e4 100644
--- a/sdks/python/tests/library_integration/aisuite/test_aisuite.py
+++ b/sdks/python/tests/library_integration/aisuite/test_aisuite.py
@@ -79,6 +79,7 @@ def test_aisuite__openai_provider__client_chat_completions_create__happyflow(
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=PROJECT_NAME,
         spans=[
             SpanModel(
@@ -139,6 +140,7 @@ def test_aisuite__nonopenai_provider__client_chat_completions_create__happyflow(
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=PROJECT_NAME,
         spans=[
             SpanModel(
@@ -196,6 +198,7 @@ def test_aisuite_client_chat_completions_create__create_raises_an_error__span_an
         },
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=PROJECT_NAME,
         error_info={
             "exception_type": ANY_STRING(),
@@ -271,6 +274,7 @@ def f():
         output=None,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=PROJECT_NAME,
         spans=[
             SpanModel(
diff --git a/sdks/python/tests/library_integration/anthropic/test_anthropic.py b/sdks/python/tests/library_integration/anthropic/test_anthropic.py
index 68c9ee609e..562963d29d 100644
--- a/sdks/python/tests/library_integration/anthropic/test_anthropic.py
+++ b/sdks/python/tests/library_integration/anthropic/test_anthropic.py
@@ -84,6 +84,7 @@ def test_anthropic_messages_create__happyflow(
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=expected_project_name,
         spans=[
             SpanModel(
@@ -137,6 +138,7 @@ def test_anthropic_messages_create__create_raises_an_error__span_and_trace_finis
         metadata={"created_from": "anthropic", "model": None, "base_url": ANY},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=ANY_BUT_NONE,
         error_info={
             "exception_type": ANY_STRING(),
@@ -216,6 +218,7 @@ def f():
         output=None,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name="anthropic-integration-test",
         spans=[
             SpanModel(
@@ -286,6 +289,7 @@ async def async_f():
         output=None,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=ANY_BUT_NONE,
         spans=[
             SpanModel(
@@ -360,6 +364,7 @@ def test_anthropic_messages_stream__generator_tracked_correctly(
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -434,6 +439,7 @@ def run_stream(client, messages):
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -464,6 +470,7 @@ def run_stream(client, messages):
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -525,6 +532,7 @@ def test_anthropic_messages_stream__get_final_message_called__generator_tracked_
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -587,6 +595,7 @@ def test_anthropic_messages_stream__get_final_message_called_after_stream_iterat
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -651,6 +660,7 @@ async def async_f():
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -715,6 +725,7 @@ async def async_f():
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -776,6 +787,7 @@ def test_anthropic_messages_create__stream_argument_is_True__Stream_object_retur
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -847,6 +859,7 @@ async def async_f():
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
diff --git a/sdks/python/tests/library_integration/crewai/test_crewai.py b/sdks/python/tests/library_integration/crewai/test_crewai.py
index 6282008a26..aa1785d71a 100644
--- a/sdks/python/tests/library_integration/crewai/test_crewai.py
+++ b/sdks/python/tests/library_integration/crewai/test_crewai.py
@@ -50,6 +50,7 @@ def test_crewai__happyflow(
         output=ANY_DICT,
         project_name=project_name,
         start_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         tags=["crewai"],
         spans=[
             SpanModel(
diff --git a/sdks/python/tests/library_integration/dspy/test_dspy.py b/sdks/python/tests/library_integration/dspy/test_dspy.py
index 9168315d36..6fff32fd0a 100644
--- a/sdks/python/tests/library_integration/dspy/test_dspy.py
+++ b/sdks/python/tests/library_integration/dspy/test_dspy.py
@@ -59,6 +59,7 @@ def test_dspy__happyflow(
         metadata={"created_from": "dspy"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=expected_project_name,
         spans=[
             SpanModel(
@@ -131,6 +132,7 @@ def test_dspy__openai_llm_is_used__error_occurred_during_openai_call__error_info
         metadata={"created_from": "dspy"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=project_name,
         spans=[
             SpanModel(
@@ -236,6 +238,7 @@ def f(x):
         output={"output": "the-output"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=project_name,
         spans=[
             SpanModel(
@@ -352,6 +355,7 @@ def f():
         metadata=None,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_STRING(),
diff --git a/sdks/python/tests/library_integration/genai/test_genai.py b/sdks/python/tests/library_integration/genai/test_genai.py
index 698b5369fd..82899d4c6e 100644
--- a/sdks/python/tests/library_integration/genai/test_genai.py
+++ b/sdks/python/tests/library_integration/genai/test_genai.py
@@ -93,6 +93,7 @@ def test_genai_client__generate_content__happyflow(
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=expected_project_name,
         spans=[
             SpanModel(
@@ -151,6 +152,7 @@ def test_genai_client__async_generate_content__happyflow(fake_backend):
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -212,6 +214,7 @@ def f():
         input={},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=expected_project_name,
         spans=[
             SpanModel(
@@ -278,6 +281,7 @@ async def f():
         input={},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -341,6 +345,7 @@ def test_genai_client__generate_content_stream__happyflow(fake_backend):
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name="genai-integration-test",
         spans=[
             SpanModel(
@@ -399,6 +404,7 @@ async def stream_example():
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -458,6 +464,7 @@ def f():
         output=ANY_BUT_NONE,  # tracked generator
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -530,6 +537,7 @@ async def stream_outside_of_parent_function_example():
         output=ANY_BUT_NONE,  # tracked generator
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
diff --git a/sdks/python/tests/library_integration/guardrails/test_guardrails.py b/sdks/python/tests/library_integration/guardrails/test_guardrails.py
index 8112a0629e..1e56fdac57 100644
--- a/sdks/python/tests/library_integration/guardrails/test_guardrails.py
+++ b/sdks/python/tests/library_integration/guardrails/test_guardrails.py
@@ -44,6 +44,7 @@ def test_guardrails__trace_and_span_per_one_validation_check(
         metadata={"created_from": "guardrails", "model": "gpt-3.5-turbo"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=expected_project_name,
         spans=[
             SpanModel(
diff --git a/sdks/python/tests/library_integration/haystack/test_haystack.py b/sdks/python/tests/library_integration/haystack/test_haystack.py
index b55d5383ff..d06b648b75 100644
--- a/sdks/python/tests/library_integration/haystack/test_haystack.py
+++ b/sdks/python/tests/library_integration/haystack/test_haystack.py
@@ -89,6 +89,7 @@ def test_haystack__happyflow(
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=expected_project_name,
         spans=[
             SpanModel(
diff --git a/sdks/python/tests/library_integration/langchain/test_langchain.py b/sdks/python/tests/library_integration/langchain/test_langchain.py
index 998a1a9d18..5e83ebc409 100644
--- a/sdks/python/tests/library_integration/langchain/test_langchain.py
+++ b/sdks/python/tests/library_integration/langchain/test_langchain.py
@@ -62,6 +62,7 @@ def test_langchain__happyflow(
         },
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=expected_project_name,
         spans=[
             SpanModel(
@@ -199,6 +200,7 @@ def test_langchain__distributed_headers__happyflow(
         tags=["tag_d1", "tag_d2"],
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=project_name,
         spans=[
             SpanModel(
@@ -320,6 +322,7 @@ def f(x):
         output={"output": "the-output"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=project_name,
         spans=[
             SpanModel(
@@ -449,6 +452,7 @@ def f():
         output={"output": "output-of-manually-created-trace"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
diff --git a/sdks/python/tests/library_integration/langchain/test_langchain_anthropic.py b/sdks/python/tests/library_integration/langchain/test_langchain_anthropic.py
index 071a121287..24f892e1d4 100644
--- a/sdks/python/tests/library_integration/langchain/test_langchain_anthropic.py
+++ b/sdks/python/tests/library_integration/langchain/test_langchain_anthropic.py
@@ -68,6 +68,7 @@ def test_langchain__anthropic_chat_is_used__token_usage_and_provider_is_logged__
         },
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -184,6 +185,7 @@ def test_langchain__anthropic_chat_is_used__streaming_mode__token_usage_and_prov
         },
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
diff --git a/sdks/python/tests/library_integration/langchain/test_langchain_openai.py b/sdks/python/tests/library_integration/langchain/test_langchain_openai.py
index 21563e3c52..91494d8514 100644
--- a/sdks/python/tests/library_integration/langchain/test_langchain_openai.py
+++ b/sdks/python/tests/library_integration/langchain/test_langchain_openai.py
@@ -121,6 +121,7 @@ def test_langchain__openai_llm_is_used__token_usage_is_logged__happyflow(
         metadata={"a": "b", "created_from": "langchain"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -229,6 +230,7 @@ def test_langchain__openai_llm_is_used__streaming_mode__token_usage_is_logged__h
         },
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -330,6 +332,7 @@ async def invoke_generator(chain, inputs):
         },
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -454,6 +457,7 @@ def invoke_generator(chain, inputs):
         },
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -560,6 +564,7 @@ def test_langchain__openai_llm_is_used__error_occurred_during_openai_call__error
         },
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         error_info={
             "exception_type": ANY_STRING(),
             "traceback": ANY_STRING(),
diff --git a/sdks/python/tests/library_integration/langchain/test_langchain_vertexai.py b/sdks/python/tests/library_integration/langchain/test_langchain_vertexai.py
index 9ab2519780..d09531bba2 100644
--- a/sdks/python/tests/library_integration/langchain/test_langchain_vertexai.py
+++ b/sdks/python/tests/library_integration/langchain/test_langchain_vertexai.py
@@ -98,6 +98,7 @@ def test_langchain__google_vertexai_llm_is_used__token_usage_is_logged__happyflo
         },
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
diff --git a/sdks/python/tests/library_integration/langchain/test_langgraph.py b/sdks/python/tests/library_integration/langchain/test_langgraph.py
index 7e6ed6f60b..6c701cc67a 100644
--- a/sdks/python/tests/library_integration/langchain/test_langgraph.py
+++ b/sdks/python/tests/library_integration/langchain/test_langgraph.py
@@ -69,6 +69,7 @@ def test_langgraph__happyflow(
         },
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
diff --git a/sdks/python/tests/library_integration/llama_index/test_llama_index.py b/sdks/python/tests/library_integration/llama_index/test_llama_index.py
index 309abd5c66..d603885764 100644
--- a/sdks/python/tests/library_integration/llama_index/test_llama_index.py
+++ b/sdks/python/tests/library_integration/llama_index/test_llama_index.py
@@ -93,6 +93,7 @@ def test_llama_index__happyflow(
             metadata={"created_from": "llama_index"},
             start_time=ANY_BUT_NONE,
             end_time=ANY_BUT_NONE,
+            last_updated_at=ANY_BUT_NONE,
             project_name=expected_project_name,
             spans=ANY_BUT_NONE,  # too complex spans tree, no check
         ),
@@ -104,6 +105,7 @@ def test_llama_index__happyflow(
             metadata={"created_from": "llama_index"},
             start_time=ANY_BUT_NONE,
             end_time=ANY_BUT_NONE,
+            last_updated_at=ANY_BUT_NONE,
             project_name=expected_project_name,
             spans=ANY_BUT_NONE,  # too complex spans tree, no check
         ),
@@ -178,6 +180,7 @@ def test_llama_index__no_index_construction_logging_happyflow(
             metadata={"created_from": "llama_index"},
             start_time=ANY_BUT_NONE,
             end_time=ANY_BUT_NONE,
+            last_updated_at=ANY_BUT_NONE,
             project_name=expected_project_name,
             spans=ANY_BUT_NONE,  # too complex spans tree, no check
         ),
@@ -251,6 +254,7 @@ def test_llama_index_chat__happyflow(
             metadata={"created_from": "llama_index"},
             start_time=ANY_BUT_NONE,
             end_time=ANY_BUT_NONE,
+            last_updated_at=ANY_BUT_NONE,
             project_name=expected_project_name,
             spans=[
                 SpanModel(
@@ -339,6 +343,7 @@ def test_llama_index_stream_chat__happyflow(
             metadata={"created_from": "llama_index"},
             start_time=ANY_BUT_NONE,
             end_time=ANY_BUT_NONE,
+            last_updated_at=ANY_BUT_NONE,
             project_name=expected_project_name,
             spans=[
                 SpanModel(
diff --git a/sdks/python/tests/library_integration/openai/agents_tests/test_opik_tracing_processor.py b/sdks/python/tests/library_integration/openai/agents_tests/test_opik_tracing_processor.py
index 0a1115481b..515f0f0228 100644
--- a/sdks/python/tests/library_integration/openai/agents_tests/test_opik_tracing_processor.py
+++ b/sdks/python/tests/library_integration/openai/agents_tests/test_opik_tracing_processor.py
@@ -40,6 +40,7 @@ def test_opik_tracing_processor__happy_flow(fake_backend):
         name="Agent workflow",
         project_name=project_name,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         metadata={
             "created_from": "openai-agents",
             "agents-trace-id": ANY_STRING(startswith="trace"),
@@ -115,6 +116,7 @@ def test_opik_tracing_processor__happy_flow_conversation(fake_backend):
         name=workflow_name,
         project_name=project_name,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         metadata={
             "created_from": "openai-agents",
             "agents-trace-id": ANY_STRING(startswith="trace"),
@@ -202,6 +204,7 @@ async def test_opik_tracing_processor__handsoff(fake_backend):
         input=ANY_DICT,
         output=ANY_DICT,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         metadata={
             "created_from": "openai-agents",
             "agents-trace-id": ANY_STRING(startswith="trace"),
@@ -320,6 +323,7 @@ def get_weather(city: str) -> str:
         name="Agent workflow",
         project_name=project_name,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         metadata={
             "created_from": "openai-agents",
             "agents-trace-id": ANY_STRING(startswith="trace"),
diff --git a/sdks/python/tests/library_integration/openai/test_openai_chat_completions.py b/sdks/python/tests/library_integration/openai/test_openai_chat_completions.py
index d467fe70a5..bbca5ba8ee 100644
--- a/sdks/python/tests/library_integration/openai/test_openai_chat_completions.py
+++ b/sdks/python/tests/library_integration/openai/test_openai_chat_completions.py
@@ -90,6 +90,7 @@ def test_openai_client_chat_completions_create__happyflow(
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=expected_project_name,
         spans=[
             SpanModel(
@@ -147,6 +148,7 @@ def test_openai_client_chat_completions_create__create_raises_an_error__span_and
         },
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=ANY_BUT_NONE,
         error_info={
             "exception_type": ANY_STRING(),
@@ -224,6 +226,7 @@ def f():
         output=None,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=project_name,
         spans=[
             SpanModel(
@@ -297,6 +300,7 @@ async def async_f():
         output=None,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=ANY_BUT_NONE,
         spans=[
             SpanModel(
@@ -372,6 +376,7 @@ def test_openai_client_chat_completions_create__stream_mode_is_on__generator_tra
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=ANY_BUT_NONE,
         spans=[
             SpanModel(
@@ -435,6 +440,7 @@ async def async_f():
         output=None,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=ANY_BUT_NONE,
         spans=[
             SpanModel(
@@ -522,6 +528,7 @@ class CalendarEvent(BaseModel):
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=expected_project_name,
         spans=[
             SpanModel(
@@ -589,6 +596,7 @@ class CalendarEvent(BaseModel):
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -654,6 +662,7 @@ def test_openai_chat_completion_stream__generator_tracked_correctly(
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -717,6 +726,7 @@ def test_openai_chat_completion_stream__include_usage_is_not_enabled__usage_not_
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -787,6 +797,7 @@ def run_stream(messages):
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -814,6 +825,7 @@ def run_stream(messages):
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -881,6 +893,7 @@ def test_openai_chat_completion_stream__get_final_completion_called__generator_t
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -945,6 +958,7 @@ def test_openai_chat_completion_stream__get_final_completion_called_after_stream
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -1011,6 +1025,7 @@ async def async_f():
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -1077,6 +1092,7 @@ async def async_f():
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
diff --git a/sdks/python/tests/library_integration/openai/test_openai_responses.py b/sdks/python/tests/library_integration/openai/test_openai_responses.py
index cbdb937152..a1d1dde8c7 100644
--- a/sdks/python/tests/library_integration/openai/test_openai_responses.py
+++ b/sdks/python/tests/library_integration/openai/test_openai_responses.py
@@ -76,6 +76,7 @@ def test_openai_client_responses_create__happyflow(
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=expected_project_name,
         spans=[
             SpanModel(
@@ -134,6 +135,7 @@ def f():
         input={},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -203,6 +205,7 @@ def test_openai_client_responses_create_raises_an_error__span_and_trace_finished
         },
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=ANY_BUT_NONE,
         error_info={
             "exception_type": ANY_STRING(),
@@ -274,6 +277,7 @@ def test_openai_client_responses_create_stream__happyflow(fake_backend):
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -330,6 +334,7 @@ async def test_openai_client_responses_create_async__happyflow(fake_backend):
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -388,6 +393,7 @@ async def test_openai_client_responses_create_stream_async__happyflow(fake_backe
         metadata=ANY_DICT,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -462,6 +468,7 @@ class CalendarEvent(pydantic.BaseModel):
         tags=["openai"],
         metadata=ANY_DICT,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -527,6 +534,7 @@ class CalendarEvent(pydantic.BaseModel):
         tags=["openai"],
         metadata=ANY_DICT,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -599,6 +607,7 @@ class CalendarEvent(pydantic.BaseModel):
         },
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name=ANY_BUT_NONE,
         error_info=ErrorInfoDict(
             exception_type=ANY_STRING(),
diff --git a/sdks/python/tests/testlib/backend_emulator_message_processor.py b/sdks/python/tests/testlib/backend_emulator_message_processor.py
index aa7fac1581..d5c201b531 100644
--- a/sdks/python/tests/testlib/backend_emulator_message_processor.py
+++ b/sdks/python/tests/testlib/backend_emulator_message_processor.py
@@ -111,6 +111,7 @@ def _dispatch_message(self, message: messages.BaseMessage) -> None:
                 project_name=message.project_name,
                 error_info=message.error_info,
                 thread_id=message.thread_id,
+                last_updated_at=message.last_updated_at,
             )
 
             self._trace_trees.append(trace)
@@ -128,6 +129,7 @@ def _dispatch_message(self, message: messages.BaseMessage) -> None:
                 end_time=message.end_time,
                 project_name=message.project_name,
                 thread_id=message.thread_id,
+                last_updated_at=message.last_updated_at,
             )
 
             self._trace_trees.append(trace)
diff --git a/sdks/python/tests/testlib/fake_message_factory.py b/sdks/python/tests/testlib/fake_message_factory.py
index 66264e0914..883c09e536 100644
--- a/sdks/python/tests/testlib/fake_message_factory.py
+++ b/sdks/python/tests/testlib/fake_message_factory.py
@@ -51,11 +51,12 @@ def fake_create_trace_message_batch(
 
         # Randomly decide if the trace has ended
         has_ended = random.choice([True, False])
-        end_time = (
-            start_time + timedelta(seconds=random.randint(1, 3600))
-            if has_ended
-            else None
-        )
+        if has_ended:
+            end_time = start_time + timedelta(seconds=random.randint(1, 3600))
+            last_updated_at = end_time
+        else:
+            end_time = None
+            last_updated_at = start_time
 
         # Generate dummy input data
         input_data = {
@@ -129,6 +130,7 @@ def fake_create_trace_message_batch(
             tags=tags,
             error_info=error_info,
             thread_id=thread_id,
+            last_updated_at=last_updated_at,
         )
 
         dummy_traces.append(trace_message)
diff --git a/sdks/python/tests/testlib/models.py b/sdks/python/tests/testlib/models.py
index fd5d860bb1..b6b2cae2de 100644
--- a/sdks/python/tests/testlib/models.py
+++ b/sdks/python/tests/testlib/models.py
@@ -52,6 +52,7 @@ class TraceModel:
     )
     error_info: Optional[ErrorInfoDict] = None
     thread_id: Optional[str] = None
+    last_updated_at: Optional[datetime.datetime] = None
 
 
 @dataclasses.dataclass
diff --git a/sdks/python/tests/unit/api_objects/trace/__init__.py b/sdks/python/tests/unit/api_objects/trace/__init__.py
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/sdks/python/tests/unit/api_objects/trace/test_trace_data.py b/sdks/python/tests/unit/api_objects/trace/test_trace_data.py
new file mode 100644
index 0000000000..b9fa848bf0
--- /dev/null
+++ b/sdks/python/tests/unit/api_objects/trace/test_trace_data.py
@@ -0,0 +1,59 @@
+import datetime
+
+from opik.api_objects import trace
+from ....testlib import assert_equal, ANY_BUT_NONE
+
+
+def test_trace_data__trace_start_parameters__expected_parameters_are_set():
+    trace_data = trace.TraceData(
+        name="name",
+        project_name="project_name",
+        created_by="evaluation",
+    )
+
+    expected_parameters = {
+        "id": ANY_BUT_NONE,
+        "start_time": ANY_BUT_NONE,
+        "project_name": "project_name",
+        "_trace_start": True,
+    }
+    start_parameters = trace_data.trace_start_parameters
+
+    assert_equal(expected_parameters, start_parameters)
+
+
+def test_trace_data__trace_end_parameters__expected_parameters_are_set():
+    trace_data = trace.TraceData(
+        name="name",
+        end_time=datetime.datetime.now(),
+        metadata={"foo": "bar"},
+        input={"input": "input"},
+        output={"output": "output"},
+        tags=["one", "two"],
+        feedback_scores=[{"name": "score_name", "value": 0.5}],
+        project_name="project_name",
+        created_by="evaluation",
+        error_info={},
+        thread_id="thread_id",
+        attachments=[],
+    )
+
+    expected_parameters = {
+        "id": ANY_BUT_NONE,
+        "name": "name",
+        "start_time": ANY_BUT_NONE,
+        "end_time": ANY_BUT_NONE,
+        "metadata": ANY_BUT_NONE,
+        "input": ANY_BUT_NONE,
+        "output": ANY_BUT_NONE,
+        "tags": ANY_BUT_NONE,
+        "feedback_scores": ANY_BUT_NONE,
+        "project_name": "project_name",
+        "error_info": ANY_BUT_NONE,
+        "thread_id": ANY_BUT_NONE,
+        "attachments": ANY_BUT_NONE,
+        "_trace_start": False,
+    }
+    end_parameters = trace_data.trace_end_parameters
+
+    assert_equal(expected_parameters, end_parameters)
diff --git a/sdks/python/tests/unit/decorator/test_tracker_outputs.py b/sdks/python/tests/unit/decorator/test_tracker_outputs.py
index f1ef32db61..b3e46b1c10 100644
--- a/sdks/python/tests/unit/decorator/test_tracker_outputs.py
+++ b/sdks/python/tests/unit/decorator/test_tracker_outputs.py
@@ -38,6 +38,7 @@ def f_outer(x):
         output={"output": "outer-output"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -83,6 +84,7 @@ def f(x):
         output=None,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -118,6 +120,7 @@ def f(x):
         output={"some-key": "the-output-value"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -161,6 +164,7 @@ def f_outer(x):
         output={"output": "outer-output"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -226,6 +230,7 @@ def f_outer(x):
         output={"output": "outer-output"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -285,6 +290,7 @@ def f_2(x):
             output={"output": "f1-output"},
             start_time=ANY_BUT_NONE,
             end_time=ANY_BUT_NONE,
+            last_updated_at=ANY_BUT_NONE,
             spans=[
                 SpanModel(
                     id=ANY_BUT_NONE,
@@ -304,6 +310,7 @@ def f_2(x):
             output={"output": "f2-output"},
             start_time=ANY_BUT_NONE,
             end_time=ANY_BUT_NONE,
+            last_updated_at=ANY_BUT_NONE,
             spans=[
                 SpanModel(
                     id=ANY_BUT_NONE,
@@ -343,6 +350,7 @@ def f(x):
         output=None,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         error_info={
             "exception_type": "Exception",
             "message": "error message",
@@ -396,6 +404,7 @@ def f_inner():
         output={"output": "the-output"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -449,6 +458,7 @@ async def async_f(x):
         output=None,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         error_info={
             "exception_type": "Exception",
             "message": "error message",
@@ -515,6 +525,7 @@ def f_outer(x):
             output={"output": "outer-output"},
             start_time=ANY_BUT_NONE,
             end_time=ANY_BUT_NONE,
+            last_updated_at=ANY_BUT_NONE,
             spans=[
                 SpanModel(
                     id=ID_STORAGE["f_outer-span-id"],
@@ -534,6 +545,7 @@ def f_outer(x):
             output={"output": "inner-output-from-thread-1"},
             start_time=ANY_BUT_NONE,
             end_time=ANY_BUT_NONE,
+            last_updated_at=ANY_BUT_NONE,
             spans=[
                 SpanModel(
                     id=ID_STORAGE["f_inner-span-id-thread-1"],
@@ -553,6 +565,7 @@ def f_outer(x):
             output={"output": "inner-output-from-thread-2"},
             start_time=ANY_BUT_NONE,
             end_time=ANY_BUT_NONE,
+            last_updated_at=ANY_BUT_NONE,
             spans=[
                 SpanModel(
                     id=ID_STORAGE["f_inner-span-id-thread-2"],
@@ -614,6 +627,7 @@ def f(x):
         output={"output": "yielded-1 yielded-2 yielded-3"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -655,6 +669,7 @@ def f(x):
         output=None,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         error_info={
             "exception_type": "Exception",
             "message": "error message",
@@ -715,6 +730,7 @@ def f_outer(x):
         output={"output": "outer-output"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -790,6 +806,7 @@ def f_outer(x):
         output={"output": "outer-output"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -859,6 +876,7 @@ async def async_f(x):
         output={"output": "the-output"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -901,6 +919,7 @@ async def async_f_outer(x):
         output={"output": "outer-output"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -955,6 +974,7 @@ async def async_generator_user():
         output={"output": "yielded-1 yielded-2 yielded-3"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -1003,6 +1023,7 @@ async def async_generator_user():
         output={"output": "yielded-1 yielded-2 yielded-3"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -1058,6 +1079,7 @@ async def async_generator_user(x):
         output={"output": "generator-user-output"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -1118,6 +1140,7 @@ async def async_generator_user(x):
         output={"output": "generator-user-output"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -1189,6 +1212,7 @@ def f_outer(x):
         output={"output": "outer-output"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -1253,6 +1277,7 @@ def f(x):
         output={"output": "output-of-manually-created-trace"},
         start_time=mock.ANY,  # not ANY_BUT_NONE because we created span manually in the test
         end_time=mock.ANY,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -1300,6 +1325,7 @@ def f(x):
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
         thread_id="some-thread-id",
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -1346,6 +1372,7 @@ def f(x):
         output={"output": "f-output", "trace-output-key": "trace-output-value"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -1390,6 +1417,7 @@ def f(x):
         output={"output": "f-output"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         feedback_scores=[
             FeedbackScoreModel(id=ANY_BUT_NONE, name="trace-score-name", value=0.75)
         ],
@@ -1431,6 +1459,7 @@ def f(a, b, c=3, d=4, e=5):
         output={"some-key": "the-output-value"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -1465,6 +1494,7 @@ def f():
         output={"some-key": "the-output-value"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -1501,6 +1531,7 @@ def f(x):
         output=None,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         error_info={
             "exception_type": "TypeError",
             "traceback": ANY_STRING(),
@@ -1553,6 +1584,7 @@ def f(x):
         output={"output": "f-output"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -1613,6 +1645,7 @@ def f(x):
         output={"output": "the-output"},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -1673,6 +1706,7 @@ def do_distributed_trace(x):
         input={"x": "do_distributed_trace-input"},
         output={"output": "do_distributed_trace-output"},
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -1749,6 +1783,7 @@ def do_distributed_trace(x):
         input={"x": "do_distributed_trace-input"},
         output={"output": "do_distributed_trace-output"},
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
diff --git a/sdks/python/tests/unit/decorator/test_tracker_outputs_long_running_jobs.py b/sdks/python/tests/unit/decorator/test_tracker_outputs_long_running_jobs.py
new file mode 100644
index 0000000000..52554225a4
--- /dev/null
+++ b/sdks/python/tests/unit/decorator/test_tracker_outputs_long_running_jobs.py
@@ -0,0 +1,63 @@
+import pytest
+
+from opik.decorator import tracker
+from ...testlib import (
+    ANY_BUT_NONE,
+    SpanModel,
+    TraceModel,
+    assert_equal,
+)
+
+
+@pytest.mark.parametrize(
+    "fake_backend_with_patched_environment",
+    [{"OPIK_LOG_START_TRACE": "True"}],
+    indirect=True,
+)
+def test_track__trace_logged_at_start_and_end(fake_backend_with_patched_environment):
+    @tracker.track
+    def f(x):
+        return "the-output"
+
+    f("the-input")
+
+    tracker.flush_tracker()
+
+    EXPECTED_TRACE_TREE = [
+        TraceModel(
+            id=ANY_BUT_NONE,
+            name=None,
+            start_time=ANY_BUT_NONE,
+            project_name="Default Project",
+            last_updated_at=ANY_BUT_NONE,
+        ),
+        TraceModel(
+            id=ANY_BUT_NONE,
+            start_time=ANY_BUT_NONE,
+            name="f",
+            project_name="Default Project",
+            input={"x": "the-input"},
+            output={"output": "the-output"},
+            end_time=ANY_BUT_NONE,
+            last_updated_at=ANY_BUT_NONE,
+            spans=[
+                SpanModel(
+                    id=ANY_BUT_NONE,
+                    start_time=ANY_BUT_NONE,
+                    name="f",
+                    input={"x": "the-input"},
+                    output={"output": "the-output"},
+                    end_time=ANY_BUT_NONE,
+                    project_name="Default Project",
+                )
+            ],
+        ),
+    ]
+
+    trace_trees = fake_backend_with_patched_environment.trace_trees
+    assert len(trace_trees) == 2
+
+    assert_equal(EXPECTED_TRACE_TREE, trace_trees)
+
+    assert trace_trees[0].id == trace_trees[1].id
+    assert trace_trees[0].last_updated_at < trace_trees[1].last_updated_at
diff --git a/sdks/python/tests/unit/evaluation/metrics/test_tracked_metrics.py b/sdks/python/tests/unit/evaluation/metrics/test_tracked_metrics.py
index acd51f27e4..41a71f2d45 100644
--- a/sdks/python/tests/unit/evaluation/metrics/test_tracked_metrics.py
+++ b/sdks/python/tests/unit/evaluation/metrics/test_tracked_metrics.py
@@ -34,6 +34,7 @@ def test_metric_equals__track_enabled__happyflow(fake_backend):
         output={"output": score_result},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         spans=[
             SpanModel(
                 id=ANY_BUT_NONE,
@@ -80,6 +81,7 @@ def test_metric_equals__track_enabled__project_name_set__data_logged_to_the_spec
         output={"output": score_result},
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         project_name="metric-project-name",
         spans=[
             SpanModel(
diff --git a/sdks/python/tests/unit/evaluation/test_evaluate.py b/sdks/python/tests/unit/evaluation/test_evaluate.py
index 9a3ad9f9b2..1327b8ab46 100644
--- a/sdks/python/tests/unit/evaluation/test_evaluate.py
+++ b/sdks/python/tests/unit/evaluation/test_evaluate.py
@@ -92,6 +92,7 @@ def say_task(dataset_item: Dict[str, Any]):
             },
             start_time=ANY_BUT_NONE,
             end_time=ANY_BUT_NONE,
+            last_updated_at=ANY_BUT_NONE,
             spans=[
                 SpanModel(
                     id=ANY_BUT_NONE,
@@ -162,6 +163,7 @@ def say_task(dataset_item: Dict[str, Any]):
             },
             start_time=ANY_BUT_NONE,
             end_time=ANY_BUT_NONE,
+            last_updated_at=ANY_BUT_NONE,
             spans=[
                 SpanModel(
                     id=ANY_BUT_NONE,
@@ -305,6 +307,7 @@ def say_task(dataset_item: Dict[str, Any]):
             },
             start_time=ANY_BUT_NONE,
             end_time=ANY_BUT_NONE,
+            last_updated_at=ANY_BUT_NONE,
             spans=[
                 SpanModel(
                     id=ANY_BUT_NONE,
@@ -379,6 +382,7 @@ def say_task(dataset_item: Dict[str, Any]):
             },
             start_time=ANY_BUT_NONE,
             end_time=ANY_BUT_NONE,
+            last_updated_at=ANY_BUT_NONE,
             spans=[
                 SpanModel(
                     id=ANY_BUT_NONE,
@@ -559,6 +563,7 @@ def say_task(dataset_item: Dict[str, Any]):
         output=None,
         start_time=ANY_BUT_NONE,
         end_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         error_info={
             "exception_type": "Exception",
             "message": "some-error-message",
@@ -682,6 +687,7 @@ def test_evaluate_prompt_happyflow(
             },
             start_time=ANY_BUT_NONE,
             end_time=ANY_BUT_NONE,
+            last_updated_at=ANY_BUT_NONE,
             spans=[
                 SpanModel(
                     id=ANY_BUT_NONE,
@@ -735,6 +741,7 @@ def test_evaluate_prompt_happyflow(
             },
             start_time=ANY_BUT_NONE,
             end_time=ANY_BUT_NONE,
+            last_updated_at=ANY_BUT_NONE,
             spans=[
                 SpanModel(
                     id=ANY_BUT_NONE,
@@ -871,6 +878,7 @@ def aggregator(results: List[score_result.ScoreResult]) -> score_result.ScoreRes
             },
             start_time=ANY_BUT_NONE,
             end_time=ANY_BUT_NONE,
+            last_updated_at=ANY_BUT_NONE,
             spans=[
                 SpanModel(
                     id=ANY_BUT_NONE,
@@ -985,6 +993,7 @@ def aggregator(results: List[score_result.ScoreResult]) -> score_result.ScoreRes
             },
             start_time=ANY_BUT_NONE,
             end_time=ANY_BUT_NONE,
+            last_updated_at=ANY_BUT_NONE,
             spans=[
                 SpanModel(
                     id=ANY_BUT_NONE,
diff --git a/sdks/python/tests/unit/llm_usage/test_client_with_usage_logged.py b/sdks/python/tests/unit/llm_usage/test_client_with_usage_logged.py
index 4152c5f5c2..3f8dff3a17 100644
--- a/sdks/python/tests/unit/llm_usage/test_client_with_usage_logged.py
+++ b/sdks/python/tests/unit/llm_usage/test_client_with_usage_logged.py
@@ -22,6 +22,7 @@ def test_span__provider_supported__usage_format_is_correct__usage_converted_to_o
     EXPECTED_TRACE_TREE = TraceModel(
         id=ANY_BUT_NONE,
         start_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         name="some-name",
         spans=[
             SpanModel(
@@ -67,6 +68,7 @@ def test_span__provider_not_passed__usage_format_is_correct_for_some_provider__u
     EXPECTED_TRACE_TREE = TraceModel(
         id=ANY_BUT_NONE,
         start_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         name="some-name",
         spans=[
             SpanModel(
@@ -111,6 +113,7 @@ def test_span__unknown_provider_passed__usage_format_is_correct_for_some_provide
     EXPECTED_TRACE_TREE = TraceModel(
         id=ANY_BUT_NONE,
         start_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         name="some-name",
         spans=[
             SpanModel(
@@ -160,6 +163,7 @@ def test_span__unknown_provider_passed__usage_format_is_also_unknown__usage_flat
     EXPECTED_TRACE_TREE = TraceModel(
         id=ANY_BUT_NONE,
         start_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         name="some-name",
         spans=[
             SpanModel(
@@ -206,6 +210,7 @@ def test_span__user_added_openai_keys_to_unknown_usage_themselves__they_are_incl
     EXPECTED_TRACE_TREE = TraceModel(
         id=ANY_BUT_NONE,
         start_time=ANY_BUT_NONE,
+        last_updated_at=ANY_BUT_NONE,
         name="some-name",
         spans=[
             SpanModel(
diff --git a/sdks/python/tests/unit/test_messages.py b/sdks/python/tests/unit/test_messages.py
index 128f195afb..9e04ea9e52 100644
--- a/sdks/python/tests/unit/test_messages.py
+++ b/sdks/python/tests/unit/test_messages.py
@@ -17,6 +17,7 @@ def test_messages__all_fields_are_serializable():
         "tags": None,
         "error_info": None,
         "thread_id": None,
+        "last_updated_at": datetime.now(),
     }
 
     message = messages.CreateTraceMessage(**payload_dict)
@@ -48,6 +49,7 @@ def test_messages__not_all_fields_are_serializable():
         "tags": None,
         "error_info": None,
         "thread_id": None,
+        "last_updated_at": datetime.now(),
     }
 
     message = messages.CreateTraceMessage(**payload_dict)


Code review comments:
diff_hunk: @@ -101,3 +101,31 @@ def _update_attachments(self, attachments: List[Attachment]) -> None:
             self.attachments = attachments
         else:
             self.attachments.extend(attachments)
+
+    @property
+    def trace_start_parameters(self) -> Dict[str, Any]:
path: sdks/python/src/opik/api_objects/trace/trace_data.py
text: @user1:
Once we move the logging decision from Opik class to base_track_decorator, we may keep just something like `as_parameters()` method or a similar property that will behave similarly to __dict__ but remove the fields we don't need (e.g. created_by), no need to distinguish between start and end once we no longer need `_trace_start`

@author:
Not sure that we can move this decision here because some integrations doesn't use base_track_decorator at all.

diff_hunk: @@ -229,14 +230,20 @@ def trace(
             thread_id: Used to group multiple traces into a thread.
                 The identifier is user-defined and has to be unique per project.
             attachments: The list of attachments to be uploaded to the trace.
+            _trace_start: Internal use only.
 
         Returns:
             trace.Trace: The created trace object.
         """
+        if _trace_start is True and self._config.log_start_trace is False:
path: sdks/python/src/opik/api_objects/opik_client.py
text: @user1:
Let's move this if-condition to base_track_decorator (whether to log or no). Opik class shouldn't know about start/end, it should just log the data.

@author:
We have quite a few integrations that doesn't depend on base track decorator but rather use this method directly. You can find them in this pull request as well. Thus, if we move mentioned logic inside base track decorator then we need to either duplicate it in all related integrations or just not apply it in those integrations. Taking into account that most of users use OPIK implicitly through integrations, we would be forced to duplicate it in many places.
Not sure that this is a good idea.

diff_hunk: @@ -418,6 +418,9 @@ def _before_call(
             context_storage.set_trace_data(created_trace_data)
             TRACES_CREATED_BY_DECORATOR.add(created_trace_data.id)
 
+            client = opik_client.get_client_cached()
+            client.trace(**created_trace_data.trace_start_parameters)
path: sdks/python/src/opik/decorator/base_track_decorator.py
text: @user1:
As mentioned in another comment, this place should contain if-condition to decide - log or not.

@author:
Unfortunately, this method is not invoked in all possible scenarios. Furthermore, it is not invoked for some important integrations.

For example: DSPy, ADK, OpenAI agents, and Llama index integrations use Opik client trace() method directly rather than via base tracker decorator.