Repository: microsoft/autogen
Pull request #5890: add LLMStreamStartEvent and LLMStreamEndEvent
Author: login: EItanya
Created: 2025-03-10T18:17:53Z | Merged: 2025-03-11T22:02:46Z

<!-- Thank you for your contribution! Please review https://microsoft.github.io/autogen/docs/Contribute before opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR. If you don't have the access to it, we will shortly find a reviewer and assign them to your PR. -->

@ekzhu 

## Why are these changes needed?

These changes are needed because there is currently no way to get logging information about Streaming LLM requests/responses.

I decided to put the StreamStart event AFTER the first chunk so there aren't false positives about connections/auth.

## Related issue number

<!-- For example: "Closes #1234" -->

Closes #5730

## Checks

- [ ] I've included any doc changes needed for <https://microsoft.github.io/autogen/>. See <https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md> to build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes introduced in this PR.
- [ ] I've made sure all auto checks have passed.


Linked issue #1234: [Bug]:  Error running LLava Agent using LM Studio and Autogen
### Describe the bug

I am trying to implement this notebook: https://github.com/microsoft/autogen/blob/main/notebook/agentchat_lmm_llava.ipynb

The only difference is that I am hosting my BakLLava model using `LM Studio` and  running it behind OpenAI API. When testing the `llava_call` I get the following error:

> Error: Invalid reference to model version: http://0.0.0.0:8001/v1. Expected format: owner/name:version
> None

### Steps to reproduce

1. Install required packages
```sh
pip3 install replicate
pip3 install pillow
pip3 install matplotlib
```
2. Import packages and set `LLAVA_MODE` to "local"
```python
# More details in the two setup options below.
import json
import os
import random
import time
from typing import Any, Callable, Dict, List, Optional, Tuple, Type, Union

import matplotlib.pyplot as plt
import requests
from PIL import Image
from termcolor import colored

import autogen
from autogen import Agent, AssistantAgent, ConversableAgent, UserProxyAgent
from autogen.agentchat.contrib.llava_agent import LLaVAAgent, llava_call

LLAVA_MODE = "local"  # Either "local" or "remote"
assert LLAVA_MODE in ["local", "remote"]
```
3. Execute the `llava_call` test
```python
from autogen.agentchat.contrib.llava_agent import llava_call

config_list = [
    {
        "model": "llava",
        "api_key": "NULL",
        "base_url": "http://0.0.0.0:8001/v1",
    }
]

rst = llava_call("Describe this AutoGen framework <img https://raw.githubusercontent.com/microsoft/autogen/main/website/static/img/autogen_agentchat.png> with bullet points.",
    llm_config={
        "config_list": config_list,
        "temperature": 0
    }
)

print(rst)
```
4. The following error comes up

> Error: Invalid reference to model version: http://0.0.0.0:8001/v1. Expected format: owner/name:version
> None

### Expected Behavior

It should be able to execute the llava_call successfully and respond with something like the following:

> The AutoGen framework is a tool for creating and managing conversational agents. It allows for the creation of multiple-agent conversations, enabling complex interactions between different agents. The framework is designed to be flexible and scalable, allowing for the addition of new agents and conversations as needed.
> 
> The framework consists of three main components:
> 
> 1. Agents: These are the individual conversational entities that can be created and managed within the framework. Each agent has its own unique set of conversational capabilities and can engage in conversations with other agents.
> 
> 2. Conversations: These are the interactions between agents, which can be managed and directed by the framework. Conversations can be structured and organized to facilitate efficient communication between agents.
> 
> 3. Flexibility: The framework is designed to be flexible, allowing for the addition of new agents and conversations as needed. This flexibility enables the framework to adapt to changing requirements and facilitate the development of more complex conversational systems.

### Screenshots and logs

_No response_

### Additional Information

Autogen Version: 0.2.6
Operating System: Windows 11 Pro
Python Version: 3.11.6

To prove that the LM Studio/OpenAI API hosting of BakLLava works, the following code works fine using Autogens Enhanced Inference.

```python
#Autogen Enhanced Inference, API Unification
from autogen import OpenAIWrapper
import base64

# Function to encode the image
def encode_image(image_path):
  with open(image_path, "rb") as image_file:
    return base64.b64encode(image_file.read()).decode('utf-8')

# Path to your image
image_path = "image.jpg"

# Getting the base64 string
base64_image = encode_image(image_path)

# OpenAI endpoint
client = OpenAIWrapper(cache_seed="None", model="llava", base_url="http://0.0.0.0:8001/v1", api_key="NULL") 

# ChatCompletion
response = client.create(messages=[{"role": "user", "content": [
        {
          "type": "text",
          "text": "Whats in this image?"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": f"data:image/jpeg;base64,{base64_image}"
          }
        }
      ]
    }
  ], 
  model="llava")
# extract the response text
print(client.extract_text_or_completion_object(response))
```
Linked issue comments: user: login: sonichi
created_at: 2024-01-14T01:57:25Z
body: @BeibinLi fyi
author_association: CONTRIBUTOR
user: login: BeibinLi
created_at: 2024-01-14T07:54:53Z
body: @shaunaa126 You should use MultimodalConversableAgent instead of LLaVAAgent, because your API call supports the OpenAI API version. The LLaVA agents are designed for the original LLaVA client. 

Check the [GPT-4V notebook](https://github.com/microsoft/autogen/blob/main/notebook/agentchat_lmm_gpt-4v.ipynb) for reference. The usage should be the same (except the config_list). 
 
author_association: CONTRIBUTOR
user: login: shaunaa126
created_at: 2024-01-15T20:03:17Z
body: @BeibinLi thanks for your response, your solution worked. I was able to use MultiModalConversableAgent to make a call to my OpenAI API endpoint.

Do you know how to send an image locally instead of using image url? The following doesn't work:
```python
user_proxy.initiate_chat(assistant, message=f"Describe this image. <img src={image_path}>" )
```
author_association: NONE
user: login: BeibinLi
created_at: 2024-01-16T01:21:22Z
body: > @BeibinLi thanks for your response, your solution worked. I was able to use MultiModalConversableAgent to make a call to my OpenAI API endpoint.
> 
> Do you know how to send an image locally instead of using image url? The following doesn't work:
> 
> ```python
> user_proxy.initiate_chat(assistant, message=f"Describe this image. <img src={image_path}>" )
> ```

Can you send me error message regarding the local image issues? I don't have LM Studio to reproduce your error. Another thing to check is to give an "absolute" path instead of relative path for the message. For instance, you can try
```
image_path = os.path.abspath(image_path)
print(image_path)
assert os.path.exists(image_path)
user_proxy.initiate_chat(assistant, message=f"Describe this image. <img src={image_path}>" )
```

author_association: CONTRIBUTOR
user: login: shaunaa126
created_at: 2024-01-16T02:43:09Z
body: I tried the above and this is the error message I received, which is what I have been receiving with the prior code:

```sh
[c:\Users\Downloads\Github\image.jpg](file:///C:/Users/Downloads/Github/image.jpg)
Warning! Unable to load image from src=c:\Users\Downloads\Github\image.jpg, because [Errno 22] Invalid argument: 'src=c:\\Users\\Downloads\\Github\\image.jpg'
user_proxy (to image-explainer):
```

I am not sure if `src` is supported. Are there any examples in Autogen of sending a local image instead of a http url in the chat? I don't think this error is specific to LM Studio. Here is my full code example:

```python
import base64

# Function to encode the image
def encode_image(image_path):
  with open(image_path, "rb") as image_file:
    return base64.b64encode(image_file.read()).decode('utf-8')

# Path to your image
image_path = "image.jpg"

# Getting the base64 string
base64_image = encode_image(image_path)

image_path = os.path.abspath(image_path)
print(image_path)
assert os.path.exists(image_path)

# Load LLM inference endpoints
config_list = [
    {
        "model": "llava",
        "api_key": "None",
        "base_url": "http://0.0.0.0:8001/v1",
    }
]

assistant = MultimodalConversableAgent(
    name="image-explainer",
    max_consecutive_auto_reply=10,
    llm_config={"config_list": config_list, "temperature": 0.5, "max_tokens": 300},
)

user_proxy = autogen.UserProxyAgent(
    name="user_proxy",
    system_message="A human admin.",
    human_input_mode="NEVER",  # Try between ALWAYS or NEVER
    max_consecutive_auto_reply=0,
)

# This initiates an automated chat between the two agents to solve the task
user_proxy.initiate_chat(assistant, message=f"Describe this image. <img src={image_path}>" )

```
author_association: NONE
user: login: BeibinLi
created_at: 2024-01-16T16:39:01Z
body: It is not a HTML image tag. Just use "image_path".

For instance
`user_proxy.initiate_chat(assistant, message=f"Describe this image. <img {image_path}>" )`
`user_proxy.initiate_chat(assistant, message=f"Describe this image. <img C:/User/temp/Downloads/test.jpg}>" )`
`user_proxy.initiate_chat(assistant, message=f"Describe this image. <img xyz/test.jpg}>" )`
author_association: CONTRIBUTOR
user: login: shaunaa126
created_at: 2024-01-16T23:13:17Z
body: @BeibinLi thank you, that worked. I am able to use LM Studio to host my Multmodal LLM and run inference as well as MultimodalConversableAgent using Autogen. Great stuff. Appreciate your time.
author_association: NONE

Diff:
diff --git a/python/packages/autogen-core/src/autogen_core/logging.py b/python/packages/autogen-core/src/autogen_core/logging.py
index 6ea430f27e4d..18811f8d2b5b 100644
--- a/python/packages/autogen-core/src/autogen_core/logging.py
+++ b/python/packages/autogen-core/src/autogen_core/logging.py
@@ -29,11 +29,14 @@ def __init__(
 
             .. code-block:: python
 
+                import logging
                 from autogen_core import EVENT_LOGGER_NAME
                 from autogen_core.logging import LLMCallEvent
 
+                response = {"content": "Hello, world!"}
+                messages = [{"role": "user", "content": "Hello, world!"}]
                 logger = logging.getLogger(EVENT_LOGGER_NAME)
-                logger.info(LLMCallEvent(prompt_tokens=10, completion_tokens=20))
+                logger.info(LLMCallEvent(prompt_tokens=10, completion_tokens=20, response=response, messages=messages))
 
         """
         self.kwargs = kwargs
@@ -61,6 +64,99 @@ def __str__(self) -> str:
         return json.dumps(self.kwargs)
 
 
+class LLMStreamStartEvent:
+    """To be used by model clients to log the start of a stream.
+
+    Args:
+        messages (List[Dict[str, Any]]): The messages used in the call. Must be json serializable.
+
+    Example:
+
+        .. code-block:: python
+
+            import logging
+            from autogen_core import EVENT_LOGGER_NAME
+            from autogen_core.logging import LLMStreamStartEvent
+
+            messages = [{"role": "user", "content": "Hello, world!"}]
+            logger = logging.getLogger(EVENT_LOGGER_NAME)
+            logger.info(LLMStreamStartEvent(messages=messages))
+
+    """
+
+    def __init__(
+        self,
+        *,
+        messages: List[Dict[str, Any]],
+        **kwargs: Any,
+    ) -> None:
+        self.kwargs = kwargs
+        self.kwargs["type"] = "LLMStreamStart"
+        self.kwargs["messages"] = messages
+        try:
+            agent_id = MessageHandlerContext.agent_id()
+        except RuntimeError:
+            agent_id = None
+        self.kwargs["agent_id"] = None if agent_id is None else str(agent_id)
+
+    # This must output the event in a json serializable format
+    def __str__(self) -> str:
+        return json.dumps(self.kwargs)
+
+
+class LLMStreamEndEvent:
+    def __init__(
+        self,
+        *,
+        response: Dict[str, Any],
+        prompt_tokens: int,
+        completion_tokens: int,
+        **kwargs: Any,
+    ) -> None:
+        """To be used by model   to log the call to the LLM.
+
+        Args:
+            response (Dict[str, Any]): The response of the call. Must be json serializable.
+            prompt_tokens (int): Number of tokens used in the prompt.
+            completion_tokens (int): Number of tokens used in the completion.
+
+        Example:
+
+            .. code-block:: python
+
+                import logging
+                from autogen_core import EVENT_LOGGER_NAME
+                from autogen_core.logging import LLMStreamEndEvent
+
+                response = {"content": "Hello, world!"}
+                logger = logging.getLogger(EVENT_LOGGER_NAME)
+                logger.info(LLMStreamEndEvent(prompt_tokens=10, completion_tokens=20, response=response))
+
+        """
+        self.kwargs = kwargs
+        self.kwargs["type"] = "LLMStreamEnd"
+        self.kwargs["response"] = response
+        self.kwargs["prompt_tokens"] = prompt_tokens
+        self.kwargs["completion_tokens"] = completion_tokens
+        try:
+            agent_id = MessageHandlerContext.agent_id()
+        except RuntimeError:
+            agent_id = None
+        self.kwargs["agent_id"] = None if agent_id is None else str(agent_id)
+
+    @property
+    def prompt_tokens(self) -> int:
+        return cast(int, self.kwargs["prompt_tokens"])
+
+    @property
+    def completion_tokens(self) -> int:
+        return cast(int, self.kwargs["completion_tokens"])
+
+    # This must output the event in a json serializable format
+    def __str__(self) -> str:
+        return json.dumps(self.kwargs)
+
+
 class ToolCallEvent:
     def __init__(
         self,
diff --git a/python/packages/autogen-ext/src/autogen_ext/models/anthropic/_anthropic_client.py b/python/packages/autogen-ext/src/autogen_ext/models/anthropic/_anthropic_client.py
index 7da8508c764c..2e589bb3658c 100644
--- a/python/packages/autogen-ext/src/autogen_ext/models/anthropic/_anthropic_client.py
+++ b/python/packages/autogen-ext/src/autogen_ext/models/anthropic/_anthropic_client.py
@@ -45,7 +45,7 @@
     FunctionCall,
     Image,
 )
-from autogen_core.logging import LLMCallEvent
+from autogen_core.logging import LLMCallEvent, LLMStreamEndEvent, LLMStreamStartEvent
 from autogen_core.models import (
     AssistantMessage,
     ChatCompletionClient,
@@ -665,8 +665,18 @@ async def create_stream(
         output_tokens: int = 0
         stop_reason: Optional[str] = None
 
+        first_chunk = True
+
         # Process the stream
         async for chunk in stream:
+            if first_chunk:
+                first_chunk = False
+                # Emit the start event.
+                logger.info(
+                    LLMStreamStartEvent(
+                        messages=cast(List[Dict[str, Any]], anthropic_messages),
+                    )
+                )
             # Handle different event types
             if chunk.type == "content_block_start":
                 if chunk.content_block.type == "tool_use":
@@ -761,6 +771,15 @@ async def create_stream(
             thought=thought,
         )
 
+        # Emit the end event.
+        logger.info(
+            LLMStreamEndEvent(
+                response=result.model_dump(),
+                prompt_tokens=usage.prompt_tokens,
+                completion_tokens=usage.completion_tokens,
+            )
+        )
+
         # Update usage statistics
         self._total_usage = _add_usage(self._total_usage, usage)
         self._actual_usage = _add_usage(self._actual_usage, usage)
diff --git a/python/packages/autogen-ext/src/autogen_ext/models/azure/_azure_ai_client.py b/python/packages/autogen-ext/src/autogen_ext/models/azure/_azure_ai_client.py
index b6257bd9a8ed..93a8c2c507aa 100644
--- a/python/packages/autogen-ext/src/autogen_ext/models/azure/_azure_ai_client.py
+++ b/python/packages/autogen-ext/src/autogen_ext/models/azure/_azure_ai_client.py
@@ -6,7 +6,7 @@
 from typing import Any, Dict, List, Mapping, Optional, Sequence, cast
 
 from autogen_core import EVENT_LOGGER_NAME, CancellationToken, FunctionCall, Image
-from autogen_core.logging import LLMCallEvent
+from autogen_core.logging import LLMCallEvent, LLMStreamEndEvent, LLMStreamStartEvent
 from autogen_core.models import (
     AssistantMessage,
     ChatCompletionClient,
@@ -430,7 +430,16 @@ async def create_stream(
         completion_tokens = 0
         chunk: Optional[StreamingChatCompletionsUpdate] = None
         choice: Optional[StreamingChatChoiceUpdate] = None
+        first_chunk = True
         async for chunk in await task:  # type: ignore
+            if first_chunk:
+                first_chunk = False
+                # Emit the start event.
+                logger.info(
+                    LLMStreamStartEvent(
+                        messages=[m.as_dict() for m in azure_messages],
+                    )
+                )
             assert isinstance(chunk, StreamingChatCompletionsUpdate)
             choice = chunk.choices[0] if len(chunk.choices) > 0 else None
             if choice and choice.finish_reason is not None:
@@ -499,6 +508,15 @@ async def create_stream(
             thought=thought,
         )
 
+        # Log the end of the stream.
+        logger.info(
+            LLMStreamEndEvent(
+                response=result.model_dump(),
+                prompt_tokens=usage.prompt_tokens,
+                completion_tokens=usage.completion_tokens,
+            )
+        )
+
         self.add_usage(usage)
 
         yield result
diff --git a/python/packages/autogen-ext/src/autogen_ext/models/ollama/_ollama_client.py b/python/packages/autogen-ext/src/autogen_ext/models/ollama/_ollama_client.py
index f817a17ae6f6..0842bdf0b68d 100644
--- a/python/packages/autogen-ext/src/autogen_ext/models/ollama/_ollama_client.py
+++ b/python/packages/autogen-ext/src/autogen_ext/models/ollama/_ollama_client.py
@@ -27,7 +27,7 @@
     FunctionCall,
     Image,
 )
-from autogen_core.logging import LLMCallEvent
+from autogen_core.logging import LLMCallEvent, LLMStreamEndEvent, LLMStreamStartEvent
 from autogen_core.models import (
     AssistantMessage,
     ChatCompletionClient,
@@ -676,7 +676,7 @@ async def create_stream(
         content_chunks: List[str] = []
         full_tool_calls: List[FunctionCall] = []
         completion_tokens = 0
-
+        first_chunk = True
         while True:
             try:
                 chunk_future = asyncio.ensure_future(anext(stream))
@@ -684,6 +684,14 @@ async def create_stream(
                     cancellation_token.link_future(chunk_future)
                 chunk = await chunk_future
 
+                if first_chunk:
+                    first_chunk = False
+                    # Emit the start event.
+                    logger.info(
+                        LLMStreamStartEvent(
+                            messages=cast(List[Dict[str, Any]], ollama_messages),
+                        )
+                    )
                 # set the stop_reason for the usage chunk to the prior stop_reason
                 stop_reason = chunk.done_reason if chunk.done and stop_reason is None else stop_reason
                 # First try get content
@@ -759,6 +767,15 @@ async def create_stream(
             logprobs=None,
         )
 
+        # Emit the end event.
+        logger.info(
+            LLMStreamEndEvent(
+                response=result.model_dump(),
+                prompt_tokens=usage.prompt_tokens,
+                completion_tokens=usage.completion_tokens,
+            )
+        )
+
         self._total_usage = _add_usage(self._total_usage, usage)
         self._actual_usage = _add_usage(self._actual_usage, usage)
 
diff --git a/python/packages/autogen-ext/src/autogen_ext/models/openai/_openai_client.py b/python/packages/autogen-ext/src/autogen_ext/models/openai/_openai_client.py
index 05557e19f3e2..430d78af105e 100644
--- a/python/packages/autogen-ext/src/autogen_ext/models/openai/_openai_client.py
+++ b/python/packages/autogen-ext/src/autogen_ext/models/openai/_openai_client.py
@@ -30,7 +30,7 @@
     FunctionCall,
     Image,
 )
-from autogen_core.logging import LLMCallEvent
+from autogen_core.logging import LLMCallEvent, LLMStreamEndEvent, LLMStreamStartEvent
 from autogen_core.models import (
     AssistantMessage,
     ChatCompletionClient,
@@ -748,8 +748,18 @@ async def create_stream(
         empty_chunk_warning_threshold: int = 10
         empty_chunk_count = 0
 
+        first_chunk = True
+
         # Process the stream of chunks.
         async for chunk in chunks:
+            if first_chunk:
+                first_chunk = False
+                # Emit the start event.
+                logger.info(
+                    LLMStreamStartEvent(
+                        messages=cast(List[Dict[str, Any]], oai_messages),
+                    )
+                )
             # Empty chunks has been observed when the endpoint is under heavy load.
             #  https://github.com/microsoft/autogen/issues/4213
             if len(chunk.choices) == 0:
@@ -869,6 +879,15 @@ async def create_stream(
             thought=thought,
         )
 
+        # Log the end of the stream.
+        logger.info(
+            LLMStreamEndEvent(
+                response=result.model_dump(),
+                prompt_tokens=usage.prompt_tokens,
+                completion_tokens=usage.completion_tokens,
+            )
+        )
+
         # Update the total usage.
         self._total_usage = _add_usage(self._total_usage, usage)
         self._actual_usage = _add_usage(self._actual_usage, usage)
diff --git a/python/packages/autogen-ext/src/autogen_ext/models/semantic_kernel/_sk_chat_completion_adapter.py b/python/packages/autogen-ext/src/autogen_ext/models/semantic_kernel/_sk_chat_completion_adapter.py
index 7df5dd59be7e..4c9f7bf5ac55 100644
--- a/python/packages/autogen-ext/src/autogen_ext/models/semantic_kernel/_sk_chat_completion_adapter.py
+++ b/python/packages/autogen-ext/src/autogen_ext/models/semantic_kernel/_sk_chat_completion_adapter.py
@@ -5,7 +5,7 @@
 
 from autogen_core import EVENT_LOGGER_NAME, FunctionCall
 from autogen_core._cancellation_token import CancellationToken
-from autogen_core.logging import LLMCallEvent
+from autogen_core.logging import LLMCallEvent, LLMStreamEndEvent, LLMStreamStartEvent
 from autogen_core.models import (
     ChatCompletionClient,
     CreateResult,
@@ -577,9 +577,19 @@ async def create_stream(
         # accumulating chunk arguments for that call if new items have id=None
         last_function_call_id: Optional[str] = None
 
+        first_chunk = True
+
         async for streaming_messages in self._sk_client.get_streaming_chat_message_contents(
             chat_history, settings=settings, kernel=kernel
         ):
+            if first_chunk:
+                first_chunk = False
+                # Emit the start event.
+                logger.info(
+                    LLMStreamStartEvent(
+                        messages=[msg.model_dump() for msg in chat_history],
+                    )
+                )
             for msg in streaming_messages:
                 # Track token usage
                 if msg.metadata and "usage" in msg.metadata:
@@ -659,7 +669,7 @@ async def create_stream(
         if isinstance(accumulated_text, str) and self._model_info["family"] == ModelFamily.R1:
             thought, accumulated_text = parse_r1_content(accumulated_text)
 
-        yield CreateResult(
+        result = CreateResult(
             content=accumulated_text,
             finish_reason="stop",
             usage=RequestUsage(prompt_tokens=prompt_tokens, completion_tokens=completion_tokens),
@@ -667,6 +677,17 @@ async def create_stream(
             thought=thought,
         )
 
+        # Emit the end event.
+        logger.info(
+            LLMStreamEndEvent(
+                response=result.model_dump(),
+                prompt_tokens=prompt_tokens,
+                completion_tokens=completion_tokens,
+            )
+        )
+
+        yield result
+
     async def close(self) -> None:
         pass  # No explicit close method in SK client?
 
diff --git a/python/packages/autogen-ext/tests/models/test_anthropic_model_client.py b/python/packages/autogen-ext/tests/models/test_anthropic_model_client.py
index 74092b33f5a4..2ab8b9f40f20 100644
--- a/python/packages/autogen-ext/tests/models/test_anthropic_model_client.py
+++ b/python/packages/autogen-ext/tests/models/test_anthropic_model_client.py
@@ -73,7 +73,7 @@ async def test_anthropic_basic_completion(caplog: pytest.LogCaptureFixture) -> N
 
 
 @pytest.mark.asyncio
-async def test_anthropic_streaming() -> None:
+async def test_anthropic_streaming(caplog: pytest.LogCaptureFixture) -> None:
     """Test streaming capabilities with Claude."""
     api_key = os.getenv("ANTHROPIC_API_KEY")
     if not api_key:
@@ -86,20 +86,28 @@ async def test_anthropic_streaming() -> None:
 
     # Test streaming completion
     chunks: List[str | CreateResult] = []
-    async for chunk in client.create_stream(
-        messages=[
-            UserMessage(content="Count from 1 to 5. Each number on its own line.", source="user"),
-        ]
-    ):
-        chunks.append(chunk)
-
-    # Verify we got multiple chunks
-    assert len(chunks) > 1
-
-    # Check final result
-    final_result = chunks[-1]
-    assert isinstance(final_result, CreateResult)
-    assert final_result.finish_reason == "stop"
+    prompt = "Count from 1 to 5. Each number on its own line."
+    with caplog.at_level(logging.INFO):
+        async for chunk in client.create_stream(
+            messages=[
+                UserMessage(content=prompt, source="user"),
+            ]
+        ):
+            chunks.append(chunk)
+        # Verify we got multiple chunks
+        assert len(chunks) > 1
+
+        # Check final result
+        final_result = chunks[-1]
+        assert isinstance(final_result, CreateResult)
+        assert final_result.finish_reason == "stop"
+
+        assert "LLMStreamStart" in caplog.text
+        assert "LLMStreamEnd" in caplog.text
+        assert isinstance(final_result.content, str)
+        for i in range(1, 6):
+            assert str(i) in caplog.text
+        assert prompt in caplog.text
 
     # Check content contains numbers 1-5
     assert isinstance(final_result.content, str)
diff --git a/python/packages/autogen-ext/tests/models/test_azure_ai_model_client.py b/python/packages/autogen-ext/tests/models/test_azure_ai_model_client.py
index 986ae27bacec..43401134378d 100644
--- a/python/packages/autogen-ext/tests/models/test_azure_ai_model_client.py
+++ b/python/packages/autogen-ext/tests/models/test_azure_ai_model_client.py
@@ -181,10 +181,21 @@ async def test_azure_ai_chat_completion_client_create(
 
 
 @pytest.mark.asyncio
-async def test_azure_ai_chat_completion_client_create_stream(azure_client: AzureAIChatCompletionClient) -> None:
-    chunks: List[str | CreateResult] = []
-    async for chunk in azure_client.create_stream(messages=[UserMessage(content="Hello", source="user")]):
-        chunks.append(chunk)
+async def test_azure_ai_chat_completion_client_create_stream(
+    azure_client: AzureAIChatCompletionClient, caplog: pytest.LogCaptureFixture
+) -> None:
+    with caplog.at_level(logging.INFO):
+        chunks: List[str | CreateResult] = []
+        async for chunk in azure_client.create_stream(messages=[UserMessage(content="Hello", source="user")]):
+            chunks.append(chunk)
+
+        assert "LLMStreamStart" in caplog.text
+        assert "LLMStreamEnd" in caplog.text
+
+        final_result: str | CreateResult = chunks[-1]
+        assert isinstance(final_result, CreateResult)
+        assert isinstance(final_result.content, str)
+        assert final_result.content in caplog.text
 
     assert chunks[0] == "Hello"
     assert chunks[1] == " Another Hello"
diff --git a/python/packages/autogen-ext/tests/models/test_openai_model_client.py b/python/packages/autogen-ext/tests/models/test_openai_model_client.py
index 2f51b6e2093d..5cef46edb97f 100644
--- a/python/packages/autogen-ext/tests/models/test_openai_model_client.py
+++ b/python/packages/autogen-ext/tests/models/test_openai_model_client.py
@@ -235,22 +235,31 @@ async def test_openai_chat_completion_client_create(
 
 
 @pytest.mark.asyncio
-async def test_openai_chat_completion_client_create_stream_with_usage(monkeypatch: pytest.MonkeyPatch) -> None:
+async def test_openai_chat_completion_client_create_stream_with_usage(
+    monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
+) -> None:
     monkeypatch.setattr(AsyncCompletions, "create", _mock_create)
     client = OpenAIChatCompletionClient(model="gpt-4o", api_key="api_key")
     chunks: List[str | CreateResult] = []
-    async for chunk in client.create_stream(
-        messages=[UserMessage(content="Hello", source="user")],
-        # include_usage not the default of the OPENAI API and must be explicitly set
-        extra_create_args={"stream_options": {"include_usage": True}},
-    ):
-        chunks.append(chunk)
-    assert chunks[0] == "Hello"
-    assert chunks[1] == " Another Hello"
-    assert chunks[2] == " Yet Another Hello"
-    assert isinstance(chunks[-1], CreateResult)
-    assert chunks[-1].content == "Hello Another Hello Yet Another Hello"
-    assert chunks[-1].usage == RequestUsage(prompt_tokens=3, completion_tokens=3)
+    with caplog.at_level(logging.INFO):
+        async for chunk in client.create_stream(
+            messages=[UserMessage(content="Hello", source="user")],
+            # include_usage not the default of the OPENAI API and must be explicitly set
+            extra_create_args={"stream_options": {"include_usage": True}},
+        ):
+            chunks.append(chunk)
+
+        assert "LLMStreamStart" in caplog.text
+        assert "LLMStreamEnd" in caplog.text
+
+        assert chunks[0] == "Hello"
+        assert chunks[1] == " Another Hello"
+        assert chunks[2] == " Yet Another Hello"
+        assert isinstance(chunks[-1], CreateResult)
+        assert isinstance(chunks[-1].content, str)
+        assert chunks[-1].content == "Hello Another Hello Yet Another Hello"
+        assert chunks[-1].content in caplog.text
+        assert chunks[-1].usage == RequestUsage(prompt_tokens=3, completion_tokens=3)
 
 
 @pytest.mark.asyncio
diff --git a/python/packages/autogen-ext/tests/models/test_sk_chat_completion_adapter.py b/python/packages/autogen-ext/tests/models/test_sk_chat_completion_adapter.py
index 3ec2bcfaf400..bd30aad0f06a 100644
--- a/python/packages/autogen-ext/tests/models/test_sk_chat_completion_adapter.py
+++ b/python/packages/autogen-ext/tests/models/test_sk_chat_completion_adapter.py
@@ -396,7 +396,9 @@ async def test_sk_chat_completion_stream_with_tools(sk_client: AzureChatCompleti
 
 
 @pytest.mark.asyncio
-async def test_sk_chat_completion_stream_without_tools(sk_client: AzureChatCompletion) -> None:
+async def test_sk_chat_completion_stream_without_tools(
+    sk_client: AzureChatCompletion, caplog: pytest.LogCaptureFixture
+) -> None:
     # Create adapter and kernel
     adapter = SKChatCompletionAdapter(sk_client)
     kernel = Kernel(memory=NullMemory())
@@ -409,23 +411,28 @@ async def test_sk_chat_completion_stream_without_tools(sk_client: AzureChatCompl
 
     # Call create_stream without tools
     response_chunks: list[CreateResult | str] = []
-    async for chunk in adapter.create_stream(messages=messages, extra_create_args={"kernel": kernel}):
-        response_chunks.append(chunk)
+    with caplog.at_level(logging.INFO):
+        async for chunk in adapter.create_stream(messages=messages, extra_create_args={"kernel": kernel}):
+            response_chunks.append(chunk)
 
-    # Verify response
-    assert len(response_chunks) > 0
-    # All chunks except last should be strings
-    for chunk in response_chunks[:-1]:
-        assert isinstance(chunk, str)
+        assert "LLMStreamStart" in caplog.text
+        assert "LLMStreamEnd" in caplog.text
 
-    # Final chunk should be CreateResult
-    final_chunk = response_chunks[-1]
-    assert isinstance(final_chunk, CreateResult)
-    assert isinstance(final_chunk.content, str)
-    assert final_chunk.finish_reason == "stop"
-    assert final_chunk.usage.prompt_tokens >= 0
-    assert final_chunk.usage.completion_tokens >= 0
-    assert not final_chunk.cached
+        # Verify response
+        assert len(response_chunks) > 0
+        # All chunks except last should be strings
+        for chunk in response_chunks[:-1]:
+            assert isinstance(chunk, str)
+
+        # Final chunk should be CreateResult
+        final_chunk = response_chunks[-1]
+        assert isinstance(final_chunk, CreateResult)
+        assert isinstance(final_chunk.content, str)
+        assert final_chunk.finish_reason == "stop"
+        assert final_chunk.usage.prompt_tokens >= 0
+        assert final_chunk.usage.completion_tokens >= 0
+        assert not final_chunk.cached
+        assert final_chunk.content in caplog.text
 
 
 @pytest.mark.asyncio


Reviews:
user: login: ekzhu
body: Thanks! Shall we add checks in unit tests using `caplog` fixture? See existing examples for checking `LLMCallEvent` in the unit tests. We just need to verify in one of the tests that triggers the `create_stream` method for each client class. 
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T20:52:49Z
user: login: ekzhu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T23:56:01Z
user: login: EItanya
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T01:53:01Z
user: login: ekzhu
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T05:18:16Z
user: login: ekzhu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T21:54:40Z

Review comments:
diff_hunk: @@ -61,6 +61,95 @@ def __str__(self) -> str:
         return json.dumps(self.kwargs)
 
 
+class LLMStreamStartEvent:
+    """To be used by model clients to log the start of a stream.
+
+    Args:
+        messages (List[Dict[str, Any]]): The messages used in the call. Must be json serializable.
+
+    Example:
+
+        .. code-block:: python
+
+            from autogen_core import EVENT_LOGGER_NAME
+            from autogen_core.logging import
+
+            logger = logging.getLogger(EVENT_LOGGER_NAME)
+            logger.info(LLMStreamStartEvent())
path: python/packages/autogen-core/src/autogen_core/logging.py
user: login: ekzhu
body: This sample code is also typed checked so we need to ensure the code can run. 
created_at: 2025-03-10T23:56:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -61,6 +61,95 @@ def __str__(self) -> str:
         return json.dumps(self.kwargs)
 
 
+class LLMStreamStartEvent:
+    """To be used by model clients to log the start of a stream.
+
+    Args:
+        messages (List[Dict[str, Any]]): The messages used in the call. Must be json serializable.
+
+    Example:
+
+        .. code-block:: python
+
+            from autogen_core import EVENT_LOGGER_NAME
+            from autogen_core.logging import
+
+            logger = logging.getLogger(EVENT_LOGGER_NAME)
+            logger.info(LLMStreamStartEvent())
path: python/packages/autogen-core/src/autogen_core/logging.py
user: login: EItanya
body: Which specific piece are you referring to? I added the `import` in, and also fixed the other examples to have all required args.
created_at: 2025-03-11T01:53:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -61,6 +61,95 @@ def __str__(self) -> str:
         return json.dumps(self.kwargs)
 
 
+class LLMStreamStartEvent:
+    """To be used by model clients to log the start of a stream.
+
+    Args:
+        messages (List[Dict[str, Any]]): The messages used in the call. Must be json serializable.
+
+    Example:
+
+        .. code-block:: python
+
+            from autogen_core import EVENT_LOGGER_NAME
+            from autogen_core.logging import
+
+            logger = logging.getLogger(EVENT_LOGGER_NAME)
+            logger.info(LLMStreamStartEvent())
path: python/packages/autogen-core/src/autogen_core/logging.py
user: login: ekzhu
body: > I added the import in, and also fixed the other examples to have all required args.

That's all I was referring to. I meant we have CI action that runs type checks on the code blocks in API doc. 
created_at: 2025-03-11T21:54:40Z
author_association: CONTRIBUTOR

PR comments:
user: login: EItanya
created_at: 2025-03-10T22:19:53Z
body: > Thanks! Shall we add checks in unit tests using `caplog` fixture? See existing examples for checking `LLMCallEvent` in the unit tests. We just need to verify in one of the tests that triggers the `create_stream` method for each client class.

Done
author_association: CONTRIBUTOR
user: login: ekzhu
created_at: 2025-03-11T05:15:09Z
body: Fantastic. We should also trace the LLM call and stream spans.



Created https://github.com/microsoft/autogen/issues/5895
author_association: CONTRIBUTOR
user: login: EItanya
created_at: 2025-03-11T12:40:07Z
body: > Fantastic. We should also trace the LLM call and stream spans.
> 
> 
> 
> Created https://github.com/microsoft/autogen/issues/5895

So I thought about that and I'm actually not sure. Instrumentation libraries already exist for many of these and they allow the user to more easily control it. 

For example OpenAI: https://pypi.org/project/opentelemetry-instrumentation-openai/

I created a demo using the instrumentation for OpenAI and HTTPX and it worked super well!

In the long term it might actually make sense to take this approach with Autogen as well, but that's a whole separate discussion.

I can of course also add this context on the issue you created.

In terms of telemetry I'm actually fairly interested in metrics as well. Does Autogen emit any metrics, or is there any plan to do so?
author_association: CONTRIBUTOR
user: login: codecov[bot]
created_at: 2025-03-11T22:03:05Z
body: ## [Codecov](https://app.codecov.io/gh/microsoft/autogen/pull/5890?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=microsoft) Report
Attention: Patch coverage is `50.00000%` with `32 lines` in your changes missing coverage. Please review.
> Project coverage is 75.51%. Comparing base [(`2cc8c73`)](https://app.codecov.io/gh/microsoft/autogen/commit/2cc8c73d3b83911892903001d64425298cc94da4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=microsoft) to head [(`5e99ef7`)](https://app.codecov.io/gh/microsoft/autogen/commit/5e99ef79887a609e21ef3219ff76c17589f1ea8a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=microsoft).
> Report is 1 commits behind head on main.

| [Files with missing lines](https://app.codecov.io/gh/microsoft/autogen/pull/5890?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=microsoft) | Patch % | Lines |
|---|---|---|
| [.../packages/autogen-core/src/autogen\_core/logging.py](https://app.codecov.io/gh/microsoft/autogen/pull/5890?src=pr&el=tree&filepath=python%2Fpackages%2Fautogen-core%2Fsrc%2Fautogen_core%2Flogging.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=microsoft#diff-cHl0aG9uL3BhY2thZ2VzL2F1dG9nZW4tY29yZS9zcmMvYXV0b2dlbl9jb3JlL2xvZ2dpbmcucHk=) | 31.25% | [22 Missing :warning: ](https://app.codecov.io/gh/microsoft/autogen/pull/5890?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=microsoft) |
| [.../autogen\_ext/models/anthropic/\_anthropic\_client.py](https://app.codecov.io/gh/microsoft/autogen/pull/5890?src=pr&el=tree&filepath=python%2Fpackages%2Fautogen-ext%2Fsrc%2Fautogen_ext%2Fmodels%2Fanthropic%2F_anthropic_client.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=microsoft#diff-cHl0aG9uL3BhY2thZ2VzL2F1dG9nZW4tZXh0L3NyYy9hdXRvZ2VuX2V4dC9tb2RlbHMvYW50aHJvcGljL19hbnRocm9waWNfY2xpZW50LnB5) | 16.66% | [5 Missing :warning: ](https://app.codecov.io/gh/microsoft/autogen/pull/5890?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=microsoft) |
| [...xt/src/autogen\_ext/models/ollama/\_ollama\_client.py](https://app.codecov.io/gh/microsoft/autogen/pull/5890?src=pr&el=tree&filepath=python%2Fpackages%2Fautogen-ext%2Fsrc%2Fautogen_ext%2Fmodels%2Follama%2F_ollama_client.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=microsoft#diff-cHl0aG9uL3BhY2thZ2VzL2F1dG9nZW4tZXh0L3NyYy9hdXRvZ2VuX2V4dC9tb2RlbHMvb2xsYW1hL19vbGxhbWFfY2xpZW50LnB5) | 16.66% | [5 Missing :warning: ](https://app.codecov.io/gh/microsoft/autogen/pull/5890?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=microsoft) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #5890      +/-   ##
==========================================
- Coverage   75.65%   75.51%   -0.14%     
==========================================
  Files         191      191              
  Lines       12974    13032      +58     
==========================================
+ Hits         9815     9841      +26     
- Misses       3159     3191      +32     
```

| [Flag](https://app.codecov.io/gh/microsoft/autogen/pull/5890/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=microsoft) | Coverage Δ | |
|---|---|---|
| [unittests](https://app.codecov.io/gh/microsoft/autogen/pull/5890/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=microsoft) | `75.51% <50.00%> (-0.14%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=microsoft#carryforward-flags-in-the-pull-request-comment) to find out more.

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/microsoft/autogen/pull/5890?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=microsoft).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=microsoft).

<details><summary>🚀 New features to boost your workflow: </summary>

- ❄ [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- 📦 [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
author_association: NONE