Repository: microsoft/autogen
Pull request: feat: add structured output to model clients
Resolves #5933

1. Add a new `type[BaseModel]` option for `json_output`  in `ChatCompletionClient.create` and `ChatCompletionClient.create_stream`, using them for the output type of structured output.
2. Add support for this new option for `OpenAIChatCompletionClient`, `AzureOpenAIChatCompletionClient`, `OllamaChatCompletionClient`, `LlamaCppChatCompletionClient`, and `ChatCompletionClientCache`. 
3. Add unit tests for the supported clients.
4. Add `structured_output` field to `ModelFamily`, and update everywhere. 

For future:
1. Add support for structured output for `AzureAIChatCompletionClient`.
2. Add new field `parsed_content` to `CreateResult` to carry the parsed output type. 

Problem statement:
Add structured output to model client API
### Confirmation

- [x] I confirm that I am a maintainer and so can use this template. If I am not, I understand this issue will be closed and I will be asked to use a different template.

### Issue body

1. `output_type` to `ChatCompletionClient` APIs.
2. Add `structured_output` as a new `ModelInfo` field.
3. Add `structured_output` field to existing built-in model infos.
4. Use the new `output_type` argument in built-in model clients.

Hints from issue discussion:

Commit message: add structured output to model clients

Patch to review:
diff --git a/python/packages/autogen-core/src/autogen_core/models/_model_client.py b/python/packages/autogen-core/src/autogen_core/models/_model_client.py
index 84f008d7990e..9900c8b150b8 100644
--- a/python/packages/autogen-core/src/autogen_core/models/_model_client.py
+++ b/python/packages/autogen-core/src/autogen_core/models/_model_client.py
@@ -107,6 +107,8 @@ class ModelInfo(TypedDict, total=False):
     """True if the model supports json output, otherwise False. Note: this is different to structured json."""
     family: Required[ModelFamily.ANY | str]
     """Model family should be one of the constants from :py:class:`ModelFamily` or a string representing an unknown model family."""
+    structured_output: Required[bool]
+    """True if the model supports structured output, otherwise False. This is different to json_output."""
 
 
 def validate_model_info(model_info: ModelInfo) -> None:
@@ -122,6 +124,15 @@ def validate_model_info(model_info: ModelInfo) -> None:
                 f"Missing required field '{field}' in ModelInfo. "
                 "Starting in v0.4.7, the required fields are enforced."
             )
+    new_required_fields = ["structured_output"]
+    for field in new_required_fields:
+        if field not in model_info:
+            warnings.warn(
+                f"Missing required field '{field}' in ModelInfo. "
+                "This field will be required in a future version of AutoGen.",
+                UserWarning,
+                stacklevel=2,
+            )
 
 
 class ChatCompletionClient(ComponentBase[BaseModel], ABC):
@@ -135,6 +146,7 @@ async def create(
         # None means do not override the default
         # A value means to override the client default - often specified in the constructor
         json_output: Optional[bool] = None,
+        output_type: Optional[type[BaseModel]] = None,
         extra_create_args: Mapping[str, Any] = {},
         cancellation_token: Optional[CancellationToken] = None,
     ) -> CreateResult: ...
@@ -148,6 +160,7 @@ def create_stream(
         # None means do not override the default
         # A value means to override the client default - often specified in the constructor
         json_output: Optional[bool] = None,
+        output_type: Optional[type[BaseModel]] = None,
         extra_create_args: Mapping[str, Any] = {},
         cancellation_token: Optional[CancellationToken] = None,
     ) -> AsyncGenerator[Union[str, CreateResult], None]: ...
diff --git a/python/packages/autogen-ext/src/autogen_ext/experimental/task_centric_memory/utils/chat_completion_client_recorder.py b/python/packages/autogen-ext/src/autogen_ext/experimental/task_centric_memory/utils/chat_completion_client_recorder.py
index 34dd1616de72..6ba2d57d03e7 100644
--- a/python/packages/autogen-ext/src/autogen_ext/experimental/task_centric_memory/utils/chat_completion_client_recorder.py
+++ b/python/packages/autogen-ext/src/autogen_ext/experimental/task_centric_memory/utils/chat_completion_client_recorder.py
@@ -13,6 +13,7 @@
     RequestUsage,
 )
 from autogen_core.tools import Tool, ToolSchema
+from pydantic import BaseModel
 
 from .page_logger import PageLogger
 
@@ -88,6 +89,7 @@ async def create(
         *,
         tools: Sequence[Tool | ToolSchema] = [],
         json_output: Optional[bool] = None,
+        output_type: Optional[type[BaseModel]] = None,
         extra_create_args: Mapping[str, Any] = {},
         cancellation_token: Optional[CancellationToken] = None,
     ) -> CreateResult:
@@ -155,6 +157,7 @@ def create_stream(
         *,
         tools: Sequence[Tool | ToolSchema] = [],
         json_output: Optional[bool] = None,
+        output_type: Optional[type[BaseModel]] = None,
         extra_create_args: Mapping[str, Any] = {},
         cancellation_token: Optional[CancellationToken] = None,
     ) -> AsyncGenerator[Union[str, CreateResult], None]:
diff --git a/python/packages/autogen-ext/src/autogen_ext/models/anthropic/_anthropic_client.py b/python/packages/autogen-ext/src/autogen_ext/models/anthropic/_anthropic_client.py
index 2e589bb3658c..7d62d0227244 100644
--- a/python/packages/autogen-ext/src/autogen_ext/models/anthropic/_anthropic_client.py
+++ b/python/packages/autogen-ext/src/autogen_ext/models/anthropic/_anthropic_client.py
@@ -61,6 +61,7 @@
     validate_model_info,
 )
 from autogen_core.tools import Tool, ToolSchema
+from pydantic import BaseModel
 from typing_extensions import Self, Unpack
 
 from . import _model_info
@@ -413,6 +414,7 @@ async def create(
         *,
         tools: Sequence[Tool | ToolSchema] = [],
         json_output: Optional[bool] = None,
+        output_type: Optional[type[BaseModel]] = None,
         extra_create_args: Mapping[str, Any] = {},
         cancellation_token: Optional[CancellationToken] = None,
     ) -> CreateResult:
@@ -435,6 +437,9 @@ async def create(
             if json_output is True:
                 create_args["response_format"] = {"type": "json_object"}
 
+        if output_type is not None:
+            raise ValueError("output_type is currently not supported for Anthropic models")
+
         # Process system message separately
         system_message = None
         anthropic_messages: List[MessageParam] = []
@@ -568,6 +573,7 @@ async def create_stream(
         *,
         tools: Sequence[Tool | ToolSchema] = [],
         json_output: Optional[bool] = None,
+        output_type: Optional[type[BaseModel]] = None,
         extra_create_args: Mapping[str, Any] = {},
         cancellation_token: Optional[CancellationToken] = None,
         max_consecutive_empty_chunk_tolerance: int = 0,
@@ -594,6 +600,9 @@ async def create_stream(
             if json_output is True:
                 create_args["response_format"] = {"type": "json_object"}
 
+        if output_type is not None:
+            raise ValueError("output_type is currently not supported for Anthropic models")
+
         # Process system message separately
         system_message = None
         anthropic_messages: List[MessageParam] = []
diff --git a/python/packages/autogen-ext/src/autogen_ext/models/anthropic/_model_info.py b/python/packages/autogen-ext/src/autogen_ext/models/anthropic/_model_info.py
index a945b7c9f978..059af5097632 100644
--- a/python/packages/autogen-ext/src/autogen_ext/models/anthropic/_model_info.py
+++ b/python/packages/autogen-ext/src/autogen_ext/models/anthropic/_model_info.py
@@ -12,6 +12,7 @@
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.CLAUDE_3_7_SONNET,
+        "structured_output": False,
     },
     # Claude 3.7 Sonnet latest alias
     "claude-3-7-sonnet-latest": {
@@ -19,6 +20,7 @@
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.CLAUDE_3_7_SONNET,
+        "structured_output": False,
     },
     # Claude 3 Opus (most powerful)
     "claude-3-opus-20240229": {
@@ -26,6 +28,7 @@
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.CLAUDE_3_5_SONNET,
+        "structured_output": False,
     },
     # Claude 3 Sonnet (balanced)
     "claude-3-sonnet-20240229": {
@@ -33,6 +36,7 @@
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.CLAUDE_3_5_SONNET,
+        "structured_output": False,
     },
     # Claude 3 Haiku (fastest)
     "claude-3-haiku-20240307": {
@@ -40,6 +44,7 @@
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.CLAUDE_3_5_SONNET,
+        "structured_output": False,
     },
     # Claude 3.5 Sonnet
     "claude-3-5-sonnet-20240620": {
@@ -47,6 +52,7 @@
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.CLAUDE_3_5_SONNET,
+        "structured_output": False,
     },
     # Claude Instant v1 (legacy)
     "claude-instant-1.2": {
@@ -54,6 +60,7 @@
         "function_calling": False,
         "json_output": True,
         "family": ModelFamily.CLAUDE_3_5_SONNET,
+        "structured_output": False,
     },
     # Claude 2 (legacy)
     "claude-2.0": {
@@ -61,6 +68,7 @@
         "function_calling": False,
         "json_output": True,
         "family": ModelFamily.CLAUDE_3_5_SONNET,
+        "structured_output": False,
     },
     # Claude 2.1 (legacy)
     "claude-2.1": {
@@ -68,6 +76,7 @@
         "function_calling": False,
         "json_output": True,
         "family": ModelFamily.CLAUDE_3_5_SONNET,
+        "structured_output": False,
     },
 }
 
diff --git a/python/packages/autogen-ext/src/autogen_ext/models/azure/_azure_ai_client.py b/python/packages/autogen-ext/src/autogen_ext/models/azure/_azure_ai_client.py
index 93a8c2c507aa..373e7ea6b754 100644
--- a/python/packages/autogen-ext/src/autogen_ext/models/azure/_azure_ai_client.py
+++ b/python/packages/autogen-ext/src/autogen_ext/models/azure/_azure_ai_client.py
@@ -52,6 +52,7 @@
 from azure.ai.inference.models import (
     UserMessage as AzureUserMessage,
 )
+from pydantic import BaseModel
 from typing_extensions import AsyncGenerator, Union, Unpack
 
 from autogen_ext.models.azure.config import (
@@ -302,12 +303,15 @@ async def create(
         *,
         tools: Sequence[Tool | ToolSchema] = [],
         json_output: Optional[bool] = None,
+        output_type: Optional[type[BaseModel]] = None,
         extra_create_args: Mapping[str, Any] = {},
         cancellation_token: Optional[CancellationToken] = None,
     ) -> CreateResult:
         extra_create_args_keys = set(extra_create_args.keys())
         if not create_kwargs.issuperset(extra_create_args_keys):
             raise ValueError(f"Extra create args are invalid: {extra_create_args_keys - create_kwargs}")
+        if output_type is not None:
+            raise ValueError("output_type is currently not supported for AzureAIChatCompletionClient")
 
         # Copy the create args and overwrite anything in extra_create_args
         create_args = self._create_args.copy()
@@ -395,6 +399,7 @@ async def create_stream(
         *,
         tools: Sequence[Tool | ToolSchema] = [],
         json_output: Optional[bool] = None,
+        output_type: Optional[type[BaseModel]] = None,
         extra_create_args: Mapping[str, Any] = {},
         cancellation_token: Optional[CancellationToken] = None,
     ) -> AsyncGenerator[Union[str, CreateResult], None]:
@@ -402,6 +407,9 @@ async def create_stream(
         if not create_kwargs.issuperset(extra_create_args_keys):
             raise ValueError(f"Extra create args are invalid: {extra_create_args_keys - create_kwargs}")
 
+        if output_type is not None:
+            raise ValueError("output_type is currently not supported for AzureAIChatCompletionClient")
+
         create_args: Dict[str, Any] = self._create_args.copy()
         create_args.update(extra_create_args)
 
diff --git a/python/packages/autogen-ext/src/autogen_ext/models/cache/_chat_completion_cache.py b/python/packages/autogen-ext/src/autogen_ext/models/cache/_chat_completion_cache.py
index 59a2538b942d..921bc461d7b1 100644
--- a/python/packages/autogen-ext/src/autogen_ext/models/cache/_chat_completion_cache.py
+++ b/python/packages/autogen-ext/src/autogen_ext/models/cache/_chat_completion_cache.py
@@ -131,6 +131,7 @@ async def create(
         *,
         tools: Sequence[Tool | ToolSchema] = [],
         json_output: Optional[bool] = None,
+        output_type: Optional[type[BaseModel]] = None,
         extra_create_args: Mapping[str, Any] = {},
         cancellation_token: Optional[CancellationToken] = None,
     ) -> CreateResult:
@@ -163,6 +164,7 @@ def create_stream(
         *,
         tools: Sequence[Tool | ToolSchema] = [],
         json_output: Optional[bool] = None,
+        output_type: Optional[type[BaseModel]] = None,
         extra_create_args: Mapping[str, Any] = {},
         cancellation_token: Optional[CancellationToken] = None,
     ) -> AsyncGenerator[Union[str, CreateResult], None]:
diff --git a/python/packages/autogen-ext/src/autogen_ext/models/llama_cpp/_llama_cpp_completion_client.py b/python/packages/autogen-ext/src/autogen_ext/models/llama_cpp/_llama_cpp_completion_client.py
index dc77721145fd..d50f0bae0068 100644
--- a/python/packages/autogen-ext/src/autogen_ext/models/llama_cpp/_llama_cpp_completion_client.py
+++ b/python/packages/autogen-ext/src/autogen_ext/models/llama_cpp/_llama_cpp_completion_client.py
@@ -1,3 +1,4 @@
+import asyncio
 import logging  # added import
 import re
 from typing import Any, AsyncGenerator, Dict, List, Literal, Mapping, Optional, Sequence, TypedDict, Union, cast
@@ -30,6 +31,7 @@
     Llama,
     llama_chat_format,
 )
+from pydantic import BaseModel
 from typing_extensions import Unpack
 
 logger = logging.getLogger(EVENT_LOGGER_NAME)  # initialize logger
@@ -256,9 +258,11 @@ async def create(
         # None means do not override the default
         # A value means to override the client default - often specified in the constructor
         json_output: Optional[bool] = None,
+        output_type: Optional[type[BaseModel]] = None,
         extra_create_args: Mapping[str, Any] = {},
         cancellation_token: Optional[CancellationToken] = None,
     ) -> CreateResult:
+        create_args = dict(extra_create_args)
         # Convert LLMMessage objects to dictionaries with 'role' and 'content'
         # converted_messages: List[Dict[str, str | Image | list[str | Image] | list[FunctionCall]]] = []
         converted_messages: list[
@@ -283,12 +287,28 @@ async def create(
             else:
                 raise ValueError(f"Unsupported message type: {type(msg)}")
 
+        if output_type is not None and json_output:
+            raise ValueError("output_type and json_output cannot be used together.")
+        if output_type is not None:
+            create_args["response_format"] = output_type.model_json_schema()
+        if json_output:
+            create_args["response_format"] = {"type": "json_object"}
+
         if self.model_info["function_calling"]:
-            response = self.llm.create_chat_completion(
-                messages=converted_messages, tools=convert_tools(tools), stream=False
+            # Run this in on the event loop to avoid blocking.
+            response_future = asyncio.get_event_loop().run_in_executor(
+                None,
+                lambda: self.llm.create_chat_completion(
+                    messages=converted_messages, tools=convert_tools(tools), stream=False, **create_args
+                ),
             )
         else:
-            response = self.llm.create_chat_completion(messages=converted_messages, stream=False)
+            response_future = asyncio.get_event_loop().run_in_executor(
+                None, lambda: self.llm.create_chat_completion(messages=converted_messages, stream=False, **create_args)
+            )
+        if cancellation_token:
+            cancellation_token.link_future(response_future)
+        response = await response_future
 
         if not isinstance(response, dict):
             raise ValueError("Unexpected response type from LlamaCpp model.")
@@ -372,6 +392,7 @@ async def create_stream(
         # None means do not override the default
         # A value means to override the client default - often specified in the constructor
         json_output: Optional[bool] = None,
+        output_type: Optional[type[BaseModel]] = None,
         extra_create_args: Mapping[str, Any] = {},
         cancellation_token: Optional[CancellationToken] = None,
     ) -> AsyncGenerator[Union[str, CreateResult], None]:
@@ -403,7 +424,9 @@ def count_tokens(
 
     @property
     def model_info(self) -> ModelInfo:
-        return ModelInfo(vision=False, json_output=False, family="llama-cpp", function_calling=True)
+        return ModelInfo(
+            vision=False, json_output=False, family="llama-cpp", function_calling=True, structured_output=True
+        )
 
     def remaining_tokens(
         self,
diff --git a/python/packages/autogen-ext/src/autogen_ext/models/ollama/_model_info.py b/python/packages/autogen-ext/src/autogen_ext/models/ollama/_model_info.py
index 8e5495dcf97d..158a0008fc6c 100644
--- a/python/packages/autogen-ext/src/autogen_ext/models/ollama/_model_info.py
+++ b/python/packages/autogen-ext/src/autogen_ext/models/ollama/_model_info.py
@@ -6,73 +6,300 @@
 # TODO: fix model family?
 # TODO: json_output is True for all models because ollama supports structured output via pydantic. How to handle this situation?
 _MODEL_INFO: Dict[str, ModelInfo] = {
-    "all-minilm": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "bge-m3": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "codegemma": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "codellama": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "command-r": {"vision": False, "function_calling": True, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "deepseek-coder": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
+    "all-minilm": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "bge-m3": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "codegemma": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "codellama": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "command-r": {
+        "vision": False,
+        "function_calling": True,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "deepseek-coder": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
     "deepseek-coder-v2": {
         "vision": False,
         "function_calling": False,
         "json_output": True,
         "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "deepseek-r1": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.R1,
+        "structured_output": True,
+    },
+    "dolphin-llama3": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "dolphin-mistral": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "dolphin-mixtral": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "gemma": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "gemma2": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "llama2": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
     },
-    "deepseek-r1": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.R1},
-    "dolphin-llama3": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "dolphin-mistral": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "dolphin-mixtral": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "gemma": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "gemma2": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "llama2": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
     "llama2-uncensored": {
         "vision": False,
         "function_calling": False,
         "json_output": True,
         "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "llama3": {
+        "vision": False,
+        "function_calling": True,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "llama3.1": {
+        "vision": False,
+        "function_calling": True,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "llama3.2": {
+        "vision": False,
+        "function_calling": True,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "llama3.2-vision": {
+        "vision": True,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "llama3.3": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "llava": {
+        "vision": True,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "llava-llama3": {
+        "vision": True,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "mistral": {
+        "vision": False,
+        "function_calling": True,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "mistral-nemo": {
+        "vision": False,
+        "function_calling": True,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "mixtral": {
+        "vision": False,
+        "function_calling": True,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
     },
-    "llama3": {"vision": False, "function_calling": True, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "llama3.1": {"vision": False, "function_calling": True, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "llama3.2": {"vision": False, "function_calling": True, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "llama3.2-vision": {"vision": True, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "llama3.3": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "llava": {"vision": True, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "llava-llama3": {"vision": True, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "mistral": {"vision": False, "function_calling": True, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "mistral-nemo": {"vision": False, "function_calling": True, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "mixtral": {"vision": False, "function_calling": True, "json_output": True, "family": ModelFamily.UNKNOWN},
     "mxbai-embed-large": {
         "vision": False,
         "function_calling": False,
         "json_output": True,
         "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
     },
     "nomic-embed-text": {
         "vision": False,
         "function_calling": False,
         "json_output": True,
         "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "orca-mini": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "phi": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "phi3": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "phi3.5": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "phi4": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "qwen": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "qwen2": {
+        "vision": False,
+        "function_calling": True,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "qwen2.5": {
+        "vision": False,
+        "function_calling": True,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "qwen2.5-coder": {
+        "vision": False,
+        "function_calling": True,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
     },
-    "orca-mini": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "phi": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "phi3": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "phi3.5": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "phi4": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "qwen": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "qwen2": {"vision": False, "function_calling": True, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "qwen2.5": {"vision": False, "function_calling": True, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "qwen2.5-coder": {"vision": False, "function_calling": True, "json_output": True, "family": ModelFamily.UNKNOWN},
     "snowflake-arctic-embed": {
         "vision": False,
         "function_calling": False,
         "json_output": True,
         "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "starcoder2": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "tinyllama": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "wizardlm2": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "yi": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
+    },
+    "zephyr": {
+        "vision": False,
+        "function_calling": False,
+        "json_output": True,
+        "family": ModelFamily.UNKNOWN,
+        "structured_output": True,
     },
-    "starcoder2": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "tinyllama": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "wizardlm2": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "yi": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
-    "zephyr": {"vision": False, "function_calling": False, "json_output": True, "family": ModelFamily.UNKNOWN},
 }
 
 # TODO: the ollama model card for some of these models were incorrect. I made a best effort to get the actual values, but they aren't guaranteed to be correct.
diff --git a/python/packages/autogen-ext/src/autogen_ext/models/ollama/_ollama_client.py b/python/packages/autogen-ext/src/autogen_ext/models/ollama/_ollama_client.py
index 272d97d2d73e..e9f603dea7f3 100644
--- a/python/packages/autogen-ext/src/autogen_ext/models/ollama/_ollama_client.py
+++ b/python/packages/autogen-ext/src/autogen_ext/models/ollama/_ollama_client.py
@@ -396,6 +396,7 @@ async def create(
         *,
         tools: Sequence[Tool | ToolSchema] = [],
         json_output: Optional[bool] = None,
+        output_type: Optional[type[BaseModel]] = None,
         extra_create_args: Mapping[str, Any] = {},
         cancellation_token: Optional[CancellationToken] = None,
     ) -> CreateResult:
@@ -565,6 +566,7 @@ async def create_stream(
         *,
         tools: Sequence[Tool | ToolSchema] = [],
         json_output: Optional[bool] = None,
+        output_type: Optional[type[BaseModel]] = None,
         extra_create_args: Mapping[str, Any] = {},
         cancellation_token: Optional[CancellationToken] = None,
         max_consecutive_empty_chunk_tolerance: int = 0,
diff --git a/python/packages/autogen-ext/src/autogen_ext/models/openai/_model_info.py b/python/packages/autogen-ext/src/autogen_ext/models/openai/_model_info.py
index c838df9cd554..20a4177ff3c9 100644
--- a/python/packages/autogen-ext/src/autogen_ext/models/openai/_model_info.py
+++ b/python/packages/autogen-ext/src/autogen_ext/models/openai/_model_info.py
@@ -25,144 +25,168 @@
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.O3,
+        "structured_output": True,
     },
     "o1-2024-12-17": {
         "vision": False,
         "function_calling": False,
         "json_output": False,
         "family": ModelFamily.O1,
+        "structured_output": True,
     },
     "o1-preview-2024-09-12": {
         "vision": False,
         "function_calling": False,
         "json_output": False,
         "family": ModelFamily.O1,
+        "structured_output": True,
     },
     "o1-mini-2024-09-12": {
         "vision": False,
         "function_calling": False,
         "json_output": False,
         "family": ModelFamily.O1,
+        "structured_output": False,
     },
     "gpt-4o-2024-11-20": {
         "vision": True,
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.GPT_4O,
+        "structured_output": True,
     },
     "gpt-4o-2024-08-06": {
         "vision": True,
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.GPT_4O,
+        "structured_output": True,
     },
     "gpt-4o-2024-05-13": {
         "vision": True,
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.GPT_4O,
+        "structured_output": False,
     },
     "gpt-4o-mini-2024-07-18": {
         "vision": True,
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.GPT_4O,
+        "structured_output": True,
     },
     "gpt-4-turbo-2024-04-09": {
         "vision": True,
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.GPT_4,
+        "structured_output": False,
     },
     "gpt-4-0125-preview": {
         "vision": False,
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.GPT_4,
+        "structured_output": False,
     },
     "gpt-4-1106-preview": {
         "vision": False,
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.GPT_4,
+        "structured_output": False,
     },
     "gpt-4-1106-vision-preview": {
         "vision": True,
         "function_calling": False,
         "json_output": False,
         "family": ModelFamily.GPT_4,
+        "structured_output": False,
     },
     "gpt-4-0613": {
         "vision": False,
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.GPT_4,
+        "structured_output": False,
     },
     "gpt-4-32k-0613": {
         "vision": False,
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.GPT_4,
+        "structured_output": False,
     },
     "gpt-3.5-turbo-0125": {
         "vision": False,
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.GPT_35,
+        "structured_output": False,
     },
     "gpt-3.5-turbo-1106": {
         "vision": False,
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.GPT_35,
+        "structured_output": False,
     },
     "gpt-3.5-turbo-instruct": {
         "vision": False,
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.GPT_35,
+        "structured_output": False,
     },
     "gpt-3.5-turbo-0613": {
         "vision": False,
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.GPT_35,
+        "structured_output": False,
     },
     "gpt-3.5-turbo-16k-0613": {
         "vision": False,
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.GPT_35,
+        "structured_output": False,
     },
     "gemini-1.5-flash": {
         "vision": True,
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.GEMINI_1_5_FLASH,
+        "structured_output": True,
     },
     "gemini-1.5-flash-8b": {
         "vision": True,
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.GEMINI_1_5_FLASH,
+        "structured_output": True,
     },
     "gemini-1.5-pro": {
         "vision": True,
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.GEMINI_1_5_PRO,
+        "structured_output": True,
     },
     "gemini-2.0-flash": {
         "vision": True,
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.GEMINI_2_0_FLASH,
+        "structured_output": True,
     },
     "gemini-2.0-flash-lite-preview-02-05": {
         "vision": True,
         "function_calling": True,
         "json_output": True,
         "family": ModelFamily.GEMINI_2_0_FLASH,
+        "structured_output": True,
     },
 }
 
diff --git a/python/packages/autogen-ext/src/autogen_ext/models/openai/_openai_client.py b/python/packages/autogen-ext/src/autogen_ext/models/openai/_openai_client.py
index 430d78af105e..fdca288d5dd7 100644
--- a/python/packages/autogen-ext/src/autogen_ext/models/openai/_openai_client.py
+++ b/python/packages/autogen-ext/src/autogen_ext/models/openai/_openai_client.py
@@ -406,6 +406,7 @@ async def create(
         *,
         tools: Sequence[Tool | ToolSchema] = [],
         json_output: Optional[bool] = None,
+        output_type: Optional[Type[BaseModel]] = None,
         extra_create_args: Mapping[str, Any] = {},
         cancellation_token: Optional[CancellationToken] = None,
     ) -> CreateResult:
@@ -422,10 +423,16 @@ async def create(
         use_beta_client: bool = False
         response_format_value: Optional[Type[BaseModel]] = None
 
+        if output_type is not None:
+            response_format_value = output_type
+            use_beta_client = True
+
         if "response_format" in create_args:
             value = create_args["response_format"]
             # If value is a Pydantic model class, use the beta client
             if isinstance(value, type) and issubclass(value, BaseModel):
+                if response_format_value is not None:
+                    raise ValueError("response_format and output_type are mutually exclusive")
                 response_format_value = value
                 use_beta_client = True
             else:
@@ -628,6 +635,7 @@ async def create_stream(
         *,
         tools: Sequence[Tool | ToolSchema] = [],
         json_output: Optional[bool] = None,
+        output_type: Optional[Type[BaseModel]] = None,
         extra_create_args: Mapping[str, Any] = {},
         cancellation_token: Optional[CancellationToken] = None,
         max_consecutive_empty_chunk_tolerance: int = 0,
@@ -639,6 +647,7 @@ async def create_stream(
             messages (Sequence[LLMMessage]): A sequence of messages to be processed.
             tools (Sequence[Tool | ToolSchema], optional): A sequence of tools to be used in the completion. Defaults to `[]`.
             json_output (Optional[bool], optional): If True, the output will be in JSON format. Defaults to None.
+            output_type (Optional[Type[BaseModel]], optional): The type of the output as a Pydantic model. Defaults to None. If set, structured output is used and a serialized JSON object is returned in the final :class:`CreateResult`.
             extra_create_args (Mapping[str, Any], optional): Additional arguments for the creation process. Default to `{}`.
             cancellation_token (Optional[CancellationToken], optional): A token to cancel the operation. Defaults to None.
             max_consecutive_empty_chunk_tolerance (int): [Deprecated] The maximum number of consecutive empty chunks to tolerate before raising a ValueError. This seems to only be needed to set when using `AzureOpenAIChatCompletionClient`. Defaults to 0. This parameter is deprecated, empty chunks will be skipped.
@@ -671,10 +680,16 @@ async def create_stream(
         use_beta_client: bool = False
         response_format_value: Optional[Type[BaseModel]] = None
 
+        if output_type is not None:
+            response_format_value = output_type
+            use_beta_client = True
+
         if "response_format" in create_args:
             value = create_args["response_format"]
             # If value is a Pydantic model class, use the beta client
             if isinstance(value, type) and issubclass(value, BaseModel):
+                if response_format_value is not None:
+                    raise ValueError("response_format and output_type are mutually exclusive")
                 response_format_value = value
                 use_beta_client = True
             else:
diff --git a/python/packages/autogen-ext/src/autogen_ext/models/replay/_replay_chat_completion_client.py b/python/packages/autogen-ext/src/autogen_ext/models/replay/_replay_chat_completion_client.py
index b09e13b4b280..efa0c8e9684a 100644
--- a/python/packages/autogen-ext/src/autogen_ext/models/replay/_replay_chat_completion_client.py
+++ b/python/packages/autogen-ext/src/autogen_ext/models/replay/_replay_chat_completion_client.py
@@ -139,7 +139,11 @@ def __init__(
             validate_model_info(self._model_info)
         else:
             self._model_info = ModelInfo(
-                vision=False, function_calling=False, json_output=False, family=ModelFamily.UNKNOWN
+                vision=False,
+                function_calling=False,
+                json_output=False,
+                family=ModelFamily.UNKNOWN,
+                structured_output=False,
             )
         self._total_available_tokens = 10000
         self._cur_usage = RequestUsage(prompt_tokens=0, completion_tokens=0)
@@ -159,6 +163,7 @@ async def create(
         *,
         tools: Sequence[Tool | ToolSchema] = [],
         json_output: Optional[bool] = None,
+        output_type: Optional[type[BaseModel]] = None,
         extra_create_args: Mapping[str, Any] = {},
         cancellation_token: Optional[CancellationToken] = None,
     ) -> CreateResult:
@@ -198,6 +203,7 @@ async def create_stream(
         *,
         tools: Sequence[Tool | ToolSchema] = [],
         json_output: Optional[bool] = None,
+        output_type: Optional[type[BaseModel]] = None,
         extra_create_args: Mapping[str, Any] = {},
         cancellation_token: Optional[CancellationToken] = None,
     ) -> AsyncGenerator[Union[str, CreateResult], None]:
diff --git a/python/packages/autogen-ext/src/autogen_ext/models/semantic_kernel/_sk_chat_completion_adapter.py b/python/packages/autogen-ext/src/autogen_ext/models/semantic_kernel/_sk_chat_completion_adapter.py
index 75efe9ba6d68..5d976c7646b5 100644
--- a/python/packages/autogen-ext/src/autogen_ext/models/semantic_kernel/_sk_chat_completion_adapter.py
+++ b/python/packages/autogen-ext/src/autogen_ext/models/semantic_kernel/_sk_chat_completion_adapter.py
@@ -283,7 +283,7 @@ def __init__(
         self._prompt_settings = prompt_settings
         self._sk_client = sk_client
         self._model_info = model_info or ModelInfo(
-            vision=False, function_calling=False, json_output=False, family=ModelFamily.UNKNOWN
+            vision=False, function_calling=False, json_output=False, family=ModelFamily.UNKNOWN, structured_output=False
         )
         validate_model_info(self._model_info)
         self._total_prompt_tokens = 0
@@ -438,6 +438,7 @@ async def create(
         *,
         tools: Sequence[Tool | ToolSchema] = [],
         json_output: Optional[bool] = None,
+        output_type: Optional[type[BaseModel]] = None,
         extra_create_args: Mapping[str, Any] = {},
         cancellation_token: Optional[CancellationToken] = None,
     ) -> CreateResult:
@@ -465,6 +466,9 @@ async def create(
         Returns:
             CreateResult: The result of the chat completion.
         """
+        if output_type is not None:
+            raise ValueError("output_type is not currently supported in SKChatCompletionAdapter")
+
         kernel = self._get_kernel(extra_create_args)
 
         chat_history = self._convert_to_chat_history(messages)
@@ -546,6 +550,7 @@ async def create_stream(
         *,
         tools: Sequence[Tool | ToolSchema] = [],
         json_output: Optional[bool] = None,
+        output_type: Optional[type[BaseModel]] = None,
         extra_create_args: Mapping[str, Any] = {},
         cancellation_token: Optional[CancellationToken] = None,
     ) -> AsyncGenerator[Union[str, CreateResult], None]:
@@ -574,6 +579,9 @@ async def create_stream(
             Union[str, CreateResult]: Either a string chunk of the response or a CreateResult containing function calls.
         """
 
+        if output_type is not None:
+            raise ValueError("output_type is not currently supported in SKChatCompletionAdapter")
+
         kernel = self._get_kernel(extra_create_args)
         chat_history = self._convert_to_chat_history(messages)
         user_settings = self._get_prompt_settings(extra_create_args)
diff --git a/python/packages/autogen-ext/tests/models/test_azure_ai_model_client.py b/python/packages/autogen-ext/tests/models/test_azure_ai_model_client.py
index 43401134378d..7c55df6d9d3e 100644
--- a/python/packages/autogen-ext/tests/models/test_azure_ai_model_client.py
+++ b/python/packages/autogen-ext/tests/models/test_azure_ai_model_client.py
@@ -88,6 +88,7 @@ def azure_client(monkeypatch: pytest.MonkeyPatch) -> AzureAIChatCompletionClient
                 "function_calling": False,
                 "vision": False,
                 "family": "unknown",
+                "structured_output": False,
             },
             model="model",
         )
@@ -101,6 +102,7 @@ def azure_client(monkeypatch: pytest.MonkeyPatch) -> AzureAIChatCompletionClient
             "function_calling": False,
             "vision": False,
             "family": "unknown",
+            "structured_output": False,
         },
         model="model",
     )
@@ -117,6 +119,7 @@ async def test_azure_ai_chat_completion_client_validation() -> None:
                 "function_calling": False,
                 "vision": False,
                 "family": "unknown",
+                "structured_output": False,
             },
         )
 
@@ -129,6 +132,7 @@ async def test_azure_ai_chat_completion_client_validation() -> None:
                 "function_calling": False,
                 "vision": False,
                 "family": "unknown",
+                "structured_output": False,
             },
         )
 
@@ -141,6 +145,7 @@ async def test_azure_ai_chat_completion_client_validation() -> None:
                 "function_calling": False,
                 "vision": False,
                 "family": "unknown",
+                "structured_output": False,
             },
         )
 
@@ -267,6 +272,7 @@ async def _mock_function_call_create(*args: Any, **kwargs: Any) -> ChatCompletio
             "function_calling": True,
             "vision": False,
             "family": "function_calling_model",
+            "structured_output": False,
         },
         model="model",
     )
@@ -354,6 +360,7 @@ async def _mock_create_noop(*args: Any, **kwargs: Any) -> ChatCompletions:
             "function_calling": False,
             "vision": True,
             "family": "vision_model",
+            "structured_output": False,
         },
         model="model",
     )
@@ -436,6 +443,7 @@ async def _mock_create_r1_content(
             "function_calling": False,
             "vision": True,
             "family": ModelFamily.R1,
+            "structured_output": False,
         },
         model="model",
     )
diff --git a/python/packages/autogen-ext/tests/models/test_llama_cpp_model_client.py b/python/packages/autogen-ext/tests/models/test_llama_cpp_model_client.py
index 1898fe7a7e8e..864100bf49df 100644
--- a/python/packages/autogen-ext/tests/models/test_llama_cpp_model_client.py
+++ b/python/packages/autogen-ext/tests/models/test_llama_cpp_model_client.py
@@ -2,6 +2,8 @@
 import sys
 from typing import TYPE_CHECKING, Any, ContextManager, Generator, List, Sequence, Union
 
+from llama_cpp import ChatCompletionRequestResponseFormat
+from pydantic import BaseModel
 import pytest
 import torch
 
@@ -21,6 +23,13 @@
     pytest.skip("Skipping LlamaCppChatCompletionClient tests: llama-cpp-python not installed", allow_module_level=True)
 
 
+class AgentResponse(BaseModel):
+    """A response from the agent."""
+
+    thoughts: str
+    content: str
+
+
 # Fake Llama class to simulate responses
 class FakeLlama:
     def __init__(
@@ -30,16 +39,25 @@ def __init__(
     ) -> None:
         self.model_path = model_path
         self.n_ctx = lambda: 1024
+        self._structured_response = AgentResponse(thoughts="Test thoughts", content="Test content")
 
     # Added tokenize method for testing purposes.
     def tokenize(self, b: bytes) -> list[int]:
         return list(b)
 
     def create_chat_completion(
-        self, messages: Any, tools: List[ChatCompletionMessageToolCalls] | None, stream: bool = False
+        self, messages: Any, tools: List[ChatCompletionMessageToolCalls] | None, stream: bool = False, response_format: ChatCompletionRequestResponseFormat | None = None,
     ) -> dict[str, Any]:
         # Return fake non-streaming response.
 
+        if response_format is not None:
+            assert self._structured_response is not None
+            # If response_format is provided, return a different format.
+            return {
+                "usage": {"prompt_tokens": 1, "completion_tokens": 2},
+                "choices": [{"message": {"content": self._structured_response.model_dump_json()}}],
+            }
+
         return {
             "usage": {"prompt_tokens": 1, "completion_tokens": 2},
             "choices": [{"message": {"content": "Fake response"}}],
@@ -80,6 +98,21 @@ async def test_llama_cpp_create(get_completion_client: "ContextManager[type[Llam
         assert usage.completion_tokens == 2
         assert result.finish_reason in ("stop", "unknown")
 
+@pytest.mark.asyncio
+async def test_llama_cpp_create_structured_output(
+    get_completion_client: "ContextManager[type[LlamaCppChatCompletionClient]]",
+) -> None:
+    with get_completion_client as Client:
+        client = Client(model_path="dummy")
+        messages: Sequence[Union[SystemMessage, UserMessage]] = [
+            SystemMessage(content="Test system"),
+            UserMessage(content="Test user", source="user"),
+        ]
+        result = await client.create(messages=messages, json_output=True)
+        assert isinstance(result.content, str)
+        assert AgentResponse.model_validate_json(result.content).thoughts == "Test thoughts"
+        assert AgentResponse.model_validate_json(result.content).content == "Test content"
+
 
 # Commmented out due to raising not implemented error will leave in case streaming is supported in the future.
 # @pytest.mark.asyncio
diff --git a/python/packages/autogen-ext/tests/models/test_openai_model_client.py b/python/packages/autogen-ext/tests/models/test_openai_model_client.py
index 5cef46edb97f..09bb407c478e 100644
--- a/python/packages/autogen-ext/tests/models/test_openai_model_client.py
+++ b/python/packages/autogen-ext/tests/models/test_openai_model_client.py
@@ -204,7 +204,13 @@ async def test_custom_model_with_capabilities() -> None:
         model="dummy_model",
         base_url="https://api.dummy.com/v0",
         api_key="api_key",
-        model_info={"vision": False, "function_calling": False, "json_output": False, "family": ModelFamily.UNKNOWN},
+        model_info={
+            "vision": False,
+            "function_calling": False,
+            "json_output": False,
+            "family": ModelFamily.UNKNOWN,
+            "structured_output": False,
+        },
     )
     assert client
 
@@ -217,7 +223,13 @@ async def test_azure_openai_chat_completion_client() -> None:
         api_key="api_key",
         api_version="2020-08-04",
         azure_endpoint="https://dummy.com",
-        model_info={"vision": True, "function_calling": True, "json_output": True, "family": ModelFamily.GPT_4O},
+        model_info={
+            "vision": True,
+            "function_calling": True,
+            "json_output": True,
+            "family": ModelFamily.GPT_4O,
+            "structured_output": True,
+        },
     )
     assert client
 
@@ -469,11 +481,10 @@ async def _mock_parse(*args: Any, **kwargs: Any) -> ParsedChatCompletion[AgentRe
     model_client = OpenAIChatCompletionClient(
         model=model,
         api_key="",
-        response_format=AgentResponse,  # type: ignore
     )
 
     # Test that the openai client was called with the correct response format.
-    create_result = await model_client.create(messages=[UserMessage(content="I am happy.", source="user")])
+    create_result = await model_client.create(messages=[UserMessage(content="I am happy.", source="user")], output_type=AgentResponse)
     assert isinstance(create_result.content, str)
     response = AgentResponse.model_validate(json.loads(create_result.content))
     assert (
@@ -530,11 +541,10 @@ async def _mock_parse(*args: Any, **kwargs: Any) -> ParsedChatCompletion[AgentRe
     model_client = OpenAIChatCompletionClient(
         model=model,
         api_key="",
-        response_format=AgentResponse,  # type: ignore
     )
 
     # Test that the openai client was called with the correct response format.
-    create_result = await model_client.create(messages=[UserMessage(content="I am happy.", source="user")])
+    create_result = await model_client.create(messages=[UserMessage(content="I am happy.", source="user")], output_type=AgentResponse)
     assert isinstance(create_result.content, list)
     assert len(create_result.content) == 1
     assert create_result.content[0] == FunctionCall(
@@ -603,12 +613,11 @@ async def _stream() -> AsyncGenerator[MockChunkEvent, None]:
     model_client = OpenAIChatCompletionClient(
         model=model,
         api_key="",
-        response_format=AgentResponse,  # type: ignore
     )
 
     # Test that the openai client was called with the correct response format.
     chunks: List[str | CreateResult] = []
-    async for chunk in model_client.create_stream(messages=[UserMessage(content="I am happy.", source="user")]):
+    async for chunk in model_client.create_stream(messages=[UserMessage(content="I am happy.", source="user")], output_type=AgentResponse):
         chunks.append(chunk)
     assert len(chunks) > 0
     assert isinstance(chunks[-1], CreateResult)
@@ -712,12 +721,11 @@ async def _stream() -> AsyncGenerator[MockChunkEvent, None]:
     model_client = OpenAIChatCompletionClient(
         model=model,
         api_key="",
-        response_format=AgentResponse,  # type: ignore
     )
 
     # Test that the openai client was called with the correct response format.
     chunks: List[str | CreateResult] = []
-    async for chunk in model_client.create_stream(messages=[UserMessage(content="I am happy.", source="user")]):
+    async for chunk in model_client.create_stream(messages=[UserMessage(content="I am happy.", source="user")], output_type=AgentResponse):
         chunks.append(chunk)
     assert len(chunks) > 0
     assert isinstance(chunks[-1], CreateResult)
@@ -787,7 +795,13 @@ async def _mock_create(*args: Any, **kwargs: Any) -> ChatCompletion | AsyncGener
     model_client = OpenAIChatCompletionClient(
         model="r1",
         api_key="",
-        model_info={"family": ModelFamily.R1, "vision": False, "function_calling": False, "json_output": False},
+        model_info={
+            "family": ModelFamily.R1,
+            "vision": False,
+            "function_calling": False,
+            "json_output": False,
+            "structured_output": False,
+        },
     )
 
     # Successful completion with think field.
@@ -860,7 +874,13 @@ async def _mock_create(*args: Any, **kwargs: Any) -> ChatCompletion | AsyncGener
     model_client = OpenAIChatCompletionClient(
         model="r1",
         api_key="",
-        model_info={"family": ModelFamily.R1, "vision": False, "function_calling": False, "json_output": False},
+        model_info={
+            "family": ModelFamily.R1,
+            "vision": False,
+            "function_calling": False,
+            "json_output": False,
+            "structured_output": False,
+        },
     )
 
     # Warning completion when think field is not present.
@@ -1324,11 +1344,10 @@ class AgentResponse(BaseModel):
     model_client = OpenAIChatCompletionClient(
         model="gpt-4o-mini",
         api_key=api_key,
-        response_format=AgentResponse,  # type: ignore
     )
 
     # Test that the openai client was called with the correct response format.
-    create_result = await model_client.create(messages=[UserMessage(content="I am happy.", source="user")])
+    create_result = await model_client.create(messages=[UserMessage(content="I am happy.", source="user")], output_type=AgentResponse)
     assert isinstance(create_result.content, str)
     response = AgentResponse.model_validate(json.loads(create_result.content))
     assert response.thoughts
@@ -1348,11 +1367,10 @@ class AgentResponse(BaseModel):
     model_client = OpenAIChatCompletionClient(
         model="gpt-4o-mini",
         api_key=api_key,
-        response_format=AgentResponse,  # type: ignore
     )
 
     # Test that the openai client was called with the correct response format.
-    stream = model_client.create_stream(messages=[UserMessage(content="I am happy.", source="user")])
+    stream = model_client.create_stream(messages=[UserMessage(content="I am happy.", source="user")], output_type=AgentResponse)
     chunks: List[str | CreateResult] = []
     async for chunk in stream:
         chunks.append(chunk)
@@ -1383,7 +1401,6 @@ def sentiment_analysis(text: str) -> str:
     model_client = OpenAIChatCompletionClient(
         model="gpt-4o-mini",
         api_key=api_key,
-        response_format=AgentResponse,  # type: ignore
     )
 
     response1 = await model_client.create(
@@ -1393,6 +1410,7 @@ def sentiment_analysis(text: str) -> str:
         ],
         tools=[tool],
         extra_create_args={"tool_choice": "required"},
+        output_type=AgentResponse,
     )
     assert isinstance(response1.content, list)
     assert len(response1.content) == 1
@@ -1414,6 +1432,7 @@ def sentiment_analysis(text: str) -> str:
                 ]
             ),
         ],
+        output_type=AgentResponse,
     )
     assert isinstance(response2.content, str)
     parsed_response = AgentResponse.model_validate(json.loads(response2.content))
@@ -1440,7 +1459,6 @@ def sentiment_analysis(text: str) -> str:
     model_client = OpenAIChatCompletionClient(
         model="gpt-4o-mini",
         api_key=api_key,
-        response_format=AgentResponse,  # type: ignore
     )
 
     chunks1: List[str | CreateResult] = []
@@ -1451,6 +1469,7 @@ def sentiment_analysis(text: str) -> str:
         ],
         tools=[tool],
         extra_create_args={"tool_choice": "required"},
+        output_type=AgentResponse,
     )
     async for chunk in stream1:
         chunks1.append(chunk)
@@ -1477,6 +1496,7 @@ def sentiment_analysis(text: str) -> str:
                 ]
             ),
         ],
+        output_type=AgentResponse,
     )
     chunks2: List[str | CreateResult] = []
     async for chunk in stream2:
@@ -1518,6 +1538,7 @@ async def test_hugging_face() -> None:
             "json_output": False,
             "vision": False,
             "family": ModelFamily.UNKNOWN,
+            "structured_output": False,
         },
     )
 
@@ -1532,6 +1553,7 @@ async def test_ollama() -> None:
         "json_output": False,
         "vision": False,
         "family": ModelFamily.R1,
+        "structured_output": False,
     }
     # Check if the model is running locally.
     try:
diff --git a/python/packages/autogen-ext/tests/models/test_sk_chat_completion_adapter.py b/python/packages/autogen-ext/tests/models/test_sk_chat_completion_adapter.py
index bd30aad0f06a..0f694b8492ac 100644
--- a/python/packages/autogen-ext/tests/models/test_sk_chat_completion_adapter.py
+++ b/python/packages/autogen-ext/tests/models/test_sk_chat_completion_adapter.py
@@ -453,7 +453,9 @@ async def test_sk_chat_completion_default_model_info(sk_client: AzureChatComplet
 @pytest.mark.asyncio
 async def test_sk_chat_completion_custom_model_info(sk_client: AzureChatCompletion) -> None:
     # Create custom model info
-    custom_model_info = ModelInfo(vision=True, function_calling=True, json_output=True, family=ModelFamily.GPT_4)
+    custom_model_info = ModelInfo(
+        vision=True, function_calling=True, json_output=True, family=ModelFamily.GPT_4, structured_output=False
+    )
 
     # Create adapter with custom model_info
     adapter = SKChatCompletionAdapter(sk_client, model_info=custom_model_info)
@@ -522,7 +524,9 @@ async def mock_get_streaming_chat_message_contents(
     adapter = SKChatCompletionAdapter(
         mock_client,
         kernel=kernel,
-        model_info=ModelInfo(vision=False, function_calling=False, json_output=False, family=ModelFamily.R1),
+        model_info=ModelInfo(
+            vision=False, function_calling=False, json_output=False, family=ModelFamily.R1, structured_output=False
+        ),
     )
 
     result = await adapter.create(messages=[UserMessage(content="Say hello!", source="user")])


Code review comments:
diff_hunk: @@ -395,13 +399,17 @@ async def create_stream(
         *,
         tools: Sequence[Tool | ToolSchema] = [],
         json_output: Optional[bool] = None,
+        output_type: Optional[type[BaseModel]] = None,
         extra_create_args: Mapping[str, Any] = {},
         cancellation_token: Optional[CancellationToken] = None,
     ) -> AsyncGenerator[Union[str, CreateResult], None]:
         extra_create_args_keys = set(extra_create_args.keys())
         if not create_kwargs.issuperset(extra_create_args_keys):
             raise ValueError(f"Extra create args are invalid: {extra_create_args_keys - create_kwargs}")
 
+        if output_type is not None:
+            raise ValueError("output_type is currently not supported for AzureAIChatCompletionClient")
path: python/packages/autogen-ext/src/autogen_ext/models/azure/_azure_ai_client.py
text: @user1:
For the azure client, I  json schema is in the code and they also mention it in the docs, would be good to add a todo at least

https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/json-mode?tabs=python

@user2:
The Azure AI client sdk might be tricky .. since it supports any arbitrary model from the AI foundry (many of them may not support structured output). See docs https://github.com/MicrosoftDocs/azure-docs-sdk-python/blob/main/docs-ref-services/preview/ai-inference-readme.md 


@user1:
At an api level I remember they use a single field akin to response format where we specify either json mode or the json schema. At an API level I think we should support it, then whether the model in case has support I think is something we or the user would need to handle through the model info.

@author:
Agreed. Add TODO for future PR as it is a bit involved.

diff_hunk: @@ -256,9 +258,11 @@ async def create(
         # None means do not override the default
         # A value means to override the client default - often specified in the constructor
         json_output: Optional[bool] = None,
+        output_type: Optional[type[BaseModel]] = None,
path: python/packages/autogen-ext/src/autogen_ext/models/llama_cpp/_llama_cpp_completion_client.py
text: @user1:
Just thinking out loud. Could we maybe use the json_output as `json_output: bool|type[BaseModel]|None= None` to keep it in a single argument since for the apis it is generally the same? I think a lot of sdks made a mistake in creating json_mode as an argument instead of keeping it closer to the response_format argument of the api. Response format can always be expanded to more options and we may not want to keep adding keys in the function for every change.

The argument name `json_output` is not the best but given that the basemodel is the representation of a json schema I also dont think it would be completely inadequate. 

@author:
Good idea. Working on it.

diff_hunk: @@ -435,6 +437,9 @@ async def create(
             if json_output is True:
                 create_args["response_format"] = {"type": "json_object"}
 
+        if output_type is not None:
path: python/packages/autogen-ext/src/autogen_ext/models/anthropic/_anthropic_client.py
text: @user1:
Is this a placeholder? I believe anthropic supports structured outputs.

@user1:
NVM, it requires some prompt workarounds 

@user2:

Were you able to find any documentation on this?

My investigations shows they do not directly support structured output types.
Anthropic supports something called "json mode", does not guarantee parsable json like OAI or other models do.
The closes is a bit of a hack .. https://docs.anthropic.com/en/docs/build-with-claude/tool-use/overview#json-mode 

- pass in output_type as tool 
- set tool_choice 
- extract the tool response 

I am not sure we want to support that. Given how widespread this is, I expect anthroipic to support it soon.



@user1:
Yes, that seems correct. WHat I read is that is seems good at following examples in the prompt with very low error rate, but the support is not at the api level.