Repository: microsoft/autogen
Pull request: add tool_call_summary_msg_format_fct and test
## Why are these changes needed?

This change introduces support for dynamic formatting of tool call summary messages by allowing a user-defined `tool_call_summary_format_fct`. Instead of relying solely on a static string template, this function enables runtime generation of summary messages based on the specific tool call and its result. This provides greater flexibility and cleaner integration without introducing any breaking changes.

### My Use Case / Problem

In my use case, I needed concise summaries for successful tool calls and detailed messages for failures. The existing static summary string didn't allow conditional formatting, which led to overly verbose success messages or inconsistent failure outputs. This change allows customizing summaries per result type, solving that limitation cleanly.

## Related issue number

Closes #6426

## Checks

- [x] I've included any doc changes needed for <https://microsoft.github.io/autogen/>. See <https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md> to build and test documentation locally.
- [x] I've added tests (if relevant) corresponding to the changes introduced in this PR.
- [x] I've made sure all auto checks have passed.


Problem statement:
Insufficient static string template for ToolCallSummaryMessage
### What happened?

**The Problem** 
I have the following use case:
A tool call with huge payload if successful. Model must not reflect on it. 
If tool call is successful, I just need have a tool call summary message like "Tool Foo has been executed successfully", whereas I need all details for the failure case (i.e.{result}). This was not possible because of the static string template we pass to the assistant agent.

**Proposal** 
We could make it more dynamic by injecting a tool_call_summary_format_fct which will be evaluated at runtime and gives more control over the summary messages.

**Implementation (_assistant_agent.py)**
```
@staticmethod
    def _summarize_tool_use(
        executed_calls_and_results: List[Tuple[FunctionCall, FunctionExecutionResult]],
        inner_messages: List[BaseAgentEvent | BaseChatMessage],
        handoffs: Dict[str, HandoffBase],
        tool_call_summary_format: str,
        tool_call_summary_format_fct: Callable[[FunctionCall, FunctionExecutionResult], str] | None,
        agent_name: str,
    ) -> Response:
  ...

        def default_summary_template(call: FunctionCall, result: FunctionExecutionResult) -> str:
            return tool_call_summary_format

        summary_template_fct = tool_call_summary_format_fct or default_summary_template

        tool_call_summaries = [
            summary_template_fct(call, result).format(
                tool_name=call.name,
                arguments=call.arguments,
                result=result.content,
                is_error=result.is_error,
            )
            for call, result in normal_tool_calls
        ]
...
```

Thats basically it, it could be easily integrated without any breaking changes, I already did everything required.

**How to use it (test_assistant_agent.py)**
```
async def _throw_function(input: str) -> str:
    raise ValueError("Helpful debugging information what went wrong.")

@pytest.fixture
def model_info_all_capabilities() -> ModelInfo:
    return {
        "function_calling": True,
        "vision": True,
        "json_output": True,
        "family": ModelFamily.GPT_4O,
        "structured_output": True,
    }


@pytest.mark.asyncio
async def test_run_with_tool_call_summary_format_function(model_info_all_capabilities: ModelInfo) -> None:
    model_client = ReplayChatCompletionClient(
        [
            CreateResult(
                finish_reason="function_calls",
                content=[
                    FunctionCall(id="1", arguments=json.dumps({"input": "task"}), name="_pass_function"),
                    FunctionCall(id="2", arguments=json.dumps({"input": "task"}), name="_throw_function"),
                ],
                usage=RequestUsage(prompt_tokens=10, completion_tokens=5),
                thought="Calling pass and fail function",
                cached=False,
            ),
            "pass and fail",
            "TERMINATE",
        ],
        model_info=model_info_all_capabilities,
    )

    def conditional_string_templates(function_call: FunctionCall, function_call_result: FunctionExecutionResult) -> str:
        if not function_call_result.is_error:
            return "SUCCESS: {tool_name} with {arguments}"

        else:
            return "FAILURE: {result}"

    agent = AssistantAgent(
        "tool_use_agent",
        model_client=model_client,
        tools=[_pass_function, _throw_function],
        tool_call_summary_format_fct=conditional_string_templates,
    )
    result = await agent.run(task="task")

    first_tool_call_summary = next((x for x in result.messages if isinstance(x, ToolCallSummaryMessage)), None)
    if first_tool_call_summary is None:
        raise AssertionError("Expected a ToolCallSummaryMessage but found none.")

    assert (
        first_tool_call_summary.content
        == 'SUCCESS: _pass_function with {"input": "task"}\nFAILURE: Helpful debugging information what went wrong.'
    )
```

Let me know if its considered helpful, if so I can provide the PR @ekzhu  

### Which packages was the bug in?

Python AgentChat (autogen-agentchat>=0.4.0)

### AutoGen library version.

Python dev (main branch)

### Other library version.

_No response_

### Model used

_No response_

### Model provider

None

### Other model provider

_No response_

### Python version

None

### .NET version

None

### Operating system

None

Hints from issue discussion:

Commit message: add tool_call_summary_msg_format_fct and test

Patch to review:
diff --git a/python/packages/autogen-agentchat/src/autogen_agentchat/agents/_assistant_agent.py b/python/packages/autogen-agentchat/src/autogen_agentchat/agents/_assistant_agent.py
index c189021a2e2c..1e179adced73 100644
--- a/python/packages/autogen-agentchat/src/autogen_agentchat/agents/_assistant_agent.py
+++ b/python/packages/autogen-agentchat/src/autogen_agentchat/agents/_assistant_agent.py
@@ -75,6 +75,7 @@ class AssistantAgentConfig(BaseModel):
     model_client_stream: bool = False
     reflect_on_tool_use: bool
     tool_call_summary_format: str
+    tool_call_summary_format_fct: Callable[[FunctionCall, FunctionExecutionResult], str] | None = None
     metadata: Dict[str, str] | None = None
     structured_message_factory: ComponentModel | None = None
 
@@ -652,6 +653,7 @@ def __init__(
         model_client_stream: bool = False,
         reflect_on_tool_use: bool | None = None,
         tool_call_summary_format: str = "{result}",
+        tool_call_summary_format_fct: Callable[[FunctionCall, FunctionExecutionResult], str] | None = None,
         output_content_type: type[BaseModel] | None = None,
         output_content_type_format: str | None = None,
         memory: Sequence[Memory] | None = None,
@@ -756,6 +758,7 @@ def __init__(
                 stacklevel=2,
             )
         self._tool_call_summary_format = tool_call_summary_format
+        self._tool_call_summary_format_fct = tool_call_summary_format_fct
         self._is_running = False
 
     @property
@@ -803,6 +806,7 @@ async def on_messages_stream(
         model_client_stream = self._model_client_stream
         reflect_on_tool_use = self._reflect_on_tool_use
         tool_call_summary_format = self._tool_call_summary_format
+        tool_call_summary_format_fct = self._tool_call_summary_format_fct
         output_content_type = self._output_content_type
         format_string = self._output_content_type_format
 
@@ -873,6 +877,7 @@ async def on_messages_stream(
             model_client_stream=model_client_stream,
             reflect_on_tool_use=reflect_on_tool_use,
             tool_call_summary_format=tool_call_summary_format,
+            tool_call_summary_format_fct=tool_call_summary_format_fct,
             output_content_type=output_content_type,
             format_string=format_string,
         ):
@@ -976,6 +981,7 @@ async def _process_model_result(
         model_client_stream: bool,
         reflect_on_tool_use: bool,
         tool_call_summary_format: str,
+        tool_call_summary_format_fct: Callable[[FunctionCall, FunctionExecutionResult], str] | None,
         output_content_type: type[BaseModel] | None,
         format_string: str | None = None,
     ) -> AsyncGenerator[BaseAgentEvent | BaseChatMessage | Response, None]:
@@ -1078,6 +1084,7 @@ async def _process_model_result(
                 inner_messages=inner_messages,
                 handoffs=handoffs,
                 tool_call_summary_format=tool_call_summary_format,
+                tool_call_summary_format_fct=tool_call_summary_format_fct,
                 agent_name=agent_name,
             )
 
@@ -1232,6 +1239,7 @@ def _summarize_tool_use(
         inner_messages: List[BaseAgentEvent | BaseChatMessage],
         handoffs: Dict[str, HandoffBase],
         tool_call_summary_format: str,
+        tool_call_summary_format_fct: Callable[[FunctionCall, FunctionExecutionResult], str] | None,
         agent_name: str,
     ) -> Response:
         """
@@ -1239,15 +1247,22 @@ def _summarize_tool_use(
         """
         # Filter out calls which were actually handoffs
         normal_tool_calls = [(call, result) for call, result in executed_calls_and_results if call.name not in handoffs]
-        tool_call_summaries: List[str] = []
-        for tool_call, tool_call_result in normal_tool_calls:
-            tool_call_summaries.append(
-                tool_call_summary_format.format(
-                    tool_name=tool_call.name,
-                    arguments=tool_call.arguments,
-                    result=tool_call_result.content,
-                )
+
+        def default_summary_template(call: FunctionCall, result: FunctionExecutionResult) -> str:
+            return tool_call_summary_format
+
+        summary_template_fct = tool_call_summary_format_fct or default_summary_template
+
+        tool_call_summaries = [
+            summary_template_fct(call, result).format(
+                tool_name=call.name,
+                arguments=call.arguments,
+                result=result.content,
+                is_error=result.is_error,
             )
+            for call, result in normal_tool_calls
+        ]
+
         tool_call_summary = "\n".join(tool_call_summaries)
         return Response(
             chat_message=ToolCallSummaryMessage(
@@ -1354,6 +1369,7 @@ def _to_config(self) -> AssistantAgentConfig:
             model_client_stream=self._model_client_stream,
             reflect_on_tool_use=self._reflect_on_tool_use,
             tool_call_summary_format=self._tool_call_summary_format,
+            tool_call_summary_format_fct=self._tool_call_summary_format_fct,
             structured_message_factory=self._structured_message_factory.dump_component()
             if self._structured_message_factory
             else None,
@@ -1385,6 +1401,7 @@ def _from_config(cls, config: AssistantAgentConfig) -> Self:
             model_client_stream=config.model_client_stream,
             reflect_on_tool_use=config.reflect_on_tool_use,
             tool_call_summary_format=config.tool_call_summary_format,
+            tool_call_summary_format_fct=config.tool_call_summary_format_fct,
             output_content_type=output_content_type,
             output_content_type_format=format_string,
             metadata=config.metadata,
diff --git a/python/packages/autogen-agentchat/tests/test_assistant_agent.py b/python/packages/autogen-agentchat/tests/test_assistant_agent.py
index 666deb511be6..eb20a072673a 100644
--- a/python/packages/autogen-agentchat/tests/test_assistant_agent.py
+++ b/python/packages/autogen-agentchat/tests/test_assistant_agent.py
@@ -32,7 +32,7 @@
     SystemMessage,
     UserMessage,
 )
-from autogen_core.models._model_client import ModelFamily
+from autogen_core.models._model_client import ModelFamily, ModelInfo
 from autogen_core.tools import BaseTool, FunctionTool, StaticWorkbench
 from autogen_ext.models.openai import OpenAIChatCompletionClient
 from autogen_ext.models.replay import ReplayChatCompletionClient
@@ -56,10 +56,68 @@ async def _fail_function(input: str) -> str:
     return "fail"
 
 
+async def _throw_function(input: str) -> str:
+    raise ValueError("Helpful debugging information what went wrong.")
+
+
 async def _echo_function(input: str) -> str:
     return input
 
 
+@pytest.fixture
+def model_info_all_capabilities() -> ModelInfo:
+    return {
+        "function_calling": True,
+        "vision": True,
+        "json_output": True,
+        "family": ModelFamily.GPT_4O,
+        "structured_output": True,
+    }
+
+
+@pytest.mark.asyncio
+async def test_run_with_tool_call_summary_format_function(model_info_all_capabilities: ModelInfo) -> None:
+    model_client = ReplayChatCompletionClient(
+        [
+            CreateResult(
+                finish_reason="function_calls",
+                content=[
+                    FunctionCall(id="1", arguments=json.dumps({"input": "task"}), name="_pass_function"),
+                    FunctionCall(id="2", arguments=json.dumps({"input": "task"}), name="_throw_function"),
+                ],
+                usage=RequestUsage(prompt_tokens=10, completion_tokens=5),
+                thought="Calling pass and fail function",
+                cached=False,
+            ),
+        ],
+        model_info=model_info_all_capabilities,
+    )
+
+    def conditional_string_templates(function_call: FunctionCall, function_call_result: FunctionExecutionResult) -> str:
+        if not function_call_result.is_error:
+            return "SUCCESS: {tool_name} with {arguments}"
+
+        else:
+            return "FAILURE: {result}"
+
+    agent = AssistantAgent(
+        "tool_use_agent",
+        model_client=model_client,
+        tools=[_pass_function, _throw_function],
+        tool_call_summary_format_fct=conditional_string_templates,
+    )
+    result = await agent.run(task="task")
+
+    first_tool_call_summary = next((x for x in result.messages if isinstance(x, ToolCallSummaryMessage)), None)
+    if first_tool_call_summary is None:
+        raise AssertionError("Expected a ToolCallSummaryMessage but found none")
+
+    assert (
+        first_tool_call_summary.content
+        == 'SUCCESS: _pass_function with {"input": "task"}\nFAILURE: Helpful debugging information what went wrong.'
+    )
+
+
 @pytest.mark.asyncio
 async def test_run_with_tools(monkeypatch: pytest.MonkeyPatch) -> None:
     model_client = ReplayChatCompletionClient(


Code review comments:
diff_hunk: @@ -652,6 +653,7 @@ def __init__(
         model_client_stream: bool = False,
         reflect_on_tool_use: bool | None = None,
         tool_call_summary_format: str = "{result}",
+        tool_call_summary_format_fct: Callable[[FunctionCall, FunctionExecutionResult], str] | None = None,
path: python/packages/autogen-agentchat/src/autogen_agentchat/agents/_assistant_agent.py
text: @user1:
Let's use `tool_call_summary_formatter` instead. 

Also, we need to add this to the argument list in the API doc. 

@author:
done

diff_hunk: @@ -1232,22 +1239,30 @@ def _summarize_tool_use(
         inner_messages: List[BaseAgentEvent | BaseChatMessage],
         handoffs: Dict[str, HandoffBase],
         tool_call_summary_format: str,
+        tool_call_summary_format_fct: Callable[[FunctionCall, FunctionExecutionResult], str] | None,
         agent_name: str,
     ) -> Response:
         """
         If reflect_on_tool_use=False, create a summary message of all tool calls.
         """
         # Filter out calls which were actually handoffs
         normal_tool_calls = [(call, result) for call, result in executed_calls_and_results if call.name not in handoffs]
-        tool_call_summaries: List[str] = []
-        for tool_call, tool_call_result in normal_tool_calls:
-            tool_call_summaries.append(
-                tool_call_summary_format.format(
-                    tool_name=tool_call.name,
-                    arguments=tool_call.arguments,
-                    result=tool_call_result.content,
-                )
+
+        def default_summary_template(call: FunctionCall, result: FunctionExecutionResult) -> str:
path: python/packages/autogen-agentchat/src/autogen_agentchat/agents/_assistant_agent.py
text: @user1:
```suggestion
        def default_tool_call_summary_formatter(call: FunctionCall, result: FunctionExecutionResult) -> str:
```

@author:
done

diff_hunk: @@ -75,6 +75,7 @@ class AssistantAgentConfig(BaseModel):
     model_client_stream: bool = False
     reflect_on_tool_use: bool
     tool_call_summary_format: str
+    tool_call_summary_format_fct: Callable[[FunctionCall, FunctionExecutionResult], str] | None = None
path: python/packages/autogen-agentchat/src/autogen_agentchat/agents/_assistant_agent.py
text: @user1:
We can't put this in the config because not every callable is serializable, e.g., a normal function. We need to make a note in the API doc that currently the custom `tool_call_summary_formatter` does not support serialization. 

To support serialization, it needs to be a class. It can be a future work. 

@author:
Done. I would take care of the refactoring into an appropriate serializable class as well for one of the next releases.