alfin06/swegent-issues-filtered huggingface dataset:
State Management does not work with `OpenAIAssistantAgent`

### What happened?

[This](https://microsoft.github.io/autogen/stable/user-guide/agentchat-user-guide/tutorial/state.html#managing-state) tutorial works for `AssistantAgent` not for `OpenAIAssistantAgent`. It looks like the `save_state` and `load_state`  methods use the default impl [here](https://github.com/microsoft/autogen/blob/main/python/packages/autogen-agentchat/src/autogen_agentchat/agents/_base_chat_agent.py#L189) which return

`{'type': 'BaseState', 'version': '1.0.0'}`

It works as expected for `AssistantAgent`.

### What did you expect to happen?

State management should work for `OpenAIAssistantAgent` similar to `AssistantAgent`. 

### How can we reproduce it (as minimally and precisely as possible)?

```
import asyncio
from autogen_ext.agents.openai import OpenAIAssistantAgent
from autogen_core import CancellationToken

from autogen_agentchat.messages import TextMessage
from openai import AsyncAzureOpenAI
import os


async def example():

    async_model_client = AsyncAzureOpenAI(
        api_key = os.environ.get("API_KEY"),
        azure_endpoint = os.environ.get("API_BASE"),
        api_version= "2024-07-01-preview",
    )

    agent = OpenAIAssistantAgent(name="FunnyAgent",
                            description="Tells jokes",
                            client=async_model_client,
                            model="gpt-4o",
                            instructions="You tell jokes.",
                        )

    response = await agent.on_messages(
        [TextMessage(source="user", content="Tell me a joke")], CancellationToken()
    )

    print(response.chat_message.content)

    agent_state = await agent.save_state()
    print(agent_state)

asyncio.run(example())

```

Observe that the agent state is returned as

```
{'type': 'BaseState', 'version': '1.0.0'}
```

### AutoGen version

0.4.2

### Which package was this bug in

Extensions

### Model used

gpt-4o

### Python version

3.12.8

### Operating system

MacOS 15.2

### Any additional info you think would be helpful for fixing this bug

_No response_

Labels: help wanted, proj-extensions
State: closed
Structured message type in AgentChat

### What feature would you like to be added?

AgentChat agents can emit and return a message that contains an object rather than a text field.

For example:

```python

ContentTypeT = TypeVar("ContentTypeT", bound=BaseModel)

class StructuredMessage(BaseChatMessage, Generic[ContentTypeT]):
    content: ContentTypeT
    source: str
```

So, we can instantiate a message like `StructuredMessage[Leger](content=leger, source="magentic-one-orchestrator")` for magentic one orchestrator to emit to the stream. #5127 

We can also make it possible for an agent to return a structured tool call input as a structured message. 

### Why is this needed?

Many applications require agent to communicate via structured messages, and agents may emit structured "inner thoughts" for better observability. 

Right now, the only way to emit "structured message" is through a serialized JSON in a `TextMessage` field. A structured message type helps the consumer understand what to expect in the message itself. This also allows for filtering at the consumer of the stream to decide whether to display the message, e.g., to the console.

Labels: proj-agentchat, structured output
State: closed
Tool result content is incorrectly serialized as a string when calling bedrock

### Description

When converting messages to the bedrock API, tool responses appear to have their entire content serialized as a JSON string within a `text` content block, rather than handling each individual piece of content appropriately.

This breaks handling of image responses from tools, because the base64 of the image ends up just being sent to the model as text.

This seems very similar to https://github.com/vercel/ai/issues/3379, but with bedrock rather than anthropic. 

The current behaviour produces requests like:

```
{
  "role": "user",
  "content": [
    {
      "toolResult": {
        "toolUseId": "tooluse_PBqR3AclQ2eObQsXbNzx5w",
        "content": [
          {
            "text": "[{\"type\":\"text\",\"text\":\"Tool Response\"},{\"type\":\"image\",\"data\":\"MQ==\",\"mimeType\":\"image/png\"}]"
          }
        ]
      }
    }
  ]
}
```

### Code example

```js
import { bedrock } from '@ai-sdk/amazon-bedrock';
import { generateText, tool } from 'ai';
import { z } from 'zod';


async function main() {

  const result = await generateText({
    model: bedrock("us.anthropic.claude-3-7-sonnet-20250219-v1:0"),
    messages: [{
      role: 'user',
      content: [{ type: "text", text: "Please download this image https://picsum.photos/200/300 and tell me what you see" }]
    }],
    tools: {
      submit: tool({
        description: 'Download an image',
        parameters: z.object({
          url: z
            .string()
            .describe('The image URL'),
        }),
        execute: async ({ url }) => {
          const bytes = await (await fetch(url)).bytes();
          return { bytes }
        },
        experimental_toToolResultContent(result) {
          return [{
            type: "image",
            data: Buffer.from(result.bytes).toString("base64"),
            mimeType: "image/jpeg",
          }]
        },
      }),
    },
    maxSteps: 5,
  });

  console.log(result.text);
}

main().catch(console.error);
```

The model will completely hallucinate the contents of the image, because it only has access to the base64 data as a string.

### AI provider

@ai-sdk/amazon-bedrock v2.1.1

### Additional context

I think the issue is caused by this line https://github.com/vercel/ai/blame/9b081a7a2241e9796d8b8561e71fc5ba9dd2bdf6/packages/amazon-bedrock/src/convert-to-bedrock-chat-messages.ts#L138, which should probably be reusing the existing content serialisation logic rather than JSON.stringify.

Labels: bug, ai/provider, provider/amazon-bedrock
State: closed
`SKChatCompletionAdapter` cannot be used with `AssistantAgent`

The model client adapter cannot be used with `AssistantAgent` due to the required kernel parameter that must be passed to the create call.

Can we instead perhaps optionally bind the kernel at instantiation time and allow it to be overridden at time of calling create?

Labels: needs-triage, proj-extensions
State: closed
FunctionTool should support passing in a `partial`

`functools.partial` objects have a `func` property that allows retrieving the inner function and reading annotations. The applied args should then be removed from the schema prior to generating it

Labels: proj-core
State: closed
[Bug] 限制历史消息数功能逻辑缺陷

### 📦 部署环境

Docker

### 📌 软件版本

v1.38.0

### 💻 系统环境

macOS

### 🌐 浏览器

Chrome

### 🐛 问题描述

限制历史消息数功能打开时，设置为0应当是没有上下文，而不是不限制上下文数量。

现在实际上当启用限制功能时，最低为0，代码逻辑在：
lobehub/lobe-chat/src/app/(main)/chat/(workspace)/@conversation/features/ChatList/ChatItem/index.tsx [68:75]
  const enableHistoryDivider = useAgentStore((s) => {
    const config = agentSelectors.currentAgentChatConfig(s);
    return (
      config.enableHistoryCount &&
      historyLength > (config.historyCount ?? 0) &&
      config.historyCount === historyLength - index
    );
  });
可以看到限制为0时，会始终返回false

### 📷 复现步骤

启用一个话题时，启用限制历史消息数功能，设置限制为0即可复现

### 🚦 期望结果

当启用限制历史消息数功能，且限制为0时，限制应当如下：

user: A
AI：B
---结束对话
user:new A
AI: new B

### 📝 补充信息

_No response_

Labels: released
State: closed
support openai responses api file inputs

### Feature Description

#### Context  
With OpenAI's **Response API** now supporting file inputs ([docs here](https://platform.openai.com/docs/guides/pdf-files)), it makes sense to align the Vercel AI SDK with this update. The SDK already supports OpenAI’s Response API, including tools like web search, so extending support to **file input** and the `input` attribute while preserving the SDK's generic structure would be beneficial.  

#### Why is this important?  
- OpenAI's Response API has a **different structure**, using an `input` array instead of `messages`.  
- Other models, like **Gemini Flash**, already support file input, but their format differs. For example, with Gemini, we handle file uploads like this:  

    ```js
    messages: [
      {
        role: "user",
        content: [
          {
            type: "text",
            text: "Help me with this file",
          },
          {
            type: "file",
            data: fileBase64,
            mimeType: "application/pdf",
          },
        ],
      },
    ]
    ```  
- However, OpenAI’s **Response API** follows a different approach, replacing `messages` with `input`:  

    ```js
    input: [
      {
        role: "user",
        content: [
          {
            type: "input_file",
            filename: "draconomicon.pdf",
            file_data: `data:application/pdf;base64,${base64String}`,
          },
          {
            type: "input_text",
            text: "What is the first dragon in the book?",
          },
        ],
      },
    ]
    ```  

#### Proposal  
1. **Maintain the `messages` format in the SDK**, ensuring a consistent developer experience across models.  
2. **Convert `messages` to `input` internally** when using OpenAI's `(<model>).responses` method.  
3. **Reuse the existing `FilePart` structure** ([docs here](https://sdk.vercel.ai/docs/reference/ai-sdk-core/core-message#filepart)) for file inputs, adapting it to OpenAI's Response API format.  
4. Ensure the implementation remains flexible to accommodate multiple models while abstracting away provider-specific differences.  

Would love to hear thoughts on whether this is already on the radar and if there are any challenges in supporting this. Happy to contribute!


### Use Cases

_No response_

### Additional context

_No response_

Labels: feature, ai/provider, provider/openai
State: closed
何时支持Qwen3VL

目前Qwen3VL的权重（235B和30B）都以开源，可以在https://huggingface.co/collections/Qwen/qwen3-vl-68d2a7c1b8a8afce4ebd2dbe被找到。

另外，Qwen3VL关于MoE和稠密架构的代码也都开源了，请问作者大大有没有计划支持Qwen3VL呢

State: closed
Support Gemini image generation

### Feature Description

https://ai.google.dev/gemini-api/docs/image-generation

Gemini 2.0 Flash Exp now supports image output

### Use Cases

_No response_

### Additional context

_No response_

Labels: feature, ai/core, ai/provider, provider/google
State: closed
Typo in dotnet/samples/Hello/HelloAIAgents/Program.cs

### What happened?

autogen\dotnet\samples\Hello\HelloAIAgents>dotnet run
fails:

Unhandled exception. System.InvalidOperationException: No connection string named 'HelloAIAgents' was found. Ensure a corresponding Aspire service was registered.

### What did you expect to happen?

autogen\dotnet\samples\Hello\HelloAIAgents>dotnet run
runs fine

### How can we reproduce it (as minimally and precisely as possible)?

autogen\dotnet\samples\Hello\HelloAIAgents>dotnet run

The issue is a typo in dotnet/samples/Hello/HelloAIAgents/Program.cs.
Fix:
-builder.Configuration["ConectionStrings:HelloAIAgents"] = Environment.GetEnvironmentVariable("AZURE_OPENAI_CONNECTION_STRING");
+builder.Configuration["ConnectionStrings:HelloAIAgents"] = Environment.GetEnvironmentVariable("AZURE_OPENAI_CONNECTION_STRING");


### AutoGen version

Commit b375d4b18c88eedb6fade6b2ec50912bb2ca1e82

### Which package was this bug in

Core

### Model used

gpt-4o

### Python version

_No response_

### Operating system

WSL

### Any additional info you think would be helpful for fixing this bug

_No response_

Labels: dotnet
State: closed
Allow returning thought in the tool call responses

### What feature would you like to be added?

Some models include an additional text message in the tool call response. [for example](https://docs.anthropic.com/en/docs/build-with-claude/tool-use):

```
{
  "role": "assistant",
  "content": [
    {
      "type": "text",
      "text": "<thinking>To answer this question, I will: 1. Use the get_weather tool to get the current weather in San Francisco. 2. Use the get_time tool to get the current time in the America/Los_Angeles timezone, which covers San Francisco, CA.</thinking>"
    },
    {
      "type": "tool_use",
      "id": "toolu_01A09q90qw90lq917835lq9",
      "name": "get_weather",
      "input": {"location": "San Francisco, CA"}
    }
  ]
}

```

We need to allow including that message in the response, so that it can be properly surfaced to agent and users.

The proposal in #5173 is to change the `content` field to `Union[str, FunctionCalls]` instead of `Union[str, List[FunctionCall]]` so that additional fields could be included in `FunctionCalls`:

```
@dataclass
class FunctionCalls:
    function_calls: List[FunctionCall]
    thought: Optional[str] = None
```

### Why is this needed?

to include model response properly

Labels: api-break-change, proj-core, proj-extensions, proj-agentchat
State: closed
.NET runtime interface to match python - add registration to interface and InMemoryRuntime

### What feature would you like to be added?

add RegisterAgentType to the agent runtime (its already in the backend server, but in memory we don't need it because we just load the agents from relection. 

### Why is this needed?

In .NET when the agents are all in one process we can use reflection to register agents and technically don't need registration. But we desire to match with the python runtime interface so I'm adding it. this may also enable scenarios like the one @colombod has proposed in which the in memory runtime could at some point self-federate into a larger agent system with distributed runtimes. 

Labels: dotnet
State: closed
Perplexity provider generateText crashes

### Description

perplexity is currently returning `null` as `finish_reason` on some of the calls (likely error on their side). The zod validation schema on `generateText` expects a non-nullable string which causes the whole response to fail. 

![Image](https://github.com/user-attachments/assets/5592b92d-8114-4382-9edb-b95bb70642db)

```
  Error message: [
    {
      "code": "invalid_type",
      "expected": "string",
      "received": "null",
      "path": [
        "choices",
        0,
        "finish_reason"
      ],
      "message": "Expected string, received null"
    }

```

### Code example

_No response_

### AI provider

_No response_

### Additional context

_No response_

Labels: bug, ai/provider, provider/perplexity
State: closed
Parsed reasoning support for Groq adapter

### Feature Description

Groq supports reasoning models like Deepseek, and it has the option to return the reasoning process, but when using tool calling it is returned in a separate field and not in the response text directly, which is not supported right now by the adapter.

### Use Cases

_No response_

### Additional context

_No response_

Labels: feature, ai/provider, provider/deepseek, provider/groq
State: closed
[Bug] 启用artifact插件导致的意外页面跳转

### 📦 部署环境

Docker

### 📌 软件版本

v1.42.1

### 💻 系统环境

Windows

### 🌐 浏览器

Chrome

### 🐛 问题描述

启用artifact插件，并指示（Claude Haiku）进行绘图后，页面意外跳转，详见录屏。
另外，在删除该对话后（使用“清空当前对话消息”），页面右侧的artifact框不会随着对话被删除而自动消失。
https://github.com/user-attachments/assets/97b99c76-7c81-4cce-b3c5-8ff0263649a8



### 📷 复现步骤

_No response_

### 🚦 期望结果

_No response_

### 📝 补充信息

_No response_

Labels: released
State: closed
CreateResult to include an optional field for reasoning token to support R1

An additional option field for `thoughts`. 

For R-1 family of models, we will parse the output to extract <think></think> content, and use the remaining content in the `content` field.

Labels: needs-triage, proj-extensions
State: closed
add support for some models whose tool call arguments is not a json object

### What feature would you like to be added?

Some of the models' tool call arguments does not return a json object but a dictionary. 

For example, AssistantAgent using DeepSeek-R1-Distill-Qwen-1.5B on HF inference api, inner_messages:
[ToolCallRequestEvent(source='Weather_Agent', models_usage=RequestUsage(prompt_tokens=173, completion_tokens=19), content=[FunctionCall(id='0', arguments={'location': 'Hangzhou'}, name='get_weather')], type='ToolCallRequestEvent'), ToolCallExecutionEvent(source='Weather_Agent', models_usage=None, content=[FunctionExecutionResult(content='**Error: the JSON object must be str, bytes or bytearray, not dict'**, call_id='0')], type='ToolCallExecutionEvent')] 

As a reference, gpt-4o-mini inner_messages:
[ToolCallRequestEvent(source='Weather_Agent', models_usage=RequestUsage(prompt_tokens=69, completion_tokens=16), content=[FunctionCall(id='call_xxx', arguments='{"location":"Hangzhou"}', name='get_weather')], type='ToolCallRequestEvent'), ToolCallExecutionEvent(source='Weather_Agent', models_usage=None, content=[FunctionExecutionResult(content='Hangzhou, nice!', call_id='call_6F0V1emkKJCleuTmo0PKpzge')], type='ToolCallExecutionEvent')] 

Add support for dictionary type argument can allow the usage of these models. A temporary fix (for dict only) could be done at autogen_agentchat/agents/_assistant_agent.py:
```
async def _execute_tool_call(
        self, tool_call: FunctionCall, cancellation_token: CancellationToken
    ) -> FunctionExecutionResult:
        """Execute a tool call and return the result."""
        try:
            if not self._tools + self._handoff_tools:
                raise ValueError("No tools are available.")
            tool = next((t for t in self._tools + self._handoff_tools if t.name == tool_call.name), None)
            if tool is None:
                raise ValueError(f"The tool '{tool_call.name}' is not available.")
            if isinstance(tool_call.arguments, dict):
                arguments = tool_call.arguments
            else:
                arguments = json.loads(tool_call.arguments)
            result = await tool.run_json(arguments, cancellation_token)
            result_as_str = tool.return_value_as_string(result)
            return FunctionExecutionResult(content=result_as_str, call_id=tool_call.id)
        except Exception as e:
            return FunctionExecutionResult(content=f"Error: {e}", call_id=tool_call.id)
```
will have inner_messages:
[ToolCallRequestEvent(source='Weather_Agent', models_usage=RequestUsage(prompt_tokens=173, completion_tokens=19), content=[FunctionCall(id='0', arguments={'location': 'Hangzhou'}, name='get_weather')], type='ToolCallRequestEvent'), ToolCallExecutionEvent(source='Weather_Agent', models_usage=None, content=[FunctionExecutionResult(content='Hangzhou, nice!', call_id='0')], type='ToolCallExecutionEvent')] 

### Why is this needed?

support for some models whose tool call is not a json object, e.g., DeepSeek on HF

Labels: proj-extensions
State: closed
Support for provider-defined tools with Amazon Bedrock

### Feature Description

As per the [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-anthropic-claude-37.html#model-parameters-anthropic-claude-37-thinking-blocks) docs, there is now support for some "new Anthropic defined tools":

![Image](https://github.com/user-attachments/assets/79e983fc-5ef0-44e0-9477-6fc0f38979d5)

I poked around and noticed that Bedrock doesn't currently support any provider-defined tools: https://github.com/vercel/ai/blob/1040a8882284475423fc0e883220252f97bc00b9/packages/amazon-bedrock/src/bedrock-prepare-tools.ts#L31-L33

This wasn't immediately intuitive to me given that e.g. `text_editor_20250124` works out of the box with `@ai-sdk/anthropic` and then realized it just hasn't been added yet.

Support for this at some point would be great 💪

### Use Cases

_No response_

### Additional context

_No response_

Labels: feature, ai/provider, provider/anthropic, provider/amazon-bedrock
State: closed
isLastToolInvocationFollowedByText() should ignore text parts that originate from the same model response as the tool call

### Description

### 📄 Description  
`gemini-2.0-flash-001` sometimes generates function calls with a **leading text part that is just a newline character (`\n`)**.  

This causes an issue in `useChat()` where **tool call results are not submitted automatically**, and the conversation gets stuck. The user then has to manually continue the conversation to submit the tool call result.

I have configured `useChat()` with `maxSteps` set to 10 and `streamText()` in the backend with `maxSteps` set to 1 (default). This means that the client will handle the resubmit.

---

### ⚠️ Problem  
Sometimes (not always), Gemini responds in the following format:

```json
{
  "id": "msg-llkj9PNWMkILH5gkJRFMjlu3",
  "createdAt": "2025-03-19T17:22:50.373",
  "role": "assistant",
  "content": "\n",
  "parts": [
    {
      "type": "tool-invocation",
      "toolInvocation": {
        "state": "result",
        "step": 0,
        "toolCallId": "ezv9IDtNtQrqvptU",
        "toolName": "updateState",
        "args": {
          "op": "replace",
          "path": "/basic/document_name/fi",
          "value": "Testing testing"
        },
        "result": "State updated."
      }
    },
    {
      "type": "text",
      "text": "\n"
    }
  ],
  "toolInvocations": [...],
  "revisionId": "ogxXwFFQmJomF6lT"
}
```

After this message is streamed and handled by the `useChat()` hook, **it does not get submitted back to the model for the next step**, because [shouldResubmitMessages()](https://github.com/vercel/ai/blob/46c85b5a1e08ee0887c9fe9f27da6db6a8db12f7/packages/ui-utils/src/should-resubmit-messages.ts#L4) returns `false`.

The reason is that [isLastToolInvocationFollowedByText()](https://github.com/vercel/ai/blob/46c85b5a1e08ee0887c9fe9f27da6db6a8db12f7/packages/ui-utils/src/should-resubmit-messages.ts#L34) returns `true` due to the trailing text part `"\n"` – even though it was created **before** the tool call result.

---

### 💡 Suggested Fix  
A potential fix would be to **ignore trailing text parts that are empty or just whitespace/newlines** inside `isLastToolInvocationFollowedByText()` logic.

Alternatively, a more robust solution would be to **ignore text parts that originated from the model's tool call message** as those texts does not mean that the model has answered to the results yet.

---

### ✅ Expected Behavior  
If a tool invocation result is followed only by a text part that was created in the same model response as the tool call itself, tool call results should be automatically submitted back to the model.

---

### 🙏 Request  
Please consider improving `isLastToolInvocationFollowedByText()` to either:

- Ignore text parts that are just whitespace, or  
- Ignore text parts that were part of the model's original tool call message

### Code example

_No response_

### AI provider

@ai-sdk/google 1.1.12

### Additional context

_No response_

Labels: bug, ai/provider, provider/google
State: closed