alfin06/swegent-issues-filtered huggingface dataset:
Mistral models do not seem to work with tools

### Description

Using the streamText functionality with Mistral models, I cannot get them to call a tool. Whenever there should be a call (other models from openai, anthropic or google genAi do the call), the response is empty text (see example response below). The tool is never called. 

Tools are indicated as usable for Mistral models in the Ai SDK docs. Mistral API has the ability to call tools.

To communicate with the backend, I use the 'useChat' hook on the frontend.

### Code example

```js
const result = await streamText({
    model: mistral("mistral-small-latest"),
    system: `You are a helpful assistant. Check your knowledge base before answering any questions.
    Only respond to questions using information from tool calls.
    if no relevant information is found in the tool calls, respond, "Sorry, I don't know."
    `,
    messages: messages,
    tools: {
      getInformation: tool({
        description: `get information from your knowledge base to answer questions.`,
        parameters: z.object({
          question: z.string().describe('the users question'),
        }),
        execute: async ({ question }) => findRelevantContent(question),
      }),
    },
    onFinish: ({ response }) => {
      for (let message of response.messages) {
        console.log("message", message);
      }
    },
  });

  return result.toDataStreamResponse();
```


Response :  
```
message {                                                                                   
  role: 'assistant',                                                                        
  content: [ { type: 'text', text: '' } ],                                          
  id: 'msg-HyGPrBvgekywYuDqp2DNfFUO'       
}
```

### AI provider

@ai-sdk/mistral@1.1.8

### Additional context

Using : 
@ai-sdk/react@1.1.14 
ai@4.1.39
nextjs@15.1.6

Labels: bug, ai/provider
State: closed
Add Streaming Support for OpenAI o1 Model in streamText

### Feature Description

Hi, OpenAI has recently announced streaming support for the o1 model. However, the streamText function currently does not work with o1, and we have to use smoothStream as a workaround to simulate streaming.

It would be great to add native support for o1 streaming in streamText, ensuring a seamless real-time experience without requiring additional modifications.

Is this feature planned for an update? Looking forward to your thoughts. Thanks! 🚀

### Use Cases

_No response_

### Additional context

_No response_

Labels: feature, ai/provider
State: closed
Handle ErrorMessage in invoker

We need to handle the newly introduce `ErrorMessage` in the invoker. To begin with we should log the error message that is being sent.

Labels: invoker
State: closed
Auth headers getting overwritten when providing headers object with Vertex AI

### Description

I'm trying to use `@ai-sdk/google-vertex` alongside Cloudflare's AI Gateway, 

The following works as-intended with Google's Generative AI provider:

```typescript
export const googleGenerativeAiSdk = createGoogleGenerativeAI({
  apiKey: GCP_API_KEY,
  baseURL: `${CLOUDFLARE_AI_GATEWAY_BASE_URL}/google-ai-studio/v1beta`,
  headers: {
    "cf-aig-authorization": `Bearer ${CLOUDFLARE_AI_GATEWAY_TOKEN}`,
  },
});
```

However the same logic fails when using Vertex AI alongside Google's Vertex AI.

```typescript
export const vertexAiSdk = createVertex({
  baseURL:
    CLOUDFLARE_AI_GATEWAY_BASE_URL +
    `/google-vertex-ai/v1/projects/${GCP_SERVICE_ACCOUNT_KEY.project_id}/locations/${GCP_REGION}/publishers/google`,
  googleAuthOptions: {
    credentials: GCP_SERVICE_ACCOUNT_KEY,
  },
  project: GCP_SERVICE_ACCOUNT_KEY.project_id,
  location: GCP_REGION,
  headers: {
    "cf-aig-authorization": `Bearer ${CLOUDFLARE_AI_GATEWAY_TOKEN}`,
  },
});
```
Which results in 
> APICallError [AI_APICallError]: Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.



### Code example

_No response_

### AI provider

@ai-sdk/google v1.1.16

### Additional context

Through testing, I've discovered that if I remove the `headers` object from the `createVertex` then the request succeeds, however I am forced to disable authentication for my Cloudflare Gateway. 

If I try to manually generate the ID token and provide it in the headers object, the request succeeds.

```typescript
 headers: async () => ({
    authorization: "Bearer " + (await GoogleAuthClient.getAccessToken()),
    "cf-aig-authorization": `Bearer ${CLOUDFLARE_AI_GATEWAY_TOKEN}`,
  });
```

I still believe this to be a bug, however, because the behavior works correctly in one library and not in the other, and it doesn't make sense for `@ai-sdk/google-vertex` to collect authorization credentials from me while still requiring me to generate my own ID token in the header.

Labels: bug
State: closed
Add anthropic reasoning support to bedrock

### Feature Description

Add anthropic reasoning support to bedrock. `vertexAnthropic` supports reasoning via:

```typescript
providerOptions: {
    anthropic: {
        thinking: { type: 'enabled', budgetTokens: 1024 }
     }
},
```

Adding support to the bedrock provider as well would be beneficial



### Use Cases

Having reasoning support for anthropic claude models in bedrock would be nice.

### Additional context

Based off of my two second look, you can add an additional field like below to the `BedrockChatSettings` and have the reasoning enabled:

```
{
	"additionalModelRequestFields": {
		"reasoning_config": {
			"budget_tokens": 1024,
			"type": "enabled"
		}
	}
}
```

I took a quick stab at this in #4980 for reference. My next comment has some testing notes.

Labels: feature, ai/provider
State: closed
AttributeError: module 'autogen_ext.models.openai' has no attribute 'AzureTokenProvider'

When following the official guide of Magentic-one, I have encountered the exception.

To reproduce the exception:

```bash
env "CHAT_COMPLETION_CLIENT_CONFIG=$(cat config.json)" LOGLEVEL=DEBUG python examples/example.py --logs_dir ./logs
```

Content of `config.json`:

```json
{
    "provider": "AzureOpenAIChatCompletionClient",
    "config": {
        "model": "gpt-4o-2024-05-13",
        "azure_endpoint": "https://{your-custom-endpoint}.openai.azure.com/",
        "azure_deployment": "{your-azure-deployment}",
        "api_version": "2024-06-01",
        "azure_ad_token_provider": {
            "provider": "autogen_ext.models.openai.AzureTokenProvider",
            "config": {
                "provider_kind": "DefaultAzureCredential",
                "scopes": [
                    "https://cognitiveservices.azure.com/.default"
                ]
            }
        }
    }
}
```

In order to resolve this issue, I have made the following change, but unsure if it is appropriate or effective.

```python
# file: python/packages/autogen-core/src/autogen_core/_component_config.py
# in: class ComponentLoader, def load_component

        _typ = ComponentConfigImpl[BaseModel]
        try:
            _val = module.__getattribute__(class_name) # AttributeError: module 'autogen_ext.models.openai' has no attribute 'AzureTokenProvider'
        except:
            print("Falling back to default class:", cls.__name__)
            _val = cls
        component_class = cast(_typ, _val)
```

Labels: needs-triage
State: closed
appendResponseMessages doesn't handle reasoning response message types

### Description

The function skips/ignores messages of type `reasoning` so they get lost

### Code example

_No response_

### AI provider

_No response_

### Additional context

_No response_

Labels: bug
State: closed
Anthropic prompt caching does not work for user message when set with providerOptions

### Description

Looks like the function (convertPartToLanguageModelPart) (https://github.com/vercel/ai/blob/9197aa5b18c50668118b9f2b9551239d4ca3b06c/packages/ai/core/prompt/convert-to-language-model-prompt.ts#L214) only relies on experimental_providerMetadata and not on providerOptions.

```
if (part.type === 'text') {
    return {
      type: 'text',
      text: part.text,
      providerMetadata: part.experimental_providerMetadata,
    };
  }
```

### Code example

So in my code this does not work -
```
part.providerOptions = { anthropic: { cache_control: { type: 'ephemeral' } } };
```

While this is ok -

```
part.experimental_providerMetadata = { anthropic: { cache_control: { type: 'ephemeral' } } };
```


### AI provider

Anthropic

### Additional context

Trying to use prompt caching

Labels: bug, ai/core
State: closed
Problems with API calls

I used langflow to build a conversation AI that contains ConversationBufferMemory. When I refer to the sample code and deploy the API on the server for public calls, how does Langflow distinguish the conversations of different users and carry the Memory information?

State: closed
Missing tiktoken dependency in AutoGen Studio

### What happened?

Cloning main in a devcontainer, trying to launch, results in the following error due to missing dependency `tiktoken`:
 
```bash
$ autogenstudio ui --port 8081
Traceback (most recent call last):
  File "/home/vscode/.local/bin/autogenstudio", line 5, in <module>
    from autogenstudio.cli import run
  File "/workspaces/autogen/python/packages/autogen-studio/autogenstudio/__init__.py", line 1, in <module>
    from .database.db_manager import DatabaseManager
  File "/workspaces/autogen/python/packages/autogen-studio/autogenstudio/database/__init__.py", line 1, in <module>
    from .component_factory import Component, ComponentFactory
  File "/workspaces/autogen/python/packages/autogen-studio/autogenstudio/database/component_factory.py", line 25, in <module>
    from autogen_ext.models.openai import AzureOpenAIChatCompletionClient, OpenAIChatCompletionClient
  File "/home/vscode/.local/lib/python3.12/site-packages/autogen_ext/models/openai/__init__.py", line 1, in <module>
    from ._openai_client import AzureOpenAIChatCompletionClient, BaseOpenAIChatCompletionClient, OpenAIChatCompletionClient
  File "/home/vscode/.local/lib/python3.12/site-packages/autogen_ext/models/openai/_openai_client.py", line 23, in <module>
    import tiktoken
ModuleNotFoundError: No module named 'tiktoken'
```

### What did you expect to happen?

AutoGen studio to launch

### How can we reproduce it (as minimally and precisely as possible)?

1. Clone main in a dev container based on `mcr.microsoft.com/devcontainers/python:1-3.12-bullseye`
2. Run
  ```bash
    $ autogenstudio ui --port 8081
  ```

### AutoGen version

115fefa132116f62c428e4dfe43d8f23f71fe306

### Which package was this bug in

AutoGen Studio

### Model used

_No response_

### Python version

3.12

### Operating system

Debian GNU/Linux 11 (bullseye)

### Any additional info you think would be helpful for fixing this bug

PR coming shortly

Labels: proj-studio
State: closed
finish_reason of non-openai models not handled in OpenAIChatCompletionClient

### What happened?

I plan to use openrouter with autogen given it provides openai api compatible models. I am able to run openai using openrouter. however, if i try to run claude or gemini, i run into this:

-----
ValidationError: 1 validation error for CreateResult
finish_reason
  Input should be 'stop', 'length', 'function_calls' or 'content_filter' [type=literal_error, input_value='end_turn', input_type=str]
    For further information visit https://errors.pydantic.dev/2.10/v/literal_error

-----

### What did you expect to happen?

I expect a way to easily specify stop reasons and map them to openai standard ones so that we can use any model whose request schema is compatible to openai but response schema may not be.

### How can we reproduce it (as minimally and precisely as possible)?

config_list=[
  {
    "model": "openai/gpt-4o",
    "base_url": "https://openrouter.ai/api/v1",
    "api_key": openrouter_key,
    "api_type": "openai",
    "model_info": {
      "vision": True,
      "function_calling": True,
      "json_output": False
    }
  },
  {
    "model": "anthropic/claude-3.5-sonnet",
    "base_url": "https://openrouter.ai/api/v1",
    "api_key": openrouter_key,
    "api_type": "anthropic",
    "model_info": {
      "vision": True,
      "function_calling": True,
      "json_output": False
    }
  }
]

config = config_list[1]  # Use the openai model config

# Extract relevant parameters
model = config["model"]
api_key = config["api_key"]
base_url = config["base_url"]
model_info=config.get("model_info", {})
api_type = config["api_type"]
# Initialize the OpenRouterClient
model_client = OpenAIChatCompletionClient(
    model=model,
    api_key=api_key,
    base_url=base_url,
    model_info=model_info,
    api_type=api_type
)

from autogen_agentchat.agents import AssistantAgent

# Initialize the AssistantAgent
agent = AssistantAgent(
    name="assistant",
    model_client=model_client,
    system_message="You are a helpful assistant skilled in multimodal tasks."
)


from autogen_core.models import UserMessage

# Run the task with the pre-constructed messages
response = await run_task(agent, agentchat_messages)
print(response)

---------------

Result:
ValidationError: 1 validation error for CreateResult
finish_reason
  Input should be 'stop', 'length', 'function_calls' or 'content_filter' [type=literal_error, input_value='end_turn', input_type=str]
    For further information visit https://errors.pydantic.dev/2.10/v/literal_error

---------------

Works well when i change "config = config_list[1]" to "config = config_list[0]" ie. use openai

### AutoGen version

0.4

### Which package was this bug in

AgentChat

### Model used

claude

### Python version

3.12

### Operating system

_No response_

### Any additional info you think would be helpful for fixing this bug

I believe there should be a way to map response keys from non-openai models in "CreateResult" so that they can be used

Labels: needs-triage, proj-extensions
State: closed
Support declarative configs for BaseChatAgents (and derivatives) - UserProxy, AssistantAgent, WebSurferAgent

Support a workflow where an Agent's configuration can be made declarative

```python

# dump
agent = AssistantAgent(
    name="assistant",
    model_client=model_client,
    handoffs=["flights_refunder", "user"],
    tools=[web_search],
    system_message="Use tools to solve tasks.",
)
agent_config = agent.dump_component() 

```

```python
{'name': 'assistant',
 'model_client': {'provider': 'autogen_ext.models.openai.OpenAIChatCompletionClient',
  'component_type': 'model',
  'version': 1,
  'component_version': 1,
  'config': {'model': 'gpt-4o'}},
 'tools': [],
 'handoffs': [{'target': 'flights_refunder',
   'description': 'Handoff to flights_refunder.',
   'name': 'transfer_to_flights_refunder',
   'message': 'Transferred to flights_refunder, adopting the role of flights_refunder immediately.'},
  {'target': 'user',
   'description': 'Handoff to user.',
   'name': 'transfer_to_user',
   'message': 'Transferred to user, adopting the role of user immediately.'}],
 'model_context': {'provider': 'autogen_core.model_context.UnboundedChatCompletionContext',
  'component_type': 'chat_completion_context',
  'version': 1,
  'component_version': 1,
  'config': {}},
 'description': 'An agent that provides assistance with ability to use tools.',
 'system_message': 'Use tools to solve tasks.',
 'reflect_on_tool_use': False,
 'tool_call_summary_format': '{result}'}
```


```python
# load 
reloaded_agent = AssistantAgent.load_component(agent_config)
```



Labels: proj-agentchat
State: closed
InitialMessages settings are not reflected in messages.

### Description

In useChat, initialMessages is no longer reflected in messages.
Instead of initialMessages, initial values are set in useChat's setMessages.

### Code example

```
export default function Chat({
    id, 
    initialMessages,
}: {
    id?: string | undefined;
    initialMessages?: Message[];
}) {

    const { input, status, handleSubmit, messages, stop, setInput, setMessages } =
    useChat({
        id, // use the provided chatId
        // initialMessages, // initial messages if provided  
        sendExtraMessageFields: true, // send id and createdAt for each message
        generateId: createIdGenerator({ prefix: 'msgc', size: 16 }), // id format for client-side messages  
        onFinish(message, { usage, finishReason }) {
            console.log('Usage', usage);
            console.log('FinishReason', finishReason);
        },
    });

    useEffect(() => {
        if (initialMessages)
            setMessages(initialMessages);
    }, [initialMessages, setMessages]);

```

### AI provider

@ai-sdk/react@1.1.19
ai@4.1.47

### Additional context

_No response_

Labels: bug, ai/ui
State: closed
Documentation on how to use structured output with AgentChat agents

### What feature would you like to be added?

Support for open ai structured output mode in agent chat. The ability to give an agent a Pydantic class that it must adhere to in the response.

This could be dumped to json as a message for intermediate agents?

### Why is this needed?

This would make integrating AutoGen with api requests easier. e.g. consider an answer agent at the end of a complex agentic group chat. It might need to formulate the answer in a specific json format, once it takes into account all the previous messages.

OpenAI structured data mode makes the likelihood of a correct JSON structure far more likely than prompting alone.

Labels: documentation, proj-agentchat
State: closed
Support declarative configs for BaseGroupChat (and derivatives) - RoundRobinGroupChat, SelectorGroupChat, MagenticOne etc

Support a workflow where an AgentChat team configuration can be made declarative

```python

# dump
assistant_agent = AssistantAgent(
    name="assistant_agent",
    model_client=OpenAIChatCompletionClient(
        model="gpt-4o-2024-08-06", 
    ),
    handoffs=["john_glover"]
)

term = MaxMessageTermination(2)
agent_team = RoundRobinGroupChat([assistant_agent], termination_condition=term)
agent_config = agent_team.dump_component()

```


```json
{
  "provider": "autogen_agentchat.teams.RoundRobinGroupChat",
  "component_type": "team",
  "version": 1,
  "component_version": 1,
  "description": null,
  "config": {
    "participants": [
      {
        "provider": "autogen_agentchat.agents.AssistantAgent",
        "component_type": "agent",
        "version": 1,
        "component_version": 1,
        "config": {
          "name": "assistant_agent",
          "model_client": {
            "provider": "autogen_ext.models.openai.OpenAIChatCompletionClient",
            "component_type": "model",
            "version": 1,
            "component_version": 1,
            "config": { "model": "gpt-4o-2024-08-06" }
          },
          "handoffs": [
            {
              "target": "john_glover",
              "description": "Handoff to john_glover.",
              "name": "transfer_to_john_glover",
              "message": "Transferred to john_glover, adopting the role of john_glover immediately."
            }
          ],
          "model_context": {
            "provider": "autogen_core.model_context.UnboundedChatCompletionContext",
            "component_type": "chat_completion_context",
            "version": 1,
            "component_version": 1,
            "config": {}
          },
          "description": "An agent that provides assistance with ability to use tools.",
          "system_message": "You are a helpful AI assistant. Solve tasks using your tools. Reply with TERMINATE when the task has been completed.",
          "reflect_on_tool_use": false,
          "tool_call_summary_format": "{result}"
        }
      }
    ],
    "termination_condition": {
      "provider": "autogen_agentchat.conditions.MaxMessageTermination",
      "component_type": "termination",
      "version": 1,
      "component_version": 1,
      "config": { "max_messages": 2 }
    }
  }
}

```

Labels: proj-agentchat
State: closed
Invalid finish_reason error in tool calls with anthropic endpoint

### What happened?

Models such as anthropic support only sequential function calling. Thus when we pass multiple tools while using an anthropic model, the function caling of subsequent tools is not handled.

and if we try to handle it ourselves, it fails again at finish_reason
ValidationError: 1 validation error for CreateResult
finish_reason
  Input should be 'stop', 'length', 'function_calls', 'content_filter' or 'unknown' [type=literal_error, input_value='end_turn', input_type=str]
    For further information visit https://errors.pydantic.dev/2.10/v/literal_error


### What did you expect to happen?

Either autogen allows the developer to handle the sequential call. Or autogen handles sequential call itself the way  it handles parallel tool calls for openai

### How can we reproduce it (as minimally and precisely as possible)?

async def run_task(agent, messages):
    """
    Sends messages to the agent and handles tool calls.

    Args:
        agent (AssistantAgent): The initialized AssistantAgent.
        messages (list): The conversation history.

    Returns:
        str: The final response after tool execution.
    """
    # Send messages to the agent
    cancellation_token = CancellationToken()

    response = await agent.on_messages(messages=messages,cancellation_token=cancellation_token)
    print("here 0")
    print(response)
    if response.inner_messages:
        print("here 0.1")
        for message in response.inner_messages:
            print(response.chat_message.content)
            print(type(response.chat_message.content))
            print("*********")
            if isinstance(message, ToolCallRequestEvent):
              tool_result_message = ToolCallSummaryMessage(
                          content=str(response.chat_message.content),
                          source="tool"
                      )
              response = await agent.on_messages(messages=[tool_result_message], cancellation_token=cancellation_token)
              break
    print("here 4")
    return response.chat_message.content


Above code works for openai. for anthropic, i will need to remove the break and handle subsequent function calling which results in failure

### AutoGen version

0.4.2

### Which package was this bug in

AgentChat

### Model used

sonnet3.5

### Python version

3.12

### Operating system

_No response_

### Any additional info you think would be helpful for fixing this bug

_No response_

State: closed
[Bug] 腾讯混元AI 点击获取模型列表的时候报错

### 📦 部署环境

Docker

### 📌 软件版本

v1.36.43

### 💻 系统环境

Windows

### 🌐 浏览器

Chrome

### 🐛 问题描述

![image](https://github.com/user-attachments/assets/5db6ecf2-7679-4dcb-ba89-8712bb5503d8)
已经确定了APIKey是正常的，连通性检查是正常的。
![image](https://github.com/user-attachments/assets/be355b86-9921-442c-a22d-19cdf6b85917)


### 📷 复现步骤

_No response_

### 🚦 期望结果

_No response_

### 📝 补充信息

_No response_

Labels: released
State: closed
[Bug] 讯飞星火 点击获取模型列表的时候报错

### 📦 部署环境

Docker

### 📌 软件版本

v1.36.43

### 💻 系统环境

Windows

### 🌐 浏览器

Chrome

### 🐛 问题描述

![image](https://github.com/user-attachments/assets/e29d2547-d059-47b2-bb24-ce76f4ba7136)
已经确定了APIKey是正常的，连通性检查是正常的。
![image](https://github.com/user-attachments/assets/c9996096-35fd-4d3f-8b06-7553aee5cf21)


### 📷 复现步骤

_No response_

### 🚦 期望结果

_No response_

### 📝 补充信息

_No response_

Labels: released
State: closed