Repository: dust-tt/dust
Pull request #12023: Add: ability to include generated images in the conversation.
Author: login: Fraggle
Created: 2025-04-16T07:36:34Z | Merged: 2025-04-17T08:34:59Z

## Description

Added support to include images in the conversation.
- Expand function call return value type to include structured output (supported by claude and required for it to see images).
- Make images "includable".
- Improve handling of tool results by supporting structured output for claude.
- OpenAI do not support structured output but figure out what to do with the image url in the text content.

## Tests

Did several tests, also checked that it still directly read images when they are submitted by the user.

## Risk

Medium, but the API is backward compatible.

## Deploy Plan

Deploy `core` then `front`


Diff:
diff --git a/core/src/providers/anthropic.rs b/core/src/providers/anthropic.rs
index a00db37ccce7..e2f7ab955be6 100644
--- a/core/src/providers/anthropic.rs
+++ b/core/src/providers/anthropic.rs
@@ -64,10 +64,27 @@ impl ToString for AnthropicChatMessageRole {
     }
 }
 
+#[derive(Serialize, Deserialize, Debug, Clone, PartialEq)]
+#[serde(rename_all = "snake_case")]
+pub enum AnthropicContentToolResultContentType {
+    Text,
+    Image,
+}
+
+#[derive(Debug, Serialize, Deserialize, PartialEq, Clone)]
+#[serde()]
+struct AnthropicContentToolResultContent {
+    r#type: AnthropicContentToolResultContentType,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    text: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    source: Option<AnthropicImageContent>,
+}
+
 #[derive(Debug, Serialize, Deserialize, PartialEq, Clone)]
 struct AnthropicContentToolResult {
     tool_use_id: String,
-    content: Option<String>,
+    content: Vec<AnthropicContentToolResultContent>,
 }
 
 #[derive(Debug, Serialize, Deserialize, PartialEq, Clone)]
@@ -229,31 +246,37 @@ async fn fetch_image_base64(image_url: &str) -> Result<(String, AnthropicImageCo
         },
     ))
 }
-
 async fn fetch_and_encode_images(
     messages: Vec<ChatMessage>,
 ) -> Result<HashMap<String, AnthropicImageContent>, anyhow::Error> {
     let futures = messages
         .into_iter()
         .filter_map(|message| {
-            if let ChatMessage::User(user_msg) = message {
-                if let ContentBlock::Mixed(mixed_content) = user_msg.content {
-                    let inner_futures = mixed_content
-                        .into_iter()
-                        .filter_map(|content| {
-                            if let MixedContent::ImageContent(ic) = content {
-                                let url = ic.image_url.url.clone();
+            let mixed_content = match message {
+                ChatMessage::User(user_msg) => match user_msg.content {
+                    ContentBlock::Mixed(content) => Some(content),
+                    _ => None,
+                },
+                ChatMessage::Function(func_msg) => match func_msg.content {
+                    ContentBlock::Mixed(content) => Some(content),
+                    _ => None,
+                },
+                _ => None,
+            };
 
-                                Some(async move { fetch_image_base64(&url).await })
-                            } else {
-                                None
-                            }
-                        })
-                        .collect::<Vec<_>>();
-                    return Some(inner_futures);
-                }
-            }
-            None
+            mixed_content.map(|content| {
+                content
+                    .into_iter()
+                    .filter_map(|content| {
+                        if let MixedContent::ImageContent(ic) = content {
+                            let url = ic.image_url.url.clone();
+                            Some(async move { fetch_image_base64(&url).await })
+                        } else {
+                            None
+                        }
+                    })
+                    .collect::<Vec<_>>()
+            })
         })
         .flatten()
         .collect::<Vec<_>>();
@@ -336,14 +359,44 @@ impl<'a> TryFrom<&'a ChatMessageConversionInput<'a>> for AnthropicChatMessage {
                 })
             }
             ChatMessage::Function(function_msg) => {
+                let content: Vec<AnthropicContentToolResultContent> = match &function_msg.content {
+                    ContentBlock::Text(t) => vec![AnthropicContentToolResultContent {
+                        r#type: AnthropicContentToolResultContentType::Text,
+                        text: Some(t.clone()),
+                        source: None,
+                    }],
+                    ContentBlock::Mixed(m) => m
+                        .into_iter()
+                        .map(|mb| match mb {
+                            MixedContent::TextContent(tc) => {
+                                Ok(AnthropicContentToolResultContent {
+                                    r#type: AnthropicContentToolResultContentType::Text,
+                                    text: Some(tc.text.clone()),
+                                    source: None,
+                                })
+                            }
+                            MixedContent::ImageContent(ic) => {
+                                if let Some(base64_data) = base64_map.get(&ic.image_url.url) {
+                                    Ok(AnthropicContentToolResultContent {
+                                        r#type: AnthropicContentToolResultContentType::Image,
+                                        source: Some(base64_data.clone()),
+                                        text: None,
+                                    })
+                                } else {
+                                    Err(anyhow!("Invalid Image."))
+                                }
+                            }
+                        })
+                        .collect::<Result<Vec<AnthropicContentToolResultContent>>>()?,
+                };
+
                 // Handling tool_result.
                 let tool_result = AnthropicContent {
                     r#type: AnthropicContentType::ToolResult,
                     tool_use: None,
                     tool_result: Some(AnthropicContentToolResult {
                         tool_use_id: function_msg.function_call_id.clone(),
-                        // TODO(2024-06-24 flav) This does not need to be Optionable.
-                        content: Some(function_msg.content.clone()),
+                        content: content,
                     }),
                     text: None,
                     source: None,
diff --git a/core/src/providers/chat_messages.rs b/core/src/providers/chat_messages.rs
index cc1c4d574e2a..6bc5aa1d1b12 100644
--- a/core/src/providers/chat_messages.rs
+++ b/core/src/providers/chat_messages.rs
@@ -89,7 +89,7 @@ pub struct AssistantChatMessage {
 #[derive(Debug, Serialize, Deserialize, PartialEq, Clone)]
 #[serde(deny_unknown_fields)]
 pub struct FunctionChatMessage {
-    pub content: String,
+    pub content: ContentBlock,
     pub function_call_id: String,
     pub name: Option<String>,
     pub role: ChatMessageRole,
diff --git a/core/src/providers/helpers.rs b/core/src/providers/helpers.rs
index 4d3957cddc12..972c4024e542 100644
--- a/core/src/providers/helpers.rs
+++ b/core/src/providers/helpers.rs
@@ -1,5 +1,5 @@
 use super::{
-    chat_messages::{AssistantChatMessage, ChatMessage, ContentBlock, UserChatMessage},
+    chat_messages::{AssistantChatMessage, ChatMessage, UserChatMessage},
     llm::ChatMessageRole,
 };
 
@@ -44,7 +44,7 @@ pub fn strip_tools_from_chat_history(messages: &Vec<ChatMessage>) -> Vec<ChatMes
             }
             ChatMessage::Function(message) => {
                 new_messages.push(ChatMessage::User(UserChatMessage {
-                    content: ContentBlock::Text(message.content.clone()),
+                    content: message.content.clone(),
                     role: ChatMessageRole::User,
                     name: message.name.clone(),
                 }));
diff --git a/core/src/providers/mistral.rs b/core/src/providers/mistral.rs
index 9ef29557eb60..297dbc3e971b 100644
--- a/core/src/providers/mistral.rs
+++ b/core/src/providers/mistral.rs
@@ -168,7 +168,35 @@ impl TryFrom<&ChatMessage> for MistralChatMessage {
             }),
             ChatMessage::Function(function_msg) => Ok(MistralChatMessage {
                 role: MistralChatMessageRole::Tool,
-                content: Some(function_msg.content.clone()),
+                content: match &function_msg.content {
+                    ContentBlock::Mixed(m) => {
+                        let result = m.iter().enumerate().try_fold(
+                            String::new(),
+                            |mut acc, (i, content)| {
+                                match content {
+                                    MixedContent::ImageContent(_) => {
+                                        Err(anyhow!("Vision is not supported for Mistral."))
+                                    }
+                                    MixedContent::TextContent(tc) => {
+                                        acc.push_str(&tc.text);
+                                        if i != m.len() - 1 {
+                                            // Add newline if it's not the last item.
+                                            acc.push('\n');
+                                        }
+                                        Ok(acc)
+                                    }
+                                }
+                            },
+                        );
+
+                        match result {
+                            Ok(text) if !text.is_empty() => Some(text),
+                            Ok(_) => None, // Empty string.
+                            Err(e) => return Err(e),
+                        }
+                    }
+                    ContentBlock::Text(t) => Some(t.clone()),
+                },
                 tool_calls: None,
                 tool_call_id: Some(sanitize_tool_call_id(&function_msg.function_call_id)),
             }),
diff --git a/core/src/providers/openai_compatible_helpers.rs b/core/src/providers/openai_compatible_helpers.rs
index f2fdf8729c8e..6f703a3507f2 100644
--- a/core/src/providers/openai_compatible_helpers.rs
+++ b/core/src/providers/openai_compatible_helpers.rs
@@ -778,24 +778,30 @@ fn to_openai_messages(
                 let all_contents_are_text = contents
                     .iter()
                     .all(|c| matches!(c, OpenAIContentBlock::TextContent(_)));
+                let is_tool_result = m.tool_call_id.is_some();
 
                 OpenAIChatMessage {
                     role: m.role,
                     name: m.name,
                     tool_call_id: m.tool_call_id,
                     tool_calls: m.tool_calls,
-                    content: match (contents.len(), all_contents_are_text, squash_text_contents) {
+                    content: match (
+                        contents.len(),
+                        all_contents_are_text,
+                        squash_text_contents,
+                        is_tool_result,
+                    ) {
                         // Case 0: there's no content => return None
-                        (0, _, _) => None,
+                        (0, _, _, _) => None,
                         // Case 1: there's only a single text content => use string format
-                        (1, true, _) => Some(OpenAIChatMessageContent::String(
+                        (1, true, _, _) => Some(OpenAIChatMessageContent::String(
                             match contents.into_iter().next().unwrap() {
                                 OpenAIContentBlock::TextContent(tc) => tc.text.clone(),
                                 _ => unreachable!(),
                             },
                         )),
                         // Case 2: There's more than one content, all contents are text and we want to squash them => squash them
-                        (_, true, true) => Some(OpenAIChatMessageContent::String(
+                        (_, true, true, _) => Some(OpenAIChatMessageContent::String(
                             contents
                                 .into_iter()
                                 .map(|c| match c {
@@ -805,8 +811,21 @@ fn to_openai_messages(
                                 .collect::<Vec<String>>()
                                 .join("\n"),
                         )),
-                        // Case 3: there's more than one content, the content isn't text or we don't want to squash them => keep structured format
-                        (_, _, _) => Some(OpenAIChatMessageContent::Structured(contents)),
+                        // Case 3: it's a tool result, openai only supports string format for tool results, return the stringified content as a string
+                        (_, _, _, true) => Some(OpenAIChatMessageContent::String(
+                            contents
+                                .into_iter()
+                                .map(|c| match c {
+                                    OpenAIContentBlock::TextContent(tc) => tc.text.clone(),
+                                    OpenAIContentBlock::ImageContent(ic) => {
+                                        ic.image_url.url.clone()
+                                    }
+                                })
+                                .collect::<Vec<String>>()
+                                .join("\n"),
+                        )),
+                        // Case 4: there's more than one content, the content isn't text or we don't want to squash them => keep structured format
+                        (_, _, _, _) => Some(OpenAIChatMessageContent::Structured(contents)),
                     },
                 }
             }
diff --git a/front/admin/cli.ts b/front/admin/cli.ts
index 7d69db959d3e..8739840dda69 100644
--- a/front/admin/cli.ts
+++ b/front/admin/cli.ts
@@ -366,7 +366,10 @@ const conversation = async (command: string, args: parseArgs.ParsedArgs) => {
             if (m.role === "function") {
               return {
                 ...m,
-                content: m.content.slice(0, 200) + "...",
+                content:
+                  typeof m.content === "string"
+                    ? m.content.slice(0, 200) + "..."
+                    : m.content,
               };
             }
             return m;
diff --git a/front/lib/actions/conversation/include_file.ts b/front/lib/actions/conversation/include_file.ts
index 244b7e33c5fb..6d1e88c0b49e 100644
--- a/front/lib/actions/conversation/include_file.ts
+++ b/front/lib/actions/conversation/include_file.ts
@@ -26,11 +26,20 @@ import type {
   ConversationType,
   FunctionCallType,
   FunctionMessageTypeModel,
+  ImageContent,
   ModelConfigurationType,
   ModelId,
   Result,
+  TextContent,
+} from "@app/types";
+import {
+  assertNever,
+  CoreAPI,
+  Err,
+  isImageContent,
+  isTextContent,
+  Ok,
 } from "@app/types";
-import { CoreAPI, Err, isTextContent, Ok } from "@app/types";
 
 const CONTEXT_SIZE_DIVISOR_FOR_INCLUDE = 4;
 
@@ -106,7 +115,10 @@ export class ConversationIncludeFileActionType extends BaseAction {
     conversation: ConversationType,
     model: ModelConfigurationType
   ): Promise<
-    Result<{ fileId: string; title: string; content: string }, string>
+    Result<
+      { fileId: string; title: string; content: ImageContent | TextContent },
+      string
+    >
   > {
     // Note on `contentFragmentVersion`: two content fragment versions are created with different
     // fileIds. So we accept here rendering content fragments that are superseded. This will mean
@@ -123,13 +135,16 @@ export class ConversationIncludeFileActionType extends BaseAction {
           return new Ok({
             fileId,
             title: f.title,
-            content: CONTENT_OUTDATED_MSG,
+            content: {
+              type: "text",
+              text: CONTENT_OUTDATED_MSG,
+            },
           });
         }
 
         const r = await renderFromAttachmentId(conversation.owner, {
           contentType: f.contentType,
-          excludeImages: true,
+          excludeImages: false,
           conversationAttachmentId: conversationAttachmentId(f),
           model,
           title: f.title,
@@ -139,14 +154,17 @@ export class ConversationIncludeFileActionType extends BaseAction {
         if (r.isErr()) {
           return new Err(`${r.error}`);
         }
-        if (!isTextContent(r.value.content[0])) {
-          return new Err(`File \`${fileId}\` has no text content`);
+        if (
+          !isTextContent(r.value.content[0]) &&
+          !isImageContent(r.value.content[0])
+        ) {
+          return new Err(`File \`${fileId}\` has no text or image content`);
         }
 
         return new Ok({
           fileId,
           title: f.title,
-          content: r.value.content[0].text,
+          content: r.value.content[0],
         });
       }
     }
@@ -172,7 +190,7 @@ export class ConversationIncludeFileActionType extends BaseAction {
     conversation: ConversationType;
     model: ModelConfigurationType;
   }): Promise<FunctionMessageTypeModel> {
-    const finalize = (content: string) => {
+    const finalize = (content: string | ImageContent[]) => {
       return {
         role: "function" as const,
         name:
@@ -183,14 +201,14 @@ export class ConversationIncludeFileActionType extends BaseAction {
       };
     };
 
-    const textRes =
+    const fileRes =
       await ConversationIncludeFileActionType.fileFromConversation(
         this.params.fileId,
         conversation,
         model
       );
-    if (textRes.isErr()) {
-      return finalize(`Error: ${textRes.error}`);
+    if (fileRes.isErr()) {
+      return finalize(`Error: ${fileRes.error}`);
     }
 
     if (this.tokensCount === null) {
@@ -217,7 +235,13 @@ export class ConversationIncludeFileActionType extends BaseAction {
       );
     }
 
-    return finalize(textRes.value.content);
+    if (isTextContent(fileRes.value.content)) {
+      return finalize(fileRes.value.content.text);
+    } else if (isImageContent(fileRes.value.content)) {
+      return finalize([fileRes.value.content]);
+    }
+
+    assertNever(fileRes.value.content);
   }
 }
 
@@ -356,31 +380,40 @@ export class ConversationIncludeFileConfigurationServerRunner extends BaseAction
       return;
     }
 
-    const coreAPI = new CoreAPI(config.getCoreAPIConfig(), logger);
-    const tokensRes = await coreAPI.tokenize({
-      text: fileRes.value.content,
-      providerId: model.providerId,
-      modelId: model.modelId,
-    });
+    let tokensCount: number | null = null;
+    if (isTextContent(fileRes.value.content)) {
+      const coreAPI = new CoreAPI(config.getCoreAPIConfig(), logger);
+      const tokensRes = await coreAPI.tokenize({
+        text: fileRes.value.content.text,
+        providerId: model.providerId,
+        modelId: model.modelId,
+      });
+
+      if (tokensRes.isErr()) {
+        yield {
+          type: "conversation_include_file_error",
+          created: Date.now(),
+          configurationId: agentConfiguration.sId,
+          messageId: agentMessage.sId,
+          error: {
+            code: "conversation_include_file_error",
+            message: `Error including conversation file: ${tokensRes.error}`,
+          },
+        };
+        return;
+      }
 
-    if (tokensRes.isErr()) {
-      yield {
-        type: "conversation_include_file_error",
-        created: Date.now(),
-        configurationId: agentConfiguration.sId,
-        messageId: agentMessage.sId,
-        error: {
-          code: "conversation_include_file_error",
-          message: `Error including conversation file: ${tokensRes.error}`,
-        },
-      };
-      return;
+      tokensCount = tokensRes.value.tokens.length;
+    } else if (isImageContent(fileRes.value.content)) {
+      tokensCount = 0;
+    } else {
+      assertNever(fileRes.value.content);
     }
 
     // Store the tokens count and file title on the action model for use in the rendering of the
     // action for the model (token count) and the rendering of the action details (file title).
     await action.update({
-      tokensCount: tokensRes.value.tokens.length,
+      tokensCount,
       fileTitle:
         fileRes.value.title?.length > 255
           ? `...${fileRes.value.title?.slice(-252)}`
@@ -397,7 +430,7 @@ export class ConversationIncludeFileConfigurationServerRunner extends BaseAction
         params: {
           fileId,
         },
-        tokensCount: tokensRes.value.tokens.length,
+        tokensCount,
         fileTitle: fileRes.value.title,
         functionCallId,
         functionCallName: actionConfiguration.name,
diff --git a/front/lib/actions/mcp.ts b/front/lib/actions/mcp.ts
index 4744a668283c..50e4707f08c4 100644
--- a/front/lib/actions/mcp.ts
+++ b/front/lib/actions/mcp.ts
@@ -12,6 +12,7 @@ import {
   augmentInputsWithConfiguration,
   hideInternalConfiguration,
 } from "@app/lib/actions/mcp_internal_actions/input_schemas";
+import { isResourceWithName } from "@app/lib/actions/mcp_internal_actions/output_schemas";
 import { getMCPEvents } from "@app/lib/actions/pubsub";
 import type { DataSourceConfiguration } from "@app/lib/actions/retrieval";
 import type { TableDataSourceConfiguration } from "@app/lib/actions/tables_query";
@@ -510,10 +511,16 @@ export class MCPConfigurationServerRunner extends BaseActionConfigurationServerR
         c.resource.mimeType &&
         isSupportedFileContentType(c.resource.mimeType)
       ) {
+        let fileName = c.resource.uri.split("/").pop() ?? "generated-file";
+
+        if (isResourceWithName(c.resource)) {
+          fileName = c.resource.name;
+        }
+
         const r = await processAndStoreFromUrl(auth, {
           url: c.resource.uri,
           useCase: "conversation",
-          fileName: c.resource.uri.split("/").pop() ?? "generated-file",
+          fileName,
           contentType: c.resource.mimeType,
         });
         if (r.isErr()) {
diff --git a/front/lib/actions/mcp_internal_actions/output_schemas.ts b/front/lib/actions/mcp_internal_actions/output_schemas.ts
new file mode 100644
index 000000000000..e222ef7f9639
--- /dev/null
+++ b/front/lib/actions/mcp_internal_actions/output_schemas.ts
@@ -0,0 +1,9 @@
+type ResourceWithName = {
+  name: string;
+};
+
+export const isResourceWithName = (
+  resource: object
+): resource is ResourceWithName => {
+  return "name" in resource && typeof resource.name === "string";
+};
diff --git a/front/lib/actions/mcp_internal_actions/servers/image_generation.ts b/front/lib/actions/mcp_internal_actions/servers/image_generation.ts
index 0f17fceef2d3..297cde76d1d9 100644
--- a/front/lib/actions/mcp_internal_actions/servers/image_generation.ts
+++ b/front/lib/actions/mcp_internal_actions/servers/image_generation.ts
@@ -33,6 +33,12 @@ const createServer = (auth: Authenticator): McpServer => {
         .describe(
           "A text description of the desired image(s). The maximum length is 4000 characters."
         ),
+      name: z
+        .string()
+        .max(32)
+        .describe(
+          "The name of the image. The maximum length is 32 characters."
+        ),
       quality: z
         .enum(["standard", "hd"])
         .optional()
@@ -55,7 +61,7 @@ const createServer = (auth: Authenticator): McpServer => {
           "The size of the generated images. Must be one of 1024x1024, 1792x1024, or 1024x1792"
         ),
     },
-    async ({ prompt, quality, style, size }) => {
+    async ({ prompt, name, quality, style, size }) => {
       // Crude way to rate limit the usage of the image generation tool.
       //
       const remaining = await rateLimiter({
@@ -92,12 +98,15 @@ const createServer = (auth: Authenticator): McpServer => {
         user: `workspace-${auth.getNonNullableWorkspace().sId}`,
       });
 
+      const fileName = name + ".png";
+
       const content = images.data.map((image) => ({
         type: "resource" as const,
         resource: {
           mimeType: "image/png",
           uri: image.url!,
-          text: `Your image was generated successfully.`,
+          text: `Your image ${fileName} was generated successfully.`,
+          name: fileName,
         },
       }));
 
diff --git a/front/lib/api/assistant/jit_utils.ts b/front/lib/api/assistant/jit_utils.ts
index a65e257d7b40..6a87b9cfd486 100644
--- a/front/lib/api/assistant/jit_utils.ts
+++ b/front/lib/api/assistant/jit_utils.ts
@@ -25,10 +25,6 @@ import {
 function isConversationIncludableFileContentType(
   contentType: SupportedContentFragmentType
 ): boolean {
-  // We allow including everything except images.
-  if (isSupportedImageContentType(contentType)) {
-    return false;
-  }
   if (isDustMimeType(contentType)) {
     return isIncludableInternalMimeType(contentType);
   }
diff --git a/front/lib/api/assistant/utils.ts b/front/lib/api/assistant/utils.ts
index 9cc7c353773d..3a29b3a6257c 100644
--- a/front/lib/api/assistant/utils.ts
+++ b/front/lib/api/assistant/utils.ts
@@ -6,12 +6,16 @@ export function getTextContentFromMessage(
 ): string {
   const { content } = message;
 
+  if (!content) {
+    return "";
+  }
+
   if (typeof content === "string") {
     return content;
   }
 
-  if (!content) {
-    return "";
+  if (isImageContent(content)) {
+    return content.image_url.url;
   }
 
   return content
diff --git a/front/types/assistant/generation.ts b/front/types/assistant/generation.ts
index 5b4cda8afb77..8a1d3e8a705d 100644
--- a/front/types/assistant/generation.ts
+++ b/front/types/assistant/generation.ts
@@ -19,7 +19,7 @@ export interface ImageContent {
   };
 }
 
-interface TextContent {
+export interface TextContent {
   type: "text";
   text: string;
 }
@@ -71,7 +71,7 @@ export interface FunctionMessageTypeModel {
   role: "function";
   name: string;
   function_call_id: string;
-  content: string;
+  content: string | Content[];
 }
 
 export type ModelMessageTypeMultiActions =


Reviews:
user: login: Fraggle
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T10:51:39Z
user: login: dust-agent[bot]
body: Great work on adding image support! Just spotted a small style issue.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T17:02:26Z
user: login: dust-agent[bot]
body: Thanks for this PR adding support for including generated images in conversations! This is a solid implementation that handles both Claude and OpenAI models appropriately.

**Strengths:**
- Clean implementation of structured output support for Claude with proper base64 handling
- Good backwards compatibility maintained for the API
- Reuses existing content fragment infrastructure
- Aligns well with JIT datasources design

**Suggestions:**
1. Consider documenting the supported structured output formats for different providers since they have slightly different requirements. This would help with future maintenance.

2. Consider adding more robust error handling for image processing failures, since image generation can be flaky. The current implementation handles basic cases but may benefit from more graceful degradation.

**Future Improvements to Consider:**
- Could explore upgrading to newer image generation models like Fireworks AI's Flux which is reportedly better than DALL-E
- May want to add a way to preview/verify generated images before including them in conversations, similar to the action approval flow being developed

**Minor Issue:**
- The Rust struct field `fileId` should follow snake_case convention and be renamed to `file_id`

Overall this is a solid PR that adds important functionality while maintaining code quality and backwards compatibility. I'm leaving this as a comment rather than requesting changes since the only real issue is the field naming convention which has already been noted.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T18:50:55Z
user: login: spolu
body: If I understand correctly the goal is to model properly generated images as tool output when feeding back conversation state to the model.

What's the story for providers that don't support it? I don't see a google diff here?

Overall LGTM deferring final review to @flvndvd or @fontanierh 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T20:42:48Z
user: login: flvndvd
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T07:47:10Z

Review comments:
diff_hunk: @@ -64,10 +64,27 @@ impl ToString for AnthropicChatMessageRole {
     }
 }
 
+#[derive(Serialize, Deserialize, Debug, Clone, PartialEq)]
+#[serde(rename_all = "snake_case")]
+pub enum AnthropicContentToolResultContentType {
+    Text,
+    Image,
+}
+
+#[derive(Debug, Serialize, Deserialize, PartialEq, Clone)]
+#[serde()]
+struct AnthropicContentToolResultContent {
+    r#type: AnthropicContentToolResultContentType,
+    #[serde(skip_serializing_if = "Option::is_none")]
path: core/src/providers/anthropic.rs
user: login: Fraggle
body: this one lost me an hour
created_at: 2025-04-16T10:51:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -64,10 +64,27 @@ impl ToString for AnthropicChatMessageRole {
     }
 }
 
+#[derive(Serialize, Deserialize, Debug, Clone, PartialEq)]
+#[serde(rename_all = "snake_case")]
+pub enum AnthropicContentToolResultContentType {
+    Text,
+    Image,
+}
+
+#[derive(Debug, Serialize, Deserialize, PartialEq, Clone)]
+#[serde()]
+struct AnthropicContentToolResultContent {
+    r#type: AnthropicContentToolResultContentType,
+    #[serde(skip_serializing_if = "Option::is_none")]
path: core/src/providers/anthropic.rs
user: login: flvndvd
body: Serde is 🔥 
created_at: 2025-04-17T07:34:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -510,10 +510,15 @@ export class MCPConfigurationServerRunner extends BaseActionConfigurationServerR
         c.resource.mimeType &&
         isSupportedFileContentType(c.resource.mimeType)
       ) {
+        const fileName =
+          "name" in c.resource && typeof c.resource.name === "string"
path: front/lib/actions/mcp.ts
user: login: flvndvd
body: `nit` Typeguard
created_at: 2025-04-17T07:46:12Z
author_association: CONTRIBUTOR

PR comments:
user: login: Fraggle
created_at: 2025-04-17T04:36:47Z
body: > If I understand correctly the goal is to model properly generated images as tool output when feeding back conversation state to the model.

Exactly, currently they are presented as part of the serialized JSON output which isn't handled correctly by Claude. Claude expect a tool output "content" to be either a string or an array of content blocks with type text or image (like a user message content blocks) => [doc](https://docs.anthropic.com/en/docs/build-with-claude/tool-use/overview#handling-tool-use-and-tool-result-content-blocks) (bullet point 3.)

> What's the story for providers that don't support it? I don't see a google diff here?

I expected rust to be my guard since I changed the type of the global `FunctionChatMessage` struct.
It did error in mistral and the open ai compatible providers helper which I fixed. I imagine that google is using the open ai compatible layer but I'll double check.
author_association: CONTRIBUTOR
user: login: Fraggle
created_at: 2025-04-17T04:39:19Z
body: > I imagine that google is using the open ai compatible layer but I'll double check.

[Yep, it uses the open ai compatible helpers.](https://github.com/dust-tt/dust/blob/a9b633c73592a7c313592d52afbcae598bf158cb/core/src/providers/google_ai_studio.rs#L146)
author_association: CONTRIBUTOR
user: login: spolu
created_at: 2025-04-17T06:46:50Z
body: Copy that. Thanks 🙏 
author_association: CONTRIBUTOR