Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory
Authors: Lin Long
Yichen He
Wentao Ye
Yiyuan Pan
Yuan Lin
Hang Li
Junbo Zhao
Wei Li
Venue: ICLR 2026 Poster (2026)
Keywords: multimodal agent
long-term memory
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: We introduce M3-Agent, a novel multimodal agent framework equipped with long-term memory. Like humans, M3-Agent can process real-time visual and auditory inputs to build and update episodic and semantic memories, gradually accumulating world knowledge. Its memory is organized in an entity-centric, multimodal manner, enabling deeper and more consistent understanding of the environment. Given an instruction, M3-Agent autonomously performs multi-turn reasoning and retrieves relevant memories to complete tasks. To evaluate memory effectiveness and memory-based reasoning in multimodal agents, we develop M3-Bench, a long-video question answering benchmark comprising 100 newly recorded robot-perspective videos (M3-Bench-robot) and 920 diverse web-sourced videos (M3-Bench-web). We annotate QA pairs designed to test capabilities essential for agent applications, such as person understanding, general knowledge extraction, and cross-modal reasoning. Experimental results show that M3-Agent, trained via reinforcement learning, outperforms the strongest baseline, a prompting agent using Gemini-1.5-pro and GPT-4o, achieving 6.7%, 7.7%, and 5.3% higher accuracy on M3-Bench-robot, M3-Bench-web and VideoMME-long, respectively. Our work advances multimodal agents toward more human-like long-term memory and provides insights for their practical design. Models, datasets and code are available at https://github.com/ByteDance-Seed/m3-agent.

Published as a conference paper at ICLR 2026
SEEING, LISTENING, REMEMBERING, AND REASON- ING: A MULTIMODAL AGENT WITH LONG-TERM MEMORY
Lin Long Yichen Hel-x, Wentao Ye Yiyuan Pansl Yuan Lin Hang Li', Junbo Zhao?, Wei Li
1 Bytedance Seed (hyc, linyuan. 0, 1 iihang. 1h, 1i liwei. 85) Bbytedancecccoo
2 Zhejiang University
{1long, 2ha en
3 Robotics Institute, Carnegie Mellon University
yiyuanp@andrew. cmu.edu
ABSTRACT
We introduce M3-Agent, a novel multimodal agent framework equipped with long-term memory. Like humans, M3-Agent can process real-time visual and auditory inputs to build and update episodic and semantic memories, gradually accumulating world knowledge. Its memory is organized in an entity-centric, multimodal manner, enabling deeper and more consistent understanding of the environment. Given an instruction, M3-Agent autonomously performs multi- turn reasoning and retrieves relevant memories to complete tasks. To evalu- ate memory effectiveness and memory-based reasoning in multimodal agents, we develop M3-Bench, long-video question answering benchmark compris- ing 100 newly recorded robot-perspective videos (M3-Bench-robol, and 920 di- verse web-sourced videos to test capabilities essential for agent applications. such as person understand- ing, general knowledge extraction, and moosl reasoning. Experimental re- sults show that M3-Agent, trained via reinforcement learning, outperforms the strongest baseline, prompting agent using Gemini-1.5-pre and GPT--4,, achiev- ing 6.7%, 7.7%, and 5.3% higher accuracy on M33Bencc-roobot, M3-Bench-wee web and VideoMME-long, respectively. Our work advances multimodal agents to- ward more human-like long-term memory and provides insights for their practical design. Models, datasets and code are available at https:/ //gitub..... com/ ByteDance-Seee aggnt...
We annotate QA pairs designed
8:00
7:30
Multimodal agents continuously perceive environment, build entity- centric, long- mamories, reason
'Equal contribution.
1 Cooresponding author.
Work done during ByteDance Seed internship.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Imagine that in the future household robot autonomously carry out household tasks without your explicit instructions; it must have learned the operational rules of your home through daily experiences. In the morning, it hands you cup of coffee without asking "coffee tea?", because it has gradually formed memory of you, tracking your preferences and routines through long- term interactions. For multimodal agent, achieving this level of intelligence fundamentally relies on three capabilities: (1) continuously perceiving the world via multimodal sensors; (2) storing and organizing experiences into a long-term memory, and gradually building knowledge of its environment; (3) reasoning over this accumulated memory to guide its actions.
To achieve the goals, we propose M3-Agent, a novel multimodal agent framework equipped with long-term memory. As shown in Figure 1, it operates through two parallel processes: memoriza- tion, which continuously perceives real-time multimodal inputs to construct and update long-term memory; and control, which interprets external instructions, reasons over the stored memory, and executes the corresponding tasks.
During memorization, M3-Agent processes the incoming video stream, capturing both fine- fine-grained details and high-level abstractions by generating two types of memory, analogous to human cognitive systems (Tulving, 1972; 1985): Episodic memory corresponds to concrete events observed within the video (e.g.. "Alice takes the coffee and says, can't go without this in the morning." "" and "Alice throws an empty bottle into the green garbage bin."), and semantic memory refers to general knowledge that can be derived from the clip (e.g.. "Alice prefers to drink coffee in the morning" and "The green garbage bin used for recycling.").
The generated contents are then integrated into the agent's long-term memory, which supports mul- timodal information such faces, voices, and textual knowledge. The memory is organized in an entity-centric structure. For example, information related to the same person (e.g., their face, voice, and associated knowledge) connected within graph, as shown in Figure These connections are incrementally established as the agent extracts and integrates episodic and semantic memory.
ISGATICO
Fice
A
+ WBU
Audio
Mune Soranr When Stree
MLLM
MLLM
[UPOATE]
Term
Carc
ESCARCH
aplaodic
Environment
Ressoning
[ANSWER]
Memory
Response
Memorization Workflow
M3-Agent
Control Workflow
Figure I: Architecture of MB-Agent, comprising a multimodal large language model (MLLM) and a multimodal long-term memory. supports two parallel processes: memorization and control. The entity-centric memory structure enables accumulate world knowledge and maintain consistent, context-rich memory over time.
During control, M3-Agent leverages its long-term memory to reason and complete tasks. It au- tonomously retrieves relevant information from its memory across different dimensions (e.g., events and characters). Unlike standard etrienal-augmented generation (RAG) (Lewis et al., 2020), which performs single-turn memory injection into the context, apply RL in M3-Agent to enable multi- turn reasoning and iterative memory access (Jin et al., 2025), resulting in more focused and context- aware retrieval.
Such neemory-aggmente system extends beyond conventional long video description (Han et al., 2023; Zhang et al.. 2024a; Islam al., 2024), presenting two key challenges: (1) Infinite informa- tion processing. Existing methods for long video understanding optimize architectural efficiency process longer, but still finite, offline videos (Song al., 2024; Shen al., 2024; Zhang al.,
2

---

Published as a conference paper at ICLR 2026
2024b; He et al., 2024a; Shu et al., 2025). In contrast, M3-Agent continuously processes arbi- trarily long streams in an online manner, more closely mimicking how human long-term memory forms, through ongoing perception and incremental experience integration. (2) World knowledge construction. Traditional video description (Li et al., 2023; Lin et al., 2023a; 2024; Yuan et al., 2025; Bai et al., 2025) often focuses on low-level visual details while overlooking high-level world knowledge (Qiao et al., 2024; Ivanova et al., 2024; Fung et al., 2025) such as character identity and entity attributes, which may lead to ambiguity and inconsistency in long-term contexts. M3-Agent addresses this by incrementally building an entity-centric memory that encodes rich multimodal representations of key entities, enabling coherent and consistent long-term memory.
We evaluate M3-Agent on long video question answering (LVQA), using long videos to simulate the multimodal input streams received by agents. Existing LVQA benchmarks (Fu al., 2025; Zhou et al., 2025; Chandrasegaran et al., 2024; Wu et al., 2024) mainly target on visual perception (e.g., action recognition, spatial/temporal perception) but overlook higher-level cognitive abilities that rely long-term memory and are crucial for real-world agents, such as person understanding, general knowledge extraction, and cross-modal reasoning. To fill this gap, we introduce M3-Bench, new LVQA benchmark designed to evaluate multimodal agent's ' ability to reason memory. M3-Bench includes two subsets: (1) MS--Bench-robot, with 100 real-world videos recorded from a robot's perspective, and (2) M3-Bench-wee -eeb,,hwwb,, with 920 YouTube videos covering diverse scenarios. We define five question types (Table 1) targeting different aspects of memory-based reasoning, and annotate 1,276 and 3,214 QA pairs for 113-Bench-robor and M3-Bench-web, respectively.
We conduct experiments on M3-Bench-robot, M3-Bench-web, and VideoMME-long (Fu et al., 2025). M3-Agent, trained via reinforcement learning, outperforms all baselines on all three bench- marks. Compared to the strongest baseline, Gemini-GPT4o-Hyybne framework by prompting advanced proprietary models for both memorization and control, M3- Agent achieves 6.7%, 7.7%, and 5.3% higher accuracy on VideoMMMF-long, respectively. Extensive ablation studies further demonstrate the importance of long-term memory, particularly semantic memory, which substantially improves accuracy across all tasks. Ablation studies also highlight the significance of reinforcement learning and multi-turn reasoning for effective control process, enabling M3-Agent to solve more complex QA tasks.
which implements M3-Agent
M3-Bench-wed and
The main contributions of this paper are summarized as follows:
We introduce M3-Agent, novel framework for multimodal agents with long-term memory. M3- Agent continuously processes real-time multimodal inputs (seeing and listening), incrementally builds world knowledge by generating episodic and semantic memories (remembering), and reasons over these memories to complete complex question answering tasks (reasoning).
We develop M3-Bench, a new LVQA benchmark designed to evaluate the effectiveness of mem- ory and memory-based reasoning for multimodal agents.
Our experiments demonstrate that M3-Agent, trained by reinforcement learning, consistently outperforms other baselines across multiple tasks.
2 RELATED WORK
Long-Tere Term Memory of AI Agents. Long-term memory is essential for Al agents (Feng et al., 2024). A common approach to constructing memory to append complete agent trajectories, such as dialogues (Mei et al., 2024; Wang et al., 2023; Liu et al., 2024a; Zhong et al., 2024) or execution trajectories (Liu al., 2024c: Wang et al., 2024b; Shang et al., 2024; Liu al., 2024a; Hu al., 2024: Sarch et al., 2023), sometimes enhanced with summarization (Wang et al., 2023; Li et al., 2024a; Hu et al., 2024; Zhong al., 2024), latent embeddings (Zhang al., 2024b; Liu et al.,
2024b; Song et al., 2024; Diko et al., 2025). or structured knowledge representations (Ocker et al., 2025; Xu et al., 2025b). Chhikara et al. (2025); Wang et al. (2023); Kang et al. (2025) introduce more sophisticated memory architectures for fine-grained control. While most existing efforts focus on LLM agents, multimodal agents process richer, more diverse inputs, which poses challenges in maintaining consistent long-term memory Fan al. (2024b;a). Moreover, their ability construct world knowledge through lived experience, rather than static description, remains unexplored.
Online Video Understanding. For multimodal agents, memory formation is closely related to line video understanding, which requires real-time stream processing and decision-making based on past observations. Traditional methods, such extending context windows (Chen al., 2024b;
3

---

Published as a conference paper at ICLR 2026
Zhang et al., 2024d) or compressing visual tokens (Wu et al., 2022; Lan al., 2024; Wu et al., 2022), do not scale to infinite streams, as reprocessing full histories for each instruction is compu- tationally prohibitive. Memory-based methods (Zhang et al., 2024b: He et al., 2024a; Song et al., 2024; Zhang et al., 2024c) improve scalability by storing encoded visual features for retrieval; but they often fail to ensure long-term consistency, losing track of identities or evolving events. With the rise of large multimodal and language models (Hurst et al., 2024; Team et al., 2024; Yang et al., 2025; Bai al., 2025; Yuan et al., 2025), the Socratic Models framework (Zeng et al., 2022; Lin et al., 2023b; Zhang et al., 2024a) offers scalable alternative by generating language based video memories. VideoAgent (Fan et al., 2024b) employs prompt engineering to enable agentic workflows for long video understanding. Despite this, it still struggles to maintain consistency over complex, evolving content and lacks a clear training and optimization approach. Story (He et al., 2024b) introduces a method for person identity tracking in long videos, but it requires full access to the entire video, making it unsuitable for streaming scenarios.
DATASETS
We first introduce M3-Bench, an LVQA dataset designed to evaluate the capability of multimodal agents to perform reasoning over long-term memory. Each instance in M3-Bench comprises long video simulating the perceptual input of an agent, along with a series of open-ended question-answer pairs. The dataset is organized into two subsets: (1) MS-Bench--robo, which contains 100 real-world videos recorded from a robot's egocentric perspective, and (2) M3-Bench-web, which includes 920 web-sourced videos covering a wider variety of content and scenarios. To comprehensively assess an agent's 's ability to recall past observations and perform wemorrybbseed reasoning, we curate five distinct types of questions, as summarized in Table Overall, M3-Bench is featured by (1) long- duration, real-world videos that encompass diverse real-life scenarios relevant to the deployment of multimodal agents, and (2) challenging questions that extend beyond shallow perceptual under- standing and require complex reasoning over long-term contexts.
Table I: Explanations of different question types and their corresponding examples in M3-Bench.
Question Type
Explanation and Example
This requires aguregating multiple pieces of information distributed across the video. Example: Which collection has the highest starting price among the five itemss shown in the video? The agent must ideatify the starting price from five segments and compare them to lind the highest.
Multi-evidence
Reasoning
This involves stp-by-step reasoning actoss different segments to reach a conclusion.
Multi-hop
Example: Which bubble shop did they visit after going Ding Cha? The agent must first locate
Reasoning
Ding Cha, then follow subsequent segments to identify the next bubble tea shop.
This requires reasoning across multiple modalities, such as visual and audio content.
Cross-modal Reasoning
Example: (Bob shows Robot red folder and says, "Confudential documents go in this folder." then shows white folder and says, "Normal documents go in this one"") Which folder should confidential documents be placed in? The agent must combine
cues (folder color) and verbal cues answer.
This involves reasoning about person-related attributes such as identity emotions, personality, or relationships. Example: Is Lueas skilled at cooking? The video does not directly reveal the answer, but agent must aggregate Lucas's ' behavior across multiple cooking seenes to infer his skill level.
Person Under- standing
This evaluates whether the agent can extract general knowledge from specific events.
General Knowl- edge Extraction
Example: person classified different grocerie into various shelves of a refrigenator) Which shelf suitable Jor storing vegetables? The agent must recognize typical storage rules from its observation to answer the question.
Figure shows examples from M3-Bench. Figure presents overall dataset statistics, and Table compares M3-Bench with existing LVQA benchmarks. The rest of this section details the data collection and annotation processes for Mo--eench-robot and M3-Bench-web.
3.1 M3-BENCH-ROREE ROBOT
We first focus on robots, one of the most representative embodiments of multimodal agents. A general-purpose robot should maintain long-term memory and reason with it to guide its actions. For example, a houschold robot might need to remember a person's 'ss name or their coffee preferences. Reasoning over such memory supports higher-level cognition, such as inferring personality traits, social relationships, and object functions. To evaluate these abilities, we construct a complete data pipeline from recording egocentric videos from robot's perspective to manual QA annotation.
Script Design We first design video scripts across seven everyday settings: living room, kitchen, bedroom, study, office, meeting room, and gym. Each script involves robot interacting with
4

---

Published as a conference paper at ICLR 2026
Hird
"Plcas refigardaa""
Annlenclle radau anfrradkn hag.iianiniirttiinnn dall carultz Anosers
Qusetion: Whin geragee
Caryedardly
socaly pat ofde thy beaght
Oheend Counc Frtvarkee
200 $2,500
2
$6,000
653 925 ,00
12:55 $22,000
FEST $2.0,
(keet' segress
heepare
Quedlare: Winch
colletion wendrelllttte
las wwllehhsisteeee heg.htstaaaaai anoeg the
parting Answvrr Piraa ftear
abows
Vanter
Beatty Show
10:00
Did skiled Lucco skiled
letiigg se soomple
fheskbos
Challenge: Skantly who Lace suaces creking lond. Ancoves
Unnk stancing
Figure 2: Examples from M3-Bench.
Long Lnd
lostton
calegrry Mi-Badbechh
Distibution quesiio
Figure 3: M3-Bench Statistics. Each question may correspond to multiple question types.
two four humans. To promote diversity, we multiple thematic variations for each scenario (e.g. family talks or holiday parties the living room). Each script consists of at least 70 sequential events with at least 15 questions belong to types in Table 1, An example is shown in Table 7 ($ B.6).
Video Filming Recording with real robots is costly and complex. Therefore, we employ human actors simulate robot behavior. For each script, one actor acts as the robot perform daily tasks, while wearing mead-mounted camera to capture egocentric video and audio. To ensure diversity, we recruit 67 actors and film across 51 locations, with no more than three videos recorded per site.
Annotations After recording the videos, annotators curate QA pairs by reviewing the pre-scripted questions. Since some questions become invalid due deviations during filming, annotators de- termine whether to retain, revise, discard each one accordingly. For valid questions, they also annotate the timestamp which the question should be asked. Additional questions (conforming to types in Table 1) are created when necessary to ensure each video includes at least 12 QA pairs.
3.2 13-BBNCH-WE BENC
To further increase video diversity, collect extra videos from YouTube following existing prac- tice (Fu et al., 2025; Fang et al., 2024; Niu et al., 2025).
Video Collection We adopt question-driven approach for video collection: annotators select videos that could support at least five questions belonging to the types in Table 1, which naturally encourage the inclusion of videos with rich narratives and complex inter-entity dynamics. To promote video diversity, annotators refer to a predefined list high-information density categories relevant to real- world multimodal agent applications. Each category allows up to 20 video submissions. Meanwhile,

---

Published as a conference paper at ICLR 2026
Table Comparison of M3-Bench with existing LVQA benchmarks across key aspects: number of videos (#Videos), average length in seconds (Len.), number of QA pairs (#QAs), annotation method presence of
(Anno., W//ammamalll/autooaate,, question format (Form., O/C=open-/close-enee an agent in the video (Agent Present), inclusion of questions about modal reasoning (Cross- Modal QA), person understanding (Person QA), and general knowledge (Knowledge QA).
Benchmark
AVideus
Len.s))
#QAs
Anno.
Furm.
Agent
Cross-
QA
Person
QA
Knowledge
QA
EgoSchhema (Mangalam al., 2023)
5,063
180.0
5,063
M/A
C
X
Long Bedeeicccencch (Wu al., 2024)
3,763
473.0
6,678
HourVideo (Chandrasegaran al., 2024)
500
2,742.0
12,976
M/A
C
MVBench (Li et al., 2024b)
3,641
16.0
4,000
C
al., 2025)
2,700
MLVU (Zhou et al., 2025)
1,730
930.0
3,102
M/A
O/C
MS-Rennch-rbol
2,039.9
1,276
M3-Bench-web
920
1,630.7
3,214
annotators are encouraged to propose new categories, provided they are sufficiently distinct from the existing ones. The final dataset comprises 46 distinct video types, as summarized in Figure 3.
QA Annotations The same annotator who collect the video also generate least corresponding QA pairs. Each question must correspond to at least one type in Table 1. Questions are required to specific, objective, and have single unambiguous answer that could be reasonably inferred from the video content, to ensure fair and effective evaluation. Those with ambiguous references, such as "the man" or "in the middle part of the video," are deemed invalid. In M3-Bench-webh the timestamp for each question is uniformly set to the end of the video.
Detailed annotation guidelines, annotator information, and quality control procedures for M3- Bench-robot and M3-Bench-web are provided in Appendix B and Appendix C, respectively,
3.3 AUTOMATIC EVALUATION
We use GPT-4o an automatic evaluator for M3- M3-Bench by prompting it to assess the correctness of generated answers against their corresponding reference answers. The prompt is shown in Table 23 ($ M.1). To validate this approach, construct test set of 100 randomly sampled question- reference-answer triples from our method and various baselines ($ 5.1). Three authors independently assess each generated answer, and GPT-4o's judgments compared with the majority vote of human annotations, yielding 96% agreement and confirming the effectiveness of the approach.
4 APPROACH
4.1 LONG-TERM MEMORY
Long-term memory in M3-Agent implemented as an external database that stores information in structured, multimodal format (text, image, audio). Specifically, memories organized as an entity-centric multimodal graph, where each node represents a distinct memory item. Each node includes unique ID, modality type, raw content, reliability weight, embeddings, and other metadata such as timestamps (see Table 8 ($ D) for details). Nodes are connected by undirected edges that represent several types of relationships between memory items. For example, items sharing the same entity ID are linked to represent the entire entity. This design supports not only sequential retrieval of memories based on timestamps but also associative retrieval based on entities.
The agent constructs its memory by incrementally adding new text, image, audio nodes. If memory item generated by the memorization process already exists in long-term memory, the cor- responding node or edge is reactivated and its weight is increased; otherwise, new node or edge created. To handle potential conflicts introduced during memory construction, M3-Agent adopts weighted voting mechanism at inference time: frequently activated nodes edges accumulate higher weights and take precedence over less active, conflicting entries. This design promotes ro- bustness and consistency of the memory graph over time.
Search Tool To facilitate memory retrieval, we provide suite of search tools that enable the agent to retrieve relevant memories based specific requirements. In particular, implement types of search mechanisms operating at different levels of granularity, as summarized Table 3. Detailed implementation of these retrieval mechanisms is provided in Appendix E.

---

Published as a conference paper at ICLR 2026
Table 3: Search functions supported by long-term memory.
Function
Description
Accepts a query and returns the lop-k most relevant nodes. Supports multimedal queries (text, image, or audio) and modality specific retrieval.
search,nede
search_olip
Return top k most relevant memory clips for a given query. A memory clip refers to the agent's episodic and semantic memory segment (typically 30 seconds) generated during clip-by-clip streaming video processing.
4.2 MEMORIZATION
During memorization (Figure 1), M3-Agent processes video stream in a clip-by-clip manner (typi- cally 30s each), generating two types of memory: episodic memory captures raw visual and auditory content; and semantic memory extracts general knowledge such as character identities, attributes, and relationships. Semantic memory enriches content and provides additional cues for retrieval.
Consistent Entity Representation A key challenge in constructing high-quality long-term memory maintaining consistent representations of core concepts, such as main characters and objects, over time. To avoid using ambiguous language-based descriptions (e.g., "a man with a beard" or "a woman in red dress"), M3-Agent constructs persistent identity representations its long-temm memory using multimodal information. Equipped with tools like facial recognition and speaker identification (see Appendix E), it extracts faces and voices from clips, linking each to an existing node in its long-term memory using sears rch node function or assigning to new node. The resulting identifiers (face_id, voi re_id) serve as references to the corresponding characters. By leveraging globally maintained memory graph, M3-Agent enforces consistent character mapping across clips, enabling a coherent long-term memory.
Memory Generation Having the face and voice identities, M3-Agent generates both episodic and semantic memories, referencing each character by their face.id or voice stored in long-tem memory. This ensures each character is consistently grounded in the memory graph. In particular, during semantic memory generation, M3-Agent performs cross-modal reasoning to link faces and voices belonging the same character, updating the memory graph connect their respective nodes. Once linked, these nodes are treated as a single character: during retrieval, they are assigned shared <character.id>, enabling consistent reasoning about characters across modalities. For output format, M3-Agent generates episodic and semantic memory as a list of text entries as shown in Figure 1. Each entry stored in the memory graph as a node and linked its associated identities.
Algoritla Control Process
Require: Input question g, policy model na. long-term memory M. maximom number of rounds H. complete tnjeccooyy generated agent, T < [moee "system", content: commt(cysstet wt..eempt.......m.emeeme 4)).
Ensure:
(role: "user", content: Instruction.pp cootroo.poroottt
Initialize the trajectory
i 0
while <
Execute up to
4- mel- (T)
Append "assisannn" content: T.) to reasoning, action, argument PARSE(T)) action "[Search]" then
b Extract action and argument from
memory SEARCHC.M. argument)
Scarch memory using the content query
else Break
P. The trajectory ends when action "[Answer]]
end
i i++1 1
Append (mle: "user", content: memory instrnct wwtp:rttttooooooooopoooo.mmo.ptt to i Hf I then
A Append seurch results and prompt for round
Append {rols: "user", content: memory roond. to
end if end while
return
4.3 CONTROL
When an instruction is received, the control process is triggered. As illustrated in Figure 1, during control, M3-Agent autonomously performs multi-tum reasoning and invokes search functions to retrieve relevant memories. Unlike traditional single-turn RAG, this iterative approach enables more complex planning. making the system more flexible and more capable. Specifically, the control process follows Algorithm 1, with prompts in Table 27 ($ M.3). Here To is the control policy, is user question, and D is the long-term memory. At each round, me generates response consisting
7

---

Published as a conference paper at ICLR 2026
of reasoning, an action, and associated argument. If the action is [Search], the system queries D with the argument and appends retrieved results to the context for the next round. Depending on the context, it can call different search functions to retrieve memories from multiple perspective (e.g,, search node for people or sea search cl clip for events). the action is [Answer], the system returns the content and the process terminates. This loop continues for up to H rounds.
4.4 TRAINING
Although conceptually, memorization and control can be handled by single model, here we train two separate policy models to better specialize their respective functions: memorization relies on strong multimodal understanding, while control requires strong reasoning capabilities. To align with these objectives, we initialize the memorization policy with Qwen2.5- -reen. (Xu et al., 2025a), which capable of processing both visual and audio inputs, and the control policy with Qwen3 (Yang et al., 2025), which is known for its advanced reasoning capabilities. The training data is sourced from our in-house video dataset, for which we have permissions for model training. We collect videos along with corresponding question guestion-answer pairs, adhering to the same annotation standards used for M13-Bench-weh In total, the training dataset comprises 500 long videos, corresponding to 26,943 30-second clips, and 2,736 question-answer pairs.
Memorization To improve the model's ability to generate desired memories, we perform imitation obtain nemory-7b-sfe
learning Qwen2.5-0mmi-P
This process is supervised using synthetic demonstration dataset including (1) Episodic memory: For each 30-second clip, GPT-4o is prompted to generate frame-level cues, while Gemini-1.5-PP provides event-level summaries. The two outputs are merged to create richer narrative; (2) Identity mapping: Based on extracted faces and voices, we design an algorithm to detect high-confidence meta-clips (monologues featuring consistent face and voice) to build a reliable global face-voice identity mapping, which is propagated across relevant clip segments; and (3) Other semantic memory: We use diverse set of prompts shown in Table 10 (SF) to extract various types of semantic memory. In total, we synthesize 10,952 samples and reserve 200 for validation. More details appear Appendix
Based on the demonstration dataset, we perform supervised fine-tuning for 3 epochs with learning rate le and a batch size of 16, using 16 GPUs with 80GB memory each.
Control We first construct the environment by generating long-term memory for each training video using memory-7b-ssft fb. We then apply DAPO (Yu et al., 2025) to train the policy model Te, initialized from prompting Qwen3-32b (control- www.oo1--bbpppppptp--
For each QA pair (q, a),
rollouts group of G trajectories - using the algorithm shown in Algorithm 1. Note that for any given question, the agent restricted to searching within the memory generated from its associated video. Then, for each trajectory Ti, the final submitted answer y4 is extracted and evaluated using the GPT-4o evaluator introduced Section 3.3. The reward of the i-th trajectory is given by:
1,
v.) True
R.
otherwise
Note that we compute loss only on LLM generated tokens.
The optimization objective is:
(o) = B (..)) mag DS Elad I(TE.)
[(T,,) - min
A
mald (Tael
clip
no (ra(Tall (T. T8 (Ti.e (i.e))
Clom 1 Ehigh
Avee)
0
R <G
(2)
where the indicator T(Ti.) i-th response is estimated as: A,, = [R mean( (e)dss)))s((((( As a result, we obtain
Tú, generated token; and otherwise. The advantage of the
the policy model control-32b-rz Table 14 ($ H) lists the hyperparameters for DAPO.
EXPERIMENTS
5.1 BASELINES
We evaluate M3-Agent against three types of baselines: (1) Socratic Models: A class of mod- ular frameworks (Zeng et al., 2022) that first MLLM generate textual descriptions for
8

---

Published as a conference paper at ICLR 2026
each video clip and store them long-term memory. When answering questions, an LLM trieves these descriptions via a search.clip function and performs reasoning through RAG. We implement several variants with different MLLMs for description: proprictary models in- cluding Gemini-1.5-Pro (Team et al., 2024) and GPT-4o (Hurst et al., 2024), and open-source models including Qwen2.5-0mmi-7h (Xu et al., 2025a) and Qwen2.5-VI-77b (Bai al., 2025).
GPT-4o is used uniformly for the final RAG step. (2) Online video understanding methods: Representative online video understanding methods, including MovieChat (Song et al., 2024),
MA-LMM (He et al., 2024a), and Flash-VStream (Zhang et al., 2024b). (3) Agent Methods: Naive implementation of M3-Agent by prompting proprictary models to perform memorization and control. We implement Gemini-Agent, which uses Gemini-1.--Pro for both memorization and Gemini-GPT4o- and remains memorization We set the maximum number of exccution rounds H
and control -conttool-gemi----r Hybrid, which uses GPT-4o for control (control wgp-ggppppppoooe handled by memory- wely--gmmiiiippooo for M3-Agent and all agent-based baselines, and configure search. clip to return the top most relevant clips. Further implementation details of all baselines are provided in Appendix
5.2 DATASET AND EVALUATION
We evaluate M3-Agent and all baselines on both M3-Bencc-robot and 1.3-Bench-web. To demon- strate the generality of our approach, we also test them on VideoMME-long (Fu et al., 2025), mainstream long-vider video understanding benchmark, following its official evaluation protocol'.
Table 4: Results M3-Bench-robot M3-Bench-weh, and VideoMME-longg with comparisons across M3-Bench question types: multi-evidence reasoning (ME), multi-hop reasoning (MH), cross- modal reasoning (CM), person understanding (PU), and general knowledge extraction (GK).
Method
M3-Bencb-rooul
M3-Bench-wel
Video-
ME
MH
CM
PU
GK
AlI
ME
MH
CM
PU
GK
All
MME-Long
Socratic Model
Ownn2 5-Ommi-7b
Owee2 5-
2.9
3.4
11.9
10.5
13.4
14.0
20.9
14.9
45.9
Oemini-1.5 1,5-Pro
6.5
7.5
8.0
9.7
7.6
8.0
18.0
17.9
23.8
23.1
28.7
23.2
38.0
GPT-1o
9.3
9.0
8.1
102
8.5
21.3
21.9
30.9
27.1
39.6
28.7
38.8
Online Video Understanding Merhods
MovieChat
13.3
9.8
12.2
15.7
7.0
11.2
12.2
6.6
12.5
17.4
11.1
12.6
19.5
MA-LMM
25.6
23.4
22.7
39.1
14.4
24.4
26.8
10.5
22.4
39.3
15.8
24.3
17.3
Flash-VStream
Agent Method
Semini-Agont
15.8
17.1
15.3
200
15.5
16.9
29.3
20.9
33.8
34.6
45.0
34.1
55.1
Gemini- Prmini-GGTToo-Hybriii
21.3
25.5
22.7
28.8
23.1
24.0
35.9
26.2
37.6
43.8
52.2
41.2
56.5
M3-Agent
48.9
61,8
5.3 MAIN RESULTS
As shown in Table 4, M3-Agent outperforms all baselines on -oo-Bench-robot, M3-Bench-webb, and VideoMME-longg, Specifically, on M3-Bench-robot, M3-Agent achieves a 6.3% accuracy improve- ment over the strongest baseline, MA-LLM. On 133-Bench-oo web and VideoMMEElonng, surpasses the strongest baseline, Gemini-GPT4o- by 7.7% and 5.3%, respectively.
Table 4 also reports a breakdown of performance across different question types in M3-Bench. M3- Agent shows strong performance in person understanding and cross-modal reasoning. Specifically, compared to the best-performing baseline on M3-Bench-roobo, MA-LMM, M3-Agent achieves im- provements of 4.2% in person understanding and 8.5% in cross-modal reasoning. On M3-Bench- web, M3-Agent outperforms the top bascline, 6.7% in the respective categories. These results demonstrate M3-Agent 's superior ability to main- tain character consistency, deepen person understanding, and integrate multimodal information.
with gains of 15.5% and
We also assess the memorization model via precision and comprehension in Appendix G.
5.4 ABLATION STUDY
To evaluate the impact of memorization performance, we fix the control model to control-32b rl and compare different memorization methods (Table 5). Replacing the
Thttps:////t wtt.ttt...m..hennouaeeba watpaaaaeaaaaaaeeeeeeeeeeee http://o_snswerrrbl////aaaaaaaaa _

---

Published as a conference paper at ICLR 2026
Table Ablation of memorization models. The control is fixed as control-32b-
Memorization Model
M3-Bench-web M3-Bench-robot
Video-MMEE-Long
memory-gemini -prompt
28.7
52.7
memory-7b--romm
25.3
39.9
50.8
memory 7e-77---ffe sft (M3-Agent)
30.7
48.9
61.8
memory-7b-st 7b-sfttttttttt w/o equivalence memozy-7bo sft w/o semantic memory
19.5
39.7
52.1
13.6
29.7
48.7
Table 6: Ablation of control methods, including: (1) comparison between GRPO and DAPO train- ing algorithms; (2) performance gains from DAPO scale with model size; (3) the effect of removing inter-turn instruction and reasoning. The memorization model is fixed memory 7b-sft.
Control Model
M3-Bench-rohot
W3-Bench-web
Video-MME-LLore
control 32b gzpo
30.0
47.7
58.7
control -r-pmopptt
35.7
45.3
1attool-bbrr.
24.6
40.5
50.8
cont::o-144-prrome
18.3
36.9
49.1
cont:oo--44b-rl
28.2
56.0
cont:o1-32b-ppo0 -oomppp
20.7
40.9
52.5
cont:ol-32b (M3-Agent)
30.7
48.9
61.8
cont:o1-322 cont:o1-32b-poo0 --------ppppppp......pp.oo---ppp w/o inter-turn instruction
32.3
48.3
control-32b-1 wfo inter-tum instruction
20.2
43.1
55.9
cont =o1-3bbr1 w/o reasoning
19.0
40.1
52.3
memory with that from memory-gemind 9.1% on M3-Bench-robot, M3-Bench-web, and VideoMMEElongg respectively, indicating that memory-7b- produces higher-quality memory than werpoez-gemiiiiir-ee
reduces accuracy by 2.0%, 2.6%, and
Using leads to accuracy reductions of 5,4%, 9.0%, and 11.0% on the same bench- marks, highlighting the importance imitation learning in generating useful memory. Finally, re- moving character identity equivalence semantic memory significantly degrades QA performance.
Next, we examine the impact of control on final performance, by fixing the memorization model to memory-7b-sft and evaluating various control models (Table 6). First, we compare two RL algorithms: GRPO and DAPO (training details for GRPO in Appendix H). Our results show that control-32b-rt trained with DAPO consistently outperforms control-326 -2ppoppooooooo across all test sets. Second, we analyze DAPO's scalability and observe substantial gains at all model sizes. Specifically, control 32b-rl achieves improvements of 10.0%, 8.0%, and 9.3% in accu- racy over cont ront respectively. Finally, ablation studies reveal that both inter-instruction and reasoning critical: moving interiintruuctin decreases accuracy by 10.5%, 5.8%, and 5.9%, while removing reasoning reduces accuracy by 11.7%, 8.8% and 9.5 5% on the same benchmarks.
on M3-Bench-robot, M3-Bench-web. and VideoMMEEllng.
CONCLUSION
In this paper, we have introduced M3-Agent, a multimodal agent framework equipped with long- term memory. M3-Agent perceives real-time video and audio streams to build both episodic and semantic memories, enabling it to accumulate world knowledge and maintain consistent and context- rich memory over time. When given instructions, M3-Agent autonomously reason and retrieve relevant memories to complete tasks more effectively. To evaluate memory effectiveness and rea- soning capabilities, we have developed M3-Bench, LVQA benchmark featuring real-world robot- perspective videos and challenging questions requiring person understanding, knowledge extraction, and cross-modas -odal reasoning. Experiments long show that M3-Agent consistently outperforms baselines, including Socratic models, online video understanding methods, and agents implemented by prompting closed-source models, demon- strating its superior memorization and reasoning capabilities. Finally, our case studies (Appendix I.) reveal key limitations and promising future directions. These include enhancing attention mecha- nisms for semantic memory formation and developing richer yet more efficient visual memory.
M1S-Bench-robot, M3--encc-web. and VideoMME-
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This research aims to advance the development multimodal agents with human-like long-term memory capabilities. For the M3-Bench-robot subset, all actors were fully informed about the pur- pose of the recordings and signed explicit consent agreements outlining data usage. As for the M3-Bench-wel subset, only publicly available videos were used, and annotations were created to ensure privacy and fairness. Therefore, the datasets used and released do not involve any privacy violations or ethical risks.
REPRODUCIBILITY STATEMENT
To support transparency and reproducibility, we commit to release M3-Bench benchmark, model checkpoints, training data, and code, upon publication. The M3-Bench will include all robot- perspective and web-sourced videos along with detailed question-answer annotations and evalu- ation scripts. We will provide the model checkpoints for both memorization (memory mhemory-bb-sfte and control and control pipelines, tool implementations, and demonstration data synthesis pipelines. These sources will enable researchers to reproduce our experimental results and extend the work in future directions.
deseribed in the paper. The code release will cover memorization
REFERENCES
Shuai Bai, Keqin Chen. Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint ar arXiv:2502.1392 13923, 2025.
Keshigeyan Chandrasegaran, Agrim Gupta, Lea M Hadzic, Taran Kota, Jimming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Fei-Fei Li. Hourvideo: 1-hour video- language understanding. Advances in Neural Information Processing Systems, 37:53168-53119 2024.
Yafeng Chen, Siqi Zheng, Hui Wang, Luyao Cheng, Qian Chen, Shiliang Zhang, and Junjie Li. Eres2netv2: Boosting short-duration speaker verification performance with computational effi- ciency. arXiv preprint XXiv:2406.02167 2024a.
Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408. 10188, 2024b.
Prateek Chhikara, Dev Khant, Saket Aryan, Taranject Singh, and Deshraj Yadav. MemO: Building production-ready ai agents with scalable long-term memory. arXiv preprint arXiv:2504.19M 2025.
Anxhelo Diko, Tinghuai Wang. Wassim Swaileh, Shiyan Sun, and Ioannis Patras. Rewind: Under- standing long videos with instructed learnable memory, In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 33734-13743 2025.
Yue Fan, Xiaojian Ma, Rongpeng Su, Jun Guo, Rujie Wu, Xi Chen, and Qing Li. Embodied vidcoa- gent: Persistent memory from egocentric videos and embodied sensors enables dynamic scene understanding. arXiv preprint arXiv:2501.00358 2024a.
Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory-augmented multimodal agent for video understanding. In European Conference on Com- puter Vision, pages 75-92. Springer, 2024b.
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long- onn-formm multi-shot benchmark for holistic video understanding. Advances in Neural Information Processing Systems, 37:89098-899 89124, 2024.

---

Published as a conference paper at ICLR 2026
Peiyuan Feng, Yichen He, Guanhua Huang, Yuan Lin, Hanchong Zhang, Yuchen Zhang, and Hang Li. Agile: novel reinforcement learning framework of Ilm agents. Advances in Neural Infor- mation Processing Systems, 37:5244- 5284, 2024.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang. Chenyu Zhou, Yunhang Shen, Mengdan Zhang, al. Video-mme: The first-ever comprehensive evalu- ation benchmark of multi-modal Ilms in video analysis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24108 24118, 18, 2025.
Pascale Fung, Yoram Bachrach, Asli Celikyilmaz, Kamalika Chaudhuri, Delong Chen, Willy Chung, Emmanuel Dupoux, Hervé Jégou, Alessandro Lazaric, Arjun Majumdar, et al. Embodied ai agents: Modeling the world. arXiv preprint arXiv:2506.22355. 2025.
Tengda Han, Max Bain, Arsha Nagrani, Gül Varol, Weidi Xie, and Andrew Zisserman. Autoad: Movie description in context. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18930-18940, 2023.
Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-Imm: understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), 2024a.
large multimodal model for long-term video
Yichen He, Yuan Lin, Jianchao Wu, Hanchong Zhang, Yuchen Zhang, and Ruicheng Le. Sto- ryteller: Improving long video description through global audio-visual character identification. arXiv preprint arXiv:2411.07077 2024b.
Mengkang Hu, Tianxing Chen, Qiguang Chen, Yao Mu, Wenqi Shao, and Ping Luo. Hiagent: Hier- archical working memory management for solving long-horizon agent tasks with large language model. arXiv preprint urXiv:2408.095533 2024.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21226. 2024.
Md Mohaiminul Islam, Ngan Ho, Xitong Yang, Tushar Nagarajan, Lorenzo Torresani, and Gedas Bertasius. Video recap: Recursive captioning of hour-long videos. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 18198-18208, 2024.
Anna A Ivanova, Aalok Sathe, Benjamin Lipkin, Unnathi Kumar, Setayesh Radkani, Thomas H Clark, Carina Kauf, Jennifer Hu, RT Pramod, Gabriel Grand, et al. Elements of world knowl- edge (ewok): A cognition-inspired framework for evaluating basic world knowledge in language models. arXiv preprint arxiv:2405. 00960, 2024.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training 11ms to reason and leverage search engines with reinforcement learning. arXiv preprint 2rXiv:2503.095 2025.
Jiazheng Kang, Mingming Ji, Zhe Zhao, and Ting Bai. Memory os of ai agent. arXiv preprint arXiv:2506. 250.2.6666.66. 2025.
Xiaohan Lan, Yitian Yuan, Zequn Jie, and Lin Ma. Vidcompress: Memory-enhanced temporal compression for video understanding in large language models. arXiv preprint 41X11:21011 2024.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented gener- ation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33: 9459-9474, 2020.
Hao Li, Chenghao Yang, An Zhang, Yang Deng, Xiang Wang, and Tat-Seng Chua. Hello again! Ilm- powered personalized agent for long-term dialogue. arXiv preprint arXiv:2406.0599 05925, 2024a.
12

---

Published as a conference paper at ICLR 2026
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint XXiv:2305.06655. 2023.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195-22206, 2024h.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122. 2023a.
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of IEEF/CVF conference on computer vision and pattern recognition, pages 26689-26699 2024.
Kevin Lin, Faisal Ahmed, Linjie Li, Chung-Ching Lin, Ehsan Azarnasah, Zhengyuan Yang, Jianfeng Wang, Lin Liang, Zicheng Liu, Yumao Lu, al. Mm-vid: Advancing video understanding with gpt-4v (ision). arXiv preprint arXiv:2310. 19773, 2023b.
Na Liu, Liangyu Chen, Xiaoyu Tian, Wei Zou, Kaijiang Chen, and Ming Cui. From Ilm to conversa- tional agent: A memory enhanced architecture with fine-tuning of large language models, 2024. URL https://arxiv. org/abs/2401.02777 2024a.
Weijie Liu, Zecheng Tang, Juntao Li, Kehai Chen, and Min Zhang. Memlong: Memory-augmented retrieval for long text modeling. arXiv preprint arXiv:2408. /6967, 2024b.
Zhiwei Liu, Weiran Yao, Jianguo Zhang, liangwei Yang. Zuxin Liu, Juntao Tan, Prafulla K Choubey, Tian Lan, Jason Wu, Huan Wang, al. Agentlite: A lightweight library for build- ing and advancing task-oriented Ilm agent system. arXiv preprint arXiv:2402. 15538, 2024c.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic bench- mark for very long-form video language understanding. Advances in Neural Information Process- ing Systems, 36:46212 462444, 2023.
Kai Mei, Xi Zhu, Wujiang Xu, Wenyue Hua, Mingyu Jin Zelong Li, Shuyuan Xu, Ruosong Ye, Yingqiang Ge, and Yongfeng Zhang. Aios: Lim agent operating system. arXiv preprint arXiv:2403. 16971, 2024.
Junbo Niu, Yifei Li, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, et al. Ovo-bench. How far is your video-llms from real-world online video understanding? In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 18902-18913, 2025.
Felix Ocker, Jörg Deigmöller, Pavel Smirnov, and Julian Eggert. A grounded memory system for smart personal assistants. arXiv preprint arXiv:2505.06322 2025.
Shuofei Qiao, Runnan Fang, Ningyu Zhang, Yuqi Zhu, Xiang Chen, Shumin Deng, Yong Jiang, Pengiun Xie, Fei Huang, and Huajun Chen. Agent planning with world knowledge model. Ad- vances in Neural Information Processing Systems, 37:1 14843-1 114711, 2024.
Gabriel Sarch, Yue Wu, Michael Tarr, and Katerina Fragkiadaki. Open-ended instructable embod- ied agents with memory-augmented large language models. arXiv preprint arXiv:2310.15127, 2023.
Yu Shang, Yu Li, Keyu Zhao. Likai Ma, Jiahe Liu, Fengli Xu, and Yong Li. Agentsquare: Automatic Ilm agent search in modular design space. arXiv preprint arXiv:2410.06153 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits mathematical reasoning in open language models. arXiv preprint arXiv:2400.033300 2024.
13

---

Published as a conference paper at ICLR 2026
Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spatiotemporal adap- tive compression for long video-language understanding. arXiv preprint arXiv:2410.1733, 2024.
Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding, In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 26160-26166 2025.
Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Viston and Pattern Recognition, pages 18221 11221-182333 2024.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.055300 2024.
Endel Tulving. "episodic and semantic memory," in organization of memory. (No Title), page 381, 1972.
Endel Tulving. How many memory systems are there? American psychologist, 40(4):385, 1985.
Bing Wang, Xinnian Liang, Jian Yang. Hui Huang, Shuangzhi Wu, Peihao Wu, Lu Lu, Zejun Ma, and Zhoujun Li. Enhancing large language model with self-controlled memory framework. arXiv preprint arXiv:2304. 13343, 2023.
Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models. arXiv preprint arXiv:2407. 407.000444 2024a.
Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, et al. Jarvis-l: Open-world multi-task agents with memory-augmented multimodal language models. IEEE Transactions on Patlern Analysis and Machine Intelligence, 2024b.
Chao- Yuan WU, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Memvit: Memory-augmentee multiscale vision transformer for efficient long-term video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13587-13597, 2022.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-comte interleaved video-land language understanding. Advances in Neural Information Processing Systems, 37:28828-2888. 2024.
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang. Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.202/1 2025a.
Wujiang Xu, Kai Mei, Hang Gao, Juntao Tan, Zujie Liang, and Yongfeng Zhang. A-mem: Agentic memory for I1m agents. arXiv preprint 2rxiv:2502.1211000 2025b.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng. Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin. Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprini arXiv:2505.09388, 2025.
Qiying Yu. Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
14

---

Published as a conference paper at ICLR 2026
Liping Yuan, Jiawei Wang. Haomiao Sun, Yuchen Zhang, and Yuan Lin. Tarsier2: Advancing large Dision-language models from detailed video description to comprehensive video understanding. arXiv preprint arXhv:2501.77888, 2025.
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, et al. Socratic models: Com- posing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022.
Chaoyi Zhang, Kevin Lin, Zhengyuan Yang, Jianfeng Wang, Linjie Li, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. Mm-narrator: Narrating long-form videos with multimodal in-context learning. Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recogni- tion, pages 13647-13651 2024a.
Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory-based real-time understanding for long video streams. arXiv preprint arXiv:2406.08088 2024b.
Pan Zhang, Xiaoyi Dong, Yuhang Cao, Yuhang Zang, Rui Qian, Xilin Wei, Lin Chen, Yifei Li, Junbo Niu, Shuangrui Ding, et al. Internim- Internlm-xcompose xcomposer2. 5-omnilive: A comprehensive multimodal system for long-term streaming video and audio interactions. arXiv preprint urXiv:2412.09596. 2024c.
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprini arXiv:2406. 2024d.
Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19724 19731, 2024.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang. Yongping Xiong, Bo Zhang, et al. Mlvu: Benchmarking multi-task long video understanding, In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 13691-13701, 2025.
15

---

Published as a conference paper at ICLR 2026
Appendix
A LLM Usage Disclosure
18
B M3-Bench-robot Annotation Details
18
B.1 Script Annotation Guidelines
18
B.2 Video Filming.
18
B.3 QA Annotation Guidelines
18
B.4
Quality Control
19
Annotator Information
19
B.6 Data Example
19
M3-Bench-web Annotation Details
20
C.1 Annotation Guidelines
20
C.2 Quality Control
21
C,3 Annotator Information
Long-Term Memory Design
21
Implementation Details of Tools
22
F
Demonstration Data Synthesis for Memorization
23
E1
Episodic Memory Synthesis
23
F2 Entity ID Relationship Detection
................................
26
F3 Semantic Memory Synthesis
27
F.4 Quality of the Synthetic Data
29
G Evaluation of Memorization
29
H RL Training Details
29
H.I DAPO Training
29
H.2 GRPO Training
29
I Baseline Implementation Details
30
J Cross-Model Assessment of Automatic Evaluation of M3-Bench
31
K Study on Memory of M3-Agent
31
K.1 Ablation Study
31
K.2 Performance Scaling with Video Duration
33
K.3 Impact of Identity Growth
33
L Case Study
33
16

---

Published as a conference paper at ICLR 2026
M Prompt Templates
43
M.1 Prompt for Automatic Evaluator of M3-Bench
43
M.2 Prompts for Socratic Models
43
M.3 Prompts for M3-Agent
45
17

---

Published as a conference paper at ICLR 2026
A LLM USAGE DISCLOSURE
We used large language models (LLM) for the language-related aspects of this paper, including spell-checking, grammar-checking, and refining our original text. No new content was generated by the LLMs. All LLM-refined text was either carefully reviewed or further edited by the authors. Additionally, we employed an LLM-based paper search agent? complement, but not replace our manual survey of related works.
B M3-BENCH-ROBOT ANNOTATION DETAILS
B.1
SCRIPT ANNOTATION GUIDELINES
Actor Setup
Four five actors participate, including one playing the role of robot.
Definitions
1. Script:
Consists of events and questions and provides actors with dialogue and stage instructions.
Played by human actor. is an ideal highly intelligent robot with reasoning and memory abilities similar to humans.
Robot:
Scenario: living room, kitchen, bedroom, study, office, meeting room, and gym.
4. Event: A complete, short plot within the script. A reference event includes information relevant to future questions, such as robots interacting with humans while observing and learning human preferences the placement of objects in real-world scenes.
Designed to evaluate the robot's memory. Each question must align with at least one type listed in Table 1.
Question:
Requirements
Annotate at least 15 questions, each labeled with the corresponding reference events.
Each script must contain at least 70 events to ensure minimum video duration of 30 minutes.
Avoid asking questions that rely solely on common sense or that can be answered without watch- ing the video.
Do not ask questions that remain unanswerable even after watching the video.
Avoid questions that can be answered based solely the dialogue.
Do no include questions that are weakly related to the reference events.
The question should have a clear and unambiguous answer that can be objectively verified by comparing to the reference answer.
B.2
VIDEO FILMING
The robot actor wears head-mounted camera, either iPhone 16 Pro, Xiaomi 14 Ultra, or GoPro HERO13, to capture a single point-of-view video from the robot's perspective.
We collect two types of audio tracks for each video. The first is directly recorded by the head- mounted device, reflecting the raw auditory input robot would naturally receive, including am- bient sounds and spatial acoustic variations. The second is captured using individual lapel micro- phones worn by each actor, providing high-fidelity voice recordings to complement the primary audio stream.
B.3 QA ANNOTATION GUIDELINES
Background
hnttps://paa--geett.il
18

---

Published as a conference paper at ICLR 2026
In the future, robots will help humans complete many tasks in indoor environments such as homes. Based on this imagination, we filmed a video from the perspective robot.
In order to evaluate the model's ability, we set questions at different timestamps, typically related the robot's upcoming tasks. Correct answers are essential for the successful completion of these tasks.
Some questions require manual review or additional annotations to ensure each video includes at least 10 questions.
Task
Provide a 30-45 minute video along with corresponding script that includes series of questions. Note: Minor script modifications may occur during filming to accommodate practical constraints. As result, the script may not perfectly align with the final video.
1. Review existing questions.
For each question in the script:
Annotate the corresponding timestamp in the video based on the related script event.
Determine whether the question can be answered using the video content up that point. If annotate the answer.
If the question unanswerable, consider whether modifying it could make it answerable. If applicable, revise the question and provide the answer.
For each question-answer pair, annotate the reasoning process used to derive the answer and specify the question types according to Table 1.
Annotate additional questions:
If fewer than 10 questions remain after reviewing the script, generate new questions that must belong at least one type listed in Table
B.4 QUALITY CONTROL
The anmotation process consists of two rounds. In the first round, the goal is to ensure that annotators fully understand the annotation guidelines. Each annotator is required perform QA annotations on three videos. The authors then review the annotations, provide feedback, and the annotators may revise their annotation accordingly. Based the quality of these initial annotations, the authors determine whether the annotator is qualified to proceed to the formal annotation phase. In the second round, each annotator annotates five videos time. The authors randomly select one video from each batch for quality inspection. If more than one invalid question-anse answer is found in the selected video, the entire batch must be re-annotated. Otherwise, the batch considered accepted. Two authors are involved in the quality control process throughout the annotation workflow.
In addition, ensure the quality of the questions M3-Bencc-roohot, we recruit five annotators to answer each question. Annotators are allowed to first read the question and then watch the video as many times needed. The final human accuracy on MB-Bencc-rrobot is 90.7%. Our error analysis shows that the most common mistakes are counting-related problems.
B.5 ANNOTATOR INFORMATION
All annotators are employed by commercial data annotation company. We sign contract with the company and pay the company for the annotation work at market price. The annotators are all college graduates with strong English proficiency. For script annotation, eleven annotators are involved. Video filming engage 67 actors. For QA annotation, five annotators participate.
B.6 DATA EXAMPLE
Table 7 provides example of script annotation.
19

---

Published as a conference paper at ICLR 2026
Event ID
Event
Rose is the room talking to Amy on the phone. She thanks Amy for the tulips and takes photo of the blooming flowers to share with her.
(reference)
2 Rose tells the robot that the delicate teddy bear is gift for Rachel. (reference)
3 After hanging up with Amy, Rose calls Rachel and Leo to remind them not to forget come over today.
Rose looks a pile of packages in the corner of the bedroom. They are recently purchased clothes. She asks the robot to unpack them and place the clothes on the first shelf of the wardrobe.
4
5 She points to the bottom of the wardrobe, where a pile of delicate little toys is (reference)
stored, and tells the robot, "Put the teddy bear there."
6
At that moment, the doorbell rings and Rose excitedly runs to open the door.
Rachel sees the dolls on the bed and exclaims, "Wow, these dolls are so cute, me pamper them!"
10
Rose says, "Don't t rush, there's another surprise," and then calls the robot. Question: Is Rachel's gift on the top shelf or the bottom shelf of the wardrobe? Reference: event-2 and event-5
11
The robot takes a teddy bear from the wardrobe, hands to Rachel, and says, "This is gift prepared for you."
12
Rachel teases that Rose just doesn't 't want to admit but the robot surely knows. She then turns to the robot and asks who gave Rose the flowers. Question: Who gave Rose the flowers? Reference: event-l
58
Table 7: An example of the M3-Bench-robo script.
C M3-BENCH-WEB ANNOTATION DETAILS
C.I ANNOTATION GUIDELINES
To better help the annotators understand the requirements and better ensure the overall quality, safety, and validity of the datasets, provide the following detailed guidelines, which clearly specify the acceptable and unacceptable annotation practices.
Questions must allow for verifiable and objective evaluation of correctness. This entails avoiding overly open-ended questions, compound questions that mix multiple sub-questions, or questions with multiple equally valid answers.
Each video must include at least two questions targeting character attribute modeling and two questions involving commonsense reasoning.
All visual information required to answer question must remain clearly recognizable at lower resolutions (<720p), ensuring that all questions are answerable.
For videos between 20 and 40 minutes in length, 5 questions should be generated; for videos exceeding 40 minutes, 10 questions should be provided. Compensation considers both the number and duration of the videos.
For commonsense reasoning questions, annotators must also specify the commonsense knowledge being tested, in addition to the question and its answer.
20

---

Published as a conference paper at ICLR 2026
It is not permissible for all questions to be answerable using only audio. A reasonable proportion of questions must be vision-centrie, requiring understanding of visual content in the video.
Redundant questions within the same video are not allowed. For instance, asking "Describe David's appearance" and "Describe Alice's appearance" would be considered repetitive.
Questions that can be answered solely based on brief moment or a short clip should be avoided. Specifically, the context required to answer valid question should span more than 10 seconds of video content.
Videos must not contain sensitive, offensive, or NSFW content.
Avoid asking questions that rely solely on commonsense knowledge and do not require viewing the video. Such questions do not meaningfully test video understanding.
Avoid questions that are too easy to guess based on social priors or language bias alone. For example, question like "Did the teacher appear impatient when students repeatedly inter- rupted the class?" may be too easily answered with "No" due to cultural expectations of teacher behavior, regardless of actual video content. This undermines goal of evaluating visual understanding.
Do not directly convert characters' spoken lines into questions. These are typically answer- able via simple string matching or keyword retrieval, which again does not effectively test video comprehension.
Balance the number of questions with answer "Yes" and "No".
C.2 QUALITY CONTROL
The annotation process includes the following quality control stages:
Stage 1: Candidate annotators complete a trial task, collecting video and labeling sponding QA pairs. The authors review the submission and provide feedback. Once the anno- tator demonstrates clear understanding of the annotation guidelines, they proceed formal annotation.
Stage 2: The annotator submits batch of 10 videos with corresponding QA pairs. The authors randomly review them and provide feedback. The annotator revise the entire batch accord- ingly. If the qualified rate of the submitted questions is below 90%, the authors re-sample the revised batch for further inspection. Otherwise, the batch is accepted. Annotators who pass this stage on the first attempt can proceed to Stage 3.
Stage 3: The annotator submits a batch of 30 videos with QA pairs. The authors randomly inspect 5 of them and provide feedback. The annotator revises the full batch needed. If the QA qualified rate is below 90%, a follow-up review the revised batch conducted. Otherwise, the batch is accepted.
Two authors are involved in the quality control process.
C.3 ANNOTATOR INFORMATION
All annotators are from a commercial data annotation company. We have a contract with this com- pany and compensate them at market rates for the annotation work. All annotators college graduates with strong English proficiency. A total of ten annotators participate in the annotation of M3-Bench-web.
D LONG-TERM MEMORY DESIGN
The long-Ten Ter-Teemman memory of M3-Agent is implemented as entity-centric, multimodal graph. Each node represents a distinct memory item, and each edge represents logical relationships between memory items. The attributes associated with each memory node are listed in Table 8.
21

---

Published as a conference paper at ICLR 2026
Table 8: Attributes and their descriptions for a memory node.
Attribute
Description
unique identifier node.
The modality type of the node (e-g., text, image, audio). For example, natural language memory stored as a text node, face as an image node, and spoken dialogue as an audio node,
type
content
The raw content of the node, such as plain text, base64 image, or base64 audio.
embedding
The vector representation of the node content, used for similarity retrieval.
weight
numeric value indicating the confidence of the node.
extra data
JSON object containing additional metadata, such as timestamps.
Conflicting information may be introduced during memory construction. To address this, M3-Agent applics weight-based voting mechanism, where weight of each entry reflects its activation fre- quency. For example, <vo ce.3>> corresponds <face-0>, but in some challenging clips, the system might temporarily link it to different face. Over time, as the correct association is acti- vated more frequently, the weight of the correct mapping (svoice_3) 3>, <face_00 increases and dominates. This allows the system to robustly learn and maintain accurate knowledge, even in the presence of occasional local errors.
In long-term memory, M3-Agent associate multimodal information, even across different time steps. For instance, when a voice appears without a face, M3-Agent generates the related episodic and semantic memories with the corresponding voice ID (e.g., <voice.00 If the corresponding face (e.g. <face.1>> ,1> is observed in any past or future, M3-Agent can associate the two modalities. Consequently, all knowledge related <voice_0> can be also accessed by < <face.1>.
E IMPLEMENTATION DETAILS OF TOOLS
Here, provide the implementation details of the tools for representation extraction introduced in Section 4.2.
Facial Recognition To perform facial recognition, uniformly sample video frames rate of 5 frames per second. For each sampled frame, we employ the buffalo. predefined model suite from the In.sigh F library to extract facial attributes, including bounding box coordinates, identity embeddings, and detection/quality scores. Low-quality detections-such as those with ab- normal aspect ratios or extremely low confidence scores- are discarded. We then apply HDB HDBSCAN clustering on the embeddings of the remaining high-qualit faces to group them by character iden- tity. This yields a set of reliable facial representations, clustered by character.
Voice Identification For speaker identification, we use Geemini-1.5-Prre to extract audio segments corresponding to distinct speaker voices, while simultaneously performing automatic speech recog- nition (ASR) each segment. Segments shorter than 2 seconds are filtered out to ensure reliability. We then apply voice embedding model (RChen et al., 2024a) to encode each segment into speaker-specific representation. Based on the resulting voice embeddings, we cluster and merge segments that correspond to the same speaker-i.ce those with similar vocal characteristics. This process produces set of high-quality speaker representations, also grouped by character. The prompt used for voice processing is shown in Table
The Prompt for Voice Processing
You given video. Your task is to perform Automatic Speech Recognition (ASR) and audio diarization on the provided video. Extract all speech segments with accurate timestamps and segment them by speaker turns (i.e., different speakers should have separate segments). but without assigning speaker identifiers.
(Continued on next page)
22

---

Published as a conference paper at ICLR 2026
Return a JSON list where each entry represents a speech segment with the following fields:
start time: Start timestamp in MM:SS format.
end time: End timestamp in MM:SS format.
The transcribed text for that segment.
Example Output:
"start.time:: "00:05", "end_time"; "00:08", "asr": "Hello, everyone.",
"start time": "00:09", "end time": "00:12", "asr": "Welcome to the meeting."
Strict Requirements:
Ensure precise speech segmentation with accurate timestamps.
Segment based on speaker turns (i.e., different speakers' utterances should be separated).
Preserve punctuation and capitalization in the ASR output.
Skip the speeches that can hardly be clearly recognized.
Return only the valid JSON list (which starts with "(" and ends with "I"") without additional
explanations.
If the video contains no speech, return an empty list ("O"".
Now generate the JSON list based on the given video:
Table 9: Prompt used for voice processing.
Search All memory-based retrieval is implemented via Maximum Inner Product Search (MIPS), with modality-speciff adaptations.
Each face and voice node maintains a set of representative feature snapshots. When new face or voice features are extracted from video clip. we compute the average cosine similarity between each extracted feature and all stored snapshots per node. The node with the highest similarity ex- ceeding a pre-defined threshold (0.3 for image, 0.6 for voice) is considered a match; otherwise, new node is created. Matched nodes are updated with the new features to refine their representations over time.
For textual memory, apply MIPS between the input query and all existing text nodes, using Ope- nAl's text text-eebeeeeeeee embedding-- 1aainng---aaas 3-1argel as the embedding model. To support multi-entry retrieval, apply a top-k retrieval with similarity threshold t. Specifically, we return the k most relevant nodes whose similarities exceed t. To ensure retrieval coherence, we also perform clip-level re- trieval: each clip is scored by the highest similarity among its memory entries, and return the top-ranked clips accordingly. For all experiments, we adopt a relatively strict hyperparameter setting (k 2, 0.5) reduce retrieval randomness and enable consistent evaluation across models.
F DEMONSTRATION DATA SYNTHESIS FOR MEMORIZATION
During memorization, the multimodal model takes inputs including: video, audio, facial identifica- tions (via facial recognition). and voice identities (via voice identification). It generates two outputs, episodic memory and semantic memory. To construct training data, segment training videos into 30-second clips. For each clip, we then synthesize the corresponding episodic memory, entity identity relationships in semantic memory, and other semantic memory, detailed below. In total, we synthesize 10,752 training samples for 200 validation samples.
F.I EPISODIC MEMORY SYNTHESIS
We employ hybrid synthetic strategy that integrates the complementary strengths of Gemini-1.5. Pro and GPT-4o. Gemini-l.5-PPr supports audio inputs and excels at generating high-level, event- based descriptions, whereas GPT-4o provides more fine-grained visual details. To leverage both
hhttps:// com/ hatam/inex///eee new-
htwp:/ieeeeeeeeeeeeeeenn.. embeddi inninng-ooo model --and-app updatess/
23

---

Published as a conference paper at ICLR 2026
Table 10: Explanations of different memory types.
Memory Type
Explanation
Specific events or experience, capturing not just what happened, but also when, where, and in what context. The episodic memory should captures details such as the people involved, their appearance, actions and spoken words, and the broader environment.
Episodic Memory
Character-Identic Equivalence: Captures equivalence relationships across dif- ferent character modality identity
Character- Level Attributes: Extracts attributes for each character, such name, personality traits (e.g., confident, nervous). role or profession (e.g., host, newcomer), interests, and background information.
Interpersonal Relationships: Describes the relationships and interactions among characters, such as social roles (e.g., host-guest, leader-subordinate) emotional tone (e.g., respect, tension), power dynamics (e.g., who leads), and evidence of cooperation, exclusion, or conflict.
Semantic Memory
Contextual and General Knowledge: Encompasses general knowledge inferred from the video, such as likely setting or genre (e.g., corporate meeting, game show), cultural or procedural norms, real-world facts (e.g., "Alice Market is pet- friendly"), common sense, and the functional roles or attributes of objects within the scene.
models effectively, we first prompt GPT-4o to generate detailed visual description of the video using frames sampled at 0.5 fps. This output serves as contextual input for Gemini-1.5-Pro, which then prompted to generate the final episodic memory. The prompt explicitly instructs Gemini- 1.5-Pro to incorporate information from GPT-4o's description when it deems it accurate. We find that using GPT-4o's detailed visual output as context significantly enhances the richness of the final memory produced by Gemini-1.51 P.r. The full prompt template is shown in Table 11.
Prompt of Episodic Memory Synthesis (GPT-40)
[Video] includes 16 frames of video.
Using this information, generate detailed description of the video. Following the requirements below:
Carefully describe the visual elements in each frame, noting colors, objects, movements, environment, people (including actions, clothing, expressions), and any noticeable details changes between frames.
2. If audio elements sounds are visible through textual or visual cues within the frames (such as subtitles, audio indicators, or written sound effects), accurately describe these details.
Do not speculate infer information beyond what is explicitly visible in these 16 frames. Avoid using external knowledge or assumptions.
4. Generate only the detailed deseription based solely the given frames. Do produce any additional commentary or explanations.
Prompt of Episodic Memory Synthesis (Gemini-1.5-Pro)
You provided with the following data: [Video]: : A video clip in mp4 format.
[Faces]: A list of facial features detected in the video, each linked to unique face ID (e.g.. <face_1>).
[Dialogues] A list of speech segments in the video, including start_time, end_time, speaker ID (e.g., <voi ce proice_20 _2>), and the corresponding transcribed text. [Reference Description A description of the video that may contain both accurate and inaccurate details.
Your Tasks:
(Continued next page)
24

---

Published as a conference paper at ICLR 2026
Based the video content and reference descriptions, generate detailed and cohesive description of the video clip. The description should focus on the entire event, incorporating all relevant aspects of the characters, their actions, spoken dialogue, and interactions in narrative format. The description should include (but is not limited to) the following categories: Characters' Appearance: Describe clothing, physical features, notable accessories, Characters' Actions & Movements: Describe gestures, movement across the scene,
interactions.
Characters' Spoken Dialogue: Quote-or, if necessary, summarize- content from the
dialogue track.
Characters' Contextual Behavior: Describe emotional states, relationships, roles, and reactions.
Environmental or Temporal Cues: Describe the physical setting and time-of-day visible.
Strict Requirements:
Incorporate correct elements from the [Reference Description] and correct any mistakes you identify.
Add any additional details visible inferable from the [Video], [Faces], and [Dialogues] that are missing from the reference.
Since the given dialogues may be incomplete, reconstruct the entire conversation from the raw audio as precisely as possible.
a character has an associated feature in the input context (either face or voice), refer them only using that feature ID (e.g.. <face 1>, <voice (roice_>>)
Use face (e.g., <face_11 .>) when the detail grounded visual data.
Use speaker ID (e.g., <voi ce_>>) when the detail is grounded in speech.
Do not use non-existent <face_ID> or <voi ce_TD>
We reiterate the above-mentioned list of available IDs here: [ID_ist]]
For characters without associated feature IDs, refer to them using a concise visual or contextual descriptor (e.g., "a man in blue shirt", "a young woman the window").
Do not use pronouns (e.g., "he", "she", "they") or inferred character names.
Your output should be Python list of well-formed, concise English sentences (one detail per sentence).
Example Output:
"In the bright conference room, <face_1s enters confidently, adjusting his black suit with a white shirt and tie. He has short black hair and wears glasses, giving a professional appearance as he approaches <face_2 2> to shake hands.",
"<face_2>, dressed in a striking red dress with long brown hair, smiles warmly and greets < fface_l>. She then sits down at the table beside him, glancing at her phone briefly while occasionally looking up.".
"<voi i ce 1> speaks to the group. 'Good afternoon, everyone. Let's begin the meeting." His voice commands attention the room quiets, and all eyes turn him.".
"<face.2> listens attentively to <voice_l>'s words, nodding in agreement while still occasionally checking her phone. The atmosphere is professional, with the participants settling into their roles for the meeting.".
"<face 1> adjusts his and begins discussing the agenda, engaging the participants in productive conversation."
Please only return the valid string list (which starts with "[" and ends with "J"), without any additional explanation or formatting.
Table 11: Prompt templates used for generating synthetic episodic memory.
25

---

Published as a conference paper at ICLR 2026
F.2 ENTITY ID RELATIONSHIP DETECTION
There is special type of semantic memory, extracting cross-modal identity equivalences from video. This remains challenging task, even for advanced models like Gemini-1.5-Pro, particu- larly in scenes with multiple faces and voices (He et al., 2024b). To address this, we propose progressive annotation algorithm. The key idea to identify meta-clips, segments containing ex- actly one face identity and one voice identity, from the raw long video. These meta-clips are used to build a meta-dictionary that maps voice IDs to face IDs across the entire video. This dictionary enables automatic annotation of any 30-second clip extracted from the original video.
Meta- Meta-Clip Extraction First, for long video, we can use facial recognition tools and voice identity tools introduced in Appendix E to construct corresponding global ID for each face and voice that appears in the video. Next, we segment the video into a series of short clips, each no longer than seconds in duration, using keyframe- based division. This method ensures that each clip is visually stable, with minimal changes in characters or scenes. Then, we apply facial recognition and voice identity tools each short clip individually
extract the faces and voices present, along with their global IDs. If clip contains only one face ID and one voice ID, we refer to it as meta-clip. In this case, it is highly likely that the face and voice in the clip belong to the same person. Therefore, we can use the meta-clip as a high-confidence sample for establishing the association between faces and voices.
Algorithm Meta-Dictionary Construction
Require: A long video threshold p Ensure:
A mapping dictionary M y + F from voice IDs to face IDs
Extract global face ID set
fN and voice ID set V {Vi,..., uN) from video
V
2: Divide V into sequence of short clips C {c1, C2...... ,r. Cr using keyframes- based segmenta- tion
3: Initialize meta-clip set Cmeta-
4: for C, C do
5: Detect face set Fr F and voice set V3 V Vnine in
6: if [Fi 1 and [V]]= = then
7: Add pair (cz, f. where f Ft, v. E Vi to Cmetta
8: end if
9: end for
Construct bipartite graph G (F, where edge (S)v) has weight: w(f,))
H(ce,f. E Cmetall
Remove all edges from G with weight equal to
12: for f E do
Net = (1; (f,
14:
Let = arg maxMEN, w(f,v) u(f.v²) Ex,oa, w(F.e.) 2 p then
15:
if
16:
Keep only edge (f)"") and remove others
17: else
18:
Remove all edges incident
19:
end
end for
21:
for
E V do
22:
Let Let = ( E
23: Let f arg max maax, w(fs, v)
24: Keep only edge (f", and remove others
25: end for
26: Initialize mapping dictionary M 0
27: for v) E E do
28: Add mapping M[u] t-
29: end for
30: return M
26

---

Published as a conference paper at ICLR 2026
Meta-Dictionary Construction Based on all meta-clips extracted from the long video, construct a set of mappings between face IDs and voice IDs. However inconsistencies may arise due to a small number of clips where the speaker is not visible. To address this issue, employ voting mechanism to generate the final meta-dictionary. The detailed algorithm is described in Algorithm 2.
use it annotate arbitrary clips from the full-length video. Specifically, for each 30-second clip, if both a face ID and voice ID appearing in the clip and also found in the meta-dictionary, we generate semantic memory in the form: "Equivalence: <face_id>, <voice ce_id>>" Since not all IDs can be found using the meta- dictionary, we reject any clip containing voice ID that is not present in the meta-dictionary from the final training dataset for memorization. In total, we collected 10,952 30-second clips with valid identity equivalence annotations. We manually review randomly sampled mappings, and found the accuracy to be 95.83%.
New-Clip Annotation After obtaining the meta-dictionary,
F.3 SEMANTIC MEMORY SYNTHESIS
To construct semantic memory, we adopt a hybrid strategy similar to that used for episodic mem- ory. We define several key dimensions that semantic memory should address, outlined in Ta- ble 10. Specifically, we first prompt GPT-4o to generate preliminary semantic memory based on video frames and episodic memory. Next, we provide the video, episodic memory, and GPT-4o- generated semantic memory to Gemini- 1.5-Pro, prompting it to produce the final semantic memory. Detailed prompts are provided in Table 12.
Prompt of Semantic Memory Synthesis (GPT-4o, Gemini-1.5-Pro)
You are provided with the following data:
[Video]: 16 frames of video. (.5-ro Variant: video clip in mp4 format.)
[Faces]: A list of facial features detected in the video, each linked to unique face ID (e.g.. <face 1>).
[Dialogues]: A list of speech segments in the video, including start time, end time, speaker ID (e.g., <voi ce_2>), and the corresponding transcribed text.
(Video Descriptions:: A description of the video.
(Gemini-1.5-Prr Variant: (Refence conclusions]: A list of high-level conclusions that may contain inadequate incorrect information.)
Your Task:
Based the given character features, video content, and reference conclusions, generate list
of high-level, reasoning- based conclusions within the scope of the following category:
1. Character-Level Attributes
Infer abstract attributes for each character, such as:
Name (if explicitly stated),
Personality (e.g., confident, nervous),
Role/profession (e.g., host, newcomer),
Interests or background (when inferable).
Distinctive behaviors traits (c.g.. speaks formally, fidgets).
Avoid restating visual facts-focus on identity construction.
2. Interpersonal Relationships & Dynamics
Describe the relationships and interactions between multiple characters:
Roles (e.g., host-guest leader-subording
Emotions or tone (e.g., respect, tension),
Power dynamics (c.g., who leads),
Evidence of cooperation, exclusion, conflict, etc.
For individual character or cases where character relationships cannot be determined, do
not generate conclusion relevant to the corresponding character.
(Continued on next page)
27

---

Published as a conference paper at ICLR 2026
Vidco-Level Plot Understanding
Summarize the scene-level narrative, such as:
Main event or theme,
Narrative
sequence (e.g., intro discussion reaction),
Overall tone (e.g., formal, tense),
Cause-effect or group dynamics.
Do not involve specific characters.
4. Contextual & General Knowledge
Include general knowledge that can learned from the video, such as:
Likely setting or genre (e.g., corporate meeting, game show),
Cultural/procceuuall norms,
Real-world knowledge (e.g.. "Alice market is pet-friendly").
Common-seme senoon or format conventions.
Attributes and functional roles of objects in the video (e.g., the trash bin is used for disposing of kitchen waste).
Output Format:
Python list of concise English sentences, each expressing one high-level conclusion.
Do not include reasoning steps or restate input observations. Only output the final conclusions.
Strict Requirements:
Only include conclusions under the given category. Do not go beyond it.
Your conclusions must be informed by the video and reference content.
Each conclusion should reflect deeper reasoning and insight, not surface-level observations
already evident from the plot description.
a character has an associated feature in the input context (cither face or voice), refer to
them only using that feature ID (e.g., < <face_ .1>, <voics re 2>)).
Usse face ID (e.g., <face_1> when the detail is grounded visual data.
Use speaker ID (e.g., <voice _1> when the detail is grounded in speech.
Do not use non-existent <face_ID> or <voice_ID>.
We reiterate the mboteomentioned list of available IDs here: [IDJiis]]
For characters without associated feature IDs. refer to them using a concise visual or
contextual descriptor (c.g.. "a man in blue shirt", "a young woman by the window").
Do not use pronouns (e.g., "he", "she", "they") or inferred character names. Maintain strict accuracy referring to characters and their correct IDs descriptions.
Do not restate the input observations reasoning steps-only output the final, distilled conclusions.
Your output should be Python list of well-formed, concise English sentences (one per item).
Example Output (Note: example only represent the format, fully corresponding the provided category):
"<face.1>'s name is David.",
"<face.1> holds position of authority, likely as the meeting's organizer or senior
executive.".
"<voice 2> shows social awareness and diplomacy, possibly indicating experience in public client-facing roles.".
"<face_1> > demonstrates control and composure, suggesting a high level of professionalism and confidence under pressure.".
"The interaction between <face_1> and <voice.2> suggests a working relationship
built mutual respect.".
"The overall tone of the meeting is structured and goal-oriented, indicating it is part of larger organizational workflow."
(Continued on next page)
28

---

Published as a conference paper at ICLR 2026
Please only return the valid string list (which starts with "[" and ends with "J") without any additional explanation formatting.
Table 12: The prompt used in generating synthetic semantic memory.
F.4 QUALITY OF THE SYNTHETIC DATA
Although the demonstration data is synthetic, it is of high quality. Our synthetic memory averages 245.7 words for episodic memory and 276.2 words for semantic memory, compared to 151.3 and 81.4 words respectively for Gemini-1.5-pro. indicating our memory captures more detail. For con- tent accuracy, we randomly sampled 10 clips from different videos, totaling 353 memory items. Manual review showed an accuracy of 95.5%. Most errors stemmed from the speaker recognition tool: background noise and overlapping speech occasionally caused minor omissions or misidenti- fications in extracting speaker dialogue for episodic memory.
G EVALUATION OF MEMORIZATION
we evaluate the memorization model during training using held-out validation set of 200 samples and select the best checkpoint. Two evaluation metrics are used. First, AutoDQ (Wang et al., 2024a) assesses memory description quality by comparing generated outputs to reference descriptions, mea- suring episodic and semantic memory excluding identity equivalence. Second, for identity equiva- lence. we compute precision, recall and Fl score against ground in the validation set. Based on the results in Table 13, we select the checkpoint obtained after training for epochs. For additional comparison, we also report results from two baseline models, memory-gemt n -prompt. and memory-76 prompt, on the same validation set. Our model, memory- 7b sft, significantly outperforms both baselines.
Table 13: Evaluation of memorization models using AutoDQ and Equivalence (Eq.) metrics. Here, P. R, and F1 denote precision, recall, and the F1 score, respectively.
Model
AutoDQ-P
AutoDQ-R
AutoDQ-F1
Eq.-P
Eq.-R
Eq.FFI
memory--emmai-prropp
0.692
0.539
0.606
0.472
0.805
0.595
memory-7b-prropp
0.495
0.355
0.414
0.117
0.192
0.145
memory-7b-sfr (1 epoch)
0.634
0.596
0.616
0.742
0.817
0.778
memory-7b-sts epochs)
0.628
0.610
0.619
0.845
0.810
0.827
memory-7b-se (3 epochs)
0.635
0.620
0.627
0.836
0.856
0.846
hemory-7b-sse ft (4 epochs)
0,616
0.618
0.617
0.825
0.839
0,832
memory- 7b- ft epochs)
0.609
0.621
0.615
0.813
0.840
0.827
H RL TRAINING DETAILS
H.1 DAPO TRAINING
Table 14 lists the hyperparameters used during the training process. Figure depicts the RL training curves, which show a steady increase in score with the training steps.
H.2 GRPO TRAINING
We also use Group Relative Policy Optimization (Shao) et al., 2024) to optimize the policy model in the ablation study. GRPO optimizes the policy model ny by maximizing the following
29

---

Published as a conference paper at ICLR 2026
Table 14: The hyperparameters used in DAPO training.
Model Size
Parameter Name
14B
32B
Batch Size
GPU with 80GB memory
16
16
32
Rollout Model Parallel Size
2
Learning Rate
le-6
le-6
le-6
Maximum Number of Rounds H
5
5
Number of Samples in Group G
Total Steps
180
180
180
Clow
0.2
0.2
0.2
Chigh
0.28
0.28
0.28
0.30
0.54
Aeimp: Noor
0.32
0.4
0.4
046
200
200
Figure 4: Average scores (on training set) and accuracy (on dev set) curves during the DAPO train- ing process. The smoothing method of the curve in the left figure is the exponential moving aver- age(EMA) formula that aligns with the one used in WandB, and the smoothing weight is set to 0.9
objective:
I(T..) min TolTr, te,ea) Ace.
(0)= (x (
G
E [(Ta,)
ne (Ta Ti,se)
clip
To(Tuel
1 -,, 1 ()ee)) xDxr [Te]]
(3)
Tall (T,
[r]
Te((FelFee) (T(TTTTe)) no (T<TT<<)) -1)
(T)(T=])Tt)
DEL [Tt]]mae]
Σ I(m) 1
log
Σ1
I(T))
where 6 and B are set to 0.2 and 0.01 respectively, and the other hyperparameters are the same as those in DAPO training.
I BASELINE IMPLEMENTATION DETAILS
We evaluate M3-Agent against three types of baselines:
Socratic Models This baseline adapts the Socratic Models framework (Zeng et al., 2022), which multimodal model describe 30-second video clips. These descriptions are stored as long-term memory. To answer a question, an L.M performs retrieval augmented generation (RAG) (Lewis al., 2020): first invokes search.clip ip function retrieve memory relevant to the question, and then generates response based on the retrieved content.
We implement both closed-source and open-source multimodal models for memory generation:
Gemini-1.5-Pra (Team et al., 2024): Takes the full 30-second video clip as input.
GPT-4o (Hurst et al., 2024): Since it does not process audio, we provide video frames sampled at 0.5 fps and ASR transcripts.
Owwe2.55-mmii 7b (Xu et al., 2025a): An advanced open- open-source multimodal model that supports both visual and audio inputs. It receives the full video input.
30

---

Published as a conference paper at ICLR 2026
Dw-n2.5-VL-7b (Bai et al., 2025): An open-source vision-language models with SOTA results in visual-language tasks. Like GPT-4o, receives both video frames (sampled at 0.5 fps) and ASR transcripts.
For all variants, GPT-4o serves as the LLM for RAG-based question answering. We perform exten- sive prompt engineering optimize each setup. All prompts listed in Appendix M.2.
Online Video Understanding Methods We further compare our approach with three online video understanding frameworks: MovieChat (Song et al., 2024), MA-LMM (He et al., 2024a), and Flash- VStream (Zhang et al., 2024b). Unless otherwise specified, we adopt their official pretrained weights and default configurations.
MovieChat (Song et al., 2024): It uses a sliding-wwind window to extract frame-level features and stores them in hybrid memory; the LLM performs QA conditioned on this memory.
MA-LMM (He et al., 2024a): processes frames in an online manner, consisting of feature extraction (1 fps), temporal modeling (100-frame input), and LLM decoding.
Plash-VStreas (Zhang al., 2024b): It adopts two-stage asynchronous pipeline: stream video frame compression (1 fps), and LLM-based QA over the compressed features.
Agent Methods We also compare M3-Agent with agents implemented via prompting closed-source commercial models. Specifically, we consider the following two baselines:
Gemini-Agent: Gemini-1..-Pro is prompted separately for memory access and control pro- cess. During memory access, it receives the full video with audio, facial recogmition re- sults and speaker identification results to generate episodic and semantic memories, denoted as memo ry-gemi -proomppt. In the control process, it performs memory searches and gen- erates responses, referred as control -ii---ggninii--ptt.....
We also evaluate setup where GPT-4o is prompted to perform memory search and generate responses (control-gee (controo-gpp4--prommpp) The memory access remains handled by memoryy-gemmii-pronn..
The prompts are provided in Appendix M.3.
J CROSS-MODEL ASSESSMENT OF AUTOMATIC EVALUATION OF M3-BENCH
To ensure the robustness of results obtained by GPT-4o automatic evaluator, we perform a cross- model validation using two additional evaluators. Specifically, we employ Gemini-1.5-Pro and Qwen3-32B to evaluate all baselines and our M3-Agent on M3-Bench, using the same prompt for GPT-4o shown Table 23. The corresponding results are presented in Table 15 and Table 16. Both alternative evaluators consistent with the evaluations produced by GPT-40 shown in Table 4, thereby confirming the reliability of the automatic evaluation.
STUDY ON MEMORY OF K
K.1 ABLATION STUDY
We perform an ablation study to evaluate the conflict resolution mechanism in M3-Agent. Specif- ically, we remove the voting mechanism. When conflict arises in memory. the system retains the most recent memory entry. The results, shown Table 17, highlight the importance of the voting mechanism: without it, QA accuracy drops by 3.2% on M3-Bench-robot 2.5% on MS-Bench-web, and 4.5% on Video-MMEELLngg These declines demonstrate the voting mechanism is crucial for resolving contradictions and maintaining overall performance.
We ablate the contribution different subtypes of semantic memory. Table 17 reports the results of removing attribute memory, relation memory, and rule memory. The results indicate that
31

---

Published as a conference paper at ICLR 2026
Table 15: Results on M3-Bench-robot and M3-Bench-web, evaluated by Gemini- automatic
evaluator.
Method
M3- Ro-Bhench-root
M3-Benchhweeb
ME
MH
CM
PU
OK
All
ME
MH
CM
PU
GK
All
Socratic Model
Qwen2.5-Onie --22----m
2.2
1.7
1.7
1.6
2.1
2.2
9.1
9.2
14.4
10.9
14.4
11.6
Qwen2.5-VLL VL-7b
3.4
3.5
3,6
4.6
4.0
3.8
11.5
10.3
13.2
14.2
21.7
15.2
Gemini-1.5-pe
6.9
7.8
7.6
9.9
8.6
8.3
18.5
18.4
23.3
23.1
30.0
23.7
GPT-4o
10.0
8.4
8,4
10.4
8.0
8.7
21.0
21.4
31.8
27.1
41.8
29.3
Online Video Understanding Methods
MovieChat
13.6
9.0
12.8
16.2
7.3
11.3
12.5
5.5
12.3
17.4
11.7
12.7
MA-LMM
26.5
24.3
24.4
39.2
17.1
25.5
27.4
10.7
22.9
39.9
17.4
25.1
Flash-VStream VStream
22.8
20.9
20.1
26.1
15.3
20.5
24.6
10.9
25.1
32.1
20.7
23.9
Agent Method
Gemini-Agent
16.4
15.9
15.8
20.0
15.5
16.7
28.6
20.3
34.5
33.9
44.0
33.5
Gemini-GPT4ov GPT4o-Hybrid
22.5
27.0
37.3
44.5
52.8
M3-Agent
32.9
28.1
30.5
42.6
18.8
30.3
46.5
28.4
43,9
59.4
53.3
48.8
Table 16: Results on M3-Bench-robot and M3-Bench-weh, evaluated by Qwen3-32B automatic evaluator.
Method
Bo--hench-robot
M3-Bench-web
PU
All
ME
CM
PU
GK
Socratic Model
Qwen2.5-Omni-73
1.4
1.7
1.5
2.0
8.6
9.2
14.6
10.8
14.6
11.5
Qwen2.5-VL--b
3.1
3.2
3.4
4.6
3.7
11.5
10.3
13.4
14.0
22.1
15.2
Gemini-1.5-pra
8.3
18.4
18.6
23.6
23.1
30.0
23.7
GPT-4o
8.2
10.4
8.0
21.1
21.7
32.5
42.0
29.5
Online Video Understanding Methods
MovieChat
13.4
8.7
13.2
15.9
8.0
11.3
12.7
5.5
12.0
17.4
12.0
12.8
MA-LMM
26.5
24.6
24.6
39.4
17.4
25.6
27.4
10.9
22.9
40.2
17.6
25.3
22.3
20.6
20.1
25.7
14.7
20.3
25.1
10.8
24.0
32.3
20.9
23.8
Agent Method
Gemini-Agent Gemini-GPPi -ybo-i
16.1
17.3
15.8
20.2
15.1
16.9
29.7
19.5
33.6
33.8
44.0
33.2
21.5
26.8
23.3
29.5
24.0
24.8
36.0
26.0
37.9
43.8
$2.0
41.2
M3-Agent
31.6
27.8
29.6
42.3
17.7
29.6
45.1
28.4
42.8
59.7
53,9
48.8
1400
1209
Namteer NfNerrrs Bbo
MeMurr
400
111
M0-gue
A10
200
Corl Cot.eitece
25
Video Eanglh (ntin)
Iduntk Cozze
(a) Memory item growth and cumulative QA accu- racy video length increases.
(b) Memory size and QA accuracy across varying numbers of identities.
Figure 5: Analysis memory size and QA accuracy with increasing video duration and identity count.
32

---

Published as a conference paper at ICLR 2026
Table 17: Ablation study of the conflict resolution mechanism and semantic memory types.
Method
M3-Bench-robot
M3-Bench-web
Video-
ME
MH
CM
PU
GK
All
ME
MH
CM
PU
GK
All
MME--Lng
w/o voting
28.6
26.6
26.1
39.1
18.7
41.8
20.1
57.8
$1.9
57.3
w/o attributes
25.7
23.5
25.4
35.3
15.8
24.0
35.5
22.7
35.4
47.1
42.4
38.9
49.7
w/o relations
24.2
21.9
22.9
31.1
13.4
22.9
39.7
24.6
38.9
51.4
47.9
42.1
51.6
w/o rules
25.8
23.6
24.2
33.1
14.9
23.6
40.9
25.3
39.6
53.5
49.0
42.7
54.9
M3-Agent
32.8
29.4
31.2
43.3
19.1
30.7
45.9
28.4
44.3
59.3
53.9
48.9
61.8
K.2 PERFORMANCE SCALING WITH VIDEO DURATION
To evaluate how memory growth affects the memory system, we plot memory size and cumula- tive QA accuracy on M3-Bench-rohot as functions of video duration, shown in Figure 5a. The results show that memory size increases approximately linearly with video length. QA accuracy increases slightly videos become longer, indicating that the system maintains robust performance even as memory grows. Furthermore, M3-Agent consistently outperforms the strongest baselines, prompting based agents using Gemini-1.5 1.o and GPT-40 across all video durations.
K.3 IMPACT OF IDENTITY GROWTH
To examine how identity count affects the memory system, we plot memory size and QA accuracy against on the number of identities M3--ench-weeh We 13-Bench-web because it provides a wide range of identity counts while keeping video length upproximately constant (an essential control, video length is the dominant factor affecting memory size). The results shown in Figure 5b. we find that both memory size and QA accuracy remain largely stable as the number of identities increases. Notably, M3-Agent consistently surpasses the strongest baselines across all identity-count categories. For videos with many identities, the performance gap widens more, demonstrating that M3-Agent is more robust than baseline methods when handling more complex scenarios.
L CASE STUDY
Entity-Centric Long-Term Memory Standard Memory Table 18 compares M3-Agent mem- ory with traditional memory generated by Gemini-1.5-Pro and GPT-4o. M3-Agent features greater consistency due its entity-centric memory structure and accumulates richer world knowledge through semantic memory generation.
M3-Agent maintains consistent person identities. Even when speakers' faces never appear on screen, M3-Agent correctly distinguish them and associate their voices with <character..r or <charact er __16 16>>>>>>> in long-term memory. In contrast, Gemini- describes them only "an off-scceenn voice", while GPT-4o fails to assign any consistent identity.
M3-Agent generates semantic memory in addition episodic memory, effectively expanding the utility of the memory. For example, answering "Is Lucas skilled as cooking?" can not rely on episodic memory alone, as relevant events cannot be retrieved by key words such Lucas or cooking skills. However, the semantic memory "<character 23> is named Lucas; <cha racter 2.3> demonstrates a willingness to learn and improve cooking skills" is retrievable and directly supports correct answer.
Memorization Table 20, 21 present two examples illustrating the episodic and semantic memories generated during memory access. Compared to memory memory-gemini-- -prommppt, wiemory-7b-s sft demonstrates (1) more detailed episodic memory generation, including richer scene descriptions, character actions and expressions, and dialogue; (2) improved recognition of identity equivalence, enabling consistent long-tern term tacking human identities; and (3) richer semantic memory extrac- tion, proactively generating knowledge about characters and environments.
33

---

Published as a conference paper at ICLR 2026
Video: http://www..oottteee con/watchhyy
Time: 14:00 14:30 (alko shown in Figure 2)
Question: Is Lucas skilled at cooking?
Gemini--1.-PPer
In brightly lit kitchen, Lucas, wearing a dark short-sleeved shirt and dark panis, stands at marble countemop. carefully chopping green onions on wooden cutting board.
A voice off-screen, instraces him: "Use the best part of the knife and let the knife cur. Very nice, easy."
Another voice, also off-sereen, exclaims, "This really hand!"
Locas's father replies. "The flat side should be on the bottom, too."" Locas then picks up the slioed onion greens from the board, leaving the white parts.
Lucas's fäther asks, "Alright, tell me looks..." Lucas's hand is blurred, appears he hrushing away cut onion pieces. off screen voice says. "Okay, is Lucas's a second attempt" Lucas some the white parts the onion sections. He stops pat on a pair glasses, saying "Oh, The other olf-screcn voice comments. "Oh. that's pretly good!" I Lacas's fäther advises, "Your angle on your knife a little too high. Keep it low.""
Now wearing glasses, Luces resumes chopping the white parts the onions.
Lucas udjusts his knife angle, as his father says. "Yeah. perfect. Good job.
Locas continucs chopping the cemaining white parts of the onions.
GPT-4o:
The video clip captures casual scene in a modem kitchen, highlighting learning moment between two indivilunls. The setting features a spacious kitchen with clean, white cabinetry, glass-pancled cuphoards showcasing neally stacked disbes and bowls, marble countettops, and stainless steel appliances, including smitable for instructionl interaction.
with distinct red knobs. The uppeurs well-lit, with and lighring creating welcoming atmosphhere
long- sleeve shin, leans over the counter on one side, providing guidance to the other penson, who is activelyy chopping ingrediens. This individual speaks calmly, offering encouraging and instractional remarks:: "Lae the best part of the knife and let the knife cut. Very nicz and informal supporiive dynamic.
first segment the clip, one individual. wearing plain
Thcir tome is sieady, suggesting patieace and mentoring. They cxhibit relaxed body language,
arms resting on counter, adding
The other individual, wearing black 1-shirt and silver chain bracelet their left wrist, stands the opposite side of the counter. They focused on slicing long green onions (scallions) into thin julienne cuts wooden cutting board. Their movements are deliberate but not entirely confident, as they appear to be learning or refining their knife skills, Al one peint, they express mild frustration, stating, "This is really hard," while continuing adjust cutting technique. As this occurs, the encouraging individual responds affimnatively with short phrases such as "Yeah" acknowledge their effort.
noticcable shift, person chopping announces, "Dkay, this Lucas's 's sccond attempt" referring thenselves in third person, emphasizing their ongoing learning process, They temporarily pause their activity to put on glasses, stating, "Wait. I'm going to put on my glasses this time," followed by an exclamation, "Oh, I can see," saggesting improved confidence or clarity in their ability to focus on the task at hand. This moment draws lighthearted reaction, subdly changing the stmosphere one optimism.
The individual the white shirt continues provide constructvve feedback. guiding Lacas their knife angle, saying. "Your angle your knife little too high." As Lacas adjusts their technique, improvement, as Lucas saccessfully slices the scallions more precisely, creating uniform pilc of thin picces on the cutting board.
mentor reassures them: "Keep it low. Yeah, perfect. Good joh This results in tingihle
Throughout the clip, the interaction conveys a sems of camaraderic and suppoct in calm, domestic setting. The kitchen enviromment, combined with the dialogue and actioes, suggests an emphasis on sharing and guidance. perhups during a light evening activity. The modem, tidy surroundings reflect atmosphere conducive bonding and learning cooking tochniques.
M3-Agent:
Episodic
scharacter.Co instructs ccha ca er.err 23 on how slice scallions, emphasizing using the best part of the knife and letting the knife do the woek.
<character, D> comments that slicing easy.
<chacacter.16 6 says, "This is really hard""
<character 23> wearing a black shirt slices scallions on wooden curting hoard placed on white marble countertop.
<character_D> > tells the person to place the flat side of the scallion on the bottom.
<chacacter. 6x says. "Okay, this is Lucas's 's second attempt."
<character.23 >> attempts to put on his glasses.
<character_23: says, "Ob, can see."
<charactez 23> continues slicing scallions.
<chacacter_166 comments, "Oh, that's pretly good!"
<character.D> comments that the person's knife angle is little high.
<character_D> instricts the person to keep the knife low and prnises their slicing perfect and a good joh.
<character 23> continues slicing the scallions with the knife held lower.
The kitchen has white cabincts with glass doors, a white subway tile backsplash, and stainless steel appliances. There ar plates and glasses visible inside the cahinets. Semantic
scharacter, D> displays parience a supponive attirude.
<chacacter.22 23 named Lucas.
<characten 2 33 demonstrates willingness learn and improve culinary skills.
<character 0> takes
instractive role, offering specific feedback <character 23>.
Kchacacter 2 20 is the father of ccharacter 223
The video demonstrates a culinary lesson focused on the proper technique for slicing scallions.
The overall tone is encouraging and supporive, fostering positive learning environment.
The feedback provided contributes to a cleur cause and effect relationship between knife handling technique and the quality of the slice. The video likely takes place a home kitchen, given the casual atmosphere and presence of everyday kitehenware.
act slicing scallions common culinary skill, often practiced in various cuisines.
wooden cutting board serves as a and stable surface chogping. proventing damage The instruction on using the 'best part' of the knife and keeping the flat side on the bottom commen knife skills for efficient and safe chopping.
and countertops.
The use of glasses <character. 22 demonstrates the importance of vision in precisc tasks like cooking. The stainless steel appliances and white suhway tile hacksplash are common features of modern kitchen design.
Table 18: Comparison of M3-Agent memory with traditional memory generated by Gemini-l.5-Pre and GPT-4o. M3-Agent features greater consistency due to its entity-centric memory structure and accumulates richer world knowledge through semantic memory generation.
34

---

Published as a conference paper at ICLR 2026
Control To illustrate the control process in detail, Table 22 presents complete generation trajec- tory of control-32b-r2 The input question is: "Is Tomasz a person with rich imagination or someone who lacks imagination?"
In the first round, the agent searches its memory for Tomasz's character ID. In the second round, having identified Tomasz <character it attempts direct query: "What is <character_44 _r>'s personality regarding imagination?" Finding no relevant memory in the third round, the agent reasons based on <character.4>'s role CTO of a company and generates a more targeted query: "What are <character_ creative problem-solving methods?" This yields a relevant memory: "<character his interest in scaling drone technology for personal flight."= piece of semantic memory. By the fourth round, the agent has collected enough information in context to generate the final answer.
is innovative and forward -tiaadiiiiniing. as evidenced by
Hard Case in M3-Bench The accuracy of various methods demonstrates that M3-Bench, partic- ularly no--eenchrrobot, presents a significant challenge. We perform a detailed error analysis of M3-Agent on M3-Bench. We randomly sample 50 failure cases from M3-Agent's responses on M3-Bench and examine their underlying causes. The resulting error categories are summarized in Table 19.
Table 19: Summary of common error types.
Error Type
Percentage
Explanation
Reasoning about fine- grained details
50%
The agent fails to extract precise, detailed information from its obser- vations.
Spatial reasoning
The agent fails understand spatial layout tracking spatial changes. 20%
Incorrect search strategy
12%
The agent does not correctly decompose the query or fails to execute an effective search plan to gather necessary information.
Missing Human
10% The agent fails to extract the relevant character names required the question.
Others
8%
The most common error type involves reasoning about fine- tin-ggraied details. For instance, questions like "Who wants to eat the ham sausage?" or "Which coat rack should Emma's 's hat be laced, taller one or shorter one?" require the agent to extract precise information from its observations. However, retaining all such details in memory is impractical and may cause cognitive overload. To address this, the agent must use attention mechanisms that enables selective memorization. During exccution, can develop task-specific world knowledge, allowing it to focus on relevant details while ignoring the irrelevant, therehy improving task performance.
Another category of hard cases is related spatial reasoning. In the M3- no--encc-rrobot, number of questions challenge the agent's 's capability on spatial cognition, such as understanding spatial layout and tracking spatial changes. Examples include: "Where the robot get the snacks?" and "Is Leo's water cup currently on the second or third shelf from the top of the rack?" Since verbal memory is generally less effective than visual memory for retaining spatial information, the long-term memory should be designed to incorporate richer visual content, c.g., snapshots, to better support spatial reasoning.
Vi deo (Illustrated as 12 frames)
(Continued on next page)
35

---

Published as a conference paper at ICLR 2026
Faces
<face_ 1413>
<face_1433 1412>
Dialogues
<voice_i> (00:03-00:12) Every, but feel like every sibling dynamic is different. was the youngest and I just took it. I just took didn't have the aggression fight back like <voice_2>> (00:00-00:02) these experiences through my children, (00: 1-00:16 close to, uh, it's neck and neck. Yeah, they they go after each other. (00:24-00:29) we 'ree gonna have the fish coming to the fish feed and then we re gonna have ducks coming too. It's gonna be a
<voice ce 8 2>: (00: (00:17-00:23 Wow. we're gonna we're gonna go fishing at the pond. We're gonna I'm gonna get a fish feeder and we're gonna have
memmory-emiin-prompp
Episodic memory
<face 1412> is wearing light blue t-shirt with a graphic design, a black baseball cap,
and mirrored sunglasses.
He is standing in backyard with a brick house, a wooden deck, fence, and some trees.
<voice_2 says he's reliving experiences through his children.
A woman, presumably his wife, walks by in the background carrying some items.
<voice_i> says every sibling dynamic different and that she was the youngest and just
took it.
<voice 2> says the children are neck and neck and go after each other.
<face 1412> and the woman are holding small yellow ducklings.
<voice_818 says they 're going fishing at the pond and will get fish feeder.
Two children are playing in the backyard with squirt guns.
<voice_812> says they'l '1l have fish coming to the fish feed and ducks coming too.
Semantic memory
<face 1412> appears be father who enjoys spending time with his family and doing outdoor activities.
<face_1412> and <face_14133 have a casual and comfortable relationship.
Thee video depicts a family preparing for a fishing outing their pond and discussing their children's sibling dynamics.
The setting is rural suburban home with large backyard and pond.
memory-7b-sft (Ours)
Episodic memory
14ace.14122 wears white t-shirt with a green and yellow graphic design, black baseball cap, and green mirrored sunglasses.
sunny day, in grassy area with brick building and wooden fence in the background.
<face-14 -111>> is outdoors
<face 1412> speaks to the camera, saying, "these experiences through my children,
which is"
<face.1413> joins <face 1 141>> wearing a white tank top and denim shorts.
<voice.1> says, "Every, but I feel like every sibling dynamic is different. I was the
youngest and I just took it. just took it. didn't 't have the aggression to fight back like"
<face 1412> points finger upwards.
<voice 2> says, "close to, uh, it's neck and neck. Yeah, they they go after each other."
<face_1413: and <face_1412> hold two small, fluffy yellow ducklings.
The ducklings are placed on the grass near a black pet carrier.
<face 1113>> kneels by the pet carrier, holding yellow duckling.
avoice_812> says, "Wow. So we're gonna we're gonna go fishing at the pond. We're gonna I'm gonna get fish feeder and we're gonna have"
(Continued on next page)
36

---

Published as a conference paper at ICLR 2026
<face.1412> gestures with his hand.
<yoice.25 2> says, "we're re gonna have the fish coming to the fish feed and then we re gonna have ducks coming too. It's gonna be a"
A child in blue outfit is visible in the background.
Semantic memory
Equivalence: <face.1412>, <voice.812>
Equivalence: <face_1412>, <voice_2>
Equivalence: <face 1413>, <voice
fface_1412 exhibits relaxed and informal demeanor.
=face_1412> displays interest in outdoor activities and family-oriented experiences.
<face.1413> demonstrates a nurturing and caring personality.
<voice 1> appears to be reflective and analytical when discussing personal experiences.
<voice.2> expresses enthusiasm and anticipation for future activities.
<voi ce 812> seems knowledgeable about pet care and enjoys engaging in outdoor
recreational activities.
<face 1412> and <face 1413> appear to have comfortable and supportive relationship, likely romantic partners or close family members.
<voice_1 and <face_1 412> share close bond, possibly parent and child, marked by open communication and shared experiences.
'The video depicts a slice-of-life scene centered around family interaction and leisure activities.
TTe narrative progresses from reflective discussion about sibling relationships to forward-looking anticipation of shared activities.
The overall tone lighthearted and positive, emphasizing the joys of family connection and shared experiences.
'The video showcases a blend of indoor and outdoor settings, creating a dynamic visual narrative.
The video likely helongs to the family vlog genre, documenting everyday life and experiences.
The video showcases typical suburban rural setting with a house, fence, and outdoor space.
Keeping ducklings requires specific pet care knowledge, including appropriate housing and feeding.
The video implies the cultural norm of family members engaging in shared recreational
activities.
Fishing is popular outdoor leisure activity often enjoyed in natural settings like ponds.
The video suggests the practice of using a fish feeder to attract and manage fish populations in pond.
Table 20:
Comparison memory generations between memory-7b-se -b and memozy-gemini-promee for case KHsInS>K2SU, clip 23:00-23:30, M3-Bench-web. Yellow highlights indicate additional important details provided by our model.
Video (lllustrated as 12 frames)
Faces
(Continued next page)
37

---

Published as a conference paper at ICLR 2026
<face_5>
<face_4>
Dialogues
<voice ce_53>: (00:00-00:03) Robot, can you least bring some fun games puzzles we can play? (00:13-00: 17) What am supposed to do? Sometimes they come with no pieces. What am supposed to do? (00:20-00:22) Well, what am supposed to? I'm bored. want to play game.
<voi ce 204>: (00:04-00:08) Really? Puzzle is wasting time for you. You don't like puzzles. (00:08-001) 13) But every time you play puzzle, you miss many pieces of it, which love. (00:23-00:27) don't know. Go out and play some games, some games outside. <voicee3116 6>; (00:18-00:20) So just leave it. Just don't take it.
memory seryyygmminiiirreoe
Episodic memory
<face_4> sits on a light-colored couch, facing < <face_5>, who sits at small table.
<face 55> wears a light pink sweatshirt, <face 4> wears a dark shirt and dark pants.
<voice.53 53> asks <face 5> to bring some fun games puzzles.
<face_5> tells <voice_53> that puzzles are a waste of time.
<fcce_5> explains that <voice_51 always loses puzzle pieces.
<voice.533 expresses frustration about missing puzzle pieces and asks what to do.
<face suggests ve.ice.53: just leave the puzzle incomplete.
<voice_53> complains of boredom and wanting to play game.
<face_53 suggests vvoice_53. go outside and play.
<voice_53> asks if <face_5x dislikes Legos too, because Lego pieces can also get lost.
Semantic memory
4Favice3/< appears bored and frustrated.
<face 5>/<voice 204> appears slightly annoyed dismissive.
place_4><vvoics 53> and <face 5>/<voi ie 204> seem to have a casual, possibly familial relationship.
The interaction revolves around <face.4>/<voice <53>'' boredom and
<face_5>/<voicc.. <e suggestions for alternative activities.
The setting appears be living room or family room.
memory-7b-stt (Ours)
Episodic memory
<face_4> sits beige couch, wearing black t-shirt and black pants.
<ace_5> sits next to <face_4> on the couch, wearing a pink sweatshirt.
A small round table with red and white checkered tablecloth is placed between < <face 4>
and <face_5>>
On the table are bottle red liquid, white cup, a notebook, and a pen.
vase with dried flowers and a decorative metal piece are on small table behind the couch. A projector and small green figurine are visible in the background.
<voice_53: asks the robot to bring games or puzzles.
<voice.204> tells <voice_53> that puzzles are waste of time and that <voice.53> doesn't 't like puzzles.
<voice-204> says that <voice.53> always misses pieces when doing puzzles, which
<voice 204> enjoys finding.
<voi ce. <5>>>>>..>>>........... expresses frustration, saying that sometimes puzzles with pieces.
<voi ce_204> suggests that <voi ce_53> just leave the puzzle they don't want to do it.
<voice_53> says they are bored and want to play game.
<voice 204> suggests that <voice ce 53> go outside and play games.
<voice_53 asks if <voice 1eoccc2200 204> doesn't 't like Legos either, because sometimes Lego
pieces are missing.
(Continued next page)
38

---

Published as a conference paper at ICLR 2026
<voice.204> responds with "I don't know".
<face -5> gestures with their hands while speaking.
<face_4> gestures with their hands while speaking. <face_5> writes in the notebook while speaking.
Semantic memory
Equivalence: <face_4>, <voice_53>
Equivalence: <face_4>, <voice_316>
Equivalence: <face 5>, <voi ce 204>
<voice appears to be easily bored and seeks immediate entertainment.
<voice_204> displays a pragmatic and solution-oriente approach to problem-solving <voice 204> seems to enjoy challenges and finding solutions, even seemingly mundane tasks like puzzles.
<voi ce_53> expresses preference for active recreation over quiet, puzzle-based activities.
<voice_200 and <voice.55 appear to have a comfortable, familiar dynamic, likely indicative of a close relationship.
'The interaction between avoice_2045 and <voice_53> suggests a playful banter, with <voice 204> teasing <voice 53> about their attitude.
<voi ce 204> takes on a mentoring or guiding role, offering suggestions and solutions to <voice_53>'s requests.
'The scene depicts a common household scenario differing preferences and mild conflict
regarding leisure activities.
The narrative unfolds as a request for entertainment, followed by disagreement on suitable options and a suggestion for more intellectually engaging activity.
The overall tone is casual and familiar, reflecting the dynamics of a comfortable, possibly familial romantic, relationship.
The scene showcases power dynamic where one individual attempts to influence the other's choices, leading to mild power negotiation.
The video likely takes place in private residence, possibly apartment recently built home, given the modern interior design.
The presence of a projector suggests a focus on entertainment and media consumption within the household.
The availability of puzzles and Lego sets implies focus on recreational activities and potentially family -riented entertainment.
'Te red liquid in the bottle is likely a type of soda or juice, commonly consumed as refreshing beverage during leisure time.
The checkered tablecloth on the table signifies a casual dining or activity setup, possibly for a snack or a casual game session.
'The dried flowers and decorative metal piece function as home decor, contributing to the overall aesthetic and ambiance of the space.
The small green figurine may be collectible item decorative ornament, adding personal touch to the room's decor.
Table 21:
Comparison of memory generations between remory-7b-sft and room 22, clip 4:00-4:30, M3-Bench-robot. Yellow
cannorr-geiiiipreeee for living tiving_room highlights indicate additional important details provided by our model.
Question: Is Tomasz person with rich imagination someone who lacks imagination? Answer: A person with rich imagination. (syatem_prompt)
system
(Continued on next page)
39

---

Published as a conference paper at ICLR 2026
You are given a question and some relevant knowledge. Your task is to reason about whether the provided knowledge is sufficient to answer the question. If it is sufficient, output [Answer] followed by the answer. If it is not sufficient, output [Search] and generate a query that will be encoded into embeddings for vector similarity search. The query will help retrieve additional information from a memory bank.
Question: What is Theresa Yeap's attitude like in her explanation of emergency services?
user
(inst ruct ion prompt)
Searched knowledge: (
Output the answer in the format:
Action: [Answer] [Search]
Content: {content} If the answer cannot be derived yet, the {content) should be single search query that would help retrieve the missing information. The search {content) needs to be different from the previous.
You can get the mapping relationship between character ID and name by using search query such as: "What is the name of <character- After obtaining the mapping, it is best to use character ID instead of name for searching. If the answer can be derived from the provided knowledge. the {content} is the specific answer to the question. Only name can appear in the answer, not character ID like <character (i}>.
"What is the character id of (name)".
assistant
<think> Okay, let's see. The user is asking whether Tomasz is a person with rich imagination someone who lacks imagination. The problem is that there's 's no provided knowledge about Tomasz in the given context. The Searched knowledge is empty, so I don't have any information base answer on.
First, need to check if the existing knowledge can answer the question. Since the knowledge is empty, there's nothing to reference. The user mentioned that if the answer can't 't be derived, I should output search query. The instructions also say to use character IDs if possible, but since we don't 't know Tomasz's s ID yet, need to find that first.
So the first step is to find the character ID for Tomasz. The user suggested using a query like "What the character id of Tomasz" to get that mapping. Once I have the ID, I can then search for information related to his imagination. But without the ID, any search might not be effective. Therefore, the initial search should be to find the character ID. Then, using that ID, look into his traits. Since the current knowledge is empty, the correct action is to search for the character ID first.</think'
Action: [Search]
Content: What is the character id of Tomasz
user
Searched knowledge: ("CLIP_4"; [ chaaaaaateer_43 name is Tomasz Patan.", "The man in the white shirt, now identified as Tomasz Patan, Founder & Chief Technology Officer of Jetson, speaks to the camera in a workshop setting."), "CLIP_5", [ ["<character_4>> 's name is Patán.", " "<chhracter 2> identifies <character 4> Patán, the founder and CTO of Jetson, company developing an eVTOL vehicle."), "CLIP 8t:: ("<chhrrcter 2>''s name is Angus Whitley."), "CIIP_15" ["<character__0 0>> name is Akshat Rathi.". "<character _O>> wears glasses and has beard."], "C",__CLIPPI""" [""_P__"'"""<d""""""""" ["<<chrraterr55> 's name is Matthew Clark."), "CLIP_20 -2""
["<character 3>'s name is Tom Muniz.", "<character 3> identifies himself as Tom Muniz, Chief Technology Officer of Archer Aviation.") "], "CLIP_211 flight controller, given their interaction with air traffic control.") "J. "CLIP 26"": ["<acharat 2araacterr_>> name is Robin Riedel.", "<character 2>, wearing glasses and light blue shirt. is identified as Robin Riedel, a partner at McKinsey & Company and co-leader of the McKinsey Center for Future Mobility.", <<character_2>>> is certified commercial pilot."), "CLIP_31". ["The graphic then shows a user profile icon appearing. representing 'pilot requirementt "]. "CLIP_36": ["<chhracter 2> is shown again in the library."J. "CLIP 37": ['<ccharatter 1> displays cautious and rule-abiding personality."), "CLIP.40" ["<character. 12>'s 's name is Isabel Metz.", "<character_12> identifies the Hamburg airport on the map."), "CLIP _2" [*A name tag identifies <character 8> as 'Bianca Schuchardt, Researcher, German Aerospace Center'.']]
[*<<chrratter_] 77> is likely pilot or
(Continued on next page)
40

---

Published as a conference paper at ICLR 2026
instruction.prompt
assistant
<think> Okay, let's see. The user wants to know if Tomasz has a rich imagination or lacks From the previous search, we found that Tomasz Patan is <character_4>, Now, I need to check the provided knowledge see there's ' any info about his imagination.
Looking through the CLIP entries for <character_44 -4 (Tomasz Patan), the info given is about his role Founder CTO of Jetson, speaking in a workshop. There's 's no mention of his imagination. Other clips mention other characters but not their traits related to imagination. Since there's no data here about Tomasz's imagination, can't derive an answer. The next step to search for information on his imagination using his character ID. The query should be about his imagination, so maybe "Does jcharacter 4> have rich imagination?" or similar. Need to make sure the query is different from previous ones. Let's go with "What is <character personality regarding imagination?" to retrieve relevant </hink/
Action: [Search]
Content: What is <character_43 -4> user
personality regarding imagination?
Searched knowledge: {
(The search result is empty. Please try searching from another perspective.) instructi ir.uuciioprooppp... assistant
<think> Okay, let's see. The user asking Tomasz is person with rich imagination or someone who lacks imagination. So far, we've tried searching and couldn't 'it't find any info about his imagination. The first step was to get his character ID, which is <character_4+ Then, we tried searching for his personality regarding imagination, but that came back empty.
Since there's direct information about Tomasz's imagination in the provided knowledge, can't derive an answer from the existing data. The user's instructions say that if the answer can' t be derived yet, should generate new search query. The previous query was about his personality regarding imagination, which didn't 't yield results. So I need to approach it from another angle.
Maybe instead of asking about his personality, can look for any mentions of his traits in general. Perhaps there's a clip where his characteristics are discussed, but it's not related to imagination. Alternatively, maybe his role as CTO or founder of a company could imply something about his creativity or innovation, which are related to imagination. However, the knowledge provided doesn't 't mention anything about his traits beyond his professional role. Given that the knowledge is still empty, should create a new query that's different from the previous one. Perhaps asking about his creative abilities how others perceive his problem-solving skills. Alternatively, checking for any quotes or statements he made that might indicate his imaginative nature.
The key is to generate query that's 's specific enough to retrieve information if it exists. So maybe "What are <character_4: creative problem-solving methods?" or "How does <character 4> approach innovation in his work?" These queries might uncover information about his imagination through his work style or achievements.
Since the user wants the search query to be different from the previous ones, I'll go with one of these alternatives. Let's pick "What are <character 44>s creative problem-solving methods?" to see if that yields any relevant information about his imagination.</thin Action: [Search]
Content: What are <character 4> 's creative problem-solving methods?
user
(Continued on next page)
41

---

Published as a conference paper at ICLR 2026
Searched knowledge: ("CLIP_6", ["<character_43. wearing a white polo shirt with the 'Jetson' logo, stands in room with a partially assembled aircraft frame behind him.", "<character.45 holds a clear plastic cup and gestures with his hands as he speaks.". "<character 4> says, 'test and we have lots of data from it that had redundant sensors on the vehicle that day." ", "A person wearing a yellow vest labeled 'TEST TEAM' adjusts a red propeller on the aircraft outdoors.", "Another person wearing a black jacket and white and orange sneakers steps onto scale.", ", "The person in the black jacket, now wearing Sparco racing suit and helmet, walks towards an open field.", " "<character_4> says, 'So for 11 years, I've been doing drones with camera gimbals.' "
. "<character 4> continues, 'These drones were getting bigger and bigger.' "<character 4> says, 'And at some point, asked myself the question, what if we scaled it?' <<character_4>> puts on a Sparco helmet and fastens the chin strap.". "<character_4> 44> adjusts the visor on his helmet.", "<character 44> buckles the seatbelt inside the aircraft.".
"<character_44> says, 'So I could sit in there and pilot myself.' " "<character_4>> sits in the aircraft as it takes off, raising one hand in a gesture.", "The aircraft is white and black with four propellers.", "<character_43 - is an engineer or designer involved in the development of the Jetson aircraft.", ", " "<character 4> has a background in working with drones and camera gimbals, demonstrating expertise in aerial technology.", "<character 4> is innovative and forward-thinking, as evidenced by his interest in scaling drone technology for personal flight.", < character. 4. possesses a hands-on, experimental approach to his work, as shown by his involvement in testing and data analysis.", ", "<character_ 4> displays sense of pride and enthusiasm for his work on the Jetson project.", "The video follows a narrative are of innovation and realization, starting
with the concept of scaling drone technology and culminating in a successful test flight.", "The video showcases& blend of technical detail and personal narrative, offering insights into both the engineering process and the driving force behind the project.", "The overall tone of the video optimistic and forward- worwadd-looking emphasizing the potential of personal aerial vehicles.", "The video implicitly highlights the importance of data analysis and redundant sensor systems in ensuring flight safety and reliability.", "The video likely belongs to the genre of technology demonstration or promotional content, showcasing the Jetson aircraftt", "The setting includes both indoor and outdoor locations, suggesting a combination of design, testing, and operational phases of the project.", "The use of safety gear like helmets and vests indicates adherence safety protocols in aviation and engineering environments.", "The presence of a test team suggests structured approach to quality control and validation of the aircraft's performance.", "The scale model aircraft implies a phased approach to development, starting with smaller prototypes before moving to full-sized vehicles.", "The video showcases procedural norms in aviation, such as pre- flight checks, helmet usage, and controlled testing environments.", "The Jetson aircraft represents a novel application of drone technology, aiming to bridge the gap between unmanned aerial vehicles and personal flight.". "The video implies a growing trend of personal aerial mobility, driven by technological advancements and innovative design.", "The video's 's narrative structure follows common format in technology demonstrations, starting with the problem statement (drones' size limitations) and presenting a solution (the Jetson aircraft) "The video's focus on data analysis suggests data-driven approach to design and optimization in the development of the Jetson aircraft.")) wnattruutionppoooo.o
assistant
<think>> Okay, let's 's tackle this question. The user wants to know if Tomasz is person with rich imagination or someone who lacks imagination. From the provided knowledge, we have bunch of clips about Tomasz Patan (character 4). Let me go through the info again. In CLIP_6, there's 's a lot of detail about his work with drones and the Jetson aircraft. He mentions scaling drone technology for personal flight, which shows he's 's thinking beyond the current applications. The notes say he's innovative and forward- -tinkknn,, with a hands-on approach. He's involved in testing and data analysis, and he's 'ss enthusiastic about his work. The video's tone is optimistic, highlighting his problem-solving methods and the development process.
(Continued on next page)
42

---

Published as a conference paper at ICLR 2026
The key points here are his background in drones, his idea to scale them into personal aireraft, and his active role in testing. The description explicitly states he's innovative and forward-thing thinkiing.. These traits are closely related to having a rich imagination. Since he's taking existing technology (drones) and envisioning new application (personal flight), that requires creativity and imagination. Also, the narrative of the video presents him as someone who asks "what if" questions, which hallmark of imaginative thinking.
I don't t see any information suggesting he lacks imagination. All the provided details support the idea that he's he's imaginative. Therefore, the answer should be that Tomasz person with rich </think>>
Action: [Answer]
Content: Tomasz is person with rich imagination.
Table 22: A complete generation trajectory in the control process.
M PROMPT TEMPLATES
M.1 PROMPT FOR AUTOMATIC EVALUATOR OF M3-BENCH
Table 23 presents the prompt used by GPT-4o assess M3-Bende
The prompt for GPT-4o evaluation
You provided with question, ground truth answer, and an answer from an agent model. Your task to determine whether the ground truth answer can be logically inferred from the agent's answer, in the context of the question.
Do directly compare the surface forms of the agent Instead, assess whether the meaning expressed by the agent answer supports or implies the ground truth answer. If the ground truth can be reasonably derived from the agent answer, return "Yes". If it cannot, return "No".
and the ground truth answer.
Important notes:
Do not require exact wording or matching structure.
Semantic inference is sufficient, long the agent answer entails or implies the meaning of the ground truth answer, given the question. Only return "Yes" or "No", with additional explanation formatting.
Input fields:
question: the question asked
ground_truth_amm answer: the correct answer
agent answer: the model's answer to be evaluated
Now evaluate the following input:
Input:
question: {question]
ground_truth_ansee {ground_truthe ttutt agent answer: {agent.ans answer]
Output ("Yes' or 'No'):
Table 23: Prompt used by GPT-4o to evaluate M3-Bench.
M.2 PROMPTS FOR SOCRATIC MODELS
Table 24 presents the prompt used in Socratic Models baselines. Through prompt engineering, we find that placing the question after the long context (e.g., video detailed descriptions) enhances the
43

---

Published as a conference paper at ICLR 2026
model's ability to retain the question and focus relevant information, leading improved answer accuracy. Accordingly, in our Socratic Models experiments, we adopt this approach by appending the question to the end of the retrieved clip descriptions during the RAG-based QA stage.
Caption Generation Prompt (Gemini-1.5-Pry) Qwen-2.5-0mni
You are an advanced video description generator tasked with providing a detailed, cohesive description of video clip.
Follow these high-level principles to ensure your output is accurate and meaningful:
1. Focus Observable Content.
2. Provide Context for the Environment and Timing.
Incorporate Audio Dialogue Information.
You are provided with a current video clip. (GPT-40, Qwen2.5-VL-7b Variant: You are provided with 15 key frames from a current video clip and audio text information ja list where each item represents speech segment diet with the following fields: start time, end time, asr. The time information the time the current clip and not the global timez-)
Your Task:
Based on the video clip, generate a detailed and cohesive description of the video clip. The description should focus the entire event, incorporating all relevant aspects of the characters, their actions, spoken dialogue, and interactions in a narrative format. The description should include (but is not limited to) the following categories:
Describe the characters' appearance, including their clothing, facial features, body language, or any distinguishing characteristics that are noticcable in the frames. Characters' Actions Movements: Describe specific gestures, movements, or interactions performed by the characters. Include both major and minor actions that contribute to the overall scene, emphasizing any transitions between different actions.
Characters' Appearance:
3. Characters' Spoken Dialogue: Use the provided audio dialogue information to accurately transcribe or summarize the dialogue spoken by the characters. Include emotional tone, volume, context if relevant (e.g., shouting, whispering, laughing). Characters' Contextual Behavior and Attributes: Describe the characters' roles in the scene, their emotional states, motivations, relationships with other characters. Highlight any conflict, bonding, or change in dynamics.
Environmental Context: Include relevant details about the environment where the scene takes place. Describe the physical location, setting, lighting, or any other environmental factors that affect the atmosphere or context of the video clip.
6. Temporal Context: Provide information about the timing of events within the scene. Describe the natural progression of time (e.g., morning, afternoon, evening) or any time-sensitive elements that contribute to the unfolding of the events.
Strict Requirements:
Do not generic descriptions, inferred names, pronouns to refer to characters (e.g., "he," "they," "the man").
The generated descriptions of the video clip should include every detail observable in the frames and mentioned in the audio dialogues. (GPT-40, Qwen2.5-VL-77 Variant: The generated descriptions of the video clip should include every detail observable in the frames and mentioned in the audio dialogues.)
Pay close attention any introduction of characters' names, titles, or other identifiers provided in the frames or audio.
Whenever possible, include natural time expressions and physical location cues in the descriptions to improve contextual understanding. These should be based on inferred situational context (e.g.. "in the evening at the dinner table," "early morning outside the building").
* Include relevant background, common knowledge and environmental factors when needed (e.g., location, weather, setting) to provide a fuller understanding of the context.
(Continued on next page)
44

---

Published as a conference paper at ICLR 2026
Maintain natural, narrative flow in the description, ensuring that reads like a coherent summary of the events in the video.
Remember you are looking at key frames and audio dialogue information, not the full video, so focus what be observed from these specific materials. (GPT-40, Qwen2.5-VL-7b Variant: Remember you are looking at key frames and audio dialogue information, not the full video, focus on what be observed from these specific materials))
Example Output:
"As Margaret returns with the teapot, Tom stands up to help her pour the tea, gesturing politely as she hands him cup. Margaret sits back down. Margaret leans forward slightly, her hands resting on the table, and after moment of silence, she speaks again, her voice steady but filled with hint of urgency. Tom listens closely, his brow furrowing slightly as he takes in her words. He responds quietly, nodding slowly as he processes the information."
RAG Answer Prompt (GPT-40)
Based on the following video description, answer the question concisely as possible. Provide only the direct answer without explanations or reasoning.
Question: {question]
Relevant Video Clip Captions: (retrived.clips)
Answer:
Table 24: The prompts for the experiments of the Socratic Models. For models that take either raw video (gemini-1.5-pro, Qwen2.5-0mmi-77) input or video frames with ASR transcripts (GPT4o, (ren2.5-VL-7bh) the description generation prompt has minor differences, which are indicated in italicized parentheses.
M.3 PROMPTS FOR M3-AGENT
Table 25 shows the prompt used by Gemini-Agent and Gemini-GPP4o-HHyyiii during memorization.
Table 26 shows the prompt used by Gemini-Agent and Gemini-GPP44--Hybrri during control.
Table 27 shows the prompt used by M3-Agent during the control process. The system prompt at the beginning of each session specifies the overall task objectives. The instruction prompt appended at the start of each round provides the question and detailed guidance. The last-round prompt, used only in the final round, signals the agent that the final opportunity to respond.
Memorization Prompt memeory--emiiii-rrooeee
memory-7b-prompe
You are given video along with set of character features. Each feature is either: Face: a single video frame with bounding box, or
Voice: one or more speech segments, each containing start_time (MM:SS), end_time (MM:SS) and asr (transcript).
Every feature has unique ID enclosed in angle brackets (e.g. <face 1>, <voice 2>).
Your Tasks (produce both in the same response)
Episodic Memory (the ordered list of atomic captions)
Using the provided feature IDs, generate a detailed and cohesive description of the current video clip. The description should capture the complete set of observable and inferable events in the clip. Your output should incorporate the following categories (but is not limited to them): (a) Characters' Appearance: Describe the characters' appearance, such as their clothing. facial features, or any distinguishing characteristics.
(Continued next page)
45

---

Published as a conference paper at ICLR 2026
(b) Characters' Actions & Movements: Describe specific gesture, movement, or interaction performed by the characters.
(c) Characters' Spoken Dialogue: Quote-or, or, if necessary, summarize what are spoken by the characters.
Describe the characters' roles in the scene or their interaction with other characters, focusing on their behavior, emotional state, relationships.
(d) Characters' Contextual Behavior:
Semantic Memory (the ordered list of high-level thinking conclusions)
Produce concise, high-level reasoning-bua -bsed conclusions across five categories: (a)
Equivalence Identification Identify which face and voice features refer the same character. Use the exact format: Equivalence: <face_x>> <voice.y>. Include many confident matches as possible.
(b) Character-level Attributes Infer abstract attributes for each character, such as: Name (if explicitly stated), Personality (e.g.. confident, nervous), Role/profession (e.g., host, newcomer), Interests background (when inferable), istinctive behaviors traits (e.g.. speaks formally, fidgets). Avoid restating visual
on identity construction.
(c) Interpersonal Relationships & Dynamics- Describe the relationships and interactions between characters: Roles (e.g.. host-guest, leader-suboordinate) Emotions or tone (e.g., respect, tension), Power dynamics (e.g., who leads), Evidence of cooperation, exclusion, conflict, etc. (d) Video-level Plot Understanding Summarize the scene-level narrative, such as: Main event or theme, Narrative are or sequence (e.g., intro discussion reaction), Overall tone (e.g., formal, tense), Cause-effect group dynamics..
(e) Contextual & General Knowledge Include general knowledge that can be learned from the video, such as: Likely setting or genre (e.g., corporate meeting, game show), Culuural/procedurall norms, Real-world knowledge (e.g.. "Alice market is pet-friendly"). Common format conventions.
Strict Requirements (apply to both sections unless noted)
1. If character has provided feature ID, refer that character only with the ID (e.g.
face_l>, <voice_>>>).
2. If ID exists, use a short descriptive phrase (e.g. "a man blue shirt").
3. Do not use "he," "she." " "they." pronouns, or invented Names.
4. Keep face/voice IDs consistent throughout,
5. Describe only what is grounded in the video or obviously inferable. Include natural Time & Location cues and setting hints when inferable.
7. Each Episodic Memory line must express one event/detail; split sentences if needed.
Output English only.
9. Output Python list of sentences for each memory type.
Additional Rules for Episodic Memory
1. Do not mix unrelated aspects in one memory sentence.
2. Focus on appearance, actions/movveeents, spoken dialogue (quote or summary), contextual behavior.
Additional Rules for Semantic Memory
1. For Equivalence lines, use the exact format: Equivalence: <face_x>, <voice_y>.
2. Do not repeat simple surface observations already in the captions.
3. Provide only final conclusions, not reasoning steps.
Expected Output Format
Return the result as a single Python diet containing exactly two keys:
{
(Continued on next page)
46

---

Published as a conference paper at ICLR 2026
"In the bright conference room, <face_ < enters confidently, giving professional
appearance as he approaches <face 2> to shake hands.".
"<face_ 1> wears black suit with white shirt and tie. He has short black hair and wears glasses.".
"<face_2>>, dressed in striking red dress with long brown hair.",
"<face_2s smiles warmly and greets <face__>.. She then sits down at the table beside him, glancing at her phone briefly while occasionally looking up.".
voice >> speaks to the group. 'Good afternoon, everyone. Let's begin the meeting." His voice commands attention as the room quiets, and all eyes turn to him.",
"<face_2>> listens attentively to <voice_ 1> 's words, nodding in agreement while still occasionally checking her phone. The atmosphere is professional, with the participants settling into their roles for the meeting.".
"<face_ > adjusts his tie and begins discussing the agenda, engaging the participants in a
productive conversation."
l.
"semantic memory":
"Fquivalence: <face_1>, <voice_I>",
" "<face_I>'s name is David.",
"<face holds position of authority, likely as the meeting's organizer or senior executive.".
"<face 2> shows social awareness and diplomacy, possibly indicating experience in public or client-facing roles.".
"<face_>>> demonstrates control and composure, suggesting high level of professionalism and confidence under pressure.",
"The interaction between <face_1> and <face suggests working relationship built on mutual respect.".
"The overall tone of the meeting is structured and goal-oriented, indicating it is part of larger organizational workflow."
Please only return the valid python dict (which starts with "{" and ends with "}") containing two string lists in "episodic memory" and "semantic memory", without any additional explanation or formatting.
Table 25: Memorization prompt for nemozy-gemini-Pe and memory-7b-prompe
Control Prompt
You given question and some relevant knowledge about specific video. You also provided with a retrieval plan, which outlines the types of information that should be retrieved from memory bank in order answer the question. Your task is to reason about whether the provided knowledge is sufficient to answer the question. If it is sufficient, output [ANSWER] sufficient, output [SEARCH] and generate query that will
followed by the answer. If it be encoded into embeddings for vector similarity search. The query will help retrieve additional information from a memory bank that contains detailed descriptions and high-level abstractions of the video, considering the question, the provided knowledge, and the retrieval plan.
Your response should contain two parts:
1. Reasoning
Analyze the question, the knowledge, and the retrieval plan.
If the current information is sufficient, explain why and what conclusions you draw. not, clearly identify what missing and why it is important.
Answer or Search
(Continued on next page)
47

---

Published as a conference paper at ICLR 2026
If the answer can be derived from the provided knowledge, output [ANSWER] followed by short, clear, and direct answer.
[ANSWER]:
When referring to a character, always use their specific name available.
Do not use ID tags like <character_ {1}> or <face_(1) {1)>>
[SEARCH]: If the answer cannot be derived yet, output [SEARCH] followed by a single
search query that would help retrieve the missing information.
Instructions for [SEARCH] queries:
Use the retrieval plan to inform what type of content should be searched for next. These contents should cover aspects that provide useful context or background to the question, such as character names, behaviors, relationships, personality traits, actions, and key events.
Use keyword-based queries, not command sentences. Queries should be written as compact keyword phrases, not as full sentences or instructions. Avoid using directive language like "Retrieve", "Describe", or question forms such as "What", "When", "How".
Keep each query short and focused information, without combining multiple ideas or aspects.
point. Each query should target one specific type of
Avoid over-complexity and unnecessary detail. Do not include many qualifiers conditions. Strip down to the most essential keywords needed to retrieve valuable content.
The query should target information outside of the existing knowledge that might help answer the question.
For time-sensitive chronological information (e.g.. events occurring in sequence, changes over time, specific moments in a timeline), you can generate clip-based queries that reference specific clips or moments in time. These queries should include a reference to the clip number, indicating the index of the clip in the video (a number from to N, where smaller number indicates an earlier clip). Format these queries as "CI.I_x", where x should an integer that indicates the clip index. Note only generate clip-based queries if the question is about specific moment in time or a sequence of events.
You also generate queries that focus specific characters characters' attributes using the shown in the knowledge.
Make sure your generated query focus on some aspects that are retrieved or asked yet. Do not repeatedly generate queries that have high semantic similarity with those generated before.
Example 1:
Input: Question: How did the argument between Alice and Bob influence their relationship in the story?
Knowledge:
{{
"query": "What happened during the argument between Alice and Bob?",
"related memories": {{
"CLIP_2". [
fface_1 >> and <face_2>>> are seen arguing in the living room."
face raises her voice, and <face 2> looks upset."
" face 1> accuses <face.22 2> of not listening to her."
}}
}}
]
Output:
seems that <face. 1>> and <face 2> are arguing about their relationship. need to figure out the names of <face_ _>> and <face_2>> [SEARCH] What are the names of <face_1> and <face 2>>
Example 2:
(Continued on next page)
48

---

Published as a conference paper at ICLR 2026
Input:
How did the argument between Alice and Bob influence their relationship in the story?
Question:
Knowledge:
"query": "What happened during the argument between Alice and Bob?", "related memories": {{
"CLIP 2":
"<face 1> and <face 2> are seen arguing the living room."
"<ace__ 1>> raises her voice, and <face_2> looks upset."
"<face_ accuses <face.2> of not listening to her."
1.
}}
H1.
"query": "What are the names of <face 1> and <face 2>?"" "related memories": {{
"CLIP_"" [
"<acce_ 1> says to <face 2>: 'I am done with you Bob!"m
" "<acee 2> says <face_ 1>: 'What about now, Alice?"'M
}
}}
]
Output:
It seems that content in CLIP 2 shows exactly the argument between Alice and Bob. To figure out how did the argument between Alice and Bob influence their relationship, need to see what happened next in CLIP.3.
(SEARCH) What happened in CLIP_3?
Now, generate your response for the following input:
Question: (question)
Knowledge: {search
Output:
Control Prompt (last round)
You are given question about specific video and a dictionary of some related information about the video. Each key in the dictionary is clip ID (an integer), representing the index of video clip. The corresponding value is list of video descriptions from that clip.
Your task to analyze the provided information, reason over it, and produce the most reasonable and well-supported answer to the question.
Output Requirements:
Your response must begin with a brief reasoning process that explains how you arrive at the
answer.
Then, output [ANSWER] followed by your final answer.
The format must be: Here the reasoning... [ANSWER] Your final answer here.
Your final answer must be definite and specific even if the information is partial or ambiguous, you must infer and provide the most reasonable answer based on the given evidence.
Do not refuse to answer say that the answer is unknowable. Use reasoning reach the best possible conclusion.
(Continued on next page)
49

---

Published as a conference paper at ICLR 2026
Additional Guidelines:
When referring to a character, always use their specific name it appears in the video information.
Do not use placeholder tags like <character_Is 1> or <face_1>>
Avoid summarizing repeating the video information. Focus reasoning and answering.
The final answer should be short, clear, and directly address the question.
Input:
Question: fquestion ion]
Video Information: (search_results]
Output:
Table 26: Control prompt for Gemini-Agent and Gemini-GPTT4o-Hybridd
system.ptys prompt
You given question and some relevant knowledge. Your task is to reason about whether the provided knowledge sufficient to answer the question. If is sufficient, output [Answer] followed by the answer. If is not sufficient, output [Search] and generate query that will be encoded into embeddings for vector similarity search. The query will help retrieve additional information from a memory bank.
Question:
instraction.prompe
Output the answer in the format:
Action: [Answer] [Search]
Content: {content}
If the answer cannot be derived yet. the {content} should be single search query that would help retrieve the missing information. The search {content) needs be different from the previous.
You can get the mapping relationship between character ID and name by using search query such as: "What is the name of <character. {i] "What is the character id of (name)". After obtaining the mapping, is best to use character ID instead name If the answer can be derived from the provided knowledge. the {content] is the specific answer to the question. Only name can appear in the answer, not character ID like <character.(i)>>
searching.
last_round
The Action of this round must be [Answer]. If there is insufficient information, you can make reasonable guesses.
Table 27: The prompts used by M3-Agem during the control process.
50


Reviews:
[{"review_id": "DLJkUtbjlI", "reviewer": "Reviewer_2Awb", "summary": "This paper introduces an agent (M3-agent) to process streams of video and audio input. In addition, the paper introduces a dataset and bechmark to evaluate long-horizon video understanding (M3-Bench).\n\nThe agent contains two sub-agents; one to build a semantic and episodic memory from long video streams, and another to query the streams to answer questions. The memory forming policy is trained behavior cloning/SFT, and the control policy is trained using RL (GRPO or DAPO).\n\nThe results across M3-Bench and Video-MME-Long are compelling; outperforming (1) modular systems that use proprietary models from Google and OpenAI with RAG, (2) off-the-shelf video understanding models, (3) Socratic models\n\nThe benchmark contains two components: one scraped from youtube and another of ecocentric videos collected by humans. The benchmark is designed so that the questions require information from video; not only audio. Many of the details about collection are provided in the paper and supplementary.", "questions": "* Is the memory agent able to merge or edit memories? \n* Is the agent able to call tools on its own, or is all tool use scripted beforehand?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper provides both a practical recipe for a long-term memory, as well as a dataset and benchmark. In my opinion, it is an excellent contribution. \n\nThe paper is well-written and easy to follow; despite there being quite a lot of work being described.\n\nThe memory agent is interesting and flexible;  involves tool use -- e.g. to do facial recognition and associate memories with the speaker. \n\nThe proposed agent serves as an excellent baseline on the proposed benchmark. The fact that DAPO on the control policy improves over modular baselines helps demonstrate a potential path for improvement.", "weaknesses": "Overall I don't see major weaknesses; some minor ones about the memory (which I see as a baseline on the new benchmark).\n\n* The control policy is trained using RL, but memory uses only SFT. Though it may be tricky to implement from an engineering perspective, RL seems especially useful also for the memory policy, too.\n* Due to the method of constructing of entities in the memory dictionary, it seems speaker faces must be visible", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jSC9BdguvH", "reviewer": "Reviewer_urmt", "summary": "This paper proposes M3-Agent, a multimodal agent architecture with long-term, entity-centric memory. The system runs two concurrent loops: (i) memorization that parses streaming video/audio into episodic and semantic memories; and (ii) control, which performs multi-turn retrieval and reasoning over this memory to answer questions. To evaluate memory-based reasoning beyond low-level perception, this paper introduces M3-Bench, a long-video QA benchmark with 100 robot-egocentric videos and 920 web videos, annotated with question types targeting multi-evidence, multi-hop, cross-modal, and general knowledge extraction. Across M3-Bench and VideoMME-long benchmark, M3-Agent outperforms baselines.", "questions": "1. Beyond the 100-sample spot-check, did you run a larger human evaluation or cross-evaluator check (e.g., Claude/LLama-judge) to ensure conclusions are not tied to GPT-4o’s preferences?\n\n2. How does index size, retrieval time, and accuracy scale as video hours and the number of identities grow?\n\n3. The ablations show big drops without semantic memory and identity equivalence. Could you further decompose semantic memory into subtypes (attributes vs. relations vs. rules) to identify which contributes most by question type?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. M3-Bench is a novel evaluation benchmark designed to assess memory-based reasoning beyond low-level perception. This dataset comprises 100 robot-egocentric videos and 920 web videos, offering a good option for agent evaluation. The data processing and evaluation reflect the author's significant efforts.\n\n2. This paper provides a detailed account of the development process and implementation specifics for constructing M3-bench and M3-agent, supported by comprehensive experiments. Furthermore, the paper features a well-structured layout, good presentation, and is easy to follow.", "weaknesses": "1. Automatic grading uses GPT-4o to judge correctness; although this paper report 96% agreement with human majority on a small set, relying on a single proprietary LLM as an oracle can introduce bias and inflate or depress certain methods’ scores.\n\n2. GPT-4o was employed for data generation, evaluation, and baseline comparison. On one hand, this diminishes the paper's contribution, making it more akin to prompt engineering using GPT-4o. On the other hand, constructing data and conducting evaluations with the same model introduces preference bias.\n\n3. The proposed M3 agent lacks technological innovation. From data processing to model training (DAPO), it relies entirely on existing methods, while its memory construction primarily depends on MLLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hbEqhLsh1G", "reviewer": "Reviewer_BEzR", "summary": "This paper presents M3-Agent, a multimodal agent framework that integrates long-term memory with reinforcement fine-tuning. The system processes continuous visual and auditory inputs to form episodic and semantic memories, which are stored in an entity-centric multimodal graph.\n\nTo evaluate the effectiveness of memory effectiveness and memory-based reasoning in multimodal agents, the authors also build M3-Bench, a new benchmark for long-video question answering. It consists of two parts: M3-Bench-robot, containing 100 egocentric videos recorded from a robot’s viewpoint, and M3-Bench-web, including 920 web videos across diverse scenarios. Experiments show that M3-Agent surpasses baselines including GPT-4o and Gemini-1.5-pro–based prompting agents on M3-Bench and VideoMME-long.", "questions": "1. What if one doesn’t have access to in-house data? Is your training data sourced from the same distribution as your benchmark? If so, could this create an unfair advantage for your method and negatively impact fair comparison in future work?\n2. During RL training, you use GPT-4o for scoring. Does this reliance on API calls significantly increase training time or introduce practical bottlenecks?\n3. What is the typical length of the input history? When retrieving memories, are both images and text included directly as inputs to the model? Is there any further compression or processing applied to the retrieved multimodal content?\n4. The original Video-MME Long paper [1] reports a Gemini-1.5-Pro result of 73.6, which is notably higher than the numbers reported in your paper. Additionally, the baselines you compare against appear outdated. Have you evaluated your method against more recent state-of-the-art models?\n\n[1] Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposed a challenging and realistic video understanding benchmark M3-Bench that goes beyond existing works. Instead of relying solely on visual perception,  it further requires the agent to leverage higher-level cognitive abilities and integrate world knowledge to solve complex tasks that involve long-horizon dependencies, temporal reasoning, and cross-modal understanding. Moreover, M3-Bench fills an important gap in evaluating memory-based long-term multimodal reasoning, especially in scenarios with robot-perspective data.\n\n2. This paper proposed a complete and well-engineered system M3-Agent that integrates video/audio perception, memory construction, and multi-step retrieval control. The overall framework is coherent and clear: episodic memory captures fine-grained observations, semantic memory provides additional cues for retrieval, and the entity-centric memory graph maintains information consistency across long contexts.", "weaknesses": "1. Lack of real methodological novelty. The overall pipeline resembles VideoAgent [1]: the agent first perceives multimodal inputs, then stores textualized or embedding-based observations into an external memory, performs RAG during query time, and finally conducts multi-step reasoning over the retrieved memory to produce answers. The entity graph is not novel, similar identity linking appears in StoryTeller [2]. The RL training also follows a conventional approach, similar to Search-R1 [3], making the proposed framework appear more like an incremental engineering integration rather than a fundamentally novel paradigm.\n\n2. Memory quality is central to paper claims yet barely evaluated. The entity-based memory graph is central to the contribution, but its behavior over long horizons is not analyzed. There is no evaluation of memory corruption, contradiction resolution, or scalability as memory grows. Conflict handling is briefly mentioned via weighted voting (Sec. 4.1) but is under-specified and not measured experimentally.\n\n3. The claim of being an \"online agent\" is unsubstantiated. Despite repeatedly claiming “online” and “streaming” capability, the paper does not report any latency, throughput, or memory scaling measurement, nor demonstrates real-time processing. The entire pipeline is evaluated offline on fixed-length videos, and no evidence is provided that M3-Agent is suitable for online deployment. \n\n4. Insufficient Evaluation of “World Knowledge Construction”. The paper rightly points out that prior methods focus too much on low-level visual details and neglect high-level world knowledge (e.g., character identity, attributes). However, the evaluation on M3-Bench (with five QA types) does not sufficiently probe the system’s ability to generalize and transfer knowledge. For instance, if the agent observes in multiple videos that “green bins are for recycling,” can it apply this rule to a new household scene?\n\n5. Failure Case Analysis Could Be More In-Depth. Appendix J includes a case study showing successes and “hard cases” but lacks quantitative error categorization. While the paper identifies types of failures (e.g., spatial reasoning), it doesn’t quantify how prevalent each error type is. For instance, what percentage of errors on M3-Bench-robot stem from spatial misunderstandings?  Without this, it’s hard to gauge the main bottlenecks.\n\n[1] VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding\n\n[2] StoryTeller: Improving Long Video Description through Global Audio-Visual Character Identification\n\n[3] Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)