Scaling up Memory for Robotic Control via Experience Retrieval
Authors: Ajay Sridhar
Jennifer Pan
Satvik Sharma
Chelsea Finn
Venue: ICLR 2026 Poster (2026)
Keywords: Robot Learning
Memory
Vision-Language-Action Models
Primary area: applications to robotics, autonomy, planning
TL;DR: We enable existing vision-language-action models (VLAs) to solve long-horizon tasks that require minutes of memory by finetuning a VLM to act like a high-level planner and select task-relevant keyframes.

Abstract: Humans rely on memory to perform tasks; our goal is to endow robot policies with the same ability. Naively conditioning on long observation histories is computationally expensive and brittle under covariate shift, while indiscriminate subsampling of history leads to irrelevant or redundant information. We propose a hierarchical policy framework, where the high-level policy is trained to select and track previous task-relevant keyframes from its experience. The high-level policy uses selected keyframes and the most recent frames when generating text instructions for a low-level policy to execute. This design is compatible with existing vision-language-action (VLA) models and enables the system to efficiently reason over long-horizon dependencies. In our experiments, we fine-tune Qwen2.5-VL-7B-Instruct and $\pi_{0.5}$ as the high-level and low-level policies respectively, using demonstrations supplemented with minimal language annotations. Our approach, MemER, outperforms prior methods on three real-world long-horizon robotic manipulation tasks that require minutes of memory. Videos and code can be found at https://jen-pan.github.io/memer/.

Published as a conference paper at ICLR 2026
MEMER: SCALING UP MEMORY FOR ROBOT CON- TROL VIA EXPERIENCE RETRIEVAL
Ajay Sridhar* Jennifer Pan" Stanford University (ajaysri, Jrpanjestanford.ede
Satvik Sharma
Chelsea Finn
ABSTRACT
Humans rely memory perform tasks; our goal is endow robot policies with the same ability. Naively conditioning on long observation histories is com- putationally expensive and brittle under covariate shift, while indiscriminate sub- sampling of history leads to irrelevant or redundant information. We propose a hierarchical policy framework, where the high-level policy is trained to select and track previous task-relevant keyframes from its experience. The high-level policy uses selected keyframes and the most recent frames when generating text instruc- tions for a low-level policy to execute. This design is compatible with existing vision- Laiionlanguage-aee actiinn (VLA) models and enables the system to efficiently reason over long-horizon dependencies. In our experiments, 7B-Instruct and mp.5 as the high-level and low-level policies respectively, using demonstrations supplemented with minimal language annotations. Our approach, robotic ma-
fine-tune Qwen2.5-VE
MemER, outperforms prior methods on three real-world nipulation tasks that require minutes of memory. Videos and code can be found https://jen-Poo pan. gitb... 10/ m
INTRODUCTION
In recent times, we have seen significant strides in the Sangugg--folowwngg and generalization capa- bilities of robotic manipulation policies (Brohan et al. (2023); Intelligence et al. (2025); Kim et al. (2024); NVIDIA et al. (2025)) While these policies are getting better for real-world deployment, eritical limitation remains: the absence of long-ten -org-term memory. Memory allows humans to handle the inherent partial observability found in their environment. For instance, if a person wanted to make a sandwich, they would have to recall where they saw the jar of peanut butter or the knife, especially if these items were not recently viewed. The ability to form and retrieve long-term visual memories is crucial step towards robots solving complex, multi-step tasks. The goal of this paper is provide an effective way to enable existing generalist policies to solve tasks that require long-term visual memory.
Because conditioning on long sequences of high-dimensional image and video sequences is compu- tationally expensive, many existing generalist end-to-end policies are trained with little no visual history. The high memory cost makes training prohibitively expensive and model deployment un- usably slow. Furthermore, long observation histories can often introduce a form of overfitting shortcut reliance on spurious correlations between inputs and demonstrator actions (Torne et al.) 2025). The policy misgeneralizes under its own state distribution, leading to performance degrada- tion during deployment due to compounding covariate shift between states visited by the demon- strator policy and the learned policy. The suboptimal policy will generate histories that differ from those seen by the expert, which is only made worse observation history lengthens.
Some past works have shown it is possible to expand the observation context of their policy via auxiliary losses (Tome et al. 2025), or by finetuning pretrained foundation models for action pre- diction with native memory capabilities (Fang et al., 2025). Although these methods significantly increase the types of tasks robot can execute, they are challenging to naively scale to long histo- ries. To overcome this, policies must learn to filter out the task-relevant information from the full
"guul contribution

---

Published as a conference paper at ICLR 2026
historical context to prevent the memory footprint from exploding on tasks that require long-range dependencies.
To this end, propose approaching long-term memory for robotic policies with hierarchical framework. The high-level policy is a fine-tuned video-understandinng VLM trained to output action subtasks and, most importantly, important information will need to remember to solve the task. The low-level policy is generalist robot policy fine-tuned to execute the subtasks specified by the high-level policy. Together, the low- level policy handles the robot-specific challenges of the task that require high-frequency inference such as kinematic control, and the high-level policy manages planning and memory -pemoiy--pecific aspects of the task such as deciding what object or tool to manipulate next based on the high-level task and memory. We take advantage of the fact that these open-source VLMS are finetuned on large amounts of video understanding data. With this strong prior, we find that we only need a relatively small number of teleoperated robot demonstrations with additional annotations to adapt these VL.Ms to accomplish robot-specific memory-
select keyframes from its fixed recent context that represent
tasks (Bai et al. 2025
Our contribution is MemER, a framework for scaling up Memory in robotic control via Experience Retrieval. We demonstrate MemER's ss ability to utilize task-relevant past information effectively on three complex long- horizon tasks that require up to a couple of minutes of memory. To the best of our knowledge, our real-world robotic tasks necessitate reasoning over more image observations than prior work.
2 RELATED WORK
Memory in Robotics and Long-Context Policies. Memory is essential for generalist robots to complete complex tasks. Prior work primarily studies memory in the context of comparatively short-horizon tasks. For Somple,,TTrne, et al. (0025) and et al. (2025) use different approaches extend the context of imitation policies from a few frames at most two dozen. Our work in- vestigate tasks that require building memory from hundreds of frames. Unlike previous approaches, our method can choose to include task-relevant frames in the context spanning the entire episode. Another body of work investigates the compression of images in the policy's 's context. [Li et al. (2025a) compresses similar observations in pixel space, which is effective for stationary camera se- tups but struggles with wrist-mounted cameras that are necessary for most dexterous manipulation tasks. Memory has also played a major role in robotic navigation research. Some navigation works represent memory with an explicit geometric and/or semantic map of the environment Heenry et al. 2012); Yu et al. (2024). However, spatial maps of the environment are hard to apply in manipulation tasks since the robot often modifies the environment. Other works directly prompt API-based VLMs with video context to decide where the robot should navigate (Chiang et al.] 2024) [Sharma et al.) 2023 Chen al., 2024). We found that existing API-based VL.Ms not sufficient to reason about robot affordances for our long-horizon, memory-aware tasks (Table 2 so we resort to finetuning open-weight models.
Foundation Models and Long-Horizen Tasks in Robotics. Recent progress in vision-language- action models (VLAs) have allowed for impressive generality in robotics. VLAs combine web-scale pretraining with expressive action decoding mechanisms to execute real-world tasks. Conceptually, two paradigms have emerged. The first is a single end-to-end model that directly maps images and
language task to actions (Intelligence et al. 2025 Brohan et al.) 2023 Fan et al. 2025). The second hierarchical approach that uses a high-level policy to output an intermediate representation to guide low-level policy. Possible intermediate representations include latent embeddings (Shentu) cct al. 2025 Wen et al. 2025: Wu et al.. 2024), language subtasks (Shi ct al. 2025 2024), and way- points (Li ct a 2025b). Prior work has shown that hierarchical approaches improve performance on long-horizon tasks (Shi et al., 2025) Wu et al., 2024, Wen et al., 2025) by introducing a tempo- ral abstraction: a high-level policy operates
lower frequency and decomposes complex task into simpler subgoals (Zhang et al., (2023). The low-level policy can then focus on high-frequency, reactive motor control to achieve the current subgoal. Other methods that integrate L.sed high-level planning with task and motion planning, imitation learning, and RL (Dalal et al.) 2024] Zhou al. (024)) similarly tackle long- horizon manipulation via task decompositions, but rely on a well-specified planning stack and do not provide an explicit long-term memory mechanism over raw visual histories.
2

---

Published as a conference paper at ICLR 2026
High-Level Instruction 4
Selected Keyframes
Keyframe Filter
Candidate Kayframes Je
High-Level Policy (VLM)
framet
R₂
Language Subtask
Low-Level Policy (VLA)
Actions
Joints
Figure Architecture of MemER. The high-level policy processes task instructions, selected keyframes (if any), and recent images from base and wrist-mounted cameras generate low-level language subtasks and candidate keyframes (if any). The low-level policy uses the subtask, current image, and robot joint states to produce actions. The candidate keyframe(s) are processed by the keyframe filter to obtain the selected keyframes for input during the next step of inference.
Our work builds on the second paradigm, using language subtasks as the intermediate representation. What distinguishes MemER from prior hierarchical approaches is the addition of stable, persistent memory mechanism that preserves salient long-range dependencies while keeping inference fast enough for real-world deployment. We show that this memory system is essential for complex, long-horizon tasks that span multiple minutes, on which standard hierarchical methods with little or no memory reliably fail.
Video Keyframe Selection. Outside of robotics, previous work in computer vision has also studied incorporating longer contexts for VLMs (Goletto al. (2024): Manigrasso al. (2025) )) Similar to our work, other works have used keyframe selection to improve vidco understanding and question answering (Yu al. 2023 Ranasinghe et al., (025). Many such methods incur a high per-frame cost because they estimate frame importance via separate multimodal-LLAn calls. These methods are not directly applicable for robotic tasks because with increasing video context lengths they would not meet the task's latency constraint during inference. Hu et al. (2025a) uses lightweight models to score all frames in a single pass, which reduces per-frame cost but lacks the ability to continu- ously stream image observations. Departing from existing VLM work for VQA, Hu et al. (2025a) uses non-uniform frame sampling through a lightweight scoring model; in contrast, we achieve non-uniform sampling without additional models. Designed for real-world robotics, method emphasizes low-cost inference and streaming support.
3 MEMER
3.1 PRELIMINARIES
Language-Conditionn Control Policies. Language-condittione robot policies typically trained to model the conditional distribution p(A,lo:), where A, = [ae, G1+1, a++ is chunk of actions modeled from the current timestep
timesteps the future (Zhao et al. 2023) and is the robot's current sensor observation. The current observation is usually formulated Ot [r. 4,, (t), where J, 17 .2...., .., [7]] are images from multiple cameras, 4 is the language instruction, and qr the proprioceptive inputs from the robot (i.e. joint angles and gripper state) [Bacck et al. 2024 Team et al. 2024)..
Memory- Based Tasks. We consider set of tasks such that robot policy must leverage past in- formation to successfully complete them due to partial observability in the environment. other words, robot policy trained to model πT( (A,lor) could not complete the task, but policy trained to model (AA, (o:e) could.
Hierarchical Policies. In order execute complex, long-horizon tasks, follow Shi et al. (0055) and hierarchically decompose the robot policy into a low-level control policy (mT) and a high-level
3

---

Published as a conference paper at ICLR 2026
policy (IT)) that generates instructions.
x (Aelor) TI (A. (Av (Te, 1/, gol) (7/) Lt, 4t)
The high-level policy models the conditional distribution TT (4) 4), where 1/ is the current lan- guage subtask, which we also refer to as subtasks, that the low-level policy conditions on to complete the overall instruction 4. The high-level policy could be represented as separate VLM [Sh et al.
(2025) or share the same weights as the low-level control policy (Intelligence et al. 2025). In our method, the high-level policy will be responsible for reasoning about memory.
Data Collection Similar to prior work, we use language subtasks I to label each observation in trajectory (Shi et al. 2024 2025; Intelligence et al. (025). We end up with a dataset of trajectories of the following tuple (Ie, gt. 4, at) train our high-level and low-level policies. For example, we can take the task "search for ketchup" and break it down into the following subtasks: "look in left bin", "look in right bin", and "take out ketchup from right bin." " Examples of the subtasks and ground-truth frames for each task are shown in the Figure 3 In our expert data-collection setup, the operator executes a prescribed primitive and presses key upon completion to advance to the next primitive. Consistent with previous work, we supplement the low-level policy training set with 10-15 intervention demonstrations to improve robustness at deployment
al. 2025b); Kim al.) (025). To collect the intervention data, we first initialize the trajectories in common failure states we have seen the low-level policy reach, then we teleoperate the robot back into an in in-distribution state. Since the low-level policy only uses the current frame, we can have the robot start from bad states then demonstrate the correct behavior from those states.
3.2 HIGH-LEVEL, POLICY
Our method builds on the common hierarchical VLA paradigm, as in Shi et al. (2025), and extends it with the ability to tackle long-horizon tasks that require memory. We choose to adopt a hier- archical apporach because open source VLMs like Wreen..--L-7B-Instteee
have strong video understanding prior from the video datasets they have been trained on, and thus can be adapted for memory-based planning (Bai et al., 2025). We to both nominate candidate keyframes and predict subtasks for the low-level policy during closed- loop control, as shown in Figure I The candidate keyframes are then filtered for redundancy and added to a group of selected keyframes that the high-level policy conditions on continuously when predicting the next subtask and candidate keyframes. Concretely, at each timestep, we feed our high-level memory policy 1) the last N frames per camera R2 I1:+ where N is the integer context- window shared across cameras, 2) the high-level task instruction 4. and 3) previously se- lected keyframes K, S Tot--N+1, where practically KK < 8. The high-level policy then predicts two things: (i) the current subtask to execute & and (ii) the candidate keyframes Je C Re, a subset of frames from the recent context. All together, our high-level policy models (', Jel Rr, Kt). The low-level policy conditions on I/ to predict the direct joint velocities for the robot as described in Section 3.3 In parallel, KL+1, the selected keyframes for the next timestep of high-level infer- ence, are calculated from the sequence of all candidate keyframes predicted since the start of the trajectory Jo (Jo JJ,...,JJJ Jt) using simple 1D single-linkage clustering algorithm described in the following paragraphs.
finetuned VLM as the high-level policy
Building Visual Memory. To build visual memory, our keyframe filter consolidates observations that have just exited our recent context, RG, into our selected keyframes K,. The filtering process operates on the temporal indices of these keyframes. The approach is task-agnostic and allows us to have coverage of all frames in the stream. By timestep t, the high-level policy has given us Jo (J+) the sequence of candidate sets nominated up to timestep We begin by extracting the temporal index of every frame nominated in this sequence and pooling them into single tem- porally ordered list Go Importantly, we preserve duplicate indices, as this allows the subsequent median selection to aggregate repeated nominations and yield the most representative frame from each cluster.
We then create clusters, C₁ for all candidate indices in Go.t by grouping keyframe indices that are at most d apart from each other. For example, if Goat has temporal indices {1, 3. 4, 10) and d 5, then would have two clusters C1 = {1. 3, 3, 4) and C2 = {10}. After constructing the sequence of clusters, (C,, C2....), we select the median index from each C, to be that cluster's representative keyframe. The keyframes corresponding to these final median indices represent K,,
4

---

Published as a conference paper at ICLR 2026
Recentilomss
Timestapt
111
Froms 45 sooes
4811
114
Farms to mamory
cluster1
distamcc 5
cluster
Figure 2: ID single-linkage over nominated frames. At each timestep, the high-level policy nom- inates candidate keyframe(s), as highlighted in orange. All candidate keyframes are aggregated across time with ID single-linkage using a merge distance ters. For each cluster, the colored bars indicate nominations for the observation at that timestamp, with bar height proportional to the number of nominations received. We select one representative frame per cluster by taking the median keyframe of all the candidates, and add that frame to memory.
= 5 frames, yielding disjoint clus-
the selected keyfran frames for timestep t. For efficiency, clusters that have indices less than f- N- ++1-- 1-d do not need to be recalculated. Figure I visualization of the clustering and keyframe selection during deployment.
3.3 PRACTICAL IMPLEMENTATION OF MEMER
Training the Low-Level Policy. For our low-level robot policy, we finetune a version xo.s Intelligence et al. 2025) trained the DROID dataset Khazatsky et al. (2025). Given have trajectories of the tuple (It, qt. L2, ar), we can train our low-level policy to model the conditional distribution Ti (A, Le qt, (2). We choose finetune the Ta.5 checkpoint trained the DROID dataset due to its strong out-of-the-box behavior on the DROID setup we use to conduct all of our experiments. Consequently, we find that we need only 50 demos of long-horizon trajectories to finetune a strong low-level policy. We finetune single low-level policy across all three tasks. Refer to Appendix A for the specific training parameters.
Training the High-Level Policy. For our high-level policy, we finetune Qwen2.5 VL--BB -L-7------ to predict two things: 1) the current subtask execute and 2) any task-relevant keyframes remember from the most recent frames (as described in Section 3.2p. We finetune a single high-level policy on all three tasks, and we observe this gives the added benefit of stronger object generalization (see Appendix F for comparisons with the single-task variant of MemER). We freeze the weights of the vision encoder and projection layer during finetuning for training efficiency and to preserve the visual prior.
Annotating Keyframes for the Leve Policy. To label keyframes for each task, we employ cemi-automatic annotation procedure. First, we extract the boundary frames between consecutive subtasks, which are simply the last frame of each subtask segment. Next, we review a small number of demonstrations (-3) to determine simple annotation rule per subtask-deciding whether or to keep the last frame of that subtask segment as ground-truth keyframe, since these transition points usually contain visually informative state. For example, the rule may indicate selecting the last frame in "look inside the center bin." " or no frame for "reset scooper position." Once established, these rules are fixed per subtask and automatically applied to all demonstrations of each task; this process is not manual, framf effort, but a quick, one-time setup that makes keyframe labeling practically free. The resulting set of keyframes forms the ground-truit -truh targets used to train the high-level policy, See Appendix E for the specific keyframe annotation rules for all of the subtasks.
Closed-Loop Deployment. Our policy decomposition the following:
5

---

Published as a conference paper at ICLR 2026
T(Aelous) = =i (AelLee qu, Jile Ki)
(2)
The interaction between the low-level and high-level policy for closed loop deployment is shown in Figure [ The low-level policy predicts actions chunks at ~2Hz, while the the high-level policy Th predicts keyframes and subtasks at roughly ~1Hz. We run both policies on their own server. Like Shi et al. (2025), we choose to run the policies asynchronously, as we find it to improve perfor- mance. While the high-level policy is predicting the next primitive, the low-level policy conditions on the latest predicted primitive. We add the image observations sampled at 2Hz to queue, and then send this queue to the high-level policy to query the next primitive prediction after the current high-level policy prediction is complete. Following Anonymous Following/Anonymous (2025), we found that linearly inter- polating the weights of the finetuned high-level policy with its base model -iven..5-VL-Innstei improves performance of the hierarchical policy on most tasks (more details in Appendix H+).
EXPERIMENTS
In this section, aim evaluate the extent which method and alternative approaches can tackle long-horizon manipulation tasks that require some form of memory. We first describe our tasks and evaluation protocols, then we discuss the following questions:
1. To what extent our approach tackle tasks that require memory, in comparison to memory-less policy (i.e. current robot foundation models), a human high-level (Human HL) policy, and other naive approaches?
2. How does our high-level policy, fine-tuned from an open-source VLM, compare to propri- etary off-the-shelf vision-language models?
3. How does representing memory via images compare to other modalities?
We design three complex, real-robot tasks that entail using memory in multiple distinct ways, in- cluding remembering object locations, keeping track of previously completed actions, and counting repeated task steps, as illustrated in Fig. Since all of the tasks are long- horizon,,o,,, we record different metrics for each task to provide a granular view of task completion.
Object Search. In the task, randomly place three five objects across three opaque bins. Then, the robot is sequentially given three objects to find; each new instruction is issued only after the robot has attempted to retrieve the previous object. Our goal is an optimized search: the robot remembers which bins it has already examined (and what it saw), skips re-searching them, and explores additional bins only as needed; it should proceed directly to the target bin if it has already been searched. This task requires cross-episodic memory as finding each object is its own 4, thus requiring recall of information gathered while executing prior instructions. We train and test with the same set of 15 objects, which are various small toys. Evaluation metric. We measure task completion by two criteria for each of the three objects: successful retrieval and adherence to the optimal path without unnecessary exploration, for maximum score of (2 points per object).
Counting Scoops. In this task, the robot is asked to fill two separate bowls with scoops of food. Specifically, the robot is asked to place an exact number of scoops of two different ingredients into different bowls. The robot needs to keep track of how many scoops have already been obtained per ingredient. This counting task has appeared in prior work (Torne et al. 2025). we modify it to require much longer-horizon reasoning by increasing the potential number of scoops and ingredients to scoop from. This task is challenging because the keyframes corresponding to each ingredient nearly indistinguishable-piin keyframe can cause the high-level policy to misjudge its progress. We train and test with peanuts and jelly beans. Evaluation metric. Task completion is measured by the absolute value of the difference between the number of scoops requested and obtained, for each ingredient. Here, lower metric is better. We also report the 0-1 success rate for satisfying the instruction.
look almost identical after each scoop-so so missing or duplicating
Dust & Replace. this task, the robot asked to remove objects from a two-tiered shelf, pick up a duster, dust each shelf, and replace the objects to their original positions. Between dusting the two shelves, we return the duster to its reset position, making unclear from recent context which shelf has already been dusted. This task is challenging because the robot must simultancously remember

---

Published as a conference paper at ICLR 2026
TASK
"search blook
worden Black"
OBJECT SEARCH
SUBTASK
look Insido the left tin
kestchup from left bin and place white pin
look inside center bin
look inaide right bin
from right bin piace in white bin
take block
take wooden block from right bie and
place in wiit bin
EXECUTION
'put three of peanuts in the baxd, and three of beana the blue bowf
COUNTING
place scoop peanuts green
place secop peanuts the green
place scoop of
place sooop of jelly beans blue
place scoop of jelly beans blue
place a scoop of jully beans blue
SUBTASK
peanuta green
ROBOT EXECUTION
TASK
purole "seovvo the itams from the sheives, dust the shotves, and place the ltems back the sholves"
DUST REPLACE
SUBTASK
shelf
ROBOT EXECUTION
Figure 3: Task used in our evaluation. Across three domains. we evaluate complex instructions, intermediate subtasks, and keyframe predictions. We report performance across 20 trials per task per method.
100
97
97
100
96
96
Task Task Progross (%)
80
62
64
52
58
58
48
28
26
20-
Object Search
Counting
Replace
No History
Short History
Long History
MemER (Ours)
Human HL
Figure 4: Main Results. Our method clearly outperforms the no history, short history (8 frames of context), and long history (32 frames of context) baselines on the three aan-horizoo memory-based tasks. on par with the human high level policy.
two types of information: the original locations of the objects and which shelf, if any, has already been dusted. We train and test with a set of objects, which are various plushies. Evaluation metric. Task completion is measured by the binary success of each object being replaced correctly the shelf and each shelf being dusted, for a max score of 4.
Our robot setup resembles that within DROID (Khazatsky al. 2025 having Franka arm, parallel jaw gripper and two cameras: a third-person ZED camera and a wrist-mounted miniZED camera. For all tasks the गi operates ~ ~H and the मं operates at ~2Hz. क outputs an
Evaluation Setup:

---

Published as a conference paper at ICLR 2026
100
90
95
78
Taak Progress (%)
80
57
55
61
40-
20
Object Search
Counting
Dust Replace
Average
Short History Text
MemER Text
MemER (Ours)
Figure 5: Modality Results. Using only images to represent the memory performs better than the baselines that use only text or text and images. We hypothesize that the high-level policy over- indexes the text tokens in the memory, causing it to miss important details in the visual input.
Method
Object Search
Counting # wrong scoops (4)
Dust & Replace
times object retrieved (f)
# times used optimal path (?)
Dust bottom shelf (t)
Dust top shelf (1)
Replace bottom object (t)
Replace top object (1)
MemER (Ours)
57
19
20
No history
32
25
61
4
7
Short History
31
14
12
Long History
47
41
12
11
11
12
12
Human HL
58
58
19
19
18
17
Table 1: Detailed Main Results. Online evaluation of our method and the baselines for Q1. We provide task-specific evaluation metrics and the counts across 20 trials for each component of the task. Bold marks the best non-oracle method in each row. and indicate higher and lower is better, respectively.
action chunk A, of 15 actions, and we execute 8 open loop before replanning. The cameras stream 320 180 resolution images 15Hz, but we subsample 2Hz input hierarchical policy.
4.1 MAIN RESULTS
approach tackle tasks that require memory compared to other methods? All evaluated methods incorporate a Th and Tú, and the baselines change the input context of the high-level policy TTs while using the No history: memory-less high-level policy that only views the current frame (i.e. current robot foundation models). similar to (Shi al. 2025) 2) Short History: policy that views only the recent N frames (N = 8 for our setup) 3) Long History: policy that naively receives longer context (4x that of Short History or subtasks. The Human HL policy establishes rough estimate of the upper bound performance for all tasks, with failures stemming from the low-level policy. From Figure 4 we see that No History and Short History baselines perform poorly as all of the tasks simply require more context than what was provided. The Long History baseline shows that increasing the context can slightly help, but 32 frames (~16 seconds of memory) incurs inference cost of second, which approaches the limit of what can be tolerated in closed-loop settings. The Long History policy still performs on average 34% worse than MemER, necessitating strategies such rather than naively extending the context. Lastly, our method has > 95% on all tasks with the most common failure case being failures in the low-level policy executing the subtask, which can be rectified with better low-level correction data.
Q1: To what extent
mi We compare the following baselines: 1)
32 recent frames), and Human HL: a human provides the correct
method that consolidate keyframes
Q2: How does our high-level policy, fine-tuned from open-source VLM, compare to pro- prictary off-the-shelf VLMs? Since our approach outperforms other selections of na and performs similar the Human HL policy, we investigate our method is necessary given existing state-of-
8

---

Published as a conference paper at ICLR 2026
Method
Object Search
Counting
Dust & Replace
Trajectory acc. (1)
Boundary acc. (1)
Trajectory
Boundary acc. (1)
Trajectory acc. (1)
Boundary
acc. (1)
acc. (1)
MemER (Ours)
0.80
0.76
0.67
0.65
0.87
0.86
GPT-5
0.15
0.16
0.43
0.47
0.67
0.63
Gemini Robotics-FR 1.5
0.21
0.23
0.13
0.14
0.19
0.22
Table 2: Comparison with API-Based VLMs. Offline evaluations of the per-task trajectory and
boundary accuracy of subtask predictions between MemER, GPT-5, and Gemini Robotics-ER 1.5
(Team et al., 2025), to compare our finetuned high-level policy from an open-source VLM against proprietary VLMs.
Method
Input Components
Object Search
Counting
& Replace
# times object retrieved (1)
# times used optimal path (t)
# wrong scoops
Dust top shelf
Replace
Replace
Image Keyframes
bottom object (?)
top
History
Subtasks
(1)
shelff
object (f)
MemHi (Ouns)
$7
1
20
19
18
20
Short History + Text
40
28
10
16
16
7
10
McmER Text
49
13
20
18
17
20
Table 3: Detailed Modality Results. Online evaluation across methods ablating the textual modal- ity. Bold marks the best method. f and & indicate higher and lower is better, respectively.
the-art VLMs may already have this capability. We test both GPT-5 and Gemini Robotics-ER 1.5 (Team et al., 2025) given the former's strong multimodal reasoning performance and the latter's robotics-specific agentic capabilities. Because the API latency for both ranged from 10-15 sec- onds, these API-based high-level policies led to complete failures when we deployed it in the same closed-loop evaluation the other baselines, which require latencies of less than 1 second to react accordingly to the environment.
To still offer means of comparison between Th and the API-based high-level policies, de- signed an offline experiment using a held-out set of trajectories generated by the low-level policy commanded by ground-truth subtasks 1,- This simulates closed-loop execution under realistic be- haviors (i.c. retries after missed grasps, pauses, and jerky motions), while allowing the model to build its visual memory in the same way. We carefully craft the prompt include specific task- relevant instructions and an explicit list of all possible actions that the low-level policy can follow, and ask the model to choose among them (Appendix C]. Just like our setup, the model takes in the N = 8 most recent frames of context and selected keyframes K, at every timestep, and out- puts the subtask I for the low-level policy to execute and candidate keyframes Jt. We measure trajectory accuracy, which is how often the correct subtask is predicted at each timestep in the tra- jectory, since we know the ground at that moment. We also measure boundary accuracy, computed as the trajectory accuracy within a fixed window centered on transition points between subtasks. These are critical moments that expose the high-level policy's grasp of task progress by knowing when to move task; correct timing in transitioning between subtasks plays a major role in proper coordination with the low-level policy during deployment. From Table 2 observe that both zero-shot API-based models perform poorly compared to our finetuned Qwen2.5-Vi model, primarily failing by predict- ing too many non-informative candidate keyframes, reflecting its limited ability to identify which frames are truly useful. Consequently, even with a significantly stronger base VLM such as GPT-5 or Gemini Robotics ER 1.5, the model lacks the capacity to interpret robot-specifie perceptual cues and identify meaningful keyframes, resulting in less accurate subtask predictions and necessitating additional fine-tuning.
subtask command that the low-level policy is exccuting
the next sub-
Q3: How does representing memory via images compare to other modalities?
We now discuss which modalities are best suited for building memory-visual, textual, or both. Storing memory in text offers natural benefits as it's interpretable and much more condensed. We test two additional methods that use text memory, in the form of the predicted subtask 7/ that associated with each of the selected keyframes in Kg: 1) Short History + Text uses the most recent

---

Published as a conference paper at ICLR 2026
N = 8 frames and predicted subtasks and 2) MemER Text interleaves the predicted subtasks and visual keyframes in memory. shows the input for each baseline.
We see from Figure (left) that relying on textual memory underperforms compared to vision- only approach. Specifically, replacing the visual memory with text (Short History + Text) leads to the most significant performance drop. Furthermore, adding text to visual memory (MemER Text) provides no benefits, consistently under-performing across all tasks, especially the Counting task. We find that both the baselines' subtask predictions are brittle, largely due overreliance on the most recently predicted subtask stored in memory. This leads to failures when policy retries or freezes shift the recent context out of distribution. In such cases, the model tends to overfit to the canonical ordering of subtasks observed in expert demonstrations and misidentifies the subtask being executed given the current environmental state. In contrast, directly grounding predictions in the current observation combined with the robust visual memory proves more reliable.
For the Short History + Text baseline, the language-bas formation required to successfully complete the task. For example, in the Object Search task, the predicted language subtasks only specify the objects the robot has previously been asked to locate or is currently searching for, but have no reference to objects it has seen that may need to be retrieved in subsequent episodes. For the MemER + Text baseline, the model disproportionaaely attends to the text stored in memory, which can be incorrect for the reasons stated above, and subsequently ignores important information stored in visual memory. Such behavior has been noted before in (Zheng et al. [025] iil Lee et al. 2025). Thus, from our tasks, find that visual memory alone pro- vides the most robust representation, though exploring multimodal memory remains an interesting future direction.
subtasks do not capture all of the in-
DISCUSSION AND FUTURE WORK
We introduced MemER, a hierarchical laiio--angguaee-action framework that scales memory via experience retrieval. A high-level memory policy processes streamed observations, nominates keyframes to retain, and emits language subtasks that a low-level controller executes. A simple on- line consolidation strategy converts per-timestep candidate keyframes into a compact, stable episodic memory that is fed back into the high-level policy. Across three real-world, long-horizon manip- ulation domains, MemER significantly improves performance on tasks requiring minutes of recall while retaining low-latency inference and strong compatibility with existing VLA backbones.
Despite its benefits, our approach has several limitations. We continuously accumulate informative keyframes but currently lack a mechanism to discard them when they become too numerous an issue that may arise for tasks requiring hours of memory. Enabling the high-level policy to reason about which keyframes to not only add but also delete for modifiable long-term memory is an excit- ing direction for future work. Aghajoharii et al. (2025) proposes an approach that uses reinforcement learning to train an LLM to maintain a fixed-size memory state throughout its chain-of-thought reasoning. Adapting this idea could provide promising way to endow MemER with a learned memory management system that includes an explicit forgetting mechanism. In addition, our mem- ory is limited to visual observations; incorporating other sensory modalities such as tactile or audio is promising extension. Finally, we study a single robot embodiment, and extending to mobile manipulation and multi-room tasks, where memory must interleave spatial mapping with episodic recall, would bring the system closer to buman--like memory. We view MemER as step toward robot policies that decide what to remember and leverage those memories when needed for effective long-horizon control.
6
ACKNOWLEDGMENTS
This research was supported by ONR grant N00014-22-1-2621. the Robotics and AI Institute, and compute from the Stanford Institute for Human-Centered AI (HAI), We thank Dorsa Sadigh, John Yao, Moo Jin Kim, Lucy Xiaoyang Shi, Brian Kim, Homer Walke, Marcel Tome, Yuejiang Liu, Anikait Singh, Andy Tang, and Jennifer Grannen for their informative discussions. We thank Yue- jiang Liu, John Yao, Alex Swerdlow, and Ria Doshi for their feedback on earlier versions of the paper. We also thank Physical Intelligence for providing beta access to the n0.5 model. Ajay Sridhar is supported by the NSF Graduate Research Fellowship.
10

---

Published as a conference paper at ICLR 2026
7 REPRODUCIBILITY STATEMENT
We link the code for creating the training data and practical real-world deployment on our website.
8 ETHICS STATEMENT
We are not presently aware of significant ethical issues arising from this work.
THE USE OF LARGE LANGUAGE MODELS (LLMs)
We only used LLMs to rephrase and polish the text for clarity and readability.
REFERENCES
Milad Aghajohari, Kamran Chitsaz, Amirhossein Kazemnejad, Sarath Chandar, Alessandro Sor- doni, Aaron Courville, and Siva Reddy. The markovian thinker: Architecture-agnostic linear 065557
scaling of reasoning, 2025. URL
Anonymous. Robust fine-tuning of vision-language-action robot policies via parameter merging. In ICLR 2026 Conference Submission (OpenReview), 2025. URL https://oppeneviiii net/ forummid==w.Jw$552oM conference submission on OpenReview.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL https://arxivv htgg/bb//iae 2502. 13923
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubezak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. language-action flow model for general robot control, 2024. URL hhttps://arxiss http://aaxx....a/aaaaab/ org/abs/ 2410. 24164
A vision-
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexan- der Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-2: Dision-languagee action models transfer web knowledge to robotic control, 2023. URL https://arxiv. http://rxiiv.orr//b////2/0/0 15818
Annie S. Chen, Alec M. Lessing, Andy Tang, Govind Chada, Laura Smith, Sergey Levine, and Chelsea Finn. Commonsense reasoning for legged robot adaptation with vision-language models,
2024. URL httpps://axxxi http://aaxxxvvvooaaaaaaa argg/bs//// 2407. 02666
Hao-Tien Lewis Chiang, Zhuo Xu, Zipeng Fu, Mithun George Jacob, Tingnan Zhang, Tsang- Wei Edward Lee, Wenhao Yu, Connor Schenck, David Rendleman, Dhruv Shah, Fei Xia, Jas- mine Hsu, Jonathan Hoech, Pete Florence, Sean Kirmani, Sumeet Singh, Vikas Sindhwani. Carolina Parada, Chelsea Finn, Peng Xu, Sergey Levine, and Jie Tan. Mobility vla: Mul- timodal instruction navigation with long-context vims and topological graphs, 2024. URL https://arxiv/ http://aaxk...or/ab///0.7..0000
11

---

Published as a conference paper at ICLR 2026
Murtaza Dalal, Tarun Chiruvolu, Devendra Chaplot, and Ruslan Salakhutdinov. Plan- leamm::: Language model guided rl for solving long horizon robotics tasks, 2024. URL (tLlhhtss///arii org/abs/24g0 .1154
Yiguo Fan, Pengxiang Ding, Shuanghao Bai, Xinyang Tong, Yuyang Zhu, Hongchao Lu, Fengqi Dai, Wei Zhao, Yang Liu, Siteng Huang, Zhaoxin Fan, Badong Chen, and Donglin Wang. Long- vla: Unleashing long-horizon capability of vision language action model for robot manipulation, 2025. URL https://arxiv
wwterriivv.rrrveeee httg/bb//258..... 2508. 19958
Haoquan Fang, Markus Grotz, Wilbert Pumacay, Yi Ru Wang, Dieter Fox, Ranjay Krishna, and Jiafei Duan. Sam2act: Integrating visual foundation model with a memory architecture for robotic manipulation, 2025. URL https://arxiv. wwt. 18564
Gabriele Goletto, Tushar Nagarajan, Giuseppe Averta, and Dima Damen. Amego: Active memory from long egocentric videos. In European Conference on Computer Vision, pp. 92-110. Springer, 2024.
Peter Henry, Michael Krainin, Evan Herbst, Xiaofeng Ren, and Dieter Fox. Rgb-d mapping: Using kinect-style depth cameras for dense 3d modeling of indoor environments. International Journal of Robotic Research IJRR, 31:647-663, 04 2012. doi: 10. 0111777027836499113341
Kai Hu, Feng Gao, Xiaohan Nie, Peng Zhou, Son Tran, Tal Neiman, Lingyun Wang, Mubarak Shah, Raffay Hamid, Bing Yin, and Trishul Chilimbi. M-llm based video frame selection for efficient video understanding, 2025a. URL //arxiv.. wwwpa/awwwwooooooooooooooooroorr www.oogg////////o/oo/ooo /bss///22 19680
Zheyuan Hu, Robyn Wu, Naveen Enock, Jasmine Li, Riya Kadakia, Zackory Erickson, and Aviral Kumar. Rac: Robot learning for horizoo tasks by scaling recovery and correction, 2025b.
URL
abt: 2009. 07953
Kang il Lee, Minbeom Kim, Seunghyun Yoon, Minsung Kim, Dongryeol Lee, Hyukhun Koh, and Kyomin Jung. Vlind-bench: Mcasuring language priors in large asionnlanguage models, 2025.
URL https://arxxv.crtt org abs/ 2406. 08702
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubezak, Tim Jones, Liyiming Ke. Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. Wu.s: vision-language-actte action model with open-world generalization, 2025. BRLLhhtps:///ake orgg/ abs/2504.16055
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Young- woon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag Sanketi, Archit Sharma, Cody Simpson, Quan Vuong. Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Ro- han Baijal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, Vitor Guizilini, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Muhammad Zubair Irshad, Donovon Jackson, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mirchandani, Daniel Morton, Tony Nguyen, Abigail O'Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, An- drew E. Wang, Yilin Wu. Annie Xie. Jingyun Yang, Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayara- man, Joseph Lim, Jitendra Malik, Roberto Martín-Martin, Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip. Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. Droid: A large-scale in-the-wild robot manipulation dataset, 2025. URL wwtp:/iwww.ggg/////000000 12945
12

---

Published as a conference paper at ICLR 2026
Ji Woong Kim, Juo-Tung Chen, Pascal Hansen, Lucy X. Shi, Antony Goldenberg. Samuel Schmidgall, Paul Maria Scheikl, Anton Deguet, Brandon M. White, De Ru Tsai, Richard Cha, Jeffrey Jopling, Chelsea Finn, and Axel Krieger. Srt-h: A hierarchical framework for autonomous surgery via language conditioned imitation learning, 2025. URL. https://arxiv.org/// 2505. 10251
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong. Thomas Kollar, Ben- jamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source vision-language-actinn model, 2024. URL
2406 09246
Haoxuan Li, Sixu Yan, Yuhan Li, and Xinggang Wang. Towards fast, memory-based and data- efficient vision-language policy, 2025a.
http:/:://aaaii.oor///////////// 10322
10322
Yi Li, Yuquan Deng, Jesse Zhang, Joel Jang, Marius Memmel, Raymond Yu, Caelan Reed Garrett, Fabio Ramos, Dieter Fox. Anqi Li, Abhishek Gupta. and Ankit Goyal. Hamster: Hierarchical action models for open- world robot manipulation, 2025b. URL wtt.ttp:://aiii..rrrr.r 05485
Zaira Manigrasso, Matteo Dunnhofer, Antonino Furnari, Moritz Nottebaum, Antonio Finocchiaro, Davide Marana, Rosario Forte, Giovanni Maria Farinella, and Christian Micheloni. Online episodic memory visual query localization with egocentric streaming object memory, 2025. URL https://arxiv http:://aaxii..or/ab/////////// 16934
NVIDIA, :.
Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi "Jim" Fan, Yu Fang. Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang. Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li. Zongyu Lin, Kevin Lin, Guilin Liu, Edith Llontop, Loic Magne, Ajay Mandlekar, Avnish Narayan, Soroush Nasiriany, Scott Reed, You Liang Tan, Guanzhi Wang. Zu Wang, Jing Wang, Qi Wang, Jiannan Xiang, Yuqi Xie, Yinzhen Xu, Zhenjia Xu, Seonghyeon Ye, Zhiding Yu, Zhang, Hao Zhang, Yizhou Zhao, Ruijie Zheng, and Yuke Zhu. Gr00t nl: An open foundation model for generalist humanoid robots, 2025. URL 2503. 14734
Kanchana Ranasinghe, Xiang Li, Kumara Kahatapitiya, and Michael S. Ryoo. Understanding long videos with multimodal language models, 2025. http://ttss//aaxv.oog///////////
16998
Satvik Sharma, Huang Huang. Kaushik Shivakumar, Lawrence Yunliang Chen, Ryan Hoque, Brian Ichter, and Ken Goldberg. Semantic mechanical search with large vision and language models, 2023. URL cnttps://arxiin http://awxwvvoooorbrs//ssa wrgg/bb//302..o... 12915.
Yide Shentu, Philipp Wu, Aravind Rajeswaran, and Pieter Abbeel. From Ilms to actions: Latent codes bridges in hierarchical robot control, 2025. URL https://arriv.orr/abbb// 04798
Lucy Xiaoyang Shi, Zheyuan Hu, Tony Z. Zhao, Archit Sharma, Karl Pertsch, Jianlan Luo, Sergey Levine, and Chelsea Finn. Yell at your robot: Improving on-the-fly from language corrections, 2024. URL htttps://axxiv..ot//
2403. 12910
Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, Adrian Li-Bell, Danny Driess, Lachy Groom, Sergey Levine, and Chelsea Finn. Hi robot: Open-ended instruction following with hierarchical vision-language-action models, 2025. URL htttp:://axxiv.orr//assss
19417
Gemini Robotics Team, Abbas Abdolmaleki, Saminda Abeyruwan, Joshua Ainslie, Jean- -aptiste Alayrac, Montserrat Gonzalez Arenas, Ashwin Balakrishna, Nathan Batchelor, Alex Bewley, Jeff Bingham, Michael Bloesch, Konstantinos Bousmalis, Philemon Brakel, Anthony Brohan, Thomas Buschmann, Arunkumar Byravan, Serkan Cabi, Ken Caluwaerts, Federico Casarini, Christine Chan, Oscar Chang, London Hiappellet-Volpini, Jose Enrique Chen, Xi Chen, Hao- Tien Lewis Chiang, Krzysztof Choromanski, Adrian Collister, David B. D' A'mbrsi., Sudeep Dasari, Todor Davchev, Meet Kirankumar Dave, Coline Devin, Norman Di Palo, Tianli Ding, Carl Doersch, Adil Dostmohamed, Yilun Du, Debidatta Dwibedi, Sathish Thoppay Egambaram,
13

---

Published as a conference paper at ICLR 2026
Michael Elabd, Tom Erez, Xiaolin Fang, Claudio Fantacci, Cody Fong, Erik Frey, Chuyuan Fu, Ruiqi Gao. Marissa Giustina, Keerthana Gopalakrishman, Laura Graesser, Oliver Groth, Agrim Gupta, Roland Hafner, Steven Hansen, Leonard Hasenclever, Sam Haves, Nicolas Heess, Bran- don Hernaez, Alex Hofer, Jasmine Hsu, Lu Huang, Sandy H. Huang, Atil Iscen, Mithun George Jacob, Deepali Jain, Sally Jesmonth, Abhishek Jindal, Ryan Julian, Dmitrry Kalashnikov, M. Emre Karagozler, Stefani Karp. Matija Kecman, J. Chase Kew, Donnie Kim, Frank Kim, Junkyung Kim, Thomas Kipf, Sean Kirmani, Ksenia Konyushkova, Li Yang Ku, Yuheng Kuang, Thomas Lampe, Antoine Laurens, Tuan Anh Le, Isabel Leal, Alex X. Lee, Tsang- Wei Edward Lee, Guy Lever, Jacky Liang. Li-Heng Lin, Fangchen Liu, Shangbang Long, Caden Lu, Sharath Maddi- neni, Anirudha Majumdar, Kevis-Koking Maninis, Andrew Marmon, Sergio Martinez, Assaf Hur- witz Michaely, Niko Milonopoulos, Joss Moore, Robert Moreno, Michael Neunert, Francesco Nori, Joy Ortiz, Kenneth Oslund, Carolina Parada, Emilio Parisotto, Amaris Paryag, Acorn Poo- ley, Thomas Power, Alessio Quaglino, Haroon Qureshi, Rajkumar Vasudeva Raju, Helen Ran, Dushyant Rao, Kanishka Rao, Isaac Reid, David Rendleman, Krista Reymann, Miguel Rivas, Francesco Romano, Yulia Rubanova, Peter Pastor Sampedro, Pannag R Sanketi, Dhruv Shah, Mo- hit Sharma, Kathryn Shea, Mohit Shridhar, Charles Shu, Vikas Sindhwani, Sumeet Singh, Radu Soricut, Rachel Sterneck, Ian Storz, Razvan Surdulescu, Jie Tan, Jonathan Tompson, Saran Tun- yasuvunakool, Jake Varley, Grace Vesom, Giulia Vezzani, Maria Bauza Villalonga, Oriol Vinyals, René Wagner, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Chengda Wu, Markus Wulfmeier, Fei Xia, Ted Xiao, Annie Xie, Jinyu Xie, Peng Xu, Sichun Xu, Ying Xu, Zhuo Xu, Jimmy Yan, Sherry Yang, Skye Yang, Yuxiang Yang. Hiu Hong Yu, Wenhao Yu, Wentao Yuan, Yuan Yuan, Jingwei Zhang, Tingnan Zhang, Zhiyuan Zhang, Allan Zhou, Guangyao Zhou, and Yuxiang Zhou. Gemini robotics 1 1.5: Pushing the frontier of generalist robots with advanced embodied reasoning, thinking, and motion transfer, 2025. URL hnttps://arxiis / abs//5/s////5// 2510 03342
Octo Model Team, Dibya Ghosh. Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yun- liang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy, 2024. URL https://arxivv.rr/aba 2405. 12213
Marcel Torne, Andy Tang, Yuejiang Liu, and Chelsea Finn. Learning long-context diffusion policies via past-token prediction, 2025. URL
095611
Junjie Wen, Yichen Zhu, Jinming Li, Zhibin Tang. Chaomin Shen, and Feifci Feng. Dexvla: Vision- language model with plug-in diffusion expert for general robot control, 2025. URL https:// wwxiv.org//bb//50000 05855
Kun Wu, Yichen Zhu, Jinming Li, Junjie Wen, Ning Liu, Zhiyuan Xu, Qinru Qiu, and Jian Tang. Discrete policy: Learning disentangled action space for multi-task robotic manipulation. arXiv preprint arXiv:2409. 18707, 2024.
Haoming Ye, Yunxiao Xiao, Cewu Lu, and Panpan Cai. Pretraining a unified pddl domain from real-world demonstrations for generalizable robot task planning, 2025. URL httphhttt///rtte //arxiv. org/abs/2507. 21545
Justin Yu, Kush Hari, Kishore Srinivas, Karim El-Refai, Adam Rashid, Chung Min Kim, Justin Kerr, Richard Cheng, Muhammad Zubair Irshad, Ashwin Balakrishna, Thomas Kollar, and Ken Goldberg. Language-embeddded gaussian splats (legs): Incrementally building room-scale repre- sentations with mobile robot. In 2024 IEEE/RS International Conference on Intelligent Robots and Systems (IROS), pp. 13326- 13332, 2024. doi: 10. 10110990R00535522024.0000222222.
Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained imagelllannmmmaaaaaaaaaaaaa model for 06988
video localization and question answering, 2023.
Zichen Zhang, Yunshuang Li, Osbert Bastani, Abhishek Gupta, Dinesh Jayaraman, Yecheng Jason Ma, and Luca Weihs. Universal visual decomposer: Long-horizon manipulation made easy, 2023. URL
08581
Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware, 2023. https::/aaxxv.org/abb/////////// 137785
14

---

Published as a conference paper at ICLR 2026
Xu Zheng, Chenfei Liao, Yuqian Fu, Kaiyu Lei, Yuanhuiyi Lyu, Lutao Jiang, Bin Ren, Jialei Chen, Jiawen Wang, Chengxin Li, Linfeng Zhang. Danda Pani Paudel, Xuanjing Huang, Yu-Gang Jiang, Nicu Sebe, Dacheng Tao, Luc Van Gool, and Xuming Hu. Milms are deeply affected by modality bias, 2025. URL httptt/tss://ai......m www.:/org/gggooee/aa hwnggabb//25555 18657
Zihan Zhou, Animesh Garg, Dieter Fox, Caelan Garrett, and Ajay Mandlekar. Spire: Synergis- tic planning, imitation, and reinforcement learning for long-horizon manipulation, 2024. URL http:://arxii.orr//bbssss 18065
15

---

Published as a conference paper at ICLR 2026
A MODEL INITIALIZATION AND HYPERPARAMETERS
Training the High-Levell Policy The hyperparameters for fine-tuning the Qwen2.5-VL-7B-Anstmtt high-level policy are detailed in Table
Table 4: Hyperparameters for High-Level Policy (Qwen2.5-VL-7B-Instrrcc) fine-tuning.
Hyperparameter
Value
Learning Rate
6e-5
Optimizer
Adam W
B1
0.9
B3
0.999
Weight Decay
Gradient Clip Norm
1.0
LR Schedule
Cosine
Warmup Ratio
0.05
Batch Size
256
Training
4500 gradient steps
Compute
96 H200 GPU hours
Frozen Layers
Vision Encoder, Projection Layer
Trainable Layers
LLM Backbone
Training the Low-Level Policy The hyperparameters for fine-tuning the ma.s low-level policy are detailed in Table 5 The model is fine-tuned from the public To.5 checkpoint trained on the DROID dataset (Khazatsky et al. 2025).
Table Hyperparameters for Low-Level Policy (To.s) fine-tuning.
Hyperparameter
Value
Learning Rate
2.5e-5
Optimizer
AdamW
B1
0.9
B2
0.95
Weight Decay
0
Gradient Clip Norm
1.0
LR Schedule
Cosine
Warmup Steps
1000
Batch Size
128
Training Steps
18000
Compute
48 H200 GPU hours
B DATA COLLECTION AND LABELING THE SUBTASKS.
For collecting the robot trajectory data follow the data collection procedure with the Oculus teloperation set in Khazatsky et al. (2025). To make the primitive labeling process for data collection easy possible, we generate the subtasks associated with the task before collection data. For instance, for the counting task, if we wanted to scoop 3 scoops of peanuts in the blue bowl and scoop of jelly beans in the blue bowl, would generated list of subtasks for the whole trajectory. This includes a pick up scooper primitive, primitive for each individual scoop, reset scooper primitive between the scoops, and drop scooper primitive. While collecting the data, we just need to follow what the current primitive is asking, and we only need to indicate when primitive ends with simple keyboard input. We also automate the randomization of the high-level task to avoid human biases when collecting data.
16

---

Published as a conference paper at ICLR 2026
Konka
Statetiiing
Object Search
Counting
Dust & Replace
Figure 6: Images of the three tasks.
C
PROMPTS FOR GPT-5 GEMINI ROBOTICS-ER. 1.5 EVALUATION
Object Search System Prompt
You are an Al assistant controlling single-arm robot to search for specific objects amongst 3 bins. When exploring the bins for objects, look in the order of left bin, center bin, then right bin. You will receive images from two cameras: one for global view and one on the robot's wrist for detailed view. You will be provided with recent images that show the most recent actions the robot has executed. You will also be provided with selected keyframe images which are frames of particular importance from all the actions the robot has executed so far. Based on these, choose an action from the provided list for the robot to execute to best achieve the user's task instruction. Provide the exact action from the list without any explanation.
You will select your action from the following list:
look inside the <LOCATION> bin take the <OBJECT> from the <LOCATION> bin and place it in the white bin
<LOCATION> is one of "left", "center", or "right". <OBJECT> is one of "green tape", "red block", "corn". "baguette", "blue block", "fried chicken". "milk carton", "ketchup", "craser", "grapes". "strawberry", "tomato", "pear", "wooden block". or "olive oil".
You will also return list of values from 1-8 to index which of the frames from the most recent actions seem to of particular importance for the robot remember. For this task, recalling what objects are in each bin is critical, so you should return a list of indices, if any, from the most recent frames that provides good view of bin.
Return JSON with:
current_ subtask the action that should be executed the current timestep, selected from the above list using the stated <OBJECT> and <LOCATION> values keyframe__ rasee_osei_epeee tions: list frame positions from 1-8, any, from the recent frames to keep track of which objects are in each bin
17

---

Published as a conference paper at ICLR 2026
Counting System Prompt
You are an Al assistant guiding a single-arm robot to obtain a specific amount of scoops of two different ingredients. You will reset the scooper between each scoop, and drop the scooper when all scoops across both ingredients have been obtained. You will receive images from two cameras: one for global view and one the robot's wrist for detailed view. You will provided with recent images that show the most recent actions the robot has executed. You will also be provided with selected keyframe images which are frames of particular importance from all the actions the robot has executed so far. Based on these, choose action from the provided list for the robot execute best achieve the user's task instruction. Provide the exact action from the list without any explanation.
You will select your action from the following list:
pick up the scooper
place scoop of <OBJECT> in the <COLOR> bowl reset scooper position drop the scooper
<OBJECT> is one of "peanuts" or "jelly beans". <COLOR> one of "green" "blue".
You will also return list of values from 1-8 to index which of the frames from the most recent actions seem to of particular importance for the robot remember. For this task, recalling how many scoops of each ingredient have been obtained is critical, so you should return a list of indices, if any, from the most recent frames that provides a good view of completed scoop.
Return a JSON with:
current__ suutt__uubtaat tubtask: the action that should be executed the current timestep, selected from the above list using the stated <OBJECT> and <COLOR> values
keyf kayerame_pae iate_pommisiiio tions: list of frame positions from 1-8, if any, from the recent frames to keep track of scoops
Dusting System Prompt
You are an Al assistant guiding a single-arm robot to take an object off each shelf (bottom shelf then top shelf), pick up the duster, dust the bottom shelf, reset the duster, dust the top shelf, put down the duster, and replace the objects back to their original places (bottom shelf then top shelf). You will receive images from two cameras: one for a global view and one on the robot's wrist for a detailed view. You will be provided with recent images that show the most recent actions the robot has executed. You will also be provided with selected keyframe images which are frames of particular importance from all the actions the robot has executed so far. Based on these, choose an action from the provided list for the robot to execute to best achieve the user's task instruction. Provide the exact action from the list without any explanation.
You will select your action from the following list:
remove the object on the bottom shelf
remove the object the top shelf
pick up duster
dust bottom shelf
reset duster
dust top shelf
put down duster
place the <OBJECT> on the bottom shelf place the <OBJECT> the top shelf
<OBJECT> is one of "panda plushic", "purple plushic". "zebra plushie", "elephant plushic", "lion plushic", "smily face ball", "hello kitty plushic", "baby shoe", "milk carton".
18

---

Published as a conference paper at ICLR 2026
You will also return list of values from 1-8 to index which of the frames from the most recent actions seem to be of particular importance for the robot to remember. For this task, recalling where the items were originally placed on the shelves and which shelves have been dusted is critical, so you should return a list of indices, if any, from the most recent frames that provides good indication of either.
Return JSON with:
current_ sunttt_uuttskk the action that should be executed the current timestep. selected
from the above list using the stated <OBJECT> values
keyframe_pos tions: list frame positions from 1-8, if any, from the recent frames to keep track of where the objects were originally placed on the shelves and which shelves have been dusted
D INFERENCE SPEED AND MEMORY COST
An important aspect of using keyframes as sparse memory representation is the ability to maintain fast inference and low memory usage for solving tasks that require reasoning over hundreds of frames. We run all of our experiments an NVIDIA GeForce RTX 4090 GPU with our finetuned Owen2.5-VV-Inntruccee
model. In Figure we show how the latency for the high-level policy changes as we increase the number of keyframes in its context [. from O to 8 (we also keep the recent context Rg] = 8). We see that the inference speed always stays below 0.8s per prediction. Empirically, we found any high-level policy that can predict at IHz or faster to perform the best on real-world deployments, so we are well within this limit. Additionally, our VRAM usage is within the 24GB limit for 4090, so we can run our high-level policy on single card.
However; we were to use naive method of retaining long-range dependencies by simply keeping track of more recent frames, after increasing the number of recent frames beyond 32, the inference cost exceeds 1.0s (the IHz threshold), it would not coordinate well with the low-level controller in real-world setting.
in Figure that the inference cost quickly blows up. Specifically,
In Table 6 we show the inference speed for the low-level policy, Tu.s- To.s can also run on a 4090 GPU well above the desired speed of 2Hz. We only need two 4090s to run our hierarchical policy.
Model Configuration
Inference Time (s)
VRAM (GB)
0.088 + 0.001
6.25
MemER (8 recent keyframes)
0.787 + 0.066
15.93
No History (1 recent frame)
0.532 + 0.065
15.55
Short History (8 recent frames)
0.591 + 0.064
15.64
Long History (32 recent frames)
0.874 + 0.065
16.01
Table 6: Comparison of inference speed and VRAM usage across models for the high-level (Qwen2.5-VL -L---LLsruutt-7BB and low-level (To.s) policy on a 4090 GPU. We run 20 trials per value.
MemER: inference Time Keyframes
MemER: VRAM Usage Keyframes
15.90
0.75
VRZMM (GB)
15.85
15,80
0.6
15.65
3
2
Number of Keyframes
Number of Keyframes
Figure 7: Plot of the average inference inference speed (Left) and VRAM usage (Right) for the MemER high-level policy. We evaluate D0ven2.5-vLL7BB-Instte for the high-level policy. All con- figurations include 8 recent frames ([R,) 8); the x-axis shows the number of additional keyframes added to context. We run 20 trials for each data point.
19

---

Published as a conference paper at ICLR 2026
Naive Context Sceling: inference Time
Neive Context: Scaling: VRAM Usage
19.5
19.0
3.5
3.0
2.0
1.5
16.5
16.0
05
15.5
2 21
23
2
2
2
2
Context Length
Context Length
Figure 8: Plot of the average inference speed (Left) and VRAM usage (Right) for naively scaling context in the high-level policy. We evaluate Qwen2.5-VL. 1L-.5--V--B----tttee for the high-level policy. This baseline uses only recent frames (no additional keyframes); the x-axis shows the total context length [Re] from (29) to 256 (2*). We run 20 trials for each data point.
ANNOTATION RULES FOR KEYFRAMES
we use a simple annotation rule for each subtask to build the set of keyframes that constitute the ground-trute truth targets. We take the last frame of the following subtasks:
As described in Section 35
Object Search
"look inside the <LOCATION> bin"
Counting
"place scoop of <OBJECT> the <COLOR> bowl" "
Dust & Replace
"remove the object the bottom shelf"
"remove the object on the top shelf"
"dust bottom shelf"
"dust top shelf"
"place the <OBJECT> on the bottom shelf"
"place the <OBJECT> the top shelf"
The last frames of these subtasks represent what the policy needs remember.
F CROSS-TASE OBJECT GENERALIZATION
To evaluate the benefits of multi-task training, compare single-task version of MemER (sep- arate policy trained for each task) with our multi-task version. For these experiments, we finetune Twen2.5-VV--Instuucce
for the single- and multi-task versions of the high-level policies. We eval- uate on our object-centric tasks: the Object Search and Dust & Replace tasks, using objects from Figure TO First, establish baseline by evaluating both versions of MemER their original task setups. As shown in Figure (left), their performance is roughly similar.
The primary benefit of multi-task training is revealed when evaluating cross-task object generaliza- tion. For this evaluation, we swap all of the objects between the tasks (e.g., using Object Search objects for the Dust & Replace task, and vice-versa). This creates new object-task combinations that the models have not seen during training. The results in Figure ( (right) demonstrate the clear advantage of using the multi-task model (82% success), generalizing much more effectively to out- of-domain combinations than the single-task version (59% success). This demonstrates that our method learns a generalizable skill of "what to remember" that transfers to new scenarios, rather than just overfitting to the original training demonstrations.
20

---

Published as a conference paper at ICLR 2026
Single- vs Multi-Task
Cross-Tase Task Object Generalization
100
96
59
Task Progress (%)
70
Task Progress
40
20
Object Search Dust & Replace
Average
Object Search
Dust Replace
Average
Single-Task MemER
MultiTTask MemER (Ours)
Single-Task MemER Muli-Task MemER (Ours)
Figure 9: (Left) Single- Multi-Task Results. The performance of the single- and multi-task versions of MemER on the Object Search and Dust & Replace task are similar. (Right) Cross-Tase Object Generalization Results. We swap objects between the Object Search and Dust & Replace (see Figure [0) tasks during evaluation. The multi-task policy can generalize to the new object-task combinations during evaluation despite never seeing them in training.
Figure 10: (Left) Objects for the Object Search task. (Center) Objects for the Counting task. (Right) Objects for the Dust & Replace task.
21

---

Published as a conference paper at ICLR 2026
G KEYFRAME SELECTION ALGORITHM
Algorithm Selecting Keyframes from Candidates
Input: A sequence of candidate keyframe sets Jor (J........,
The merge distance
Output: A list of the selected keyframes K
1: function
d)
2:
Gosz Sort SeeeenndiissFFooFramaa (J:.)
Extract the temporal indices. Handle case with no candidates.
3: if Gozt is empty then
4: return 0
5: end
6: Clusters 4-
7: Courvent f- [0]]
8: for i
Go.t do
Build the clusters.
9:
if Go:eli] Go - 1 < then
Append Go:e [i] Cearrent
11:
else
Append Ceurrent to Clusters Ceurrent f- [ end
13:
Start new cluster.
14:
15:
end for
16:
Append Ceurrent to Clusters
17:
T Teelected t
18:
for each cluster C in Clusters do Emedian &- Median(C)
Di Select the median index of each cluster.
19:
20:
Append imedion to Teelected end for K, f GodFameesroolldicesTTcccaaa
22:
return K,
24: end function
22

---

Published as a conference paper at ICLR 2026
100
97
95
96
96
88
Task Progress (%)
80
70
63
57
40
20
0
Object Search
Counting
Dust Replace
Average MemER (Ours)
Short History Text
MemER Text
Figure 11: Merging our fine-tuned high-level policy's weights with its base model's weights im-
proves maintains performance all tasks.
H HIGH-LEVEL POLICY PARAMETER MERGING
An important factor contributing to the success our policy is the strong video understanding prior in Qwen2.5-VL-7B Instructttt (Bai et al.] 2025). However, training the high-level policy to accurately predict the language subtasks used by the low-level policy requires roughly 5,000 gradient steps. After this amount of fine-tuning, the high-level policy tends to lose some robustness low- level policy freezes and retry behaviors, due to its training data consisting solely of optimal expert demonstrations. Concurrent work suggests that linearly interpolating the weights of a generalist pretrained model with those of the same model fine-tuned on narrow, task-specific data can help preserve the pretrained model's robustness and generalization, while still allowing adaptation the new task (Anonymous) (025). We find this also applies to the high-level policy. Specifically, we set the weights of our high-level policy as:
0= a1 Ope + + 0+
where pee is the weights of Bwwen255vV--7B-Instiie and On is the weights of this model fine-tuned on all three memory-beed based tasks. We follow AAnoyymous (2025) and set a = 0.8 for all basclines test. Figure II shows that model merging improves maintains performance across all tasks.
I FREQUENCY DOMAIN-BASER CLUSTERING EXPERIMENTS
We originally explored more seuristic-based methods for selecting keyframes, but found them to be much less reliable than semantically selected keyframes. MemER's keyframes come from a trained high-level VLM that nominates task-relevant frames, and the clustering step consolidates these into a compact memory. On the other hand, frequency-domain methods operate on low-level intensity/spectral changes, so they mostly detect visual motion rather than subtask boundaries that correspond to meaningful visual states. Because the VLM sees language and multi-view context, it can learn ignore viewpoint noise and only nominate genuinely informative frames. Frequency- domain changes spike on any large camera motion or background shift, so they over-trigger on irrelevant movement in egocentric manipulation.
To illustrate this, we ran UniDomain's 's Ye et al., 2025) clustering method using energy-based ex- trema in sliding window, and observed that the selected keyframes are much less informative. We take random Counting Task demonsiration struct memory buffer that is the same size as our method, which means that a lot of salient information will be lost if it occurs within 350 frames of local minimum/mi miximummmaximum (Figure [2). we allow the window to be smaller but still sparse enough such that the high-level policy runs at 1Hz (32 frames of memory, window size of 35), we get incredibly noisy and uninformative keyframes (Figure
an example and include the results below. To con- window size needed be >350,
23

---

Published as a conference paper at ICLR 2026
Thamd tryhueee
Nanáa Salected Keyftrres
Figure 12: (Top) UniDomain's 's clustering method with a window size of 351. (Bottom) MemER's s selected keyframes.
UniDoradin (53
Figure 13: (Top) UniDomain's clustering method with window size of 35. (Bottom) MemER's selected keyframes.
24

---

Published as a conference paper at ICLR 2026
J ATTEMPTS AT A UNIFIED HIERARCHICAL MODEL
Additionally, our approach trains two models for the hierarchical policy Shi et al. (022 while more elegant method would use the same of weights for unified policy Black al. (2024).
We initially tried to train a unified model by fine-tuning To.5 for subtask prediction from videos.
This failed as N0.5 lacked the proper video understanding pretraining to memory- -ased subtask prediction from just 50 demos. We also tried to train Qwen2.5VL to predict low-level actions in addition to subtasks and keyframes, but we found difficult to balance all training objectives on our limited dataset. However, it would be worthwhile to take the ideas from MemFR, such as keyframe prediction and management, and scale them to large pretraining runs for VLAs to train the next generation of memory-based VLAs.
We found that existing pre-trained models are quite specialized: *0.5 has strong action priors but very poor video-understanding priors, while Qwen2.5-VI. has the reverse behavior. A unified model struggled to effectively learn both memory-aware subtask prediction and action prediction from our limited data (50 demos/task). We attempted two unified variants:
Fine-tuning To.5 (VLA) for high-level memory reasoning: We tried fine-tuning To.5 to predict subtasks and keyframes from frames of context (the setup for the high-level policy in MemER) in addition to action predictions. This failed, as the model lacked the necessary video-understanding pre-training reason about long-horizon context.
Fine-tuning Qwen-VI VL (VLM) for low-level actions: We tried fine-tuning Qwen2.5-VIL to predict low-level actions using the FAST tokenizer, in addition subtasks and keyframes. We found it extremely unstable to train with both the action-generation and video-planning losses.
Our hierarchical design is pragmatic solution that leverages the distinct strengths of both pre- trained models, and this modularity is what enables MemER to succeed in complex, multi-minute tasks from only 50 demonstrations. Moreover, our method is compatible with existing VLA models to allow the system to efficiently reason over long- mor-hooizon dependencies.
25


Reviews:
[{"review_id": "bRchgY7BuV", "reviewer": "Reviewer_eE1T", "summary": "This paper proposes MemER, a hierarchical framework that endows VLAs with long-term memory via experience retrieval. A high-level VLM policy processes recent frames and retrieved keyframes to generate language primitives and nominate new keyframes for storage. A low-level VLA policy executes these primitives based on the current frame. This sparse retrieval mechanism allows the robot to successfully perform complex, multi-minute tasks that depend on recalling distant past events.", "questions": "- The most important concern is that the dual-policy architecture is computationally heavy and overly complex, requiring two separate, large models (a VLA and a VLM) to be run in parallel. The paper fails to justify this separation or explore a more efficient, unified architecture where a single VLM backbone could perform both high-level planning (keyframe nomination, primitive generation) and low-level feature extraction, which is a significant missed opportunity.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The hierarchical design, where a high-level policy learns to explicitly nominate salient keyframes for retrieval, is a novel and highly scalable architecture for managing truly long-horizon (multi-minute) dependencies.\n- The experiments are well-designed for long-horizon tasks, and the ablation comparing visual memory ($K_{img}$) to textual memory ($K_{text}$) provides a crucial insight into the limitations of using language as a lossy memory representation.", "weaknesses": "- The method's generalizability is unproven, as it was only evaluated on three custom, in-domain tasks and lacks benchmarks on standard, multi-task datasets like RoboCasa or LIBERO.\n- The framework introduces significant system complexity and data annotation overhead, as it requires labeling both language primitives and ground-truth keyframes for training the high-level policy.\n- The paper provides no computational analysis, making it impossible to assess the inference latency or memory cost of this dual-policy system, which is a critical factor for real-world deployment.\n- The comparison to GPT-5 was only conducted offline (Table 2), and it's unclear if this setup accurately reflects the challenges of closed-loop control or simply highlights the need for finetuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dGV8StheRj", "reviewer": "Reviewer_bnmy", "summary": "- A hierarchical VLA (Vision-Language-Action) framework where the high-level policy retrieves and tracks keyframes from past experience.\n\n- Efficient memory management via online keyframe selection and filtering, reducing redundancy and computational cost.\n\n- Real-world evaluation on three long-horizon tasks requiring minutes of memory: Object Search, Counting Scoops, and Dust & Replace.", "questions": "- How does the keyframe extraction algorithm proposed in this work compare to frequency domain-based clustering methods (such as Fourier transform or wavelet transform followed by clustering), for example, with *UniDomain*?\n- As tasks become longer, does the computational overhead of maintaining an explicit visual memory buffer adversely impact the performance of the high-level policy by reducing its inference frequency?\n- Does the approach presented in this paper offer significant advantages compared to existing long-context strategies, such as introducing sink tokens?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Successfully tackles real-world robotic tasks that require reasoning over several minutes of past experience (hundreds of frames), a significant step beyond prior work limited to a few dozen frames.\n- The hierarchical design with intelligent keyframe selection avoids the high cost of processing long, raw video sequences, enabling low-latency inference (~1 Hz for high-level policy) suitable for closed-loop control.", "weaknesses": "- The framework was evaluated on a single robot arm and on memory within a single task. Its scalability to mobile manipulation, multi-room navigation, and cross-task memory recall remains unexplored.\n- The approach is inherently limited to the specific task it was fine-tuned on and lacks the capacity for broader scaling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YUu25l043m", "reviewer": "Reviewer_JibS", "summary": "This paper presents MemER, a hierarchical architecture for Vision-Language-Action (VLA) modeling. The low-level policy is a general VLA model, while the high-level policy employs a Vision-Language Model (VLM) to predict primitives and identify key frames. The core contribution is the proposed experience retrieval mechanism, which operates at the high-level policy, enabling the VLM to leverage critical historical information for more accurate primitive prediction.", "questions": "- Could the authors more clearly articulate the specific innovative contributions of this paper? Given that the architecture is simple and the high-level training methods appear familiar, a clearer distinction from existing work is necessary to establish the technical merit.\n\n- The authors discuss several related VLA models that utilize memory in the Related Works section. Why are these models not included as baselines for comparison in the experimental evaluation? Including these relevant memory-based approaches would provide a more rigorous validation of MemER's claimed state-of-the-art performance in memory-intensive tasks.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is easy to follow, and the figures and tables are clear and easy to understand.\n- There is a strong motivation for this work. Memory is essential for real-world, long-horizon tasks, a need that is often overlooked in existing VLA literature.\n- The approach is simple and effective. The low-level VLA model requires minimal or even no training, with the bulk of the effort focused on fine-tuning the high-level VLM. This results in a low overall training cost.\n- The experimental validation provided is thorough and comprehensive.", "weaknesses": "- The architecture of MemER appears overly simple. The core advancements appear to lie primarily in the training methodology and data preparation for the high-level VLM, and these specific improvements do not seem to be particularly novel. This raises concerns about the overall innovation of the paper.\n\n- The data preparation phase seems to be highly resource-intensive, relying on manual annotation of primitives and the segmentation of trajectories.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dGMV3mp75D", "reviewer": "Reviewer_uYHS", "summary": "This work addresses the problem that robotic policies often rely only on the current observation or a few recent frames and thus lack long-term memory. It proposes a hierarchical policy framework: the high-level policy selects keyframes or memories from past experiences and generates language-based subtasks or instructions based on both the current observation and the retrieved keyframes; the low-level policy then receives the current image, the robot’s current state, and the subtask produced by the high-level policy to execute the concrete actions.", "questions": "1. How does the high-level policy determine which frames are keyframes? When the task types or scenes vary significantly, is this keyframe selection mechanism still generalizable, or does it require task-specific tuning?\n\n2. The paper includes baselines such as “Short History (8 frames)” and “Long History (32 frames).” I’m curious why the long-history setup leads to such a large improvement. Moreover, since simply adding more historical frames performs worse than MemER, why does “more history” not yield the same gains as the keyframe retrieval mechanism?\n\n3. After the high-level module generates subtasks and keyframes, isn’t the way the low-level policy directly uses this information somewhat too simple?\n\n4. Although the paper proposes using memory to tackle long-horizon problems, it does not compare MemER conceptually with other VLA-based or related long-horizon approaches [1, 2, 3]. Even if experimental comparison is difficult, a conceptual discussion would provide readers with deeper insights.\n\n5. The paper mentions that keyframes are accumulated into memory but currently lacks a mechanism for “deletion” when the memory becomes too large. Are there any promising methods to address this issue?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The system design is simple, effective, and scalable.\n\n2. The results show a significant improvement in long-horizon task success rates.\n\n3. The writing is clear and easy to follow.", "weaknesses": "1. The evaluation of scalability aspects such as memory size and retrieval latency could be more detailed.\n\n2. Providing a conceptual comparison among different types of approaches addressing the long-horizon problem would offer readers deeper insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)