SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
Authors: Haoquan Fang
Markus Grotz
Wilbert Pumacay
Yi Ru Wang
Dieter Fox
Ranjay Krishna
Jiafei Duan
Venue: ICML 2025 poster (2025)
Keywords: Robot Learning
Behavior Cloning
Imitation Learning
Memory-based Robotics Transformer
Primary area: applications->robotics
TL;DR: Memory-based Multi View Robotics Transformer

Abstract: Robotic manipulation systems operating in diverse, dynamic environments must exhibit three critical abilities: multitask interaction, generalization to unseen scenarios, and spatial memory. While significant progress has been made in robotic manipulation, existing approaches often fall short in generalization to complex environmental variations and addressing memory-dependent tasks. To bridge this gap, we introduce **SAM2Act**, a multi-view robotic transformer-based policy that leverages multi-resolution upsampling with visual representations from large-scale foundation model. SAM2Act achieves a state-of-the-art average success rate of **86.8% across 18 tasks** in the RLBench benchmark, and demonstrates robust generalization on *The Colosseum* benchmark, with only a **4.3% performance gap** under diverse environmental perturbations. Building on this foundation, we propose **SAM2Act+**, a memory-based architecture inspired by SAM2, which incorporates a memory bank, an encoder, and an attention mechanism to enhance spatial memory. To address the need for evaluating memory-dependent tasks, we introduce ***MemoryBench***, a novel benchmark designed to assess spatial memory and action recall in robotic manipulation. SAM2Act+ achieves an average success rate of **94.3% on memory-based tasks** in *MemoryBench*, significantly outperforming existing approaches and pushing the boundaries of memory-based robotic systems.
Project page: [sam2act.github.io](https://sam2act.github.io/).

SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
Haoquan Fang! Markus Grotz!
Wilbert Pumacay² Yi Ru Wang'
Dieter Fox" 11
Ranjay Krishna" 14 Jiafei Duan
Abstract
1. Introduction
Robotic manipulation systems operating in di- verse, dynamic environments must exhibit three critical abilities: multitask interaction, general- ization to unseen scenarios, and spatial mem- ory. While significant progress has been made in robotic manipulation, existing approaches of- ten fall short in generalization to complex en- vironmental variations and addressing memory- dependent tasks. To bridge this gap, we intro- duce M2AcAtt, a multi-view rohotic transformer- based policy that leverages multi-resolution up- sampling with visual representations from large- scale foundation model. SAM2Act achieves a tfate-of-the-ar art average success rate of 86.8 % across 18 tasks in the RLBench benchmark, and demonstrates robust generalization on The Colosseum benchmark, with only a 4.3% % per- formance gap under diverse environmental per- turbations. Building on this foundation, we pro- pose SAM2Act+, a memory-basee architecture inspired by SAM2, which incorporates memory bank, encoder, and attention mechanism to enhance spatial memory. To address the need for evaluating memory-dependeer tasks, we introduce MemoryBench, novel benchmark designed to assess spatial memory and action recall robotic manipulation. SAM2Act+ achieves an average success rate of 94.3% % memory-based tasks in MemoryBench, significantly outperforming existing approaches and pushing the boundaries of memory- based robotic systems. Project page: nam2act.githuble
The world in which we live is diverse and constantly chang- ing, encompassing a wide variety of objects, scenes, and en- vironmental conditions. Consider the seemingly simple task of following a recipe when cooking: we can seamlessly per- form the action of picking it up and sprinkling it into the pan, recognize salt even if it comes in different types of container, and remember whether we have already added salt. Humans excel in such environments because they can interact with their surroundings to achieve specific goals, generalize to unseen scenarios, and retain knowledge from past experi- ences (Smith & Gasser, 2005). These interaction, generalization, and memory- serve guiding principles for developing robotic systems capable of operat- ing in similarly complex environments.
multitask
Significant progress has been made in robotic manipulation through prior work. Early methods, such as the Transporter Network (Zeng et al., 2021) and CLIPoort (Shridhar et al., 2022), demonstrated effective 2D action-centric manipu- lation but were limited in their ability handle spatially complex tasks. More recent approaches, such as PerAct (Shridhar et al., 2023) and RVT (Goyal et al., 2023), have pushed toward 3D-based manipulation. PerAct employs a multitask transformer that interprets language commands and predicts keyframe poses, achieving strong results across variety of tasks. RVT builds on this foundation by adopt- ing 2.5D representation, improving training efficiency and inference speed. Its successor, RVT-2, further enhances per- formance with coarse-to-fine strategy, increasing precision for high-accuracy tasks. Despite these advances, important challenges remain, including improving multitask perfor- mance, enhancing generalization to novel environment con- figurations, and integrating memory mechanisms for tasks requiring episodic recall.
'Equal advising Uniivessity of Washington Univeersidad Católica San Pablo "NVIDIA "Allen Institute for Ar- tificial Intelligence.
We introduce SAM2Act, multi-view robotics transformer- based policy that enhances feature representation by inte- grating multi-resolutionn upsampling with visual embeddings from large-scale foundation models. Built on the RVT-2 multi-view transformer, SAM2Act achieves strong multitask success and generalization. Building on this foundation, we
Correspondence to: Haoquan Fang <hqfang@ es. washington.cduu, Jiafci Duan <du- anji @cs. wrssswashinnto..eee edu..
Proceedings of the 4dnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
Training Data
Precision
Tar keep.
Pick up plan
Truan
Deploy
Memory
Generalization
Open LEpeeS
Figure 1. SAM2Act is multi-view, language-condiinee conditioned behavior cloning policy trained with fewer demonstrations, Given language instruction, can execute high-precision tasks, such as turning the tiny knob on lamp. also generalizes various environmental variations, such changes in lighting conditions. Through further training with our proposed memory architecture, it now evolves into SAM2Act+, which now capable solving tasks that require implicit spatial memory-such as remembering where robot previously stored the pliers, depicted in the above figurc.
introduce SAM2Act+, which incorporates a memory-based architecture inspired by SAM2's approach. Using memory bank, an encoder, and an attention mechanism, SAM22Act enables episodic recall to solve spatial meeoory-deenndee manipulation tasks. We evaluate SAM2Act and SAM2Act+ using MemoryBench, a new benchmark suite that tests policies' spatial memory capabilities and the ability to re- tain and recall past actions. SAM2Act+ achieves an average success rate 94.3% across all tasks on GemoryBence Be,ch, with an average accuracy of 94.3%, outperforming next high- est baseline by huge margin of 39.3%. Furthermore, we assess the generalization capabilities of SAM2Act on The Colosseum (Pumacay et al., 2024), a benchmark designed to test robotic manipulation under various environmental perturbations. SAM2Act demonstrates robust performance on The Colosseum with an average decrease of 4.3% across all perturbations, highlighting its ability to generalize effectively in diverse and challenging scenarios. Lastly, our approach outperforms the baseline methods in real-world evaluations while exhibiting comparable generalization and spatial memory capabilities.
cost. However, they depend on pretrained image encoders and fail in tasks requiring high precision, robust spatial inter- action, resilience to environmental and camera variations (Pumacay et al., 2024). Recent work addresses these limi- tations with 31> perception. Methods like PolarNet (Chen et al., 2023), M2T2 (Yuan et al., 2023), and Manipulate- Anything (Duan et al., 2024) reconstruct point clouds, while C2F-ARM (James & Abbeel, 2022) and PerAct (Shridhar et al., 2023) use voxel-based 3D representations. Act3D (Gervet al., 2023) and ChainedDiffuser (Xian et al., 2023) adopt multi-scale 3D features. RVT (Goyal et al., 2023) introduces 2.5D multi-view images for faster training, re- fined by RVT-2 (Goyal et al., 2024) with a coarse-to-fine architecture for improved precision. Our work, SAM2Act, combines RVT-2's spatial reasoning with enhanced virtual images from the SAM2 visual encoder, achieving high pre- cision and generalization across diverse tasks.
2.2. Visual Representations for Robot Learning
Robotics research heavily relies on visual representations from computer vision process high-dimensional inputs and improve policy learning. Visual representations integrated into robot learning through pre-training (Majum- dar et al., 2023; Ma et al., 2022; Nair et al., 2022), co- training (Laskin et al., 2020a; Yarats et al., 2021; Laskin et al., 2020b; Shang et al., 2024), or frozen encoders (Shah & Kumar, 2021; Wang et al., 2022a; Zhang et al., 2024a), all of which effectively support policy training. These repre- sentations also enhance invariance, equivariance, and out- of-distribution generalization (Wang et al., 2022b; Pumacay et al., 2024: Dasari et al., 2023). SAM-E (Zhang et al., 2024a) demonstrates the use of a pre-trained SAM en- coder for robotic manipulation by leveraging image em- beddings for policy learning. Expanding on this, our ap- proach employs the SAM2 visual encoder to generate im- age embeddings for robotic transformers and utilizes its multi-resolution features to improve convex upsampling for next-action prediction.
In summary, this work makes three key contributions. First, we introduce novel model formulation that leverages visual foundation models to solve high-precision, memory- dependent manipulation tasks. Second, we propose MemoryBench, a evaluation benchmark for assessing spa- tial memory in behavior cloning models. Finally, we present empirical results and insights on the model's 's per- formance across both simulation and real-world tasks.
2. Related Work
2.1. 3D-based Robotic Transformer for Manipulation
2D-based methods (Zhao et al., 2023; Chi et al., 2023; Zeng et al., 2021; Brohan et al., 2022; Shridhar et al., 2022) are effective for simple pick-and-place tasks due to fast train- ing, low hardware requirements, and minimal computational
2

---

SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
2.3. Memory in Robotics
model in memory-based tasks. These principles guided the development of our spatial memory-based tasks.
Memory is a fundamental component of human cognition, and equipping generalist robotic agents with episodic and se- mantic memory is crucial for enabling them to perform com- plex tasks effectively (Jockel et al., 2008). Early research on memory in robotics primarily addressed navigation tasks, re- lying on semantic maps that were often constrained in scope (Henry et al., 2012; Bowman et al., 2017; Chaplot et al., 2020). Other work explicitly model the memory and its rep- resentation for rohot cognitive architecture (Peller- Konnrdd et al., 2023). Recent advancements leverage representations derived from vision-language models (VLMs) and Large Vision Models (LVMs), utilizing voxel maps or neural fea- ture fields to encode, store, and retrieve information (Huang et al., 2024; 2023; Duan et al., 2024; Liu et al., 2024). Alter- native methods represent semantic memory for manipulation tasks using Gaussian splats encode spatial information (Kerbl et al., 2023; Shorinwa et al., 2024). In contrast, our approach draws inspiration from the framework of Partially Observable Markov Decision Processes (POMDPs) (Lauri et al., 2022), incorporating memory directly into the training process. By integrating spatial memory from past actions into the agent's belief state, we enhance the robustness and adaptability of learned policies.
3.2. Spatial Memory-based Tasks
MemoryBench extends the RLBench simulator to pro- vide scripted demonstrations for three spatial mem- ory tasks: reopen.drawer, putcklock.baco and bearrange.block
Each task is designed to evaluate a specific aspect of spatial memory and adheres to the prin- ciples outlined in Section 3.1. To introduce complexity, these tasks include two to four variations and additional steps-suse as pressing a button mid- sequence -that dis- rupt the Markov property. This forces the agent to rely on memory rather than solely on immediate observations.
The pooenndraave task evaluates the agent's ability to recall 3D spatial information along the z-axis. Initially, one of three drawers (top, middle, or bottom) is open. The agent must close the open drawer, press a button on the table, and then reopen the same drawer. After the button is pressed, all drawers are closed, and the scene becomes visually indistinguishable, requiring the agent to use mem- ory to identify the correct drawer. This task tests the agent's 's ability to recall spatial states over a temporal sequence. The wat.block.bbce task tests the agent's 's ability to re- member 2D spatial information on the x-y plane. Four red patches are placed on table, with a block initially posi- tioned on one of them. The agent should move the block to the center of the patches, press button, and return the block to its original position. The agent must rely on its memory of the block's 's initial location to succeed, demonstrating its capability to encode and retrieve 2D spatial information.
3. MemoryBench: A Memory Benchmark for Robotic Manipulation
We introduce MemoryBench, a benehmark designed to systematically evaluate the spatial memory capabilities of robotic manipulation policies. In Section 3. we begin by outlining the logic and rules behind task design. We will then describe the tasks we have developed in Section 3.2.
The rearrange rearrange.blokk block task evaluates the agent's s ability to perform backward reasoning by recalling and reversing prior actions. Initially, one block is placed on one of two red patches, while the other patch remains empty. A second block is positioned at the center of both patches. The agent must move the second block to the empty patch, press but- ton, and then relocate the first block off its patch. Success- fully completing this task requires the agent to determine which block to move without having interacted with the correct one in previous actions, thereby testing its capacity for backward spatial memory reasoning. These tasks collec- tively evaluate both forward and backward spatial reasoning across 3D (z-axis) and 2D (x-y plane) spaces. By intro- ducing non-Markovian elements, they emphasize the need for memory representations to solve complex sequential decision making problems (more details in Appendix G).
3.1. Task Design
Unlike standard RLBench tasks (James et al., 2020), many of which involve long-horizon scenarios, our tasks are specifically designed to require spatial memory. Without such memory, the agent would be forced to rely on random actions. To create these tasks, we intentionally violate the Markov assumption, which states that in Markov Decision Process (MDP), the next observation depends solely on the current observation and action:
P(oc+1 01, 2,,4,,..,,,,,. a²) = P(o4+1 or, as).
This assumption implies that knowing only on and a is sufficient to predict oc+1- However, in our tasks, we de- sign scenarios where two distinct action histories lead to the same observation or, but require different subsequent actions. This forces the agent to recall which action history led to o, to perform the correct next action. Furthermore, we standardized the language instructions to prevent unin- tentional leakage of spatial information that could aid the
decision-making
4. Method
Our method, SAM2Act, enables precise 3D manipulation with strong generalization across environmental and object- level variations. Building upon the RVT-2 framework
3

---

SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
Memory Bench
"Ukee the dieecr, thes reseent the pernially parking beraa herreas..
The tiac so caree and baroea." Real-world Tasks
"Mare be blxc on paic to the copty
pems hace carved off the
Turn the lamp
Maah buttons
Stock Mocks
odl Phak the batton
kemp
the
ooaage
Pash the button that previcualy had the bock from of twkee
Figure 2. Simulation and Real Tasks. We demonstrate the effectiveness of SAM2Act+ in solving memory- -based tasks by evaluating it against baselines on the three benchmark memory tasks (shown at the top). Additionally, we validate our approach using Franka Panda robot four real-world tasks (shown at the bottom), including tests under out-of-distribution perturbations.
(Goyal et al., 2024), SAM2Act introduces key architec- tural innovations that enhance visual feature representation and spec-ipe reasoning. The architecture reconstructs point cloud of the scene, renders it from virtual cameras at orthogonal views, and employs two-stage multi-view transformer (coarse-to-fine) to predict action heatmaps. The coarse branch generates zoom-in heatmaps to localize re- gions interest, while the fine branch refines these into pre- cise action heatmaps. SAM2Act leverages the pre-trained SAM2 encoder (Ravi et al., 2024) extract multi- rustutons image embeddings, which are further refined through the multi-resolution upsampling technique to predict accurate translation heatmaps with minimal information loss. To address tasks requiring spatial memory, SAM2Act+ extends the SAM2Act architecture by incorporating memory-based components. These include Memory Bank, Memory En- coder, and Memory Attention, enabling the model to encode historical actions and condition current observations. This memory-bane based policy enhances the agent's ability to pre- dict actions based past contextual information, signifi- cantly improving performance in tasks that require sequen- tial decision-making.
4.1. SAM2Act: Multi-Resolution Upsampling for Enhanced Visual Feature Representation
A distinctive feature of SAM2Act is the incorporation of the SAM2Act Module into the manipulation backbone for training, illustrated in Figure 4. The coarse and fine SAM2Act Modules share the same architecture, with the fine branch generating additional features to predict actions beyond translation, while the coarse branch focuses exclu- sively on translation. Point-cloud representations are re- constructed from raw image inputs, and virtual images are generated from three viewpoints using virtual cameras. In- stead of directly inputting these images into the multi-view transformer, their RGB channels duplicated and pro- cessed by the SAM2 (Ravi et al., 2024) image encoder, which produces object-centric nulti-resslluion embeddings. These embeddings, generated at three resolution levels, are combined with virtual images containing RGB, depth, 3D translation coordinates, and language instructions before being fed into the multi-view transformer. Details of how adapt the MVT can be found in Appendix A.
To adapt the SAM2 image encoder to our domain, we fine- tune it using Low-Rank Adaptation (LoRA) (Hu et al., 2021) )) with a default rank of 16, which enables domain adaptation with minimal computational cost while main- taining model efficiency. Additionally, to fully leverage the multi-resolution embeddings produced by the SAM2 image encoder, introduce multi-resolutin upsampling method. This method uses the embeddings as auxiliary
In the following sections, we detail the SAM2Act architec- ture (Section 4.1), including its multi-resooution upsampling mechanism (Figure 4). We also present the SAM2Act+ tension, which integrates memory-bart based components for solving spatial memory tasks (Section 4.2).

---

SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
SAM2Act for General Manipulation Tasks
Stage1 1
SAM2Act Module (Coarse)
Input
Outpuu Stat
6AV0 Erceder
(Ccered
6 Zoomin
Dkads faree Wildiir
Mexide prent
Oripwe tait
yut calccie
SAM2Act+ for Memory- Specific Tasks
Stage 2
SAMZAct- Module (Coarse)
Frozen
Finetuned
Input (Sequentia)
Output
Fore Orodd
Anteetion
Prddis Heamap
SAMO Prage
SAMDAAC Modka
Crea Take Doud
Prodect beet
Mensar
Mernary
snin for Tuu
Figure 3. Overview of the SAMZAct (top) and SAM2Act+ (bottom) architectures. The SAM2Act architecture leverages the SAM2 image encoder to generate prompt-conditioned, multi-resolution embeddings, fine-tuned with LoRA for efficient adaptation to manipulation tasks. A niew transformer aligns spatial coordinates with language instructions, while a cascaded multi-rcsolution upsampling mechanism refines feature maps and generates accurate translation heatmaps. SAM2Act+ extends this architecture by incorporating memory-based components, including the Memory Encoder, Memory Attention, and Memory Bank, into the coarse branch. These components enable memory-driven reasoning by processing historical heatmaps and integrating prior observations, allowing the agent to predict actions based on stored contextual information. Observations are reconstructed into point clouds, rendered into three virtual images, and lifted into 3D translation points, enabling precise spatial reasoning across both architectures.
Insouctie
Mutti Recolution Upcampiing
Muli: Vio Trandomer
Eacoder
Polnt Cood
vitu altroors
Figure 4. SAMZAct Module and multi-resolution upsampling mechanism. A cascade of three convex upsamplers processes feature maps at increasing resolutions, integrating multi-resolution embeddings from the SAM2 image encoder through elementwise addition and layer normalization. The upsamplers progressively refine features, doubling spatial dimensions cach stage, generate accurate translation heatmaps while capturing fine-grained spatial details critical for manipulation tasks.
inputs to enhance the generation of translation heatmaps, therehy improving spatial precision and overall system per- formance. The multi-resolution upsampling mechanism, also detailed in Figure 4, leverages cascaded convex up- samplers to progressively refine feature maps across resolu- tions. Let X E BBCx 'H' xw denote the feature maps at stage and E' E X H''''' the corresponding multi- resolution embedding from SAM2. Also let U(.) denote the upsampling operator that doubles the spatial dimensions. The feature maps are updated at each stage as follows:
operator U is defined as:
BB xH'xw" Bx( (C (2x)
At each stage, the output of the upsampler is combined with the corresponding multi-resolltion embedding E' from the SAM2 encoder, ensuring alignment between the multi- resolution features and the decoder's spatial refinement pro- cess. A layer normalization step follows each addition sta- bilize training and maintain feature coherence. This results in direct integration of the embeddings into the translation heatmap generation process. The cascading structure refines features across multiple resolutions, capturing ine-grained spatial details critical for manipulation tasks.
x LayerNorm (UUX)))) @ E'),
where ( represents element- wise addition. The upsampling
5

---

SAM2Act:
Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
Algorithm 1 Forward Pass of SAM2Act+ Module
cuses on generating heatmaps that provide richer contextual information for recalling past actions. The fine branch, in contrast, primarily emphasizes small objects or localized regions, which typically contain less information relevant to memory-based tasks.
Initialize: Number of steps N, number of views V. empty memory bank Q with V separate FIFO queues, size of memory bank M, input X
for i 1 to N do
for j = 1 to do
Training. To train SAM2Act+, we fine-tune the coarse branch by integrating the three memory components (and train them from scratch) with the multi-resolutionn up- sampling module. During fine-tuning, consecutive ac- tion keyframes are sampled as input, training the multi- resolution upsampler to predict new translations conditioned on memory. The memory components function similarly to their implementation in SAM2 for object tracking, with one key distinction: the input to the Memory Encoder. Instead of using image embeddings from the SAM2 image encoder, input feature embeddings generated by the multi-view transformer (not conditioned by memory). This adapta- tion ensures that memory encoding incorporates niew information while maintaining independence in handling stored representations. Virtual images are treated indepen- dently during memory encoding and attention, with each view's memory encoded separately. Feature embeddings from each view attended using their corresponding stored memories, preserving spatial and contextual align- ment while leveraging fused multi-view information. This structured approach prevents cross-view interference and enhances the model's ability to reason over sequential tasks. The memory-based forward pass for SAM2Act+ is outlined in Algorithm 1. By incorporating the memory mechanism, SAM2Act+ enhances performance in scenarios requiring long-term reasoning, enabling the agent to make informed decisions based on historical context.
Get embeddings Erow from MVT T Tm (X,)
Retrieve past memories Mota from Qij
Get memory-conditioned embeddings &mem from Memory Attention m Moid)
Predict translation heatmap H with Upsampling U (Emem)
Encode new memory Mnew using Memory Encoder Emem (H. Erow)
Store new memory Qlil + Q61 U [Mnew)
if [Q[]]| = M then
Q - 2:)
end if end for end for
4.2. SAM2Act+: Action Memory Architecture for Improved Spatial Awareness in Past Observations
To extend the SAM2Act architecture (Section 4.1) with memory-based capabilities inspired by SAM2, we intro- duce SAM2Act+, a task-specific variant designed for solv- ing memory-based tasks. SAM2Act+ integrates the three core memory components from SAM2-Memory Atten- tion, Memory Encoder, and Memory Bank into the coarse branch of SAM2Act. Originally developed for object track- ing in SAM2, these components are adapted to align with the needs of SAM2Act+, enabling the agent retain prior actions and observations for sequential decision-making. In SAM2, the Memory Encoder processes predicted ob- ject masks, while the Memory Attention module fuses im- age embeddings with positional information from previous frames. SAM2Act+ adopts a similar structure: the predicted heatmaps, which serve as binary indicators of spatial posi- tions in the image, function analogously object masks. This conceptual alignment ensures seamless integration of memory mechanisms, allowing the agent to leverage stored information to predict subsequent actions based on histori- cal context. A detailed description of the Memory Attention and Memory Encoder modules can be found in Appendix A.
5. Experiments
We study SAM2Act and SAM2Act+ in both simulated and real-world environments. Specifically, we are interested in answering the following questions:
$ 5.2
How does SAM2Act compare with state-of-the-art 3D manipulation policies?
$ 5.3
Can SAM2Act generalize across object and environ- mental perturbations?
$ 5.4
Can SAM2Act+ solve spatial memory- based tasks that other baselines cannot?
$ 5.5
How well does SAM2Act and SAM2Act+ perform on real-world tasks?
Architecture. The SAM2Act+ architecture is illustrated in Figure 3. After pretraining SAM2Act in Stage 1, we freeze the SAM2 image encoder and the multi-view transformer in the coarse branch, as these components effectively generate robust embeddings for multi-view images in manipulation tasks. We also freeze the entire fine branch, given its proven ability to predict fine- -ree-grrdnd actions accurately. The reason why we only fine-tune tune the coarse branch because it fo-
5.1. Experimental Setup
We benchmark SAM2Act in both simulated and real-world environments. The simulated environments serve as a con- trolled platform to ensure reproducible and fair comparisons. The real-world experiments demonstrate the applicability
6

---

SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
Table Multi-Task Performance on RLBench. We evaluate 18 RLBench tasks (James et al.. 2020), reporting success rates across all
tasks among 3D keyframe- based behavior cloning (BC) policies. We report stats of 4 evaluations for SAM2Act. Our method, SAM2Act, outperforms all baselines, achieving significant performance margin of 5.8% over RVT-2 (Goyal al., 2024), the prior state-of of the-ahtr art 3D
keyframe-basee BC policy. Against all existing approaches, SAM2Act remains the state-of-the-art. See full comparisons in Appendix C.
Method
Aug. Suecoss
Avg. Rank
Close Jar
Drag Stick
Insert Peg
Mcat off Geill
Open Drawer
Place Cups
Place Wine
Push Buttons
PerAct (Stridhar et al., 2023)
49.4 4.3
4.6
55.2 4.7
89.6 4.1
5.6 4.11
70.4 = 2,0
88.0
2.4
44.8 7.8
1. 3.0
RVT (Goyal et al., 2023)
62.9 + 3.7
3,6
52.0 2.5
99.2 1.6
11.2 3.0
25
712 6.9
40 2.5
5.2
100.0 + 0.0
RVT-2 (Goyaa et 2024)
81.4 3.1
1.9
100.0
1.
40.0 0.0
1.7
3.5
108.0 1: 0.0
SAM- (Zhing st, 2024a)
70.6+07
2.6
82.4+336
100.0
4.6
952233
98.+
944-46
1000
SAMIAci (Oum)
86.8 :: 0.5
1.8
99.0-1:2.0
2.0
84.0 : 5.7
2.3
83.0 6.0
47.0:::00 6:60
93.0=3.8
100.0 :: 0.0
Method
Put in Caphoard
Put in Drawer
Put In Safe
Screw Bolh
Slide Block
Sort Shape
Stack Hlocks
Stack Cupe
Sweep to Dustpan
Tarn Tap
ParAct (Sividhar et al., 2023)
4..0
4..2++4.7
$4.0 3.6
17.6 2.0
74.00
13.0
16.8 4.7
26.4 + 3.2
2.4 + 2.0
52.0 0.0
88.0 + 4.4
RVT (GGoyal 2023)
3.2
5.7
4:
48.0 5.7
2.5
28.5
26.4
=200=00 0.0
93.6
RVT-2 (Giayal al., 20244)
66.0+4.
96.0++00 0.0
96.0++22 2.8
$8.0 4.9
92.0
35.0 7.1
80.0 2.8
60.0+59
100.0
99.0 1.7
SAM-E (Zhnng al., 20244)
64.0 2x
22.0
95.2 3.3
78.4 3.6
95.2= 1.8
34.4 6.1
26.4 4.6
0.0 + 0.D
100.0 + 1.0
100.0 + 0.0
SAMZAct (Ours
75.0 + 3.8
99.0 2.0
98.0 + 2.3
89.0 2.0
86.0 :0:0:44 4.0
64.0 4.6
76.0 86
78.0 4.0
99.0 2.0
96.0 5.7
of the method to real- settings. Section 5.1 details our experimental setup and outlines the evaluation methodology. Training details can be found in Appendix B.
cies are evaluated four times to obtain standard deviations. Tasks details can be found in Appendix F and Appendix G.
3D Baselines. We benchmark SAM2Act and SAM2Act+ against the current ofate-of-the-art 3D next-best-pose predic- tion model, RVT-2. RVT-2 is multi-view roboties trans- former that leverages coarse-to-fnne approach on the con- structed point cloud to predict the next best action heatmap. We also compare with RVT (Goyal et al., 2023), PerAct (Shridhar et al., 2023), and SAM-E (Zhang et al., 2024a).
Simulation Setup. All simulated experiments were con- ducted in the CoppeliaSim environment via PyRep, using 7-DoF Franka Emika Panda robot in tabletop setting. Ob- servations were captured from five RGB-D fameras-front, left shoulder, right shoulder, overhead and wrist-each at 128 px x 128 px. The robot receives a keyframe specify- ing translation and quaternion orientation and utilizes an OMPL-based motion planner to move to the target pose.
5.2. Performances Across 18 RLBench Tasks
Real-robot Setup. We validate SAM2Act in real-world scenarios using Franka Emika Panda robot with Robotiq 2F-85 gripper and a exocentric Intel RealSense D455 depth sensor (more in Appendix I). We study four manipulation tasks, aligning three with RVT-2 for comparison and in- troducing a new memory- -aaed task. We use the software stack as in (Grotz et al., 2024). For each task, we collect 10-15 demonstrations via kinesthetic teaching and scripted execution with scene and object variations. As in Figure 2, we evaluate SAM2Act against RVT-2 for tasks (c) and SAM2Act+ for memory task (d). Each task undergoes 10 in-distribution and 10 out-o--distributiin trials, including environmental perturbations, measuring total success.
Table 1 compares SAM2Act with prior keyframe based 3D BC methods on the RLBench benchmark. Overall, SAM2Act achieves an average success rate of 86.8 80.5, surpassing the previous best (RVT-2) by 5.4%. closer look at individual tasks reveals that SAM2Act ranks first in 9 out of 18 tasks and remains highly competitive in 7 others, coming within one successful attempt or 4% of the best performance. These tasks include Close Jar, Drag Stick, Meat Off Grill, Place Wine, Screw Bulb, Sweep to Dustpan, and Turn Tap. The largest margin of improvement occurs in Insert Peg, where SAM2Act exceeds RVT-2 by 44 % (approximately 2.1x), and in Sort Shape, where it out- performs RVT-2 by 29%. Both tasks require precise ma- nipulation, underscoring the effectiveness of SAM2Act's mustuurrsooltton upsampling strategy. These results estab- lish SAM2Act as a leading policy for complex 3D tasks, highlighting its ability to handle high-srecision manipula- tions an area where prior methods have struggled. Ablation studies are performed on SAM2Act in Appendix E.
18 RLBench & MemoryBench Tasks. To evaluate the general performance of SAM2Act and the memory capabil- ities of SAM2Act+, we conducted simulation experiments on two benchmarks: subset of 18 tasks from RLBench and MemoryBench. RLBench is a standard multi-task manipulation benchmark, from which we selected 18 tasks well-studied in prior work. MemoryBench is a curated set of three tabletop manipulation tasks in CoppeliaSim that require the trained policy to have both semantic and spatial memory of past scenes and actions. In both benchmarks, each task is defined by a language instruction with 2-60 variations (c.g., handling objects, locations, and colors). We collected 100 demonstrations per task for training and held out 25 unseen demonstrations per task for testing, All poli-
5.3. Semantic Generalization across Tasks
The results evaluated in Section 5.2 were obtained by train- ing and testing models within the same environment. How- ever, to truly assess generalization performance, policies must remain robust against both environmental and object- level perturbations. We therefore trained SAM2Act and the baseline methods on 20 tasks from The Colosseum
7

---

SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
Table 2. The Colosseum results. Task-average success rate percentage change for SAM2Act and other baselines across 13 perturbation factors from The Colosseur, relative to evaluations without perturbations. Results of 3 evaluations are reported for all models. Our approach, SAM2Act, demonstrates the lowest average percentage change across all perturbations, with a minimal drop of -4.3%3.6%% highlighting its robustness in handling various environmental and object-level perturbations. The full result table is shown in Appendix D.
Method
Average
MO-Color 1
RO-Color 1
MO-Texture I
RO-Texture 1
MO-Size I
RO-Size
RVT-2 (Goyal et 2024)
29.55+2.8
.20.7+100
-118-08
-1333+4.
-11.4+3.7
-132=3.1
-17.7+0.1
SAMZAct SAM)
2..721..
26.110..7
15.72-29
15,0433.3
16.5662
1187219
1198
SAM2Act (wln Multi-res Input)
-19.1+4.5
-15.54664
-13.5=46
-20400.5
-16.6+6.1
-21,3-755
-126477
SAM2Act (Ours)
4..443.6
.1142.5
-07772
33.422
24.7246.1
-15.==5.0
0,9268
Method
Light Colue 1
Table Culor 1
Table Texture
Distractor
Backgoound Texture
Camera Pose
All Pecturbations
RVT-2 (Goyal et al., 2024)
1.3
26654444
.1462444
4945.3
4.4+4.0
-19,52288
-779+1177
SAMEAct (SAM2 SAM)
-16.341.2
23.515,3
-12333.1
0612.
-.4432
20.7-122
-7.5.225
SAMZAst (wlo Mulli-res Input)
-7.133.6
-18316.1
-17,5133
-.6.1..55
-5.71...55
-19.1:455
-73.8 12:2
SAMZAzt (Ours)
4,5.4.4
11:255
-3.75.52
17417
-1.52277
.4333.5
-58.31444
benchmark and tested them under 13 different perturbation categories over three runs. SAM2Act exhibits the small- est performance drop compared to the baselines, with average decrease of 4.3% (standard deviation of 3.59%). No- tably, it proves particularly robust to environmental pertur- bations- - such as changes in lighting, table color/texture, the addition of distractors, and even camera pose- - while also maintaining competitive performance under object-level per- turbations (see more analysis in Section F.2).
5.5. Real-robot Evaluations
Table 4 presents our real-world experiment results, where our method achieves a 75% task success rate, compared to 43% for RVT-2 SAM2Act significantly outperforms the baseline in high-precision tasks (60% vs 0%). It excels in memory-based tasks, such as (d) Push the same button, which requires recalling the button's previous location. Here, SAM2Act achieves 70% success, while RVT- 2, relying on random guessing, scores 40%. We also test models' generalization against perturbations like lighting changes. distractors, and position variations. Additional details are in the Appendix I, with real-world rollout videos available on our project website.
5.4. Performance on MemoryBench
In Table 3, we evaluate SAM2Act+ against SoTA 3D BC model, RVT-2 on MemoryBench, training all models in single-task setting to isolate memory-related challenges opening the wrong drawer rather than unrelated mid- task failures). This setup ensures that performance differ- ences stem from memory capabilities. For random agent, the expected success rates are determined by the number of possible choices per task: 33% for zeopen.drave (three drawers), 25% for put olok back (four patches), and 50% for zearrange_ble variations in task complexity, fixed training data, and imbal- anced task distributions lead to slight deviations from these baselines. Our proposed memory-based model, SAM2Act+, demonstrates a strong understanding of spatial memory, achieving an average success rate of 94.3% across all tasks. outperforms SAM2Act (without memory) by huge margin of 39.3% on MemoryBench, highlighting the sig- nificant impact of explicit memory modeling.
Table 4. Real-world results. We compare RVT2 against SAM2Act for the first three tasks and SAM2Act+ on the last real-world tasks (indicated with *). evaluating performance both in-distribution and out-of-distriattion during test time.
In-Distribution
Out-Distribution
Task
RVT-2
SAM2Aci
RVT-2
SAM2Act
(a) tum on the lamp
0/10
0/10
6/10
(two blocks). However,
(b) push button sequeace
4/10
9/10
1/10
9/10
(c) stack cubes
8/10
8/10
3/10
3/10
(d) push the same button
4/10
2/10
6/10
6. Conclusion & Limitation
We introduce SAM2Act, a multi-view, language- conditioned behavior cloning policy for 6-DoF 3D manipulation manipulation, enabling high-precision manipulations while generalizing effectively to unseen perturbations. Building on this foundation, we propose SAM2Act+, a memory-basee based multi-view lunguage-oonitioone robotic transformer- based policy that equips the agent with spatial memory awareness, allowing it to solve spatial memory- based tasks. While both SAM2Act and SAM2Act+ achieve SOTA performance across multiple benchmarks, challenges remain in extending them to dexterous continuous control. Additionally, SAM2Act+ relies on a fixed memory window length, which differs from task to task, limiting its adaptability to tasks of varying length. Despite these challenges, we believe SAM2Act+ is an important step towards memory- generalist manipulation policies.
Table 3. Performance on MemoryBench. We report the success rates for the three spatial memory tasks in Memo Rench.. Our method, SAM2Act+, significantly outperforms all haseline meth- ods that lack an explicit memory mechanism, achieving an average improvement of 37.6% across all three tasks. Note that there is an update with Memor TeeooryBence Bench,, see more in Appendix H.
Matheds/ Toocs
Arg. Sussse: (0) Ruseen Daver
(b) Pu Black Back (c) Reamage Bluck
RVI-3
$4.0+ + 53
60.0 00
50.0 +00+:2 2.1
500 3.
SAMIACt (Ours)
55.0 2: 24.3
:0:000
35.0 3.8
820 00:222 23
SAMZAct (Cuss)
94,3 2 9.0
$40 0.0000
100.0 = 0.0
910 20
8

---

SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
Acknowledgements
Dasari, S., Srirama, M. K., Jain, U., and Gupta, A. An unbiased look datasets for visuo-motor pre-training. In Conference on Robot Learning, pp. 1183-1198. PMLR, 2023.
Jiafei Duan supported by the Agency for Science, Tech- nology and Research (A*STAR) National Science Fellow- ship. Wilbert Pumacay is supported by grant 234-2015- FONDECYT from Cienciactiva of the National Council for Science, Technology and Technological Innovation (CONCYTEC-PERU), This project is partially supported by Amazon Science We would also like to thank Winson Han from the Allen Institute for Artificial Intelligence for helping with the figure and icon design, and Jieyu Zhang from the University of Washington for assisting with the design of the model architecture and training pipeline.
Duan, J., Yuan, W., Pumacay, W., Wang, Y. R., Ehsani, K., Fox, D., and Krishna, R. Kanipulate-anythinn Au- tomating real-world robots using vision-language models. arXiv preprint arXiv:2406. 18915, 2024.
Garcia, R., Chen, S., and Schmid, C. Towards general- izable vision-language robotic manipulation: A bench- mark and Ilm-guided 3d policy, ArXiv, abs/2410.01345, 2024. URL ttttps:// www.pp..saaaa...... commantssccollas
Impact Statement
Gervet, T., Xian, Z., Gkanatsios, N., and Fragkiadaki, K. Act3d: Infinite resolution action detection transformer for robotic manipulation. arXiv preprint arXiv:2306.17817. 2023.
This paper presents a novel model architecture for robotic manipulation, integrating a visual foundation model with a memory-baae based design. Our approach has the potential en- able general-purpose household robots capable of assisting with daily tasks, contributing to broader societal benefits. We do not anticipate any direct ethical or societal concerns arising from this work.
Goyal, A., Xu. J., Guo, Y., Blukis, V., Chao, Y.-W., and Fox, D. Rvt: Robotic view transformer for 3d object manipulation. In Conference on Robot Learning, pp. 694- 710. PMLR, 2023.
References
Goyal, A., Blukis, V., Xu, J., Guo, Y., Chao, Y.W.,, and Fox, D. Rvt-2: Learning precise manipulation from few demonstrations, arXiv preprint arXiv:2406.088. 085445 2024.
Bowman, S. L.,, Atanasov, N., Danilidis, K., and Pappas, G. Probabilistic data association for semantic slam. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 1722-1729. IEEE, 2017.
Grotz, M., Shridhar, M., Chao, Y.-W., Asfour, T., and Fox, D. Peract2: Benchmarking and learning for robotic bi- manual manipulation tasks. In CoRL 2024 Workshop on Whole-body Control and Bimanual Manipulation: Appli- cations Humanoids and Beyond, 2024. URL https:
Brohan, A., Brown, N., Carbajal, J.. Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A.,
Hsu, J., et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.068/1 2022.
/ // //oenreview. net http://rrmminnnn0000aaaa-- Z00pttt
Guhur, P.-L., Chen, S., Pinel, R. G., Tapaswi, M., Laptev, I., and Schmid, C. Instraction-driven history-aware poli- cies for robotic manipulations. ArXiv, abs/2209.04899, 2022. URL https: /ppi. semanticschola. org / II
Chaplot, D. Gandhi, D. P., Gupta, A., and Salakhutdinov, R. R. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems, 33:4247 2020.
Henry, P., Krainin, M., Herbst, E., Ren, X., and Fox, D. Rgb- d mapping: Using kinect-style depth cameras for dense 3d modeling of indoor environments. The international journal of Robotics Research, 31(5):647-663, 2012.
Chen, S., Garcia, R., Schmid, C., and Laptev, Polarnet: point clouds for language-guided robotic manipulation. arXiv preprint arXiv:2309. 2023.
Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., and Song, S. Diffusson policy: Visuomo- tor policy learning via action diffusion. The International Journal of Robotics Research, pp. 02783649241273668, 2023.
Heo, B., Park, S., Han, D., and Yun, S. Rotary position embedding for vision transformer. In European Conference on Computer Vision, 2024. URL https: //api. .emant www.choar..rg/oorooor..oe 268536717.
Dao, T. Flashattention-2: Faster attention with better paral- lelism and work partitioning. ArXiv, (et/2307008691 2023. URL https://api www.pp.seeaatnscchhh.ooooooooooe org/CoorpuSDD:25936.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang. S., Wang, L.,, and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.00 09685. 2021.
9

---

SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
Huang, H., Lin, F,, Hu, Y., Wang, S., and Gao, Y. Copa: General robotic manipulation through spatial con- straints of parts with foundation models. arXiv preprint DrXiv:2403.08248, 2024.
Langley, Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference Machine Learning (ICML 2000). pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Huang, W. Wang, C., Zhang, R., Li, Y., Wu, J., and Fei-Fei, L. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.059735 2023.
Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., and Srinivas, A. Reinforcement learning with augmented data. Advances in neural information processing systems, 33: 19884-19895, 2020a.
James, S. and Abbeel, P. Coarse-to-fine q-attention with learned path ranking. arXiv preprint arXiv:2204.01571, 2022.
Laskin, M., Srinivas, A., and Abbeel, Curl: Contrastive unsupervised representations for reinforcement learning. In International conference on machine learning, pp. 5639-5650. PMLR, 2020b.
James, S., Ma, Z., Arrojo, R., and Davison, J. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019-3026, 2020.
Lauri, M., Hsu, D., and Pajarinen, J. Partially observable markov decision processes in robotics: A survey. IEEE Transactions on Robotics, 39(1):21-44) 2022.
James. S., Wada, K., Laidlow, T.. and Davison, A. J. Coarse-to-fine q-attention: Efficient learn- ing for visual robotic manipulation via discretisa- tion. 2022 IEEF/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 13729-13738, 2021. URL https: semantccschola. htt/CorpusID:2356655
Liu, P., Guo, Z., Warke, M., Chintala, S., Paxton, C., Shafiul- lah, N. M. M., and Pinto, L. Dynamem: Online dynamic ppatio-semantte memory for open world mobile manipu- lation. arXiv preprint arXiv:241 1.04999, 2024.
Ma, Y. J., Sodhani, S., Jayaraman, D., Bastani, O., Kumar, V., and Zhang, A. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.00033 2022.
Jang, E., Irpan, A., Khansari, M., Kappler, D., Ebert, F., Lynch, C., Levine, S., and Finn, C. Bc-z: Zero-shot task generalization with robotic imitation learn- ing. ArXiv, abs/2202.02005, 2022. URL. https:
Majumdar, A., Yadav, K., Arnaud, S., Ma, J., Chen, C., Silwal, S., Jain, A., Berges, V.PP., Wu, T., Vakil, J., et al. Where are we in the search for an artificial visual cortex for embodied intelligence? Advances in Neural Informa- tion Processing Systems, 36:655-677 677,66777760 2023.
//api. www.nticscooolaatt.....occoc htt.:riiiicccolaaroooooooooooooo org/CorpusID: 237257594.
Jockel, S., Weser, M., Westhoff, D., and Zhang. J. Towards an episodic memory for cognitive robots. In Proc. of 6th Cognitive Robotics workshop at 18th European Conf. Arlificial Intelligence (ECAI), pp. 68-74. Citeseer, 2008
Nair, S., Rajeswaran, A., Kumar, V., Finn, C., and Gupta, R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12600 2022.
Ke, T.-W., Gkanatsios, N., and Fragkiadaki, K. 3d diffuser actor: Policy diffusion with 3d scene represen- tations. ArXiv, 10022402.10885. 2024. URL ht htsp:: //api. wwwanticccoollaatt..eeeccccc wttp:chhhlaar...cooooooooo Corpppuss. 267740176.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H. Q. Szafraniec. M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.-.. B., Li, S.WW.. Misra, I., Rabbat, M. G., Sharma, V., Synnaeve, G., Xu. H., Jégou, H., Mairal, J., Labatut, P., Joulin, A.. and Bojanowski, P. Dinov2: Learning robust visual features without supervision. ArXiv, abs/2304.07199, 2023. URL https: /api. semant. iommattcsccoll isschllr. http/CoopuID::25111110
Kerbl, B., Kopanas, G., Leimkühler, T., and Drettakis, G. gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139-1, 2023.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo. W.-Y,, Dollár, P., and Girshick, R. B. Seg- ment anything. 2023 IEEE/CVF International Con- ference on Computer Vision (ICCV), pp. 4903,4003, 2023. URL https:/ http:/ppp..eeaaaccccchooooooso/e org/CorppsSD::5575233
F., Kartmann, R., Dreher, C. R., Meixner, A., Reister, F., Grotz, M., and Asfour, T. A memory system a robot cogniitiv architecture and its implementation in armarx. Robotics and Autonomous Systems, 164:104415, 2023.
10

---

SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
Pumacay, W., Singh, I., Duan, J., Krishna, R., Thomason, J., and Fox, D. The colosseum: A benchmark for evaluating generalization for robotic manipulation. arXiv preprint arXiv:2402.0819/ 2024.
Wang, D., Walters, R., Zhu, X., and Platt, R. Equivariant q learning in spatial action spaces. In Conference Robot Learning, pp. 1713-1723, PMLR, 2022b.
Xian, Z., Gkanatsios, N., Gervet, T., Ke, T.-W., and Fragki- adaki, K. Chaineddiffuser: Unifying trajectory diffusion and keypose prediction for robotic manipulation. In 7th Annual Conference on Robot Learning, 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning trans- ferable visual models from natural language supervi- sion. In International Conference on Machine Learning, 2021. URL https: / sew..sseettcsccoo....... / ID: 2311991444
Yang, I.., Kang, B., Huang, 7., Zhao, Z., Xu, X., Feng. J., and Zhao, H. Depth anything v2. ArXiv, abs/2406.0944/ 2024. URL https: /appi http:///appp...ssssssseeeeeeeeee semantccscoolaar org/ (OorpusID:20444000
Ravi, N., Gabeur, V., Hu, Y.TT., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L.., al.
Yarats, D., Kostrikov, I., and Fergus, Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International conference on learning representations, 2021.
Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.0077 007144 2024.
Shah, R. and Kumar, V. Rrl: Resnet representation for reinforcement learning. arXiv preprint arXiv:2107.0338/ 2021.
Yuan, W., Murali, A., Mousavian, A., and Fox, D. M212: Multi-task masked transformer for object-centric pick and place. arXiv preprint arXiv:2311.0092. 2023.
Shang, J., Schmeckpeper, K., May, B. B., Minniti, M. V., Kelestemur, T., Watkins, D., and Herlant, L. Theia: Dis- tilling diverse vision foundation models for robot learning. 2024.
Zeng, A., Florence, P., Tompson, J., Welker, S., Chien, J., Attarian, M., Armstrong. T., Krasin, I., Duong, D., Sindhwani, V., et al. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference Robot Learning, pp. 747-747. PMLR, 2021.
arXiv preprint arXiv:2407.2011
Shorinwa, O., Tucker, J., Smith, A., Swann, A., Chen, T., Firoozi, R., Kennedy, M. D., and Schwager, M. Splat- mover: Multi-stage. open- open-vocabularyy robotic manipula- tion via editable gaussian splatting. In 8th Annual Con- ference on Robot Learning, 2024.
Zhang, J., Bai, C., He, H., Xia, W., Wang. Z., Zhao, B., Li, X., and Li, X. Sam-e: Leveraging visual foundation model with sequence imitation for embodied manipula- tion. arXiv preprint arXiv:2403. 19586, 2024a.
Zhang, X., Liu, Y., Chang, H., Schramm, L., and Boular- ias, A. Autoregressive action sequence learning for robotic manipulation. IEEE Robotics and Automa- tion Letters, 10:4898-4905, 2024b. URL https:
Shridhar, M., Manuelli, L., and Fox, D. Cliport: What and where pathways for robotic manipulation. In Conference on robot learning, pp. 894-906. PMLR, 2022.
Shridhar, M., Manuelli, L., and Fox, D. Perceiver-actor:
wrt.mannicccooaaa....mmm //app//eeatttccccooll..ooooooppp oroooppuuD 273162644.
A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pp. 785-799. PMLR, 2023.
Zhao, T. Z., Kumar, V. Levine, S., and Finn, C. Learn ing fine-grained bimanual manipulation with low-cost hardware. arXiv preprint
2023.
Smith, and Gasser, M. The development of embodied cognition: Six lessons from babies. Artificial life, 11(1-2); 13-29, 2005.
Su, J., Lu, Y., Pan, S., Wen, B., and Liu, Y. Roformer: Enhanced transformer with rotary position embed- ding. ArXiv, abs/2104.09864, 2021. URL https: sew.iisemannicccchec 233307138.
org/CorppsID:
Wang, C., Luo. X., Ross, K., and Li, D. Vr13: A data- driven framework for visual deep reinforcement learning. Advances in Neural Information Processing Systems, 35: 32974-32988, 2022a

---

SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
A. Model Architecture
We will explain our model architecture in detail, including Multi- Transformer, Memory Attention, Memory Encoder, and Memory Bank. The multi-resolution is already explained in Section 4.1.
Multi-Viee Transformer. The two MVTs used in the coarse and fine branches have the same architecture. Very similar to the MVT proposed by (Goyal et al., 2023), the input to the transformer consists of a language description of the task, virtual images of the scene point cloud, and the image embeddings (at the lowest resolution) generated by the SAM2 image encoder. The text is transformed into token embeddings using the pre-trained CLIP (Radford et al., 2021) model, while the virtual images are converted into token embeddings through patchify and projection operations. Similarly, the image embeddings are converted into token embeddings via a projection layer. For each virtual image, tokens corresponding to the same image are processed through four attention layers. Finally, the processed image tokens, along with the language tokens, are jointly processed using an additional four attention layers. The resulting image tokens are then used to infer the 3D action.
Memory Attention. Akin to the memory attention in SAM2 (Ravi et al., 2024), the purpose of this module is to condition the current observation features on both past observation features and predicted actions, specifically translation. Notably, features from each view are processed independently. We stack four transformer blocks, with the first one taking the image embedding output of MVT from the current observation as input. Each block applies self-attention, followed by cross-attention to memories of past observation features and predicted actions, stored in a memory bank (described below), and ends with a multi-layer perceptron (MLP). For both self- and cross-attentions we use vanilla attention operations, enabling us to leverage recent advances in efficient attention kernels (Dao, 2023). In addition to sinusoidal absolute positional embeddings, 2D spatial Rotary Positional Embedding (RoPE) (Su et al., 2021; Heo et al., 2024) are incorporated in both self-attention and cross-attention layers. We also reduce the dimension size from the original 256 to 128 to align with the image embedding dimension of the MVT output.
Memory Encoder. The memory encoder constructs memory features by downsampling the output translation heatmap using a convolutional module and summing it element- wise with the unconditioned observation embedding from the multi-view transformer (not shown in Figure 3). This is followed by lightweight convolutional layers to integrate the information. Instead of employing an additional image encoder, our memory encoder reuses the image embeddings produced by the MVT (not the SAM2 image encoder) and fuses them with the predicted translation information generate memory features. This design enables the memory features to leverage rich representations that incorporate language, semantic, and spatial features from multiple views, making them more suitable for encoding action memories. Originally, this module was designed to encode an image embedding with multiple object masks within the same frame. However, we do not utilize this functionality. Instead, we encode one memory per view, where each memory is generated by encoding a single heatmap with a corresponding image embedding from each view.
Memory Bank. The memory bank preserves past translation predictions associated with previous observations in the video by maintaining FIFO queue of up to N recent memories. Each view has its own independent memory bank, as memories are stored and retrieved separately for different views. These memories are represented as spatial feature maps. Additionally, in our memory bank, the memory features are projected to a dimension of 64.
B. Training Implementation
All models are trained on 32 NVIDIA H100/A 100 GPUs. In some cases, we also train on 16 or 8 NVIDIA H100/A100 GPUs, but ensure fairness by maintaining the same total batch size across all settings.
B. SAMZAct
We use the same way data and demo augmentation methods and training pipeline as in RVT2 (Goyal et al., 2024) train SAM2Act (stage 1). The training hyperparameters are shown in Table 5. We use this set of hyperparameters to train on RLBench and The Col colooseum.
B.2. SAM2Act+
We use a different strategy for sampling batch of data for training. Previous sampling strategies randomly select batch of independent observations, allowing the model to predict the next action based on each observation independently. However, for SAM2Act+, we aim for the agent to predict the next action based on both the current and past observations. To achieve
12

---

SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
Table Training Hyperparameters of SAM2Act on RLBench and The Colosseum. The batch size stands for total batch size across all GPUs. For the learning rate, we follow the scaling strategy used in RVT2 (Goyal et al., 2024), where the learning rate is scaled by the batch size as 1.25e 5 bs.
Hyperparameters
SAM2Act Training
batch size
256
learning rate
3.2e-3
optimizer
LAMB
learning rate schedule
cosine decay
weight decay
le-4
warmup steps
2000
training steps
56.25K
training epochs
90
LoRA rank
16
this, we must sample batch of data that is spatio-temporally consistent. To implement this, we randomly sample consecutive observations from a random episode. The forward pass is then performed sequentially from the first to the last observation. The details of the forward pass are provided in Algorithm
When adopting this new sampling method during training, one immediate effect a significant reduction in data diversity per batch. This can be detrimental, especially when dealing with tasks with numerous variations. We attempted to train the standard SAM2Act model on RLBench tasks using this new sampling method, but the convergence time was excessively long. To address this, we propose method, then fine-tune it with the new sampling approach. This strategy effectively mitigates the issue of slow convergence, significantly reducing training time.
training pipeline: first, we pre-train the model using the previous sampling
As mentioned in Section 5.4, we train all methods on MemozyBench in a single-task setting. However, finding a training configuration that optimizes all tasks challenging. To address this, we use a universal set of hyperparameters for training but evaluate models across all epochs and select the rest-performing one for evaluation. We follow the same approach to determine the optimal pre-trained weights for SAM2Act before fine-tuning SAM2Act+. In addition, the window size of the memory mechanism is also decided to be different for each task in Memory We keep the batch size the same as the window size during training, and thus the learning rate will be bit different as they are related with batch size. The detailed training hyperparameters are listed in Table 6.
Table 6. Training Hyperparameters of SAM2Act and SAMZAct+ on MemoryBench. Note that the batch size refers to the total batch size across all GPUs For SAM2Act+ training on the scopen.drawee task, we use maximum window size of 8, resulting in Per-GPU batch size of and a total batch size of 256. Similarly, for the other two tasks, where the maximum window size is 10, the total batch size 10
320 in total. The leamnngg rate follows the same scaling rule mentioned in Table 5.
Hyperparameters
SAM2Act Training
SAM2Act+ Training
batch size learning rate
256
256 (reopen.drawer), 320 (other two)
3.2e-3
3.2e-3 (reopen.drawer), 4e-3 (other two)
optimizer
LAMB
LAMB
learning rate schedule
cosine decay
cosine decay
weight decay
le-4
1e-4
warmup steps
2000
2000
training steps
6.25K
12.5K
training epochs
10
20
LoRA rank
16
16
13

---

SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
C. Full Comparisons for RLBench 18 Tasks
The full comparisons of SAM2Act with existing approaches on RLBench 18 tasks are shown in Table 7.
Table 7. Full Comparisons of Multi-Tass Performance on RI.Bench, We evaluate 18 RLBench tasks (James et al., 2020), reporting success rates across all tasks among all existing approaches, not limited to 3D keyframe- based behavior cloning (BC) policies. We report
stats of 4 evaluations for SAM2Act. Our method, SAM2Act, outperforms all baselines, achieving performance margin of 1.9% over
ARP (Zhang et al., 2024b), the prior state-of of the art approach.
Method
Arg. Succes:
Arg. Rank
Close Jur
Drag Stick
Insert Peg
Meat off Grill
Open Drawer
Place Cups
Place Wine
Pach Battons
Image BC (CNN) (ang ai., 2022)
1.3
12.4
0.D
0,9
0.0
0.0
4.0
0.0
0.0
0.0
Image BC (ang)aaang
2022)
1.3
126
0.0
0,0
0.0
0.0
0.0
0.0
0.0
0,0
CIF-ARM-BC
(021)
20.1
20.0
200
0.
HiveRommer (Cuhurr 2023)
45.3
9.6
$20
76.0
0.0
108.0
$2.0
00
80.0
54.0
NaaNerrChaan et al. 2023)
46.4
9.1
36.0
92.0
4.11
100.11
84.0
0.0
40.0
96.0
PerAct (Shrälhur et al., 2023)
2:43
8.9
55.2 4.7
89.6 4.1.
5.6 4.1
70,4 = 2.0
88,0 5.7
244 3.2
44,8 = 7.8
92.8 2:3.1
MCD (Grrett 2023)
65.0
92.0
92.0
27.0
94.0
93.0
3.0
80,0
99,0
(Goyal et 2023)
62,9
-
68.0
71.2
100.0
RVT-3 (Gryal at. 20244)
$14:1:
3.7
1000 0.0
99.
1.7
40.0 0.0
99.0 1.7
74.0 :: 11.8
58.0
95.0 ::
100.0 0.0
3D Difluse Actor (Ko al., 2024)
81.3
3.9
96.0 - 2.5
1000
+ 0.0
65.6 +-11
96.8 1/6
88.6 4.1
24.0 340-77 7/6
99.6 4.8
98.4 +
3D LOTUS (Gancia al., 2024)
$3.1
3.7
96.0 1 0.0
100.0 1 0.0
68.5 1. 3.6
98,4 2.22
83.6 7.3
40.8 1: 12.1
91.2 1: 1.66
100.0 1 0.0
ARP! (Zhanger 202/b)
$4,9
32
95.2
99.3
78.4
91.6
92.8
48.8
96.0
100,0
70.6
81.4
18.4
98.2
0.0
100.0
SAMBAcI
868 0.5
900-
99.0
$4.0 57
98.0
83.0
600
47.0 60
93.0
10RL0
Method
Put Cuphoard
Drawer
Put
Screw
Block
Shape
Stack Blocks
Stuck Cups
Sweep to Dustpan
Tup
Imuge-BC (CNN)
2023)
0.0
4.0
0.0
0.0
0.0
0.
0.0
0.0
8.0
Image BC (WTT) Uang et tl., 2022)
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
16.0
CIF ARM BC Oames al., 2021)
0.0
4,0
12.0
8.0
16.0
8.0
0.0
0.0
0.0
68.0
HiveFormer (Gabur 2002)
52.0
68.0
76.0
8.0
64.0
80
80
0.
28.0
80,0
PobaNee
2023)
92.0
44.0
56.0
12.0
40
80
$2.0
PerAst (Shiklur atal. 2023)
$1.2 2 4.7
84.0=
74.0 13.0
16.8 4.7
3.2
$2.0
88.0
Ar3D (Gervet et al., 2025)
$1.01
98.0
95.0
47.0
93.0
8.
12.0
9.0
92.11
94,0
RVT (Gyyal et 2023)
49.5 3.2
38.0 1: 5.7
91.2 3.0
48.0 5.7
36,0 26.0222 2.5
28,8
3.9
26,4 2 8.2
72,0 1 0.0
93.6 2: 4.1
RVT2 (Goyal al.. 2024)
66.01144.5
96.0
96.0
88.0
92.0 2.8
35.0
80.0 2.8
69.0 5.9
100,0 1.0.
99.0 1.7
Diffuser Actor (Ke 2004)
2024)
4..
82.4
44.0
47.2
99:2 1.6
BDALCITU (Gancis
78.4++4. 4.6
97.6
95.2
888 +
99.2 1.8
34.4
58.4
75.2 7.7
96.0
90.4 4.6
ARP (Zhagget et al., 30248)
69.6
98.4
86.4
89.6
92.8
46.4
63.2
97.6
96.0
SAM E (Zhing all. 2024
64.0.12.88
92.0 1. 5.7
95.2 3.3
1.36
98.2 1.8
34.4 6.1
26.4 4.6
0.0 : 0.0
100,0 1: 0.0
100.0 1 0.0
SAMIAcI (Duns)
75.0++3.8
99.0 + 2.0
98.0 2.3
89.0 2,0
$6.0 4.0
64.0 4.6
76.0- 86
78.0
99.0 2.0
96.0 5.7
D. Full Results for The Colosseum
The full results of SAM2Act on The Colosseum are shown in Table 8.
Table 8. Full Results of SAM2Act for Various Perturbations on The Colosseum, Mean and std of evaluations are reported.
Tink Nare
No Varatino
All Variatioas
MO Cabee
BD Cobe 380 Testor: 80 Tosturs
MD Siz
RD Sioc
Light Ooder
Taki Cake
Balk: Testore
Doinater
Backgrozed Iestare
Canera Poc
baketall heop
201
327
233 25
I06U
373.1225
Us
85.7
11412 LLD
1010:1d.
1ax0:210 uLC
247 2:4
100
1060
96334
010-30
100:-00
960-00
435--35
960-1000
950-5000
13-22
03:100
50 pd
30 00
13 233
00-000
00-00
0D:440
60--65
66 03-03
D0 60-60
on -0-00 60
go foe ihe lidae
05.32445
4.3
20
0L1 23
23
$402250
1:5:25
155
16.213
147=15
235
213:2:3
1000 = 60)
bodky
00:000
133=13
23
133
$10
267
67:45
21.12:3
95.1
155
1301330 3m
13.1 50
10
155
64n D0
110
160
15.3
001:01
00:108
03:100
0A IN
DO n1
100100
0D:140
00:100
15 3.3
DO 1 64
spandower
067 23
30.725511
150200
$5:0000
100:0000
155
087423
Jak bulegd
1200
11.3
255
400=000
56.7 0.0
520
33.3 255
26.5
767-2 2.1
25112-40 41
547
52.1 TEA
057
11.
760 31.
15.0
23,5
45.3 5 16.5
ploc wis 31 www.k....aee.. Jecatin
2.
367
2 23
851 23
91T 4.6
11.5 4.
157 2:4A
03:5 4.
16.7 2:4A
V20-1..9
66332 36.3
26.7 56.
25
46
613
100
83.7-4 1.
$40-10 40
587--13
23
23
96:-000
730--30
960-00 00
357
460:-00
9.0-1000
straghim oxpe
76.22992
675.23
4-3
140:2:00
14
11
453
9172992
260:2000
scAp ches
1..:2.23
00:300
20
23
160:2:0
25
11.
11.
250
117:446
200-166
253
003-166
35
900-1 69
417.442
357-35
1674-16
511-16
360-29
900-000
4-1-1-66
E. Ablation on SAM2Act
E.1. RLBench
We conduct ablation experiments the proposed SAM2Act,, focusing on two key aspects: the SAM2 image Encoder and multi-resolution upsampling. We evaluate the model under three different configurations:
(i) Replacing the SAM2 image encoder with the SAM image encoder and removing the multi-resolution upsampling, as the SAM image encoder does not produce multi-resolution outputs. (ii) Replacing the multi celtiiresolution upsampling with the original convex upsampling from RVT-2 (Goyal et al., 2024). (iii) Removing SAM2's 's multi-resolution image embedding inputs to the multi-resolution upsampling while keeping the multi-resolution upsampling itself.
14

---

SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
Note that SAM-E (Zhang et al., 2024a) proposed 3D behavior cloning policy that integrates RVT and the SAM image encoder, along with an action-sequence policy head. We attempted to extend this method to the more powerful RVT2 backbone for comparison. However, its action-sequence policy proved incompatible with the coarse-to-fine pipeline, resulting in very slow convergence under SAM-E's training setup. To ensure a fair comparison, we also extended SAM-E while keeping its original hyperparameters (notably, a LoRA rank of 4, whereas ours is 16). We trained both versions and found that SAM-E's configuration performed better. Therefore, we adopted their configuration and reported the results accordingly, which also applies to Section 5.3. For all other ablation experiments, the training configuration are kept the same.
Ablation results on RLBench are presented in Table 9. All variants of SAM2Act exhibit lower performance than the original version. Removing SAM2's multi-resolution image embedding inputs results in 1.1% drop in the average success rate. Replacing the entire multi-resolution upsampling with the original convex upsampling leads to a 2.6% decrease. Substituting the SAM2 image encoder with the SAM image encoder (Kirillov et al., 2023) causes a 6.0% drop compared to SAM2Act and a 3.4% drop compared to SAM2Act with the original convex upsampling, where the only differences are the image encoder and some training hyperparameters. the same setting, we further replace the SAM2 image encoder to latest image encoders, DINOv2 (Oquab et al., 2023) and Depth Anything V2 (Yang et al., 2024), while both of them show a large drop compared to the original SAM2Act. These results indicate that all of our architectural innovations significantly enhance the agent's s ability across multiple manipulation tasks.
Table SAM2Act Abaltion Performance on RLBench. We report the success rates RLBench tasks (James et al., 2020), along
with the average success rate and ranking across all tasks. Table shows that SAM2Act outperforms all of its variations.
Ang. Streess
Arg. Rank
Close
Drag
Prg
Open Drawer
Place Cupe
Place
Puch
SAMEACISAMM SAM)
96.0
94.0
72.0
38
SAMEAAISAMM Depth Anything V2)
100.0
98.0 2.3
3.8
93.0 388
96.0 0.0
SAMZAcI (SAM2
DINOw2))
$2. 0.5
3.8
97.0+:2.0
98.0 2.3
69.0 3.8
99.0 2-2
800 533
90.0 7.7
80.0 388
960 0.
SAM2AcI (Driginal Upsampling)
$4.2 0.0
3.4
100.0 0.0
100.0 0.0
91.0 3.8
90.0 2.0
780 9.5
29.0 6.0
88.0 5.7
960+ 0.0
SAMEAcT (win Malni res Inpur)
15.7 = 0.3
2.7
95.0 2.0
95001.0.0
86,0 5,5
2.3
990 00::20 2.0
4300 1005
9601 = 0.0
108.0 1: 0.0
SAMEAct
868 0.5
3:00 2,0
99.03:24 2.0
8..0
9..0-12.3
830 : 800
47.0 6.60
95.0 38
108.0
Method
Put in Cupboard
Put in Deawwrr
Put in Safe
Screw Bulh
Slide Black
Sort Shape
Stack Blocks
Stack Caps
Swee to Dustgan
Tarn Tap
SAM2AcI (SAM) - SAM)
12.0 - 8.6
96.0 + 2.3
99.8 2.0
92.0 + 5.7
93.0 3.8
41.0 + 3.8
73.0 - 3.8
71.0 2.0
95.0 + 3.3
95.0 + 2.0
SAMEAcI (SAM2 - Depti Anylinn V3)
7800 2.3
96.0 + 3.8
95.0 :::::220 2.0
90.0 6 2,3
67.0 2.0
45.0 4: 5.0
600 =0:::3
91.0 2.0
108,0 0.0
90.0 :: 4.0
SAMEAct (SAM2- DINOO2)
$0.0 3.3
99.0 2.0
93.0 3:38
89:0 2.0
77.0 2.0
4..0:1.8.
690 5.0
79.0 6.8
96.0 2.3
920 3:33
SAM2Aci (Drginal Upanmping))
9.3
388
6.0
SAM2AcT(wo Muld Input)
96.0
1..02...
54.0
9000
92.0
SAMZAsI
75.0
99.0
98.0 2.3
80.0
0::0:::0
7600 86
78.0
9:00=22.
96.0
E.2. The Colosseum
We also conducted the same ablation experiments on The Colosseum generalization benchmark, as shown in Table 2. The experimental setup remains the same as in Table 9, except that we did not test the variant of SAM2Act with the original convex upsampling. The results in Table 9 show that removing SAM2's multi-resolution image embedding inputs leads to 14.8% drop in performance, representing relative decrease of 344.2%. This highlights the effectiveness of SAM2's multi-resolution image embeddings in providing robust visual representations, significantly enhancing SAM2Act's generalization ability.
F. RLBench Tasks
We follow the multi-task, multi-variaiion simulated experiment setup of PerAct (Shridhar et al., 2023), RVT (Goyal et al., 2023), and RVT-2 (Goyal et al., 2024), using 18 RLBench tasks with 249 unique variations in object placement, color, size, category, count, and shape. A summary of the 18 RLBench tasks is provided in Table 10. For more detailed description of each task, please refer to PerAct (Shridhar et al., 2023).
G. MemoryBench Tasks
In the following we provide details of the MenoryBench tasks.
(a) Reopen drawer
Task Description: The robot is instructed remember the drawer slot that was initially opened, and closed and then press the button on the table, before finding back the previously opened drawer to re-open
15

---

SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
Table The 18 RLBench Tasks for Multi-task Experiment. report on language template, average number extracted
keyframes, the task variations, and the variation type.
Task name
Language Template
Avg. Keylrames
#of Variations
Variation Type
put in drawer
"put the item in the drawer"
12.0
3
placement
reach and drag
"use the stick to drag cube onto the target"
6.0
20
color
turn tap
"turn tap"
2.0
2
placement
slide to target
"slide the block to target"
4.7
4
color
open drawer
"open the drawer"
3,0
placement
put in cupboard
"put the in the cupboard"
5.0
category
place in shape sorter
"put the in the shape sorter"
5.0
5
shape
put money in safe
"put the money away in the safe on the t shelf"
5,0
3
placement
push buttons
"push the button, [then the button]"
3.8
50
color
close jar
"close the - jarr"
6.0
20
color
stack block
"stack blocks"
14.6
60
color,count
place cups
"place cups on the cup holder"
11.5
count
place wine at rack
"stack the wine bottle the of the rack"
5.0
placement
screw bulb
"screw in the - light bulb"
7.0
20
color
sweep to dustpan
"sweep dirt to the dustpan" "put the ring on the spoke" "take the off the grill"
4.6
2
size
insert peg
5.0
20
color
off grill
5.0
2
calegory
stack cups
"stack the other cups top of the cup"
10.0
20
color
Success Metric:
The task is considered successful once the initial opened drawer has been re-opened.
Objects: A drawer and button.
Variation Number:
Keyframes: 8
Language Instructions: "Close the drawer, then reopened the previously opened drawer while pushing the button in between."
(b) Put block back
Task Description: The robot is instructed move the block the centre, then push the button, then move the block back to its initial position.
Success Metric: The task is considered successful once the initial block has been moved back to its initial pose.
Objects: Four patch, one block and one button.
Variation Number:
Keyframes: 11
Language Instructions: ""Pul the block to the centre and then back to its initial position while pushing the button in
between." "*
(c) Rearrange block
Task Description: The robot is instructed move the block in the centre to the empty patch, and then press the button, and then move the alternative block to the centre..
Success Metric: The task is considered successful once the alternative block has been moved to the centre.
Objects: Two patch, two blocks and one button.
16

---

SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
Table Properties the Real-world Tasks. report on language template, average number of extracted keyframes, the number of items that the robot can interact with, the task variations, and the variation type.
Task
Language template
keyframes
items
variations
variation type
(a) turn the lamp
"turn the lamp"
4.5
1
placement placement
(b) push buttons in sequence
"push the red button, then the green button"
3
(c) stack cubes
"stack the cube on the cube"
4.0
5
3
calegoryyplasement color.placement
(d) push the right button
"push the button closest the blue block"
3
Variation Number:
Keyframes: 10
Language Instructions: "Move the block not the patch to the empty patch, then press the button, then move the block that has not been moved off the patch."
H. MemoryBench Update
We updated the zeopen.drawer task in MemoryBench for the following reasons. During training on the original data, we observed that the gripper often collided with the drawer handle when closing the drawer. prevent this, we introduced an additional waypoint for the closing motion, mirroring the procedure used for opening the drawer. Consequently, retrained all policies specifically on this updated task. Furthermore, standardize the memory window size across all three tasks, we also retrained SAM2Act+ on this task using window size of 10, which led improved performance. All results are updated to Table 3.
I. Real-world Experiments
In the following we provide details of the real-world setup and tasks. Figure 5 illustrates the real-world setup. Table summarizes the properties of the real-world tasks.
(a) Turn on the lamp
Task Description: The robot is instructed to turn on lamp by rotating its knob.
Success Metric: The task is considered successful the lamp has been turned on by rotating the knob.
Objects: A single lamp.
Coordination Challenges: High precision is required to properly rotate the knob.
Language Instructions: "Turn on the lamp."
"
(b) Push buttons in sequence
Task Description:
The robot must press the red button first and then the blue button.
Success Metric: The task is considered successful if the buttons pressed in the specified order: red, then blue. A third button is present but should remain unpressed.
Objects: Three buttons in front of the robot.
Coordination Challenges: Ensuring the robot presses the correct buttons sequence without pressing the third button.
Language Instructions: "Push the red button and then the blue button." "
(c) Stack blocks
Task Description: The robot must place one specified block on top of another specified block.
17

---

SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
Figure 5. Real-world Robot Setup. A Franka Panda rohot with Robotiq Gripper. A RealSense D455 depth sensor captures the scene.
Success Metric:
The task is successful if the designated block is stacked on the correct target block.
Objects: Three single-colored blocks.
Coordination Challenges: Precision in picking and placing, plus correct language understanding to identify which block goes where.
Language Instructions: "Stack the <ilem>> block on the <item> block." "
(d) Push the same button
Task Description: The robot must first identify and press the button closest to the blue block, then press the same button again after the block removed.
Success Metric: The task is successful if the robot presses the correct button twice. Pressing the other button any point results in failure.
Objects: Two buttons and one blue block (marking proximity).
Coordination Challenges: After the first button press, the blue block is removed; the robot must remember the button location to press again.
Language Instructions: "Push the button that is closest to the blue block. Press the same button again." "
18


Reviews:
[{"review_id": "be5axjXGM3", "reviewer": "Reviewer_yA68", "summary": "The paper introduces SAM2Act, a multi-view, language-conditioned behavior cloning policy for 6-DoF 3D robotic manipulation, which integrates a visual foundation model (SAM2) with a memory architecture to enhance feature representation and task-specific reasoning. SAM2Act leverages multi-resolution upsampling and visual embeddings from SAM2 to achieve high-precision manipulations and robust generalization across environmental and object-level variations. Additionally, the authors propose SAM2Act+, an extension of SAM2Act that incorporates memory-based components (Memory Bank, Memory Encoder, and Memory Attention) to enable episodic recall and solve spatial memory-dependent tasks. The paper also introduces MemoryBench, a benchmark for assessing spatial memory in behavior cloning models. Empirical results demonstrate that SAM2Act and SAM2Act+ achieve state-of-the-art performance across multiple benchmarks, highlighting their effectiveness in complex manipulation tasks and their ability to generalize to unseen perturbations.\n\n**Post Rebuttal:**\n\nThanks for additional results provided in the rebuttal, most of my concerns are addressed. Overall, the overall technical novelty of this paper is somehow limited. However it's unique contributions (such as the memorybench and analysis on memory-related tasks) could be interesting to some group of the community. I've decided to raise my score to Weak Accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank **Reviewer yA68** for the detailed and insightful feedback. We appreciate the recognition of our innovative use of SAM2 for generalization, the intuitive memory module design, and our comprehensive benchmarking. We're especially grateful for the positive remarks on our real-world evaluations, also echoed by Reviewer *24ak* and *auqZ*. We also value the reviewer’s recognition of MemoryBench as a meaningful contribution to addressing critical research questions, as well as the acknowledgment that our claims are well-supported by the evidence—an assessment shared across reviewers.\n\nDespite this, the reviewer raised several concerns and suggestions, which we address below:\n\n**The model architecture appears to have limited originality, as the SAM2Act module and coarse-to-fine pipeline closely follow RVT-2, and the memory module in SAM2Act+ strongly resembles SAM2’s tracking module.**\n\nWe acknowledge that our work naturally builds upon prior research, embracing the principle of 'standing on the shoulders of giants,' much like RVT-2 evolved from RVT through targeted architectural changes, or SAM-E integrated elements from RVT with SAM's image encoder and action-chunking module. Our approach similarly leverages existing architectures but introduces critical, innovative adaptations that significantly enhance performance. In particular, our novel contribution is in effectively addressing the challenging problem of memory, representing one of the first systematic attempts in this area. Our key adaptations that make SAM2Act and SAM2Act+ effective are: (1) Multi-Resolution Upsampling, which leverages SAM2’s multi-scale image embeddings to boost RLBench performance and generalization (Tables 2 & 7), and (2) Memory Task Adaptation, where we extend SAM2’s tracking module to multi-view, multi-step settings by treating action heatmaps as object masks and integrating MVT embeddings into memory. This extension, which required careful design and extensive experimentation, is novel and essential—naively using the base models without these changes fails, as shown in our ablations, Table 7.\n\n**Given the performance drop when replacing SAM2 with SAM—falling below even the RVT-2 baseline on Colosseum and RLBench (Tables 2 & 7)—to what extent does the method's effectiveness rely on SAM2 rather than the proposed contributions?**\n\nWe appreciate the reviewer’s observation and agree that the SAM2 encoder contributes significantly to performance. The drop observed when replacing SAM2 with SAM aligns with findings from the SAM2 and SAM2-Adapter papers (to be cited in future version), which show that SAM2’s lightweight Hiera encoder yields stronger embeddings for segmentation and downstream tasks. However, our ablations also demonstrate that the strong performance of SAM2Act arises from the combination of SAM2 with the other novel contributions from our proposed method and not solely from the encoder.\nOur primary contributions include innovations like **multi-resolution upsampling**, which adeptly leverages multi-resolution embeddings. As shown in Table 2, the improved generalization on Colosseum is primarily driven by multi-resolution upsampling. Without multi-resolution embeddings, performance matches the SAM-based variant, highlighting our architectural contributions as the main driver of generalization. Moreover, when comparing the SAM-based variant to RVT-2, their overlapping performance intervals (80.8 ± 1.9 vs. 81.4 ± 3.1) indicate no statistically significant differences. In summary, our approach combines several novel contributions that together enhance both performance and practicality.\n\n**Could the authors clarify the distinction between general long-horizon tasks and the specific tasks in MemoryBench, particularly highlighting the unique challenges that make a dedicated memory module necessary?**\n\nRobotic manipulation tasks typically follow the Markov assumption, where the optimal action depends solely on the current observation. Even in long-horizon tasks key information is directly observable and might not need memory. In contrast, MemoryBench is explicitly designed to violate this assumption—tasks are ambiguous, and visually identical states may require different actions based on prior interactions. We appreciate the reviewer’s suggestion of LIBERO, it emphasizes action-based models, which differ from our keyframe-based approach in SAM2Act. To investigate the relation between long-horizon and memory-based tasks, we curated four cube-stacking tasks with increasing complexity and keyframe length. We observed consistent performance degradation for both SAM2Act and SAM2Act+ as horizon length increased, suggesting that memory-based challenges extend beyond task length alone. (Results on our [anonymous website](https://sam2act-icml.github.io/)).\n\nWe sincerely hope our responses have adequately addressed all of the reviewers' concerns. If so, we would greatly appreciate your consideration in raising your ratings.", "claims_and_evidence": "All the claims made in the paper are supported by corresponding evidence.", "methods_and_evaluation_criteria": "The proposed pipeline, including the utilization of SAM2 for improved generalizability and the proposed memory module for tackling memory-specific tasks, is intuitive and makes sense.\n\nThe evaluation benchmarks are comprehensive, and supported by further real-world evaluations. The new benchmark proposed, MemoryBench, also addresses important research questions.", "theoretical_claims": "No theoretical claims are made in the paper.", "experimental_designs_or_analyses": "It would be better for the authors to highlight the difference between *long-horizon manipulation tasks* and tasks that require the trained policy to have both semantic and spatial memory as introduced in the MemoryBench.  More detailed discussion would be helpful in highling contributions in this paper.", "relation_to_broader_scientific_literature": "The utilization of SAM2 feature for various downstream tasks is adopted in prior literature [1].\n\n\n[1] Chen, Tianrun, et al. \"Sam2-adapter: Evaluating & adapting segment anything 2 in downstream tasks: Camouflage, shadow, medical image segmentation, and more.\" arXiv preprint arXiv:2408.04579 (2024).", "essential_references_not_discussed": "Many designs proposed in the paper are largely inherited from RVT-2[1], except for the memory module. I believe it would be better for the authors to highlight unique designs that distinguished this paper from RVT-2. Otherwise, the overall technical novelty seems to be limited.\n\n[1] Goyal, Ankit, et al. \"RVT-2: Learning precise manipulation from few demonstrations.\" RSS 2024.", "other_strengths_and_weaknesses": "**Strength:** The proposed MemoryBench addresses an important aspect of the robot policies: the spatial memory capabilities. The evaluation suite is also thoroughly discussed in the paper.\n\n**Weaknesses:** The originality of model architecture design is constrained. Given the SAM2Act module and the coarse-to-fine pipeline is largely borrowed from RVT-2, and the memory module in SAM2Act+ is similar to the tacking module in SAM2.", "other_comments_or_suggestions": "None", "questions_for_authors": "1. As shown in Table 2 and Table 7, replacing SAM2 with SAM results in a significant performance decline. The generalizability evaluated in Colosseum and the average success rate in RLBench for the SAM-based variant are even lower than those of the RVT-2 baseline. Does this outcome suggest that the performance of the proposed method is largely attributed to the SAM2 encoder? Further clarifications can help better understand technical contribution and novelty of the paper.\n2. Could the authors clarify the distinction between long-horizon tasks in general and the tasks in MemoryBench, as well as the unique challenges posed by tasks that necessitate the use of a memory module? Additionally, would the memory module also bring improvements to long-horizon tasks more broadly, such as those in the LIBERO-Long benchmark?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iB9aUoAOvO", "reviewer": "Reviewer_24ak", "summary": "This submission introduces SAM2Act, a transformer-based coarse-to-fine behavior cloning policy for language-conditioned robot manipulation. SAM2Act exploits the multi-resolution features of SAM2 vision foundation model, through a novel up-sampling scheme which enables high-precision manipulation behavior. Additionally, the SAM2Act+ employs a spatial memory module, enabling it to solve more complex memory-based (non-Markovian) tasks. To evaluate the latter, a novel robot manipulation benchmark (extension to RLBench) is proposed, featuring 3 carefully designed tasks that specifically evaluate the model's spatial memory capabilities.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the **Reviewer 24ak** for recognizing the novelty of our approach, specifically in leveraging multi-resolution features from SAM2, and our innovative up-sampling scheme designed to facilitate high-precision manipulation policy learning. Additionally, we are grateful that the reviewer acknowledged our effective use of SAM2's memory mechanism to address complex, memory-based tasks, as well as our carefully designed MemoryBench, explicitly created for evaluating spatial memory capabilities. We are pleased that the reviewer found our claims well-supported by clear and convincing evidence (agreed by Reviewer *yA68* and *auqZ*). Moreover, we thank the reviewer for emphasizing the timeliness and relevance of our research within robotic manipulation(similar to  Reviewer *yA68* and *auqZ*), and for highlighting the novel insights our study provides towards training generalist robotics agents(agreed by Reviewer *auqZ*).\n\nDespite this, the reviewer raised several concerns and suggestions, which we address below:\n\n**How does the proposed architecture differ from world models? Can SAM2Act+ be adapted to serve as a backbone for training world models?**\n\nWorld models (e.g., DINO-WM, TD-MPC) differ significantly from SAM2Act in both objectives and training paradigms. While world models are trained to predict future observations and latent dynamics using various objective functions, SAM2Act is trained via behavior cloning. Although both incorporate memory and visual features, SAM2Act targets more realistic scenarios that violate the Markov assumption, unlike world models as mentioned by the reviewer. Moreover, world models typically decouple visual encoding and memory during training, whereas SAM2Act+ tightly integrates visual embeddings with spatial memory in an end-to-end fashion. While SAM2Act+ could potentially serve as a backbone for world modeling—by replacing the behavior cloning head with dynamics prediction and incorporating self-supervised objectives—that direction is beyond the scope of this work. Due to ICML's no-revision policy during rebuttal, we'll discuss this in a future version.\n\n**Is SAM2 the best choice of a vision foundation model used as backbone for robotic manipulation as compared to other visual encoders?**\n\nOur choice of SAM2 as the vision backbone is strongly justified by both prior work and our own ablation studies. In the SAM-E paper, a thorough evaluation compared various visual encoders—including CLIP, DINO, and the robotics-specific R3M—and found that the SAM image encoder outperformed the others when paired with an RVT-based action sequence prediction module. Since SAM2 builds on SAM by introducing multi-resolution features, it is expected—and indeed confirmed—to deliver superior performance and also supported from in-depth studies from the original SAM2 paper and SAM2-Adapter. To further validate this, we replaced SAM2 with alternative encoders suggested by the reviewer. Our experiments show that when using original upsampling (a fair comparison, as only SAM2 provides multi-resolution embeddings), **DINOv2 achieves 82.2 ± 0.5** and **DepthAnythingV2 achieves 81.1 ± 1.2** on the RLBench 18 tasks. Even the ablated version of SAM2Act, which uses the SAM2 encoder without multi-resolution output, already outperforms both alternatives with a success rate of **84.2 ± 0.9**. Moreover, when leveraging SAM2’s multi-resolution outputs, we anticipate an even larger performance gap in its favor. Thus, the evidence strongly supports SAM2 as the best choice for our application, as its multi-resolution capabilities are uniquely advantageous for the robot manipulation tasks addressed in our work.\n\n**Clarification on whether the reopen-drawer and put_block_back tasks effectively evaluate different aspects of spatial memory (namely, 3D versus 2D spatial information) as claimed?** \n\nWe realize our description of spatial memory evaluation may have caused confusion. Our intention with the two tasks is not to distinguish 3D from 2D spatial memory, but rather to isolate distinct components of spatial memory. Specifically:\n- **Reopen_Drawer:** This task is designed to assess whether the agent can retain and recall information along the vertical (z-axis) dimension. It tests the agent’s ability to remember which specific drawer was interacted with previously.\n- **Put_Block_Back:** This task evaluates the agent’s memory of the horizontal layout (x and y axes), ensuring it can accurately reposition objects within the overall spatial configuration.\n\nBoth tasks involve processing 2D pixel input but focus on different spatial dimensions within a 3D environment. By decoupling these aspects, our benchmark evaluates how effectively an agent integrates spatial cues across different axes. We hope this clarifies the confusion regarding the spatial memory tasks.\n\nWe hope our responses have addressed all of the reviewers' concerns. If so, we would appreciate your consideration in raising your ratings.", "claims_and_evidence": "In my opinion, the claims made in this submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "In my opinion, the proposed methods and evaluation criteria are meaningful for the task at hand.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "In my opinion, the experimental design and analysis is convincing.", "relation_to_broader_scientific_literature": "This submission integrates an existing vision foundation model (SAM2) with novel components (multi-view policy and memory architecture) to enhance it with language-guided robot manipulation capabilities. The proposed model formulation is novel and pushes the limits of SOTA in high-precision and memory-dependent manipulation tasks. Furthermore, the proposed spatial memory-based benchmark tasks can be used by the community to provide a more target evaluation of robot manipulation policies on memory-aware tasks.", "essential_references_not_discussed": "In my opinion, the related works that are essential to understand the key contributions of the paper have been cited. \nA more enhanced discussion of the positioning of the proposed methodology to approaches from different manipulation related literature may enhance the understanding of the reader (please see question below for more specific suggestions).", "other_strengths_and_weaknesses": "Strengths:\n- The manuscript studies a very interesting and timely problem.\n- The proposed solution is novel and insightful and successfully takes advantage of the latest advances in vision foundation models (offering tremendous generalisation capabilities) integrating them with novel elements that enhance their capabilities on high-precision and memory-aware robot manipulation.\n- The proposed benchmark (extension to RLBench) can also facilitate the targeted evaluation of manipulation policies specifically on memory-aware tasks. \n\nComments (related to questions below): \n- World models (e.g. the latest DINO-WM), although serving a different purpose, also typically feature a combination of a visual encoder, with a memory component and policy (although many works tend to adhere to the Markovian assumption). It is unclear how the proposed approach relates to such approaches, or if the proposed model architecture can potentially be adopted for world modelling. \n- The choice to build on top of SAM2, although proven effective due to the multi-resolution features provided and ablated against previous versions of SAM, could have been further ablated by the use of different visual encoders (e.g. DINOv2 or DepthAnything features) to indicate the generality of the proposed approach, and offer more insights on which Vision Foundation Models are most suitable for robot manipulation.\n- It is unclear whether reopen_drawer and put_block_back tasks of the proposed benchmark effectively do evaluate different aspects of spatial memory (namely 3D vs 2D spatial information) as claimed. This is because the z-axis information of which drawer to reopen can potentially be encoded in the 2D pixel-space, unless if the position of the chest of drawers all together is randomized between the 2 stages of the experiment.", "other_comments_or_suggestions": "Not Applicable", "questions_for_authors": "1. How does the proposed architecture differ from the architecture that are used to build world models (vision model + memory component) and how can the proposed architecture (due to its similarity) be adapted as a backbone architecture for training world models. It may be beneficial to add this discussion on the manuscript. \n2. Is SAM2 the best choice of a vision foundation model to be used as a backbone for the integration of the proposed robot manipulation components (and why) ? How do other vision foundation models compare to SAM2 as visual encoders for the task at hand?\n3. Is there any evidence or insights that the proposed benchmark tasks of reopen_drawer and put_block_back evaluate different aspects of spatial memory?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z0Co4OE1re", "reviewer": "Reviewer_auqZ", "summary": "This paper addresses the problem of robotic manipulation with memory. To that end, the authors combine the RVT-2 manipulation framework with the SAM2 memory-enabled segmentation model. I'll only adjust the \"coarse\" module which predicts the rough spatial region of interest for manipulation; the memory is only incorporated into the coarse module, not the \"fine\" grip-prediction module. The high-level architectural combination works as follows:\n\n1. RVT-2 takes a bunch of random camera views and synthesizes a point cloud, from which three standard orthogonal camera angles are re-rendered (XY / YZ / XZ planes). There is then a learned component which takes in these RGB images and a language instruction and outputs are three action heatmaps from the same viewpoints. The spatial intersection of the extrusions is the region of interest that is then passed to the downstream \"fine\" module.\n\n2. The authors want to ultimately enable the integration of SAM2's memory / object tracking capability into this architecture. So they introduce a \"SAM2Act\" module which produces the action heatmaps. There is some fancy upsampling going on, but the key idea is that they pass the instruction, original RGB images, and the SAM2 embeddings of those images all into a multi-view transformer (\"MVT\"), which then outputs a latent vector that is upsampled into the heatmaps. This new architecture alone shows some decent performance improvements over RVT-2 without even considering the memory.\n\n3. Now memory is incorporated as follows, independently for each view. The basic idea is that past input image embeddings and output heatmap embeddings are stored pairwise in a FIFO queue. They take the heatmap image embedding output of the MVT and do a few layers of cross-attention with the memory, and then output a new heatmap image. This allows the module to condition on the observation history.\n\nThe authors show that the SAM2Act module alone results in a ~5% success rate bump for existing benchmarks, and with memory enabled reduces the failure rate by ~80% on some memory-requiring benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank **Reviewer auqZ** for recognizing the contributions of our work. We are pleased the reviewer found novelty in our method, noting its ~5% improvement in success rate and state-of-the-art performance on benchmarks, as well as our memory components reducing failures by ~80% on memory-intensive tasks. We also appreciate that reviewer acknowledged our comprehensive experiments and thorough evaluation.\n\nDespite this, the reviewer raised several concerns and suggestions, which we address below:\n\n**The baselines are a little lacking.** \n\nWe appreciate the reviewer's feedback on the baselines. Our evaluation spans three benchmarks: RLBench (18 tasks), The Colosseum, and MemoryBench. We achieve state-of-the-art results on RLBench and The Colosseum. For MemoryBench, we tested OpenVLA, which obtained a 0% success rate across all tasks (All baselines on our [anonymous website](https://sam2act-icml.github.io/)). OpenVLA demonstrated basic interactions (e.g., approaching objects) but failed key actions like closing drawers or grasping, due to visual ambiguities—identical scenes requiring different actions (e.g., opening vs. closing). Conventional VLAs struggle as they rely solely on visual and language cues without temporal context. Our method integrates timestep information into proprioceptive inputs, resolving these ambiguities effectively. Consequently, even RVT-2 and SAM2Act (without explicit memory) succeed in such tasks, though occasionally erring on subtasks requiring actual memory. Extensive baseline comparisons and ablations confirm our method’s robustness on MemoryBench, highlighting areas for VLA improvement. Full results tables will appear in the final manuscript.\n\n**Clarification on how the higher-resolution embeddings from the SAM2 encoder are passed to the multi-resolution resampling methods in Stage 2?**\n\nWe apologize for the confusion and clarify briefly: Multi-resolution embeddings from the SAM2 encoder integrate into Stage 2 as in Stage 1 (see simplified Figure 4). Lower-resolution (res 16) embeddings pass through MVT and Memory Attention (conditioned on Memory Bank embeddings) before multi-resolution upsampling. Higher-resolution embeddings skip MVT and Memory Attention, entering upsampling directly. The resulting action heatmap is encoded into memory, indirectly involving all resolutions in memory processing.\n\n**What are the cases where SAM2Act+ still fails on MemoryBench?**\n\nOur analysis of the reopen_drawer task indicates that failures primarily stem from memory issues rather than execution errors. Specifically, SAM2Act+ often reopens the wrong drawer, showing reliable command execution but difficulty recalling the correct drawer. This aligns with our benchmark's focus on isolating memory-related errors. We found that shadows in the simulation occasionally caused ambiguous visual cues, complicating memory recall. To test this, we regenerated the dataset without shadows and, after retraining, SAM2Act+ achieved 100% success. This confirms that visual ambiguity from shadows was the primary cause of memory failures and demonstrates our approach effectively resolves these memory challenges.\n\n**Can you speculate about why the robustness is so improved in Section 5.3? Is it the pre-trained knowledge from the SAM2 image encoder that helps distinguish semantic meaning despite visual perturbations?**\n\nSimilar response to Question 2 from **Reviewer 24ak**; please kindly refer to that for more details.\n\n**For SAM2Act+ why does the fine branch not have the memory component?**\n\nOur design separates coarse and fine branches based on information density. The coarse branch uses scene-level point clouds to generate virtual images and employs a memory module for spatially consistent heatmaps, critical for memory-dependent tasks. The fine branch processes localized, rapidly changing views focused on immediate interactions, where incorporating memory would disrupt spatial consistency.\n\n**Suggestion to refine our figures and clarify our writing and highlighting key results and impacts**\n\nWe thank the reviewer for their detailed suggestions and fully agree that their proposed improvements to the figures and manuscript will strengthen our paper. Due to ICML’s no-revision policy during rebuttal, we will incorporate these enhancements in future versions.\n\n**Why make the views independent in the memory architecture?**\n\nWe treat views independently in the memory architecture for two reasons. First, the MVT effectively integrates multi-view cues by processing each view both independently and dependently through attention layers. Second, consistent with SAM2’s design, avoiding cross-view dependencies prevents memory complications due to significant camera angle changes. This approach simplifies memory integration and enhances robustness and reliability.\n\nWe hope our responses have adequately addressed all of the reviewers' concerns. If so, we would greatly appreciate your consideration in raising your ratings.", "claims_and_evidence": "I think all the claims made are well-supported by the experiments. Namely:\n\n1. SAM2Act+ enables memory-dependent manipulation where baselines fail.\n2. SAM2Act is more robust across different environmental perturbations.\n3. SAM2Act outperforms existing policies on established baselines which don't involve memory.", "methods_and_evaluation_criteria": "The authors compare on both existing benchmarks and propose their own benchmark which relies on memory. The included baseline methods are appropriate (if somewhat limited) and the results are compelling.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Overall I liked found the experiments to be quite thorough, especially the ablations between SAM2Act and SAM2Act+. One additional ablation I would be curious about is the performance of incororporating memory without the SAM2 image encoder step. My intuition here is that these are actually orthogonal, and we could simply have the MVT take in an instruction + RGB images, output an embedding, perform cross-attention with old images & heatmaps (any embedding module could be used here), and then upsample the resulting final embedding. But I don't think this experiment is necessary for the paper to be valuable.", "relation_to_broader_scientific_literature": "Vision-Language-Action models are a promising research direction right now in robotics (Octo, $\\pi_0$, OpenVLA). These models have generally made markovian assumptions; i.e. the optimal action is independent of past observations given the current observation. This is of course not realistic for real-world autonomy. If a robot turns around, it shouldn't forget what's behind it. To the best of my knowledge, this is the first serious attempt to incorporate memory into these VLA models and comprises an important step for the field.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Weaknesses:\n1. It's a bit hard to understand the paper as-is without referencing the RVT-2 paper. The figures especially could maybe use some more labels and clarity. For instance, the three action heatmaps in Figure 3 are not labelled and I actually only saw a solid blue square until I looked closely.\n2. The baselines are a little lacking (e.g. Octo and related follow-up work).", "other_comments_or_suggestions": "1. I think some of the result summaries are a little editorialized. For example, line 436 states that SAM2Act+ outperforms SAM2Act by a \"huge\" margin of 37.6%. I find this to be stylistically a little off-putting and would prefer to let the (impressive) numbers speak for themselves. Or perhaps add a sentence emphasizing that the failure rate was \"~81% lower than that of SAM2Act and ~83% lower than the RVT-2 baseline,\" or \"less than one-fifth that of SAM2Act and the RVT-2 baseline.\" This is just my preference, and I think it's also fine if the authors want to keep the language as-is.\n2. Conversely, the abstract is dramatically understated: \"SAM2Act+ achieves competitive performance on MemoryBench.\" I'd suggest giving some concrete numbers here to spark reader interest.\n3. There's a point of confusion for me about how the higher-resolution embeddings from the SAM2 encoder are passed to the multi-resolution upsampling method in Stage 2. Namely, are the original embeddings of the RGB image passed through, ignoring the memory module? Or is the memory somehow involved here? Specifically, in Figure 3 I'm expecting an arrow from the SAM2 Image Encoder directly to the Multi-Resolution Upsampling block containing the higher-resolution embeddings, bypassing the memory attention and the MVT. I guess these were just left out for simplicity of the diagram?\n4. I'd suggest using bold font a little more judiciously in Section 5.", "questions_for_authors": "1. What are the cases where the SAM2Act+ still fails on MemoryBench? Is it due to failures in memory (opening the wrong drawer), or failure in execution (trying and failing to open the drawer)?\n2. I'm trying to understand the implications of the SAM2Act-fine module not having a memory component. Does that mean that a memory-enabled instruction like \"move the doll by grabbing the limb closest to the robot, then grab the teddy bear by the same limb\" would not be possible? I don't think this is a major issue if true but I want to check my understanding of the limitations here.\n3. Why make the views independent in the memory architecture? It seems that the memory attention module should have the capacity to use or ignore other views as appropriate.\n4. Can you speculate about why the robustness is so improved in Section 5.3? Is it the pre-trained knowledge from the SAM2 image encoder that helps distinguish semantic meaning despite visual perturbations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes SAM2Act, a mutli-view transformer based policy that leverages better visual representations from large-scale foundation model. The paper is well motivated with comprehensive experiments. Although there are some concerns from reviewers, authors did a good job addressing most concerns during rebuttal. Please incorporate all the comments (clarification, novelty, baselines, failure case analysis, etc.) in the revised version.
Decision: Accept (poster)