Exploring the Limits of Vision-Language-Action Manipulation in Cross-task Generalization
Authors: Jiaming Zhou
Ke Ye
Jiayi LIU
Teli Ma
Zifan Wang
Ronghe Qiu
Kun-Yu Lin
Zhilin Zhao
Junwei Liang
Venue: NeurIPS 2025 poster ()
Keywords: Vision-Language-Action
Cross-task Generalizable Manipulation
Primary area: applications
TL;DR: Proposing a zero-shot cross-task manipulation generalization benchmark and a novel generalizable VLA method.

Abstract: The generalization capabilities of vision-language-action (VLA) models to unseen tasks are crucial to achieving general-purpose robotic manipulation in open-world settings.
However, the cross-task generalization capabilities of existing VLA models remain significantly underexplored.
To address this gap, we introduce **AGNOSTOS**, a novel simulation benchmark designed to rigorously evaluate cross-task zero-shot generalization in manipulation. 
AGNOSTOS comprises 23 unseen manipulation tasks for test—distinct from common training task distributions—and incorporates two levels of generalization difficulty to assess robustness. 
Our systematic evaluation reveals that current VLA models, despite being trained on diverse datasets, struggle to generalize effectively to these unseen tasks. 
To overcome this limitation, we propose **Cross-Task In-Context Manipulation (X-ICM)**, 
a method that conditions large language models (LLMs) on in-context demonstrations from seen tasks to predict action sequences for unseen tasks.
Additionally, we introduce a **dynamics-guided sample selection** strategy that identifies relevant demonstrations by capturing cross-task dynamics. 
On AGNOSTOS, X-ICM significantly improves cross-task zero-shot generalization performance over leading VLAs, achieving improvements of 6.0\% over $\pi_0$ and 7.9\% over VoxPoser.
We believe AGNOSTOS and X-ICM will serve as valuable tools for advancing general-purpose robotic manipulation.

Exploring the Limits of Vision-Languagg-AAtion
Manipulation in Cross-task Generalization
Jiaming Zhou HKUST(GZ)
Ke Ye HKUST(GZ))
Jiayi Liu HKUST(GZ)
Teli Ma HKUST(GZ))
Zifan Wang HKUST(GZ))
Ronghe Qiu
Kun- Yu Lin The University of Hong Kong
Zhilin Zhao
HKUST(GZZ)
Sun Yat-sen University
Junwei Liang' HKUST(GZZ) and HKUST
Abstract
The generalization capabilities of vision- iition-language-cction (VLA) models to unseen tasks are crucial to achieving general-purpont robotic manipulation in open-world settings. However, the cross-task generalization capabilities of existing VLA models remain significantly underexplored. To address this gap, introduce AGNOSTOS, a novel simulation benchmark designed to rigorously evaluate zero- shot cross-task generalization in manipulation. AGNOSTOS comprises 23 unseen manipulation tasks for test, which are distinct from common training task distribu- tions, and incorporates levels of generalization difficulty to assess robustness. Our systematic evaluation reveals that current VI.A models, despite heing trained on diverse datasets, struggle to generalize effectively these unseen tasks. To overcome this limitation, we propose Cross- Task In-Context Manipulation (X- ICM), method that conditions large language models (LLMs) on in-context demonstrations from seen tasks to predict action sequences for unseen tasks. Addi- tionally, we introduce a dlynamics-uuided sample selection strategy that identifies relevant demonstrations by capturing cross-task dynamics. On AGNOSTOS. X- ICM significantly improves zero-shot cross-task generalization performance over leading VLA models, achieving improvements of 6.0% over To [1] and 7.9% over VoxPoser (2). believe AGNOSTOS and X-ICM will serve as valuable tools for advancing peeneral-purpors robotic manipulation. Project page: http:://jaming zhou, thbb..gitub.llo///NNSOOS
1 Introduction
10 have motivated new era of robotic manipulation by integrating visual perception, language understanding, and action generation. Through large-scale pre- pre-training on diverse data, including human videos [ [2]. real or simulated cross-embont coossemmodd robotic demonstrations [3] 14), VLA models can effectively generalize across visual variations within known tasks (i.e., within-tase generalization of seen tasks), such as handling objects in novel scenes or with altered properties. However, the true promise of VI.A models lies in their capacity to generalize across tasks: to handle previously unseen combinations of objects, goals, and actions without prior exposure. This capability, zero-shot cross-task generalization, is essential for real-world deployment, where robots are expected to tackle novel tasks they arise dynamically.
Vision-Language-Actinn (VLA) models 3
2
Despite the rapid progress in VLA research, most prior work generalization testing in real-world environments. However, these evaluations typically non- reproducible, and rarely target cross-task (i.c., unseen task) zero-shot generalization. Recently, many
has focused on
"Corresponding author:
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

AGNOSTOS: Cross-task shoo Generalization Benchmark
phone on basc
close laptop lid
lamp on
put books on bookshelf
opca grill
put ruhbish in bin
lamp off
put knite on board
close microware
toilet roll stand toilct down
close fridge
put umbrella stand
Level-1 Testing: unseen tasks, with similar objects motions
take off saucepan
buskethall in houp
straighten rope
beat the buzz
unplug charger
water plants
take ush out of computer
scoop with spatula
turn OVCI on
take plate off colored dish rack
Level-2 Testing: unseen tasks, no similar objects and motions
Human- Human-Video
Pross-Embodiment
RLBench In-Domain Training
Pre- Training
Robotic Data Pre- Training
Benchmarking existing different types of Vision-Language-aee models
The proposed AGNOSTOS benchmark evaluates zero-shot cross-task generalization through two difficulty levels. Level-1 testing involves 13 unseen tasks sharing partial similarity (objects or motions) with seen tasks. Level-2 testing has 10 unseen tasks from entirely novel scenarios, requiring stronger generalization capabilities. We systematically assess three broad categories of lision-language-actio action models, revealing critical limits in their ability to adapt to unseen tasks.
Figure 1:
have been devoted to developing comprehen- sive simulated benchmarks, which could be utilized to evaluate the generalization of VLAA models. While promising, these efforts mainly assess within-task generalization, leaving zero-shot cross-task generalization largely unexplored.
works 15
24
25
26
To address this critical gap, we present AGNOSTOS, a novel benchmark for evaluating zero-shot cross-task generalization in robotic manipulation. Built on RLBench [18), our benchmark comprises 23 unseen tasks that are carefully curated to differ from 18 commonly used seen training tasks (27 28 As shown in Figure|] T to probe different aspects of generalization, these unseen tasks are categorized into two difficulty levels:
Level-1 (13 tasks) shares partial semantics (e.g., similar objects like "cups" motions like "put") with seen tasks.
Level-2 (10 tasks) introduces entirely novel scenarios with no overlapping objects actions.
We benchmark three broad categories of VLA models:
1. Models 2 2 9
trained on large-scale real-world robotic demonstrations 29 built on multimodal large language models (MLLMs);
2. Models [30 3 32
pre-trained on large-scale human action video datasets M
3. Models 33
27
trained purely on in-domain RLBench data with advanced architectures.
Our empirical findings reveal key limitation: none of the existing models generalize effectively to unseen tasks, highlighting the need for approaches that directly address cross-task generalization.

---

Motivated by the success of in-context learning [5 36 37 in large language models (LLMs), we propose a Cross-task In-context Manipulation (X-ICM) method address the challenges of zero- shot cross-task generalization. X-ICM uses demonstrations from seen tasks as in-context examples to prompt L.MS to generate action plans for unseen tasks. A key challenge under this setup selecting relevant demonstrations; irrelevant prompts fail to activate appropriate knowledge in the I.M., leading to poor cross-task predictions. To address this, we introduce a dynamics-guide strategy. This approach leverages learned representations of task dynamics captured by predicting final observations from initial states and task descriptions- identify relevant demonstrations across tasks. Guided by the learned dynamics, the dynamics-awan prompts can be formed that enable X-ICM to elicit high-quality action predictions from LLMs, even for unfamiliar, unseen tasks.
sample selection
Our contributions are threefold:
We introduce AGNOSTOS, the first benchmark systematically evaluate zero-shot cross- task generalization in robotic manipulation for VI.A models, with 23 unseen tasks spanning two levels of generalization difficulty.
We propose X-ICM, novel method that combines in-context prompting and dynamics- guided sample selection to enhance zero-shot cross-task transfer in VLA models.
We conduct extensive evaluations of diverse VLA models on AGNOSTOS, uncovering fundamental limitations in current approaches and demonstrating the superior generalization of X-ICM.
2 Related Works
LwwgLage-gggteeeeneo Models. Generalizable robotic manipulation models [ 4 5
38
39 40
44. 50 [5], designed enable robots to understand instructions and interact with the physical world, have largely followed two main paradigms. The first involves modular-based approaches 52 53 54 55
2
43
44
45
46
47
48
57 58) where different MLLM components handle perception, language understanding, planning, and action execution. For example, VoxPoser 2 uses MLLMs synthesize composable 3D value maps for manipulation. The second paradigm focuses on end-to-end approaches [ 29 9 training a policy model to directly map raw sensory inputs (e.g.. vision, language instructions) robot actions. The success of these models heavily depends on the scale and diversity of training data, which comes from various data sources, including large-scale
56
human action videos [T] and large-scale cross-embodiment robotic data (2 [59] Based on the data, Open V VLA K is the first fully open-sourced work that significantly promotes the development of the VLA community. To [ proposes a VLM-based flow matching architecture, which is trained on a diverse dataset from multiple dexterous robot platforms.
Benchmarks for Dison-Language-Action Models. Evaluating the generalization capabilities of VLA models requires robust and comprehensive benchmarks. While existing VLA models mainly focus on real-world testing, which suffers from reproducibility issues and rarely focuses systematically on shot generalization to unseen tasks. Recently, many simulated benchmarks [15] 16 [7]18] 19
20 21 23
24, 25| 26 have been proposed to offer a controlled environment for rigorous evaluation.
Colosseum (24] enables evaluation of models across 14 axes of perturbations, including changes in color and texture, etc. GemBench (25 designs four levels of generalization, spanning novel placements, rigid and articulated objects, and complex long-horizon tasks. We find that these benchmarks have predominantly concentrated on evaluating visual variations within tasks, assessing how well models can generalize to new scenes or altered attributes of objects within the same task. However, these benchmarks do not focus on the more challenging aspect of zero-shot cross-task evaluation, where models must generalize to entirely new tasks with unseen combinations of object categorics and motions. This gap in evaluation limits our understanding of the true generalization capabilities of VLA models. Motivated by this, this work proposes the first zero-shot cross-task manipulation benchmark, expanding the evaluation scope of the generalization capabilities of VLA models.
In-context Learning with Large Language Models (LLMs). LLMs 60 35
65
66
have demonstrated remarkable capability known as in-context learning (ICL) 35 36
37,, where
they can learn to perform new tasks based on a few examples provided within the prompt, without requiring updates to parameters. The potential of ICL. is being explored in robotics 67 68
69 70
71 72). Prior works like KAT (77 and RoboPrompt [Z]] have shown that off-the-shelf L.I.Ms can
predict robot actions directly using within-task in-context samples. By extending their paradigms to
3

---

cross-task setting, this work specifically focuses on the zero-shot cross-task generalization problem. Our X-ICM model uses demonstrations from seen tasks as in-context examples to prompt LLMs for action generation in completely unseen tasks. Under the zero-shot cross-task setting, the selection crucial for a robust generalization. Our X-ICM method addresses this challenge by introducing a dynamics- guudedd sample selection strategy, ensuring that the selected seen demonstrations are relevant to the unseen tasks, thereby stimulating the cross-task generalization capabilities of LLMs.
of in-context samples 77] 74
75 76
3 AGNOSTOS: Zero-shot Cross-task Generalization Benchmark
To systematically assess the zero-shot cross-task generalization ability of vision- sasion-lnguuac-action (VLA) models, we introduce AGNOSTOS, reproducible benchmark built upon the RLBench simulation environment. AGNOSTOS features 18 seen tasks for training and 23 unseen tasks for rigorous generalization testing.
Training. For training, we adopt the standard set of 18 RLBench tasks that are widely used in prior
work 27 28 Examples of these seen tasks are shown in Figure A1 We collect 200 language-
conditioned demonstrations per task, resulting in 3600 demonstrations in total. These demonstrations enable VLA models to be fine-tuned to reduce the domain and embodiment gaps between pre-training data and RLBench data.
Testing, As illustrated in Figure I AGNOSTOS comprises 23 held-out unseen tasks with semantics that are disjoint from the seen set (videos of all tasks are available the Supplementary Materials). We categorize the unseen tasks into two difficulty levels. Level-1: 13 tasks that share partial semantic similarity with seen tather in object types (e.g.. cups) or motion primitives (e.g., stacking). Level-2: 10 tasks that exhibit no overlap in either object categories or motion types, requiring broader compositional reasoning and semantic extrapolation. Details on task curation and difficulty categorization are provided in Section A1.1 of the Appendix. For each unseen task, we perform three test runs with different seeds, each consisting of 25 rollouts, and report the mean and standard deviation of success rates.
To explore generalization boundaries, AGNOSTOS evaluates three broad families of VLA VLARmodels:
1. Foundation VLA models: trained on large-scale real-world cross-embodiment robotic data (29 or built upon LLM or VLM models, including Vpe 8 RDT
9 T
LLARVA 77., SAM2Act [4,, 3D-LOTUS++ [25] and VoxPoser 2
2. Human-video VLA models: pre-trained on large-scale human action videos to capture rich buman-object interactions for downstream robotic fine-tuning, including R3M [30].
D4R 3T, R3M-Align
33), and A4R-Align [32]
3. In-domain VLA models:
trained from scratch on RLBench's 's 18 seen tasks with task- specific model architectures. These serve as strong baselines without domain mismatch. including PerAct (27), RVT (28), RVT2 [33], Sigma-Agent (78), and Instant Policy [69].
To ensure a fair comparison, we fine-tune all models on the same 18 seen tasks when the models involve the embodiment gaps, following the official protocols of each method. A detailed description of the fine-tuning process and evaluation on AGNOSTOS is provided in Section/A1.22or the Appendix. Table [ compares AGNOSTOS with existing robotic manipulation benchmarks in terms of their support for cross-task evaluation, i.e., considering test tasks that have different object categories or motions from training tasks. While many benchmarks focus within-task visual generalization, few explicitly support zero-shot cross-task generalization test, and even fewer include tasks as challenging as Level-2 testing scenarios. Even benchmarks that include some form of cross- task evaluation often test on a narrow set of tasks, and typically evaluate only in-domain models, neglecting foundation and human-vides pre- pre-trained VLA models. AGNOSTOS fills this gap by supporting broad model coverage and introducing novel, difficult tasks in Level-2, providing a more comprehensive and diagnostic assessment of cross-task generalization.
We only consider evaluating the VIA models that are fully open-source.
We carefully categorize their test tasks into our defined two-level difficulty task sets.

---

Table 1: Comparison of manipulation benchmarks that focuses on cross-task generalization testing. The comparison evaluates whether each benchmark includes cross task testing, the types of VLA models evaluated, and the number Level-1 and Level-2 unseen tasks included.
Cross-task Zero-shot Gemeralization Test
No. of No. of train tasks
Benchmark
Simulator
test
Cross
Evaluated VLA models
Level-1
Level-2
tasks
lask
In-domain Human- video Foundation unseen tasks unseen tasks
RLBench- 18Task
27
RLBench
18
x
CALYIN
PyBullet
34
Culosseum
24
RLBench
20
20
X
VLMBench
23
RLBench
8
8
X
Ravens 80
Py Buller
(9
10
10
X
3
VIMA-Bench
Ravens
13
GemBench
25
RLBench
16
44
X
15
AGNOSTOS (Ours)
RI Bench
18
23
V
13
10
4 X-ICM: Cross-task In-context Manipulation Method
To push the boundaries of zero-shot cross-task generalization in vision-language-actionn (VLA) models, we propose method called Cross-task In-context Manipulation (X-ICM). Leveraging the cross-tast generalization capabilities of LLMs, X-ICM utilizes demonstrations from seen tasks as in-context examples. The dynamic characteristics of these examples are used prompt the LLM to predict action sequences for unseen tasks. In contrast to prior works [71] 67 that apply LLMs' in-context learning to within-task generalization, our work is the first to extend this paradigm to a zero-shot cross-task setting. A central challenge in this setting is that the selection of in-context demonstrations significantly affects generalization performance. To address this, we design a dynamics-guidee sample selection module that measures similarities between dynamic representations of seen and unseen tasks to guide the selection process, resulting in improved cross-task generalization.
4.1 Problem Definition and Method Overview
We tackle the problem of zero-shot cross-task robotic manipulation by exploiting the in-context learning ability of off-the-shelf LLMs. We assume access to a dataset of demonstrations from seen tasks, denoted as D = (V,", , L where N is the total number of seen demonstrations, 0,TT] }, and L; are the visual observation sequence, action
V," {or sequence, and language description of the i-th seen demonstration (T the sequence length, which varies for each demonstration). For an unseen task with the given initial visual observation v", and language description L", our zero-shot cross-task manipulation method can be formulated follows:
" }, A {a
A Aired fat... ..k LLM v L.)),
where Fael is the cross-task in-context demonstration selection process, P is the cross-task in-context prompt construction process, and Awred the predicted action sequence for the tested unseen task.
Figure 2 outlines our X-ICM framework, which comprises two modules. The Dynamics-guided Sample Selection module introduces a dynamics diffusion model, and then retrieves effective demonstrations based dynamic similarities between seen and unseen demonstrations. The Cross- task In-context Prediction module constructs the LLM prompt using retrieved demonstrations to predict action sequences for the unseen task. Next, describe each module in detail below.
4.2 Dynamics-guided Sample Selection module
Selecting effecttive in-context examples is essential for achieving robust cross-task generalization. We propose a two-stage Dynamics-guinn guidee Sample Selection module that learns dynamic representations from demonstrations using diffusion models, and retrieves examples that most relevant to the current unseen task based on the similarities between the dynamic representations of demonstrations.
Diffusion- based dynamics modeling stage. To effectively capture the dynamic representations within each demonstration, we train dynamics diffusion model G on all seen demonstrations. For each demonstration (e.g., the i-th), the dynamics diffusion model G takes the initial visual observation
5

---

Dynamics-guided Sample Selection module
close jar
18 seen tasks 3.6k demos
Dynamics Diffusion
initial obs
final obs
in-context demos (seen tasks)
Dynamics
feature similarity
retrieve from seen demos
Diffusion
wuter plant
object extraction
object & key-action extraction
(pplant: [60, 70, 49].
'target': [50, 33, 15). 'block', 152, 50, 19]; [[53,
'waterer:: [19, 64, 26]
57, 17 36, 53, 0 550, 33, 15, 36, 53, 0]]
unseen task
In-context prompt
Robot
You Franka Panda robot with parallel gripper. We provide you with some demos from some seen tasks, the format [task instruction, observation] [[7-imm action 1, dim action 2, 7-dim action TJ. Then you will receive an unseen task instruction with a observation, and you need to output list of 7-dim actions that match the trends the demos. Do output anything else. TL 10,,0f 11 [A] A5.
I [O), Oz. --- 11-14 ... A3x
[unssonn 10',0- 1
O LLM
Cross-task In-context Prediction module
Figure 2: X-ICM Method Overview. X-ICM employs dynamics-guider sample selection module retrieve effective demonstrations from seen tasks for each tested unseen task. These demonstrations are then used by the cross-task in-context prediction module to construct the prompt that drives the LLM to predict the corresponding action sequence.
" and language description L as inputs, and predicts the future observation that matches the final visual observation . The model G is initialized from InstructPix2Pix [82 and is optimized by:
min E.,,e-Wio.n [||c co(vf, 2, of,1, 11
(2)
where is the diffusion timestep, is the noise, of,r. is the noised final observation, and eg is the noise predictor in model G. By predicting the future, the inherent dynamics of each demonstration can be effectively modeled. Figure A6 shows the generation results of our dynamics diffusion model.
Dynamics-guided retrieving stage. Once trained, use & extract dynamic feature fü for the i-th seen demonstration:
f = pis J ps.lang] = G I. L2) E IR 2x 1024
(3)
where f p..
E R 1024 is the predicted latent feature of the target visual observation, f f..lang E R 1024 is the textual feature of the language deseription. For the unseen task, we similarly compute its feature f Then, we compute cosine similarities between features and select the top- seen demonstrations with the highest similarity:
f" f
R,
iE (1, NJ,
(4)
Iute arg topk (1..... (R.),
(5)
where RS represents the cross-task dynamics similarity score between the tested unseen task and the i-th seen demonstration. And the set Lidr contains the indices of K seen demonstrations that achieve the highest similarity scores. These seen demonstrations are retrieved to construct the cross-task in-context prompt.

---

4.3 Cross-task In-context Prediction module
For tested unseen task, we use the obtained K most-relevant seen demonstrations to construct the in-context prompt for cross-task action prediction with LLMs. Our Cross-task In-context Prediction module follows the practices in existing within-task in-context manipulation methods [7 67 but we are the first to extend this paradigm to the zero-shot cross-task setting.
Textualizing key information. As shown at the bottom of Figure for each selected seen demon- stration, we extract the 3D positions of the centers of objects and the key-action sequence. For the 3D positions of objects' centers, we use GroundingDINO S detect the 2D positions of objects, and then use the depth, intrinsic, and extrinsic information of the camera to obtain the 3D coordinates of objects' centers under the robot-base coordinate system. Additionally, we evenly divide the workspace into 100 x 100 x 100 grids, and then normalize the 3D coordinates of the objects' centers into the grid's coordinate system. Thus, for the m-th object O in the k-th seen demonstration, we textualize it as O "objectname: [x, y. =1" (e.g., "block: [52, 50, 19]"). For the key-action sequence, following (44, we select the actions when the gripper state changes or the joint velocities are near zero. In this way, the execution process of demonstration can be determined by several key-actions. We use the 7-dimensional state of the end-effector to represent the robot action, i.e., the 3D positions, the roll-pitch-yaw angles, and the gripper's open/close. The 3D positions of the end-effector are also normalized into the grid's coordinate system of the workspace. And the angles of end-effector are discretized into 72 bins, with each occupying 5 degrees. Thus, for the t-th key-action A in the k-th seen demonstration, we textualize it as A "[æ, y, 2, roll, pitch, yaw, gripper]" (e.g.. "[53, 57, 17, 0, 36, 53, 0l").
Constructing in-context prompt. With the above textualization of object and key-action information, for each seen demonstration (e.g., the k-th), we can textualize it as the mapping from its language description and object context to its key-action context:
"[ seee 10 ... .% ...... 3] [A] A/.
(6)
We textualize all K' selected seen demonstrations in this manner, and concatenate them in descending order based on their dynamics similarity scores to the tested unseen task. For the tested unseen task, we textualize it using its language description and object context, i.e. "[,anseen, {0² .................. Om O'....... ..... }] -" Finally, the cross-task in-context prompt is formed by con- catenating the system prompt, the textualized all seen demonstrations, and the textualized tested unseen task (see the bottom of Figure 2 and example in Figure A7 of the Appendix). This prompt will serve as the input of LLMs, which incentivizes the cross-task generalization capability of LLMss to predict the key-action sequence for the tested unseen task.
Experiments
Implementation Details. For our X-ICM method on the AGNOSTOS benchmark, we use a total of N = 3600 seen demonstrations. During in-context prompt construction, we select K = 18 demonstrations. We mainly use off-the-shelf Owen2.5-Instruet [65] models with 7B and 72B parameters, referred to as X-ICM (7B) and X-ICM (72B), respectively. These are deployed using two or eight A6000 GPUs For fair comparison with existing zero-shot baselines (e.g.. VoxPoser (2), in simulation we use the ground-trutt positions of objects. Ablation on using different sizes of LLMs is presented in Sec A2.2 of the Appendix.
5.1 Benchmarking Vision- Lssion-Language-Actinn Models
Table the zero-shot cross-task performance of our X-ICM models on 23 unseen tasks from the benchmark, including 13 Level-1 and 10 Level-2 tasks. These tasks are designed to evaluate the generalization ability of VLA models under varying levels of difficulty. We compare our models against a diverse set of baselines, including models trained on in-domain RLBench data, human videos, and LLM/VLM- foundations. Key observations include:
X-ICM (7B) and X-ICM (72B) achieve average success rates of 23.5% and 30.1 respec- tively, outperforming all existing VIA models.
On Level-1 tasks, X-ICM (7B) surpasses the prior SoTA To 1 by 6.9%. For Level-2 tasks, performance gains are more pronounced with the 72B model.
7

---

Table 2: Cross-task zero-shot manipulation performance on 23 unseen tasks, where the column headers show the abbreviation of each unseen task (see full task names in TaTle[AT] N/A indicates that the tested tasks are removed since they overlap with the training tasks of the methods. Level-1 and Level-2 represent tasks with two different difficulty levels. The prefix indicates second best.
Level-1 tasks
methods
Tollet
Knife Fridge Microwave Laptop
Phone
Seat
LampON LampOn Book Umbrella Grill
PerAct 27
0.0
5.3
37.3
64.0
2.7
0.0
72.0
0.0
1.3
0.0
1.3
8.0
54.7
in-donuin taining
RVT 28
0.0
2.7
50.7
26.7
50.7
2.7
40.0
0.0
1.3
0.0
1.3
0.0
6.7
Sigma-Agent
78
0.0
9.3
56.0
9.3
30.7
1.3
65.3
1.3
0.0
0.0
0.0
1.3
4.0
RVT2 33
0.0
1.3
0.0
17.3
42.7
1.3
62.7
2.7
1.3
0.0
1.3
5.3
34.7
Posiay [69
0.0
1.3
13.3
4.0
4.0
18.7
24.0
0.0
0.0
0.0
0.0
0.0
0.0
bumnan vidce pretaining
D4R [3]
0.0
8.0
32.0
30.7
24.0
0.0
65.3
20.0
4.0
0.0
0.0
0.0
0.0
R3M 30
0.0
0.0
37.3
22.7
25.3
1.3
62.7
6.7
4.0
0.0
0.0
0.0
0.0
D4R- Align 32
0.0
2.7
45.3
74.7
24.0
0.0
41.3
0.0
0.0
1.3
0.0
0.0
0.0
R3M -lign
32
0.0
4.0
49.3
25.3
21.3
0.0
49.3
0.0
5.3
0.0
0.0
1.3
1.3
Open 8
0.0
5.3
38.7
40.0
57.3
0.0
53.3
12.0
1.3
0.0
10.7
0.0
VLA foundations
KDT
0.0
0.0
46.7
13.3
14.7
0.0
30.7
0.0
0.0
1.3
0.0
8.0
0.0
TO
0.0
5.3
85.3
24.0
40.0
1.3
64.0
18.7
8.0
1.3
0.0
33.3
1.3
LLARVA
0.0
0.0
12.0
0.0
6.7
0.0
40.0
0.0
0.0
0.0
0.0
0.0
0.0
3D.1 .OTUS
25
0.0
6.7
N/A
N/A
N/A
0.0
6.7
0.0
0.0
0.0
0.0
13.3
5.3
3D-LOTUS++ 25
0.0
5.3
N/A
N/A
N/A
9.3
68.0
10.7
0.0
0.0
0.0
29.3
13.3
SAM2Act 34]
0.0
0.0
36.0
40.0
6.7
6.7
62.7
6.7
0.0
1.3
1.3
9.3
0.0
VoxPoser 2
0.0
0.0
0.0
0.0
5.3
8.0
28.0
88.7
25.3
0.0
0.0
0.0
82.7
X-ICM (7B)
1.3
26.7
22.7
45.3
33.3
57.3
48.0
58.7
50.7
1.3
0.0
8.0
18.7
X-ICM (72B)
6.7
69.3
12.7
58.7
34.0
68.0
51.3
86.7
74.7
2.0
1.3
5.3
18.7
Level-2
tasks
Level-1
Level-2 avg (std)
All
methods
USB
Lid
Plate
Ball
Scoop
Rope
Oven
Bucz
Plants
Charger
avg (std)
avg (std)
PerAct 27
58.7
2.7
0.0
0.0
0.0
0.0
1.3
4.0
6.7
2.7
(1.4)
(1.)
14.0 (0.9)
in donainn training
RVT 28
0.0
4.0
8.0
5.3
11.3 (1.4)
12.8 (0.2)
Sigma- Agent
7%
88.0
0.0
0.0
0.0
0.0
0.0
4.0
8.0
5.3
1.3
13.7 (1.6)
10.7 (1.7)
12.4 (0.4)
RVT2 33
22.7
40.0
0.0
0.0
0.0
0.0
0.0
1.3
1.3
1.3
13.1 (0.4)
(1.3)
10.3 (0.6)
InstaniPolicy 669
26.7
1.3
0.0
0.0
0.0
0.0
0.0
1.
0.0
0.0
4.3 (4.2)
(1.4)
3.7 (3.0)
human viles
preteaizing
D4R [31
98.7
0.0
0.0
0.0
0.0
0.0
1.3
1.3
1.3
14.1 (0.3)
10.7 (0.2)
(2.6)
R3M 30
48.0
0.0
0.0
0.0
0.0
8.0
2.7
2.7
1.3
12.3
6.3 (0.9)
9.7 (0.6)
124R-Abgn Abin 32
89.3
1.3
0.0
0.0
0.0
0.0
8.0
6.7
0.0
1.3
14.5 (1.0)
10.7 (0.2)
12.8 (0.6)
R3M-Aligm [32
90.7
0.0
1.3
0.0
0.0
0.0
2.7
13.3
4.0
0.0
12.9 (0.7)
11.2 (0.7)
12.2 (0.3)
OpenVLA [
0.0
0.0
0.0
0.0
6.7
5.3
2.7
16.9 (1.3)
9.2 (0.7)
13.6
VLA foundalions
RDT [)]
100.0
29.3
0.0
0.0
0.0
8.0
2.7
0.0
10.4 (0.5)
14.4 (0.9)
12.1 (0.4)
To 11
97.3
0.0
1.3
0.0
0.0
0.0
14.7
5.3
1.3
0.0
(0.4)
12.0 (0.9)
117.5 (0.4)
LLARVA [77]
24.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
4.5 (0.1)
2.4 (0.0)
3.6
3D-LOTUS 25
85.3
0.0
1.3
0.0
0.0
0.0
0.0
4.0
0.0
3.2 (0.5)
9.1 (0.7)
6.2 (0.5)
LOTUUS+ 25
30.7
0.0
8.0
8.7
6.7
(1.6
15.1 (1.1)
14.4 (1.0)
SAM2Act
34
920
49.3
0.0
0.0
0.0
0.0
1,3
6.7
4.0
5.3
13.1 (0.4)
14.0 (1..9
VoxPoser 2
32.0
76.0
0.0
8.0
0.0
0.0
0.0
1,3
0.0
4.0
18.1 (0.4)
12.1 (0.4)
(0.2)
Ours
X-ICM (7B)
98.7
20.0
9.3
0.0
6.7
16.0
27
5.3
(1.9
16.9 (1.3)
(3.6
X-ICM (728)
13.3
36.0
0.7
20.7
7.3
2.7
37.6 (1.4)
20.3 (1.7)
30.1 (1.0)
While some VLA foundation models show decent performance, particularly To on Level-1 1
tasks and SAM2Act [4V3DD-LTTS++ [25] on Level-2 tasks. they fall short of the broad generalization capabilities demonstrated by our X-ICM models.
All prior models completely fail on at least eight of the 23 tasks. In contrast, X-ICM (7B) fails on only two, and X-ICM (72B) succeeds on all.
5.2 Ablation Studies
Dynamics-guided sample selection. We assess the impact of the dynamics- -uiided--uieedd sample selection module by comparing X-ICM (72B) with and without it (denoted as w/o sel, using random sampling) at the tep-left of Table 3 Incorporating the module notably boosts performance and robustness across both task levels. This confirms its effectiveness in identifying informative demonstrations that enhance the L.MM's cross-task generalization ability. Further analysis, including visualizations from the dynamics diffusion model and comparisons of dynamic features, provided in Appendix A2.
8

---

Table 3: Effects of dynamics-guided sample selection mod- and different model sizes.
12 number in-comlext somples
Models
Level-1
Level-2
All
35
32
X-ICM (72B) w/o sel
30.7 (4.7)
(2.2)
25.2 (3.2)
X-ICM (72B)
37.6 (1.4)
20.3 (1.7)
30.1 (1.0)
21.2
Table Rifect using different LLM models.
LLMs
Level-1
Level-2
All
20
21.3
15.5
20.5
20.2
19.5
Deepseek-R T-Disttll Qwen-Th
(1.7
7.9 (0.3) 9.5 (0.5)
Llama3.0 3.0-8B3.-------- B--Si--rrru
Instruct
(0.7)
11.7 (1.6) 15.1 (0.3)
Level-1
Level-2
All
Ministral-Be -aaaal----Baa 1t--Instruct-2100
22.9 (0.7) 14.8 (0.3) 19.5 (0.5)
Figure 3: The effect of varying the number of in-context demonstrations.
InternLM3--8 M3--BBInsttucct
(0.7) 13.3 (0.4) 21.8 (0.5)
Qwwen2.5-7BBie -nntttttett
(8.6 16.9 (1.3)
23.5 (1.6)
Effect of the number of in-context demonstrations. We explore how performance varies with the number of in-context demonstrations using the Qwen2.5-72B-Inse Insstrcct model. Figure [ shows that performance improves rapidly the number increases from to 12, after which it plateaus. This suggests that a moderate number of relevant demonstrations is critical. Too many may introduce irrelevant noise and diminish gains.
Comparison of LLM backbones. Table compares the performance X-ICM (7B) when replacing Qwen2.5-7B-Insee Instrructt with alternative 7B/8B models, including Deepseek-R1 -iii------itiiieeeeee [55 Llama3.0-81 Instrutttt [86), -umistra-8B--nsttuutt-
[87), and Internl.M3-8B--nnttn (88]. The results show significant variability in performance, underscoring the importance of choosing powerful and well-aligned LLM backbone.
5.3 Real-world Experiments
To evaluate real-world applicability, we test X-ICM on five physical manipulation tasks using an xArm7 robot arm, DH-Robotics gripper, and a third-person Orbbec camera: put block into bin, push button, puf bottle into box, stack blocks, and stack cups. We collect 20 demonstrations for each of the five tasks. During testing, we use the demonstrations from the other four tasks construct the cross-task in-context prompt, enabling zero-shot generalization. The Qwen2.5-72B III-rrutt LLM is used as the backbone and the number of seen demonstrations is set to 18. Each task is executed 20 times, and we report average success rates. Results shown the bottom of Figure demonstrate the strong real-world zero-shot cross-task performance of our approach. We provide additional details in Section A3 of the Appendix, and showcase some testing videos that include both successful and failed cases in the Supplementary Materials.
In addition, we also evaluate the proposed X-ICM model on long-horizon tasks under the zero-shot cross-task setting. We use tasks put block into bin, push button, put bottle into box, stack blocks, and pull out the middle block as seen tasks, where we have 10 demonstrations for each task. The long-horian long-horizon task we evaluated clean the table, according the robot's visual observation, we decompose the long-horizon task into three sequential sub-tasks, ie, stack cups, put the stacked cups into plate, place the mango on top of the stacked cups. We evaluated this task over 20 rollouts, with sub-task success contingent the success of all prior sub-tasks. For our X-ICM method, the success rates on sub-tasks "stack cups", "put the stacked cups into plate", "place the mango on top of the stacked cups." are 40%, 25%. and 5%, respectively. Thus, the overall success rate for the oogghorizzon task 5%. These results align with expectations, as the sequential nature long-horizon tasks leads cumulative failure. The cross-task zero-shot setting exacerbates these difficulties. These findings highlight the need for more advanced algorithms to handle oong-horizon tasks zero-shot scenarios.
put block into bin
push button
put bottle into box
stack blocky
stack cups
Success: 70%
Success: 10%
Success: 30%
Success: 50%
Success: 30%
Figure 4: Results of five real-world tasks. The tests are conducted in a zero-shot cross-task manner.

---

6 Conclusions and Discussions
Conclusions. In this work, we have introduced AGNOSTOS, a benchmark for evaluating zero-shot cross-task generalization in robotic manipulation. With 23 unseen tasks across two difficulty levels, AGNOSTOS provides a rigorous testbed for assessing the limits of Vision- Laaioo-Lnggage--Aaee
(VLA) models. Our evaluation of diverse VLA models reveals their significant limitations in unseen task generalization. To address this, we propose X-ICM, a cross-task task in-context manipulation method that leverages the cross-task generalization capabilities of L.MMs. The X-ICM achieves substantial improvements over existing VIA models, demonstrating robust zero-shot cross-task generalization.
Discussions. While X-ICM significantly enhances zero-shot cross-task generalization, its perfor- mance on many unseen tasks, particularly those with both novel objects and motion primitives, remains limited due to LLMs' challenges in extrapolating beyond pre-training data and in-context demonstrations. In addition, the use of visual information is limited to textualizing object infor- mation, which may ignore important visual context in the raw data. To address these limitations, future work could integrate multi-modal reasoning, combining vision, language, and action data to improve generalization to novel semantics. Additionally, leveraging generalizable concepts like object trajectories as a bridge between MI.LMs' perception and dynamic action sequence prediction could reduce cross-task generalization complexity. Scaling X-ICM to diverse robotic embodiments would further enhance its versatility across varied platforms and sensor configurations. We believe the proposed X-ICM benchmark and X-ICM method will inspire future research in generalizable robotic manipulation, paving the way for robots that can seamlessly adapt to open-word environments.
Acknowledgements. This work was supported by the National Natural Science Foundation of China (No. 62306257), the Guangzhou Municipal Science and Technology Project (No. 2024A03J0619 and No. 2024A04J4390) and the Goungzhou-HKUSTTGGG Joint Funding Program (Grant 20.3003403)00088, Education Bureau of Guangzhou Municipality.
References
[1] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. No: vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24166 2024.
[2] Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Fei-Fei. Voxposer: Composable value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023.
[3] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding. Danny Driess. Avinava Dubey, Chelsea Finn, et al. Rt-2: dasion-laguaag-actinn models transfer web knowledge robotic control. arXiv preprint arXiv:2307.15818 2023.
[4] Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Jocy Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXi::4455. 120.1123.,. 2024.
[5] Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, Chuang Gan. 3d-vla: A 3d ssiion-angguae generative world model. arXiv preprint arXiv:2403. 09631, 2024.
[6] Jonathan Yang, Catherine Glossop, Arjun Bhorkar, Dhruv Shah, Quan Vuong, Chelsea Finn, Dorsa Sadigh, and Sergey Levine. Pushing the limits of cross-embodiment learning for manipulation and navigation. arXiv preprint arXiv:2402. /9432, 2024.
[7] Cheng Chi, Siyuan Feng. Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.0411 2023.
[8] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv 2rXiv:2406.0922 096444466, 2024.
[9] Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su. and Jun Zhu. Rdt-Ib: lb: a diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.00 078644 2024.
10

---

[10] Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montserrat Gonzalez Arenas, Travis Armstrong. Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing ai into the physical world. arXiv preprint arXiv:2503.20020 2025.
[11] Kristen Grauman, Andrew Westbury. Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, al. Ego4d: Around the world in 3,000 hours of egacentric video. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 18995-19012, 2022.
[12] Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafy Afouras. Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19383-19400, 2024.
[13] Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Dani- ilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109. 13396, 2021.
[14] Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding. Siyuan Feng, Shenyuan Gao, Xindong He. Xu Huang, Shu Jiang, et al. Agibot world colosseo: large-scale manipulation platform for scalable and intelligent embodied systems, arXiv preprint arXiv:2503 06669, 2025.
[15] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Raozheh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEER/CVF conference on computer vision and pattern recognition, pages 17749-10749,... 2020.
[16] Tongzhou Mu, Zhan Ling. Fanbo Xiang. Derek Yang, Xuanlin Li, Stone Tao, Zhiao Huang, Zhiwei Jia, and Hao Su. Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations, arXiv preprint arXiv:2107. 14483, 2021.
[17] Oier Mees, Lukas Hermann, Erick Rosete- Beate and Wolfram Burgard. Calvin: A benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 733)7327-7333, 2022.
[18] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019- -026999--2,2--0 2020.
[19] Bo Liu, Yifeng Zhu, Chongkai Gao. Yihao Feng. Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36:44776-4479,, 2023.
[20] Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu, Robocasa: Large-scale simulation everyday tasks for generalist robots. arXiv preprint arXiv:2406.022 025233 2024.
[21] Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martin-Martin. Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior- lk: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. Conference on Robot Learning, pages 80-93. PMIRR 2023.
[22] Shiduo Zhang, Zhe Xu, Peiju Liu, Xiaopeng Yu, Yuan Li, Qinghui Gao, Zhaoye Fei, Zhangyue Yin, Zuxuan Wu, Yu-Gang Jiang. al. Vlabench: A large-scale benchmark for seaeguge--onnitionnn robotics manipulation with long-horizon reasoning tasks. arXiv preprint arXiv:2412.18194 2024.
[23] Kaizhi Zheng, Xiaotong Chen, Odest Chadwicke Jenkins, and Xin Wang. Vimbench: A compositional benchmark for vision-and lnn-uane manipulation. Advances in Neural Information Processing Systems, 35:665 351665-678, 2022.
[24] Wilbert Pumacay, Ishika Singh, Jiafei Duan, Ranjay Krishna, Jesse Thomason, and Dieter Fox. The colosseum: benchmark for evaluating generalization for robotic manipulation. arXiv preprint arXiv:2402.0819), 2024.
[25] Ricardo Garcia, Shizhe Chen, and Cordelia Schmid. Towards generalizable vision- language robotic manipulation: A benchmark and llm-gudded 3d policy. arXiv preprint 0rX1::2410.01345. 2024.
11

---

[26] Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian. Kangrui Wang, Qineng Wang. Teja Venkat Koripella, Marziych Movahedi, Manling Li, et al. Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. arXiv preprint 05000,2502.09560, 2025.
[27] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785-799. PMLR, 2023.
[28] Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, and Dieter Fox. Rvt: Robotic view transformer for 3d object manipulation. In Conference on Robot Learning, pages 994-710. PMLR, 2023.
[29] Quan Vuong, Sergey Levine, Homer Rich Walke, Karl Pertsch, Anikait Singh, Ria Doshi, Charles Xu, Jianlan Luo, Liam Tan, Dhruv Shah, et al. Open e-mbodiment: Robotic learning datasets and rt-x models. In Towards Generalist Robots: Learning Paradigms for Scalable Skill Acquisition CoRL2023, 2023.
[30] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203. 12601, 2022.
[31] Sudeep Dasari, Mohan Kumar Srirama, Unnat Jain, and Abhinav Gupta. An unbiased look datasets for visuo-motor pre-training. In Conference on Robot Learning, pages 1 183-1198. PMLR, 2023.
[32] Jiaming Zhou, Teli Ma, Kun-Yi Yu Lin, Zifan Wang, Ronghe Qiu, and Junwei I. Liang. Mitigating the human- robot domain discrepancy in visual pre-training for robotic manipulation. arXiv preprint arXiv:2406. 14235, 2024.
[33] Ankit Goyal, Valts Blukis, Jie Xu, Yijie Guo, Yu-Wei Chao, and Dieter Fox. Rvt-2: Learning precise manipulation from few demonstrations. arXiv preprint arXiv:2406.08854 08545, 2024.
[34] Haoquan Fang, Markus Grotz, Wilbert Pumacay, Yi Ru Wang, Dieter Fox, Ranjay Krishna, and Jiafei Duan. Sam2act: Integrating visual foundation model with memory architecture for robotic manipulation. arXiv preprint vxXiv:2501118544 2025.
[35] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners, 2020.
[36] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu,, Zhiyong Wu, Tianyu Liu, al. A survey on in-context learning. arXiv preprint arXiv:2301.002. 2022.
[37] Noam Wies, Yoav Levine, and Amnon Shashua. The learnailiity of in-context learning. Advances in Neural Information Processing Systems, 36:36637-36651, 2023.
[38] Qixiu Li, Yaobo Liang. Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, et al. Cogact: A foundational lision-language-acttoo model for synergizing cogmition and action in robotic manipulation. arXiv preprint arXiv:2411.1966 2024.
[39] Jiaming Liu, Mengzhen Liu, Zhenyu Wang. Pengju An, Xiaoqi Li, Kaichen Zhou, Senqiao Yang. Renrui Zhang, Yandong Guo, and Shanghang Zhang. Robomamba: Efficient vision-language-adee model for robotic reasoning and manipulation. Advances in Neural Information Processing Systems, 37:40085 11000085-11110 10, 2024.
[40] Jinming Li, Yichen Zhu, Zhibin Tang, Junjie Wen, Minjie Zhu, Xiaoyu Liu, Chengmeng Li, Ran Cheng. Yaxin Peng. and Feifei Feng. Improving vision-language-action models via chain-of-afforddace. arXiv preprint arXiv:2412.20451 2024.
[41] Junjie Wen, Yichen Zhu, Jinming Li, Minjie Zhu, Zhibin Tang, Kun Wu, Zhiyuan Xu, Ning Liu, Ran Cheng, Chaomin Shen, et al. Tinyvla: Towards fast, data-efficient lision-lnggage-acttio models for robotic manipulation. IEEE Robotics and Automation Letters, 2025.
[42] Delin Qu, Haoming Song. Qizhi Chen, Yuanqi Yao, Xinyi Ye. Yan Ding, Zhigang Wang, Jia Gu, Bin Zhao, Dong Wang, et al. Spatialvla: Exploring spatial representations for visual- lanuull-annuge-ccttion model. arXiv preprint ar 250: 15830, 2025.
[43] Zhongyi Zhou, Yichen Zhu, Minjie Zhu, Junjie Wen, Ning Liu, Zhiyuan Xu. Weibin Meng, Ran Cheng. Yaxin Peng, Chaomin Shen, et al. Chatvla: Unified multimodal understanding and robot control with vision-language-actiion model. arXiv preprint 2502v:2502. 1122./444 14420, 2025.
[44] Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision- language action models, arXiv preprint arXiv:2501.0972 2025.
12

---

[45] Yifan Zhong, Xuchuan Huang, Ruochong Li, Ceyao Zhang, Yitao Liang. Yaodong Yang. and Yuanpei Chen. Dexgraspvla: lssion-anguaage-atioo framework towards general dexterous grasping. arxiv preprint arXiv:2502 20900, 2025.
[46] Jian-Jian Jiang. Xiao-Ming Wu, Yi-Xiang He, Ling-An Zheng. Rethinking himanual robotic manipulation: Learning with decoupled interaction framework. In International Conference on Computer Vision, 2025.
An Zeng, Yi-Lin Wei, Dandan Zhang. and Wei-Shi
[47] Jian-Jian Jiang, Xiao-Ming Wu, Zibo Chen, Yi-Lin Wei, and Wei-Shi Zheng. igrasp: An interactive 2d-3d framework for 6-dof grasp detection. In International Conference on Pattern Recognition, 2024.
[48] Zhi Hou, Tianyi Zhang, Yuwen Xiong, Haonan Duan, Hengjun Pu, Ronglei Tong, Chengyang Zhao, Xizhou Zhu, Yu Qiao, Jifeng Dai, al. Dita: Scaling diffusion transformer for generalist vision-language-acioon policy. arXiv preprint 2rXiv:2503.1973 2025.
[49] Chengmeng Li, Junjie Wen, Yan Peng. Yaxin Peng, Feifei Feng, and Yichen Zhu. Pointvla: Injecting the 3d world into ssaionlanguage-actiinn models. arXiv preprint arXiv:2503.072 0.511,,.7111111 2025.
[50] Yangtao Chen, Zixuan Chen, Junhui Yin, Jing Huo, Pinzhuo Tian, Jieqi Shi, and Yang Gao. Gravmad: Grounded spatial value maps guided action diffusion for generalized 3d manipulation. arXiv preprint arXiv:2409. 20154, 2024.
[51] FIGURE. Helix: A saaoon-lnnuag--cttiin model for generalist humanoid control, 2025.
[52] Fangehen Liu, Kuan Fang, Pieter Abheel, and Sergey Levine. Moka: Open-vocahulary robotic manipulation through mark-based based visual prompting. In First Workshop on Vision-Languat Models for Navigation and Manipulation at ICRA 2024, 2024.
[53] Haoxu Huang. Fanqi Lin, Yingdong Hu, Shengjie Wang, and Yang Gao. Copa: General robotic ma- nipulation through spatial constraints of parts with foundation models. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9488 9498. IEEE, 2024.
[54] Yuelei Li, Ge Yan, Annabella Macaluso, Mazcyu Ji, Xueyan Zou, and Xiaolong Wang. Integrating Imm planners and 3d skill policies for generalizable manipulation. arXiv preprint arXiv:2501 2025.
[55] Xiaoqi Li, Mingxu Zhang, Viran Geng, Haoran Geng, Yuxing Long, Yan Shen, Renrui Zhang, Jiaming Liu, and Hao Dong. Manipllm: Embodied multimodal large language model for object-centric robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18061-18070, 2024.
[56] Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei- Rekep: temaiol reasoning of relational keypoint constraints for robotic manipulation. arXiv preprint arXiv:2409.01652 2024.
[57] Mingjic Pan, Jiyao Zhang, Tianshu Wu, Yinghao Zhao, Wenlong Gao, and Hao Dong. Omnimanip: Towards general robotic manipulation via object-eentric interaction primitives as spatial constraints. arXiv preprint Xiv::XX0 038441, 2025.
[58] Jiafei Duan, Wentao Yuan, Wilbert Pumacay, Yi Ru Wang, Kiana Ehsani, Dieter Fox, and Ranjay Kr- ishna. Kanipulate-anything Automating real-world robots using vision-language models. arXiv preprint arXiv:2406. 18915, 2024.
[59] Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch. Quan Vuong, Andre He, Vivek Myers. Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning (CoRL), 2023.
[60] Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
[61] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, et al. Training language models to follow instructions with human feedback, 2022.
[62] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners, 2019.
[63] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models, 2023.
13

---

[64] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain- of-thought prompting elicits reasoning in large language models, 2022.
[65] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, et al. Qwen2. technical report. arXiv preprint arXiv:2412.5115, 2024.
[66] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chenggi Deng. Chenyu Zhang, Chong Ruan, al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437. 2024.
[67] Norman Di Palo and Edward Johns. Keypoint action tokens enable in-context imitation learning in robotics. arXiv preprint arXiv:2403. 19578, 2024.
[68] Jiaqiang Ye Zhu, Carla Gomez Cano, David Vazquez Bermudez, and Michal Drozdzal. Incoro: In-context learning for robotics control with feedback loops. arXiv preprint 4rXiv:2402.05//8. 888 2024.
[69] Vitalis Vosylius and Edward Johns. Instant policy: In-context imitation learning via graph diffusion. In Proceedings of the International Conference Learning Representations (ICLR), 2025.
[70] Yecheng Jason Ma, Jocy Hejna, Chuyuan Fu, Dhruv Shah, Jacky Liang, Zhuo Xu, Scan Kirmani, Peng Xu, Danny Driess, Ted Xiao, et al. Vision language models are in-context value learners. In The Thirteenth International Conference on Learning Representations, 2024.
[71] Yida Yin, Zekai Wang. Yuvan Sharma, Dantong Niu, Trevor Darrell, and Roci Herzig. In-context learning enables robot action prediction in llms. arXiv preprint arXiv:2410. 2024.
[72] Letian Fu, Huang Huang, Gaurav Datta, Lawrence Yunliang Chen, William Chung- Ho Panitch, Fangchen Liu, Hui Li, and Ken Goldberg. In- imitation learning via next-token prediction. arXiv preprint arXiv:2408. 15980, 2024.
[73] Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning retrieve prompts for in-context learning. arXiv preprint
2021.
[74] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlmmoer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint vrXiv:2202.12837 2022.
[75] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What makes good examples for visual in-context learning? Advances in Neural Information Processing Systems, 36: 16:7731117794 2023.
[76] Yucheng Zhou, Xiang Li, Qianning Wang. and Jianbing Shen. Visual in-context learning for large vision-langual models. arXiv preprint 240. 2024.
(77] Dantong Niu, Yuvan Sharma, Giscard Biamby, Jerome Quenum, Yutong Bai, Baifeng Shi, Trevor Darrell, and Roci Herzig, Llarva: Vision-action instruction tuning enhances robot leaming. arXiv preprint arXiv:2406. 1 11855 2024.
[78] Teli Ma, Jiaming Zhou, Zifan Wang, Ronghe Qiu, and Junwei Liang. Contrastive imitation learning for language-guided multi-task robotic manipulation. arXiv preprint arXiv:2406.09738, 2024.
[79] Erwin Coumans and Yunfei Bai. Pybullet, python module for physies simulation for robotics, games and machine learning. httt::/pybullet..og org.
[80] Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning, pages 746-747. PMLR, 2021.
[81] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu and Linxi Fan. Vima: General robot manipulation with multimodal prompts. Fortieth International Conference on Machine Learning, 2023.
[82] Tim Brooks, Aleksander Holynski, and Alexci A Efros. Instractpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 18392-18402, 2023.
[83] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang. Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference Computer Vision, pages 38 S5. Springer, 2024.
14

---

[84] Stephen James and Andrew Davison. Q-attention: Enabling efficient learning for vision-based robotic manipulation. IEEE Roboticy and Automation Letters, 7(2):1612-161, 2022.
[85] Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang. Xiao Bi, al. Deepseek-rl: Incentivizing reasoning capability in ilms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
[86] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
[87] Albert Q. Jiang. Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Maric-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William EI Sayed. Mistral 7b, 2023.
[88] Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internim2 technical report. arXiv preprint arXiv:2403. 1....77299 17297, 2024.
[89] Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection. Advances in Neural Information Processing Systems, 36:72983 73007, 2023.
[90] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson. Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015-4026 2023.
[91] Hu Xu, Gargi Ghosh, Po- Yao Huang. Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre- pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2 109. http:://19./4009.0 14084, 2021.
[92] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[93] Maxime Oquab, Timothée Dareet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07119. 2023.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately reflect the main contributions and scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the Discussions Section, we discuss the limitations of this work, and demonstrate some promising directions for future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose all the information needed to reproduce the main experimental results the Method, Experiment, and Appendix sections. We will release the code upon acceptance of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
17

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: We commit to releasing the code, data, and models upon acceptance of paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cotee//mdeeeSbmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the training and test details necessary understand the results in the Method, Experiment, and Appendix sections.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We show the standard deviation of the tasks' success rates.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information on the computer resources is provided in Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips. o
Answer: [Yes]
Justification: We adhere to the NeurIPS Code of Ethics, since the paper does not include any content or practices that violate ethical guidelines
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential societal impacts of the paper in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not involve data models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper properly credits the creators original owners of all used assets, and properly respects the license and terms of use.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
datasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our new benchmark is built upon the existing RLBench environment. We provide details about our benchmark in the Benchmark section and Appendix. We will release the data for the benchmark upon acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: The task categorization process is conducted by GPT4o. The results are verified by three human experts. The verification only takes about 10 minutes. According to our country's policy, human experts have received decent remuneration.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There are no risks to human subjects, and no need for Institutional Review Board (IRB) approvals.
Guidelines:
21

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM for what should or should not be described.
22

---

Appendix
A1 AGNOSTOS Benchmark
A1.1 Task Selection and Categorization
The AGNOSTOS benchmark is built upon the RLBench simulated environment, which includes 100 pre-defined manipulation tasks. Existing robotic manipulation studies typically a subset of 18 RLBench tasks for training and close-set testing. Figure [ illustrates these 18 tasks, which we refer seen tasks. To rigorously assess the generalization capabilities of rision-lngguage-action (VLA) models on novel tasks, we select 23 unseen tasks from RLBench with distinct semantics compared to the seen tasks. Figure visualizes these 23 unseen tasks. In the Supplementary Materials, we provide video examples of all seen and unseen tasks.
put drawer
reach and drag
turn tap
slide block color target
open drawer
put groceries cupboard
the item middle drawer
the stick drag cube onto yellow target
left tap
slide the block green target
open the bottom drawer
the coffee in the cupboard
place shape in shape sorter
put money in safe
push buttons
close jar
stack blocks
place cups
the cube in the shape sorter
the money away in the safe the middle shelf
push the maroon button, them push the green button, then push the purple button
close the silver jar
stack 3 white blocks
place cups on the cup holder
place wine at rack location
light bulb in
sweep to dustpan size
insert onto square peg
meat off grill
stack cups
stack the wine bottle to the middle
in the light bulb
sweep dirt the short dustpan
put the lime spoke
take the steak off the grill
stack the other cups on top of the black cup
rack
Figure Al: Examples of 18 widely used training (seen) tasks on RLBench.
The AGNOSTOS benchmark categorizes the unseen tasks into difficulty levels. The Level-1 testing includes 13 unseen tasks, which have similar semantics to seen tasks in either objects or motions. The Level-2 testing comprises 10 unseen tasks with entirely novel semantics, distinct from the seen tasks. The task categorization is performed by GPT-4o based on the semantics similarities between seen and unseen task descriptions. The categorization results are verified by three human experts in the field of robotics. In Table A1 we list all 23 unseen tasks, where the text in each "[]" symbol is an abbreviation for the corresponding task. Under the full task name of each unseen task, show the seen task whose semantics are the most similar to the unseen task.
23

---

Table A1: 23 unseen tasks in our AGNOSTOS benchmark. The text in each "[I" symbol is an abbreviation for the corresponding task. Under the full task name of each unseen task, we show its most similar task among the 18 seen tasks.
[niff]] put knife on chopping board
[Toilet]
[Fridge]
put toilet roll on stand (place wine at rack location)
close fridge (close jar)
(put item in drawer [Laptop]
[Miwawee
[Phone]
close microwave
close laptop lid
phone on base
(close jar)
(close jar)
(place vine at rack location)
[Seat]
[[ampOff]]
[LampOn]
toilet seat down
1amp off
1amp on (pueh buttons)
(reach drag)
(push buttons)
[Umbrella]
[Book]
[Charger]
put umbrella in
put books on bookshelf (put groceries in cupboard)
unplug charger
umbrella stand
(no sinilar seen task)
(put item in draver)
[rill]]
[Bin] put rubbish bin
[USB] take out of computer
open grill
(open draver)
(put itam draver)
(no similar task)
[Plate]
[Lid]
[Ball]
take plate off colored dish rack (no similar seen task)
take lid off saucepan (no similar seen task)
basketball in hoop (no sinilar seen task)
[Scoop]
[Rope]
[Oven] turn oven on (no similar task)
scoop with spatula
straighten rope (no sinilar taak)
(no similar task)
[Buzz] beat the buzz
[Plants] water plants
(no similar seen task)
(no similar seen task)
A1.2 Evaluations existing VLA models
Due embodiment gaps (e.g., differences in robots, action spaces, and camera configurations), existing learning-based VLA models cannot effectively generalize to tasks in unseen embodiments. To address this, we fine-tune VLA models using data from RLBench's 's 18 seen tasks and evaluate their generalization on the 23 unseen tasks.
A1.2.1 Human-video pre-trained VLA models
Existing VLA models pre-trained on large-scale human action videos, such as R3M [30], D4R BI R3M-Align 322. and D4R-Align (3). claim to learn generalizable representations for robotic manipulation. HR-Align (32) adapts these models to RLBench by fine-tuning them on the 18 seen tasks, achieving strong performance on these tasks. We evaluate these adapted models provided by 32 on our 23 unseen tasks to assess their generalization.
A1.2.2 In-domain data trained VLA models
For the VLA models trained on the RLBench's 's 18 training (seen) tasks (i.c., in-domain training), including PerAct (27). RVT (28). RVT2 [33,, Sigma-Agent (78), and Instant Policy [69], we test their officially released models on our 23 unseen tasks. These VLA models serve as strong baselines on our benchmark, since they have sophisticated model designs on RLBench data and do not involve the data domain gap between their training tasks and our unseen testing tasks. Instant Policy is within-tas in-context imitation learning method that formulates policy prediction as a graph diffusion process over structured demonstrations and observations. To evaluate its cross-task generalization capability, we evaluate the released model on our 23 unseen tasks in a zero-shot manner, where randomly sampled seen demonstration is used for in-context prompting.
24

---

A1.2.3 Foundation VLA models
The foundation VLA models trained on large-scale robotic data or built upon LLM or VLM models, are expected have strong generalization capabilities on new tasks. this work, we evaluate Open- VLA [8], RDT [2] To [ LLARVA [77], SAM2Act [34], 3D-LOTUS++ [25]. and VoxPoser [2]. Below, we elaborate on our fine-tuning details, which follow the official fine-tuning guidelines.
OLe [8. We fine-tune Open VLA using 3,600 demonstrations from the 18 seen tasks, with each demonstration comprising front RGB view (size of 256 x 256) and the corresponding language instruction. We use a batch size of 16 and apply LoRA fine-tuning with rank of 32 and a learning rate of 5 10 Figure A2 shows the training loss and action accuracy during fine-tuning, indicating A2
rapid convergence within 2,000 steps. We evaluate the model on the 23 unseen tasks every 1,000 steps and select the model with the highest generalization performance.
trwin Inos
accurnyy
Figure A2: The statisties of the fine-tuning of Open
RDT 19.. For the fine-tuning of RDT, we also use the 3,600 seen demonstrations. For each demonstration, RDT takes the front and wrist RGB views as well as the language instruction as inputs, where the image size is 256 x 256. We use batch size of 16 and fine-tune the RDT for 400,000 steps by following their official guidance. Figure A3 shows the training loss and the overall average sample MSE loss (a good metric claimed by the authors) during the tinettunin, which indicates that the RDT converges well. We evaluate the RDT model every 10,000 steps, and report the highest generalization performance our 23 unseen tasks.
loss
overall, overallaavg.soe Lat_aaaape__emmmmmeaaaa
0.008
0.06
0.004
0.03
0.02
0.002
100k
2004
3004
400k
100k
200k
3004
400k
Figure A3: The statistics of the fine-tuning of RDT.
To [1. We fine-tune To with LoRA using 3,600 demonstrations, each including front, wrist, and overhead RGB views (size of 256 256) and the language instruction. Following the official protocol, we set the batch size to 64 and train for 100,000 steps. Figure A4 shows the training loss during fine-tuning. We evaluate the model every 10,000 steps and report the best generalization performance on the 23 unseen tasks.
LLARVA (77). LLARVA is model trained with instruction tuning the Open X-Embodiment dataset 29, which unifies various robotic tasks and environments. The authors further adapt LLARVA to RLBench to mitigate the embodiment gap. Therefore, we use the officially released model to evaluate the generalization performance on our 23 unseen tasks.
We directly evaluate the generalization performance of 3D-LOTUS using the pretrained model released by the authors. For 3D-LOTUS++ we follow the recommended configuration and adopt LLaMA3-8B as the LLM, and the base models of OwlViT v2 [89] and
3D-LOTTUS/D--LOUUSS
225
25

---

loss
0.3
0.2
0.1
20k
40k
60k
80k
Figure A4: The statistics the fine-tuning of To-
as the VLM components. These modules enable 3D-LOTUS+ to effectively decompose high-level instructions and ground object references in 3D space. We evaluate the model by following the official pipeline on our 23 unseen tasks.
SAM 990
SAM2Act (34,
SAM2Act builds upon the RVT-2 framework and introduces novel module that integrates semantic masks from SAM [90] into the action learning pipeline. The SAM2Act model is trained end-to-end RLBench'ss 18 training tasks. In our experiment,
use the checkpoint released by the authors and evaluate its performance directly on our 23 unseen tasks.
VoxPoser (2]. VoxPoser uses large language model to generate 3D affordance maps for motion planning. We adopt the open-sourced Owen2.5--7B-Insttrue variant as the LLM and modify the original single-task pipeline support batched evaluation custom datasets. During evaluation, object positions are directly obtained from the simulator as ground-truth. We evaluate the model in a zero-shot manner our 23 unseen tasks.
A1.3 Performance on Seen Tasks.
In Table A2 we evaluate the performance of RVT2, R3M-Align, OpenVLA, To and our X-ICM (72B) on 18 in-domain (seen) tasks, alongside their performance on 23 unseen tasks. These results demonstrate that all methods achieve high success rates on in-domain tasks after fine-tuning on seen task demonstrations, highlighting the challenge of cross-task zero-shot generalization.
18 within-task demonstrations construct in-context prompts for in-domain tasks, yielding 60.4% average success rate, which is substantially higher than its 30.1% on unseen tasks. However, our X-ICM's performance in-domain tasks lower than that of learning-basee VLA methods like (70.5%). This gap is expected, as X-ICM relies solely on the LLM's in-context learning capability without fine-tuning.
For X-ICM,
18 seen/in-domain avg (std)
23 unseen avg (std)
RVT2 R3M-Alis Align
82.3 (0.7)
10.3 (0.6)
59,2 (0.8)
12.2 (0.3)
VLe
63.1 (1.4)
13.6 (0.8)
An
70.5 (0.9)
17.5 (0.4)
X-ICM (72B)
60.4 (0.7)
30.1 (1.0)
Table A2: Performance on 18 seen tasks and 23 unseen tasks.
A2
More Analysis
A2.1
Dynamics-guided Sample Selection module
Different In-context Sample Selection Baselines. Our Dynamics-guide Sample Selection module designed to capture the dynamic semanties critical for cross-task generalization by modeling the transformation from an initial to a final state in manipulation tasks, detailed in Section 4.2. Unlike standard methods like Video-CLIP 991, which rely on video sequences and are thus incompatible our
with our setting- where only single initial frame is available during testing unseen diffusion-baan based model predicts future observations from this single frame. This enables effective modeling of task temporal dynamics essential for generalization.
26

---

put block into
push button
put bottle into box
stack blocks
stack cups
Figure AS: Demonstration examples five real-world tasks. We visualize one demonstration for each task, where show the key-observations captured by the third-view Orbbec camera.
To validate the necessity of our approach, we compared with baselines using CLIP 92 and DINOv2 93 features for sample selection, alongside a random selection strategy. The results in Table demonstrate that our method achieves higher average success rates and lower variance compared to these general-purpose features, as it is trained specifically on manipulation dynamics, unlike the broader datasets used for CLIP and DINOv2.
Level-1 avg (std)
Level-2 avg (std)
All avg (std)
Random
30.7 (4.7)
18.0 (2.2)
(3.2(
CLIP feature
32.1 (3.5)
18.9 (2.6)
26.4 (2.9)
DINOV2 feature
33.7 (3.1)
19.3 (2.0)
(2.5)
Dynamics Diffusion (Ours)
37.6 (1.4)
20.3 (1.7)
30.1 (1.0)
Table A3: Comparisons between different feature representations.
Qualitative results of the dynamics diffusion model. To facilitate the cross-task sample selection, we train dynamics diffusion model to encode the dynamic representations within each robot demonstration. The similarities of the captured dynamic representations across seen and unseen demonstrations are effective in identifying relevant demonstrations, thus guiding the cross-task sample selection process. For each demonstration, the dynamics diffusion model takes the initial observation and the corresponding language description inputs, aiming to predict the final visual observation at the completion of the demonstration. Figure A6 showcases qualitative predictions from the trained dynamics diffusion model.
27

---

groundiruth obs close the red jar
take the chicken off the grill
groundrruhh obs initial obs
predicted obs
initial obs
predicted obs
grouddruth obs
put the cube in the shape sorter
put money on the bottom shelf
initial obs
predicted obs
initial obs
predicted obs stack red blocks
groundtruth obs
stack the other cups on top of the red cup
initial obs
predicted obs groundtruth obs
initial obs
predicted obs
groundtruth obs
Figure A6: Qualitative results of the trained dynamics diffusion model. For each demonstration, the dynamics diffusion model uses the initial visual observation and language description as conditions and predicts the final visual observation upon task complction.
Using different dynamic features. Our dynamics diffusion model adopts the architecture of InstructPix2Pix [82). After training, we extract multi-modie features from each demonstration to serve as dynamic features for sample selection. this work, we use the combination of textual feature of the language description (i.e., feallang) and the predicted latent feature of the target observation (i.e., (eal as the dynamic features. Alternatively, we can use the latent feature of the initial observation (i.e., feaalvieda in). and try different combinations to form the dynamic features. Table|A4preseet combinations. The predicted latent feature of the target observation (i.e., feat out) proved most effective, validating the ability of our dynamics diffusion model to capture generalizable dynamics. Additionally, find that the latent feature of the initial observation (i.e., featvis.i is also effective, while using the textual feature the language description (i.e., featlang) does not show benefit.
the generalization performance on the 23 unseen tasks for different feature
Table A4: Generalization performance using different dynamic feature combinations. Upon our InstructPix2Pix based dynamics diffusion model, featiang denotes the language instruction feature, feat.gis.n represents the latent feature of the initial observation, and f faatfeis.out denotes the predicted latent feature of the target observation.
Tealiang
Jealwia.in
Sealviisoou
Level-1
Level-2
All
30.7
(2.0
(3.2)
32.9
17.8 (1.4)
27.5 (1.0)
X
(2.4)
19.9 (2.1)
29.7 (2.2)
V
37.1 (0.8)
17.6
28.6
V
X
36.3 (0.9)
16.4 (0.4)
27.7 (0.7)
V
37.6 (1.4)
20.3
30.1 (1.0)
38.6 (1.6)
16.7 (2.4)
29.0 (1.6)
A2.2 Different model sizes of LLMs.
We investigated the impact of model scale on performance by evaluating the X-ICM model various sizes of
7B, 14B, and 72B parameters. As shown in Table A5 larger models
28

---

generally tend to yield better performance. Performance improved noticeably the model size increased from 7B 14B. The performance gain diminished when scaling models to 72B, This suggests that while increasing model size is generally beneficial, the returns may diminish.
Table A5:
Effects of different model sizes.
Models
Level-1
Level-2
All
X-ICM (7B)
28.6 (1.9)
16.9 (1.3)
23.5 (1.6)
X-ICM (14B)
38.6 (0.5)
18.1 (2.3)
29.7 (0.7)
X-ICM (72B)
(1.4)
20.3 (1.7)
30.1 (1.0)
A2.3 Cross-task In-context Prompt
shows an example of the construction process of the cross-task in-context prompt. The prompt is concatenated by the system prompt, textualized cross-task seen demonstrations that are selected, and the textualized tested unseen task.
Figure A7
parallel gripper. provide you You are tasks, the format of [task instruction, observation] [[7-imm action 7 from
demas
system
dim action 2, 7-dim action TJ. Then you will receive an unseen task instruction with a new observation, and you need to output list of 7-dim actions that match the trends in the demos. Do not output anything else.
prompt
("instruction' slide block green target, t'target': [50, 33, 15]. 'block': [52. 50, 19]] [[53, 57, 17, O, 36, 53, [50. 33, 1S, 0, 36, $3, O]].
task
seen demos
['instrutton'' put the cube in the shape sorter, l'shupe sotter': [39, 63, 29). 'cube' [74, 53, 1871,-1173, 53, 22, 173, 53, 53, (44,
22, 0J, 173, 53, 23, 0, 36, 22, 01, [44,
68, ['instruction:: water plant, p'laatt: [60, 70, 49), 'waterer': [19, 64, 26]i]
I]].
tested
unseen
task
Cross-task In-context prompt
Figure A7:
An example of the construction our cross-task in-context prompt, which is concatenated by the system prompt. textualized cross task seen demonstrations that selected, and the textualized tested unseen task.
A3 Real-world Manipulation Experiments
In Figure A5 visualize the collected demonstrations for five real-world tasks. For each task, randomly select one demonstration for visualization. And we visualize the visual observations when the key-actions occur in each demonstration. the Supplementary Materials, showcase some testing videos that include both successful and failed cases.
A4 Broader Impacts Statements
There are no ethical issues involved in this paper. This work proposes a cross-task generalization benchmark to evaluate the generalization capabilities of existing Assionlaaguage-actiio models. The method proposed this work is purely algorithmic. Therefore, the proposed benchmark and method does not change the societal impacts of robotic manipulation.
29


Reviews:
[{"review_id": "KEpirFxuxg", "reviewer": "Reviewer_WCn9", "summary": "The paper proposes a new benchmark for cross-task generalization in VLAs they call AGNOSTOS. It is built to supplement 18 training tasks from RLBench to provide additional environments for evaluation. The authors then proceed to fine-tune existing VLA models on their training tasks and test for cross-task generalization. They discover that current methods do not do well on these tasks (as expected).\n\nTo start overcoming the limitations observed in current VLA models, the paper additionally proposes a mechanism for using off-the-shelf VLMs/LLMs with the RLBench data for in context learning. They dub their method X-ICM. This method leverages the cross-task generalization capabilities of Large Language Models (LLMs) by conditioning them on \"in-context demonstrations\" from seen tasks to predict action sequences for unseen tasks. The key insight is that this only works if appropriate in-context demonstrations are provided and to obtain such demonstrations they train a goal prediction diffusion model on the dataset (which is afterwards used for retrieval).", "questions": "Please address the weaknesses discussed above. In particular questiones regarding:\n- Statistical significance of results and providing expanded per-task results.\n- Questions regarding in-domain performance to help as a gauge for how well methods perform.\n- Questions regarding oracle performance on the proposed tasks.", "limitations": "Limitations are discussed adequately in the discussion section (given hte space constraints).", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 1, "strengths_and_weaknesses": "Pros:\n- The community is in need for more benchmarks and the proposed benchmark seems to address a whole in current evaluations.\n- The tasks seem to be designed thoughtfully and the split into tasks with partially shared semantics (Level 1) and entirely novel (Level 2) seems good.\n- They evaluate a meaningful set of pre-trained models from the literature and provide comprehensive evaluation results.\n- The explored in-context learning/policy elicitation technique seems simple and reasonable and is of interest to the community.\n\nWeaknesses:\n- The reviewer understands that large scale evaluations are costly. Still 25 rollouts with three repetitions for each method seems very little to judge statistical significance of results.\n- The real world experiments seem week / are relatively simple pick and place tasks.\n- It is unclear how good the in-context learning really is given that all methods score fairly low on most of the tasks (and no per task variances are given?). To calibrate it seems important to also give reference numbers for all methods on in-domain tasks. Where maybe the in-context learning is worse but it would help calibrate how far off the best possible result it is. It would also have been helpful to include some form of oracle score if possible (i.e. how good could a policy get at the tasks given X demonstrations)\n- Paper formatting seems off, see also comments below.\n- In some sense the benchmark is largely derivative of RLBench, most of it is \"just\" adding additional tasks to RLBench.\n- Details on how open models were fine-tuned seem scarce.\n- The retrieval of in-context examples seems relatively involved. It is unclear if such a complicated pipeline is truly necessary. No ablations / comparisons to simpler strategies is given.\n- The paper is mainly a systems paper proposing a new benchmark. Without the benchmark it would not pass the acceptance bar.\n- Some sentences seem ungrammatical, e.g. just from the first page:\nline 28: \"This capability-cross-task zero-shot generalization-is essential for real-world deployment, where robots are expected to tackle novel tasks as they arise dynamically.\" Should probably read something like \"The capability of cross-task (zero-shot) generalization is essential for real world deployment, [..]\"\nline 30: \"Despite the rapid progress in VLA research, [...] has focused on generalization testing in [...]\" -> \"Despite the rapid progress of research into VLAs [...] has focused on testing generalization\". VLA is not a noun, \"generalization testing\" isn't a thing.\nline 34: \"the generalization of VLA models.\" -> \"the generalization capabilities of VLA models\".\nline 60: \"Guided by the learned dynamics, the dynamics-aware prompts can be formed [...]\" -> \"Guided by the learned dynamics, dynamics-aware prompts can be formed [...]\"", "was_revised": true, "final_justification": "The issues brought up in my review have largely been addressed and as mentioned in my comment the additional experiments incl. in domain baselines strengthen the paper. I have adjusted my score accordingly.", "rebuttal": "**Dear Reviewer:**\n\nWe sincerely thank you for recognizing the value of our AGNOSTOS benchmark in addressing a gap in current VLA evaluations, its thoughtful task design, and our comprehensive evaluation of pre-trained models. We believe that our clarifications below can address all your concerns.\n\n&nbsp;\n\n**Q.1 Statistical significance of results.**\n\nTo address concerns about the statistical significance of our results, we expanded the evaluation to include 6 different seeds for each task, with 50 rollouts per seed, resulting in a total of 300 rollouts per task. Due to the rebuttal word limit, we present updated results for RVT2 (in-domain VLA), R3M-Align (human-video VLA), OpenVLA, $\\pi_0$ (foundation VLA), and our X-ICM (7B) across the 23 unseen tasks, as shown below. Task IDs (T1–T23) correspond to those in Table 2 of our paper.\n\nThe updated results confirm that the average success rates and standard deviations for Level-1, Level-2, and all tasks remain consistent with our original findings, with no statistically significant differences. Per-task standard deviations are included to provide transparency. High variance in some tasks is attributed to the RLBench environment’s variable scene configurations, a similar phenomenon also observed in Table I of the RVT2 paper [33].\n\n| - | T1 | T2 | T3 | T4 | T5 | T6 | T7 | T8 | T9 | T10 | T11 | T12 | T13 |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| RVT2 | 0.0 (0.0) | 0.0 (0.0) | 0.0 (0.0) | 20.0 (0.0) | 40.0 (6.9) | 1.3 (2.3) | 65.3 (2.3) | 2.7 (2.3) | 1.3 (2.3) | 0.0 (0.0) | 2.7 (2.3) | 6.7 (2.3) | 34.7 (2.3) |\n| R3M-Align | 0.0 (0.0) | 5.3 (2.3) | 46.7 (4.6) | 26.7 (2.3) | 24.0 (6.9) | 0.0 (0.0) | 53.3 (4.6) | 0.0 (0.0) | 5.3 (2.3) | 0.0 (0.0) | 0.0 (0.0) | 2.7 (2.3) | 0.0 (0.0) |\n| OpenVLA | 0.0 (0.0) |\t6.0 (4.6) | 37.8 (6.1) | 41.1 (8.0) | 56.4 (2.3) | 0.0 (0.0) | 52.8 (12.2) | 13.0 (8.0) | 1.3 (2.3) | 1.3 (2.3) | 0.0 (0.0) | 11.3 (4.1) | 0.0 (0.0) |\n| $\\pi_0$ | 0.0 (0.0) | 4.8 (1.6) | 84.8 (3.9) | 25.6 (8.6) | 40.0 (3.5) | 1.6 (1.9) | 64.8 (2.9) | 20.0 (4.3) | 7.2 (2.9) | 0.8 (1.6) | 0.0 (0.0) | 32.0 (4.3) | 0.8 (1.6)|\n| **X-ICM (7B)** | 1.6 (1.9) | 25.6 (7.8) | 22.4 (1.9) | 44.0 (4.3) | 34.4 (3.2) | 58.4 (7.8) | 47.2 (12.7) | 57.6 (7.8) | 50.4 (8.8) | 1.6 (1.9) | 0.0 (0.0) | 8.0 (7.1) | 16.8 (5.8) |\n\n| - | T14 | T15 | T16 | T17 | T18 | T19 | T20 | T21 | T22 | T23 | **Level-1 avg (std)** | **Level-2 avg (std)** | **All avg (std)** |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| RVT2 | 26.7 (9.2) | 37.3 (4.6) | 0.0 (0.0) | 0.0 (0.0) | 0.0 (0.0) | 0.0 (0.0) | 0.0 (0.0) | 1.3 (2.3) | 0.0 (0.0) | 2.7 (2.3) | 13.3 (0.6) | 6.6 (1.3) | 10.4 (0.4) |\n| R3M-Align | 89.3 (4.6) | 0.0 (0.0) | 2.7 (2.3) | 0.0 (0.0) | 0.0 (0.0) | 0.0 (0.0) | 1.3 (2.3) | 10.7 (4.6) | 4.0 (0.0) | 0.0 (0.0) | 12.2 (0.7) | 11.0 (0.5) | 11.7 (0.3) |\n| OpenVLA | 75.8 (2.3) | 0.0 (0.0) | 0.0 (0.0) | 0.0 (0.0) | 0.0 (0.0) | 0.0 (0.0) | 8.7 (2.3) | 8.3 (4.1) | 4.1 (2.3) | 0.0 (0.0) | 17.0 (1.1) | 9.7 (0.8) | 13.8 (0.9) |\n| $\\pi_0$ | 96.8 (3.9) | 0.0 (0.0) | 1.6 (1.9) | 0.0 (0.0) | 0.0 (0.0) | 0.0 (0.0) | 16.0 (4.4) | 6.4 (4.8) | 1.6 (2.0) | 0.0 (0.0) | 21.7 (0.4) | 12.2 (0.7) | 17.6 (0.4) |\n| **X-ICM (7B)** | 98.4 (1.9) | 17.6 (7.4) | 7.2 (3.9) | 8.8 (1.6) | 0.0 (0.0) | 6.4 (1.9) | 16.8 (2.9) | 2.4 (1.9) | 5.6 (1.9) | 3.20 (2.9) | **28.3 (1.0)** | **16.6 (1.2)** | **23.2 (1.1)** |\n\n&nbsp;\n\n**Q.2 More complex real-world tasks.**\n\nTo address concerns about the simplicity of real-world experiments, we evaluated two additional non-pick-and-place tasks: \"open the drawer\" and \"unscrew the bottle cap.\" To support these tasks, we expanded the real-world seen task set by including \"pull out the middle block,\" which involves a pulling action similar to \"open the drawer.\" Following the real-world testing protocol described in our paper, we achieved success rates of 20% for \"open the drawer\" and 0% for \"unscrew the bottle cap\" across 20 rollouts per task.\n\nThese results align with expectations: the pulling action in \"open the drawer\" resembles that of \"pull out the middle block,\" enabling partial success, whereas \"unscrew the bottle cap\" involves a unique twisting action absent from our seen task set, leading to failure. If we can expand the scale of real-world data collection (ie, seen tasks are diverse enough), the non-pick-and-place tasks could be solved to a certain extent. Our simulation results (i.e., Table 2) demonstrate that increasing the diversity of seen tasks improves performance on non-pick-and-place tasks.\n\n&nbsp;\n\n**Q.3 In-domain and oracle cross-domain performance.**\n\nTo address concerns about evaluating the effectiveness of in-context learning, we provide performance metrics for RVT2, R3M-Align, OpenVLA, $\\pi_0$, and our X-ICM (72B) on 18 in-domain (seen) tasks, as shown below, alongside their performance on 23 unseen tasks. These results demonstrate that all methods achieve significantly higher success rates on in-domain tasks after fine-tuning on seen task demonstrations, highlighting the challenge of cross-task zero-shot generalization.\n\nFor X-ICM, we use 18 within-task demonstrations to construct in-context prompts for in-domain tasks, yielding a 60.4% average success rate, which is substantially higher than its 30.1% on unseen tasks. However, our X-ICM's performance on in-domain tasks is lower than that of learning-based VLA methods like $\\pi_0$ (70.5%). This gap is expected, as X-ICM relies solely on the LLM’s in-context learning capability without fine-tuning.\n\nTo estimate the oracle performance of X-ICM on the 23 unseen tasks, we conducted a one-shot experiment, providing a single within-task demonstration per unseen task, selected based on semantic similarity using our dynamics-guided sample selection strategy. This achieved an average success rate of 42.8%, which can serve as an upper limit reference for cross-task zero-shot methods (our zero-shot performance: 30.1%). \n\n\n| - | **18 seen/in-domain** avg (std) | **23 unseen** avg (std) |\n| :---: | :---: | :---: |\n| RVT2 | 82.3 (0.7) | 10.3 (0.6) |\n| R3M-Align | 59.2 (0.8) | 12.2 (0.3) |\n| OpenVLA | 63.1 (1.4) | 13.6 (0.8) |\n| $\\pi_0$ | 70.5 (0.9) | 17.5 (0.4) |\n| **X-ICM (72B)** | 60.4 (0.7) | **30.1 (1.0)** |\n\n&nbsp;\n\n**Q.4 paper format and grammar correction.**\n\nThank you for the reminder on the paper format. We will move Figure 1 to the end of the abstract section. And we will remove the logo from the title of the paper. We will correct grammar issues based on your useful suggestions.\n\n&nbsp;\n\n**Q.5 Why did we choose RLBench instead of a new simulation?**\n\nWe clarify that our contribution extends significantly beyond adding tasks to RLBench.\n\nOur primary contribution lies in introducing a novel evaluation framework focused on cross-task, zero-shot generalization. This framework, supported by a large-scale evaluation of different types of VLA models, provides the first comprehensive analysis of zero-shot generalization capabilities. Our findings (e.g., Table 2) reveal critical limitations in current VLA models and provide the community with a standardized tool to measure future progress.\n\nIn addition, our choice of RLBench as the foundation for our benchmark was deliberate. RLBench is one of the most widely used manipulation environments in the community, and it provides a rich ecosystem where many influential methods are already validated. Building a benchmark on this environment allows for immediate, authoritative comparisons against prior work.\n\n&nbsp;\n\n**Q.6 Finetuning Details.**\n\nThank you for pointing out the need for more detailed descriptions of the fine-tuning process for existing VLA methods. For each VLA method, we follow the official fine-tuning guidelines and explain the details if we use different setups. We will elaborate on all the fine-tuning details in the revised version. Thank you so much for this suggestion.\n\n&nbsp;\n\n**Q.7 More in-context sample selection baselines.**\n\nWe compared our dynamics diffusion model with two simpler baselines: CLIP feature-based and DINOv2 feature-based sample selection, alongside a random selection strategy. The table below reports the average success rates and standard deviations for 23 unseen tasks.\nBoth CLIP and DINOv2 baselines outperform random selection in terms of average performance and stability, but they fall short of our dynamics diffusion model. This is because CLIP and DINOv2 can only capture static visual features (only a single initial frame is available during testing each unseen task), which fail to capture the temporal dynamics critical for cross-task generalization. In contrast, our dynamics diffusion model leverages future visual predictions to learn temporal dynamics, which is more effective in sample selection, as detailed in Section 4.2.\n\n| - | **Level-1 avg (std)** | **Level-2 avg (std)** | **All avg (std)** |\n| :---: | :---: | :---: | :---: |\n| **Random** | 30.7 (4.7) | 18.0 (2.2) | 25.2 (3.2) |\n| **CLIP feature** | 32.1 (3.5) | 18.9 (2.6) | 26.4 (2.9) |\n| **DINOv2 feature** | 33.7 (3.1) | 19.3 (2.0) | 27.4 (2.5) |\n| **Dynamics Diffusion (Ours)** | **37.6 (1.4)** | **20.3 (1.7)** | **30.1 (1.0)** |"}, {"review_id": "ZyE17cuPrj", "reviewer": "Reviewer_Qo7p", "summary": "This paper introduces AGNOSTOS, a simulation benchmark built upon RL-Bench, which includes 23 unseen manipulation tasks for evaluation. By testing several existing Vision-Language-Action (VLA) models on this benchmark, the authors observe that these models tend to generalize well to seen training tasks but perform poorly on unseen tasks. The authors propose a novel method called Cross-Task In-Context Manipulation (X-ICM) to address this limitation. This approach leverages large language models (LLMs) to predict actions by prompting them to demonstrate similar seen tasks used as in-context examples, which are selected via a dynamics-guided sampling strategy. This dynamics-guided sampling strategy first uses an image editing model based on InstructPix2Pix trained using the demonstration database of seen tasks to map (text, image) pairs to goal images, and then uses features derived from this model to compute similarity and retrieve the top-k most relevant demonstrations. On the proposed AGNOSTOS benchmark, the X-ICM method outperforms prominent VLA baselines such as $\\pi_0$ and VoxPoser, demonstrating its effectiveness in generalizing to unseen tasks.", "questions": "1. Why do the authors choose to use a diffusion model trained on the simulation benchmark dataset for feature extraction, rather than adopting a more generalizable Video-CLIP-based method? The former approach incurs additional training costs and lacks transferability to other benchmarks beyond RL-Bench, undermining the claimed generality of the proposed X-ICM framework.\n2. What is the success rate of the X-ICM framework without providing any demonstrations? This baseline is important to quantify the actual contribution of the in-context prompting. Additionally, why not consider training an end-to-end VLA model that supports in-context learning, such as ICRT [4], which is capable of acting without demonstrations and improves performance when examples are provided?\n3. The authors are encouraged to clarify their definition of cross-task zero-shot. As discussed earlier, the current formulation—using retrieved demonstrations from semantically similar tasks—does not strictly align with the conventional understanding of zero-shot generalization.\n\n[4] In-Context Imitation Learning via Next-Token Prediction", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "=== Strengths ===\n1. The paper presents some real-world experiments beyond simulation benchmarks, which makes the effectiveness of the proposed methods more convincing.\n2. The authors present a well-motivated contribution by proposing a novel simulation benchmark named AGNOSTOS. This benchmark not only reveals that current VLA models fail to generalize effectively to unseen tasks but also provides a playground for future research on cross-task generalization in VLA models.\n3. Leveraging in-context learning and generalized features of LLM, the proposed X-ICM can perform better than diverse VLA models on AGNOSTOS, showcasing its powerful cross-task zero-shot capability.\n\n=== Weakness ===\n1. The Dynamics-guided Sample Selection module appears relatively complex, and its motivation is not justified. In the current literature on multimodal RAG, especially video retrieval (e.g., VideoAgent [1], MineDreamer [2]), it is common to use more generalizable Video-CLIP-based methods to extract features and compute similarity for retrieving relevant video demonstrations. In contrast, this paper trains an image-editing model (based on data from RL-Bench) specifically for feature extraction, which may limit generalization. This approach seems misaligned with the paper’s stated goal of cross-task zero-shot generalization. Specifically, the proposed framework may not be easily transferable to new benchmarks or environments without retraining the diffusion-based image-editing model, thus undermining the claimed generality of X-ICM.\n2. The proposed X-ICM method lacks novelty, as its within-task in-context manipulation approach is not substantially different from prior work (e.g., RoboPrompt[3]). Both methods retrieve relevant video demonstrations and then pass key annotation information to an LLM or VLM to directly generate actions, limiting the contribution of X-ICM from a methodological view.\n3. The definition of zero-shot in this work is somewhat ambiguous. The X-ICM framework retrieves several demonstrations as contextual prompts for the LLM, even though these demonstrations may differ in actions/motions and objects from the target task (especially in Level 1, where partial semantic overlap exists). However, using such retrieved demonstrations as in-context examples deviates from the conventional definition of zero-shot, and may lead to confusion.\n\n[1] VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding\n\n[2] Minedreamer: Learning to follow instructions via chain-of-imagination for simulated-world control\n\n[3] In-context learning enables robot action prediction in LLMs", "was_revised": true, "final_justification": "Thanks for the comprehensive rebuttal, which addresses most of my concerns. I would like to raise my score to borderline.", "rebuttal": "**Dear Reviewer:**\n\nWe greatly appreciate your recognition of our sufficient experiments, well-motivated and significant benchmark, and the strong performance. We believe that our clarifications below can address all your concerns and provide you with new perspectives to re-assess our paper’s contributions.\n\n&nbsp;\n\n    \n**Q.1 Clarification on Dynamics-guided Sample Selection.**\n\n**(1) Clarification on the motivation:**\n\nOur Dynamics-guided Sample Selection module is designed to capture the dynamic semantics critical for cross-task generalization by modeling the transformation from an initial to a final state in manipulation tasks, as detailed in Section 4.2. Unlike standard methods like Video-CLIP, which rely on video sequences and are thus incompatible with our setting—where only a single initial frame is available during testing unseen tasks—our diffusion-based model predicts future observations from this single frame. This enables effective modeling of task temporal dynamics essential for generalization.\n\nTo validate the necessity of our approach, we compared it with baselines using CLIP and DINOv2 features for sample selection, alongside a random selection strategy. The results, shown below, demonstrate that our method achieves higher average success rates and lower variance compared to these general-purpose features, as it is trained specifically on manipulation dynamics, unlike the broader datasets used for CLIP and DINOv2.\n\n| - | **Level-1 avg (std)** | **Level-2 avg (std)** | **All avg (std)** |\n| :---: | :---: | :---: | :---: |\n| **Random** | 30.7 (4.7) | 18.0 (2.2) | 25.2 (3.2) |\n| **CLIP feature** | 32.1 (3.5) | 18.9 (2.6) | 26.4 (2.9) |\n| **DINOv2 feature** | 33.7 (3.1) | 19.3 (2.0) | 27.4 (2.5) |\n| **Dynamics Diffusion (Ours)** | **37.6 (1.4)** | **20.3 (1.7)** | **30.1 (1.0)** |\n\nWe will incorporate this analysis and ablation table into the revised manuscript to clarify the motivation and effectiveness of our approach.\n\n\n**(2) Clarification on cross-task generalization:**\n\nRegarding the generalizability of the X-ICM framework, we would like to clarify the scope of generalization we address. Due to the \"embodiment gap\" (e.g., a new robot or environment), virtually all learning-based VLA models require fine-tuning to adapt to a new data domain. Our work therefore focuses on a model's ability to generalize to unseen tasks within the same domain after this initial adaptation. Therefore, we argue that it is acceptable to train our X-ICM model on seen task data.\n\nIn addition, for direct transfer to a new benchmark without training, our dynamics diffusion model can be replaced with an off-the-shelf feature extractor like DINOv2. As shown in the ablation in our response to the previous question, this approach outperforms random selection, confirming the inherent effectiveness and transferability of our cross-task sample selection mechanism itself.\n\n&nbsp;\n\n\n**Q.2 Comparison between our X-ICM and Roboprompt.**\n\nWe clarify the key distinctions and novel contributions of our X-ICM method compared to prior work like RoboPrompt: while both leverage in-context learning, they address fundamentally different problems, and our method introduces a crucial, novel component as a result. Specifically,\n\n**A New Problem Setting:** RoboPrompt focuses on within-task generalization—learning to perform a seen task in a new scenario. Our work is the first to systematically apply and demonstrate the effectiveness of this in-context learning paradigm for the more challenging cross-task zero-shot setting. As shown in our main results (Table 2), this approach significantly outperforms existing VLA models on unseen tasks. \n\n**A Critical Selection Strategy:** In a within-task setting, randomly selecting demonstrations is a reasonable strategy, as all examples are relevant. However, in the cross-task setting, selecting relevant examples from a diverse pool of different tasks is non-trivial and critical for success. RoboPrompt does not address this challenge. Our core methodological contribution is the Dynamics-guided Sample Selection module, which is specifically designed to solve this problem by identifying dynamically similar demonstrations across different tasks. The following ablation demonstrates that applying a random selection strategy (analogous to RoboPrompt's approach) in our cross-task setting yields poor and unstable results. Our dynamics-guided selection provides a substantial performance gain, highlighting its necessity and novelty.\n\n| - | **Level-1 avg (std)** | **Level-2 avg (std)** | **All avg (std)** |\n| :---: | :---: | :---: | :---: |\n| **Random** | 30.7 (4.7) | 18.0 (2.2) | 25.2 (3.2) |\n| **Dynamics Diffusion (Ours)** | **37.6 (1.4)** | **20.3 (1.7)** | **30.1 (1.0)** |\n\n\n&nbsp;\n\n\n**Q.3 Clarification on the zero-shot definition.**\n\nWe clarify that our definition of zero-shot learning in the X-ICM framework is both rigorous and consistent with established standards in the literature [R1][R2]. Our 23 unseen tasks are divided into two difficulty levels: Level-1 tasks share partial similarity with seen tasks in either objects or motions (unseen object or unseen motion settings), while Level-2 tasks have no similarity, representing unseen object-motion compositions.\n\nFor Level-1 tasks, the partial dissimilarity (unseen objects or motions) aligns with the conventional zero-shot learning paradigm, as supported by the zero-shot human-object interaction (HOI) detection literature, which is closely related to our setting. Specifically:\n\n- CLIP4HOI (NeurIPS 2023) [R1] defines zero-shot HOI detection to include unseen object (UO), unseen verb (UV), and unseen composition (UC) settings, as noted in the caption of Table 1: “Zero-shot HOI detection results on HICO-DET. UC, UO, and UV denote unseen composition, unseen object, and unseen verb settings.” This directly supports our inclusion of unseen object and motion settings in Level-1 tasks.\n- KI2HOI [R2] similarly categorizes zero-shot HOI detection into three scenarios: unseen object, unseen action, and unseen combination, as stated in Section II.C (page 2): “Zero-shot HOI detection focuses on transferring knowledge from known HOI concepts to unseen classes. They can be categorized into three scenarios: unseen object, unseen action, and unseen combination.”\n\n[R1] CLIP4HOI: Towards Adapting CLIP for Practical Zero-Shot HOI Detection, NeurIPS 2023.\n\n[R2] Towards Zero-shot Human-Object Interaction Detection via Vision-Language Integration, Neural Networks, 2025.\n\n\n&nbsp;\n\n**Q.4 Performance without any demonstrations.**\n\nWe conducted the zero-demonstration experiment and we found that the success rates on all 23 unseen tasks are 0%. This result is expected. Without any in-context examples, the LLM has no reference to follow the action patterns required to complete the manipulation tasks.\n\n\n&nbsp;\n\n\n**Q.5 Clarification on end-to-end in-context learning.**\n\nThank you for suggesting an end-to-end training model like ICRT that supports in-context learning. To clarify, ICRT is designed for within-task in-context learning, where it leverages demonstrations to adapt to new scenarios of a seen task. Extending ICRT to our cross-task zero-shot setting, where models must generalize to entirely unseen tasks, requires significant further investigation, as this problem introduces unique challenges in sample selection and task dynamics transfer, as discussed in Sections 4.2 and 4.3.\n\nAdditionally, our results (Table 2) show that strong cross-task performance relies on a powerful VLM backbone (e.g., VLM-based $\\pi_0$ performs well while non-VLM-based RVT2 performs badly). However, training an end-to-end VLM-based model for in-context learning, such as adapting ICRT, incurs prohibitive computational costs due to the need for large-scale, diverse training data."}, {"review_id": "jH6w5P8wvV", "reviewer": "Reviewer_Xb1Z", "summary": "In this work, the authors present AGNOSTOS, a benchmark designed to evaluate cross-task zero-shot generalization for robotic manipulation. The analysis of various VLA models reveals significant limitations in generalizing to unseen tasks. To address this challenge, the authors introduce X-ICM, a cross-task in-context manipulation approach that harnesses the generalization capabilities of LLMs, achieving significant improvements over existing VLA models.", "questions": "1) **Comparison to RoboPrompt and KAT.** While I agree that existing in-context learning baselines such as RoboPrompt and KAT are not designed for cross-task zero-shot generalization scenarios, including them in the comparison would still provide valuable insight into their current performance.\n\n\n2) **“Dynamics-guided sample selection” Ablations.** This section lacks clarity. It would be helpful to evaluate multiple approaches or baselines to highlight the challenges in selecting effective in-context examples. The authors seem to have chosen diffusion models on purpose, but the reason for this choice is not clearly explained. Simpler methods could be considered, so the authors are encouraged to provide further details, motivations, and comparisons with alternative approaches that were tested but found less effective.\n\n\n3) The constructed prompt closely resembles the one used in RoboPrompt. If this is accurate, please ensure proper citation within lines 225-235.\n\n\nOverall, I liked this paper. However, it would benefit from additional justification for the proper use of X-ICM and a more thorough comparison to closely related baselines like KAT/RoboPrompt.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1) In general, the paper is clear and the technical details are easy to understand.\n\n\n2) The core idea of cross-task zero-shot generalization in vision-language-action (VLA) models and introducing a benchmark for this purpose is a valuable and timely contribution to the field.\n\n\n3) The related work section is thorough and extensive.", "was_revised": true, "final_justification": "After reading the authors' feedback and other reviewers' opinions, I would like to thank the authors for their rebuttal.\n\nMy questions have been fully addressed. Therefore, I am maintaining my score for acceptance, as this paper explores cross-task settings for in-context learning in robotics, which I consider an important direction for future research.", "rebuttal": "**Dear Reviewer:**\n\nWe sincerely thank you for recognizing the clarity of our paper and the valuable contribution of our AGNOSTOS benchmark in advancing cross-task zero-shot generalization for vision-language-action models. We believe that our clarifications below can address all your concerns.\n\n&nbsp;\n\n\n**Q.1 Comparison to RoboPrompt and KAT**\n\nSince Roboprompt and KAT are designed for within-task in-context learning and rely on random demonstration selection, we extend them to our cross-task setting by randomly selecting cross-task samples. To ensure a fair comparison, all methods use the Qwen2.5-72B-Instruct model as the LLM backbone.\nThe results, shown below, demonstrate that X-ICM significantly outperforms KAT and RoboPrompt in both average success rate and stability, owing to our dynamics-guided cross-task sample selection module.\n\n| - | **Level-1 avg (std)** | **Level-2 avg (std)** | **All avg (std)** |\n| :---: | :---: | :---: | :---: |\n| **KAT** | 24.8 (5.8) | 15.3 (4.4) | 20.7 (4.7) |\n| **Roboprompt** | 30.7 (4.7) | 18.0 (2.2) | 25.2 (3.2) |\n| **X-ICM (Ours)** | **37.6 (1.4)** | **20.3 (1.7)** | **30.1 (1.0)** |\n\n\n&nbsp;\n\n\n**Q.2 Dynamics-guided sample selection Ablations.**\n\nWe clarify the motivation for our module design and provide additional baseline comparisons to highlight its effectiveness.\n\n**(1) Motivation for Dynamics-guided Sample Selection:** Our work tackles the challenging problem of cross-task zero-shot manipulation via in-context learning, where selecting relevant cross-task demonstrations is critical. Effective sample selection requires modeling the temporal dynamics inherent in manipulation tasks. Given that only an initial RGB frame and task description are available during testing, we train a diffusion model to predict the final visual observation from the initial observation. This approach enables the model to learn feature representations that encode task temporal dynamics, facilitating robust cross-task sample selection.\n\n**(2) Ablation with Additional Baselines:** To demonstrate the effectiveness of our dynamics-guided sample selection, we compared it against two general-purpose feature extractors, CLIP and DINOv2, and a random selection baseline. The results, shown below, indicate that CLIP and DINOv2 outperform random selection but are surpassed by our dynamics diffusion model, which achieves higher performance and stability due to its focus on task dynamics.\n\n| - | **Level-1 avg (std)** | **Level-2 avg (std)** | **All avg (std)** |\n| :---: | :---: | :---: | :---: |\n| **Random** | 30.7 (4.7) | 18.0 (2.2) | 25.2 (3.2) |\n| **CLIP feature** | 32.1 (3.5) | 18.9 (2.6) | 26.4 (2.9) |\n| **DINOv2 feature** | 33.7 (3.1) | 19.3 (2.0) | 27.4 (2.5) |\n| **Dynamics Diffusion (Ours)** | **37.6 (1.4)** | **20.3 (1.7)** | **30.1 (1.0)** |\n\n&nbsp;\n\n\n**Q.3 Clarification on the prompt design.**\n\nThank you for your feedback regarding the prompt. To clarify, as our method focuses on cross-task zero-shot generalization rather than prompt engineering, we adapt the prompt structure from RoboPrompt to suit our cross-task setting. We will ensure proper citation of RoboPrompt in lines 225-235 of the revised manuscript to acknowledge this source. Thank you for bringing this to our attention."}, {"review_id": "Ej8Epi24wf", "reviewer": "Reviewer_KVSd", "summary": "This paper introduces AGNOSTOS, a new simulation benchmark built upon RLBench, which selects 23 unseen tasks as the test set and uses the standard 18-task configuration as the training set. To evaluate cross-task zero-shot generalization in manipulation, AGNOSTOS categorizes the unseen tasks into two levels based on their similarity to the training tasks. In addition to the benchmark, this paper proposes Cross-task In-Context Manipulation (X-ICM), a method specifically designed to tackle the challenges of cross-task zero-shot generalization. Compared to a broad range of mainstream Vision-Language Action (VLA) models, X-ICM achieves state-of-the-art performance on AGNOSTOS.", "questions": "1. Tasks such as straightening a rope or removing a USB device from a computer demand high spatial resolution, yet a single grid cell may encompass multiple parts from different objects, leading to ambiguity and reduced control precision. Could the authors explain further how the proposed X-ICM framework effectively handles such tasks under these discretization constraints.\n2. Include a more detailed analysis of failure cases, distinguishing between issues related to sample selection and LLM reasoning. This could provide more insight for this paper.\n3. The performance of X-ICM on complex, long-horizon tasks remains insufficiently explored, especially in realistic or real-world settings. Could the authors provide further experiment results in real-world to show X-ICM’s practical applicability?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. This paper proposes a new simulation benchmark, AGNOSTOS, to evaluate the cross-task zero-shot generalization in manipulation.\n2. X-ICM shows effective and impressive experiment results, compared to other mainstream VLA methods, e.g. pi0 and OpenVLA.\n\nWeaknesses:\n1. The construction of AGNOSTOS appears relatively straightforward, as it involves a direct partitioning of existing tasks within RLBench. The novelty of this benchmark is thus somewhat unclear.\n2. The methodological details of the proposed X-ICM approach lack sufficient clarity. For instance, it remains ambiguous how the output of the large language model (LLM) is transformed into precise low-level control commands, such as accurate gripper poses required for manipulation. A more detailed description of this mapping process is necessary to evaluate the feasibility and reproducibility of the method.\n3. Voxposer should not be categorized as a Vision-Language Action (VLA) method. While it may share some similarities in using vision and language components, its core design and objectives differ significantly from mainstream VLA models.\n4. There is a labeling error in Figure 2. where the language description in Diffusion-based dynamics modeling stage is incorrectly labeled as \"Close Jar\" instead of the correct \"Close Box.\"", "was_revised": true, "final_justification": "Thanks for the detailed rebuttal. Most of my concerns and questions have been well addressed. I have raised my rating to BA. I'd further suggest the paper to include those experiments on long-horizon tasks under realistic settings in the revision.", "rebuttal": "**Dear Reviewer:**\n\nWe greatly appreciate your recognition of our novel AGNOSTOS benchmark for evaluating cross-task zero-shot generalization and the strong performance of X-ICM compared to mainstream VLA methods. We believe that our clarifications below can address all your concerns and provide you with new perspectives to re-assess our paper’s contributions.\n\n\n&nbsp;\n\n\n**Q.1 Clarification on the construction of AGNOSTOS.**\n\n- We clarify the significant contributions of our AGNOSTOS benchmark to the VLA community:\n    - **Pioneering Cross-task Zero-shot Benchmark:** AGNOSTOS is the first systematic benchmark designed for cross-task zero-shot manipulation, addressing a critical gap in the literature where prior work has largely overlooked this challenging setting.\n    - **Revealing Generalization Limitations:** Through extensive evaluation of multiple VLA methods on AGNOSTOS, we found that existing methods achieve success rates below 20% on 23 unseen tasks (Table 2). This insight highlights the limitations of current approaches and provides a standardized framework to guide future VLA generalization research.\n- **The decision to build upon RLBench was deliberate**: RLBench is a widely adopted, high-fidelity manipulation environment used by influential works (e.g., PerAct [27], RVT2 [33]) and generalization benchmarks (e.g., COLOSSEUM [24], GemBench [25]). Leveraging RLBench ensures informative and reliable comparisons with prior methods, maintaining a cohesive ecosystem for the manipulation community rather than introducing a new simulation environment.\n\n\n&nbsp;\n\n\n**Q.2 Transformation between LLM’s input/output and low-level actions.**\n\nWe provide a detailed description of the transformation process below.\n\nWe define the robot’s workspace in the base coordinate system with ranges $X: [-0.3m, 0.7m]$, $Y: [-0.5m, 0.5m]$, and $Z: [0.6m, 1.6m]$. An end-effector action at a given timestamp is represented as a 7-dimensional vector, e.g., $[0.1m, -0.2m, 0.8m, 20°, 90°, 240°, 1]$, where the first three elements denote the 3D position (X, Y, Z), the next three denote the rotation angles around the X-, Y-, and Z-axes, and the last element indicates the gripper state (0 for closed, 1 for open).\n\nWe discretize the workspace and angles as follows:\n- The 3D workspace is divided into 100 uniform grids along each axis. For a position $p$ in range $[p_{\\text{min}}, p_{\\text{max}}]$, the discretized value is computed as: $$\\text{Grid} = \\left\\lfloor \\frac{p - p_{\\text{min}}}{p_{\\text{max}} - p_{\\text{min}}} \\times 100 \\right\\rfloor$$\n- Each rotation angle is discretized into the range $[0, 72]$ with a $5°$ interval, yielding 72 bins. The discretized angle is: $$\\text{Bin} = \\left\\lfloor \\frac{\\text{angle}}{5} \\right\\rfloor$$\n- The gripper state remains unchanged (0 or 1).\n\nFor the example action $[0.1m, -0.2m, 0.8m, 20°, 90°, 240°, 1]$, we convert it into LLM's input format:\n- X-position: $\\frac{0.1 - (-0.3)}{0.7 - (-0.3)} \\times 100 = 40$\n- Y-position: $\\frac{-0.2 - (-0.5)}{0.5 - (-0.5)} \\times 100 = 30$\n- Z-position: $\\frac{0.8 - 0.6}{1.6 - 0.6} \\times 100 = 20$\n- X-rotation: $\\frac{20}{5} = 4$\n- Y-rotation: $\\frac{90}{5} = 18$\n- Z-rotation: $\\frac{240}{5} = 48$\n- Gripper state: $1$\n\nThus, the action in LLM format is $[40, 30, 20, 4, 18, 48, 1]$. The inverse process can convert the action in LLM's format into continuous end-effector action.\n\nWe will incorporate these details into the revised manuscript to enhance clarity.\n\n&nbsp;\n\n**Q.3 Clarification on the Voxposer method.**\n\nWe clarify that since much of the literature (eg, [R3][R4]) in the community classifies Voxposer as a VLA method, we follow this classification. We will give a special explanation of the difference between Voxposer and mainstream VLA in the paper. Thank you for pointing this out. \n\n[R3] A Survey on Vision-Language-Action Models: An Action Tokenization Perspective, 2025.\n\n[R4] Vision-Language-Action Models: Concepts, Progress, Applications and Challenges, 2025\n\n&nbsp;\n\n**Q.4 Labeling error in Figure 2.**\n\nThank you, we will correct it.\n\n&nbsp;\n\n**Q.5 Clarification on the spatial perception with discretization constraints.**\n\nAs detailed in our response to Q.2, we discretize the robot’s workspace into $1 cm^2$ grids (100 grids per axis across X: [-0.3m, 0.7m], Y: [-0.5m, 0.5m], Z: [0.6m, 1.6m]). Given the sparse and scattered object distribution in the RLBench environment, the likelihood of a single $1 cm^2$ grid encompassing multiple object parts is minimal, reducing ambiguity in most cases.\n\nFor the \"USB\" tasks, the success rate is very high, and we found that this is because the detection of \"USB\" is very accurate. For the \"straight rope\" task, the detection performance for the \"rope head\" is poor, which is the main reason for its low success rate.\nTherefore, we hope that future work can focus on enhancing the 3D spatial understanding ability of models.\n\n&nbsp;\n\n**Q.6 Failure analysis (sample selection vs. LLM reasoning).**\n\nThank you for this interesting and valuable question. In order to investigate the impact of sample selection and LLM reasoning on model performance, we compared two experimental settings. One is our original X-ICM model, which follows a cross-task zero-shot setting. Another one is that in the testing of unseen tasks, we do not apply the cross-task sample selection, but instead provide a single sample within the unseen tasks, so that we can probe the performance of X-ICM under the one-shot setting.\n\nOn each task, by comparing the performance of the model in the cross-task zero-shot setting (both sample selection and LLM reasoning are involved) and the within-task one-shot setting (providing the most favorable within-task sample, LLM reasoning is more involved), we can perform the following analysis and obtain conclusions:\n- If the model performs equally in the two situations (eg, within 5% performance difference), it means that LLM's reasoning ability has a greater impact on the task;\n- If the within-task one-shot setting is significantly better (eg, more than 5% performance difference) than the cross-task zero-shot setting, it means that sample selection has a greater impact on the task;\n- If the cross-task zero-shot setting is significantly better (eg, more than 5% performance difference) than the within-task one-shot setting, it means that LLM's reasoning ability has a greater impact on the task;\n\nIn the table below, we compare the model's performance in these two situations. The table follows the same format as Table 2 in our paper (for simplicity, we use \"T(task_id)\" to represent the original task names in Table 2). From the results, we can see that, for tasks \"T3, T7, T9, T10, T12, T13, T15, T17, T19, T20, T22, T23\", the sample selection is more important; For the remaining tasks, the LLM reasoning is more important.\n\n\n| - | T1 | T2 | T3 | T4 | T5 | T6 | T7 | T8 | T9 | T10 | T11 | T12 | T13 |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| X-ICM (72B, **zero-shot**) | 6.7 | 69.3 | 12.7 | 58.7 | 34.0 | 68.0 | 51.3 | 86.7 | 74.7 | 2.0 | 1.3 | 5.3 | 18.7 |\n| X-ICM (72B, **one-shot**) | 0.0 | 58.7 | 57.3 | 33.3 | 33.3 | 61.3 | 69.3 | 70.7 | 97.3 | 10.7 | 0.0 | 57.3 | 72.0 |\n\n| - | T14 | T15 | T16 | T17 | T18 | T19 | T20 | T21 | T22 | T23 | **Level-1 avg (std)** | **Level-2 avg (std)** | **All avg (std)** |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| X-ICM (72B, **zero-shot**) | 98.7 | 13.3 | 4.7 | 36.0 | 0.7 | 16.0 | 20.7 | 7.3 | 2.7 | 2.7 | 37.6 (1.4) | 20.3 (1.7) | 30.1 (1.0) |\n| X-ICM (72B, **one-shot**) | 98.7 | 94.7 | 8.0 | 52.0 | 5.3 | 28.0 | 50.7 | 4.0 | 8.0 | 14.7 | 47.8 (0.8) | 36.4 (1.9) | 42.8 (0.7) |\n\n\n&nbsp;\n\n\n**Q.7 Clarification on long-horizon task setting.**\n\nThank you for your suggestion to explore X-ICM’s performance on complex, long-horizon tasks in realistic settings. To clarify, our work focuses on the first systematic evaluation of cross-task zero-shot generalization for VLA methods, a challenging and under-explored problem. Learning dynamic semantic generalization across tasks from robot data is inherently difficult, as VLM foundation models lack this capability. Even simple unseen tasks pose significant challenges in this setting, and long-horizon tasks, requiring sequential sub-task execution, amplify these difficulties. We believe these tasks warrant further in-depth study.\n\nAt your request, we explore this interesting problem in real-world experiments. We use tasks *\"put block into bin\"*, *\"push button\"*, *\"put bottle into box\"*, *\"stack blocks*\", and *\"pull out the middle block\"* as seen tasks, where we have 10 demonstrations for each task.\n\n**The long-horizon task we evaluated is *\"clean the table\"***, according the robot's visual observation, we decompose the long-horizon task into three sequential sub-tasks, ie, *\"stack cups\"*, *\"put the stacked cups into plate\"*, *\"place the mango on top of the stacked cups\"*. We evaluated this task over 20 rollouts, with sub-task success contingent on the success of all prior sub-tasks. \n\nFor our X-ICM method, the success rates on sub-tasks *\"stack cups\"*, *\"put the stacked cups into plate\"*, *\"place the mango on top of the stacked cups.\"* are 40%, 25%, and 5%, respectively. Thus, the overall success rate for the long-horizon task is 5%.\n\nFor comparison, we extend the RoboPrompt (within-task in-context learning) method to our cross-task zero-shot setting by randomly selecting cross-task samples. And the success rates on the three sub-tasks are 10%, 0%, and 0%, respectively. This also demonstrates the effectiveness of our method.\n\nThese results align with expectations, as the sequential nature of long-horizon tasks leads to cumulative failure. The cross-task zero-shot setting exacerbates these difficulties. These findings highlight the need for more advanced algorithms to handle long-horizon tasks in zero-shot scenarios."}]

Author rebuttal:

Meta-review: The paper presents a benchmark for cross-task zero-shot manipulation built from RLBench (18 seen training tasks; 23 unseen evaluation tasks split into Level-1 with partial semantic overlap and Level-2 with novel compositions). Baselines show poor transfer to unseen tasks. The paper then introduces an approach that elicits actions from a pretrained LLM using in-context demonstrations retrieved from seen tasks by a dynamics-guided diffusion model that predicts goal images from a single initial frame to capture temporal dynamics and outperforms the baselines. The rebuttal strengthened the evidence: reporting in-domain vs unseen performance and a one-shot “oracle”, comparing retrieval against CLIP/DINOv2/random, and adding small real-world and long-horizon studies.

The work's strengths are a timely benchmark that standardises evaluation for cross-task generalisation, a thoughtful task split, and broad empirical coverage that was materially improved during rebuttal, including clear ablations demonstrating the value of dynamics-guided retrieval. Its weaknesses are that the benchmark merely extends RLBench and the methodological advance over prior in-context pipelines (e.g., RoboPrompt) is modest and subtle. The pipeline is also comparatively complex and relies on training the diffusion component on seen tasks, which raises portability questions. Furthermore, elements of the action decoding and formatting required clarification and real-world and long-horizon performance remains preliminary. 

Even so, the benchmark fills a genuine gap and the strengthened experiments address the principal concerns. I therefore recommend accepting, with the camera-ready expected to integrate the ablations, the expanded real-world and long-horizon results and additional experiments and clarifications provided in the rebuttal.
Decision: Accept (poster)