Title: Exploring the Limits of Vision-Language-Action Manipulation in Cross-task Generalization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Vision-Language-Action

Cross-task Generalizable Manipulation
Abstract: The generalization capabilities of vision-language-action (VLA) models to unseen tasks are crucial to achieving general-purpose robotic manipulation in open-world settings.
However, the cross-task generalization capabilities of existing VLA models remain significantly underexplored.
To address this gap, we introduce **AGNOSTOS**, a novel simulation benchmark designed to rigorously evaluate cross-task zero-shot generalization in manipulation. 
AGNOSTOS comprises 23 unseen manipulation tasks for test—distinct from common training task distributions—and incorporates two levels of generalization difficulty to assess robustness. 
Our systematic evaluation reveals that current VLA models, despite being trained on diverse datasets, struggle to generalize effectively to these unseen tasks. 
To overcome this limitation, we propose **Cross-Task In-Context Manipulation (X-ICM)**, 
a method that conditions large language models (LLMs) on in-context demonstrations from seen tasks to predict action sequences for unseen tasks.
Additionally, we introduce a **dynamics-guided sample selection** strategy that identifies relevant demonstrations by capturing cross-task dynamics. 
On AGNOSTOS, X-ICM significantly improves cross-task zero-shot generalization performance over leading VLAs, achieving improvements of 6.0\% over $\pi_0$ and 7.9\% over VoxPoser.
We believe AGNOSTOS and X-ICM will serve as valuable tools for advancing general-purpose robotic manipulation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces AGNOSTOS, a new simulation benchmark built upon RLBench, which selects 23 unseen tasks as the test set and uses the standard 18-task configuration as the training set. To evaluate cross-task zero-shot generalization in manipulation, AGNOSTOS categorizes the unseen tasks into two levels based on their similarity to the training tasks. In addition to the benchmark, this paper proposes Cross-task In-Context Manipulation (X-ICM), a method specifically designed to tackle the challenges of cross-task zero-shot generalization. Compared to a broad range of mainstream Vision-Language Action (VLA) models, X-ICM achieves state-of-the-art performance on AGNOSTOS.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper proposes a new simulation benchmark, AGNOSTOS, to evaluate the cross-task zero-shot generalization in manipulation.\n2. X-ICM shows effective and impressive experiment results, compared to other mainstream VLA methods, e.g. pi0 and OpenVLA.\n\nWeaknesses:\n1. The construction of AGNOSTOS appears relatively straightforward, as it involves a direct partitioning of existing tasks within RLBench. The novelty of this benchmark is thus somewhat unclear.\n2. The methodological details of the proposed X-ICM approach lack sufficient clarity. For instance, it remains ambiguous how the output of the large language model (LLM) is transformed into precise low-level control commands, such as accurate gripper poses required for manipulation. A more detailed description of this mapping process is necessary to evaluate the feasibility and reproducibility of the method.\n3. Voxposer should not be categorized as a Vision-Language Action (VLA) method. While it may share some similarities in using vision and language components, its core design and objectives differ significantly from mainstream VLA models.\n4. There is a labeling error in Figure 2. where the language description in Diffusion-based dynamics modeling stage is incorrectly labeled as \"Close Jar\" instead of the correct \"Close Box.\"\n\nQUESTIONS:\n1. Tasks such as straightening a rope or removing a USB device from a computer demand high spatial resolution, yet a single grid cell may encompass multiple parts from different objects, leading to ambiguity and reduced control precision. Could the authors explain further how the proposed X-ICM framework effectively handles such tasks under these discretization constraints.\n2. Include a more detailed analysis of failure cases, distinguishing between issues related to sample selection and LLM reasoning. This could provide more insight for this paper.\n3. The performance of X-ICM on complex, long-horizon tasks remains insufficiently explored, especially in realistic or real-world settings. Could the authors provide further experiment results in real-world to show X-ICM’s practical applicability?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors present AGNOSTOS, a benchmark designed to evaluate cross-task zero-shot generalization for robotic manipulation. The analysis of various VLA models reveals significant limitations in generalizing to unseen tasks. To address this challenge, the authors introduce X-ICM, a cross-task in-context manipulation approach that harnesses the generalization capabilities of LLMs, achieving significant improvements over existing VLA models.\n\nSTRENGTHS AND WEAKNESSES:\n1) In general, the paper is clear and the technical details are easy to understand.\n\n\n2) The core idea of cross-task zero-shot generalization in vision-language-action (VLA) models and introducing a benchmark for this purpose is a valuable and timely contribution to the field.\n\n\n3) The related work section is thorough and extensive.\n\nQUESTIONS:\n1) **Comparison to RoboPrompt and KAT.** While I agree that existing in-context learning baselines such as RoboPrompt and KAT are not designed for cross-task zero-shot generalization scenarios, including them in the comparison would still provide valuable insight into their current performance.\n\n\n2) **“Dynamics-guided sample selection” Ablations.** This section lacks clarity. It would be helpful to evaluate multiple approaches or baselines to highlight the challenges in selecting effective in-context examples. The authors seem to have chosen diffusion models on purpose, but the reason for this choice is not clearly explained. Simpler methods could be considered, so the authors are encouraged to provide further details, motivations, and comparisons with alternative approaches that were tested but found less effective.\n\n\n3) The constructed prompt closely resembles the one used in RoboPrompt. If this is accurate, please ensure proper citation within lines 225-235.\n\n\nOverall, I liked this paper. However, it would benefit from additional justification for the proper use of X-ICM and a more thorough comparison to closely related baselines like KAT/RoboPrompt.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces AGNOSTOS, a simulation benchmark built upon RL-Bench, which includes 23 unseen manipulation tasks for evaluation. By testing several existing Vision-Language-Action (VLA) models on this benchmark, the authors observe that these models tend to generalize well to seen training tasks but perform poorly on unseen tasks. The authors propose a novel method called Cross-Task In-Context Manipulation (X-ICM) to address this limitation. This approach leverages large language models (LLMs) to predict actions by prompting them to demonstrate similar seen tasks used as in-context examples, which are selected via a dynamics-guided sampling strategy. This dynamics-guided sampling strategy first uses an image editing model based on InstructPix2Pix trained using the demonstration database of seen tasks to map (text, image) pairs to goal images, and then uses features derived from this model to compute similarity and retrieve the top-k most relevant demonstrations. On the proposed AGNOSTOS benchmark, the X-ICM method outperforms prominent VLA baselines such as $\\pi_0$ and VoxPoser, demonstrating its effectiveness in generalizing to unseen tasks.\n\nSTRENGTHS AND WEAKNESSES:\n=== Strengths ===\n1. The paper presents some real-world experiments beyond simulation benchmarks, which makes the effectiveness of the proposed methods more convincing.\n2. The authors present a well-motivated contribution by proposing a novel simulation benchmark named AGNOSTOS. This benchmark not only reveals that current VLA models fail to generalize effectively to unseen tasks but also provides a playground for future research on cross-task generalization in VLA models.\n3. Leveraging in-context learning and generalized features of LLM, the proposed X-ICM can perform better than diverse VLA models on AGNOSTOS, showcasing its powerful cross-task zero-shot capability.\n\n=== Weakness ===\n1. The Dynamics-guided Sample Selection module appears relatively complex, and its motivation is not justified. In the current literature on multimodal RAG, especially video retrieval (e.g., VideoAgent [1], MineDreamer [2]), it is common to use more generalizable Video-CLIP-based methods to extract features and compute similarity for retrieving relevant video demonstrations. In contrast, this paper trains an image-editing model (based on data from RL-Bench) specifically for feature extraction, which may limit generalization. This approach seems misaligned with the paper’s stated goal of cross-task zero-shot generalization. Specifically, the proposed framework may not be easily transferable to new benchmarks or environments without retraining the diffusion-based image-editing model, thus undermining the claimed generality of X-ICM.\n2. The proposed X-ICM method lacks novelty, as its within-task in-context manipulation approach is not substantially different from prior work (e.g., RoboPrompt[3]). Both methods retrieve relevant video demonstrations and then pass key annotation information to an LLM or VLM to directly generate actions, limiting the contribution of X-ICM from a methodological view.\n3. The definition of zero-shot in this work is somewhat ambiguous. The X-ICM framework retrieves several demonstrations as contextual prompts for the LLM, even though these demonstrations may differ in actions/motions and objects from the target task (especially in Level 1, where partial semantic overlap exists). However, using such retrieved demonstrations as in-context examples deviates from the conventional definition of zero-shot, and may lead to confusion.\n\n[1] VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding\n\n[2] Minedreamer: Learning to follow instructions via chain-of-imagination for simulated-world control\n\n[3] In-context learning enables robot action prediction in LLMs\n\nQUESTIONS:\n1. Why do the authors choose to use a diffusion model trained on the simulation benchmark dataset for feature extraction, rather than adopting a more generalizable Video-CLIP-based method? The former approach incurs additional training costs and lacks transferability to other benchmarks beyond RL-Bench, undermining the claimed generality of the proposed X-ICM framework.\n2. What is the success rate of the X-ICM framework without providing any demonstrations? This baseline is important to quantify the actual contribution of the in-context prompting. Additionally, why not consider training an end-to-end VLA model that supports in-context learning, such as ICRT [4], which is capable of acting without demonstrations and improves performance when examples are provided?\n3. The authors are encouraged to clarify their definition of cross-task zero-shot. As discussed earlier, the current formulation—using retrieved demonstrations from semantically similar tasks—does not strictly align with the conventional understanding of zero-shot generalization.\n\n[4] In-Context Imitation Learning via Next-Token Prediction\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new benchmark for cross-task generalization in VLAs they call AGNOSTOS. It is built to supplement 18 training tasks from RLBench to provide additional environments for evaluation. The authors then proceed to fine-tune existing VLA models on their training tasks and test for cross-task generalization. They discover that current methods do not do well on these tasks (as expected).\n\nTo start overcoming the limitations observed in current VLA models, the paper additionally proposes a mechanism for using off-the-shelf VLMs/LLMs with the RLBench data for in context learning. They dub their method X-ICM. This method leverages the cross-task generalization capabilities of Large Language Models (LLMs) by conditioning them on \"in-context demonstrations\" from seen tasks to predict action sequences for unseen tasks. The key insight is that this only works if appropriate in-context demonstrations are provided and to obtain such demonstrations they train a goal prediction diffusion model on the dataset (which is afterwards used for retrieval).\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n- The community is in need for more benchmarks and the proposed benchmark seems to address a whole in current evaluations.\n- The tasks seem to be designed thoughtfully and the split into tasks with partially shared semantics (Level 1) and entirely novel (Level 2) seems good.\n- They evaluate a meaningful set of pre-trained models from the literature and provide comprehensive evaluation results.\n- The explored in-context learning/policy elicitation technique seems simple and reasonable and is of interest to the community.\n\nWeaknesses:\n- The reviewer understands that large scale evaluations are costly. Still 25 rollouts with three repetitions for each method seems very little to judge statistical significance of results.\n- The real world experiments seem week / are relatively simple pick and place tasks.\n- It is unclear how good the in-context learning really is given that all methods score fairly low on most of the tasks (and no per task variances are given?). To calibrate it seems important to also give reference numbers for all methods on in-domain tasks. Where maybe the in-context learning is worse but it would help calibrate how far off the best possible result it is. It would also have been helpful to include some form of oracle score if possible (i.e. how good could a policy get at the tasks given X demonstrations)\n- Paper formatting seems off, see also comments below.\n- In some sense the benchmark is largely derivative of RLBench, most of it is \"just\" adding additional tasks to RLBench.\n- Details on how open models were fine-tuned seem scarce.\n- The retrieval of in-context examples seems relatively involved. It is unclear if such a complicated pipeline is truly necessary. No ablations / comparisons to simpler strategies is given.\n- The paper is mainly a systems paper proposing a new benchmark. Without the benchmark it would not pass the acceptance bar.\n- Some sentences seem ungrammatical, e.g. just from the first page:\nline 28: \"This capability-cross-task zero-shot generalization-is essential for real-world deployment, where robots are expected to tackle novel tasks as they arise dynamically.\" Should probably read something like \"The capability of cross-task (zero-shot) generalization is essential for real world deployment, [..]\"\nline 30: \"Despite the rapid progress in VLA research, [...] has focused on generalization testing in [...]\" -> \"Despite the rapid progress of research into VLAs [...] has focused on testing generalization\". VLA is not a noun, \"generalization testing\" isn't a thing.\nline 34: \"the generalization of VLA models.\" -> \"the generalization capabilities of VLA models\".\nline 60: \"Guided by the learned dynamics, the dynamics-aware prompts can be formed [...]\" -> \"Guided by the learned dynamics, dynamics-aware prompts can be formed [...]\"\n\nQUESTIONS:\nPlease address the weaknesses discussed above. In particular questiones regarding:\n- Statistical significance of results and providing expanded per-task results.\n- Questions regarding in-domain performance to help as a gauge for how well methods perform.\n- Questions regarding oracle performance on the proposed tasks.\n\nLIMITATIONS:\nLimitations are discussed adequately in the discussion section (given hte space constraints).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 1, "DATE": "2025-07-02"}