PixelVLA: Advancing Pixel-level Understanding in Vision-Language-Action Model
Authors: Wenqi Liang
Gan Sun
Yao He
Jiahua Dong
Suyan Dai
Ivan Laptev
Salman Khan
Yang Cong
Venue: ICLR 2026 Poster (2026)
Keywords: pixel-level understanding
vision-language-action model
robotics
multimodal large language model
Primary area: applications to robotics, autonomy, planning
TL;DR: We introduce PixelVLA, the first VLA designed to support both pixel-level reasoning and multimodal prompting with text and visual inputs. Our approach is built on a new visuomotor instruction tuning framework.

Abstract: Vision-Language-Action models (VLAs) are emerging as powerful tools for learning generalizable visuomotor control policies. However, current VLAs are mostly trained on large-scale image–text–action data and remain limited in two key ways: (i) they struggle with pixel-level scene understanding, and (ii) they rely heavily on textual prompts, which reduces their flexibility in real-world settings. 
To address these challenges, we introduce PixelVLA, the first VLA model designed to support both pixel-level reasoning and multimodal prompting with text and visual inputs. Our approach is built on a new visuomotor instruction tuning framework that integrates a multiscale pixel-aware encoder with a visual prompting encoder. To train PixelVLA effectively, we further propose a two-stage automated annotation pipeline that generates Pixel-160K, a large-scale dataset with pixel-level annotations derived from existing robot data.
Experiments on three standard VLA benchmarks and two VLA model variants show that PixelVLA improves manipulation success rates by $10.1\%\sim28.7\%$ over OpenVLA, while requiring only $1.5\%$ of its pretraining cost.  These results demonstrate that PixelVLA can be integrated into existing VLAs to enable more accurate, efficient, and versatile robot control in complex environments. The dataset and code will be released as open source.

Published as a conference paper at ICLR 2026
PIXEL ADVANCING PIXEL-LEVEL UNDER- STANDING IN VISION-LANGUAGE-ACTOON MODEL
Yao Yao Salman Khan Yang Cong?
Wenqi Liang", 1.2 Gan Sun
Jiahua Dong?, Suyan Dai?, Ivan Laptev?,
University of Trento
2 School of Automation Science and Engineering, South China University of Technology
3 Mohamed bin Zayed University of Artificial Intelligence
4 Australian National University
{1 iliangeenni0122
sungan1412, heyao0293, dongjiahual995,
congyange1 gmaal. com, {ivan, {ivan. laptev, salman khan laaannemmzuui.com ac
f Corresponding Author
Action Decoder
7D Robot Action
Action Decoder
7D Action Action
47.7
$4.0
13.5
OpenVLA
Image-level Underslanding
PixelVLA
Pixel-ievel Understanding
2r
P5
13
LLM Tokenizer
Vision Encoder
LLM Tokenizer
Pixol Encoder
Vision Encoder
78.5
Wicoow Roodt
40
80
50
What should the robat do to Put Eggplant User into
What should the robot do Uper Put <object> into Pot.
LIBERO
referring to
?
OpenttL
TrocoMLA
Fig
RubovLMM
Orto
PackLL
(a)
proposed
[c] Porformace Comparison
Figure I: We introduce PixeIVLA, vision-language-actoon (VLA) model designed for pixel-level reasoning and multimodal prompting. Unlike prior VLA models (a). which primarily rely on image- level understanding for manipulation and depend solely on textual instructions, PixeIVLA (b) ad- vances beyond these limitations by enabling fine-grained pixel-level comprehension and supporting both textual and visual prompts. This paradigm effectively enhances spatial precision and expands human-robot interaction, leading to superior performance (c) compared to baseline methods.
ABSTRACT
TisionLLaguaae-Action models (VLAs) are emerging as powerful tools for learn- ing generalizable visuomotor control policies. However, current VLAs are mostly trained on large-scale image- mmage-text-aee text-action data and remain limited in two key ways: (i) they struggle with pixel-level scene understanding, and (ii) they rely heavily textual prompts, which reduces their flexibility in real- world settings. To ad- dress these challenges, we introduce PixeIVIAA, the first VLA model designed support both pixel-level reasoning and multimodal prompting with text and visual inputs. Our approach is built on new visuomotor instruction tuning framework that integrates multiscale pixel-aware encoder with visual prompt- aware encoder. To train PixeIVLA effectively, we further propose a two-stage automated annotation pipeline that generates Pixel-160K, large-scale dataset with pixel-level annotations derived from existing robot data. Experiments on three standard VLA benchmarks and two VLA model variants show that Pix- eIVLA improves manipulation success rates by 10.1% ~ 28.7% over Open- VLA, while requiring only 1.5% of its pretraining cost. These results demon- strate that PixeIVLA can be integrated into existing VLAs to enable more accu- rate, efficient, and versatile robot control complex environments. Project page: https:: bttps://wett //wenqi linng, github. io/PixelVia.

---

Published as a conference paper at ICLR 2026
1 INTRODUCTION
Traditional robotic policy learning methods (Brohan et al. (2022); Liang et al. (2024); Chi et al. (2023); Dong et al. (2026)) rely heavily on task-specific demonstration datasets (James et al. (2020); Liu et al. (2023a)), which limits their ability to generalize to outoo--distrbuution (OOD) tasks. In contrast, saaion-lnguage-cttoon models (VLAs) (Brohan et al. (2023); Kim et al. (2024); Black et al. (2024)) leverage large-scale robot datasets together with pre-trained vision-language models (VLMs), achieving stronger generalization and instruction-followingg capabilities. For example, RT- 2 (Brohan et al. (2023)) integrates internet-scale VLMs with robotic control, enabling semantic reasoning and manipulation of novel objects. Similarly, Open VLA (Kim et al. (2024)) leverages Prismatic VLM (Karamcheti al. (2024)) backbone to conduct large-scale training the OXE dataset (O Neill et al. (2024)), leading to significant improvement in OOD generalization.
Despite recent progress, as shown in Fig. (a), most VLAs (Kim et al. (2024); Han et al. (2026); Shi et al. (2025)) inherit from VLMs that process observations only at the image level, lacking fine-grained pixel-level understanding. This gap limits spatial reasoning and weakens OOD gener- alization. In contrast, pixel-level comprehension has already been successfully validated in VLMs (Ren et al. (2024); Zhang al. (2024b)) and enables precise object perception and richer spatial awareness, which are key for robust manipulation in diverse environments. The second limitation lies in prompting. Most VLAA depend solely on textual instructions, which overlook subtle visual cues and constrain spatial awareness (Ranasinghe et al. (2024)) and multimodal human-robot in- teraction (Jiang et al. (2023); Zheng et al. (2024); Li et al. (2026)). To explore visual prompting in VLAs, VLA (Zheng et al. (2024)) improves spatial-temporal awareness with visual traces, and LLaRA (Li et al. (2025b)) encodes object locations within textual prompts to enhance region- level understanding. Nevertheless, these approaches still face challenges in achieving fin-grriine pixel-level understanding and effectively integrating diverse multimodal prompts (e.g., points, lines, regions, masks) (Wu et al. (2024b); Dong et al. (2024)).
Inspired by the successful visual instruction tuning in VLMs (Liu et al. (2023b); Karamcheti et al. (2024)), we introduce novel visuomotor instruction tuning framework to train VLA models. This framework is designed to significantly enhance the pixel-level understanding capabilities of VLAs and empower them to effectively process multimodal visuomotor control prompts. However, current robotic datasets "eiNeil et al. (2024); Khazatsky et al. (2024)) lack multimodal prompts and pixel-level annotations. Meanwhile, directly employing existing VLMs and open-set segmentation models (Karamcheti et al. (2024); Liu et al. (2024a)) to extract visual prompts and pixel-level anno- tations proves to ineffective. This due to significant domain gap between their pre-training data and robotic data, as well as the cluttered and low-quality nature of robotic images.
To tackle the above challenges, as presented in Fig. (b), we introduce PixeIVLA in this paper, the first vision-languuae-action model that achieves both pixel-level understanding and multimodal prompting. The model architecture of PixeIVLA comprises a pre-trained VLMs as backbone, a visual prompt-aware encoder, multiscale pixel-aware encoder and continuous action decoder. Specifically, in PixeIVLA, we introduce a lightweight visual prompt-aware encoder to process the diverse visual prompts (e.g., points, lines, regions, masks). Subsequently, novel multiscale pixel- aware encoder is designed to generate pixel-aware embeddings to inject pixel-level understanding into VLAs. Furthermore, we develop a continuous action representation decoder that leverages pixel-level understanding to capture fine-grained action details based on the hidden states of VLMs.
To address the challenge of synthesizing high-quality multimodal prompts and pixel-level anno- tations from cluttered, low-quality robot observations, we propose a two-stage automated annota- tion pipeline to create a pixel-annotater visuomotor instruction tuning dataset, namely Pixel-160K. Concretely, our two-stage automated annotation pipeline comprises gripper-aware region proposal stage followed by a multimodal object segmentation stage. In the first stage, a video segmenta- tion model is employed to localize the robot gripper and generate preliminary region proposals for target objects. Subsequently, the second stage leverages a large language model (LLM) and an open-vocab segmentation model to predict pixel-level annotations and produce multimodal prompts from these region proposals. Thereafter, we train PixeIVLA using the proposed visuo- motor instruction tuning framework, which incorporates continuous action training stage and pixel-level understanding enhancement stage. To evaluate the effectiveness of PixeIVLA, we inte- grate its architecture and visuomotor instruction-tuiing procedure into two widely adopted VLAs:
2

---

Published as a conference paper at ICLR 2026
VLe (O'Neill et al. (2024)) and To (Black et al. (2024)). Extensive evaluations on three VLA benchmarks demonstrate that PixeIVLA advances current VLAs to achieve superior performance in zero-shot manipulation tasks and adaptation to new robot setups, while requiring only 1.5% of the pretraining computation of Open VLA.
The main contributions of this paper are listed below:
We present PixeIVLA, a novel asioo-language-action model enabling pixel-level under- standing while supporting both textual and visual prompts. In PixeIVLA, we introduce lightweight visual prompt-aware encoder to process diverse visual prompts, a novel multi- scale pixel-aware encoder for pixel-level understanding injection, and a continuous action decoder to generate robotic action.
We design novel two-stage automated annotation pipeline to effectively create a pixel- level visuomotor instruction tuning dataset from the publicly available robot datasets, called Pixel-160K, where the pipeline comprises the gripper-aware region proposal stage and the multimodal object segmentation stage.
We introduce a novel visuomotor instruction tuning framework for training PixeIVLA, comprising continuous action training stage and pixel-level understanding enhance- ment stage. Extensive evaluations on three benchmarks and two VLA model variants show that PixeIVLA improves performance of current VLAs with relatively low training cost.
2 RELATED WORK
Aision-Language-e Action Models. Vision-language-actio models (VLAs) (Team et al. (2025); Brohan et al. (2023); Black et (2024); Ding et al. (2024); Fan et al. (2025)) have propelled robotic manipulation forward by endowing robots with the ability to understand and execute language- based instructions in diverse visual environments. Trained on numerous robot episodes, OpenVLA (Kim et al. (2024)) enables zero-shot control and adaptation for various robots. Building the foun- dational capabilities of OpenVLA various approaches have been proposed to advance robotic ma- nipulation, such SpatialVLA (Qu et al. (2025)) and ECoT (Zawalski et al. (2024)). Most prior VLAs focus on innovations in visual processing for robotic manipulation, such as introducing visual chain-of-though (CoT) reasoning mechanisms for visual planning (Zhao al. (2025)). Neverthe- less, they primarily process visual information at the image level, lacking the ability to perform detailed pixel-level visual processing required for precise robotic manipulation.
Visual Prompting in VLMs. Visual prompting methods (Zhang et al. (2023); Ren et al. (2024); Wu al. (2024a); Zhang et al. (2024b)) have recently emerged as a complementary paradigm to tex- tual prompting, allowing models to accept more fine-grate inn-ggrined supervision in the form of region-level (Guo et al. (2024)) and even pixel-level instructions (Ma al. (2024); Rasheed et al. (2024) over multimodal inputs. Shikra (Chen et al. (2023)) extends MLLMs with simple Vision-encoder-LLMM architecture that treats spatial coordinates as natural language tokens. Ferret (You et al. (2023)) en- hances region-level grounding in MLLMs through hybrid region representation and a spatial- -wwae visual sampler that supports diverse region inputs, while Ferret-v2 (Zhang et al. (2024a)) further introduces any-resolution grounding and multi-granularity visual encoding, leading improved fine-grained visual understanding and localization over prior MI.LMs. LocVLM (Ranasinghe et al. (2024)) proposes an image-space coordinate- instruction fine-tuning framework for MLLMs that explores coordinate representations and spatial objectives to inject spatial awareness. However, despite these advances, robust pixel-level understanding in VLA frameworks remains challenging, especially when aligning fine-nrained spatial cues with continuous, nigh-precision action control.
Visual Instruction Tuning in VLAs. Visual Instruction Tuning (Zhu et al. (2023); Liu et al. (2023b); Rasheed et al. (2024)) is generally divided into two steps. which are modality alignment and instruction optimization, respectively. This strategy also serves as the core paradigm for re- alizing multimodal capabilities in VLAs (Li et al. (2025b), Zheng et al. (2024); Zawalski et al. (2024)). For example, TraceVLA (Zheng et al. (2024)) introduces visual trace prompting to en- hance spatial-temporal awareness in VLAs. In contrast, LLaRA (Li et al. (2025b)) reformulates the robot action policy as visuo-textual conversations through visuomotor instruction tuning and ROVI (Li et al. (2025c)) develops an object-centric visual instruction paradigm with symbolic sketches. However, to address various visuomotor control challenges, adapting visual instruction tuning for VLAs remains major constraint.
3

---

Published as a conference paper at ICLR 2026
Robot
Continuous Action Decoder
PixeIVLA
LLM Hidden States
MUT level Fosl Ire.
Linear Projector
Plad award Eriibddiing
plantfra Kooks (3) Mulliscale Pixel-aware aware Enzoder
LLM (Llama 7B)
E
Vixual Prompting
Bnooder
E
E
MLP Projector
Muttiscale Pixel- aware Encoder
Visual Prompt- aware Encoder
(b) Visual Prompt-swars Encoder
Liama Tokenizer
Dino V2
SigLIP
Linear Projecor
Linerr Projector
MLP
Put This Eggplant in Yellow Basket
Reshee Blocks
Language Instruction
Robotic Observation
Pixel-eveel Masks Visual prompts
11M Haddin Stades
Continuous Action Decoder
Robot-Action
The model integrates three novel components: (1) visual prompt-aware encoder for processing input diverse visual prompts; (2) multiscale pixel- aware encoder that injects pixel-level information into token embeddings; and (3) a contimuous action decoder predict 7D robot actions. PixeIVLA enhances fine-grained pixel-level spatial understanding and multimodal prompt responsiveness, enabling more precise manipulation policies in visually complex scenarios.
Figure 2: Overview the PixeIVLA architecture.
PROBLEM DEFINITION: VISUOMOTOR INSTRUCTION TUNING
Inspired by the effectiveness of visual instruction tuning in VLMs (Liu et al. (2023b); Rasheed al. (2024); Wu et al. (2024a); Kang et al. (2025); Li et al. (2025a)), we aim to adapt similar process for VLAs tackle diverse visuomotor control challenges (e.g., various multimodal prompts) and achieve pixel-level understanding. Similar to A (Li et al. (2025b)), we formalize this paradigm as Visuomotor Instruction Tuning. Specifically, following OpenVL.A (Kim et al. (2024)), given series of image observations X {x' 1 Jt=1 and a language instruction L, the VLA model F.) can generate a series of robotic actions A {a' E IR [R")=- i.e., af = Fx L). For episode of length T, the likelihood of successfully completing the task through an action sequence can be calculated as:
T
P (AXXL
II
po xx L):
where T denotes the length of timestep in an episode, 0 represents the parameters of VIAA model Fe(-) and ps denotes the likelihood of generating action a' by the VLA model Fs(-). How- this robotic action generation process fails to accommodate various visual prompts and achieve fine-grained pixel-level understanding. To address these challenges, we here introduce novel visuomotor instruction tuning framework that reformulates robotic action generation as a' F ,'x''pp' p', L, V) and reformulate the likelihood in Eq. (1) for an episode of length T as:
P.A. = II (a "'',''
(2)
where P (p' E #RWYT Tr p' represents the pixel-aware mask input, and V denotes the diverse visual prompts (e.g., points, lines, regions, masks).
4 THE PROPOSED METHOD
As illustrated in Fig. 2, we present the architecture of the proposed PixeIVLA to achieve pixel- level understanding and accommodate both textual and visual prompts. Specifically, PixeIVLA integrates novel multiscale pixel-aware encoder (Sec. 4.1) that infuses pixel-level understanding into VLAs through tokenized representations, a visual prompt-aware encoder for handling diverse visual prompts (Sec. 4.1), and continuous action decoder (Sec. 4.1) for accurate robotic action
4

---

Published as a conference paper at ICLR 2026
prediction. In addition, automated annotation generation pipeline and pixel-annotated visuo- motor instruction tuning dataset Pixel-160K are presented in Sec. 4.2. Subsequently, we introduce the proposed visuomotor instruction tuning procedure for training PixcIVLA in Sec. 4.3.
4.1 PIXELVLA ARCHITECTURE
Current VLAs (Black al. (2024); Kim et al. (2024); Wang et al. (2026b:a)) are typically pre- robotic datasets (O (O'Neill et al. (2024); Wu et al.
trained on large-scale mmage-instructionaatiin (2024c)). Architecturally built upon VLMs, these models process single or multi-view images along with textual instructions. However, this foundation inherently restricts their ability to achieve pixel- level understanding or respond to detailed visual prompts, resulting in constraining VLAs for spatial comprehension and object perception.
To address these architectural constraints, as illustrated in Fig. 2, we present a novel VLA model, namely PixcIVLA. Specifically, PixeIVLA integrates four main parts: (1) vision encoder and MLP projector for visual embedding extraction, (2) a visual prompt-aware encoder and a mul- tiscale pixel-aware encoder for accommodating visual prompts and pixel-level understanding in- jection, (3) a LLM backbone and (4) a continuous action decoder for non-discrete robot ac- tion prediction. Following Open (Kim et al. (2024)), we preliminarily build our PixeIVLA on Prismatic-7B VLM (Karamcheti et al. (2024)), where a Llama 2-7B (Touvron et al. (2023)) is em- ployed as LLM backbone. The vision encoder of PixelVLA consists of pre-trained Dino (Oquab et al. (2023)) and SigLIP (Zhai et al. (2023)) models, and lightweight 2-layer MLP projector is utilized to map the output features of the vision encoder into the input space of L.LMM
Multiscale Pixel-aware Encoder. To extract pixel-level information from multiscale image features and encode the spatial positional information of visual prompts, we propose a multiscale pixel-aware encoder designed to generate both pixel-aware embeddings and prompt-aware embeddings. Specif- ically, as described in Eq. (2), for each training sample [x" p L", V"j drawn from Pixel-160K PixelVLA first encodes the image observation x" E IR NxW X x3 with the SigLIP vision encoder to 1i=1, where L denotes the number of
obtain multi-level visual features Fo {f.. E selected feature levels. As illustrated in Fig. 2 (a), the multiscale pixel-aware encoder leverages the features F and pixel-aware mask input pd E HRH to compute the pixel-aware embed- dings E. TR N,DD Here, N, is the length of pixel-aware embeddings and D denotes the feature can be computed as follows:
dimension of LLM. Specifically, the pixel-aware embeddings ED
Dd P..s
ED MLP (T)))) fo.
where MLP(-) is multilayer perceptron (MLP) layer and (-)-) denotes the linear projection in the i-th linear projector. Supervised by the action prediction loss, PixeIVLA learns to associate the pixel-level information encoded in these pixel-aware embeddings EG. with action generation, thereby enhancing the VLA backbone with pixel-level understanding.
Visual Prompt-aware Encoder. As shown in Fig. 2(b), adopt lightweight prompt encoder similar to that in SAM (Kirillov et al. (2023)) and integrate it into PixeIVLA as the visual prompting encoder. Concretely, the user-provided prompts v e # are first converted into continuous po- sitional embeddings based on their normalized image coordinates, and then combined with learned prompt-type embeddings to produce prompt features FD xD.,, where N, is the embedding length and D, is the feature dimension. These features F are further transformed by an MLP to obtain the final prompt-aware embeddings E. N. Since each embedding explicitly tied to specific location or region in the image via its coordinate-baus positional embedding, the spatial positional information of the visual prompts is preserved throughout the encoding process.
Continuous Action Decoder. Most existing VLA models (Kim et al. (2024); Zheng et al. (2024); Li et al. (2024b)) adapt autoregressive generation to predict sequential action tokens based on the pre-trained VLM backbone. In contrast, following (Black et al. (2024)), we develop contin- uous action decoder that directly predicts continuous action representations, leveraging lexel-level understanding to capture fine-gra -nniindd action details. Specifically, illustrated in Fig. 2 (b), the hidden states F' E IR N2 from the last layer of LLM backbone are sequentially processed by a linear projector, N, ResNet blocks and MI. projector obtain the actions A N.7. Here,
5

---

Published as a conference paper at ICLR 2026
Pay.
(a) Top 10 Dataset Skill
Contalners
150-
12.0-
10.0-
500
Cojects
(c) observation Pixel-18KK Figure 3: Overview of the Pixel-160K Dataset.
N, denotes the sequence length of the LLM backbone, while N: represents the chunk size used in action chunking (Zhao et al. (2023)). this way, we can effectively preserve the pixel-level under- standing learned by the pre-trained VI.M backbone while enabling the continuous action decoder to incorporate these features directly into the continuous action prediction. The resulting continuous actions then used compute an L1 regression loss that supervises the training process.
4.2 VISUOMOTOR TUNING DATA GENERATION
In this section, we introduce Pixel-160K shown in Fig. 3, a visuomotor instruction tuning dataset comprising image-text-action triplets with visual prompts and mask annotations, containing approx- imately 160K manipulation episodes to encourage VLAs for fine- graiind pixel-level understand- ing. Specifically, to address the challenge of cluttered and low-quality robot observations in robot datasets, we propose an automated annotation pipeline containing gripper-aware region proposal stage and a multimodal object segmentation stage. This pipeline enables the effective generation of visual prompts and mask annotations for each episode using the publicly available Fractal dataset (Brohan et al. (2022)) and Bridge v2 dataset (Walke al. (2023)).
Gripper-aware Region Proposal Stage. Given sequence of observations X2 x., " from the 77th episode, the first gripper-close state in the episode G, (1,2...., No, and Here. N, represents the length of the 1/tth episode. Sequentially, we can select series of gripper-close state observations C2
the corresponding observation as Xi, G. E {x x2 Xy
GN. from the whole dataset, where N, denotes the number of total episodes in the (x1 , X2 ..... N.
xx dataset. Furthermore, we assume {xf' ; ..... XN, GN. as a discrete video and apply SAM 2 (Ravi al. (2024)) to generate Ne gripper masks. Then, compute the minimal axis-aligned bounding boxes enclosing these masks, uniformly enlarge each by a fixed margin to capture local context and reduce detection noise, and take the resulting boxes as the N₂ region proposals (R,, R2,..., R. )- Here, R,, E Rd is the proposal for the 17th episode. In this way, the region proposals can be lever- aged accurately capture object from cluttered and low-quality robot observations.
Multimodal Object Segmentation Stage. Given a manipulation instruction such as "Put the Egg- plant in Yellow Basker"", employ Llama 2-7B to reason over the instruction and extract the textual description of the target object to be manipulated, e.g., "Eggplant". For the 17-h episode, we then provide the target object text along with the region proposal {R., into an open- open-vocabulary object detector Grounding DINO (Liu et al. (2024a)) and SAM (Kirillov et al. (2023)). These models detect all relevant object instances, generate their mask annotations, and associate them with the corresponding language expressions from the target object text. We then filter the predictions based on their confidence scores, retaining only the mask annotations within the bounding box that has the highest box cox-confidence. Sequentially, sampling points within the mask, generating random lines inside the object region, and extracting external bounding boxes through mask contour detection.
derive visual prompts from the object masks by randomly
Finally, we apply the proposed two-stage automated annotation pipeline to the publicly available Fractal dataset (Brohan et al. (2022)) and Bridge v2 dataset (Walke et al. (2023)). We first au-

---

Published as a conference paper at ICLR 2026
Table 1: SimplerEnv (Li et al. (2024c)) simulation evaluation results in terms of the average success rate for the Google Robot setup. VM denotes Visual Matching and VA is Variant Aggregation. and denote tuning- based methods applied the pretrained weights of Open and To, respectively.
Pick Coke Can
Move
Near
Open/Close Drawer
Average
Methods
VM
VA
VM
VA
VM
VA
VM
VA
RT-1-X --X (O'Neill et (2024))
56.7
49.0
32.3
59.7
29.4
49.4
36.9
Octo- Base (Team et al. (2024))
17.0
0.6
4.2
3.1
22.7
1.1
14.6
1.6
HPT (Wang et al. (2024))
56.0
60.0
24.0
46.7
Robo (Liu et al. (2025))
72.7
68.3
66.3
56.0
26.8
8.5
56.3
46.3
Dita (Hou al. (2025))
83.7
85.5
73.0
46.3
37.5
68.7
65.3
SpatialVI.A (Qu et al. (2025))
81.0
89.5
69.6
71.7
59.3
36.2
71.9
68.8
VLe (Kim (2024))
16.3
54.5
46.2
47.7
35.6
17.7
32.7
40.0
OpenVLA-SOV
17.5
51.9
44.6
42.3
32.8
16.8
31.6
38.6
TraceVLA (Zheng et al. (2024))
28.0
60.0
53.7
56.4
57.0
31.0
46,2
49.1
PixeIVLA
81.7
72.7
60.1
57.7
42.3
20.0
61.4
50.1
TO (Black et al. (2024))
72.7
75.2
65.3
63.7
38.3
25.6
58.8
54.8
To-SFT
70.8
72.1
64.2
61.3
36.8
28.3
57.3
53.9
PixelVLA-To
80.7
76.8
67.7
62.0
41.3
30.8
63.3
56.5
tomatically discard samples with failed mask generation (egg, empty or invalid masks) using a simple script. and then the authors rapidly inspect the remaining samples to remove those with clearly incorrect masks. In total, this process filters out approximately 19.2% of the generated samples. The resulting dataset, Pixel-160K, contains 160K robot manipulation episodes and 6.5M image-text-acee triplets with visual prompts and mask annotations.
4.3 VISUOMOTOR INSTRUCTION TUNING PROCEDURE
To advance fine-grained pixel-level understanding in VLAs, we propose a novel visuomotor in- struction tuning procedure, consisting continuous action training stage and a pixel-level under- standing enhancement stage. Concretely, the first continuous action training stage enables the VLA model acquire robust continuous action representations from large mixture of image- texti datasets. In the second stage, pixel-level understanding is explicitly enhanced by adapting the pre- trained model on Pixel- 160K dataset through LoRA adaptation (Hu al. (2022); Li et al. (2025d)). The following sections elaborate on the key designs of this two-stage training strategy.
Continuous Action Training Stage. Before training, we initialize the vision encoder, the MLP pro- jector, and the LLM backbone in PixeIVLA with the pretrained weights of VLAs (Kim et al. (2024); Black et al. (2024)), which has been trained on the large-scale mixture dataset OXE (O Neill et al. (2024)). In addition, during this stage, the visual prompt-aware encoder and the multiscale pixel- aware encoder of PixeIVLA are removed, while other modules except the continuous action decoder frozen preserve the general manipulation knowledge learned in the pretrained VLAs. To directly map the final hidden states of the last layer of LLM to continuous action values, we follow (Zhao al. (2023); Kim al. (2025)) to implement LI regression to align predicted actions generated by the proposed continuous action decoder with the ground-truth actions. Unlike Open- VLA, which represents actions as discrete tokens by normalizing each action dimension to [-1,+1] and uniformly discretizing it into 256 bins, PixeIVLA directly predicts continuous action values, thereby avoiding the loss of fine -en-grr action details introduced by discretization. Furthermore, during this stage, we train PixeIVIA on mixture of Fractal dataset and Bridge v2 dataset.
Pixel-level Understanding Enhancement Stage. Originally, most existing visuomotor instruction tuning methods (Li et al. (2025b); Kim et al. (2025); Yang et al. (2025)) focus on image-level under- standing. In contrast, at this stage, to enhance pixel-level understanding of PixeIVLA, we employ LoRA adaptation to efficiently fine-tune PixeIVLA's LLM backbone on Fixel-160K dataset, while jointly training the visual prompt-aware encoder along with the multiscale pixel-aware encoder. Meanwhile, the continuous action decoder is optimized while the remaining PixeIVLA modules remain frozen. Furthermore, we adopt the same LI regression loss and continuous action represen- tation strategy those employed in the continuous action training stage. At each training step, given

---

Published as a conference paper at ICLR 2026
Table 2: Evaluation results from the SimplerEny simulation for the WidowX robot. Gra. denotes the average grasp success rate, and Suc. is the overall task completion success rate.
Put Spoon
Put Carrot
Stack Blocks
Put Eggplant
Average
Methods
Gra.
Suc.
Gra.
Suc.
Gra.
Sue.
Gra.
Suc.
Gra.
Suc.
RT-1-X (O'Neill et al. (2024))
16.7
0.0
20.8
4,2
8.3
0.0
0.0
0.0
11.5
1.1
Octo-Base (Team et al. (2024))
34.7
12.5
52.8
8.3
31.9
0.0
66.7
43.1
46.5
16,0
Octo-Small (Team al. (2024))
77.8
47.2
27.8
9.7
40.3
4.2
87.5
56.9
58.4
29.5
RoboVLMs (Liu et al. (2025))
37.5
20.8
33.3
25.0
8.3
0.0
0.0
19.8
13.5
SpatialVLA (Qu et al. (2025))
25.0
20.8
41.7
20.8
58.3
25.0
79.2
70.8
51.1
34.4
OLe (Kim al. (2024))
4.1
0.0
33.3
0.0
12.5
0.0
8.3
4.1
14.6
1.0
Open
8.4
0.0
35.1
12.8
10.5
0.0
16.5
8.4
17.6
5.3
PixeIVLA
20.8
4.2
37.5
20.8
16.6
0.0
79.2
41.7
38.5
16.7
20 (Black et al. (2024))
45.8
25.0
50.0
91.6
62.5
53.1
27.1
To S0-SFT
45.3
26.8
28.6
4.2
52.3
18.6
88.5
59.6
53.7
27.3
PixelVLA-TO
51.7
32.4
28.7
16.7
56.8
21.7
83.3
61.7
55.1
33.8
mini-batch fx', p', L', v Ji=1 sampled from the Pixel-160K dataset, the forward process at single timestep of this stage can then be formulated as follows:
PizelV LA
Σ ||a C(H(E) Ej. E. E
where C(-) refers to the continuous action decoder and H represents the LLM backbone of Pix- eIVLA. addition, B denotes the mini-batch size and || || is the LI norm used for regression. Notably, E, Ej, E E; correspond to the visual embeddings produced by the vision encoder and MLP projector, the language embeddings from the LLM tokenizer, the pixel-aware embeddings and the prompt-aware embeddings, respectively.
5 EXPERIMENTS
We conduct experiments to investigate how PixeIVLA leverages pixel-level understanding and mul- timodal prompts enhance the performance of current VLAs in both in-domain and out-of-domain adaptation. To achieve this objective, we develop three experimental paradigms: (1) zero-shot object manipulation comparisons for oft-of-domain generalization (Sec. 5.2), (2) adaptation to new robot setups to evaluate in-domain robustness (Sec. 5.3), and (3) series of ablation studies to quantify the contribution of each individual module within PixeIVLA (Sec. 5.4).
5.1 EXPERIMENTAL SETUP
Evaluation tasks. We conduct all experiments three simulation benchmarks, i.e., SimplerEnv- Google Robot (Li et al. (2024c)), SimplerEnv-Widon (Li et al. (2024c)) and LIBERO (Liu et al. (2023a)). SimplerEnv (Li al. (2024c)) is an open-source simulation suite that facilitates repro- ducible and scalable evaluation of robot manipulation policies by explicitly addressing visual and dynamic gaps between simulation and real hardware. In light of this, we conduct zero-shot object manipulation comparisons on SimplerEnv. In addition. following OpenVLA (Kim et al. (2024)), we evaluate performance of new robot adaptation across four task suites within LIBERO (Liu al. (2023a)), iee., LIBERO-Span Spattal,, LIBERO-Object, LIBERO-Gon Goal and LIBERO-Long.
Implementation Details. evaluate the effectiveness of PixeIVLA, we apply its architecture and the proposed visuomotor instruction-tuning procedure to two widely-used VLAs, OpenVLA (Kim et al. (2024)) and To (Black et al. (2024)). Regarding the training data, PixeIVLA trained in two stages: the first stage utilizes real-robot demonstrations from the Fractal dataset (Brohan et al. (2022)) and Bridge v2 dataset (Walke et al. (2023)), while the second stage employs 160K real- robot demonstrations from the proposed Pixel- 1x0el160K dataset. For input robot observations across all datasets. PixeIVLA conditioned solely on a single third-person camera view and processes images resolution of 224x224 pixels. In all training stages, set action chunk size to 8 for
8

---

Published as a conference paper at ICLR 2026
Table 3: LIBERO Simulation Benchmark Results. We report the success rates of each method across four task suites. Models including Octo, OpenVLA, VLA, Dita, SpatialVLA and PixcIVLA adapted through fine-tuning. R. represents the success rate ranking in each task suite.
Long Suc.(1) R.(.) Suc.(1) R.(4) Suc.(1) R.(()) Suc.(t) R.(.))
Spatial
Object
Goal
Methods
Average
Diffusion Policy (Chi et al. (2023))
78.3
92.5
68.3
50.5
72.4
Octo (Team et al. (2024))
78.9
85.7
84.6
51.1
75.1
CLT-VLA (Zhao al. (2025))
87.5
91.6
87.6
69.0
81.1
Dita (Houu et al. (2025))
84.2
96.3
85.4
63.8
82.4
SpatialVLA (Qu (2025))
88.2
89.9
78.6
55.5
78.1
OpenVL.A (Kim et al. (2024))
84.7
88.4
79.2
53.7
76.5
TraceVLA (Zheng al. (2024))
84.6
89.9
78.6
55.5
78.1
PixeIVLA
88.5
90.0
85.8
82.6
86.7
the continuous action decoder, i.e., the predicted action a E R The first training stage involves training PixelVLA for 100k steps with a batch size of 32 and a learning rate of 5 x 10-4. Notably, in light of the effectiveness action expert in To. we omit the first training stage when adapting PixeIVIAA on During the second training stage, we fine-tune the LLM backbone of PixeIVIAA using LoRA adaptation with rank 32 and learning rate of x 10-3 In addition, adapt PixeIVLA to the LIBERO benchmark (Liu et al. (2023a)), we fine-tune the pre-trained model for 150K steps on each task suite using LoRA adaptation with rank 32, batch size of 32, and learning rate of 5 x 10 In addition to the two baseline VLAs, Open (Kim et al. (2024)) and Ty (Black et al. (2024)), we compare the performance of PixelVLA against other state-of oofte--f--he-aae art VLAs, such as RT-1 (Brohan et al. (2022)), HPT (Wang et al. (2024)), Octo (Team et al. (2024)), TraceVLA (Zheng et al. (2024)), Robo (Liu et al. (2025)), Dita (Hou al. (2025)) and SpatialVLA (Qu al. (2025)). To further ensure fairness, we additionally fine-tune To and Open VLA on the Fractal and Bridge datasets, obtaining baselines denoted -SFS and VL-SVL.
32. This stage trained for 200k steps with batch size of
VLA
5.2 ZERO-SHOT OBJECT MANIPULATION COMPARISONS
This subsection evaluates the zero-shot manipulation performance of our model against baseline VLAs across multiple task categories and robot platforms. As
70
Le Me/orr
Success Rate (g)
shown in Tab. 1, on the Google Robot
setup PixeIVLA achieves an average VM score of 61.4 and VA score of 50.1.
ming
For
Change
surpassing Open by 28.7/10.1 and VLASFL by 29.8/11.5 in VM/VA,
respectively. These results indicate a strong capability both pixel-level derstanding and adaptation to textual and visual prompts in out-of-domain adapta- tion. Notably, as shown in Fig. 4, Pix- eIVLA outperforms VLA and Open- VLA across various environmental varia- tions, highlighting the effectiveness of the proposed visuomotor instruction tuning procedure in addressing out-of-domain generalization.
Diarnii
Tatal
Varistions
Intial States
Figure 4: Performance comparison of Ope TraceVLA and PixeIVLA performance across vari- ous environmental variations SimplerEnv-Google Robot setup: camera orientations, lighting, back- ground, distractors, and table texture.
Similar trends are observed on the WidowX robot setup in Tab 2, where PixeIVLA-mo achieves an average grasp score of 55.1 and success score of 16.7, outperforming the baseline To by 2.0 and 6.7, respectively, and surpassing RoboVLM by 35.3 and 18.2. The results strongly affirm that Pix- eIVLA's architectural innovations including its multiscale pixel-aware encoder and integration of visual prompts, significantly enhance its zero-shot perceptual and operational capabilities. Further- more, the significant improvements of PixeIVLA and PixeIVLAA-To over the baselines OpenVL.

---

Published as a conference paper at ICLR 2026
and Ty demonstrate that incorporating the finer-grained pixel-level spatial comprehension into isting VLAs enables more effective adaptation to unseen objects.
5.3 NEW ROBOT SETUPS ADAPTATION COMPARISONS
To evaluate the adaptability of PixeIVLA to novel robotic setups and task configurations, we employ the proposed automated annotation pipeline to process the LIBERO benchmark training data (Liu al. (2023a)), yielding the LIBERO-Pixe dataset. Subsequently, as summarized in Tab. 3, Pix- eIVLA achieves state-of-the-art performance with an average success rate of 86.7 7 across all tasks, significantly surpassing strong baselines. addition, As shown in Tab. 3, PixeIVI.A outperforms VLE across all four LIBERO task suites, with particularly large gains LIBERO-Long We attribute this to the Continuous Action Decoder: action chunking helps the policy capture longer- range temporal dependencies (Liu et al. (2024b)), while continuous action prediction mitigates com- pounding discretization errors (Black et al. (2024)) over long- horizon manipulation. These superior results demonstrate the enhanced adaptability of PixeIVLA new robotic setups, highlighting the effectiveness of pixel-level visual understanding and continuous action representation in PixeIVLA. Notably, PixeIVLA achieves significant performance in the LIBERO-Long setup, demonstrating its effectiveness in long-range manipulation.
5.4 ABLATION STUDIES
Table 4: Quantitative ablation studies on Variant Ag- gregation for the Google Robot setup, evaluated in the SimplerEnv simulation environment (Li et al. (2024c)).
This subsection evaluates the effectiveness of individual components in PixeIVLA on SimplerEnv-Google Robot in terms Variant Aggregation. As shown in Tab. 4, we use Open as the baseline model. Here, Baseline+FT refers to fine-tuning OpenVLA directly on a mixture of Frac- tal dataset and Bridge v2 dataset. In ad- dition, Baseline+FT+CAT indicates train- ing VpE with the proposed contin- uous action training stage using a contin- uous action decoder, while Faseline#FT+PUF denotes fine-tuning Open with the proposed pixel-level understanding enhancement stage on the Pixel-160 dataset.
Pick
Move Near
Open/Close Drawer
Methods
Average
Coke Can
Baseline
54.5
47.7
17.7
40.0
+FT
51.9
42.3
16.8
37.0
+FT+CAT
61.3
52.3
17.7
43.8
+FT+PUE
71.1
54.7
21.3
48.0
PixeIVIA
72.7
57.7
20.0
50.1
As presented in Tab. 4, incorporating the continuous action training stage (Baselin=FF++CAT) improves the average score of 3.8% compared to Baseline, highlighting the benefits of the pro- posed continuous action decoder. Further enhancement with pixel-level understanding (Base- line+FT+PUE) yields more substantial gain of 8.0%. Compared to the single-stage Base- linc+FT+PUE, PixeIVLA adds a continuous action training stage, and this two-stage scheme leads to slight drop Open/Close Drawer, due to the catastrophic forgetting (Li et al. (2024a)) in joint two-stage optimization and the high difficulty and sensitivity this task. Ultimately, PixeIVLA outperforms Baseline=T++CAT by 6.3%. This progressive improvement validates the effectiveness of both pixel-level understanding and multimodal prompts in advancing generalization capabilities.
6 CONCLUSION
This paper proposes PixeIVLA, a vision-language-action (VLA) model, to address the limitations of existing VLAs, such as insufficient pixel-level understanding and over-reliance on textual prompts. PixeIVLA integrates multiscale pixel-awan encoder to inject pixel-level understanding. a contin- uous action decoder for generating accurate robotic actions, and a lightweight visual prompt-aware encoder to support both textual and visual prompts. In addition, two-stage automated annotation pipeline is designed to construct the Pixel-160K dataset containing 160K manipulation episodes. To advance line-grained pixel-level understanding in VLAs, we propose a novel two-stage visuo- motor instruction tuning framework to train PixeIVLA, requiring only 1.5% of the pretraining cost of Open VLA. Expensive evaluations grated into existing VLAs to achieve 10.1% ~ 28.7% improvement in manipulation success rate, effectively enhancing the spatial comprehension and complex environment adaptability of VLAs.
three VLA benchmarks show that PixeIVLA can be inte-
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is supported by the National Key Research and Development Program of China (2023YFB4704800) and National Nature Science Foundation of China under Grant (62255310.62273333)) and the Fundamental Research Funds for the Central Universities (205ZZYXXZ0032, 2024ZYGXZR024).
REFERENCES
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian lchter, et al. mo: A lasion-lnguaag--cction flow model for general robot control. arXiv preprint arXiv:2410.244 14.224664 2024.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Ri-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.068177 2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In arXiv preprint arXiv:2307.15811, 2023.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's 's referential dialogue magic. arXiv preprint arXiv:2306..5122 2023.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, pp. 02783649241273666 2023.
Pengxiang Ding, Han Zhao, Wenjie Zhang, Wenxuan Song. Min Zhang, Siteng Huang, Ningxi Yang, and Donglin Wang. Quar-vla: Aasion-language-ace action model for quadruped robots. ECCV, pp. 352-367, Springer, 2024.
Jiahua Dong, Wenqi Liang, Hongliu Li, Duzhen Zhang, Meng Cao, Henghui Ding, Salman H Khan, and Fahad Shahbaz Khan. How to continually adapt text-to-image diffusion models for flexible customization? NeurIPS, 37:1 001113005570000 2024.
Jiahua Dong, Qi Lyu, Baichen Liu, Xudong Wang, Wenqi Liang, Duzhen Zhang, Jiahang Tu, Hongliu Li. Hanbin Zhao, Henghui Ding, Yulun Zhang, Zhi Han, Nicu Sebe, Fahad Shahbaz Khan, Salman Khan, Mubarak Shah, Philip Torr, Ming-Hsuu -suan Yang, and Dacheng Tao. Learning to model the world: A survey of world models in artificial intelligence. TechRxiv, 2026.
Cunxin Fan, Xiaosong Jia, Yihang Sun, Yixiao Wang, Jianglan Wei, Ziyang Gong, Xiangyu Zhao, Masayoshi Tomizuka, Xue Yang, Junchi Yan, et al. Interleave-vla: Enhancing robot manipulation with interleaved image-text instructions. arXiv preprint ar 22X5:22505.02152, 2025.
Qiushan Guo, Shalini De Mello, Hongxu Yin, Wonmin Byeon, Ka Chun Cheung, Yizhou Yu, Ping Luo, and Sifei Liu. Regiongpt: Towards region understanding vision language model. In CVPR, pp. 13796-13806. 2024.
Zebin Han, Xudong Wang, Baichen Liu, Qi Lyu, Zhenduo Shang, Jiahua Dong. Lianqing Liu, and Zhi Han. Seqwalker: Sequential-horizon iision-and-languun navigation with hierarchical planning. arXiv preprint (tl::2601.046999 2026.
Zhi Hou, Tianyi Zhang, Yuwen Xiong, Haonan Duan, Hengjun Pu, Ronglei Tong, Chengyang Zhao, Xizhou Zhu, Yu Qiao, Jifeng Dai, et al. Dita: Scaling diffusion transformer for generalist vision- language-action policy. arXiv preprint arXiv:2503. 19757, 2025.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1,2):3, 2022.
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019 3026, 2020.
11

---

Published as a conference paper at ICLR 2026
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei- Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: robot manipulation with multimodal prompts. In ICML, pp. 14975-15022, 2023.
Zhaolu Kang, Junhao Gong, Jiaxu Yan, Wanke Xia, Yian Wang, Ziwen Wang, Huaxuan Ding, Zhuo Cheng, Wenhao Cao, Zhiyuan Feng, et al. Hssbench: Benchmarking humanities and social sciences ability for multimodal large language models. arXiv preprint arXiv:2506. 03922, 2025.
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vims: Investigating the design space of visually-conditioned language models. In ICML, 2024.
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty El- lis, et al. Droid: A large-scale in wild robot manipulation dataset. arXiv preprint arXiv:2403. 12945, 2024.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-lagguage-action model. arXiv preprint wrXiv:2406.092466 2024.
Moo Jin Kim, Chelsea Finn, and Perey Liang. Fine-tuning laioonlanguagg-acction models: Opti- mizing speed and success. arXiv preprint 19.v22 2025.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, et al. Segment anything. In ICCV, pp. 4015-4026, 2023.
Hongyu Li, Liang Ding, Meng Fang, and Dacheng Tao. Revisiting catastrophic forgetting in large language model tuning. arXiv preprint arXiv:2406.0444
2024a.
Junxian Li, Di Zhang, Xunzhi Wang, Zeying Hao, Jingdi Lei, Qian Tan, Cai Zhou, Wei Liu, Yaotian Yang, Xinrui Xiong, et al. Chemvlm: Exploring the power of multimodal large language models in chemistry area. In AAAI, pp. 415-423, 2025a.
Xiang Li, Cristina Mata, Jongwoo Park, Kumara Kahatapitiya, Yoo Sung Jang, Jinghuan Shang, Kanchana Ranasinghe, Ryan Burgert, Mu Cai, Yong Jae Lee, and Michael S. Ryoo. Llara: Su- percharging robot learning data for vision-language policy. In ICLR, 2025b.
Xiaoqi Li, Mingxu Zhang, Yiran Geng, Haoran Geng, Yuxing Long, Yan Shen, Renrui Zhang, Jiaming Liu, and Hao Dong. Manipllm: Embodied multimodal large language model for object- centric robotic manipulation. In CVPR, pp. 18061-18070, 2024b.
Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, et al. Evaluating real-world robot manipulation policies in simulation. arXiv preprint arXiv:2405.05941 2024c.
Yanbang Li, Ziyang Gong, Haoyang Li, Xiaoqi Huang, Haolan Kang, Guangping Bai, and Xi- anzheng Ma. Robotic visual instruction. In CVPR, pp. 12155-12165, 2025c.
Yuqi Li, Junhao Dong, Chuanguang Yang, Shiping Wen, Piotr Koniusz, Tingwen Huang, Yingli Tian, and Yew-Soon Ong. Mmt-ard: Multimodal multi-teacher adversarial distillation for robust vision-language models. arXiv preprint arXiv:2511.17445 2025d.
Yuqi Li, Siwei Meng, Chuanguang Yang, Weilun Feng, Jumming Liu, Zhulin An, Yikai Wang, and Yingli Tian. A comprehensive survey of interaction techniques in 3d scene generation. Authorea Preprints, 2026.
Wenqi Liang, Gan Sun, Yao He. Yu Ren, Jiahua Dong, and Yang Cong. Never-ending behavior- cloning agent for robotic manipulation. arXiv preprint arXiv:2403.008: 00336 2024.
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. NewrIPS, 36:44776-44791, 2023a.
12

---

Published as a conference paper at ICLR 2026
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36:34892-3499 2023b.
Huaping Liu, Xinghang Li, Peiyan Li, Minghuan Liu, Dong Wang, Jirong Liu, Bingyi Kang, Xiao Ma, Tao Kong, and Hanbo Zhang. Towards generalist robot policies: What matters in building vision- aaooo-llnguage-acee models. arXiv preprint arXiv:2412. 14055 2025.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang. Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang. Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In ECCV, pp. 38-55. Springer, 2024a.
Yuejiang Liu, Jubayer Ibn Hamid, Annie Xie, Yoonho Lee, Maximilian Du, and Chelsea Finn. Bidirectional decoding: Improving action chunking via closed-loop resampling. arXiv preprint arXiv:2408. 17355, 2024b.
Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, and Xiaojuan Qi. Groma: Localized visual
tokenization for grounding multimodal large language models. In ECCV, pp. 417-435. Springer, 2024.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07119,, 2023.
Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acom Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In ICRA, pp. 6892-6903, IEEE, 2024.
Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang. Jia Yuan Gu, Bin Zhao, Dong Wang, et al. Spatialvla: Exploring spatial representations for visual- language-action model. arXiv preprint 2rXiv:2501.15833 2025.
Kanchana Ranasinghe, Satya Narayan Shukla, Omid Poursaeed, Michael S Ryoo, and Tsung- Yu Lin. Learning to localize objects improves spatial reasoning in visual-Ilms. In CVPR, pp. 12977- 12987, 2024.
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming- -usan Yang. and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In CVPR, pp. 13009-13018, 2024.
Nikhila Ravi, Valentin Gabeur, Yuan-Time Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, al. Sam 2: Segment anything images and videos. arXiv preprint 27Xiv:2408.00714 2024.
Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojic Jin. Pixcllm: Pixel reasoning with large multimodal model. In CVPR, pp. 26374-26383, 2024.
Hao Shi, Bin Xie, Yingfei Liu, Lin Sun, Fengrong Liu, Tiancai Wang, Erjin Zhou, Haoqiang Fan, Xiangyu Zhang, and Gao Huang. Memoryvla: Perceptual-cogminee memory in vision-language- action models for robotic manipulation. arXiv preprint arXiv:2508.192 2025.
Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montser- rat Gonzalez Arenas, Travis Armstrong. Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing ai into the physical world. arXiv preprint arXiv:2503.20020, 2025.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405. /22 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arX1v:2307.09288, 2023.
13

---

Published as a conference paper at ICLR 2026
Homer Rich Walke, Kevin Black, Tony Z Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. Bridgedata v2: dataset for robot learning at scale. In CoRL, pp. 1723-1736 PMLR, 2023.
Lirui Wang, Xinlei Chen, Jialiang Zhao, and Kaiming He. Scaling proprioceptive-visall learning with heterogeneous pre-trained transformers. NeurIPS, 37:124420-124450 2024.
Xudong Wang, Jiahua Dong, Baichen Liu, Qi Lyu, Lianqing Liu, and Zhi Han. Lifelong embodied navigation learning. In ICLR, 2026a.
Xudong Wang, Gan Li, Zhiyu Liu, Yao Wang, Lianqing Liu, and Zhi Han. All-day multi-scenes lifelong vision-and-language navigation with tucker adaptation. In ICLR, 2026b.
Jiannan Wu, Muyan Zhong, Sen Xing. Zeqiang Lai, Zhaoyang Liu, Zhe Chen, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, et al. Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks. NeurIPS,
999999956699, 2024a.
Junda Wu, Zhehao Zhang, Yu Xia, Xintong Li, Zhaoyang Xia, Aaron Chang, Tong Yu, Sungchul Kim, Ryan A Rossi, Ruiyi Zhang, et al. Visual prompting in multimodal large language models: A survey. arXiv preprint arXiv:24099 15310, 2024b.
Kun Wu, Chengkai Hou, Jiaming Liu, Zhengping Che, Xiaozhu Ju, Zhuqin Yang, Meng Li, Yinuo Zhao, Zhiyuan Xu, Guang Yang, et al. Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation. arXiv preprint arXiv:2412.13877, 2024c.
Shuai Yang. Hao Li, Yilun Chen, Bin Wang, Yang Tian, Tai Wang, Hanqing Wang, Feng Zhao, Yiyi Liao, and Jiangmiao Pang. Instructvla: Vision-languaae-action instruction tuning from under- standing to manipulation. arXiv preprint urXiv:2507.17520 2025.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang. Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere any granularity. arXiv preprint arXiv:2310.077000 2023.
Michal Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied chain-of-thought reasoning. arXiv preprint arXiv:2407.08692 08693, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training In ICCV, pp. 11975-11986, 2023.
Ao Zhang. Hao Fei, Yuan Yao, Wei Ji, Li Li, Zhiyuan Liu, and Tat-Seng Chua. Vpgtrans: Transfer visual prompt generator across 1lms. NeurIPS, 36:20299-20319, 2023.
Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong- You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, et al. Ferret-v2: An improved baseline for referring and grounding with large language models. arXiv preprint arXiv:2404.00973 2024a.
Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. NeurIPS, 37:71737-71767, 2024b.
Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma. Song Han, Chelsea Finn, al. Cot-vla: Visual oftth-hf-thought reasoning for vision- language-action models. In CVPR, pp. 1702-1713, 2025.
Tony 7. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 12233044.37..... 13705, 2023.
Ruijie Zheng, Yongyuan Liang, Shuaiyi Huang, Jianfeng Gao, Hal Daumé III, Andrey Kolobov, Furong Huang, and Jianwei Yang. Tracevla: Visual trace prompting enhances temporllt awareness for generalist robotic policies. arXiv preprint arXiv:2412.10344 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
This supplemental material introduces additional details not mentioned in the main paper. Overall, it mainly includes the following aspects:
(1) Details of Pixel-160K dataset in Sec. A.1.
(2) More experiment setup in Sec. A.2.
(3) More implementation details in Sec. A.3.
(4) More qualitative comaprisons in Sec. A.4.
(5) Limitations in Sec. A.5.
Task Instruction
Task Instruction
Task Instruction
Whit shouls
What should robot do
What should robet (pick from middle and place counter), refer [annotations] /visual prompts: Todual
jpick and place the blue cloth:. rofer to (onnotations) (visual prompts)
the washing machine), refer (annotations) (visual prompts)
Textual
Visual
Texiual
Viuull
Visual
Robolic States
Robotic States
Robotic States
[ax, a0, &Grip]
[Ax, s0, AGripl
[ax, a0, &Grip]
Observations
Observations
Observations
episode
f-t episode
episode
Figure 5: The episode example in our Pixel-160K dataset.
A.1 PIXEL-160K DATASET
This section provides a detailed description of the Pixel-160K dataset and the proposed automated annotation pipeline. In contrast existing robot datasets composed of image-level and text in- structions, Pixel-160K offers fine-grained pixel-level annotations and supports both textual and vi- sual prompts, aiming to train VLAs for more precise pixel-level spatial understanding and diversi- fied human-robot interaction. Pixel-160K comprises 160K robot manipulation episodes and 6.5M image- text-acioon triplets enriched with visual prompts and mask annotations. Specifically, for the i-th episode E, in Pixel-160K dataset, E, (X,, A,, Pi, L, Vi). where X; = {x ) P. { A, [a As shown in Fig. 5, to effectively inject the pixel-level an- notations and visual masks, we reformulate the textual instruction-Whar should the robot do to </instruction>> } as What should the robot do to {</instruction>); refer to {</annotations>> </visual prompts>> Here, < > a placeholder and subsequently replaced with the language embeddings, the pixel-aware embeddings and the prompt-aware embeddings.
As illustrated in Fig. 6(a), the proposed automated annotation pipeline consists of two sequential stages: (a) Gripper-aware Region Proposal Stage and (b) Multimodal Object Segmentation Stage. The gripper-aware region proposal stage consists of four steps:
1. Gripper-close State Extraction: For each manipulation episode, parse the robotic state sequence Ax, A0, AGrip. Frames in which the gripper state AGrip are identified as gripper- close states, indicating that the gripper is interacting with object. The first frame in each episode selected as the key observation x,)
GG, representing the earliest moment of potential object contact.
2. Discrete Video Composition: We collect the gripper-close observations from all episodes, i.e., GNL } where N₂ is the total number of episodes. These frames then organized (xf1 x2 3.... ...,,, XN.
15

---

Published as a conference paper at ICLR 2026
Grippe-dose State Extraction
3. Gripper Detection
Target Object Reasoning
Task Instruction
LLM
AGrip 1
Qi which
ono
whice
blue
oqpoot mhe
robot jatt vessee
diject tho robor
taue prove the fax Sio podo stoveg
Robotic States
AGrip=1
and paso <n 3io
Fapper Reglose
Ai veaset
lie red foch
[ax, a0, &Grip]
Compostton
4. Region Propceals Generation
Annotations Generation
Observatione
Grounding DINO SAM
Chop
Disceda Videza
Iria Observatons
Region Proposals
(R) Grippee-worr Region Proposal Stage
10) Muitimoddi Objjct Segmentation Stage
Figure 6: The proposed automated annotation pipeline for generating visual prompts and mask annotations at scale for a given robot dataset, consisting of a gripper-aware region proposal stage and multimodal object segmentation stage.
into a discrete video set that captures diverse gripper-object interaction states across the dataset, providing compact yet informative cues for region localization.
Gripper Detection: To precisely localize the gripper within the selected frames, we apply SAM 2 (Ravi et al. (2024)). The model segments the gripper region from each observation, producing bounding boxes tightly enclosing the gripper. These gripper regions serve as reliable anchors, as the manipulated objects typically appear adjacent
in contact with the gripper.
4. Region Proposals Generation: Finally, we crop the localized gripper regions from the initial observations to form region proposals R1, R2,..., RN,, with each R,, e R denoting bound- ing box for the 1-th episode. These proposals effectively filter out irrelevant background clutter and highlight the object-relevant areas, thereby facilitating accurate segmentation in the subsequent multimodal object segmentation stage.
As illustrated in Fig. 6(b), the multimodal object segmentation stage consists of two steps:
5. Target Object Reasoning: Given a manipulation instruction, such as "Pick the vessel and place the blue cloth" or "Move the red fork to the left side of the stove", employ a large language model (LLM) to parse the instruction and extract the target object to be manipulated. The LLM identifies the first object the robot needs grasp and outputs its textual description (e.g. "vessel" or "the red fork"). This reasoning step transforms natural- natural-language task instructions into precise object queries, enabling alignment between language and perception.
6. Annotations Generation: For the 17-th episode, we feed the target object text together with the region proposal R., into an opea-vocay detection and segmentation pipeline comprising Grounding DINO (Liu et al. (2024a)) and SAM (Kirillov et al. (2023)). Grounding DINO detects candidate object bounding boxes conditioned on the target object text, grounding the language query into specific image regions. Subsequently, SAM refines these detections by generating pixel-level segmentation masks for the grounded boxes. The outputs include bounding boxes, segmentation masks, and text-object alignments. We then filter the results by selecting the mask with the highest confidence score within the region proposal, discarding low-confidence or irrelevant detections. Fi- nally, from the retained object mask, we derive visual prompts by: randomly sampling points inside the mask, generating random lines within the object area, and extracting external bounding boxes via contour detection.
A.2 EXPERIMENT SETUP
Evaluation tasks. As shown in Fig. 7, we conduct all experiments on three simulation benchmarks, (Li et al. (2024c)) and LIBERO (Liu et al. (2023a)). Specifically, SimplerEnv et al. (2024c) is an open-source simulation suite designed for reproducible and scalable evaluation of robot manipulation policies. explicitly
i.e., SerpllerEn-GGoolle Robot (Li et al. (2024c)), SimplerEnv-Widow
16

---

Published as a conference paper at ICLR 2026
SimplerEnv-Googe Robet
SimplerEnv Enddww
LIBERO
Clon
SpooS
LIBERO-C6jad
Google
WidowX Robol Setup
Franka Robol Setup
Robot
Setup
Pick Coka Can
Move Neat
Stack Blocks
Put Easplant
LIBERO-Goal
LIBERO-Long
Figure 7: Experimental Setup Overview. We evaluate PixelVLA on three simulation bench- marks: SimpleEnv- WidowX with WidowX robot, SimpleFav-Google Robot with Google robot, and LIBERO with Franka.
SimpleEn--iidoW
addresses the visual and dynamic gaps between simulation and real hardware, enabling more faithful assessments of policy generalization. SimplerEny provides two complementary evaluation settings: Visual Matching (VM), which minimizes visual appearance discrepancies improve the correlation between simulation and real-world performance; and Variant Aggregation (VA), which, inspired by domain randomization, introduces diverse visual perturbations and aggregates results across mul- tiple randomized environments to obtain more robust performance estimates. Building on these capabilities, we perform zero-shot object manipulation experiments on SimplerEny to benchmark the effectiveness of our approach.
As shown in Fig. 7, LIBERO (Liu et al. (2023a)) is a comprehensive benchmark suite designed to evaluate continual adaptation and generalization in robot manipulation. It consists of multiple task suites that focus on different aspects of knowledge transfer: LIBERO-Spatial, which requires learning new spatial relationships between identical objects; LIBERO-Object, which involves rec- ognizing and manipulating novel object types; and LIBERO-Goal, which evaluates ability to adapt new task goals given the same objects and spatial layouts. Additionally, LIBERO- Lonnnog-ongg tends the challenge to long- oong-horizon tasks, testing a robot's 's capacity for compositional reasoning and extended action planning. Together, these task suites provide systematic framework for studying declarative knowledge (spatial and object-level) and knowledge (goal-oriented behaviors) in robot learning.
A.3 IMPLEMENTATION DETAILS
Training. In the first stage of continuous action training, we initialize PixeIVLA's vision encoder, MLP projector, and LLM backbone with pretrained weights from VLAs (Kim et al. (2024); Black al. (2024)), which were trained on the large-scale OXE dataset (eiNeill et al. (2024)). To focus on learning continuous action mappings, the visual prompt-aware encoder and the multiscale pixel- aware encoder are removed, while all other modules except the continuous action decoder are frozen preserve general manipulation knowledge. The continuous action decoder maps the final hidden states of the L.LM to continuous action values. We use I.1 regression (Zhao et al. (2023); Kim et al. (2025)) to align the predicted actions with ground-truth actions. Each continuous action dimension is uniformly discretized into 256 bins and normalized to [-1, +1] for fine-grained prediction. The input observation consists of single third-person camera view resized to 224 224 pixels. Actions are predicted in chunks of 8 timesteps, with a" E R During this stage, PixelVLA is trained on mixture of Fractal (Brohan et al. (2022)) and Bridge (Walke et al. (2023)) datasets. Training runs for 100k steps with batch size of 32 and learning rate of 5 x 10-4 This stage is omitted when adapting PixeIVIAA to To due the effectiveness of its pretrained action expert.
In Pixel-level Understanding Enhancement Stage, PixelVLA is fine-tuned for pixel-level under- standing using the Pixel-160K dataset. The visual prompt-aware encoder and multiscale pixel-aware encoder are jointly trained, while LoRA adaptation is applied to the LLM backbone (rank r = 32). All other modules remain frozen except the continuous action decoder, which is optimized using L1 regression, following the same continuous action representation in the first stage. Input ob- servations are a single third-person camera view resized to 224 x 224 pixels. Actions are predicted in chunks of 8 timesteps, at E TR877 Training runs for 200k steps with batch size of 32 and
17

---

Published as a conference paper at ICLR 2026
Table 5: SimplerEnv (Li et al. (2024c)) simulation evaluation results in terms of the average success rate for the Google Robot setup. VM denotes Visual Matching and VA is Variant Aggregation. denotes tuning-basee methods applied to the pretrained weights of DLenVLA.
Pick Coke Can
Move Near
Open/Close Drwwer
Methods
Average
Horizontal
Vertical Standing Average
Average
Open Close
Average
RT-1-X (O'Neill al. (2024))
82.0
33.0
31.7
29.6
59.7
53.4
Octo- Base (Team et al. (2024))
21.0
21.0
9.0
17.0
4.2
0.9
44.4
22.7
16.8
VM
HPT (Wang et all (2024))
56.0
60.0
24.0
46.0
RoboVLMs (Liu et al. (2025))
85.0
43.0
90.0
72.7
66.3
28.7
25.0
26.8
56.3
SpatialVLA (Qu et al. (2025))
70.0
82.0
91.0
81.0
69.6
49.1
69.4
59.3
71.9
OpenVLA (Kim al. (2024))
27.0
3.0
19.0
16.3
46.2
19.4
51.8
35.6
27.7
TLA (Zheng al. (2024))
28.0
53.7
57.0
42.0
PixeIVLA
63.6
25.3
42.3
65.0
RT-1-X (O'Neill al. (2024))
56.9
20.4
69.8
49.0
32.3
$1.9
29.4
39.6
Octo-Base (Team et al. (2024))
0.5
0.0
1.3
0.6
3.1
0.0
2.1
1.1
1.1
HPT (Wang et al. (2024))
60.0
RoohoVLMs (I et al. (2025))
77.8
48.0
79.1
68.3
56.0
1.6
15.3
8.5
46.3
SpatialVLA (Qu et al. (2025))
93.3
83.1
92.0
89.5
71.7
23.3
49.2
36.2
68.8
VLe (Kim al. (2024))
71.1
27.1
65.3
54.5
47.7
15.8
19.5
17.7
39.8
Thace VL. (Zheng al. (2024))
PixelVLA
81.8
57.7
16.4
20.0
learning rate of 1 10-3, enhancing PixeIVLA's ability to localize and manipulate target objects at the pixel level.
Inference. During inference, the user only needs provide semantic instruction along with vi- sual prompts in the initial observation. Subsequently, we incorporate FastSAM to predict the mask annotation of the target object based the given visual prompts. This introduces an additional computational overhead of approximately 1% the overall model inference cost, resulting a negligible impact inference efficiency.
A.4 QUALITATIVE COMPARISONS
As shown in Tab. 5, this section presents the evaluation results of the simpler environment on the Google Robotics benchmark, which includes tasks such as Coke can manipulation (horizontal, ver- tical and standing picking) and drawer operations (open and close). PixelVLA demonstrates excep- tional performance, achieving an average VM score of 65.0, significantly outperforming OpenVLA (27.7) and VLA (42.0). In the VA category, PixelVLA achieves average again surpassing VLE (39.8) and TLa (45.0). These results indicate that PixeIVLA out- performs its counterparts in both pixel-level understanding and the adaptation to visual prompts, showcasing a remarkable capacity for generalist manipulation across the evaluated tasks. The sig- nificant margins in performance metrics underscore the effectiveness of the proposed method in addressing out-of-domain generalization challenges.
of 52.6,
A.5 LIMITATIONS
Although PixeIVLA substantially improves VIA performance through pixel-level understanding and tailored visual prompts, it remains limited in handling richer input modalities (c.g., 3D percep- tion) and more advanced forms of visual prompting, such as precise trajectory guidance, reference- image prompts, pose-conditioned prompts, compositional prompt sequences. In addition, while PixeIVLA is extensively validated across three simulated benchmarks, we expect that real-world robot experiments would further strengthen its contributions. We regard these extensions impor- tant directions for future work.
18


Reviews:
[{"review_id": "H5EqpLleQB", "reviewer": "Reviewer_FaYs", "summary": "The paper proposes PixelVLA, which besides normal language and , visual encoder, PixelVLA add a modal that enable pixelwise grounding, then claim this enable the model learn more fine-grained represenations that further benefit the understanding and action prediction.", "questions": "Besides the main conern in weakness part. I still have questions:\n1. I might need the user better claim the motivation of add the segmentation mask or visual prompting as an input, i feel these will harm the model's potential ability to learn to locate the objects and understanding the scene itself.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well organized and easy to follow.\n2. According to the results shown in Table 2, Table 3 and Table 4, it seems, add more pixel level prompting and intergrate this as a new modal can help increase the action control accuracy.", "weaknesses": "1. I am confused about the design of the whole architecture. It seems, the author proposes a two-state automated pipeline to get the pixel level segmentation, how these segmentations are used? it is not very clear that which part these segmentation masks are used for in the model.\n\n2. If the segmentation is used as a input to learn the pixel-aware embedding, i am not sure the final optimization of loss for these visual encoder is action accuracy? seems not very relevent. What is the motivation of input these mask to learn the pixel aware embedding.\n\n3. The author gives some results and try to say pixel vla is better than OpenVLA or Pi0, but that is not fair since Pixel VLA needs extra prompting from user, these extra prompt actually gives more direct spatial information of the user's goal. The intuition is the using of these informaiton will give more shortcut to the backbone learning and make it less capable to do reasoning and understand the language.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d8iwrIeaZ6", "reviewer": "Reviewer_sYs6", "summary": "This paper proposes PixelVLA, a VLA that benefits from pixel-level grounding information. Based on a typical VLA model with a vision encoder, text tokenizer, and LLM, the authors introduced a pixel encoder that is designed to handle pixel-level understanding information. Meanwhile, to finetune such an encoder, the authors proposed a two-stage automated annotation pipeline and created a pixel-annotated visuomotor instruction tuning dataset, Pixel-160k. The experiments on SimplerEnv and LIBERO show that the proposed method achieves better performance than other baselines.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written and well-structured, making it easy to follow. The experiments conducted on SimplerEnv and LIBERO are appropriate and demonstrate the effectiveness of the proposed approach. While introducing an additional pixel-level encoder could intuitively downgrade the pretrained VLM, the authors successfully solve this issue by curating a large 160K dataset and applying LoRA fine-tuning.", "weaknesses": "The introduction of pixel-level annotations can be viewed as a relatively straightforward extension of prior work on visual prompting and image-level feature adaptation (e.g., TraceVLA, LLaRA, and related approaches). As a result, the paper’s novelty is somewhat limited. Nonetheless, the work offers useful insights and has potential value for the research community, particularly as a good practice in bridging pixel-level understanding with pretrained VLMs for VLAs. \n\nMeanwhile, the authors are highly encouraged to deploy the proposed method in the real world and verify the claim.\n\nTherefore, I would recommend a weak acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TvAVsCBUBq", "reviewer": "Reviewer_DBFX", "summary": "The authors present PixelVLA, a VLA model that introduces diverse multimodal prompts to the VLA pipeline and whose architecture is designed and trained specifically for these prompts. The VLA itself consists of a vision encoder to process image observations, visual prompting encoder (from SAM) and multi-scale visual encoder to process multimodal prompts (points, boxes, lines etc), and LLM backbone and a continuous action decoder. To generate visuomotor instruction tuning data for training PixelVLA, the authors use an LLM to extract information about target objects, and use an object detector and segmentation model to extract multimodal prompts, all over the Fractal and Bridge datasets, and contribute the resulting dataset. The experiments are done over SimplerEnv and Libero, across multiple VLA backbones, to show the benefits of PixeVLA.", "questions": "l. 218 Where does the pixel-aware mask input come from?\nl. 237-239 How does the NTP loss work in this case? Is there causal masking?\nl. 252 You're only using the first gripper-close state – might that negatively affect training due to some domain shift between first gripper-close state and subsequent such states?\nSection 5.2: Were the $pi_{0}$ and OpenVLA baselines also trained in their typical fashion on the Fractal and Bridge datasets? This would ensure that the improvement is due to the PixelVLA method and instruction tuning approach rather than just seeing more data.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Meaningful technical contribution in Pixel-160k dataset and in constructing a VLA that takes advantage of multimodal prompts.\n- Experiments test two SOTA VLA architectures, showing that PixelVLA can be built on top of multiple types of VLAs.", "weaknesses": "- Unclear whether method is feasible to transfer to novel environments, due to lack of real world experiments.\n- Analysis of results are lacking and leave out failure cases; for example, why does PixelVLA perform so well on Libero Long but struggle on the Object/Goal splits (Table 3)? Why would the pixel-level understanding training damage performance on the open/close drawer task (Table 4)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l3x86InzPG", "reviewer": "Reviewer_FnpE", "summary": "The authors introduce a pixel encoder to condition VLAs on visual prompts, construct a suitable automated data annotation pipeline, and instruction tune their modified VLA to gain pixel level awareness. Their proposed visual prompt encoder conditions on multi-scale features of the robot visual observation and its resulting tokens are consumed by an LLM along with regular visual and textual tokens. The LLM outputs are mapped to robot actions with a suitable action decoder. \nThey integrate their proposed PixelVLA into two strong VLAs, OpenVLA and Pi0. Extensive experiments highlight performance improvements from their framework.", "questions": "See weaknesses. The paper is interesting and contains strong experimental evidence. However, several missing key details weakens it in current form.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1) The authors conduct extensive experiments to highlight improvements over SOTA baselines\n2) Simple idea of incorporating pixel prompts \n3) Clever multi-stage training setup to benefit from existing VLA's pretrained weights\n4) Thorough ablations", "weaknesses": "**1) Architecture is unclear**\n  - In Figure 2, part (a) \"Multiscale Pixel-aware Encoder\" also shows another Visual Prompting Decoder which is unclear. This figure needs to be explained better and updated suitably. \n  - \"Visual Prompting Decoder\" is not explained clearly. What is its exact input structure? How is does it \"preserve the spatial positional information\" of visual prompts? These need to be explained. \n  - How exactly are the visual prompts represented? Fig 2 shows them both overlaid on image and input separately. The separate input needs to be described more clearly. \n\n**2) Unclear Data Generation**\n  - \"detect the gripper to generate N_e region proposals\" - how are region proposals generated from gripper location?\n  - L262 \"we employ a LLM\" - which one? How is it prompted? \n  - \"we filter out approximately 19.2% failed samples\" - how are they filtered? Manually by humans? \n\n**3) Unclear Training Details**\n  - L307 \"uniformly discretize each continuous action representation into 256 bins\" - why? PixelVLA uses a continuous action decoder? The reasoning behind this is highly unclear. \n \n**4) Related Work**\n  - Consider discussing related VLM work Ferret (https://arxiv.org/pdf/2310.07704) and FerretV2 that performs similar visual prompt encoding to improve VLA performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)