OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction
Authors: Huang Huang
Fangchen Liu
Letian Fu
Tingfan Wu
Mustafa Mukadam
Jitendra Malik
Ken Goldberg
Pieter Abbeel
Venue: ICML 2025 poster (2025)
Keywords: vision language action model; robot foundation model
Primary area: applications->robotics
TL;DR: A vision-language model using a frozen VLM for strong zero-shot generalization ability

Abstract: Vision-Language-Action (VLA) models aim to predict robotic actions based on visual observations and language instructions. Existing approaches require fine-tuning pre-trained vision-language models (VLMs) as visual and language features are independently fed into downstream policies, degrading the pre-trained semantic alignments. We propose OTTER, a novel VLA architecture that leverages these existing alignments through explicit, text-aware visual feature extraction. Instead of processing all visual features, OTTER selectively extracts and passes only task-relevant visual features that are semantically aligned with the language instruction to the policy transformer. This allows OTTER to keep the pre-trained vision-language encoders frozen. Thereby, OTTER preserves and utilizes the rich semantic understanding learned from large-scale pre-training, enabling strong zero-shot generalization capabilities. In simulation and real-world experiments, OTTER significantly outperforms existing VLA models, demonstrating strong zero-shot generalization to novel objects and environments. Video, code, checkpoints, and dataset: https://ottervla.github.io/.

OTTER: A Vision-Langunge-Action Model with Text-Aware Visual Feature Extraction
Huang Huang
Fangchen Liu 1
Letian Fu 1 Tingfan Wu Mustafa Mukadam
Jitendra Malik
Ken Goldberg'
Pieter Abbeel
Abstract
AssionLLaguuage-Action (VLA) models aim to predict robotic actions based on visual obser- vations and language instructions. Existing ap- proaches require fine-tuning pre-trained vision- language models (VLMs) as visual and language features are independently fed into downstream policies, degrading the pre-trained semantic align- ments. We propose OTTER, novel VLA archi- tecture that leverages these existing alignments through explicit, text-aware visual feature ex- traction. Instead of processing all visual fea- tures, OTTER selectively extracts and passes only task-relevant visual features that are semantically aligned with the language instruction to the pol- icy transformer. This allows OTTER to keep the pre-trained vision-language encoders frozen. Thereby, OTTER preserves and utilizes the rich semantic understanding learned from large-scale pre-training, enabling strong zero-shot general- ization capabilities. In simulation and real-world experiments, OTTER significantly outperforms existing VLA models, demonstrating strong zero- shot generalization to novel objects and environ- ments. Video, code, checkpoints, and dataset: attps://otters: http://ttterlll.itu...bb.. git.ub.tiiiii.. io/.
"Pick the triongle"
"Plck the grees trimgle"
Taxt Encoder
Vision Encoder
Joint Vision-Tas Encoder
Jie
Policy Network
Policy Network
(6) Direct Feature Passing
(6) Text Aware Feature Extraction
Octa
OpeniV AA
Cmrer
Success
Training Tasks
Unseen Tasks
Figure 1: (Top) Different feature extraction approaches VLAA models. (a) Direct Feature Passing: existing approaches, exem- plified by Octo and OpenVLA, extract and pass visual and text tokens independently the policy network. (b) Text-Aware Fea- ture Extraction: the proposed approach, OTTER, extracts visual tokens that correspond to the text tokens, and then feeds them into the policy. (Bottom) Real-world Robot Experiments: OTTER demonstrates higher success rates on both training and unseen real- world robot pick-and-place tasks compared Octo and Open VLA. OTTER exhibits better zero-shot generalization unseen objects, maintaining strong performance across a variety of novel tasks. Kim et al., 2024). Different input modalities are usually en- coded into separate tokens: multi-view images encoded via visual feature extractors, along with tokenized language in- structions, optionally with the robot's proprioceptive states, are fed into transformer-based robot policy for end-to-end action generalization. This approach requires the policy network to connect the vision and language information and conduct precise robot control, which often presents significant challenges, especially in unseen environments.
1. Introduction
Recent advancements in Large Language Models (LLMs) and Vision-Language Models (VLMs) have inspired the ploration of scaling datasets and computational resources for vision-language-actti action (VLA) models (Collaboration et al., 2024; Khazatsky et al., 2024; Octo Model Team et al., 2024;
Existing works such as RT-2 (Brohan et al., 2023) and Open- VLA (Kim et al., 2024) have demonstrated the benefits of directly fine-tuning pre-trained VLMs on robotic datasets map vision and language to control. While these approaches leverage rich visual and language features from pre-trained encoders, this fine-tuning may interfere with pre-trained vi- sion and language features, particularly since robot datasets are far less semantically diverse compared large vision- language datasets (Schuhmann et al., 2022) where these
Equal contribution University of California, Berkeley Meta AL Correspondence to: Huang Huang <huangr@ berkeley.edu>> Fangehen Liu <fangehen liu @ew@@berkelle..eeee bed..ley.edu>,, Letian <max.fu.letinn @ berkeley.edu'>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

OTTER: A Vision-Languagcc-cttinn Model with Text-Aware Visual Feature Extraction
(6) Multi-Modal State Representation
corporates text-aware feature extraction into the vision- language-action (VLA) pipeline. At each timestep, OTTER first processes visual and textual inputs to pinpoint task- relevant visual tokens. These selected features, along with proprioceptive data, enable the policy network to concen- trate specifically on action planning. By keeping the pre- trained vision-language encoders frozen, OTTER effectively decouples task planning (selecting relevant visual features) from robot action planning (predicting appropriate actions). Both physical and simulation experiments demonstrate that OTTER outperforms existing VLA models, showing strong generalization to novel objects and environments with less performance degradation (Figure 1).
"Pick up green trangll"
CLIP Text Encoder
CUP
Embocimmnt Encoder
Text Altn Pooling
Text-Auwar Vision Extraction
(b) Transformer Based Policy
Causal Transformer
To summarize, our contributions are:
1.
We propose OTTER, a VLA model that leverages the semantic alignment capabilities of pre-trained VLMs for better generalization. By extracting text-aware vi- sual features that are semantically aligned with task instructions, OTTER preserves and utilizes the rich visual-language understanding from pre-training for effective robotic task execution.
Figure 2: OTTER Model architecture. At each timestep L, text- aware visual features fur are extracted from pre-trained CLIP model (see Figure 3). Then fot and the text tokens fi are further compressed through separate attention pooling layers, producing two representations f. and fí. respectively. The robot's proprio- ception is encoded by an MLP layer generate the embodiment representation fe- The tokens f(, fet, and fe are then concatenated to form fi, which serves as input to causal transformer. With context window of I" steps, the model autoregressively predicts the future actions (a:) at each step.
2. OTTER significantly outperforms state-of-the-art VLA models on unseen robot manipulation tasks through its zero-shot generalization capabilities. By preserving the frozen pre-trained vision-language model rather than fine-tuning, OTTER effectively leverages the semantic understanding from large-scale pre-training to handle novel objects and environments.
VLMs are trained on, often leading to noticeable drop in performance on unseen objects or environments compared to seen tasks (Brohan et al., 2023).
This finding is consistent with observations vision- language research (Kerr al., 2023; Lan et al., 2024), where fine-tuning language-aligned vision encoders, such as CLIP (Radford et al., 2021), has been shown to overfit and degrade generalization and long-tail classification per- formance. Yet, models like CLIP and SigLIP (Zhai et al., 2023) already exhibit strong vision-language alignment and impressive zero-shot performance on various downstream tasks, including fine-grained tasks like open- open-vocabulary seg- mentation (Rao et al., 2022; Lan al., 2024: Dong al., 2023). This raises the question of whether more effective strategy is to extract and utilize the pre-trained alignment rather than risk weakening it through fine-tuning.
3. Empirical results suggest that OTTER's performance on unseen tasks scales along multiple axes: through larger pre-trained vision-language encoders, increased policy network capacity, and pre-training on larger robot datasets.
2. Related Work
2.1. Vision Language Pre-training
Vision-language pre-training (VLP) seeks improve the performance of downstream tasks that involve both vision and language by training models on extensive datasets of image-text pairs. A prominent class of vision- languall models leverages contrastive learning (Alayrac et al., 2020; Cherti et al., 2023; Jia al., 2021; Radford et al., 2021; Yao et al., 2021; Yuan al., 2021: Zhai et al., 2023). Among them, CLIP (Radford al., 2021), which was trained on private WIT-400M dataset of image-text pairs, demonstrates impressive zero-shot capabilities across various downstream tasks, including image-text retrieval and image classification through text prompts. Furthermore, CLIP shows potential for application in broader fields such as decision making and robotics, where robots are required to perform language-
We seek to preserve the generalizability of VLMs for ef- fective performance in unseen scenarios. To this end, we propose OTTER, novel VLA architecture that freezes pre-trained vision and language encoders and extracts task- relevant visual features guided by language instructions. Instead of passing all visual features to the policy network, OTTER selectively extracts those semantically aligned with the task description. We use CLIP for its zero-shot capabili- ties and wide adoption, employing a lightweight selection mechanism preserve its pre-trained vision-lant Langua-aagguage un- derstanding while adapting it for robotic control.
Figure 2 illustrates the architecture of OTTER, which in-
2

---

OTTER: A Vision-Languuagc-kction Model with Text-Aware Visual Feature Extraction
specified tasks based on visual inputs.
ResNet (He al., 2016). RT-1 (Brohan al., 2022), one of the first language-conditione robot policies, uses FiL.M to encode visual and text information for action generation. However, RT-1 learns the language-vision alignment from robotic data without leveraging pre-trained models such as CLIP (Radford et al., 2021), where visual features are already aligned with text.
Recent multimodal foundation models, such Qwen-VL (Bai et al., 2023) and Llama 3-V (Dubey et al., 2024), all follow a similar pattern: they extract visual features by finetuning language-aligned vision models and train cross-attention layers to inject visual features into the language model. However, many researchers have observed that when data scarce, fine-tuning or even applying additional layers on top of CLIP (instead of using raw CLIP features) (Kerr et al., 2023; Lan et al., 2024) may result in models with weaker reasoning capabilities compared to vanilla CLIP. This motivates our approach in OTTER, where we preserve CLIP's strong vision- rision-language alignment capabilities by keeping the model frozen and extracts visual patch feature that correspond the text query in a parameter-free way.
OTTER distinguishes itself by retrieving CLIP's visual patch features that best correspond to the language task description using cosine similarity before the policy trans- former. These text-aware visual features, language features and the robot's proprioceptive state are fed into the robot policy. This approach allows the model to leverage the fine-grained features from the pre-trained vision-language models while incorporating robot-specific information.
2.2. Vision Language Action Models
3. Method
In recent years, there has been a surge of interest in develop- ing robot foundation models, largely inspired by the success of large language models (LLMs)) and vision-language mod- els (VLMs) (Devlin et al., 2018; Radford et al., 2018; 2019; Brown et al., 2020; Chowdhery et al., 2023; Achiam et al., 2023; Radford et al., 2021; Li et al., 2023). A key hypothe- sis driving this trend is that more capable robot foundation models can emerge by scaling up robot datasets, increas- ing model capacity, and co-training or pre-training models on vision and language datasets. This has led researchers in the robot learning community to train robot foundation models, investigate pre-training strategies, and iterate on model designs (Brohan et al., 2022; 2023; Kim et al., 2024; Octo Model Team et al., 2024; Jang et al., 2022; Jiang et al., 2023; Reed et al., 2022: Collaboration et al., 2024; Shah et al., 2023; Fu et al., 2024).
We propose OTTER, a vision-language-action model for learning a robot manipulation policy through extraction of text-aware vision features from a pre-trained VLM. We first describe how OTTER utilizes the vision- vision-language align- ment of pre-trained vision and language encoders to extract text-aware vision features, then provide more detailed explanation of the model architecture.
3.1. Text-Aware Visual Feature Extraction
OTTER utilizes pre-trained CLIP for vision and language features extraction. Consider ViT-based CLIP vision en- coder (Radford et al., 2021) consisting of a series of residual attention blocks. Each of these blocks takes as input col- lcction of visual tokens X = [aeass ø1,.... .., [xxx:l I where Zels represents the learnable global class token, and outputs the feature X as shown below:
Many existing VLMS (Liu et al., 2023; Laurençon et al., 2024; Karamcheti et al., 2024) use similar approach, where visual features and languages are directly passed into the LLM to generate answers. Similarly, the majority of Vision- Language-Action (VLA) models also opt for this approach, where language, vision, and robot proprioception data are separately encoded by modality-sseeciff feature extractors before being fed into single transformer policy. This method has shown promise in many language-conditionee multi-task learning models (Jiang et al., 2023; Brohan et al., 2023; Jang et al., 2022; Reed et al., 2022; Collaboration et al., 2024; Shah et al., 2023), including current open- source state-of-the-art models such as Octo (Octo Model Team al., 2024) and Open VLA (Kim et al., 2024).
q, kr Proj (LN(X))
(1)
Xsum = X + X Xati = - + Proj(Aun(q, v))
(2)
X X + FFN(LN(X..m))
Proj. LN, and FFN denote linear projection matrix, layer norm (Ba, 2016), and feed-forward network respectively. A recent work ClearCLIP (Lan et al., 2024) demonstrates that CLIP's last self-attention block's attention feature Xata con- tains cleaner semantic information than the CLIP's output feature X While ClearCLIP uses the cosine similarity between text and visual features for segmentation, we lever- age this similarity to construct task-relevant visual features for robotic control. Specifically, we the similarity scores to select and combine visual features that best align with the task instruction, creating compact representations for downstream action prediction.
In contrast to the above approach, OTTER combines vision and language input before feeding them into the robot poli- cies by extracting and passing text-aware visual features. Early works such as FiLM (Perez et al., 2018) encode text information and fuse these features into each block of a
In OTTER, we extract text por-token features from CLIP's language encoder ft (m tokens). For the visual features,
3

---

OTTER: A Vision-Languuagc-ktiion Model with Text-Aware Visual Feature Extraction
features fg. Intuitively, the softmax serves as a selection function, where patch features relevant to a particular lan- guage token are selected, and a weighted average of these patches is calculated to provide cues to where the robot policy should pay attention to. A smaller T sharpens the seftmax, concentrating the selection on the patch with the most similar feature, while a larger r produces a smoother, more evenly distributed selection across patches. Critically, only is learnalle and the entire CLIP model is frozen throughout training.
CLIP Vision Encoder
"Pick up the green trangle"
Cosine Similrtty lowary, key)
CLIP Encorde
Fused Features
3.2. Model Architecture
Parameter-Free Vision-Language Fusion
Policy Network Input We compress the extracted text- aware visual features fur into single token for each cam- era. To achieve this, we apply a learnable cross-attention pooling operation to each camera's 's ful to obtain a single feature f.I- Specifically, we use N₂ = learnable queries q, and keys and values v from sus, and compute the out- put using cross attention Xaten (g. k, v). We concatenate the N, output tokens to one single token (f))). To facili- tate better instruction following capabilities, we additionally employ another learnable cross-attention pooling on the text features fi, resulting in single text token f/ E R4, The robot's proprioceptive state is encoded through a Feed- Forward Network (FFN) to extract an embodiment feature fer At time step f, we concatenate the embodiment feature fe with the perception feature Fi and far along the channel dimension to create single token fi- This token serves as input to policy network for action prediction.
Figure 3: Text-aware Visual Features Extraction We calcu- late the similarity between the visual patch features and per-token language features, then take the softmax over the patch feature dimension. Intuitively, this gives a distribution of semantic simi- larity over all spatial locations. We then multiply the visual patch features to retrieve the visual semantic features that correspond to each token the sentence. In Appendix C, we provide more in-depth analysis and visualizations of the proposed method.
motivated by the improved ability of ClearCLIP to capture text-aligned visual features, we specifically utilize the atten- tion output Xam from the last vision attention layer, rather than the CLIP's output feature X denoting as fu (n to- kens), where
h is the total number of patch tokens from VIT. Figure 6 demonstrates how using Xam enhances the alignment between visual features and language seman- ties, illustrating the effectiveness of this approach. More details are in Appendix C.
Since the language features and the visual features have different dimensions, CLIP uses matrix per modality to project the network's s output feature to the same latent di- mension, denoted wy and w. for language and vision respectively. We normalize the text and visual features for vision-language fusion. The text features are normalized using the final layer normalization: ft = LN (fi) wr. The visual features are normalized using the post-attention layer normalization: f. LN (f))(f)) We apply L2 normal- ization to both text and visual features: ft = fllla and f. = f./ll.. flf ll2 as in standard CLIP.
Policy Network and Action Head OTTER uses a trans- former as the policy network, consisting of 4 layers and 8 heads, with a hidden dimension of 512. Fed by the com- bined features from the perception and embodiment, the model generates an action de- The model is trained with a context length of 12 steps. For each output token at given timestep, we use FFN to predict the next 12 actions. More details about our model architecture can be found in Appendix B.
Proprioception Parametrization We parameterize the pro- prioception space using 10-dimensional representation. This includes the absolute end effector translation (x, y, z), a 6DoF rotation vector, and a continuous end-effector gripper state. The 6DoF rotation vector is derived by flattening the first two rows of the 50(3) rotation matrix.
With the normalized features, we perform temperature- weighted attention:
fot (fftmma((f..T//)(C.. PF.)
(4)
Action Parametrization We employ delta end effector pose as our action parameterization. At each prediction step, the model predicts actions. Given a sequence of abso- lute end effector action transforms T1, T2. T, in tra- jectory and the current end-effector pose Ta., we define the relative transforms that the model needs to predict as T='T, T- T--,,, ... .T-IT; We then append the continu- ous absolute gripper position to each delta action. Similar
where 7 the temperature parameter. PE is the 2D sin-cos position embedding of the patch (Dosovitskiy et al., 2020), which informs the policy network of the spatial location of each patch. Same as in CLIP (Radford et al., 2021), 7 is learnable and is clipped between 0 and 100. The result- ing feature fur E Rmnxd are the text-aware visual tokens, where each row is linear combination of normalized visual
4

---

OTTER: A Lision-Language-Accion Model with Text-Aware Visual Feature Extraction
Simulation Scenes
Prysical Scenes
consider four robotic task primitives: pick up and place, poking, pouring and opening/elosing a drawer. We define task as the combination of the primitive and objects in- volved. We collect robotic datasets on multi-task scenes using a Franka robot, where there are multiple tasks that can be completed in the same scene. We consider 10 pick-and- place tasks each containing 50-80 demonstrations of human tele-operating the robot, resulting total of 724 demonstra- tions. We denote this dataset as DS-PnP. For each of the other three primitives we collect 100-200 demonstrations, with a total of 1,185 demonstrations. We denote the dataset consisting of all four primitives as DS-ALL.
Figure 4: Example scenes in the simulation (left) and in the physical environments (right) using a Franka robot.
to the proprioception representation, action using the relative end effector translation and 6DoF rotation vector, resulting in 10-dimensional action repre- sentation. When executing the predicted actions, we employ temporal ensembling (Zhao et al., 2023) in conjunction with receding horizon control (Chi et al., 2023). Through experi- mentation, we determined that an action horizon of 8 steps yields optimal performance.
express the delta
We consider a task with unseen target objects for task com- pletion as an unseen task. The training tasks involve ob- jects encountered during model training, whereas the unseen tasks test the model's ability to generalize to unseen objects or scenes, For example, the model is trained on the task of poking a wooden block and is tested on the new task of poking a radish. Example scenes in real are shown in the right column of Figure 4.
4. Experiments
We consider two classes of problems: language-conditioned multi-task learning and zero-shot generalization in unseen environments. For aunguage-conditions
We consider 19 in-distribution training tasks and 15 out-of- distribution unseen tasks across the 4 primitives (Table 6). For each task, we evaluate the model for 10 experiment trials. For each experiment trial, we vary the location of the target object and introduce 2-3 random distractor ob- jects, to evaluate the instruction following capability of the VLA models. In the unseen tasks, we provide the robot with novel target objects that are unseen during training, or novel combinations of target objects and target placement locations. This setup aims evaluate both object identifi- cation and task completion ability under more challenging and previously unseen conditions. The robot must identify and interact with the correct object based on the provided language instruction and complete the assigned task.
multi-task learn- ing, given a multi tasti setup (defined as in there are many tasks that can be performed in the same scene), the policy needs to perform the correct task corresponding to the lan- guage instruction. In the zero-shot generalization setup, the policy is provided with a language description of an unseen task, and is asked to perform the specified task in the un- seen environments. We introduce our experimental setup to evaluate the catructtionfollowiinn and text-aware visual features extraction generalization of OTTER in Section 4.1 and the baselines considered in this paper in Section 4.2.
4.1. Environment Setup
Simulation Environment We use the LIBERO bench- mark (Liu et al., 2024) for simulation evaluation. Specif- ically, we use the tasks and datasets in LIBERO-Spatial, LIBERO-Object, LIBERO- Goal, and LIBERO- -0, which contains diverse objects, scene layouts, and language in- structions. Each simulation task has 50 demonstrations. We evaluate OTTER's 's capabilities on both in-distribution tasks and unseen tasks. The nn-iisribbttinn tasks are the 30 tasks in the original LIBERO-Spatinn SSsssaaaa/lOjjee//Gaallll which can evaluate the model's 'ss multi-task learning capabilities. In addition, we also construct 10 novel tasks, where we modify the language instructions and corresponding objects of 10 original tasks from LIBERO-90 For the 10 unseen tasks, we follow the same convention in LIBERO (Liu et al., 2024) about object initialization and goal configuration by defining task bddl files. Example scenes in the simulation are shown in the left column of Figure 4.
The trial is terminated either when the task is completed or when time limit is reached. The overall performance is measured by calculating the average success rate with stan- dard error across all trials for the training and unseen tasks. The full lists of simulation and real-world environments and more experiment details can be found in Appendix A.
4.2. Baselines
To evaluate if the text-aware visual features extracted in OT- TER can better leverage the semantic understanding capa- bilities of the pre-trained VLMs, we consider four baselines that directly take all visual and language features, including three state-of-the open-sourced VLA models and one variant of OTTER:
1. Octo (Octo Model Team et al., 2024), an open-sourced transformer based policy trained from scratch on 800K trajectories from the Open X-Embodiment dataset (Col-
Real Robot Environment For real-robot evaluation, we
5

---

OTTER: A Visior-Languaag--ctiion Model with Text-Aware Visual Feature Extraction
Method
Training Tasks
Unseen Tasks
DS-PnP. For To-FasttDroid model, since our experiment setup is the Droid setup, we directly evaluate this model on our setup, denoted as mg-Fast-Droid. More details about model training and architectures are in Appendix B.
ro-Faat-Droid
61% + 5.3%
Finetuned Octo
15% + 3.4%
12% + 3.6%
OTTER w.o. CLIP vision
17% + 2.9%
11% + 2.5%
Finetuned VLA
30% + 3.9%
9% + 3.1%
For unseen pick up and place tasks, To- is able to achieve a success rate of 61 %. This is because this model is fine-tuned on Droid, which contains many pick up and place tasks. In both the training and unseen tasks, Octo struggles to accurately identify the object of interest and determine the correct placement location, leading to low success rate. We hypothesize this can be attributed to two key factors. First, Octo does not incorporate a pre-trained VLM, such as CLIP, into its network. Instead, it trains its vision encoder from scratch using a large-scale robotic dataset (OXE (Collaboration et al., 2024)), which lacks the semantic diversity found in larger vision datasets like LAION (Schuhmann et al., 2022). Second, OTTER extract text-aware visual features from the pre-trained CLIP, which results in a strong vision language association. This enables better visual grounding and generalization capabilities of OTTER to perform better on training and unseen tasks, de- spite being trained on small robotic dataset. Open and DFP-OTTER perform similarly, which is better than Octo on training tasks, but much worse than OTTER. On unseen tasks, they both fail to generalize. We hypothesize this is because it's challenging for the direct feature passing architectures to learn generalizable vision-language connec- tions on a small robotic dataset, while OTTER can utilize the extracted text-aware vision features from the pre-trained VLM. OTTER-OXE performs better than OTTER on both training and unseen tasks, suggesting that OTTER's perfor- mance scales with more data, possibly because the policy network can learn better and more precise action planning from more robotics data.
DFP-OTTER
29% 3.7%
4% 1.6%
OTTER (Finetunc CLIP)
26% + 4.0%
15% = 3.9%
OTTER w.o. Je
40% 1: 4.0%
29% 1 4.3%
OTTER W.O. f
57% + 4.4%
53% + 4.6%
OTTER (Ours)
68% + 4.3%
62% + 4.2%
OTTER-OXE (Ours)
72% 3.9%
73% + 2.8%
Table Physical Single Primitive Multi-task Experiments. For each model, we conduct physical robot pick and place exper- iments. with 100 trials on an-distribution training tasks and 70 trials unseen tasks. OTTER achieves similar success rate the in-distribution training tasks and unseen tasks, significantly outperforming the baselines, highlighting the benefits of extracting text-aware visual features and a frozen pre-trained VLM.
laboration et al., 2024).
2. OpenVL.A (Kim et al., 2024), fine-tuned Prismatic- 7B (Karamcheti et al., 2024) VLM on the Open X- Embodiment (OXE) dataset.
3. Tr-Faat-Droid (Black et al., 2024), a VLA using a pre-trained PaliGemma (Beyer et al., 2024) and a Fast (Pertsch et al., 2025) action tokenizer. This model is pre-trained on subset of OXE and the dataset, and is fine tuned the Droid (Khazatsky et al., 2024) dataset.
4. Direct Feature Passing OTTER (DFP-OTTER): a vari- ant of OTTER where the text tokens, vision tokens are passed to an attention pooling layer separately to ob- tain independent tokens, which are then concatenated with the embodiment feature fe as the input to the transformer. This baseline is constructed to inform the importance of text-aware visual feature extraction.
We evaluate all models on all four primitives. We compare the performance of Octo and Open finetuned on DS-ALL and OTTER pretrained on OXE and fine-tuned on DS-ALL, denoted as OTTER-OX- As there are more primitives, we also consider a deeper and wider OTTER model (details in Table 7), denoted as OTTER-L. For a fair comparison, we extended the context history length of Octo to 10 (Octo cannot exceed a con- text length of 10 due to its inherent design constraints) and matched its action prediction horizon to ours. As OpenVLA has many tokens per timestep, its context length cannot be extended and we use its default context length. For To- Fast-Droid model, as in the single primitive experiment, we directly evaluate this model on our setup. As most of the Droid tasks are pick and place while our setup contains other primitives, we also consider fine-tuning ne - on the DS-ALL, denoted as Finetuned -rstadd-Drod..
Multiple Primitives
5. Results
We compare OTTER against several baseline in both real- world (Section 5.1) and simulation environments (Sec- tion 5.2). We compare OTTER against several ablations in Section 5.3. In Section 5.4, we further investigate OT- TER's 'ss capabilities by scaling up models.
5.1. Real-world Experiments
Single Primitive We first evaluate all models on both the training and unseen tasks of the pick and place primitive in the real robot environment, as shown in Table 1. For fair comparisons, we fine-tune Octo and VLA on DS-PnP using the same amount of learning steps. We compare the performance of OTTER trained from scratch and OTTER- OXE pre-trained on the OXE dataset and fine-tuned on
Results are shown in Table 2. All models are evaluated
6

---

OTTER: A Vision-Languaag--cciinn Model with Text-Aware Visual Feature Extraction
Method
Pouring
Drawer
Poking
Pick and Place
Mean+tStd. Err.
Ty To-FFast-roid
0%
0%
0%
61%
29% 3.5%
Finetuned To- Fast-Droid
0%
45%
27%
51%
35% = 3.8%
Finetuned Octo
0%
0%
0%
5%
4% 1.2%
Finetuned Open VLA
0%
0%
0%
1%
0.6% + 0.5%
OTTER
63%
50%
93%
61%
67% = 3.8%
OTTER-L
65%
55%
90%
69%
71% 3.5%
OTTER-ORE -OTER-
60%
65%
93%
66%
70% -3.6%
OITER-OXE4.
77 77%
75%
93%
75%
77% = 3.3%
Table 2: Multi-primitive zero-shot generalization: train models across four manipulation primitives (pouring, drawer manipulation, poking, and pick-and-plast pladpp with total of 1,185 human tele-operated demonstration trajectories and evaluate them on 150 trials of completely unseen tasks within these primitives. Despite the inherent difficulty of zero-shot generalization across multiple primitives. OTTER achieves significantly higher success rates compared to baselines across all primitives. While baseline models struggle with generalization, particularly on pouring task (0% success rate), OTTER maintains substantial performance (60-93% success rate) unseen tasks. The results further suggest that OTTER'S 's generalization capabilities can be enhanced through increased model capacity (OTTER-L) and pre-trainn training on large robotic datasets (OTTER-ONS -XE)..
on unseen tasks for each primitive, with 10 trials for each task. While To-Fast-Drau -roid achieves decent performance on the pick and place primitives, it fails on all the other three primitives as the majority of the Droid dataset is the pick and place primitive. Finetuned To-Fast--Dree Droid achieves non-zero success rate on Drawer and Poking primitives, but still fails on the pouring primitive. There is also de- graded performance on the Pick and Place primitives for Finetuned Fo-Fasst-roohd This highlights the difficulty of our experiment setup. The performance of Octo, Open- VLA, and OTTER- OXE on the pick and place task all drop. showing the difficulty of multi-primitive learning. Notably, both Octo and VLE fail to complete any unseen tasks for the pouring, drawer, and poking tasks, likely due to a relatively small amount of demonstrations provided for each primitive. Both OTTER-OXE and OTTER-OXE-L can achieve high success rate on all four primitives on the same amount of demonstrations, indicating that using text-aware visual features extracted from a pre-trained VLM in- crease the data efficiency and enhance the generalization ability. OTTER-L outperforms OTTER average, with the performance gap increases as OTTER pre-trained on the OXE dataset, indicating that OTTER can scale with model size.
and sufficient demonstration datasets. For tasks in LIBERO- Object and LIBERO Spatial,,, DFP-OTTER is worse than OTTER because OTTER does better in localizing the ob- ject to interact with. Note that Octo and VLA pre-trained the OXE dataset but DFP-OTTER is trained from-scratch on LIBERO, so DFP-OTTER 's performance is worse. In unseen tasks, OTTER can outperform other baselines by a large margin, demonstrating its generalization capabilities to novel scenarios, which consistent with the conclusion of the real-world experiments.
5.3. Ablations
We perform ablation studies with Linge-primitive multi- task setting, where all models are trained on DS-PnP. We consider the following ablations on the design choices of OT- TER. Addition ablation studies can be found in Appendix D.
1. OTTER w.o. fe: OTTER without the embodiment representation fe. The concatenated text token fí and fused vision-language token f/c are passed as the input the transformer.
OTTER w.o. f/: OTTER without the text token fí- Only f/ and fe are concatenated as the input to the transformer.
5.2. Simulation Experiments
We compare OTTER with various baselines in simulation, as shown Table 3. For fair comparison with Open VLA and Octo, we process the simulation datasets following the procedure described in the VLE paper (Kim et al., 2024), and use their reported performance the standard LIBERO tasks. For the unseen tasks, we change 10 tasks in LIBERO-90 with different objects and distractors to test the generalizability of all the models on unseen tasks. From Ta- ble 3, we found all the models perform similarly on training tasks in LIBERO due the limited variations of the tasks
OTTER w.o. CLIP vision: OTTER using smaller ViT train from seratch instead of frozen pre-trained CLIP vision encoder.
4. OTTER (Finetune CLIP): OTTER with the CLIP ini- tialized from the pre-trained weight and fine-tuned end to end on the robotic dataset.
Real-world Results Physical results in Table suggest that
the performance on both the training tasks and the unseen
7

---

OTTER: A Vision-Language--ctiinn Model with Text-Aware Visual Feature Extraction
Method
LIBERO-Spatie
LIBERO-Object
LIBERO-Goal
Train (Average)
Unseen
Finetuned Octo
79% + 1.0%*
86% +:: 0.9%*
85% + 0.9%* +
83% 1 1.0%
26% +: 1.1%
Finetuned VLA
85% = 0.9% *
88% + 0.8%*
79% + 1.0%*
84% + 0.9%
48% + 1.0%
DFP-OTTER
79% 0.9%
80% + 1.0%
78% + 1.1%
79% + 1.0%
45% + 1.0%
OTTER (Ours)
84% 1.0%
% 1.2%
79% 1 0.9%
84% 1 1.1%
61% 1 1.1%
Table 3: Simulation results LIBERO. We evaluate OTTER and other baselines on 30 in-distribution tasks in LIBERO- Spatial/Objec//Goal and on 10 unseen tasks we constructed, each task 50 trials. The numbers marked with * are directly referred from the VLA (Kim et al., 2024) paper. The detailed training and evaluation on the unseen tasks are described in Section 4.1.
tasks drop significantly for the ablations compared to OT- TER. The performance of OTTER without the embodiment features (fe) drops 28% on the training tasks and 33% the unseen tasks, indicating that fe is vital for task comple- tion and generalization, likely because it provides a physical grounding for decision-making. Without fe- the model's understanding of embodied features, possibly linked to the spatial or physical aspects of the task, is severely impaired. OTTER without the language features (ff) experiences per- formance drop of around 10% on both training and unseen tasks, suggesting that Si provides complementary informa- tion that may help in more nuanced task understanding.
Method
LIBERO-Objed
Unseen
OTTER w.o. CLIP vision
80% + 0.7%
29% + 0.9%
OTTER W.O. fe
79% + 0.9%
48% 1 0.8%
OTTER W.o. f
71% + 1.2%
49% 1 1.0%
OTTER (Ours)
8 + 1.2%
61% E 1.1%
Table 4: Ablation results on LIBERO Object tasks and unseen tasks. We evaluate OTTER and other baselines on 100 trials of in- distribution LIBERO-Object tasks, and 100 trials of unseen tasks.
Training Unseen
70
OTTER w.o. CLIP vision has a significant performance drop of more than 50% on the training and unseen tasks in physical experiments. The results of OTTER w.o. CLIP vision is similar to Octo which also trains a vision encoder from scratch on the robotics dataset. This suggests that pre- trained VLM provides more robust and transferable visual representations. Training a vision encoder from scratch can result in poor performance, as it lacks the generalization capabilities learned from large-scale pre-training.
60
VI-1//1
Success (%)
50
VIT.8/32
VIT-B/16
40
30
20
10
50
100
150
GFLOPs
OpenVLA demonstrates that fine-tuning the vision encoder of the pre-trained VLM the robotics dataset is crucial for improving the performance. However, found that fine-tuning the pre-trained vision encoder actually degrades the model's 's vision- language understanding capabilities. The large performance discrepancy between training and unseen tasks in both VLE and OTTER (Finetune CLIP) sug- gests that fine-tuning compromises generalization ability, highlighting the benefits of OTTER's approach of extracting text-aware visual features from frozen pre-trained VL... We want to emphasize that both the effective feature ex- traction and the frozen encoder are crucial for learning a generalizable VLA, as shown by the worse performance of DFP-OTTER with a frozen VLM, OTTER (Finetune CLIP) that has a fine-tuned VLM and OpenVLA that is a VLA with direct feature passing and fine-tuned VLM.
Figure We evaluate OTTER's 's performance with improved vision language features by scaling CLIP. In particular, we train OTTER with three CLIP variants with increasing FLOPs: ViT- B/32, ViT-B/16, and ViT-L/16. We report the task performance vs. the inference FLOPs per image on training and unseen tasks. The results suggest that the OTTER can benefit from scaling up vision- language model.
The performance of OTTER w.o. Se drops about 10% on both the in-distribution and unseen tasks, indicating that fe is beneficial for task completion as it provides explicit spatial information of the robot. OTTER w.o. Jí is also no- ticeably worse. We hypothesize this is due to the object are not very realistic in simulation, so the extracted text-aware visual features in CLIP may highlight multiple objects or wrong objects. f/ can provide complementary information for the policy to interact with the correct objects.
Simulation Results Table 4 presents the simulation results of OTTER and other ablations. On the in-distribution tasks, OTTER w.o. CLIP vision and OTTER can work similarly well given sufficient demonstrations, but OTTER w.o. CLIP vision is more than 50% worse on unseen tasks, which shows the benefits of using pre-trained VLM for better generalization capabilities.
5.4. Scaling up Vision and Language Encoders
Prior work has shown that lasionn encoders exhibit improved semantic understanding capabilities as their com- putational complexity increases (Radford et al., 2021). To in- vestigate whether OTTER can leverage these improvements, we evaluated its performance using three CLIP model vari-
8

---

OTTER: A Vision-Languaag--cciinn Model with Text-Aware Visual Feature Extraction
ants of increasing computational complexity: ViT-B/32, ViT-B/16, and ViT-L/14.. As shown in Figure 5, OTTER's task success rate improves substantially as we scale up the CLIP model's FLOPs, with gains observed on both train- ing (+27.5%) and unseen pick-and-place tasks (+39.3%) when moving from ViT-B/32 to ViT-L/14. These results demonstrate that OTTER effectively utilizes larger vision- language encoders for enhanced semantic understanding and serves scalable method for integrating large-scale vision- language pre-training with learning from robotic datasets.
Impact Statement
This paper presents work whose goal is to advance the field of machine learning and robotics. There many potential soci- etal consequences our work, none of which we feel must be specifically highlighted here.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.., Akkaya, I., Aleman, F. L.., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
6. Limitations and Conclusions
Alayrac J.-B., Recasens, A., Schneider, R., Arandjelovic, R., Ramapuram, J., De Fauw, J., Smaira, L., Dieleman, S., and Zisserman, A. Self-supervised multimodal versatile networks. Advances in neural information processing systems, 33:25-37, 2020.
While OTTER demonstrates improved task completion rates compared to existing VLAss, it still faces several limitations. One significant challenge is scaling across different mor- phologies, particularly those that cannot be easily parame- terized by SE(3) transforms (i.e. robot multi-finger hand). This limitation restricts the model's adaptability to wider range of robotic platforms and task types. Furthermore, this study has not extensively explored how this method scales to long-horizon tasks and more complex scenes, which could be an important area for future research.
Ba, J. L. Layer normalization. arXiv preprint arXiv: 1607.0445, 2016.
Bai,J. Bai, S., Yang. S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C. and Zhou, J. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966. 2023.
Beyer, L., Steiner, A., Pinto, A. S., Kolesnikov, A., Wang. X.. Salz, D., Neumann, M., Alabdulmohsin, I., Tschannen, M., Bugliarello, F., Unterthiner, T., Keysers, D.,, Koppula, S., Liu, F., Grycner, A., Gritsenko, A., Houlsby, N., Kumar, M., Rong, K., Eisenschlos. J., Kabra, R., Bauer, M., Bošnjak, M.. Chen, X., Minderer, M.. Voigtlaender. P., Bica, L., Balazevic, L., Puigcerver, J., Papalampidi, P., Henaff, O., Xiong. X., Sori- cut, R., Harmsen, J., and Zhai, X. Paligemma: A versatile 3b vim for transfer, 2024. URL https: / /ttpavaaivv///////.////./.a.aav. org/abs/ abs//
In summary, present OTTER, vision- larion-langage-acttion model that leverages text-aware visual feature extraction from pre-trained vision- languale encoders. By utilizing the semantic alignments during large-scale vision-language pre-training, OTTER achieves significantly better general- ization than existing VLA models across robot manipula- tion tasks sampled from multiple motion primitives, main- taining higher success rates unseen objects and envi- ronments where previous methods fail. The experiments demonstrate that OTTER's performance scales along mul- tiple axes: through larger pre-trained vision-language en- coders, increased policy network capacity, and pre-training on larger robot datasets. These results suggest that by better preserving existing alignment in pre-trained vision-languaae encoders, rather than learning them directly from robotics data, is beneficial for developing more capable and general- izable robot learning systems.
Black, K., Brown, N., Driess, D.. Esmail, A., Equi M., Finn, C., Fusai, N., Groom, L.,, Hausman. K., Ichter, B., Jakubczak. S., Jones, T., Ke, L.. Levine, S., Li-Bell, A.. Mothukuri, M., Nair, S., Pertsch, K., Shi, X., Tanner, J., Vuong, Q. Walling, A., Wang, H., and Zhilinsky, U. To: A vision-languaae-action flow model for general robot control, 2024. URL https: 1 //arxiv.. org/ aba/2410 24164.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C.. Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., et al. Rt-1: Robotics transformer for real-world control at scale. arXiv:22/2.06817, 2022.
Acknowledgement
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choro- manski, K., Ding, T., Driess, D.,. Dubey, A., Finn, C., et al. RT-2: Languan-a action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307. 158.8 2023.
Dataset usage and model training work solely conducted at Berke- ley. Pieter Abbeel holds concurrent appointments as Professor UC Berkeley and an Amazon Scholar. This paper describes work performed at UC Berkeley and not associated with Amazon. This research was performed at the AUTOLAB at UC Berkeley in affiliation with the Berkeley AI Research (BAIR) Lab, and the CITRIS "People and Robots" (CPAR) Initiative. In their academic roles at UC Berkeley, Letian Fu, Fangchen Liu, Huang Huang, and Ken Goldberg are supported in part by by donations from Meta, Autodesk, Google, Siemens, Toyota Research Institute, Bosch, and by equipment grants from PhotoNeo, Nvidia, NSF AI4OPT Centre, and Intuitive Surgical.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in Neural Information P Processing Systems, 33:1877-1901, 2020.
2020.
Cherti, M., Beaumont, R., Wightman. R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, J. Reproducible scaling laws for contrastive language-image learn- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818-2829, 2023.
9

---

OTTER: A Vision-Language-Accii Model with Text-Aware Visual Feature Extraction
Chi, C., Feng, S., Du, Y., Xu, Z., Cousincau, E., Burchfiel, B., and Song, S. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.041 2023.
Dong, X., Bao, J., Zheng, Y., Zhang, T., Chen, D., Yang, H., Zeng, M., Zhang, W., Yuan, L., Chen, D., et al. Maskelip: Masked self- distillation advances contrastive language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10995-11005, 2023.
Chowdhery. A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A.. Barham, P.. Chung. H. W. Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, (((240)11-113, 2023.
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenhorn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
Collaboration, E., O'Neill, A., Rehman, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., Jain, A., Tung. A., Bewley, A., Herzog. A., Irpan, A., Khazatsky, A., Rai, A., Gupta, A., Wang, A., Kolobow, A., Singh, A.. Garg, A., Kembhavi, A., Xie, A., Brohan, A., Raffin, A., Sharma, A., Yavary, A., Jain, A., Balakrishna, A., Wahid, A., Burgess-
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilaaa 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Limerick, B., Kim, B., Schölkopf. B., Wulfe, B., lchter, B., Lu, C., Xu, C., Le, C., Finn, C., Wang. C., Xu, C., Chi, C., Huang. C., Chan, C., Agia, C., Pan, C., Fu, C., Devin, C., Xu, D., Morton, D., Driess, D., Chen, D., Pathak, D., Shah, D., Büchler, D., Jayaraman, D., Kalashnikov, D., Sadigh, D., Johns, E., Foster, E., Liu. F., Ceola, F., Xia, F., Zhao, F., Frujeri, F. V., Stulp, F. Zhou, G.. Sukhatme, G. S.. Salhotra, G. Yan. G., Feng, G., Schiavi, G., Berseth, G.. Kahn, G., Wang. G.. Su, H., Fang, H.-S., Shi, H., Bao, H., Amor, H. B., Christensen, H. I., Furuta, H., Walke, H., Fang, H., Ha, H., Mordatch, I., Radosavovic, I., Leal, I., Liang, J., Abou-Chakra, J., Kim, J. Drake, Peters. J., Schneider, J., Hsu, J., Bohg, J., Bingham, J. Wu, J., Gao, J.. Hu, J., Wu, J.. Wu, J.. Sun, J., Luo, J., Gu, J. Tan, J., Oh, J. Wu, J., Lu, J., Yang. J., Malik, J., Silvério, J. Hejna, Booher, J., Tompson, J., Yang. J.. Salvador, J., Lim, J.J. Han, J., Wang. K. Rao. K., Pertsch, K., Hausman, K., Go, K., Gopalakrishnan, K., Goldberg, K., Byrne K., Oslund, K., Kawaharazuka, K., Black, K., Lin, K., Zhang, K., Ehsani, K., Lekkala, K., Ellis, K., Rana, K., Srinivasan, K., Fang. K., Singh, K. P., 7eng, K.-H., Hatch, K., Hsu, K., Itti, I.., Chen, L. Y., Pinto, L.. Fei-Fei, L... Tan, L.. Fan. L. J., Ott, L., Lee. L.. Weihs, L.. Chen, M., Lepert, M., Memmel, M., Tomizuka. M., Itkina, M. Castro, M. G., Spero, M., Du, M., Ahn. M.. Yip. M. C., Zhang, M., Ding, M., Hco, M., Srirama, M. K., Sharma, M., Kim. M. J., Kanazawa, N., Hansen, N., Heess, N., Joshi, N. J., Suenderhauf, N., Liu, N., Palo, N. D., Shafiullah, N. M. M., Mees, O., Kroemer, O., Bastani, O., Sanketi, R., Miller, P. T., Yin, P., Wohlhart, P., Xu, P., Fagan, P. D., Mitrano, P., Sermanet, P., Abbeel, P.. Sundaresan, P., Chen, Q. Vuong, Q., Rafailov, R., Tian, R. Doshi, R., Mart'in -nr-iii-Miinn in, R., Baijal, R., Scalise, R., Hendrix, R., Lin, R. Qian. R. Zhang, R., Mendonca, R., Shah. R., Hoque, R., Julian, R., Bustamante, S., Kirmani, S., Levine, S., Lin, S., Moore, S., Bahl, S., Dass, S., Sonawani, S., Song, S., Xu, S., Haldar, S., Karamcheti, S., Adebola, S., Guist, S., Nasiriany, S., Schaal, S., Welker, S., Tian, S., Ramamoorthy, S., Dasari, S., Belkhale. S., Park, S., Nair, S., Mirchandani, S., Osa, T. Gupta, T., Harada, T., Matsushima. T., Xiao, T., Kollar, T. Yu, T.. Ding, T., Davehev, T., Zhao, T. Z., Armstrong, T., Darrell, T.. Chung, T., Jain, V., Vanhoucke, V., Zhan, W., Zhou, W., Burgard, W., Chen, X., Chen, X., Wang, X., Zhu, X., Geng, X., Liu, X., Liangwei, X., Li, X., Pang. Y., Lu, Y., Ma, Y. J., Kim, Y. Chebotar, Y., Zhou, Y., Zhu, Y., Wu, Y., Xu, Y., Wang. Y. Bisk, Y., Cho, Y., Lee, Y., Cui, Y., Cao, Y., Wu, Y-HH, Tang, Y., Zhu, Y., Zhang, Y., Jiang, Y., Li, Y., Li, Y., Iwasawa, Y., Matsuo, Y., Ma, Z.,, Xu, Z.,, Cui, Z. J., Zhang, Z.., Fu, Z.,, and Lin, Z. Open x-embodiment: Robotic learning datasets and rt-x models, 2024.
Fu, L., Huang, H. Datta, G., Chen, L. Y., Panitch, W. C.-H., Liu, F., Li, H., and Goldberg, K. In-context imitation learning via next-token prediction. arXiv preprint arXiv:2408. 15980, 2024.
He, K.. Zhang, X.., Ren, and Sun, J. Deep residual learning for image recognition. In CVPR, 2016.
Jang, E.,, Irpan, A., Khansari, M., Kappler, D., Ehert, F., Lynch, C., Levine, S., and Finn, C. Be-z: Zero-shot task generaliza- tion with robotic imitation learning. In Conference on Robot Learning, 2022.
Jia, C., Yang. Y., Xia, Y., Chen, Y.TT., Parekh, Z., Pham, H., Le, Q.. Sung, Y.-H., Li, Z., and Duerig, T. Scaling up visual and vision- language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904 4916. PMLR, 2021.
Jiang, Y., Gupta, A., Zhang, Z., Wang, G., Dou, Y., Chen, Y., Fei- Fei, L., Anandkumar, A., Zhu, Y., and Fan, L. VIMA: General robot manipulation with multimodal prompts. International Conference Machine Learning (ICML), 2023.
Karamcheti, S., Nair, S., Balakrishna, A., Liang, P., Kollar, T., and Sadigh, D. Prismatic vims: Investigating the design space of cisually-conditioned language models. arXiv preprint arXiv:2402.0 008666 2024.
Kerr, J., Kim, C. M., Goldherg, K., Kanazawa, A., and Tancik, M. Lerf: Language embedded radiance fields. In International Conference on Computer Vision (ICCV), 2023.
Khazatsky, A., Pertsch, K., Nair, S., Balakrishna, A., Dasari, S., Karamcheti, S., Nasiriany, S., Srirama, M. K., Chen, L. Y., Ellis, K. Fagan, P. D., Hejna, J., Itkina, M., Lepert, M., Ma, Y. J., Miller, P. T., Wu, J., Belkhale, S., Dass, S., Ha, H., Jain, A., Lee, A., Lee, Y., Memmel. M., Park, S., Radosavovic, L., Wang. K.,, Zhan, A., Black, K., Chi, C., Hatch, K. B., Lin, S., Lu, J., Mercat, J., Rehman, A., Sanketi, P. R., Sharma. A., Simpson. C., Vuong, Q. Walke, H. R., Wulfe, B., Xiao, T., Yang. J. H., Yavary, A., Zhao, T. Z.,, Agia, C., Baijal, R., Castro, M. G., Chen, D., Chen, Q., Chung, T., Drake, J., Foster, E. P., Gao, J., Herrera, D. A., Heo, M., Hsu, K., Hu, J., Jackson, D., Le, C., Li, Y., Lin, K., Lin, R., Ma, Z., Maddukuri, A.. Mirchandani, S., Morton, D., Nguyen, T., O'Nlll,, A., Scalise, R.,, Seale, Son. V., Tian, S., Tran, E., Wang, A. E., Wu, Y., Xie, A.. Yang. J., Yin, P., Zhang. Y., Bastani, O., Berseth, G., Bohg. J., Goldberg. K., Gupta, A., Gupta, A., Jayaraman, D.,. Lim, J. J.. Malik, J., Martin-Martin R., Ramamoorthy, S., Sadigh, D., Song, S., Wu, J., Yip, M. C., Zhu, Y., Kollar, T., Levine, S., and Finn, C. Droid:
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre- training of deep bidirectional transformers for language under- standing. arXiv preprint arXiv: 1ate::100000000 040004 2018.
A large-scale in wi-the-wild robot manipulation dataset, 2024.
10

---

OTTER: A Vision-Languaag--cciinn Model with Text-Aware Visual Feature Extraction
Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., Vuong, Q. Kollar, T., Burchfiel, B., Tedrake, R., Sadigh, D., Levine, S., Liang, P., and Finn, C. Openvla: An open-source vision- language-action model, 2024. URL www.hhtt///aeee. http::///aaxvvvioiaae abs 2los/20 2406. 09246.
Reed, S., Zolna, K., Parisotto, E., Colmenarejo, G., Novikov, A., Barth- G., Gimenez, M., Sulsky, Y., Kay, J., Sprin- genberg, J. T., ct al. A generalist agent. arXiv preprint arXiv:2255.06/75. 2022.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, ct Laion-5h: An open large-scale dataset for training next generation image-text models. Advances Neural Information Processing Systems, 35:25278-25222 22299444, 2022.
Lan. M., Chen, C., Ke, Y., Wang. X., Feng, L.. and Zhang. W. Clearclip: Decomposing clip representations for dense vision- language inference. In ECCV, 2024.
Laurençon, H., Tronchon, L., Cord, M., and Sanh, V. What mat- ters when building vision-language models? arXiv preprint arXiv:2405.02244 02244, 2024.
Shah, D., Sridhar, A., Dashora, N., Stachowicz, K., Black, K., Hirose. N., and Levine. S. ViNT: A Foundation Model for Visual Navigation. In 7th Annual Conference on Robot Learning (CoRL), 2023.
Li J., Li, D., Savarese, S., and Hoi, S. 22 Bootstrapping language-ing -angu pre-training with frozen image encoders and large language models. arXiv preprint ar 12592 2023,
Yao, L.. Huang, R., Hou, L.,, Lu, G.. Niu, M., Xu, H., Liang, X., Li, Z.. Jiang, X., and Xu, C. Filip: Fine-graince interactive language-iman pre pre-training. arXiv preprint arXiv:2111.07783, 2021.
Liu, B., Zhu, Y., Gao, C., Feng, Y., Liu, Q. Zhu, Y., and Stone, P. Lihero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36, 2024.
Yuan, L., Chen, D., Chen, Y.-L., Codella, N., Dai, X., Gao, Hu, H., Huang, X., Li, B., Li, C., et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv2211..11422 2021.
Liu, H. Li. C., Wu, Q., and Lee, J. Visual instruction tuning. In NewrIPS, 2023.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L., Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 11975-11986, 2023.
Octo Model Team, Ghosh, D., Walke, Pertsch, K.. Black, K., Mees, O., Dasari, S., Hejna, J., Xu, C., Luo, J., Kreiman, T., Tan, Y., Chen, I. Y., Sanketi, P., Vuong, Q. Xiao, T., Sadigh, D., Finn, C., and Levine, S. Octo: An open-source generalist robot policy. In Proceedings Robotics: Science and Systems, Delfi, Netherlands, 2024.
Zhao, T. Z.,, Kumar, V., Levine, S., and Finn, C. Learning fine- grained himanual manipulation with low-cost hardware, arXiv preprint arXiv:2304. 13705, 2023.
OpenAI. Gpt-4o system card. https://cda. openai. com/ popt-4o-ai syst seyyttemeee wwweeeerrrrreeeeeeeeeeeeeeeeeeee card. 2024. Accessed: 2024-09- 14.
Perez, E., Strub, F., De Vries, H., Dumoulin, V., and Courville, A. Film: Visual reasoning with general conditioning layer. In Proceedings the AAAI conference artificial intelligence, volume 32, 2018.
Pertsch. K.. Stachowicz. K., Ichter, B., Driess, D., Nair. S., Vuong, Q.. Mees, O.,, Finn, C., and Levine, S. Fast: Efficient action models, 2025. URL tokenization for vision-language-actinn 09747.
https://arxiv..rt
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training. 2018.
Radford, A., Wu, J., Child, R., Luan, D.. Amodei, D., Sutskever, c, al. Language models are unsupervised multitask learners. OpenAl Blog, 18))9, 2019.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agar- wal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language su- pervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Rao, Y., Zhao, W., Chen, G., Tang, Y., Zhu, Z., Huang, G., Zhou, J., and Lu, J. Denseclip: Language guided dense prediction with context-aware prompting. In Proceedings of the IEEE Con- ference Computer Vision and Pattern Recognition (CVPR). 2022.

---

OTTER: A Visior-Languaag--Aciion Model with Text-Aware Visual Feature Extraction
A. Environment Setup
A.1. Simulation Tasks
For the training tasks, use the original tasks in LIBERO-Goal, LIBERO-Spatial, and LIBERO- Object. We also build unseen evaluation lasks based on 10 original LIBERO-90 tasks, by changing language instructions and target object color and type in the task bidl files. The 10 unseen tasks are listed Table
Changes
Unseen
object type
Put the moka pot in the bottom drawer of the cabinet
object type
Put the moka pot on the wine rack
object type
Pick up the ketchup and put it in the basket
object type
Pick up the ketchup on the plate
object type
Pick up the bottle and put in the tray
object color
Put the black bowl on top of the cabinet
object color
Put the black bowl on the plate
object color
Put the red mug to the right of the plate Put the yellow and white mug in the front of the red mug Put the red mug to the front of the moka
object color
object color
Table 5: The 10 in-distribution tasks and 7 unseen tasks we used in our real-world setting.
A.2. Real-world Tasks
The full list of tasks for our real- evaluation is provided in Table 6.
In- In-Distribution
Unseen
Put potato in pot to black bowl
Put yellow cube in black bowl Pick up radish and place in grey bowl
Pickup potato
Pick up and place deer in grey bowl
Put blue bear in pink bowl Put yellow cube in grey bowl Put apple with green leaf in black bowl Pick up blue sponge and place it in steel pot Pick up black dog and place in the pink bowl
Pick up green triangle
Put tiger to black bowl
Put red cube into black bowl
Put blue cube into grey bowl
Put the red ball in black bowl
Put green triangle into pink bowl Put blue cube in pink bowl
Poke a wooden block
Poke the radish Poke the gray dog Poke the pink bowl
Poke a tiger
Poke a green triangle
Poke gray bowl
Pour from the brown cup to the gray bowl
Pour from the orange cup to the black bowl
Pour from the blue cup to the pink bowl
Pour from the blue cup to the black bowl
Pour from the yellow cup to the black bowl
Pour from the brown cup to the pink bowl
Open the drawer
Open the drawer with a tiger on top
Close the drawer
Close the drawer with a red cube inside
Table 6: The 10 in-distribution tasks and unseen tasks we used our real-world setting.
For each experiment trial of poking and pouring, we vary the location of the target object to manipulate and introduce or 3 random distractor objects. For drawer, we vary the location the drawer on each trial. Similar the pick and place primitive, for each task, we generate 10 randomized scenes.
Each trial is scored based on the robot's performance in completing the task. For the pick and place primitive, a score of 0.5 is awarded if the robot successfully picks up the correct target object, and score of 1 is given if the robot not only picks up the correct object but also places it in the correct location as specified by the instruction. If the robot fails pick up the target object picks up a distractor object, score is recorded. For other primitives, a score is recorded if task is completed, otherwise a score
given.
12

---

OTTER: A Vision-Languuag--Aciion Model with Text-Aware Visual Feature Extraction
For all models other than the OpenVL.A, cach trial allowed maximum of 30 seconds to complete. As OpenVLA is large 7B model wit lower inference speed, we give it a time limit of 60 seconds to complete a task.
B. Model and Training Details
B.1. Model Architecture for OTTER and Baselines
The details of our model parameters can be found Table 7. All the baselines share the same hyper-parameters with OTTER. For OTTER w.o. CLIP Vision, we use a VIT Encoder based on the implementation of wttttt:://// attps://githuu con/ reseaahheseearcc/// viiiiviiinn..... trans former with a VIT-T1/16 configuration with half of the number of attention layers. For OTTER w.o. f. and OTTER w.o. f(- we the same model configuration but only remove the corresponding attention pooling layers. We incorporate action chunking into predict the next 16 actions, which performs better than vanilla Open I.A which predicts only the next step. For Octo, we use the official Hugging Face Checkpoint at hf: / ra 1 -berkeleee eyy ee/octooooeeeeeeeeeoeeeeeeeeeeee sma 1-1.5 which is a comparable size with our model. During inference, we cache the CLIP feature outputs. This enables the ViT-L/14 OTTER model to perform inference at 50Hz z on single NVIDIA 3090Ti, allowing real-time control.
Open VL.A by asking
Hyperparameter
Value
CLIP Model (Real)
ViT-L/14
CLIP Model (Sim)
ViT-L/32
# Pooling Readouts
4
# Pooling Attention Heads
8
# Pooling Attention Blocks
2
# Text- Text-Pooling Output Dimension
128
# image-Pooling Output Dimension
512
# Proprio-Pooling Output Dimension
64
Causal Transformer Parameters:
# Attention Blocks
4 (8)
# Attention Heads
8
# Latent Dimension
512 (768)
# Context Length (Real)
12
# Context Length (Sim)
4
# Action Prediction Horizon (Real)
12
# Action Prediction Horizon (Sim) # Parameter
1
11,790,522 (25,516,986)
Table 7: Hyperparameters for OTTER model architecture. Values in the parenthesis shows the hyperparameters for a larger and wider OTTER for the real world experiments.
B.2. Training
We use the AdamW optimizer with a cosine learning rate decay schedule and linear learning rate warm-up. We list training hyperparameters in Table 8. All these typer-parameters shared between real-world and simulation. All the models trained on 4 NVIDIA A100 80GB GPUs.
C. Vision-Language Attention Visualization
To provide further motivations for why using Xout (per (Lan et al., 2024)) instead of the output feature map of CLIP, we compare the cosine similarity between the output of the CLIP ViT-L/166 encoder and the per-token text features three different settings: (1) fine-tuning the encoder, (2) a frozen CLIP's output features (Xout), and (3) a frozen CLIP's last attention block's feature (Xatta) as described in Section 3.1. The similarity visualization is shown in Figure 6.
It may initially seem unexpected that this type of visualization is reasonable. However, this can be explained by the fact that LayerNorm operates independently of the patch dimension, it normalizes along the channel dimension. When combined with the vision-alignment weight matrix w., the operation f. = LNpua (f) remains linear. Therefore we can linearize the final attention block:
fo LNuoutXoouttus
(1)
LN Xres + Xattn + FFN(LN((Xum)) swm)
(2)
LN Xrex) + LNpoa (Naeen )w, + LN (FFN(LN (Xam))))
(3)
13

---

OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction
Hyperparameter
Value
Learning Rate
3e-4
Warmup Steps
2000
Weight Decay
0.01
Learning Rate Scheduler
cosine
Gradient Clip Threshold
1
Batch Size
64
Total Gradient Steps
40000 (60000)
Image Resolution
224 x 224
Random Resized Ratio
[0.9, 1.1]
Random Brightness
0.2
Random Contrast
[0.8, 1.2]
Random Saturation
[0.8, 1.2]
Random Hue
0.1
Table 8: Hyperparameters used for training (pre-training on OXE).
Image Fine-tuned CLIP
Frozen CLIP X
Frozen CLIP X
uma
Blue
Cube
Blue
Cubs
Blue
Cube
Red
Ball
Red
Ball
Red
Ball
Green
Triangle
Green
Triangle
Green
Triangle
Pink
Bowl
Pink
Bowl
Pink
Bowl
Figure 6: Examples of attention maps for CLIP fine-tuned with VLA (left) and frozen CLIP's output (Xout) (middle) and frozen CLIP's attention features (Xutw.) (right). The first column shows the side view observation Fine-tune CLIP pays attention to the background and the frozen CLIP's output (Xout) noisy. In contrast, the frozen CLIP (Xatt)) pays attention to the correct object associated with the text query. These examples indicate that degrade the performance of the pre-trained CL.PP, especially when the roboties dataset is small. It also highlights the henefits of using X Xattn for fused vision-language features.
text query below each attention map.
CLIP on robotic datasets can
CLearCLIP, or Frozen CLIP Xattn, we are visualizing the LN
LNNostXatta (Xattn) wu term.
Similar to what ClearCLIP has noted, after adding residual connection and the final FFN, the features become noisy and worsen the alignment between language and visual features. The noisy attention map makes challenging for the model to identify the correct features directly from the feature map, which makes it necessary for existing VLA (i.e. VLA VLA (OpenAl, 2024)) fine-tune the CLIP extract the location of the
vision encoder. In comparison, by using Xattn. object localization becomes easier task OTTER: object by getting the sofimax across attention map without using any parameters (see Figure 3). More attention map examples on Open-X dataset are in Figure 7.
In the finetuning v.s. frozen CLIP (Xatt.) comparison, fine-tuning OTTER's 's CLIP results in overfitting to foreground-bace background separation, causing it to lose zero-shot object detection ability. This limits the model's ability to highlight the correct object, leading to significant drop in task success rates (26% 68% for training tasks and 15 62% for unseen tasks). Conversely, frozen CLIP (Xattn) preserves object detection capabilities, providing better downstream performance.
14

---

OTTER: A Lasoun-Lngguage-Action Model with Text-Aware Visual Feature Extraction
Coke
Vegetable
Carrot
Orange
Cup
Water
Dough
Lid
Figure 7: Examples of attention maps of frozen CLIP's attention features (Xattn) Open-X dataset. The bottom texts are the corresponding text tokens.
D. More Ablations
consider another ablations OTTER. Both are trained on the DS-PnP.
DFP-OTTER (CLS): another variant of OTTER that utilizes CLIP's jolsğ token rather than text-aware visual feature extraction.
2. OTTER (xatın): OTTER using standard attention pooling between the text tokens and the vision tokens to obtain the fused vision language features fie instead text-aware visual feature extraction as in Eq. 4).
From Table both DFP-OTTER (cls) and OTTER (xattn) fail to generalize to unseen tasks, highlighting the benefits using text-aware visual feature extraction to ohtain task-related vision features as the fused vision language features.
Method
DFP-OTTER (CLS) OTTER (xattn)
OTTER
Success Rate
6% + 0.8%
2% + 0.5%
62% + 4.2%
Table 9: Physical results on 70 trials on unseen pick up and place tasks for other variants of OTTER.
15


Reviews:
[{"review_id": "pZQVJ54kDC", "reviewer": "Reviewer_XXz9", "summary": "The OTTER keeps pre-trained VLMs fixed to preserve the rich semantic understanding acquired during pretraining, enabling strong zero-shot generalization to novel objects and environments, as demonstrated in both simulation and real-world experiments. This text-aware visual feature is extracted using pre-trained CLIP on the top of the VLM. The experiments are conducted on a real robot and Libero with comparison to Octo and OpenVLA.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Claims and Evidence**\n \nQ1: Clarification on \"unseen scenarios\"\n\nThank you for raising this point. In our paper, \"unseen scenarios\" explicitly refer to tasks involving entirely novel objects or novel combinations of objects and spatial configurations not encountered during training. Crucially, our evaluation—both in real-world (Section 4.1, Appendix A.2) and LIBERO simulation benchmarks—requires the model to distinguish the correct target object from visually similar distractors. Our tasks indeed evaluate visual grounding generalization based on language instructions.\n\n**Experimental Designs and Analyses** \n\nQ1: Regarding the fairness of baseline comparisons (DFP-OTTER vs. OpenVLA)\n\nWe appreciate this concern and clarify our intentions:\n1. Purpose of DFP-OTTER Baseline: it demonstrates explicitly that directly passing visual/textual features into a vanilla transformer is insufficient, underscoring the necessity of selective text-aware visual feature extraction for generalization.\n\n2. Compatibility with existing VLM-based VLAs: Our paper introduces a novel VLA architecture (OTTER) specifically designed around text-aware visual feature extraction. Adding such features directly into existing VLM-based models like OpenVLA would necessitate substantial re-training of the base VLM (e.g., Prismatic VLM [1]), exceeding our available computational resources. Exploring this integration is indeed a compelling future research direction, which we explicitly encourage.\n\n[1] Karamcheti, Siddharth, et. al. \"Prismatic VLMs: Investigating the design space of visually-conditioned language models.\" ICML 2024.\n\nQ2: Regarding plausibility of Direct Feature Passing (DFP)\n\nWe included the Direct Feature Passing (DFP) baseline specifically as a minimal ablation to clearly demonstrate the benefit of selective, text-aware visual feature extraction. Introducing additional intermediate methods like MLPs or Q-formers within the VLM would divert focus from our primary contribution, which is explicitly leveraging pre-trained CLIP for improved generalization.\n\nQ3: Clarification on experiments involving truly unseen objects\n\nOur experimental setup explicitly uses entirely novel objects, as detailed in Appendix A.2, Table 6. These unseen objects differ not only in color but also in geometry and texture. Each evaluation includes explicit distractors, ensuring robust assessment of the model’s visual-language grounding capability. OTTER's superior performance demonstrates effective grounding on genuinely novel objects.\n\n\nQ4: Availability of supplementary video material\n\nSupplementary videos demonstrating task complexity and model performance are now available at our anonymous website: https://ottericml.github.io/\n\nQ5: CLIP vs. other VLM alignment methods\n\nWhile existing VLM-based approaches (e.g., OpenVLA, Pi0-Fast) **implicitly** learn alignments between text and visual tokens through fine-tuning their respective large-scale pre-trained models (such as Prismatic or PaliGemma), our method **explicitly** formulates text-aware visual token extraction as a separate, parameter-free process leveraging the frozen pre-trained CLIP model.\nThus, the comparison requested by the reviewer is already covered by our existing baseline evaluations against state-of-the-art methods such as OpenVLA (Tables 1–3) and Pi0-Fast (added in this rebuttal). These comparisons illustrate the advantage of the explicit, selective token extraction approach compared to implicit token alignment methods embedded in other VLM-based approaches.\n\nQ6: Suitability of LIBERO for simulation evaluation\n\nWe selected LIBERO explicitly due to its usage in recent state-of-the-art VLA works (OpenVLA, π0-Fast-Droid), facilitating fair and direct comparisons. LIBERO supports evaluations involving explicit distractors under varied language instructions, thus providing meaningful assessments of visual-language generalization. \n\n**Essential References Not Discussed**\n\nQ1: Discussion of embodied CoT and CoT-VLA works\n\nWe appreciate this suggestion and will explicitly include and contrast these methods in the updated related work section of the revised manuscript.\n\nQ2: Additional comparison with π0 / its extensions\n\nWe include additional comparisons and discussions with π0-Fast-Droid in response to reviewer Yr2M, the current state-of-the-art VLA model, leveraging extensive pre-training on proprietary robot data. \n\n**Weaknesses**\n\nW1: Dependence on CLIP and complexity of real-world scenarios\n\nWe agree that evaluating more complex real-world scenarios is valuable. However, current evaluations (please see updated tables on the website) already demonstrate significant challenges for state-of-the-art models (OpenVLA, Octo, π0-Fast-Droid), validating our approach’s robustness. Exploring tasks with more complex interactions remains an important future direction, explicitly mentioned in the revision.", "claims_and_evidence": "Although OTTER claims to \"preserve the generalizability of VLMs for effective performance in unseen scenarios,\" the definition of \"unseen scenarios\" remains unclear.", "methods_and_evaluation_criteria": "The method makes sense, but the experiments are problematic; further details are provided in the following section.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "1) The baseline DFP-Otter is not a direct comparison with the base VLA. Typical VLA models use a pre-trained VLM, such as OpenVLA, and do not employ an attention pooling layer to obtain independent tokens. A fairer comparison would involve using the same OpenVLA architecture and simply adding text-aware visual features to demonstrate effectiveness, rather than building an entirely different model.\n\n2) The DFP does not appear to be a plausible architecture for passing visual features to a transformer. Other common methods in VLM—such as using an MLP (LLaVA), q-former (BLIP), c-abstractor (CVPR'24), or feature pyramid extractor (DVCP, CVPR'24)—should be considered.\n\n3) While Otter is claimed to handle many \"unseen tasks,\" the tasks presented in the appendix (real-world tasks) mainly involve attributes like object color or objects already seen during training. Since CLIP can effectively handle these cases, it would be more informative to assess its performance on truly unseen objects—for example, whether it can preserve correct visual tokens.\n\n4) The supplementary material does not include video, which leaves unclear how challenging the tasks are and whether the evaluation is robust.\n\n5) Although CLIP is used as a prior for selecting text-aware visual tokens, other methods, such as VLM, could serve the same function. A comparison with VLM would add value to the study.\n\n6) Libero represents a relatively simple simulation task. Given the emphasis on \"unseen tasks\" in this work, alternative simulations—such as a language table—might be more suitable for evaluation.", "relation_to_broader_scientific_literature": "Vision-language-action models have emerged as a popular domain in imitation learning for robotics. This paper presents a straightforward technique to enhance VLA models.", "essential_references_not_discussed": "The central idea behind Otter is to leverage text-aware visual features to enhance the effectiveness of VLA. However, several existing VLA models—for example, pi0—have not been discussed. Additionally, pertinent works that employ a chain-of-thought approach in VLA, such as embodied CoT and CoT-VLA, are also omitted. Given that using text-aware visual features contrasts with these methods, it is crucial to include them in the discussion.", "other_strengths_and_weaknesses": "Other Strengths:\nThe paper is well-written, and its idea is both straightforward and intuitively effective.\n\nWeaknesses:\nThe experimental results are underwhelming. As mentioned earlier, OTTER is highly dependent on the reliability of CLIP, and it would be beneficial to see experiments conducted in more complex real-world scenarios.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P82c658IdM", "reviewer": "Reviewer_8GXo", "summary": "OTTER is a Vision-Language-Action (VLA) model that enhances robotic task execution by leveraging the semantic alignment capabilities of pre-trained Vision-Language Models (VLMs) without fine-tuning. By extracting text-aware visual features aligned with task instructions, OTTER preserves the rich visual-language understanding from pre-training, enabling effective handling of novel objects and environments. It demonstrates strong zero-shot generalization, outperforming state-of-the-art VLA models on unseen robot manipulation tasks. Empirical results show that OTTER's performance scales with larger pre-trained vision-language encoders, increased policy network capacity, and pre-training on larger robot datasets, making it a robust solution for diverse and unseen robotic scenarios.\n\n## Update after rebuttal\nAfter reading all reviewers' comments and authors' responses, some of my confusion about technical details are explained. However, in comparison with previous work in LLaVA-style, author select only 1 token, such as CLS token, to keep the same with OTTER's setting, but it's not the setting of OpenVLA, so that I think there is a promblem of comparing with OpenVLA.\nOpenVLA builds their framework based on large language model such as LLaMA2, but OTTER is based on a Transformer-based policy model. Although OTTER has better performance than OpenVLA in experiments, OTTER's capacity of dealing with in-the-wild instructions are questionable, and maybe OpenVLA is not suitable to be a baseline of OTTER.\n\nOverall, OTTER designed a novel framework to extract and merge multi-modal features in VLA tasks efficiently and obtained an impressive performance in several tasks. Therefore, I will keep my rating unchanged.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: \"Effectiveness of text-aware visual feature extraction are not well explored.\"\n\nWe agree that explicitly isolating the impact of our proposed text-aware visual feature extraction is important. The ablation study (DFP-OTTER baseline in Tables 1 and 3 and Appendix D, Table 9) was designed to illustrate the benefit of explicitly selecting text-aware visual tokens compared to directly passing visual features to the policy transformer. Specifically, the results demonstrate that selective text-aware feature extraction substantially outperforms this direct-feature-passing baseline on unseen scenarios. \n\nQ2: \"Although the improvement in visual features are reasonable, results in Table.1 shows that embodied features have a large effect on overall performance. Thus, it's quite confusing whether the visual features are the key in this kind of tasks.\"\n\nWhile embodied features (proprioceptive information such as robot end-effector pose) significantly contribute to overall performance—given that robotic tasks inherently require spatial grounding—the visual features remain fundamentally critical. If the policy is provided with only embodied features without meaningful visual inputs, the robot cannot perceive objects at all, yielding a zero success rate. Furthermore, pi0 also takes the embodied features. As shown in the additional results in response to reviewer Yr2M, OTTER outperforms pi0, indicating the significance of text-aware visual features.\n\nQ4/W1: \"Mainstream VLMs, such as LLaVA, usually freeze CLIP encoder during training rather than finetuning. Another baseline using linearly projected frozen CLIP features shall be included.\"\n\nThank you for raising this important point. We address this from two angles:\n\n1. To investigate the effectiveness of simply using frozen CLIP features with linear projection, we conducted a specific ablation using the CLS token from the CLIP encoder as the only visual representation (see Appendix D, Table 9). For convenience, we list the relevant results here:\n| Method                      | Unseen Tasks |\n|-----------------------------|--------------|\n| OTTER (CLS token only DFP)  | 6% ± 0.8%    |\n| OTTER (Ours)                | 62% ± 4.2% |\n\nThese results show that using just the CLS token with a linear projection under the OTTER formulation—similar to mainstream frozen-encoder approaches like LLaVA—is insufficient for achieving generalization in robotic manipulation tasks.\n\n2. Furthermore, unlike mainstream VLM architectures such as LLaVA, prior state-of-the-art VLA models (e.g., OpenVLA) explicitly demonstrate that fine-tuning the visual encoder significantly improves robotic performance. As reported by OpenVLA (Section 3.4 and Table 1), fine-tuning the vision encoder leads to notably higher success rates compared to using a fully frozen vision encoder. This suggests that directly adopting the mainstream frozen-encoder approach from general VLM literature is not optimal for robotic applications, and our explicit text-aware feature extraction provides an effective alternative solution.\nWe will clearly articulate both points in our revised manuscript to thoroughly address this comparison.\n\nW2: \"Beyond text-aware visual feature extraction, visual features are further compressed before input into policy network. Details of this step are not well presented and analyzed.\"\n\nThank you for pointing this out. We compress visual features using a learnable attention pooling operation to produce a compact representation, as briefly described in Section 3.2. Specifically, the text-aware visual feature extraction step outputs multiple (m) text-aligned visual tokens (empirically, we used the first 15 text-tokens, which sufficiently covers the instructions) each of dimension 768. Directly concatenating these tokens would result in a prohibitively high-dimensional input (15 × 768) for the policy transformer, significantly increasing computational complexity. Therefore, to mitigate this issue and maintain computational efficiency, we introduce a learnable attention pooling layer to aggregate these visual tokens to reduce the dimension before passing them into the policy. This attention pooling layer has N_q = 4 query tokens and a latent dimension of 64. We further concatenated these 4 tokens into 1 token.\nWe will clarify and expand this step in the revised manuscript, explicitly discussing both the motivation behind the dimensionality reduction and the details of the attention pooling mechanism.", "claims_and_evidence": "Otter use a novel parameter-efficient text-aware vision feature extraction scheme in VLA model, rather than commonly used finetuning or projection. By selectively use visual features, visual information can be extracted and used efficiently. However, experiments and ablation studies mainly focus on effects of embodied encoder or pre-trained CLIP encoder. Effectiveness of text-aware visual feature extraction are not well explored.", "methods_and_evaluation_criteria": "The method part mainly focus on visual feature processing to solve problems in utilizing CLIP features in existing VLMs. Although the improvement in visual features are reasonable, results in Table.1 shows that embodied features have a large effect on overall performance. Thus, it's quite confusing whether the visual features are the key in this kind of tasks.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Experiments presented by this work is quite impressive, which includes environments both in simulation and real world. Also, generalization ability on unseen tasks are well tested. But in my opinion, ablation study lack of the analysis of claimed text-aware visual feature extraction. Details of DFP-OTTER can be shown and compared in supplementary.\nMeanwhile, mainstream VLMs, such as LLaVA, usually freezes CLIP encoder during training rather than finetuning. Another baseline using linearly projected frozen CLIP features shall be included.", "relation_to_broader_scientific_literature": "Otter achieves impressive performance both in simulation and real world environments, trained and unseen tasks. The benchmarking setting is quite important in the field. And it highlights the importance of well-pretrained vision encoder in VLA models.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength:\nOtter adopts current VLM paradigm and explored into VLA model. By designing a selective visual feature extraction scheme, Otter achieves impressive exprimental results.\n\nWeakness:\n1. Baseline of linearly projected CLIP features shall be compared.\n2. Beyond text-aware visual feature extraction, visual features are further compressed before input into policy network. Details of this step are not well presented and analysed.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MNLNA9WdwL", "reviewer": "Reviewer_Yr2M", "summary": "This paper aims to address a critical issue wherein the simultaneous fine-tuning of visual and language encoders during VLA training significantly exacerbates the risk of overfitting and impairs their original perceptual capabilities. To overcome this, the authors propose an explicit alignment strategy, which utilizes language features to guide the extraction of visual features, thereby obviating the need for encoder fine-tuning. Experiments on downstream tasks demonstrate the superiority and effectiveness of this proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*The authors, however, did not compare their approach with more recent baseline methods like Pi-0.*\n\nWe thank the reviewer for pointing this out. We have added experiments comparing OTTER with Pi-0 as below. \n| Method | Pouring | Drawer | Poking | Pick and Place | Mean±Std. Err. |\n|-----------|-----------|----------|----------|----------|----------|\n| π0-Fast-Droid | 0% | 0% | 0% | 61% | 29% ± 3.5% |\n| Finetuned π0-Fast-Droid | 0% | 45%|  27% | 51% | 35% ± 3.8% |\n| OTTER-OXE-L | 77% | 75% | 93% | 75% | 77% ± 3.3% |\n\nSpecifically, we consider the Pi-0 Fast [1] model trained on Droid. Since our experiment setup is the droid setup, we first evaluate the pi-0 performance directly on our setup, denoted as $\\pi_0$-Fast-Droid. As shown in the table, while $\\pi_0$-Fast-Droid achieves decent performance on the pick and place primitives, it fails on all the other three primitives as the majority of the Droid dataset is the pick and place primitive. Therefore, we also consider fine-tuning $\\pi_0$-Fast-Droid on the DS-ALL, denoted as Finetuned $\\pi_0$-Fast-Droid. Finetuned $\\pi_0$-Fast-Droid achieves non-zero success rate on Drawer and Poking primitives, but still fails on the pouring primitive. There is also a degraded performance on the Pick and Place primitives for Finetuned $\\pi_0$-Fast-Droid. This highlights the difficulty of our experiment setup. OTTER-OXE-L can achieve high success rate on all four primitives on the same amount of demonstrations, indicating that using text-aware visual features extracted from a pre-trained VLM can increase the data efficiency and enhance the generalization ability.\n\n[1] Pertsch, Karl, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. \"Fast: Efficient action tokenization for vision-language-action models.\" arXiv preprint arXiv:2501.09747 (2025).\n\n*I noticed that the tasks evaluated are predominantly simple pick-and-place scenarios.*\n\nWe have primitives other than the pick-and-place as shown in Table 2, where we consider 4 primitives, pouring, drawer, poking and pick-and-place. We have provided the videos for the experiments on https://ottericml.github.io/ to show the diversity of the evaluation tasks.\n\n*The authors could include a discussion on related VLA works that do not freeze encoders, such as CogACT.*\n\nWe thank the reviewer for pointing out this reference. We appreciate and agree with the reviewer’s discussion on CogACT and will include this in the related work of the revised draft.\n\n*Have the authors observed task-level generalization of their proposed method, and how does it compare with contemporary approaches?*\n\nIn OTTER, we define primitive as a fundamental, reusable action or skill that can be applied across multiple tasks when combined with different objects. For example, picking up and pouring are two different primitives, picking up an apple and picking up a banana are two different tasks but the same primitive. OTTER has shown strong task-level generalization as shown in Table 1 and 2. We assume the reviewer is referring to primitive-level generalization. \n\nAs the reviewer pointed out earlier, keeping a frozen pre-trained Visual-Language Model (VLM) primarily enhances visual generalization. In OTTER, our focus is on leveraging this strength to improve object-level generalization. While OTTER does not explicitly target primitive-level generalization, it can bring certain benefits in enabling it. By reducing the burden of generalizing to unseen objects, the model can allocate more capacity to learning task primitives and compositional behaviors, which enables more efficient adaptation to novel primitives. We agree that exploring primitive-level generalization is a compelling future direction, and we explicitly encourage work in this area.", "claims_and_evidence": "The experimental results presented in the paper somewhat support its claims.", "methods_and_evaluation_criteria": "Intuitively, I also believe this straightforward approach can indeed yield performance improvements.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "In Table 2, several baseline methods, such as OpenVLA and Octo, exhibit near-zero success rates. The authors, however, did not compare their approach with more recent baseline methods like Pi-0. Thus, although the proposed method achieves noticeable improvements, the poor performance of baselines may stem from architectural limitations rather than solely from encoder-related issues.", "relation_to_broader_scientific_literature": "I believe the primary contribution of this paper is highlighting the importance of freezing the original modality encoders, rather than jointly fine-tuning them, as many concurrent VLA studies have done. Joint fine-tuning of encoders tends to induce severe overfitting, which explains why such VLA methods typically exhibit poor generalization performance. This paper introduces and empirically validates an approach that avoids fine-tuning encoders, effectively demonstrating its benefit in improving generalization.", "essential_references_not_discussed": "The authors could include a discussion on related VLA works that do not freeze encoders, such as CogACT, to illustrate the potential limitations of encoder fine-tuning approaches. Specifically, methods like CogACT jointly fine-tune visual and linguistic encoders, which could lead to severe overfitting risks. This encoder fine-tuning can cause the models to lose their original perceptual and semantic generalization capabilities, making them overly specialized to specific training tasks or datasets. By analyzing such limitations, the authors could clearly demonstrate why freezing modality encoders is crucial to preserving their intrinsic representational abilities and achieving better generalization across tasks.", "other_strengths_and_weaknesses": "Overall, the authors present a simple yet effective method that emphasizes the necessity of preserving original encoder capabilities. This insight is valuable to the community, as it highlights a critical consideration often overlooked in current VLA research. Although the proposed method itself is straightforward and additional baseline comparisons would further strengthen the findings, I am currently inclined toward acceptance given its conceptual clarity and practical contribution.", "other_comments_or_suggestions": "NA", "questions_for_authors": "Have the authors observed task-level generalization of their proposed method, and how does it compare with contemporary approaches?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D9kyMRJlmM", "reviewer": "Reviewer_Gn1m", "summary": "The authors propose a novel Vision-Language-Action robotic policy called OTTER. Motivated by the computational burden that fine-tuning VLMs for robot policy use entails, as well as by observations that fine-tuning VLMS for action generalization risks weakening the pre-trained vision-language alignment, the authors propose freezing vision and language encoders, extracting task-relevant visual features guided by language instructions. They compare OTTER to previous methods Octo and OpenVLA through experiments on Libero and in a real world setting, and ablate several aspects of OTTER in the real world setting.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*W1: OTTER .. performs on par with them on the synthetic Libero benchmark...*\n\nThe Libero benchmark consists of simple tasks with fixed scenes, distractors, and minimal object variation. As a result, all methods can achieve comparable performance by fitting well to the benchmark’s constraints. Since CLIP is predominantly trained on real-world data, its vision encoder is more effective in real settings than in synthetic ones.\n\nIn our real-world experiments, OTTER was evaluated on four primitives, unseen objects, and novel distractors. Experiment videos are available at https://ottericml.github.io/, showcasing OTTER's ability to generalize across diverse environments.\n\n*W2: Several of OTTER’s design decisions are not ablated, namely: whether to use Xattn vs Xout from the CLIP vision encoder, Nq=4, and context length set to 12.*\n\nWe have included the ablation experiments on using X_out. Specifically, we trained OTTER using the X_out from CLIP on the DS-PnP and evaluated on the pick and place tasks. The results are shown in the table below:\n| Method        | Unseen Pick and Place Task Mean ± Std. Err. |\n|--------------|--------------------------------------------|\n| OTTER X_out  | 18.6% ± 4.3%                              | \n| OTTER        | 62% ± 4.2%                                |\n\nThese results demonstrate that using X_attn​ significantly improves real-world performance, aligning with findings from ClearCLIP and Appendix Figure 6. Leveraging  X_attn enhances OTTER’s generalization in real-world settings compared to using X_out. We also conducted ablations in simulation to assess the impact of context length and N_q​ on performance in Libero-object tasks. The results are shown below:\n\n| Context Length | Success Rate       |\n|----------------|--------------------|\n| 4              | 90% ± 1.2%         |\n| 8              | 90% ± 1.4%         |\n| 12             | 89% ± 1.2%         |\n| 16             | 87% ± 1.3%         |\n\n| N_q | Success Rate       |\n|-----|--------------------|\n| 2   | 87% ± 1.4%        |\n| 4   | 89% ± 1.2%        |\n| 8   | 88% ± 1.3%        |\n\nOverall, our ablation studies support the idea that OTTER’s success is not solely dependent on hyperparameter tuning for a specific setting. Instead, key design choices, such as using X_attn play a more significant role in improving generalization to real-world tasks.\n\n\n*Q1*:\nWe define primitive as a fundamental, reusable action or skill that can be applied across multiple tasks when combined with different objects. For example, picking up and pouring are two different primitives, Picking up an apple and picking up a banana are two different tasks but the same primitive.\n\n*Q2*:\nFor an image input of size `224 × 224`, the CLIP vision encoder initially generates **257 tokens** (256 from `16 × 16` patches plus 1 CLS token). We extract `N_q = 4` text-relevant visual features from `X_attn` of size `15 d`, and concatenate them into a single token.  `d` is the latent dimension. The computational complexity of processing these features is approximately `O(4 × 15 d)`. A standard Transformer’s attention mechanism has a complexity of `O(T²d)`, where `T` is the number of input tokens. For full-resolution image tokens (`T = 257`), the computational cost is `O(257²d)`.  Thus, the ratio of computational cost when using compressed visual tokens compared to processing all 257 tokens is:  `(O(4 x15d) + O(d))/O(257^2 d)  = 0.001`. This demonstrates that our approach reduces the computational cost to approximately **0.1%** of the full-token processing cost, leading to significant efficiency gains in both training and inference.\n\n*Q3*:\nThe action horizon length H=8 is the parameter used in the receding horizon control [1] for action smoothness, as in [2]. For a number of predicted actions of 12, only the first 8 numbers are used for temporal ensembles to calculate the action to execute. \n\n[1] Mayne, D. Q., & Michalska, H. (1988, December). Receding horizon control of nonlinear systems.\n\n[2] Chi, Cheng, et al. \"Diffusion policy: Visuomotor policy learning via action diffusion.\" \n\n*Q4*:\nAs in A.2, all models except OpenVLA are given a maximum of 30 seconds per trial. Due to OpenVLA's larger size and slower inference speed, it is allotted 60 seconds.\n\n*Q5*:\nOcto has 93 million parameters and an inference speed of 33 Hz. It was trained for 300k steps in 14 hours using a TPU v4-128 pod. Fine-tuning Octo on the DS-ALL dataset for 40k steps takes 6 hours using 8 A100 GPUs. OpenVLA, with 7 billion parameters and a much lower inference speed of 0.5 Hz, was trained for 14 days (21,500 A100-hours) using a cluster of 64 A100 GPUs. Fine-tuning OpenVLA on DS-ALL for 15k steps takes 15 hours on 8 A100 GPUs. OTTER, the smallest with 25.5 million parameters and the highest inference speed of 50 Hz, was pre-trained on the OXE dataset for 40k steps in 12 hours using 4 A100 GPUs. Fine-tuning on DS-ALL for 40k steps takes 6 hours on 8 A100 GPUs.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "OTTER offers an alternative to heavy fine-tuning of VLMs/LLMs for robotic policy learning.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\nS1: Technical Novelty: The idea of reducing the number of visual tokens input to the model is important for computational efficiency, and OTTER’s use of existing information from CLIP’s features to extract the most important visual tokens is elegant.\n\nS2: Real world experiments: The real world experiments are well-designed and thorough. It is often difficult to get robot policies to work in real-world settings, as shown by Rows 1-2 in Table 2, so OTTER’s strong performance is very promising.\n\nWeaknesses:\n\nW1: Generalization: OTTER vastly outperforms Octo and OpenVLA on the real world experiments but only performs on par with them on the synthetic Libero benchmark. My concern here is that OTTER is tailored to the experimental real world set up and will not see the same boosts in performance on other environments.\n\nW2: Ablations: Several of OTTER’s design decisions are not ablated, namely: whether to use $X_{attn}$ vs $X_{out}$ from the CLIP vision encoder, $N_{q} = 4$, and context length set to 12. My concern is that these design decisions/hyperparameters are tailored to the real world setting or to Libero, and whether it’s necessary to tune these to make OTTER generalize well to a different environment.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. l. 288 col. 2: What is a “primitive”?\n2. What are the computational savings of using a fraction of the visual patch tokens?\n3. Why is there a difference between the action horizon length and the number of predicted actions?\n4. Can you describe the time limit for task termination?\n5. Can you give a comparison of the number of model parameters, training time, and inference time for OTTER, Octo and OpenVLA? This is important for assessing usability.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces OTTER, a novel Vision-Language-Action (VLA) model that addresses the issue of fine-tuning pre-trained vision-language models (VLMs) for robotic tasks. OTTER aims to preserve the rich semantic understanding of VLMs by extracting text-aware visual features without requiring fine-tuning, achieving strong zero-shot generalization capabilities across novel objects and environments.

Reviewers have highlighted several strengths, including the novelty of the method and the promising real-world experimental results. The approach of leveraging pre-trained CLIP features for selective, task-relevant visual feature extraction is considered a key contribution, especially in light of its efficiency gains and strong performance on unseen tasks.


Overall, the paper offers a compelling approach with strong results but could be improved with additional experiments and a more thorough exploration of some technical details. The AC agrees with the reviewers to accept the submission.
Decision: Accept (poster)