Language-Instructed Vision Embeddings for Controllable and Generalizable Perception
Authors: Chengzhi Mao
Xudong Lin
Wen-Sheng Chu
Venue: ICLR 2026 Poster (2026)
Keywords: Vision Language Model
Primary area: foundation or frontier models, including LLMs
TL;DR: We revert the paradigm of feeding image into large language models, and show language can guide image encoder to learn more sophisticated features and combat hallucinations.

Abstract: Vision foundation models are typically trained as static feature extractors, forcing the burden of task adaptation onto large downstream models. We propose a different paradigm: instead of solely feeding visual features into language, we use language itself to dynamically guide the vision encoder. Our method, Language-Instructed Vision Embeddings (LIVE), leverages language as high-level guidance to produce task-centric embeddings at inference time—without requiring task-specific retraining. This enables the encoder to focus attention on contextually relevant aspects of the input, yielding more controllable and generalizable representations. Empirically, LIVE reduces visual hallucinations (+34 points on MMVP), outperforms vision–language models with orders of magnitude more parameters on visual question answering, and generalizes to unseen instructions and  tasks---offering a direct path toward adaptive, instruction-driven visual intelligence.

Published as a conference paper at ICLR 2026
LANGUAGE-INSTRRUCEED VISION EMBEDDINGS FOR CONTROLLABLE AND GENERALIZABLE PERCEPTION
Chengzhi Mao, Xudong Lin, Wen-Sheng Chu Google {czm, xudonglin, wschu)@google.com
ABSTRACT
Vision foundation models are typically trained as static feature extractors, placing the burden of task adaptation onto large downstream models. We propose an alternative paradigm: instead of solely feeding visual features into language mod- els, we use language itself to dynamically guide the vision encoder. Our method, sanguage-Instructed Vision Embeddings (LIVE), leverages language as high-level guidance produce task-centric embeddings inference time, removing the need for task-specific retraining. This enables the encoder to focus on contextually relevant aspects of the input, yielding more controllable and generalizable represen- tations. Empirically, LIVE reduces visual hallucinations (+34 points on MMVP), surpasses vision- Lasion-language models with orders of magnitude more parameters on vi- sual question answering, and generalizes to unseen instructions and tasks offering a direct path toward adaptive, instruction-diig driven visual intelligence.
INTRODUCTION
hallmark of human vision is its active, selective nature. Guided by internal goals task demands, we focus on relevant parts of the visual world while ignoring distractions (Posner) 1980 Desimone al. 1995). When searching for specific object or understanding particular interaction, humans implicitly "know" where and what to look for. In contrast, today's ' leading vision models, despite producing powerful general purposs features (Oquab et al.. 2023 Zhai et al.) 2023 Yu al. [022] Radford et al. 2021). lack this dynamic, intention-driven adaptability. Their representations are typically static, pre-computed without reference to the specific query they are meant to serve.
This limitation is particularly acute in vision- language models. Existing approaches, such visual prompting (Bahng et al., [222 Shtedritski et al. 2023) or fine-tuning (Mao et al. 2022). offer limited adaptability; because they are optimized for specific target tasks, they fail to interpret zero-shot language instructions. Dominant LLM-centric architectures (Liu et al. 2023 Grattafiori et al. 2024; al. 2022: Team 2024) delegate language integration to large downstream modules, incurring high computational cost while being unable to recover fine-grained details overlooked by the vision encoder, often resulting in hallucinations (Tong et al.) 2024). Recent attempts modulate vision encoders with paired captions (Lavoic et al., 2024 Xiao et al. 2025) are restricted by their reliance on descriptive text rather than true instructions, limiting their versatility and controllability. Thus, the central challenge remains: how to embed language-driven control into the vision encoder to yield adaptive, aaske representations.
We address this challenge with LIVE Language-Instrueted Instructed Vision Embeddings). a simple and effective framework for creating language-steered vision embeddings. LIVE enables dynamic, fine-grained control of a vision encoder by training it to follow textual instructions. Concretely, we use a large language model (LL.M) as the knowledge base to generate synthetic instruction resputteioo-rsooseee pairs, which we combine with images into triplets. By teaching the vision encoder to steer embeddings based on textual prompts, the model can highlight task-relevant features while suppressing unrelated ones. For instance, in Figure I instructing the model to focus on 'fuiit type' allows it to ignore typographical attacks, ensuring robust instruction-folloviin directly within the representation space.
Once trained, LIVE yields standalone. language-steen steered embeddings that downstream tasks directly- no large LLMs or task-specific fine-tuning required. Trained on synthetic ImageNet-basse data, LIVE generalizes strongly real, unseen tasks: it reduces hallucinations by 34 points

---

Published as a conference paper at ICLR 2026
State-of-the-art vision-only (SigLip v2)
Ours
iPod
Apple
Apple
Apple
i iPod
Table
Table
Table
Fence
Fence
Backyard
Backyard
Backyard
Stem
Stem
Stem
Grass
Softmax Prob %
Grass
Grass
20
40
60
80
What the fruit in the Image? What is the text in the Image?
Language Instructions
LIVE [Language-Instructed) Vision Embedding). We show SoTA foundation model struggle to distinguish between textual content and objects within an image. LIVE enables user- guided attention to specified aspects (e.g., "fruit" v.s. "text"), boosting control and prediction accuracy.
Figure 1:
MMVP (Tong et al. 2024),, and surpasses LLM- counterparts GQA (Hudson & Manning 2019) by points with less than 10% their parameters. We also measure and narrow its gap to its LLM knowledge base by up 49 points across cnoructton-folloinng benchmarks. Attention and retrieval visualizations show precise, instruction-driven control emerging inside the encoder. With up 10 times fewer parameters than LLM-heavy methods, our results suggest that embedding task instructions into the vision encoder-rather than scaling downstream modules-is an efficient path to generalizable, and controllable visual perception. More details and training data are available our project page: papp:://iiveeemeeddiiiie get.hiitth... io/
RELATED WORK
Vision Foundation Models. Recent vision foundation models often use two-tower architectures
and train contrastively with image-text pairs (Radford al. 2021 Zhai al., 2023 Tschannen
et al. 2025 /Zhai et al. (022h). While some approaches jointly optimize contrastive and generative objectives (e.g., CoCa (Yu al. 2022). Mammut (Kuo et al. (023)) or use encoder-based captioning
(e.g., Flamingo (Alayrac et al. 2022). Pali (Chen et al. 2022)), )), the vision embeddings are typically computed independently or language interaction occurs during late fusion transformers. Similarly, methods like Q-former (BLIP-2) (Li et (2022) use intermediate stages and powerful LLM decoders for image-to-ae to-tett tasks, without directly instructing the frozen image encoder with language.
Alternative paradigms like masked image-text modeling (ViLT (Kim et al. 2021 learn alignments are not optimized for retrieval embeddings, therefore they require further finetuning on the target task and cannot perform prediction in zero-shot manner. More recent architectures aim to unify approaches (X-Former (Swetha et al. (024 or leverage LLMs as decoders for richer outputs and supervision (Lin al.) [202]]Liu u et al. 2023 Beyer et al., 2024; Team et al. 2025 Grattafiori et al. 2024, Wan et al.J 2024, I'schannen et al.) 2023 Lin et al. (024). However, these methods still generally do not allow language directly control the vision embeddings and cannot perform the target task via retrieval. Alternative vision only models, like Dino (Oquab et al.) 2023 Caron et al.
2021) and JEPA (Assran al. (023), cannot handle language inputs. BRAVE Kar et al. 2024
ensembles vision encoders for improved accuracy.
Instructed Foundation Models. The growing need for adaptive vasion-language models inspired efforts in fine-tuning (Lin et al.] (023) and prompt engineering (Menon et al.) 2022). However, these approaches typically optimize either the entire model or specific prompts, restricting them single- task adaptations such as rationale explanation (Mao et al., 2023) or category classification (Mao et al. 2022). Methods like (Shtedritski et 2023 Zhong et al. 2022) allow querying visual encoders explicit markers (e.g., red circles or bounding boxes) but fail in scenarios involving overlapping or ambiguous visual concepts, as these markers only specify location without clarifying the targeted attribute (e.g., color, texture). Prior work train top down vision encoder for embodied agent, yet this not zero-shot (Eftekhar et al.) [203)) Magiclens (Zhang et al. 2024) perform self self-supervised image retrieval based instructions, yet it does not provide retrieval semantic, language space, and not ready for direct visual perception.
Multimodal retrieval methods like UnilR (Wei et al. features, our work focuses on guiding the vision encoder itself. which could serve as an enhanced vision component complement such models. (Kar et al. 2024 combines multiple vision encoders
2024) perform retrieval via late-stage fusion of
2

---

Published as a conference paper at ICLR 2026
Vision Input
Vision Encoder E
Embestiog
Feature Matching
Vision Input
Language-Instructed
Feature Matching
Vision Encoder Eine
Text Question
Text
Text Input
Text Encoder T
Encoder
Answer
Enitant
(A) CLIP, SigLip (Baseline)
(B) Ours
Figure 2: Instructive Vision Encoder Design. Prior vision-language models such as CLIP (Radford
et al. 2021] and SigLIP (Zhai et al., 2023) adopt two-tower architecture with separate vision and text encoders. We reuse the text tower encode the query, apply projection layer, and inject into the vision transformer alongside with the input image. Note that although the question and the answer are passed through the same text encoder, they are processed independently with no feature interaction between them. The text encoder remains frozen during training, while only the vision encoder updated (highlighted in yellow). Learnable embeddings are shown in pink.
to obtain better vision representations for language models. Other approaches control vision indi- rectly through post-hoc modification (Chen et al. 2024a) or in specialized domains like document recent trend fine-tuning vision LLMs for retrieval (Wei et al. [0224 al.) [024]] Liu et al. 2025). While powerful, these models inherit the substantial computational footprint of their underlying LLMs. The most related works that also modulate the vision encoder directly typically use image captions the conditioning signal (Lavoie ct al. 2024 Xiao et al., 2025). This strategy risks learning undesirable shortcuts, as the model can minimize loss by simply matching text features rather than learning true visual grounding. Our method, LIVE, explicitly decouples the guidance from the target by using task instructions that differ significantly from the target description. This design forces the model learn
retrieval (Zhou al.) 2024, Chen et al. 2024b).
sophisticated mechanism for instruction-bast based control, enabling precise manipulation of vision embeddings without the inference overhead of large LLMs or the risk of learning trivial solutions.
3 METHODOLOGY: LEARNING LANGUAGE- VISION EMBEDDINGS
Conventional vision-language models treat the vision encoder as a static feature extractor for downstream L.M. We reverse this paradigm by distilling knowledge from an L.M back into the vision encoder. Using synthetic image-query-answer triplets, we train the vision encoder with contrastive learning to produce aunguag--instructd embeddings that align with the answer's s semantics. The result is powerful, standalone vision encoder capable of zero-shot perception, removing the need for a computationally expensive LLM at inference.
3.1 LANGUAGE-INSTRUCTER VISUAL EMBEDDINGS
Standard vision-language models like CLIP (Radford al. 2021 and SigLIP ZZai al. 2023) use a two-tower architecture with vision (-)) and text encoders T(.) (see Figure 2 for an illustration). The vision encoder E' produces general-purpose embedding information from the input image However, to support diverse downstream tasks via text queries, these embeddings must be both versatile and precise. Learning such universal representations challenging due to the inherent capacity limits of the vision encoder.
E(x) designed to capture all relevant
Visual prompting (Bahng et al. 2022 Jia et al. 2022) seeks to adapt visual representations. However, visual perception is often ambiguous and context-dependent. Simple location prompts (e.g., boxes, circles (Shtedritski et al., 2023) )) offer limited control and cannot specify which aspects to focus on. For instance, region might require attention color ("What color?"), make ("What make?"), or condition ("Is it clean?"). Existing methods struggle to handle this kind of semantic ambiguity.
To address this, we propose language-conditione conditioned vision encoder, denoted Elive Instead of fixed embedding, Elive dynamically processes the image x based on the embedding of a textual instruction q. We reuse the pretrained text encoder T(-). Our instructive visual embedding is computed
(T) Fasuee (x, T(q)).
(1)
3

---

Published as a conference paper at ICLR 2026
Question
Answer
promary aciiviy taking image??
persin driving
presenger.
What type of vehicie dispicted in the image?
A vintage snowmoble..
What is the color of the snowmoblee?
The snowmobile predominantly blue and white.
What the
child's jacked?
Orange
What is the adult wearing thair nead?
A hemeet.
What is wisble the background of image??
Houses, trees, a road, and bright sun
What the
number snowmoblee?
542959
What is the color of the mug the center mo imaga?
The mug in tha center is primarlly yollow and brown.
What character featured the yellow mug?
The yellow features Donald Duck
What object behind the mugs?
A coffee maker is behind the mugs.
What is the relativ position of the red the yallo??
The red mua to the left the yellow
any the mugo?
Yes, appears
in yellow mug.
What surface mugs sttiigg
mugs sittigg on
counter.
foreground object image?
foreground ignored, image coffee
Figure 3: Triplet Training Data from LLM. We apply Gemini-2.00Flaas. (Comanici et al., 2025 to automatically generate diversified, open-world triplet data containing image, query, and answer. This method moves beyond generic questions from existing image-text datasets, enabling more nuanced and sophisticated exploration of image-specific content.
This formulation enables the vision encoder to focus on the aspects of the image most relevant to the language instruction, producing a targeted, task-specific representation. Model implementation details are provided in Section
3.2 TRAINING OBJECTIVE
We train the instruction-conditionne vision encoder Eiive by matching its output (44) with the text embedding of the corresponding correct answer a. Specifically, we want our instructed vision embedding _! = Elive ((:)) to be close to the answer text embedding (T) = (ay) if and only if (x;, q:) corresponds to answer aj.
Following (Zhai al., 2023), we employ sigmoid-based alignment loss, which yields better performance and stability than standard contrastive losses (Radford et al. [022]) Given a batch of
image-instruction pairs (x,, q:) and their corresponding answers aj. the loss is defined as:
= -iss
log
+ exp (-g1(t((z))) 2, ( +b) b))
Va E 1} encodes the match of the image-query-anse The parameters (temperature) and b (bias) are learnable parameters for calibration. We optimize the visual encoder Elive by minimizing this loss with gradient descent.
triplet (1 for match, -1 for mismatch).
3.3 KNOWLEDGE TRANSFER FROM LLM
significant challenge in training the instruction-guided encoder Eiiee the scarcity of large-scale datasets with image-instruction answer triplets (x, a.. Existing visual question answering (VQA) datasets (e.g., CC3M-VQX (Changpinyo al.) 2022)) often rely on template-base based
Despite the abundant image text paired data (Schuhmann et al. 2022 Bycon et al. 2022)
deleegenedtedd questions, which may not capture the breadth and complexity of real-world queries needed to probe deeper understanding. Our experiments in Figure (6 shows existing datasets are insufficient for training language-instrrn instructed visual embeddings with high accuracy.
To overcome this data bottleneck, we leverage the extensive world knowledge and reasoning capabili- ties in LLMs. We treat an LLM an implicit knowledge source that can identify salient aspects of an image and generate relevant questions about them. Specifically, we query powerful LLMs that accept visual inputs to generate inestion-answer pairs (q. a) conditioned on the given image. This effectively transfers the LLM's rich understanding from billions of parameters into training data for vision encoder. Crucially, this computationally intensive LLM inference is performed offline
4

---

Published as a conference paper at ICLR 2026
Vision-Only Embedding (Baseline)
conguage-insiructed Vision Embeddiig (Ours)
Is smilng tounge cut?
Input
smiling with tounge out?
Retrieval
minion smiling tounge out
minion smiling with tounge
A minion smiing with tounge
A minion smiling without tounge
Input
+
direction the duck facing?
+
direction the duck facing?
Retrieval
A duck facing left
duck facing left
duck facing left
A duck facing right
Input
How many egge there?
How many
Retrieval
4 eggs
eggs
4 eggs
eggs
Where is rabbit revanive to the eggs?
Where is rabbit relative to eggs?
Input
Retrieval
rabbit the middle
rabbit in middle eggs
rabbit in the middle the oggs
A rabbit the left of eggs
aggs
Are the flowers
Are the flowers above or from side?
Input
+
above or from
the side?
Retrieval
Flowers seon from
Flowers seen from
Flownrrs seen from above
Flawers seen from
X
Figure 4: LIVE Reduces Visual Hallucinations (MMVP Benchmark Tong al. (0024)). State- of-the-art vision-only embeddings Zhai et al. (2023) (left column) encode the entire scene without query-specific guidance, making them prone to hallucination when fine-grained precise details are required. By modulating visual computation with the input text query (right column), our method selectively focuses on relevant information, thereby reducing hallucinations and improving accuracy.
during dataset creation. At inference, only the efficient instruction-guiie vision encoder Eive is required, maintaining real-time computational efficiency for perception tasks.
For each image, prompt LLM generate multiple diverse question-answe answer pairs using the following prompt structure: Provide numbered list of interesting visual questions about the image, followed by the corresponding answers. Figure shows examples of our generated queries and answers on ImageNet, introducing diverse visual attributes and semantic concepts that were previously unavailable. This rich, detailed data enables vision models learn beyond prevalent mageeccapiooning patterns, fostering more effective and fine-grained visual comprehension. Importantly, humans often perform such queried visual tasks using System-1 (intuitive) reasoning. Our LIVE encoder designed to operate with similar efficiency to avoid the computational overhead of larger LLMs, especially visual perception applications. Moreover, when downstream tasks are known before deployment, our text embeddings can be pre-computed and cached to further save computation costs.
4 EXPERIMENT
This section details our experimental setup, benchmarks, baselines, results, and analysis designed to evaluate the zero-shot language controllability enabled by our LIVE approach.
5

---

Published as a conference paper at ICLR 2026
Image Input
Language-Instrucke Vision Embedding Retrieval (Ours)
Vision-Only Embedding Retrieval (Baseline)
What color is the ink of the pen?
The ink color not visible
Redi
Red
color is not visible Red and blue The liptop the lift of the bottle
Yallowish broww
The dog appears to be Basset Hound The plate dark allver color
lieely
35
52.5
70
68
we ignor the cmbbeell, what the main facus the image?
runway, grass, building, sky The flaid would cosist of grassy fiald
umbrela
Smith
catalog, likely advertsing
There next visible, but lorgely unmedablle an umbrelle
Airways
Plying in the sky
37 5 45 52.5 60
60
65
70
75
What baciground,
Dense and tall
Dense and tall Black
There standi upricht dry, hilly semain sparse vegation They are lying horioontally
greenery
Wood and some type of wire or mesh
65.5
50
60.5
67
What color is the motorcycie to the right of the motorcycin highighted the boundiing box?
Is the foreground He sitingg motorgycle
Yellow
(4)
The yellow around catcher It the background
Whitt
Three
Two
Two
No
41.5
40
69
73.5
the beach wearing jackett?
no
gitss
left
young
(5)
Idy
cap
light brown
ye5
68
96.5
60
60
07.5
Figure 5: LIVE'S Retrieval based on Language Instructions. Examples 1-5 show examples from ImageNet, Caltech, SUN, RefCOCO, and GQA, respectively. The instructions provided at inference time are unseen during training and highlighted in red. For both our method and the vision-only baseline SigLIP (Zhai et al. 2023), we show the top-5 retrieved text responses with bars indicating the predicted sigmoid probabilities. Our method demonstrates superior retrieval accuracy, as correctly (I) identifies non-visible elements, (2) follows instructions to ignore specified content, (3) attends to the requested factors, (4) performs basic spatial reasoning, (5) captures relationships between objects.
4.1 EXPERIMENTAL SETUP
Training Data. We train LIVE using ImageNet training images, and apply the data generation process described in Section 3.3 using ImageNet dataset and Gemini 2.0 Flash [ Geemini 2024 Comanici et al.
2025 This yields 16..4 million images-query-ane -nnwer triplets, multiple informative queries are generated per image. We also explore and compare against publicly available triplet data from PaliGemma (Beyer al. (2024) specifically leveraging CC3M-VQA (Changpinyo et al.. 2022). For datasets without explicit instructions, such as WebLI (Wang et al. (022)) and Open Images (Piergiovanni et al. 2022). we introduce a universal instruction, "caption the image".
Evaluation Benchmarks. We target tasks that require explicit language instructions to define the objective, in contrast to conventional benchmarks that rely on static vision encoders for universal zero-shot classification, and thus cannot evaluate cestruction-follwwee label taxonomies. We deduplicate both images and text instructions to ensure all our evaluations data are novel. For all datasets, we report Top-1 retrieval accuracy. MMVP (Tong et al. (2024) is recent benchmark designed to evaluate hallucinations visually similiar images that models are required to attend to the nuances to answer correctly, GQA (Hudson & Manning 2019) is a challenging visual question answering benchmark that extends beyond attribute answering and requires reasoning over scene graphs to answer relational queries.
or capabilities beyond fixed
VLMs (see example in Figure 4). pairs
'maageNet images httpp:: : /www.wwww imaagemeem.... net org/ and Gemini 2.0 Flash API are publicly available. The generated query-answer pairs are provided on our project wehpage.

---

Published as a conference paper at ICLR 2026
Image
Size
Params (M)
Q
13
Average
OpenAl ViT-L-14
Radford
2021
224
427.6
13.3
13.3
20.0
20.0
13.3
53.3
20.0
6.7
13.3
19.3
OpenAl ViT-L-14
Radlord
2021
336"
427.9
0.0
20.0
40.0
20.0
6.7
20.0
33.3
6.7
33.3
20.0
DEN VIT-H-114
Tang 2023
224
986.1
20.0
26.7
73.3
26.7
26.7
66.7
46.7
13.3
53.3
39.3
DEN VIT-H-14
Fang
2023
378
986.7
13.3
20.0
53.3
33.3
26.7
66.7
40.0
20.0
40.0
34.8
MetaCLIP ViT-L-I4
14
Xu al.
2023
224
427.6
13.3
6.7
66.7
6.7
33.3
46.7
20.0
13.3
23.7
MctaCLIP ViT-H-14 14 Xu
2023
224
986.1
6.7
13.3
60.0
13.3
6.7
53.3
26.7
13.3
33.3
25.2
EVA01 ViT-g-14 (Sun 20:2
224"
1136.4
6.7
26.7
40.0
6.7
13.3
66.7
13.3
13.3
20.0
23.0
EVA02 ViT-bigUU- I4+ (Sun al
2023
224"
5044,9
13.3
20.0
66.7
26.7
26.7
66.7
26.7
20.0
33.3
33.3
SigLIP ViT-SO-14 Zhai 2023
224
877.4
26.7
20.0
53.3
40.0
20.0
66.7
40.0
20.0
53.3
37.8
SigLIP ViT-SO-14 Zhai
2023
384
878.0
20.0
26.7
60.0
33.3
13.3
66.7
33.3
26.7
53.3
37.0
InstructBLIP [Dai
2023
336a
144200.
16.7
LLaVa (Liu 2023
336
13000.00
31.3
BRAVE Kar 2024
336"
-00300.0
42.0
IigLIP ViT-SO 14 (Ours)
384
891.0
80.0 76.7 73.3 80.0 83.3 86.7 66.7 66.7
73.3
76.3
Table 1: Zero-Shot Accuracy on MMVP-VLM benchmark Tong et al., 2024). We use bold to highlight the highest accuracy. Baseline methods are vision-only models, with their results quoted from MMVP. Following MMVP, we denote visual patterns as follows. Orientation and Direction, Q: Presence of Specific Features, 3 State and Condition, 1:: Quantity and Count, 9. Positional and Relational Context, Color and Appearance, c Structural and Physical Characteristics, A: Texts, Viewpoint and Perspective. All CL.IP-based methods using vision-only embeddings struggle on this benchmarks. By incorporating instruction- perruction-guided modulation, our method achieves 34-point zero-shot accuracy improvement over prior SOTA methods, highlighting the role of instructions in directing the vision encoder towards relevant signals and reducing hallucinations.
(b) Comparison with SoTA methods.
(a) Performance of our model variants on GQA.
VII Model
SigLIP
Fusion
Menon et al.
Ours
Model
Accuracy (%)
LIPLIP VIT-T-14
9.8
20.0
10.4
60.6
InstraciLL1P
BLIP-2 [Li al 2022
2023
44,7
SigLIP VT--B-16
SigLIP 2 B-6
12.0
14.4
20.8
12.8
13.0
19.8
67.6
71.2
BRAVE Kar a 2024
52.7
SigLIP 2 VIT-S0-14
16.4
19.6
17.8
68.2
LLava Liu ct 2025
63.3
Ours ( V-1---16
71,2
Table 2: Zero-Shot Retrieval Accuracy GQA tasks. We report top-1 accuracy (%).
To quantify the gap between LIVE and its LLM knowledge source, we use Gemini 2.0 Flash to annotate instruction answers on test data from Caltech101 (Griffin et al.] 2007). SUN397 (Xiao et al.) 2010), RefCOCO (Kazemzadeh et al. 2014), and ImageNet (Deng et al. 2009). Under this setup, Gemini 2.0 Flash achieved 100% accuracy by construction. We filter the test data to ensured no instruction overlap between training and test sets, and denote the repurposed datasets as f. Those t datasets intended to measure the fidelity of knowledge transfer from the "teacher" (Gemini) to the "student" (LIVE), rather than benchmark performance on the original target tasks, since those instructions are Gemini-generated.
Baselines. We consider a family of static vision-only embedding baselines and their variants. LigLIP (Zhai al., p We use publicly available SigLIP models up to size SO400M, representing the SoTA static, instruction-agnoste visual embeddings. Fusion: We directly add the image and text query embeddings, and use the combined representation to retrieve the text answer. Menon en al. (Menon& & Vondrick 2022): Following Menon et al., who improved retrieval by augmenting text answers with language descriptions, we append the language instructions to the answer so that the text tower is explicitly informed of the query. We further compare against L.L.-based approach LLaVa (Liu et al.) 2023), late fusion methods InstructBI.IP (Li et al.) 2022). and an ensemble-bane based state-of-the-art method BRAVE (Kar et al. (2024) that combines five generic vision encoders EVA- CLIP-g (Sun et al.) 2023), CLIP-L/I4 (Radford et al.) 2021), SILC-G/16 (Naeem et al. 2024). ViT-e (Chen et al. 2022). and DINOv2-L/14 (Oquab al. 2023 followed by further fine-tuning tuning.
Implementation Details, We initialize LIVE's vision encoder from a pretrained SigLIP and SigLIP. v2 (Zhai et al. 2023; Tschannen et al.) 2025). which outperform CLIP (Radford et al. 2021). All models are based on the transformer architecture (Vaswani et al. (017). We use SigLIP text encoder to precompute fixed instruction embeddings for both training and evaluation. These embeddings
7

---

Published as a conference paper at ICLR 2026
ImageNeti
Caltech 1011
VIT Model
SigLIP
Fusion
Menon et al.
Ours
SigLIP
Fusion
Menon al.
Ours
SigLIP ViT-T-14
25.10
32.46
33.42
73.28
10.53
11.38
22.05
37.08
SigLIP ViT-B-16
30.84
33.23
42.50
86,93
12.08
12.64
24.72
55.75
SigLIP 2 ViT-B-16
37.73
40.69
60.52
86.79
14.89
15.31
29.92
51.97
SigLIP2 VIT-SO-14
38.03
40.40
60.86
87.06
14.61
15.87
33.00
55.05
SUNI
RefCOCO1
VIT Model
SigLIP
Fusion
Menon et al.
Ours
SigLIP
Fusion
Menon
Ours
SigLIP VIT-T-14
6.99
8.87
10.90
33.16
8.52
11.74
11.01
42.73
SigLIP ViT-B-16
9.26
9.75
16.94
49.83
9.84
10.87
12.78
59.32
SigLIP 2 ViT-B-16
12.44
12.96
24.67
49.76
12.04
13.51
17.47
55.95
SigLIP 2 ViT-SO-14
13.00
14.06
25.79
52.94
9.40
10.28
14.98
54.33
Table 3: Closing the gap to the Gemini knowledge source in zero-shot instruction following. We report Top-1 retrieval accuracy on benchmarks t. where Gemini's annotations act as 100% accurate oracle. Our model is evaluated in a strict zero-shot setting, without any fine-tuning on the downstream Caltech 101, SUN, or RefCOCO datasets, unlike prior work (Beyer al. 2024: Kim et al. 2021 lb. All evaluation data are deduplicated from our synthetic training set. Our approach substantially narrows the gap to the oracle, outperforming baselines by up to 49 points.
Baseline
Images
Webli
CCSM
LIVE (Ours)
90%
67.5%
45%
22.5%
ImageNett
Caltech1011
SUNF
RelO0001
VOAr
GOA
ImageNet OCR
Figure 6: Impact of triplet training data on LIVE's accuracy. We train SigLIP v2 VIT-B-16 with four triplet datasets, Open Images, WebLI, CC3M, and ours. Ours achieves broader improvements across benchmarks. While Open Images showed no gain, WebLI increases OCR, and CC3M offered slight improvements on some tasks, our approach highlights the benefit of using LLMs to overcome traditional data limitations for training transferable vision encoders.
projected through single linear layer and injected into the vision transformer. which adds ~ 13M parameters for the ViT-So model. During training, the vision encoder, including the text projection layer, updated, while the original text tower remains frozen. We adopt the optimizer setting as SigLIP (Zhai et al.) 2022a) with learning rate of 0.001, batch size of 8192, and 122k training steps, using 256 TPUv3 cores. As in SigLIP, apply only resize augmentation during training.
4.2 RESULTS
We first evaluate LIVE the MMVP-VLM benchmark (Tong al. 2024). As shown in Table | our model achieves a 34-point accuracy gain over prior methods, including LLM-based and ensemble approaches that are up 10 times larger. Qualitative examples in Figure 4 illustrate how language instructions guide the vision encoder towards task-relevant cues, thereby reducing hallucinations (e.g., incorrectly perceiving minion's tongue). the GQA benchmark, which requires additional reasoning, LIVE outperforms both LLMs and the strongest generic vision models by 7 points, while using 10 times fewer parameters.
We further measure LIVE's accuracy gap its LLM knowledge source, Gemini 2.0 Flash. As shown in Table 3 although a 23-41 point gap to the Gemini oracle remains, LIVE consistently attains Top-1 retrieval accuracy over established vision-only embeddings these targeted tasks, despite considerable domain shifts in both images and query types.
Figure 5 shows visualizations of images, queries, and top-5 retrieved instances, all deduplicated from the training set. Our LIVE model exhibits emergent capabilities beyond its training data. For instance, image (4) shows the model correctly interprets bounding boxes identify the color of
8

---

Published as a conference paper at ICLR 2026
Vision-Only Embedding (Baseline)
Language Instructed Vision Embedding (Ours)
Languagg-Instructed
What the text in the image?
What is the fruit image?
No Instruction.
iPod
iPod
iPod
Input
Attention
Input
Attention
Input
Attention
Figure 7: Zero-Shot Language Instructions Steer Visual Attention. Unlike baseline encoders producing global attention (SigLIP, left), our LIVE uses instructions to focus dynamically. Prompting for "text" highlights the "iPod" label; prompting for "fruit" highlights only the apple, ignoring the label. This demonstrates emergent, instruction-driven control over visual encoding.
Training Groups Testing Groups
SVD
FVD
FSD
FSV
FSVD FSVD
S
V
D
SigLIP ViT-B/16
74.05
82.48
83.40
83.28
86.93
Table Ceave-One-Group-Ouu Generalization. To generalization novel instruction types, we partition our data into four categories: Fundamental Properties (F), Spatial-Textual (S), Viewpoint (V), and Dynamic Reasoning (D). We train the model while holding out each category in turn, demonstrating LIVE's ability to generalize semantically distinct, unseen instructions.
a specified motorcycle, despite never seeing bounding box annotations during training. Similarly, image (1) highlights the model's ability reason about nuanced visual details, such as recognizing that an ink color not visible, rather than defaulting to the image's dominant red color (as seen with baseline vision-only embedding). These examples illustrate fine-grained understanding and contextual reasoning previously unattainable with static vision-only embeddings.
4.3 ANALYSIS
Impact of Training Data. We benchmark model by training it individually on established Vision-language datasets, l.e., Open Images (Piergiovanni et al., 2022). WebLI (Wang et al. 2025). newly constructed Imagenet triplet dataset.
CC3M-VQA (Changpinyo al. (022), and
For datasets lacking explicit textual queries (unlike CC3M-VQA, which provides rulc-extracted query-answer pairs), we employed generie queries (e.g., "caption the image") ensure a comparable training setup. As shown in Figure models trained with our Imagenet triplet dataset significantly outperform those trained on existing image-language datasets across diverse benchmarks. These results indicate that the lack of large-scale, diverse, and targeted image-query-amme answer data has been major bottleneck for advancing instruct-aware vision embeddings. While prior work typically freezes vision model and improves the LLM, we reverse this paradigm and show that LLMs can be instead leveraged to more effectively train vision models.
Impact of Vision Encoder Size. As detailed in Table we scale the vision encoder from VIT-T (5.4M parameters) to ViT-B (86.6M) and SO400M (89 (89IM). Although performance improves with model size, the compact ViT-T model still attains competitive accuracy, indicating its potential for deployment on resource-constraingge edge devices.
Attention Map Visualization. Figure visualizes how language instructions modulate visual attention. We plot heatmaps of the attention from our language input token to the visual tokens. We contrast baseline vision-only transformer LIi ViT-SO-14, left) with ours (right). Given the same

---

Published as a conference paper at ICLR 2026
ViT-B
GQA
MMVP
Layer
67.4
69.5
Layer 4
67.8
69.4
Layer 8
68.2
68.7
Table 5: Impact of Language Injection Depth (ViT-B): Late injection favors GQA, implying relation-heavy tasks require higher-level semantics, while early injection better reduces visual halluci- nations MMVP. Ultimately, no single insertion point universally optimal across all benchmarks.
Method
Text Query (Vision Input)
Text Target
GQA
MMVP
Ablation
Neutral ("Caption the image.")
Rich Answer
13.1
65.1
Ablation 2
Specific ("What is the category of the image?")
Class Name
2.7
54.7
LIVE (Ours)
Specific Query
Rich Answer
67.4
69.5
Table 6: Impact of Text Guiding Vision Representation. We change the text query and target to isolate the effects of instruction specificity and supervision granularity.
input image (e.g., an apple labeled as "iPod"", the baseline's s attention remains instruction-agnostic, as it does condition instructions. In contrast, LIVE dynamically adjusts its focus: when instructed to find "the text", attention focuses on the "iPod" label; when asked to identify "the fruit", attention localizes on the apple. This demonstrates that LIVE learns to steer its visual processing according to the language query, enabling focused computation on ceetuuctioo-relevan regions.
Generalization Out-of-Distribution (OOD) Instructions Groups.
To conduct stricter gen- eralization test, we train and evaluate our model on semantically disjoint instruction groups. This introduces larger distributional shift than the deduplication strategy used in prior experiments. As shown in Table the model maintains strong performance even under this challenging OOD setting.
Impact of Language Injection Depth. We evaluate the effect of injecting language tokens at different depths (Layer 1, 4, and 8) of the ViT-B encoder. As shown in Table early injection (Layer 1) yields the highest performance on MMVP (69.5), where preserving fine- fine-grained visual details is critical for detecting hallucinations. In contrast, late injection (Layer 8) performs better GQA tasks benefit from higher-level semantic abstraction. These
(68.2), suggesting that results indicate language tokens actively modulate the visual features different processing stages.
Impact of Text in Guiding Vision Representation. To validate that instruction conditioning- not simply data scale- drives performance, we ablate the encoder input types (Table (). Replacing LIVE'S specific queries with neutral prompts (e.g.. "Caption the image") collapses GQA accuracy from 67.4 to 13.1, showing that vision encoders require targeted, query-driven guidance. Moreover, replacing rich, descriptive answers with standard class labels drops GQA performance to near-random levels (2.7). Together, these results confirm that LIVE's gains stem from explicit language-to-yee -toviiion instruction conditioning and descriptive supervision, rather than from the backbone architecture or dataset scale alone.
5 CONCLUSION
We introduce new paradigm for vision representation: instructing the vision encoder with knowledge from language models. Unlike conventional approaches that freeze generic vision encoder, we show injecting task-specific guidance directly into the visual system provides substantial benefits. Our method produces efficient, lightweight encoder that improves perceptual precision and mitigates hallucinations without costly retraining. Our findings suggest that advancing vision models on targeted tasks relies not only scaling, but also making them instruction-aware.
ACKNOWLEDGEMEENT
We would like to thank Guangxing Han for their invaluable insights and discussions, and Longgi Cai for crucial infrastructure support. We are also deeply grateful to Yaojie Liu for executing all the key experiments during the rebuttal phase, and Ahmed Abdelkader for providing feedback for the paper.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our work introduces instruction-aware vision encoders that accept natural-language task speci- fications. While this reduce hallucination and improve task precision, it also raises ethical considerations: Instruction following could be repurposed for harmful objectives (e.g., surveillance, targeted profiling). In our training, do not include any harmful objectives, therefore the risk shall be minimized in our model perspective.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we have provided comprehensive implementation details, network archi- tectures, and hyperparameter configurations throughout the paper and the Appendix. Because the original training codebase is deeply integrated with proprietary internal infrastructure, it cannot be directly open-sourced. Furthermore, the release of the training dataset is currently undergoing internal institutional review. Pending final open-source approval, the dataset will be made publicly available.
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc. Arthur Mensch, Katherine Millican. Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning, Advances in neural information processing systems, 35:23716-237366 2022.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint-embedding predictive architecture. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 15619-15629, 2023.
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan. and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203. 2022.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile vlm for transfer. arXiv preprint arXiv:2407.07726, 2024.
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Back, and Sae-
hoon Kim. C0yo-700m: Image-text pair dataset. attps://qitthu com/ wwt../owoo.kkceaeeeee kakaobraiiaaa/a/aa
coyo-dataset 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF international conference computer vision, pp. 9650-9660, 2021.
Soravit Changpinyo, Doron Kukliansky, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut. All you may need for vqa are image captions. arXiv preprint arXiv:2205.01883, 2022.
Haozhe Chen, Junfeng Yang, Carl Vondrick, and Chengzhi Mao. Invite: Interpret and control vision- language models with text explanations. The Twelfth International Conference on Learning Representations, 2024a.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference Computer Vision, pp. 370-387. Springer, 2024b.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.0677 2022.
Gheorghe Comanici. Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXivv2507.06261, 2025.
11

---

Published as a conference paper at ICLR 2026
Wenliang Dai, Jumnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose with instruction tuning. Advances in neural information processing systems. 36:49250-49267, 2023.
purpose vision-language models
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Robert Desimone, John Duncan, et al. Neural mechanisms of selective visual attention. Annual review of neuroscience, 18(1):193-22) 22292222221122 1995.
Ainaz Eftekhar, Kuo-Hao Zeng, Jiafei Duan, Ali Farhadi, Ani Kembhavi, and Ranjay Krishna. Selective visual representations improve convergence and generalization for embodied ai. arXiv preprint arXiv:2311.04113,, 2023.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309.17425, 2023.
Gemini. Gemini 2.0 flash models vertex ai, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:4407221783 2024.
Gregory Griffin, Alex Holub, Pietro Perona, et al. Caltech-256 object category dataset. Technical report, Technical Report 7694, California Institute of Technology Pasadena, 2007.
Drew Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 6700-6709, 2019.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. European conference on computer vision, pp. 709-727. Springer, 2022.
Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang. Deqing Wang, and Fuzhen Zhuang. E5-v: Universal embeddings with multimodal large language models. arXiv preprint 2.Xx:22477.228800 2024.
Ouguzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari. Brave: Broadening the visual encoding of vision-language models. In European Conference on Computer Vision, pp. 113-132. Springer, 2024.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp- 787-798, 2014.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and languuge transformer without convo- lution or region supervision. In International conference on machine learning, pp. 5583-5594. PMLR, 2021.
Weicheng Kuo, AJ Piergiovanni, Dahun Kim, Xiyang Luo, Ben Caine, Wei Li, Abhijit Ogale, Luowei Zhou, Andrew Dai, Zhifeng Chen, et al. Mammut: A simple vision-encoder text-decoder architecture for multimodal tasks. Transactions on Machine Learning Research, 2023.
Samuel Lavoie, Polina Kirichenko, Mark Ibrahim, Mahmoud Assran, Andrew Gordon Wilson, Aaron Courville, and Nicolas Ballas. Modeling caption diversity in contrastive vision-language pretraining. arXiv preprint arXiv:2405.0074 2024.
Junnan Li. Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping eraguage-imagee pre- training for unified vision- language understanding and generation. In International conference on machine learning, pp. 12888-18 2888-12900. PMLR, 2022.
12

---

Published as a conference paper at ICLR 2026
Xudong Lin, Gedas Bertasius, Jue Wang. Shih-Fu Chang, Devi Parikh, and Lorenzo Torresani. Vx2text: End-to-end learning of video-based text generation from multimodal inputs. In Proceed- ings of the IEEE/CVF Conference Computer Viston and Pattern Recognition, pp. 7005-7015, 2021.
Xudong Lin, Simran Tiwari, Shiyuan Huang, Manling Li, Mike Zheng Shou, Heng Ji, and Shih-Fu Chang. Towards fast adaptation of pretrained contrastive models for multi-channel video-language retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14846-14855, 2023.
Xudong Lin, Manling Li, Richard Zemel, Heng Ji, and Shih-Fu Chang. Training free deep concept injection enables language models for video question answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 22399-22416, 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-3444 -3491,, 2023.
Yikun Liu, Yajie Zhang, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiangchao Yao, Yanfeng Wang, and Weidi Xie. Lamra: Large multimodal model as your advanced retrieval assistant. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 4025--025,,10 2025.
Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding zero-shot adversarial robustness for large-scale models. arXiv preprint 22Xiv:2212.07166, 2022.
Chengzhi Mao, Revant Teotia, Amrutha Sundar, Sachit Menon, Junfeng Yang. Xin Wang, and Carl Vondrick. Doubly right object recognition: A why prompt for visual rationales. In Proceedings of the TEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2722-2732, 2023.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint arXiv:2210.071:0 2022.
Sachit Menon, Ishaan Preetam Chandratreya, and Carl Vondrick. Task bias in vision-language models. arXiv preprint 2rXiv:2212.044122 2022.
Muhammad Ferjad Naeem, Yongqin Xian, Xiaohua Zhai, Lukas Hoyer, Lue Van Gool, and Fed- erico Tombari. Sile: Improving vision language pretraining with self-distillation. In European Conference Computer Vision, pp. 38-55. Springer, 2024.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXt::2304.07193, 2023.
AJ Piergiovanni, Weicheng Kuo, and Anelia Angelova. Pre-training image-language transformers for open-vocabulary tasks. arXiv preprini arXiv:2209.04377 2022.
Michael Posner. Orienting of attention. Quarterly jouraaaljoournall of experimental psychology. 32(1):3-25, 1980.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. International conference on machine learning, pp. 8748-876 PmLR, 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 35:25278-25294, 2022.
Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red circle? visual prompt engineering for vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11987-11997, 2023.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip scale. arXiv preprini arXiv:2303. 15389, 2023.
13

---

Published as a conference paper at ICLR 2026
Simam Swetha, Jinyu Yang, Tal Neiman, Mamshad Nayeem Rizve, Son Tran, Benjamin Yao, Trishul Chilimbi, and Mubarak Shah. X-former Unifying contrastive and reconstruction learning for mllms. arXiv preprint arXhv:240713381,, 2024.
Chameleon Team. Chameleon: Hixed-modal early- foundation models. arXiv preprint arXiv:2405.098/8. 2024.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint 25X1 2025.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9568 9578, 2024.
Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer. Image captioners scalable vision learners too. Advances in Neural Information Processing Systems, 083.46830-468555 2023.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, al. Siglip Multilingual vision- vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint 2502 2025.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Bo Wan, Michael Tschannen, Yongqin Xian, Filip Pavetic, Ibrahim M Alabdulmohsin, Xiao Wang, André Susano Pinto, Andreas Steiner, Lucas Beyer, and Xiaohua Zhai. Locca: Visual pretraining with location-aware captioners. Advances in Neural Information Processing Systems, 37: 37:116355- 116387, 2024.
Xiao Wang, Ibrahim Alabdulmohsin, Daniel Salz, Zhe Li, Keran Rong, and Xiaohua Zhai. Scaling pre- training one hundred billion data for vision language models. arXiv preprint 251::2500200617.. 2025.
Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. In European Conference Computer Vision, pp. 387-404. Springer, 2024.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on IEEE, 2010.
computer vision and pattern recognition, pp.
Rui Xiao, Sanghwan Kim, Mariana-Iuliin Georgescu, Zeynep Akata, and Stephan Alaniz. Flair: Vim with fine-grained language-informed. image representations. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24884 24894, 2025.
Hu Xu, Saining Xie, Xiaoging Ellen Tan, Yao Huang, Russell Howes, Vasu Sharma, Shang-War Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data, arXiv preprint arXiv:2309. 2023.
Chengrun Yang. Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe V Le. Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint arXiv:2309.03400) 03409, 2023.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini. and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint 2200:2205.01117. 2022.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12104-12113, 2022a.
14

---

Published as a conference paper at ICLR 2026
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with tecked-image text tuning. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 18123-18133 2022b.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11975-11986, 2023.
Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, and Ming- Wei Chang. Magiclens: Self-supervised image retrieval with open-ended instructions. arXiv preprint arXiv:2403. / 19651, 2024.
Yiwu Zhong. Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, al. Regionclip: Region-based language-image pre- training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16793-16803, 2022.
Junjie Zhou, Zheng Liu, Shitao Xiao, Bo Zhao, and Yongping Xiong. Vista: Visualized text embedding for universal multi-modal retrieval. arXiv preprint 2xX1:2440.002922,0 2024.
15

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1 LIMITATIONS
Our approach enhances the controllability of visual representations using language instructions. However, its practical application and further development are subject to certain limitations, which also open avenues for future research.
A primary challenge lies in the formulation of effective textual queries to maximize performance on specific downstream applications. The process of identifying the optimal phrasing, level of detail, and linguistic structure for queries that elicit the desired visual representation changes remains an empirical endeavor. may require significant tuning for each new task dataset. This is compounded by the inherent ambiguity and richness of natural language, where subtle variations in a query can lead to different steering outcomes, not all of which may be beneficial for the target application's accuracy. We conducted initial experiments in yet a principled way to discovery effective prompt is still missing.
Optimizing Query Design for Downstream Tasks:
Figure 12
Handling of Complex and Compositional Queries. The reliance a pretrained text encoder constrains the complexity of queries our method can effectively interpret. Current pretrained text encoders, while powerful, often struggle with deeply compositional abstract textual prompts. Their encoding of nuanced relationships between multiple concepts, or negation, might not be robust. Our method, therefore, performs best with relatively simple, direct queries.
Potential for Undesired Steering Outcomes. Depending on how the users provide the instructions, the model has a risk of generating biased, unsafe, or undesirable content.
A.2 FUTURE WORK
Principled Query Optimization and Discovery: Developing systematic methods or even learnable components to automatically discover or refine queries for optimal downstream performance would significantly enhance usability. This could involve techniques from prompt engineering, reinforcement learning, or semantic search to bridge the gap between user intent and effective query formulation.
Enhancing Complex Query Understanding: Future work should focus on strategies to decompose complex textual queries into simpler, manageable sub-queries that our current framework can process. Alternativlly, exploring new architectures or fine-tuning regimes for the text encoder to better handle compositional semantics and logical operations directly within the query embedding space would be valuable pursuit. This could involve incorporating structured knowledge or symbolic reasoning alongside neural representations.
Visual Grounding with Instructions: Our method can mitigate visual hallucinations, which can be used as component in RAG systems, to help verify, ground the reasoning and prediction of L.M..
Language-Instruce Instructed Vision Generation: Our method is a language- language-instructed vision encoder, which can be used as the backbone that encode semantic information in generative models, such as Diffusion. For example, by using language-instrucee vision embeddings, one can train image editing models based on the instructions.
A.3 BROADER IMPACT
Our research on language-steeres vision embeddings has the potential for considerable positive societal impact, primarily by offering novel approaches to creating more equitable and robust AI systems. By enabling vision embeddings to be guided by language instructions, we introduce mechanism for actively mitigating biases present in training datasets. This zero-shot bias mitigation capability is significant step towards fairer Al representations, as it allows for targeted adjustments without the need for extensive retraining or dataset curation, making the development of equitable models more accessible.
Furthermore, our method enhances the robustness of vision embeddings. By training models on detailed, instructed triplets, they leam to capture nuanced, fine-grained signals from image, moving beyond a single, holistic embedding. This improved granularity can lead to models that are more adaptable and less susceptible to being misled by irrelevant superficial features. An important
16

---

Published as a conference paper at ICLR 2026
application of this enhanced instructional control is the ability direct the model defend against typographical attacks. This contributes to making vision models safer and more resilient to adversarial manipulations aimed at "jailbreaking" or deceiving them.
However, we also recognize potential negative socictal impacts. The same linguistic stcerability that allows for bias mitigation and robustness enhancement could, if misused, be employed intentionally introduce or amplify biases. malicious actor could craft instructions to make the vision embeddings unfairly prejudiced against certain groups or characteristics. Currently, our work does not include mechanism to automatically discriminate between benign and malicious instructions, nor a system refuse potentially harmful guidance. This creates risk of misuse, where the technology could be exploited to generate unfair or harmful representations, potentially leading to discriminatory outcomes if deployed in sensitive applications.
Future work should prioritize the development of safeguards against such misuse. This could involve research into methods for detecting and rejecting biased malicious instructions, establishing proto- cols for the responsible deployment of steerable vision models, and fostering deeper understanding of the societal implications as this technology matures.
A.4 SAFEGUARDS
Since our training set is repurposed from Imagenet dataset and other established benchmarks that has been extensively used by the field, they shall not contain image data with NSFW. For language instructions, one can implement a classifier for the instructions to classify if is benign or malicious straightforward safeguard.
A.5
PSEUDO CODE
We provide pseudo code for implementing our LIVE encoder and training loss.
Assuming text text_quexi query, image, text text_answer answer are input batches 2 Assuming (temperature) and (bias) are parameters
3 Models: next_query_model image_model, sext_answer_mov
Frecomputed query:
_?query_raw, out_query text_query_model (text_query)
zquery jax. lax. wtt.ssspppggdiiao (.annnt(tqqppe (_zquery_ras
9 Image embeddings steered by query:
10 2 zimg, out_img = image_model (image, query_tokens=zqurr
Text answer embeddings: _txx_raww out_txt eext_answer_mode (text_answer) # **kw omi tted ztxt lax. stwpsssooogrgreeetttteeeee gradient (zzxt__aaa raw)
16 Compute logits:
17
logits jnp. dot (zimg, 2txt.T) (batch_sizo, batoh_ logits legits * t +
18
Contrastive loss calculation: batch size zing. shapel0] eye eyp (batch_size)
mi diagl jnp. ones like (logits) (2 * eye)
24
(ml loglik jax. nn (ml * logits)
sum loss =
nll -jnp. sum (loglik, axis=-1) # NLL per sample mean (np.mean(n1)) Average loss for the batch
Figure 8: Pseudo JAX code for language-sterred vision embedding model.
17

---

Published as a conference paper at ICLR 2026
1 VIT Input: image language Query Tokens (Concise)
2 Assumes: self (Flax Module), nn (flax.1inem) jnp
5 Config: self.I, self.dtype_mne mm, self.width, self.patch_siee self.poseml
Helper; got_posemb() for positional onboddings
1. Image Patch Embeddinge
img_in jnp. amp.aaarry (image, dt dddppe__mn) patches nn. Conv http://turrssel..iittt.t.
self self.patch_siza)
10
strides= size, self.patch_sine
padding="VALID", name-"patch_conv"
Sttye=self.dtyppe dtype_mm, (img_in)
n, h, patches. patch_emb jnp. reshape (patches, (n, w, c]) Add positional embeddinge to patch embeddings
15
patch_emm +- get_posemb (self, self.posee (h, c, "patch_pos", patch_emb.dtyy dtype)
2. Process Query Tokens
query_feat_ nn. nn.initiall ieer.. (1, self.I, c), selt.dtype_mm, mm) query_emb {_roj __pos_emb
query_tokens input, e.g., (batch, query feat_ dim)
proj nn Dense (features=c sclf.I, name="query_proj"" scype=self.dtype.. (aype__m) (query_tokens)
21 proj jap. reshape (q (n, self.I, c)) 9_pos_emb self, paaa.aarrmm"uurrppsss
Concatenate quezy and patch embeddings for ViT Encoder _xisss for (batch, seq len, features)
Typically, sequence axis=1
28
encoder_input jnp, concatenate ([query_emb, patch_embl, axis=1)
eenodder_ input is then fed into the main VIT Encoder layers
Figure 9: Concise pseudo JAX code for VIT input processing with language queries. The sel is number language tokens feed into the Vit.
A.6 COMPARISON WITH EXISTING WORK
We list a comparison with existing vision language models in the followings, and visualize their architecture in Figure
A) CLIP Radford et al. (021). SigLip Zhai et al.
(2033)
LiT Zhai al. (022b)
B) Llava Liu et al. (2023)
Gemma Team et al.
(2055)
Paligemma Beyer et al. (2024).
Llama Grattafioni al. (2024)
C) CoCA Yu et al.
2022)
Cappa Tschannen al.
2023
D) VILT Kim et al.
2021
E) Falmingo Alayrac et al.
2002 BLIP Li et al.
2022
X-former Swetha et al. (0024)
(F) Ours
Our work introduces the first vision-centric encoder that uses language to modulate visual computation for encoding target tasks. We address the scarcity of high-quality image, query, and answer triplet data by transferring the knowledge from LLM such as Gemini, and we demonstrate how language can directly control the vision encoder.
A.7 THE IMPACT OF LANGUAGE INSTRUCTIONS FOR LIVE
Since our method allows feeding text instructions to the vision encoder, have the potential to serve the final task better by improving the query. We investigated the impact prompt text on the ImageNet classification accuracy of our IPIL.I So400M model variant. We show the classification
18

---

Published as a conference paper at ICLR 2026
Caption Model
Model
Vision Model
Text Model
Vision
Vision Model
Text Model
Model
(A)
(B) Encoder M.M-Decoder
Caption Contrastive Hybrid Model
Binary Classifier
Masked Loss
Caption Model
Vision Language
Vision Language
Encoder
Model
Energy Model
Vision
Text Model
Vision Emb
Text Query Emb
Text Answer Emb
Vision
Text Emb
Model
Joint Encoder
(E) Encoder Captioner Model
(F) Two Tower Model with Language- Vision Encoder (Ours)
Figure 10: Comparison with existing methods. Note that B, C, E requires large language model based decoders, D does have a embedding perform zero-shot retrieval.
Retrieval
Retrieval
Retrieval
Add
Vision Encoder
Vision Encoder
Text Model
Vision Encoder
Text Model
Model
Model
Vision Input
Text
Vision Input
Text Query
Text Answer
Vision Input
Text Query
Text Answer
Input
Baseline
(B) Fusion
(C) Menon el.
Illustration baselines compared with in our paper. We take the two tower architecture (A), add the text query embedding the embedding (B), and adding query to the text answer as description following Menon et al. (C).
Figure 11:
accuracy of different prompt in Figure 12 Due to our model's training on more sophisticated image queries, the ImageNet classification accuracy dropped to 49.32% when query prompt was used in retrieval tasks. Interestingly, by leveraging Gemini to evolve and generate different text prompts Yang al. (2023), improved the ImageNet accuracy 68.18% using the instruction query: "Classify the main object." We believe this demonstrates the potential of our instructive vision foundation model for future work in prompt optimization achieve even higher accuracy.
A.8 ADDITIONAL. EXPERIMENTAL RESULTS.
Results on Steering Visual Representations for Text Recognition, We repurposed the ImageNet dataset for text recognition task by rendering text from one ImageNet category onto image of another. A visual representation that ignores this text and instead predicts the original image's category would result in 0% accuracy. Therefore, higher accuracy directly indicates the model's 's ability to follow instructions and perform OCR retrieval. As shown in Table / significant effectiveness.
our approach demonstrates
Robustness Against Typographical Attacks. lasiunnla models like CLIP and SigLip are known to be vulnerable to typographical attacks, where target text is appended to image mislead
19

---

Published as a conference paper at ICLR 2026
70
62.5
55
47.5
40
Nene
What ls the category of the foregraund object
Main object dessification label?
Name the category the central subject
Classify the main object
Provide the category of the main abject
State the category the main object
Output the clasoffication of main object
Pertorm Classflication for the main object
Classity the most prominent object
Classify the key object.
Figure 12: Impact of Different Language Instructions for ImageNet classification task. The y-axis shows the ImageNet classification accuracy in %. The x-axis shows the language instructions for the vision encoder. By improving the query prompts, improve the downstream task accuracy by up to 20 points.
OCR Accuracy
ViT Model
Baseline
Ours
SigLIP 2 ViT-SO-14
10.48
38.99
Table 7: Zero-shot accuracy to recognizing the text on Imagenet dataset. We evaluate OCR performance when text in words, potentially of different categories to the ImageNet image, is rendered in the image. If the model only perceive the original imagenet image without attending to the added text, the accuracy will be O. While vision-only representations has a low accuracy on recognizing the text, our instructive visual embeddings allow embedding either image or text information based on instructions.
the model's representations. This vulnerability poses a significant concern for critical applications such as autonomous driving and facial authentication.
To evaluate this, we rendered a text sticker in the middle of ImageNet images, with the text explicitly stating different class name than the original image. If the model attends to this text sticker, its accuracy drops 0%. shown in Table baseline models exhibited reduced ImageNet accuracy of 48.31% % under these attacks. However, simply by adding the prompt, "Ignoring text, what is the object?", we observed a significant increase in robust accuracy, demonstrating our approach's ability to disregard typographical attacks.
Robustness Against Typographical Attacks Baseline
VIT Model
Ours
SigLIP 2 VIT-SO-14
48.31
51.48
Table Zero-Shot test accuracy on ImageNet with typographical attacks. When providing text sticker on top of the image, original image classification model has the tendency to be mislead by the text. By using text prompt to let the model to ignore the text, we can increase the robustness against typographical attack.
Instructive Visual Benchmark on All Language Instructions. In the main paper, we present the results testing unseen instructions, where exclude all the language instructions that appear in the training. Table we also show the results on all instructions, which includes also language instructions that appear in the training. Our method consistently improves accuracy the instructive
20

---

Published as a conference paper at ICLR 2026
ImageNet
Caltech 101
VIT Model
SigLip
Fusion
Menon al.
Ours
SigLip
Fusion
Menon et al.
Ours
SigLip T/14
7.84
10.85
7.93
71.72
7.52
6.64
9.49
26.50
SigLip B/16
8.45
9.35
8.17
83.51
8.83
7.56
12.3
38.74
SigLip 2 B/16
9.29
10.91
9.50
84.54
8.18
8.05
14.8
37.12
SigLip 2 So400m
9.21
10.46
9,43
85.00
8.08
8.18
15.04
37.64
SUN
RefCOCO
VIT Model
SigLip
Fusion
Menon al.
Ours
SigLip
Fusion
Menon al.
Ours
SigLip T/14
6.13
5.50
6.72
26.87
5.72
5.72
5.72
33.52
SigLip B/16
8.41
8.06
9.68
41.55
6.93
6.94
7.09
47.24
SigLip 2 B/16
10.08
10.02
14.41
41.41
7.75
7.75
9.72
45.42
SigLip 2 So400m
10.81
10.54
15.26
44.68
6.63
6.68
7.65
47.80
Table Zero-Shot Accuracy on Instructive Visual Benchmark repurposed from ImageNet, Caltech 101, SUN, and RefCOCO. We directly test our model on these datasets without any training on them. This is in contrast to prior work that require finetuning on those downstream tasks Mao
et al.. 2023): Beyer et al. (2024) to do them.
visual benchmark. Despite some instructions being encountered during training, the task's difficulty persists. This is attributed to the new image and data domains, and the fact that many tasks remain non-trivial even with instruction familiarity.
Ablation Study on Instrucist Generalization We investigate the ability of our learned embeddings generalize unseen instruction families after training on distinct set. Utilizing Gemini, we automatically categorize ImageNet instructions into four broad families: fundamental properties (F), spatial-textual symbolic tasks (S), viewpoint composition aesthetics tasks (V), and dynamic inferential interpretive reasoning tasks (D).
Table presents our results where SigLip B/16 model is trained on three of these instruction groups and evaluated on the deliberately held-out fourth group on ImageNet. While training and testing on all groups yields 86.93% accuracy, testing on our hold-out subgroups results in only 1-2 percentage point accuracy drop for three of our studies. Notably, when not training on F (fundamental properties), the model experiences a significant accuracy drop, underscoring the importance of training on instructions related to fundamental properties.
A.9 TRAINING DATASET
We conducted an in-depth analysis to understand the distribution of language instructions generated by our LLM for the ImageNet dataset. Our process involved key steps: First, we used Gemini Flash 2.0 to define 66 distinct subcategories for vision-related questions, which are depicted in Figure 13 Second, we employed Gemini Flash 2.0 assign each question within our expansive 16-million synthetic saagee-uuer--anssee triplet dataset to one of these 66 categories, or to an "others" category if it didn't fit.
The resulting distribution, visualized in Figure 13 reveals significant variations in instruction fre- quency. "Material identification via Visual Properties" was by far the most common, accounting for roughly 2.2 million data entries. In contrast, "Fractal Properties/Self-Smmiiin Analysis" was rarely observed, with only 140 associated queries.
A.10 TESTING DATASET
A.10.1
ESTABLISHED BENCHMARKS
MMVP. In this paper, we use the MMVP-VLV benchmark, which are divided into 9 visual patterns. The benchmark consists of image pairs with corresponding answer pairs to retrieve. The original MMVP only comes with text answers, no text queires. Yet since they are divided into categories with answers that has good description for the task to ask about. We create queries, which itself,
21

---

Published as a conference paper at ICLR 2026
Category Counts
ieficaatonn Praperties
Suajact/Fosture Orientation
Specific Modellnstance dletifiiation
ScenelEniirrmmen Componencs
Contest
Quantlat ve Est mation with Confidence
Mulb-Objec: Spatial Reletionships (Steti)ynammic)
Color
Verietlors
Object/Sytemm Openatonnl
Fonal Point ineerti rahion & Emphasis Terrriquee
Perceived Testure Origin (Netenil vs. Ligting)
Condition from Visual Peatures
Extegorization Subil sigra
Impart of Hyputhetical
Element Alteratiora
Complex Mulli Stop AcEi by Recogritio
Viewpoint based Ocz usion, Overiap Layering
Contentual somartics Virual
Object onality. Affordances
Overall Mood. Feeling& 5 Atmrophere
Geometric Primitive Decompestton Shapes
Subhte/Comphee Complooiived Emetton Recogrition
HistorylUsage
Motion Altrbucee /Velocty. Trajectory, Nature)
Petten/Rostree Dentyy
Interned Function Untem lirrUnicue Osjects
Arrangement
crssic/Aresseecuue Style Peradd
Social intera tirw. Roles Thew Dynemits
Causel Reesening Observed SisteuEvena
Assesoment against Citerii/Excmppic
Cacapory
Depicted Agents Gool, intent & Notivation
Symbolic iconography weaning
Ecosystem HeslthiDistress didicaters
Uept-Deeerdent Hazeria appeerancts 9. indeeseence ttem Packing Density Dense) Optical Pempective Exidence of Image Meripuldiosyorrery Structural Losc Bearing Chments/Stress Points influence HerratowFerr lon'ts leoce
becessary DolaCormpnentte for Depicted
Iutuin That Processing Inderpattaionn
hulow Slelw/hajecttrg Art opatiss
ArtictioC..turll kcentfiiation fication
Creator's Lidly Purpose! ntent
Inforred Skeleta/lntemm Structure
Cultural, Histortcal Symbol it Sigrificance
Comparative
(SimlantyiD
Ochers)
Change
Secucncing
Aesthetic Appeal Subjective Visial Response
Specific thotog appici/ttstc Techniques Used
Tormal Compositiral Rules
Outcomes Hypothetical
Cranges
Rep on on Abstract Concepus
Spetial Decth, Negative Space & Their Use Fent for En/Purpose/DDgg Inference
Symboic
Mataphorizal Moanings
Impoct Lines, Shapes Forms Eamocaition
effccvveness of Visual mnssegemaaraal conveyance
Step by Stop frocess Derivation from Visuals
Audlence voried intorpretations Visull Ahythms,
Fractal Propertee/Seetsim Similatty Analysis
0.0
Figure 13: The Histogram of Query Categories Generated in our Language-Instruccke ImageNet. We first use LLM to generate a taskonomy of visual queries. We then use LLM to label each instructions generate one of the categories. We show the counting plot. The data generated show a long tail distribution.
does not offer any additional information to distinguish the text answer, since for example, the answer pair to discriminate is "A minion smiling with tounge out" and "A minion smiling without tounge out", added query: "Is the minion smiling with tounge out" does not offer additional information
22

---

Published as a conference paper at ICLR 2026
on the tounge's status, but repeat of the answer context. Note the chance prediction accuracy MMVP is 50% due to the binary choice.
VQA v2, GQA. While target is to evaluate how language can steer the visual representations, as benchmarked by our above datasets that designed to evaluate this, like having rich query and answer pair for the same image. We also use existing visual question answering tasks like VQAv2 and GQA, which often has single query and ansewr for the same image. We subsample the first 500 samples from GQA validate our approach for all our experiments. By adding our instructions, we achieve significantly higher accuracy than vanilla models. Note that for VQA task, the query tend to already contain a lot of information about the image, therefore, the Menon et achieve higher accuracy largely due to the query allows better retrieval for the image.
A.10.2 OUR BENCIMARKS
For the following repurposed dataset to evaluate this language-instructed vision embeddings, we use the same prompt to generate the test answer and query:
Provide a numbered list of interesting visual questions about the image, Followed by the corresponding answers.
Note since those are unseen images, and new domains for four of them, the Gemini-generate questions are often very different, allowing us to perform zero-shot evaluation on both: 1) novel data category and domain but instructions could be seen before, 2) novel data category, domain, and unseen instructions. We report the (2)'s results in the main paper due to the space limitation. We will also report the accuracy for (1) in later section.
ImageNet. ImageNet validation was originally designed for evaluating classification tasks. We repurpose it to also benchmark instructive visual embeddings. The queries generated by gemini condition on the Image. In the main paper, we remove all instructions are appear in the training. Therefore, the numbers shown is unforeseen, new instructions. In addition, in the appendix, we also show the retrieval on all the instructions generated without removing the ones that overlap with the training queries. There are 145549 data for the validation data in the paper after removing the ones with instructions appear in the training. Before removing the data with seeing instructions, is 551514. We retrieve the answer from 1000 answers, which contains the groundtruth and 999 random others.
Caltech101 We repurpose the test set via Gemini, to generate open queries and corresponding answers. In the main paper we remove queries that overlap with training. We also show the results for the set without removing the overlapping ones.
SUN We repurpose the test set via Gemini, to generate open queries and corresponding answers. In the main paper we remove queries that overlap with training. We also show the results for the set without removing the overlapping ones.
RefCOCO We use the images with rendered bounding box to to create the test datasets. feed the image with bounding box to LLM to generate open queries and corresponding answers. Note that the task is zero-shot because bounding box is not given in ImageNet.
Net OCR Test We render text the ImageNet validation images, where the text are the name of a different category. Therefore, the model will have different predictions by looking at the text or the image object category itself.
A.11 ATTENTION VISUALIZATIONS
We provide more attention visualizations of our encoder in Figure 14 Guided by language instructions, without supervision on where the model shall look at, our LIVE encoder learns to focus on the part of the image that is corresponding to the language instructions.
23

---

Published as a conference paper at ICLR 2026
Input Image Instruction
Attention Map (Ours)
Input Image + Instruction Attention Map (Oure)
What is the color of the floo??
What is the fexture of the ground beneath the bird?
that color is the football helmer?
Is there collar on the gray dog?
Is thom any stom visille on the butternut squeah?
What the bird parchad an?
LIFE
"LIFE" logo located the Image?
What inside the loft bucket?
man wearing?
What action is the person the background performing?
Figure 14: Attention Visualizations of Our LIVE Encoder. Guided by language instructions, the VIT model learn to focus on relevant parts, effectively prioritizing information and ignoring distractions. This is achieved without any direct supervision on the region the model shall focus on, showing the active, selective capabilities can be automatically leamndd by encoder. Examples randomly draw from ImageNet validation set that was not trained on.
24


Reviews:
[{"review_id": "K7RNMhmAi0", "reviewer": "Reviewer_LBmL", "summary": "The paper introduces a new paradigm in vision–language modelling: rather than merely feeding image features into a large language model, they reverse this flow by using language instructions to guide the vision encoder. The proposed method, called Language-Instructed Vision Embeddings (LIVE), uses natural-language directives at inference time to steer the vision encoder into producing task-centric visual embeddings without any task-specific retraining.\nIn practice, this means the model can dynamically focus on aspects of an image relevant to the given instruction, resulting in more controllable and generalizable representations. Empirical results show substantial reinforcement: LIVE reduces visual “hallucinations” by a large margin and outperforms much larger vision–language models on visual question-answering tasks, even when faced with unseen instructions and tasks.\n\nContributions: \n- Introduces the idea of language-driven visual encoding: language is not just describing or querying, but actively guiding how the visual encoder processes the image.\n- Provides a framework where the vision encoder produces embeddings that are instruction-aware, enabling strong controllability (you decide what to focus on) and generalizability (works for new instructions/tasks).\n- Demonstrates empirically that this paradigm can deliver strong performance gains: better alignment of visual features with linguistic task goals, fewer hallucinations, and robustness to unseen tasks/instructions.", "questions": "1.\tClarifying Conceptual Novelty\n\t•\tWhat exactly is unique about LIVE’s modulation pathway (e.g., position of language injection, training objective) that cannot be reduced to prior adapter or cross-attention methods?\n\n2. Data Contamination (Weakness #2)\n\t•\tDid you perform any explicit decontamination between the Gemini-generated triplets and the evaluation benchmarks?\n\t•\tWould you consider running a controlled experiment on a held-out, human-annotated benchmark to quantify the possible inflation due to data leakage?\n\n3.\tUse of Model-Generated “Oracle” Labels\n\t•\tFor the “†” benchmarks where Gemini provides the reference answers, can you clarify the rationale for treating Gemini outputs as ground truth?\n\t•\tHow correlated are Gemini’s responses with human-labeled gold data (e.g., on a small manually annotated subset)?\n\n4.\tScale and Fairness of Comparisons\n\t•\tCan you provide more detail on the total compute budget and data volume of LIVE versus each baseline?\n\t•\tWere any baselines retrained under identical compute/data conditions, or were all numbers taken from prior publications?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper presents a well-motivated and timely contribution in the field of language-guided visual representation learning. The proposed LIVE (Language-Instructed Visual Encoder) introduces an instruction-based training paradigm that allows natural language to directly modulate the visual encoding process. This idea aligns closely with the current trend of multimodal alignment and reasoning, yet it explores an under-investigated direction—embedding-level linguistic control—rather than the more common dual-encoder or caption-generation settings.\n\nOriginality\nThe originality of LIVE lies not merely in architectural novelty, but in its reformulation of multimodal learning as instruction-conditioned representation alignment. Instead of aligning paired modalities post hoc (as in CLIP), the model internalizes linguistic guidance into the feature extraction process. This perspective opens new possibilities for controllable and interpretable vision encoders. While the high-level idea is inspired by prior “visual instruction tuning” work, the paper provides a conceptual synthesis that generalizes those methods into a unified embedding-learning view.\n\nQuality\nThe paper provides a coherent training objective, a clear architecture description, and a reasonable learning pipeline involving LLM-generated image–instruction–answer triplets. The experimental section covers multiple benchmarks, showing consistent improvements over baseline vision encoders. The methodology is easy to follow, and the ablation studies—though limited—support the claim that linguistic instruction improves cross-task generalization. In terms of engineering quality, the model is computationally efficient and can be integrated into existing multimodal pipelines.\n\nClarity\nThe manuscript is generally well-written and clearly organized. Figures and tables effectively convey the core architecture and comparative results. The motivation is explicitly stated, and the connection to prior work is well articulated. \n\nSignificance\nLIVE addresses a fundamental question in representation learning: Can visual features be directly instructed through language? The proposed framework provides early empirical evidence that such an approach is not only feasible but also beneficial. This work could inspire a new class of “instruction-aware” encoders applicable to downstream tasks such as retrieval, visual reasoning, and grounding. Its conceptual significance lies in bridging the gap between vision-language alignment and language-conditioned representation control, which is of high relevance to the ICLR community’s focus on generalizable and interpretable AI systems.", "weaknesses": "1.\tOriginality is under-argued vs. closely related paradigms.\nThe paper positions LIVE as “language-instructed” encoding inside the vision tower, but the boundary from prior lines of work (caption-conditioned encoder modulation, cross-attention/adapters, visual instruction tuning) isn’t made watertight. \n\n2. Potential data contamination between training and evaluation remains insufficiently addressed.\nAlthough the paper specifies that the 16.4 M (image, instruction, answer) triplets are generated with Gemini 2.0 Flash and even provides example prompts, it does not report any train–test decontamination analysis. Given that Gemini-generated supervision and evaluation “oracle” labels (marked †) are both derived from similar prompting pipelines, there is a tangible risk of semantic overlap or leakage into benchmarks. \n\n3.\tUse of a model-generated “oracle” undermines evaluation rigor.\nSeveral benchmarks marked with “†” use Gemini-produced answers and then measure how close LIVE gets to that oracle. This is not a neutral ground truth: it bakes the teacher’s biases into the metric and can inflate apparent progress. Replace “oracle” labels with human annotations or established datasets; if teacher labels must be used, include teacher–student agreement vs. human gold, plus audits for leading prompts and spurious shortcuts.\n\n4.\tFairness of headline gains is only partially established (scale mismatches remain).\nThe reported +34-point MMVP and large GQA gains are impressive. However, the overall training scale differs substantially—LIVE uses 16.4 M LLM-generated triplets trained on 256 TPUv3 cores—which makes it difficult to disentangle architectural improvements from data or compute advantages. Re-evaluating baselines under comparable compute/data budgets or reporting normalized gains would clarify how much of the improvement stems from the proposed mechanism itself.\n\n5.\tAblations only partially support the claimed mechanism.\nThe paper provides some evidence that language modulates visual representations—such as comparisons without instructions, attention visualizations, and encoder-size studies—but these analyses remain high-level. More controlled ablations (e.g., varying where or how language is injected, or replacing instructions with neutral text) could strengthen the causal link between “instruction conditioning” and the observed performance gains. As written, it remains difficult to fully disentangle the benefits of language-guided encoding from those of larger data or improved contrastive training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "13KNaSVZ1U", "reviewer": "Reviewer_pP9q", "summary": "This work proposes a language-instructed vision embedding (LIVE)  method, aiming at extract more controllable and generalizable visual representation upon the language guidance. Specifically, the vision encoder takes the embedding of the textual instruction q and the image as input and compute the instructive visual embedding. Extensive experimental results compared with a series of the leading methods show the improved performance of the proposed method.", "questions": "* What is the setting of the inference pipeline? What are the specific inputs of the visual encoder and the text encoder? More detailed illustrations are expected. \n\n* The proposed method highlights more on the semantics of the visual embedding, how's the sensitivity of the proposed method to the visual detail variation? \n\n* Why the proposed work is termed as \"generalizable perception\"? How does the proposed method relate and contribute to the visual perception and low-level patterns?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The proposed method proposes a new perspective of learning effective and adaptive visual representation, which is interesting. \n\n* This work is well-motivated. \n\n* The introduction is well-written.", "weaknesses": "* L156~160 demonstrate that the vision encoder always requires the query as the instruction to guide the visual representation learning. Although the concept is novel and could make sense, this strategy inevitably brings the bypass for the multimodal learning, e.g., model can   rely on the query, rather than the image itself to learn the representation for the semantic matching. Intuitively, this might be one of the cause for the performance boost. \n\n* During training, the answer is adopted. Yet, during inference, it seems that the answer may not be adopted. Is this mismatch intentionally devised and what might be the rational behind this?\n\n * This work provides extensive of performance comparison but lacks thorough ablation and model discussions in the manuscript, such as the discussion of each type of the input to the text/img encoders. \n\n* Fig.6 shows that the query steer the visual attention. Yet, the more effective the steering it is, the more important role the query plays for the representation learning, instead of visual signals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UyD2qJ3GdE", "reviewer": "Reviewer_k7wj", "summary": "This paper aims to make vision encoders, such as CLIP, to extract language-instructed vision embeddings to produce query-specific visual features for downstream applications. To address this, the authors let Gemini-2.0-Flash annotate over 16M image-question-answer pairs, and then fine-tune the original vision encoder with SigLIP's loss. Experiments under various evaluation protocols demonstrate the effectiveness of the proposed method.", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-written and easy to follow.\n2. The motivation is clear, reasonable, and meaningful.\n3. The proposed method is quite effective.", "weaknesses": "1. The improvement may mainly come from the data instead of the method. To investigate this, the following experiments are highly recommended:\n   - Train LIVE on the classic caption dataset (using \"describe this image\" as the prompt) or the original ImageNet dataset (using \"what is the category of this image\" as the prompt).\n   - Train other methods like Flamingo or BLIP-2 using the proposed data.\n\n2. How this encoder incorporates with typical MLLMs remains unclear. It is encouraged to replace the original CLIP-336 in LLaVA with LIVE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cQJTdocsK1", "reviewer": "Reviewer_mP9X", "summary": "This paper proposes Language-Instructed Vision Embeddings (LIVE), a novel paradigm that uses language instructions to dynamically guide a vision encoder during inference, enabling it to produce task-specific visual embeddings without retraining. To achieve this, LIVE reuses the text encoder from SigLIP and injects textual queries directly into the vision part, allowing it to focus on instruction-relevant image regions. After training on  around 16.4 million images-query-answer synthesized by Gemini, LIVE reduces visual hallucinations by 34 points on the MMVP benchmark and outperforms much larger vision–language models (e.g., LLaVA, InstructBLIP) on GQA, demonstrating its effectiveness.", "questions": "1. Can the trained visual encoder be used for MLLM training? Will it make current MLLM (.e.g. LLaVA) better or worse?\n2. See Weakness above", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of this paper is quite novel. By training a language-instructed visual embedding model, it addresses visual tasks more efficiently with a model of reduced parameter count, eliminating the need for task-specific training. Furthermore, it mitigates the issue of large language models (LLMs) dominating visual inference in existing multimodal large language models (MLLMs), which is a commendable contribution.\n2. The methodology is both straightforward and effective. Moreover, the synthesized triplet dataset holds significant potential for enhancing CLIP-style pretraining for the research community.\n3. The experimental results are promising, demonstrating that a considerably smaller embedding model surpasses existing MLLMs across a wide range of tasks. Additionally, the attention visualizations substantiate the efficacy of LIVE at the representational level, which is highly appreciated.", "weaknesses": "1. Generating large-scale vision-related data based on Gemini 2.0 may incur substantial costs. It would be beneficial if the authors could provide relevant details regarding the expenses associated with data synthesis and model training.\n2. The examples presented in the paper are confined to relatively simple queries. It remains an open question whether the proposed approach maintains its efficacy when applied to more complex visual problems and intricate queries (e.g., visual reasoning tasks).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)