Better Language Models Exhibit Higher Visual Alignment
Authors: Jona Ruthardt
Gertjan J. Burghouts
Serge Belongie
Yuki M Asano
Venue: Accepted by TMLR ()

Abstract: How well do text-only large language models (LLMs) align with the visual world? We present a systematic evaluation of this question by incorporating frozen representations of various language models into a discriminative vision-language framework and measuring zero-shot generalization to novel concepts. We find that decoder-based models exhibit stronger visual alignment than encoders, even when controlling for model and dataset size. Moreover, language modeling performance correlates with visual generalization, suggesting that advances in unimodal LLMs can simultaneously improve vision models. Leveraging these insights, we propose ShareLock, a lightweight method for fusing frozen vision and language backbones. ShareLock achieves robust performance across tasks while drastically reducing the need for paired data and compute. With just 563k image-caption pairs and under one GPU-hour of training, it reaches 51% accuracy on ImageNet. In cross-lingual settings, ShareLock dramatically outperforms CLIP, achieving 38.7% top-1 accuracy on Chinese image classification versus CLIP’s 1.4%. Code is available.

Published in Transactions on Machine Learning Research (01/2026)
Better Language Models Exhibit Higher Visual Alignment
Jona Ruthardt
jona. www.rruaaarr.utnn..mmm
Fundamental AI Lob, University of Technology Nuremberg
Gertjan J. Burghouts
purljen.bburghoutts@loo.oo
Intelligent Imaging, TNO
Serge Belongie Department of Computer Science, University of Copenhagen
s.belongie@di.kc dk
Yuki M. Asano
yuki.asano@utn.d
Fandamental AI Lab, Universily of Technology Nurembery
Reviewed on OpenReview: https: // openreview. net/ forum? id (d=
Project Page: https: // jonaruthraa.. geniit.gghhhht github. so/ pro/projeett.. ShareLock/
Abstract
How well do text-only large language models (LLMs) align with the visual world? We present systematic evaluation of this question by incorporating frozen representations of various language models into discriminative vision-language framework and measuring zero-shot generalization to novel concepts. We find that decoder- based models exhibit stronger visual alignment than encoders, even when controlling for model and dataset size, Moreover, language modeling performance correlates with visual generalization, suggesting that advances in unimodal LLMs can simultaneously improve vision models. Leveraging these insights, we propose ShareLock, a lightweight method for fusing frozen vision and language backbones. ShareLock achieves robust performance across tasks while drastically reducing the need for paired data and compute. With just 563k image-caption pairs and under one GPU-hour of training, it reaches 51% accuracy on ImageNet. In cross-lingual settings, ShareLock dramatically outperforms CLIP, achieving 38.7% top-1 accuracy Chinese image classification versus CLIP's 1.4%. Code is available.
1 Introduction
Large Language Models (LLMs) are solely pretrained unimodal textual data, yet are increasingly incorpo- rated into systems that perceive and interact with the natural world (Ahn et al., 2022: Driess et al., 2023; Wayve, 2023). The lack direct sensory experience raiscs fundamental questions as to what extent such models generalize across modalities and develop a meaningful understanding of visual reality. Do these models merely regurgitate visually relevant. factual knowledge from their training corpus, or do they form internal representations that correspond to real-world phenomena? Despite their successful integration into large-scale Vision-Language Models (VLMs), judging the visual capabilities already inherent to LLMs is difficult. This stems not only from differences in training recipes and proprietary data, but especially from fine-tuning with paired image text data, which blends with the visual knowledge already embedded in text-only models. In contrast, Sharma et al. (2024) and Huh et al. (2024) more immediately assess the visual nature of LLMs and highlight a non-trivial degree of visual understanding and cross- moosl alignment. These works compile proxy tasks such as generating code to represent real-world concepts (Sharma al., 2024) or correlating vision and language features (Huh et al., 2024). However, reliance on highly constrained and synthetic tasks with limited practical significance fails to gauge the aptitude of LLMs in more realistic settings.
To this end, we assess visual alignment the degree to which language model representations structurally and semantically correspond to those of vision throuhh the task of zero-shot open-vocabulary image classification, as popularized by CLIP (Radford et al., 2021). This involves learning a projection of language embeddings into the vision manifold and selecting the label whose representation most similar to a given image. To ensure rigorous evaluation of true zero-shot generalization, we enforce strict disjointness between concepts encountered during training and testing as illustrated in Figure 2 (center) (Lampert et al., 2009). This mitigates concept leakage, common issue in VLMs (Fig. 2, left). For example, Xu et al. (2024) find
1

---

Published in Transactions on Machine Learning Research (01/2026)
45.0
42.5
Visual Generalization Store [%]
40.0
37.5
35.0
SSnteeue-TT-e T6-XLL
T5-XXL
Model Family
32.5
Llama-3
30.0
Model Variant Base-LLM
Qwen2
Qwen2.5
27.5
Instruct-LL.M
gemma-2
ModerBERT
Encoder- LM
Falcon3
25.0
0
10
20
30
40
50
Massive Multitask Language Understanding (MMLU-Prro) Score
Figure 1: Visual generalization as language comprehension. Language modeling capability MMLU- Pro predicts LLM visual transfer performance (Pearson-r: 0.768). We compute visual generalization score by aligning language with vision features in a CLIP-like framework and evaluating on disjoint sets of unaligned classes across four datasets. Dot size is proportional to the LLM's parameter count.
significant conceptual overlap between CLIP's training and evaluation datasets. Other works demonstrato sharp performance drops for less-frequent or truly novel concepts (Fang et al., 2022; Udandarao et al., 2024; Parashar et al., 2024; Mayilvahanan et al., 2025). In our proposed setup, generalization relies solely on the semantic information and visual knowledge encoded language representations. By probing how well models capture visual semantics, we provide insight into their ability encode text for vision-language applications. We observe non-trivial zero-shot generalization across language model types, indicating latent visual-semantic alignment. In particular, features from modern generative LLMs outperform classic encoder-based embeddings such as BERT (Devlin et al., 2019). This advantage of decoder-based models persists when controlling for pretraining data and parameter count across architectures. Intriguingly, we find that general LLM capability, as measured by MMLU-Pro (Hendrycks et al., 2021a), correlates positively with the model's visual performance, as shown in Figure 1. Even off-the-shelf LLMs without mubeedding-speciff fine-tuning exhibit strong visual representation abilities.
Finally, we integrate frozen vision and language representations into lightweight discriminative VLM, ShareLock, demonstrating strong multimodal capabilitics across range of tasks. Despite using only fraction of the data and learnable paramelers, our method approaches the performance of fully optimized CLIP models trained on orders of magnitude more paired data. By capitalizing on the broad pretraining of modern LLMs, ShartLock achicves remarkable cross-lingual zero-shot gencralization to non-English languages, outperforming CLIP's performance of 1.4% with 38.7% for Chinese. The visual and linguistic expressiveness of L.L representations is particularly effective nuanced and fine-grained tasks, resulting compositional reasoning performance. Our results highlight the considerable extent to which language representations capture visual structure and semantics, enabling highly efficient alignment with vision embeddings using limited supervision and paramelerization. Overall, the main contributions of this work are:
Ahove-CI.PP
We provide a systematii evaluation of the visual alignment inherent to language models, using strict zero-shot image classification as a practically--elevant probing task.
Our analysis highlights modern decoder- based LLMs as effective sources visual knowledge, with semantically mcaningful representations extractable from their internal states.
With ShareLock, we incorporate frozen LLMs with high intrinsic visual alignment into a lightweight VLM, resulting in improved robustness and generalization on various classification, retrieval, multi- lingual understanding, and compositional reasoning tasks.
2

---

Published in Transactions on Machine Learning Research (01/2026)
Typical VLM Training Data
Alignment Probing Data (mutually exclusive concepts)
Contrastive Training
(open domain concept leakage)
Training: image-caption pairs
Training: aligned concepts A
"... car."
sitting on branch." "
"A photo of a cat."
LIM
"The boy pets the dog." 4
"A photo of rhino."
"... bird.""
"Orcas are black and white."
'1 photo of a bird."
Zero-Shot Evaluation
Evaluation: ImageNet- 1k etc.
Evaluation: unaligned concepts U
'A photo of goat."
photo of dog."
'A photo of a fish."
1AM
"A photo of a tion."
"A photo of dog. "
+B
"A photo of a dog.
4
photo of horse."
'A photo of horse."
Figure 2: Comparison of training modes. (Left) Web-scale VLMs training lacks concept control, weakening generalization claims and resulting in erratic drops for rare categories. (Center) Our visual alignment probing protocol enforees strict concept separation to assess true gencralization. (Right) Our ShareLock method uses lightweight projections to align frozen unimodal models via CLIP-style contrastive learning and zero-shot. evaluation.
2 Related Work
Visual Understanding of Large Language Models. LLMs can infer and reason about visual content without explicit multi-modal training (Bowman, 2023). Sharma et al. (2024) tasked LLMs to draw common objects and scenes using simple shapes, indicating spatial understanding and illustrating that LLMs can conceptualize real-world settings. Various works highlight the plausibility and utility of L.M-generated descriptions objects in the context of image classification and demonstrate that LLMs possess encyclopedic knowledge about visual characteristics (Pratt et al., 2022; Menon and Vondrick, 2023; Yang et al., 2022; Saha et al., 2024). These capabilities suggest that the extensive pretraining on large volumes of diverse textual data aids the visual understanding of LLMs. Huh et al. (2024) argue that the embodding spaces of neural networks converge lowards a shared 'platonic' represeniation of reality irrespective of the concrete optimization objectives and data modality utilized during training. Similarly, we investigate the degree of visual alignment inherent to exclusively languago-basse basedd representations but assess this in the practically more relevant context zero-shot image classification and design a rigorous benchmark to measure the true generalization capabilities facilitated by such language embeddings.
Vision-Language Alignment. Similarly to Huh et al. (2024), other recent works indicate and exploit alignment between pretrained unimodal vision and language models. Zhai et al. (2022) and Khan and Fu (2023) reveal that leveraging pretrained models and only tuning a subset of parameters can improve performance and efficiency over CLIP (Radford et al., 2021), indicating some degree of model-inherent alignment. Norelli et al. (2023) and Maniparambil et al. (2024) only rely cross-modal correlations for training-free alignment of unimodal models. However, these studies primarily restricted to encoder-basod language models. Zhang et al. (2024) incorporate decoder-based LLMs into CLIP-like framework but transform both vision and language representations, making it difficult to isolate the contribution of each modality during aligument. In contrast, we systematically ovaluate both encoder- and decoder- basod language models, examine how well their representations can be mapped to visual latent spaces, and focus on zero-shot. generalization.
3 Probing LLMs for Visual Alignment
This section details our methodology for quantitatively assessing the visual alignment of language models. We introduce the zero-shot evaluation protocol, describe the architecture aligning frozen vision and language backbones, and summarize implementation details to ensure reprodueibility.
3

---

Published in Transactions on Machine Learning Research (01/2026)
3.1 Zero-Shot Generalization
We determine the visual aptitude of language models by drawing on traditional zero-shot learning methodology (ef. Lampert et al. (2009)) and consider how their representations facilitate generalization to novel concepts. To rigorously assess true generalization performance without concept leakage from supervision with arbitrary web-scraped image-captions pairs, we split image classification datasets into aligned classes Å (training stage) and umaligned classes U (testing stage) with A = 0 (see Figure 2, center). In the absence of image-specific captions, text-based class representations f(y:) are used as supervision signals during training and for zero-shot transfer during inference. Crucially, this implies that the performance discriminate novel concepts is contingent on the validity and cross-modal contimuity of the class representations. Therefore, this setup allows us to assess the degree to which language models encode visual knowledge and semantics.
3.2 Shared Vision-LanguageLLoceel Tuning
To map textual inputs into visual latent spaces, we draw inspiration from late-fusion archilect in CLIP- like models. Texts are first encoded using a language model (-xa-) and subsequently projected into the d-dimensional latent space of the vision encoder (-mg(-) via learnable projection network (-). The latent representation for a given input image X, or caption i, is therefore computed by Zung = (ang(()) E Rd and zixt = Poxt (orx (t:)) E TRd, respectively. Their similarity sim(Zang. zxxa) is computed as the cosine similarity, given by the dot product of the normalized embeddings.
During training. only the lightweight projection network (-x is optimized, while the pretrained vision and language backbones remain frozen. A contrastive loss encourages alignment by pulling textual representations closer to their corresponding image embeddings while pushing them away from non-matching ones, in (Radford et al., 2021). For an image-text pair i in a batch with N items, it is given by
exp (sim(z, i 2.)/T)
[(i) = log Σn= exp(sim( z /
(1)
for both alternated modalities pairings (m, n) E {(txt, img), (img, txt)} and with 7 being fixed temperature parameter. Given a set of classes C and their corresponding textual class representations f()), the predicted class è for sample x; is obtained via
arg max sim(Zing; Pixa (dext (f)))) eEC
(2)
3.3 Experimental Setup
Datasets. For a comprehensive evaluation, we select four datasets: AWA2 (Xian et al., 2017), CUB (Wah et al., 2011), FGVCAireraft. (Maji et al., 2013), and ImageNet spanning natural and human-made artifacts, coarse and fine-grained categories, and varying scales (40 < |A| < 1000). ImageNet+ treats ImageNet- Ik (Deng et al., 2009) classes as aligned concepts and the 500 most populated ImageNet-21k classes as unaligned ones. For AWA2 and CUB, we use splits by Xian et al. (2017) while randomly dividing aircraft types into 50 aligned and 20 unaligned classes. We report the average per-class classification accuracy on unaligned classes cA across the datasets as measure of visual generalization ability facilitated by the language embeddings. Besides the class-name based templates proposed by Radford et al. (2021) (e.g., "a photo of a <class>"), we generate more comprehensive Wikipedia-style descriptions with an LLM and acquire human-curated information from Wikipedia to be used as class representations (details in C).
Pretrained Unimodal Backbones. Given its strong performance, broad pretraining regime, and popu- larity, the ViT-L/14 variant of the self-supervised DINOv2 model family (Oquab et al., 2023) is the default vision backbone. Global image embeddings are obtained through the CLS token. Language representations are extracted from encoder-based models through mean Loken pooling or directly via the CLS token if fine-tuned on sentence-level representation tasks. For decoder-based LLMs other than NV-Embed (Lee et al., 2024), features are extracted through last-token pooling (details in Appendix). Frozen vision and language model features are initially precomputed and stored for direct re-use in subsequent epochs.
Training. Given the data-constrainod setting, we optimize a lincar projection network using Adam (Kingma and Ba, 2014) with a learning rate following a cosine schedule with a maximum value of 10-3, Gradient clipping to global norm of and weight decay of 10-1 are applied. The loss of Eqn. is applied with
4

---

Published in Transactions on Machine Learning Research (01/2026)
Table 1: Visual generalization scores of various language models. We benchmark visual alignment of models for different class representations. Decoder-based language models outperform popular encoder-based architectures across all types of imput data. Llama-3 8B (Instr.) used for LLM generated Wikipedia articles.
Class Names
LLM
original Wikip.
Language Model [# Parameters]
Wikip.
BERT-Large [336M] (Devlin et al., 2019)
14.2
16.4
24.0
ModernBERT [395M] (Nussbaum et al., 2024)
28.9
23.9
24.8
Enc.
all-roberta-v1 [355M] (Liu et al., 2019)
31.0
34.5
11.7
SentenceT5-XXI [1B] (Ni et al., 2022)
32.8
36.1
39.6
SentenceT5-XXI [5B] (Ni et al., 2022)
36.6
39,1
42.8
Flan-UL2 [10B] (Tay, 2024)
37.9
41.0
44.0
Dec.
NV-Embed-v2 [8B] (Lee al., 2024)
39.4
42.1
43.6
Llama-3 [8B] (Dubey al., 2024)
39.8
43.9
44.5
Gemma-2 [9B] (Mesnard et al., 2024)
42.8
45.0
44.5
== 0.07, and models are trained until convergence on a randomly chosen validation split or for a maximum of 3.5k sleps with a batch size of 16,384 and five different initialization seeds. Dropout, (Srivastava al., 2014) with p 0.2 is applied during training.
4 Language-driven Visual Generalization
Utilizing the zero-shot evaluation methodology outlined, we investigate critical factors that promote the generalization of language representations to illuminate language models' visual capabilities.
LLM representations encode visual knowledge. The class-wise supervision and limited concept diversity of conventional image classification datasets can impede vision-language alignment but permit more sophisticated semantic class representations beyond simple template-based targets as typically used with CLIP-like models (Radford et al., 2021; LAION AI, 2022). We thus examine how the nature and information content of different textual class representations impact gencralization performance. The results are summarized in Table 1. As a naive language-free frree baseline, we also construct one-hot-encoded class representations. Unsurprisingly, the lack of semantic continuity connecting aligned and unaligned concepts results in a near-random generalization score of 4.5% and demonstrates that generalization in the proposed evaluation protocol is fundamentally dependent We find that the addition of auxiliary information, such as Wikipedia articles, results in improved performance for most language models, reflecting insights from previous studies (Pratt et al., 2022; Menon and Vondrick, 2023; Yang el al., 2022; Saha el al., 2024). We also find that LLM-generated articles describing a class in the style of Wikipedia ("LLM Wikip." in Table 1) can provide strong targets during multi modal alignment, achieving the best overall performance of 45.0%. Interestingly, relying on strictly human-curated data in the form of actual Wikipedia articles lends only to provide marginal benefits, for example, from 43.9% 44.5% and 42.1% - 43.6% for Llama-3 and NV-Embed. Thus, LLMs can effectively absorb and interpolate substantial amounts of factual information from their training data, positioning them valuable sources of visually relevant knowledge.
the semantic alignment of vision and language embeddings.
Decoders excel in visual concept representation. A new insight resulting from analysis is the competiliveness of decoder-based language models in representing visual concepts. Compared to language encoders commonly used in vision tasks, we find that representing inputs with decoders can result in higher performance, mirroring recently emerging trend in the language domain (Lee et al., 2024; Springer et al., 2024). However, results in Table confound factors such as model and dataset size with the different pretraining tasks. To isolate the impact of architecture, we conducted controlled comparisons with the Ettin model family (Weller et al., 2025) that only varies attention patterns and training objectives while unifying data and parameter count for encoder- and decoder-basee models. Figure 3 reveals that decoders outperform encoders by an average of 2.7 percentage points across model sizes, suggesting that autoregressive next-token prediction gives rise to more vision-aligned representations compared to masked language modeling
Deeails about the characteristics and acquisition these class representations are elaborated in Appendix
5

---

Published in Transactions on Machine Learning Research (01/2026)
32
Visual Generalization Score [%)
Decoder Representations
Encoder Representations
30
28
26
24
22
20
17M
32M
68M
150M
400M
1B
Language Model Parameter Count
Figure 3: Encoder- decoder-based language models. The models are trained on identical data with matched model size, isolating the effects of their pretraining objectives. Decoders demonstrate higher visual alignment across all model sizes compared to encoder models.
in encoder models. Beyond this intrinsic advantage, decoder models often pretrained larger-scale data and comprise more parameters, leading to more capable models in addition to architectural benefits. For instance, Gemma-2 9B achieves the highest score of 45.0% across our evaluated input types, outperforming the strongest and similarly sized encoder model, Flan- UL2, which reaches 41.0% on the same input data. Other decoders like LLaMA and NV-Embed perform on par with Gemma, whereas encoder models such as T5- and BERT-based variants lag considerably behind. These findings suggest that decoder-based LLMs can offer both architectural and practical advantages for visual representation and alignment lasks.
LLM and its visual performance are correlated. In Figure 1, we compare various LLMs by the visual generalization ability they possess, as well as their MMLU-Pro (Wang et al., 2024a) score taken from the Open LLM Leaderboard (Fourrier et al., 2024), a common metric to measure LLM performance, We find that the general capability of language models strongly corrclated with their ability to perform well on the visual tasks (Pearson coefficient r: 0.768). Within and across model families, we see improved visual generalization the capacity and capabilities of models increase. Since models steadily improve in the language domain, our evaluation protocol will be helpful in assessing whether the trend of increasing visual understanding will continue in future LLM models. If this holds, VLMs that incorporate LLMs can piggyback off developments in the language modeling domain.
CLIP
LiT
ShareLock
Encoder
Image
Text
Image
Text
Image
Text
A
S
Decoder
Trainable
RC/MLP
Frozen
Figure 4: Our ShareLock us previous methods. Compared CLIP and LiT, ShareLock utilizes frozen pretrained representations for both modalities, allowing extremely efficient. training. Using this framework, assess how "visual" frozen language models' text representations are by how strong the resulting model can generalize to entirely novel categories. We find decoder-only LLMs to yield strong performances for zero-shot generalization and hence incorporate them as the text backbones.

---

Published in Transactions on Machine Learning Research (01/2026)
5 LLMs for General-Purpose VLMs
The previous section demonstrated that LLMs trained for textual next-token prediction encode visually- aligned semantics in their representations. Next, we investigate whether this inherent visual alignment can be leveraged to obtain effective and efficient multimodal models by integrating off-the-shelf LLMs directly into CLIP-like VI.Ms. For this, we relax the strict zero-shot setup and utilize larger-scale image-captine datasets (see Figure 2, left) to explore the benefits and limitations of general-purpose VLMs that fuse existing frozen foundation models with minimal data and compute.
5.1 Experimental Setup
Methodology. We propose integrating LLMs into a CLIP-like framework through "Shared Vision- Language-Locket Tuning" (ShareLock), extending the architecture and training setup introduced in Section 3. As illustrated in Figure 4, this design places LLM at the core of the model and maps its expressive reprosentations into the feature space of a frozen vision model. If not noted otherwise, Llama-3 8B (Dubey et al., 2024) and DINOv2-L (Oquab et al., 2023) are used as frozen backbones. Given the greater availability of training data compared to the strict zero-shot setting, we expand the projection network Pixe (-) applied on top of the frozen language features to four layers with a hidden dimension of 4096. The additional parameters, combined with RcLU activations between layers, enhance ShareLock's ability to capture complex, non-lincar cross-modal correspondences. Additionally, the maximum number of optimization steps is increased to 5000.
Datasets. Our investigation focuses leveraging LLMs with minimal additional paired data and explores how unimodal embeddings can drive robust multimodal performance with minimal supervision and alignment. As a result, our evaluation is limited to comparably small paired datasets. COCO Captions. COCO Captions (Chen et al., 2015) contains 83k images with multiple human-written captions per image, from which we randomly sample during training. CC3M. Conceptual Captions (Sharma ct al., 2018) comprises 2.8M filtered web-scraped image-alt-text pairs. We also utilize a smaller subset with more balanced concept coverage designed for LLaVA (Liu et al., 2023). CC12M. CC12M (Changpinyo et al., 2021) expands CC3M by 12M image-text pairs, enabling evaluation at larger scales. Due to expired links, our version contains about 8.5M samples. If not noted otherwise, CC3M is used to compare design choices.
Computational efficiency and storage. Precomputing features for the CC3M-Llava subset (563k pairs) using Llama-3 8B requires around 8 hours on a single A100 40GB GPU. Extracting DINOv2 features and optimizing the MLP-based projection network take approximately 1 GPU hour each, totaling roughly 10 GPU hours. This is paired with significant storage reduction from over 80GB for the raw data to just 12GB for precomputed features. For classification at inference, language targets are computed once per label set and inference cost is dominated by the vision encoder. Consequently, CLIP, LiT, and ShareLock have comparable FLOPs and latency when using the same backbone architecture (e.g., VIT-L/1402242244: ~ 160 GFLOPs), despite ShareLock's larger LLM.
Evaluation. We employ a comprehensive suite of evaluations to assess ShareLock's capabilities across a wide range of tasks. Based on the publicly available CLIP Benchmark (LAION AI, 2022), we gauge the models' zero-shot classification abilities across diverse datasets: ImageNet-Ik (Deng et al., 2009), ImageNet-R(endition
(Hendrycks et al., 2021b), AmageNet-A(dversattti S(ketch) (Wang et al., 2019), Oxford-IIIT Pets (Parkhi et al., 2012), Oxford Flowers (Nilsback and Zisserman, 2008), Stanford Cars (Krause et al., 2013), and FGVCAircraft (Maji et al., 2013). We also provide qual- itative text-to-image retricval results on ImageNet for CC3M-trained models. Moreover, the challenging compositionality Winoground task (Thrush et al., 2022) is explored.
(Hendrycks et al., 2021c), ImageNet-
Baselines and comparisons. We compare our straightforward and economical way of incorporating LLMs into VLMs to more conventional CLIP-like methods, particularly emphasizing data-efficient alignment approaches. Alongside the original ViT-B/16 variant of CLIP (Radford et al., 2021), test against several CLIP-like models trained on public datasets of different scales and with modified lcarning objectives that all share the standard dual-encoder architect ure where both image and lext encoders are trained from scratch with a contrastive loss (Fig. 4, left) (Fan et al., 2023; Gadre et al., 2023; Mu et al., 2022). Leveraging pretrained models, we evaluate how ShareLock compares to LiT (Zhai et al., 2022) and ASIF (Norelli et al., 2023) by reproducing these methods on smaller-scale datasets. LiT corresponds to the partially frozen setup in Figure 4 (center), where the vision encoder is locked but the text tower and projection are fine-tuned.
7

---

Published in Transactions on Machine Learning Research (01/2026)
Table 2: Open-vocabulary zero-shot classification accuracy on various datasets. Especially low-data regimes, the frozen LLM features (Llama-3 8B) utilized by ShareLock enable it outperform CLIP, LiT and ASIF baselines and achieves performances competitive with models trained on significantly more paired data, such L (384M).
Model
Dataset
Size
IN-1k
IN-R
IN-A
IN-S
Pets
Flowers
Cars
Aircraft
Avg
LIT
COCO
83k
21.4
35.5
21.4
18.9
23.5
7.0
2.0
2.1
16.5
ASIF
COCO
83k
9.4
14.4
6.9
7.0
1.3
2.8
6.5
Loak
COCO
83k
36.9
49.0
37.0
29.8
34.5
10.5
4.4
26.2
LIT
CC3M Subset
563k
44.5
70.0
58.3
39.5
25.5
34.4
2.5
2.4
34.6
ASIF
CC3M Subset
563k
21.6
27.7
24.4
14.9
11.7
6.4
2.3
2.1
13.9
ShareLock
CC3M Subset
563k
51.5
71.9
63.6
43.2
33.0
39.2
6.5
39.2
CLIP
CC3M
2.8M
16.0
17.6
3.6
6.4
13.0
10.8
0.8
1.4
8.7
SLIP
CC3M
2.8M
23.5
26.8
6.8
12.1
17.0
13.5
1.2
1.3
12.8
LaCLIP
CC3M
2.8M
21.3
23.5
5.0
10.6
15.8
15.7
1.6
1.6
11.9
LiT
CC3M
2.8M
46.8
72.8
59.4
40.8
31.1
42.4
3.7
2.6
37.4
Loake
CC3M
2.8M
54.5
74.7
65.9
46.0
36.0
38.9
41.3
DataComp
CPool-S
3.84M
3.0
1.3
4.0
1.6
1.4
2.4
CLIP
CC12M
12M
41.6
52.6
10.7
28.8
64.2
36.7
24.1
2.5
32.6
SLIP
CC12M
12M
41.7
55.2
13.8
30.7
56.7
34.1
22.4
3.0
32.2
LaCLIP
CC12M
12M
49.0
63.8
14.7
39.4
72.5
43.2
36.2
5.5
40.5
LiT
CC12M
8.5M
68.2
50.6
76.8
51.9
13.5
6.0
ShareLock
CC12M
8.5M
62.0
78.5
70.1
51.6
71.3
56.3
15.0
10.9
52.0
DataComp
CPool-M
38.4M
23.0
28.0
4.3
15.1
29.9
22,4
22.0
1.7
18.3
DataComp
CPool-T
384M
55.3
65.0
20.2
43.2
77.8
53.3
67.7
48.7
CLIP
Proprictary
400M
68.4
77.6
48.2
89.0
71.2
64.7
24.4
61.7
ShareLock follows the fully frozen setup in Figure 4 (right), freezing both vision and language encoders and learning only lightweight projection networks. For L.T baselines, we initialize the language encoder with pretrained BERT-Base weights (Devlin et al., 2019), following Zhai et al. (2022). When comparing ShareLock with LiT and ASIF, the same precomputed features (except for LiT's language inputs) are used.
5.2 Comparison to Conventional VLMs
Classification. LLMs can directly be leveraged profitably in rision-centric tasks and outperform conven- tional models trained on similar small-scale datasets as demonstrated by the IN-1k accuracies Table 2. ShareLock's 54.5% accuracy on CC3M substantially exceeds both CLIP (16.0%) and LiT (46.8%), despite the latter using the same vision features and fully fine-tuning the language component. Even with larger datasets like CC12M, where full fine-tuning becomes more viable, minimal transformations tations maintain a competitive advantage of 3% - 15% over LiT and CLIP. Compared to the training-free ASIF, optimizing a small number of parameters proves advantageous and forgoes the reliance on large and diverse reference datasets and extensive compute during inference.
top of LLM represen-
Beyond competitive performance on general-purpose classification, leveraging strong representations from pretrained models enables increased robustness out-of-distribution image inputs as seen in columns "IN-R" to "TN-A" of Table surpassing the robustness of the original CLIP model despite being exposed to fraction of the training data (8.5M vs. 400M).
The fine-grained nature of certain classification problems (cols. "Pet" to "Aircraft* in Tab. 2) demands larger-scale datasets with more diverse and nuanced concepts included as minute visual differences may be insufficiently captured the text space. Consequently, low performance on small datascts can be observed across all methods. Nonetheless, the LLM representations still contain visually valuable signals, enabling ShareLock to surpass other methods trained on the same data in 15/ cases and demonstrating effective utilization of intrinsic LLM knowledge generalize truly novel concepts.
Multilingual Understanding, Most popular multimodal datasets consist primarily of English captions. Consequently, VLMs like CLIP and LiT experience significant performance drops when performing inference in other languages, as shown Table on multilingual ImageNet LAION AI (2022). contrast, the broader and typically multilingual pretraining of LLMSs enables ShareLock to harness cross-linguistic image-text.
8

---

Published in Transactions on Machine Learning Research (01/2026)
Table Multilingual zero-shot classification accuracy. Leveraging extensive pretraining and consistent representations, ShareLock allows cross-lingual transfer on ImageNet without extra alignment.
Model
Datasct
[Size]
EN
CN
JP
IT
LiT
COCO
83k
21.4
0.2
0.2
3.6
Lock
COCO
83k
36.9
20.0
11.2
15.8
CLIP
CC12M
12M
41.6
0.1
0.1
7.9
LIT
CC12M
8.5M
59.9
0.1
12.9
ShareLock
CC12M
8.5M
62.0
38.7
19.8
39.3
DataComp
CPool-M
38.4M
23.0
0.2
0.3
4.7
DataComp CPool-L
384M
55.3
0.7
1.5
15.2
CLIP
Proprietary
400M
68.4
1.4
4.1
21.7
Table 4: Compositional reasoning on Winoground [Accuracy]. Strong frozen language features alone do not address systemic shortcomings inherent contrastive alignment approaches when comes spatial conceptual relationships, but enable ShareLock to outperform all alternative methods on image selection.
Model
Dataset
[ize]
Text.
Image
Group
Human
89.5
88.5
85.5
Chance
25.0
25.0
16.7
LIT
COCO
83k
21.3
7.3
3.5
ASTF
COCO
83k
18.8
5.3
ShareLock
COCO
83k
20.5
12.5
6.5
CLIP
CC12M
12M
22.3
9.5
5.3
LIT
CC12M
8.5M
22.0
6.5
4.0
ShareLock
CC12M
8.5M
25.0
12.5
9.5
DataComp
CPool-M
38.4M
25.0
8.3
6.3
DataComp
CPool-L
384M
27.0
9.5
7.0
CLIP
Propriatary
400M
30.8
10.8
8.3
consistencies, greatly mitigating the performance loss in non- Non-English languages. Even with substantially fewer training samples, ShareLock surpasses the original CLIP model, achieving an accuracy of 38.7% versus 1.4% for Chinese and 19.8% versus 4.1% for Japanese. In comparison, both LiT and CLIP models similarly trained on CC12M demonstrate nead-random performance in these languages. This transfer of capabilities makes the inclusion of LLMs especially attractive for low-resource languages with little available or high-quality multimodal data.
Compositionality. Late-fusion VLMs often struggle with nuanced textual and fine-grained compositional differences, as seen in benchmarks like Winoground (Thrush et al., 2022) and SugarCrepe (Hsich et al., 2023). Despite the intriguing linguistic and generative abilities of LLMs,, their representations fail to adequately reflect fine linguistic differences in the vision-language contrastive setting. While ShareLock improves image scores over CLIP (12.5 vs. 10.8) and thus more reliably selects the correct image given textual description, it still falls short of significant above-random performance and remains far from human-level capability. However, the low performance on compositionality tasks might partly be an architectural limitation, as recent works (Zhang et al., 2024; Jose et al., 2024) have indicated limitations of solely aligning language representation to the vision space, as suggested by Zhai et al. (2022). Thus, also applying transformations on top of vision features can be beneficial, especially retrieval and detail-oriented settings.
Data scaling. Figure 5 illustrates that ShareLock achieves comparable performance to CLIP and DataComp models while using orders of magnitude less data. Utilizing frozen LLM features is especially effective in low-data regimes, consistently outperforming conventional CLIP-like models, further underlining their visually- relevant semantic content and capacity to facilitate generalization. Additionally, tuning far fewer parameters enables substantially larger batch sizes (sce Table 5), which has been shown to improve contrastive lcarning performance (Zhai et al., 2022). Frozen backbones also enables the integration of large-scale LLMs into CLIP-like architectures while maintaining training efficiency.
9

---

Published in Transactions on Machine Learning Research (01/2026)
ShamLock (ouns
ImnogNeesl Perfermure
CLI
50
DataComp-
Table 5: Maximum batch size @80GB.
DataComp
Frozen backbones permit ShareLock to utilize siguificantly larger batch sizes.
Language Backbone
Method
BERT-base
Llama-3 (8B)
10"
10a
20'
102
10"
CLIP
1,600
Unique Captonn
LIT
2,450
LookeLock
61, 000
57, 000
Figure Scaling of image-text dataset size. Share- Lock outperforms other models despite using notably fewer datapoints.
CLIP
LiT
ShareLock
photo banana.
Sm:0R
0.66
(im
horseback riding.
Sm 045
(5ml 0.00
4m 0.54
Sim 0.2
Sim 3:5
cim 3:68
lighthouse caught in heavy seas.
Sm
STRE STI-IE22
STI
A photo of BMW
Sim:
9m:0447
5m:0.0
Skm
CITI CAR
235
230
Figure 6: Qualitative comparison. We show qualitative top-3 retrieval results for CLIP, LiT and ShareLock models trained on CC3M. Green border color indicates correctly retrieved samples.
5.3 Qualitative Results
In addition to quantitative evaluations, Figure 6 demonstrates ShareLock's strong text-image alignment across diverse prompts, showing advantages over CC3M-trained CLIP and LiT models for both fine- aue-grained (e.g., "a photo of BMW") and abstract (c.g., "f.... heavy seas", ")) queries.
5.4 Component Analysis
Choice of language model As the nature and quality of the frozen language features are of great significance the proposed architecture, we examine the choice of language model on the CC3M dataset. Reflecting the insights from investigation in Section 4, Table 6a highlights the potential of decoder-basod models for vision-language tasks. Although BERT encoders serve the starting point in LiT models, they perform poorly without fine-tuning. Similarly, all-roberta-vl (Liu et al., 2019) improves significantly but remains inferior to LLM-based representations, despite being highlighted by Maniparambil et al. (2024) for its high inherent visual alignment. In contrast, frozen decoder- based representations consistently surpass BERT-based ones, with gains of 40% to 350%, showcasing the richness of strong LLM representations from the more extensive pretraining and larger parameter counts. Even naive last-token-pooling is suitable for extracting visual information from LLMs and the more sophisticated multi-token embedding approach of NV-Embed (Lee et al., 2024) fails to materialize in consistent and notable performance gains over off- off-the-shelf Llama and Gemma models.
Choice of vision encoder As ShareLock agnostic the utilized vision encoder, we compare variants with differing architectures and supervision regimes Table 6b. Since language embeddings map the vision space, encoder choice crucial factor, as reflected in notable performance differences. Unlike DINOv2, the
10

---

Published in Transactions on Machine Learning Research (01/2026)
Table 6: Comparison unimodal backbones.
Strong and comprehensive features rule generalization.
(a) Comparison of language backbones. Decoder- based LLMs facilitate generalization most. Popular en- coders lag behind when not tuned.
(b) Comparison of vision backbones. Self-supervised backbones transfer best. across dataset and benefit from increased model capacity.
Lang. Model
IN1k
IN-R
IN-A
Pets
Cars
Vision Model
INIk
IN-R
IN-A
Pets
Cars
Doc. Enc.
BERT-
44.3
71.7
59.6
18.3
2.2
Self-sup. Sup.
ResNet101
5.1
all-roberta-vl
49.5
72.7
61.5
27.4
4.3
Con
64.7
60.6
54.9
28.8
8.4
VIT-L
55.0
38.0
16.0
29.3
6.1
Llama-3 8B
54.5
74.7
65.9
36,0
7.5
Gemma-2 9B
56.4
76.0
68.1
49.9
6.9
DINOv2-8
45.6
49.7
33.3
NV-Embeed-22
57.0
75.9
46.3
8.0
-B
52.1
64.1
50.9
43.1
4.4
-L 54.5
74.7
65.9
36.0
7.5
-G
56.3
77.7
69.9
36.2
6.6
other models were only trained on ImageNet and exhibit lower robustness and generality on other datasets, highlighting the benefits broad pretraining across diverse concepts even without explicit supervision. One advantage of CLIP is its favorable scaling characteristics when increasing the size of the vision encoder (Radford et al., 2021). To validate if comparable trends are present in ShareLock, we vary DINOv2- baseed backbones ranging from Small to Giant vision transformers (cf. Tab. 6b). Indeed, ShareLock also profits from scaling up the vision backbones with the average scores increasing by 33% and 11%, when moving from the Small to the Base and from the Base to the Large DINOv2 models, respectively. However, the benefits of scale start to level off thereafter, and only marginal differences are present when utilizing representations from the Giant vision encoder.
Choice of projection network depth. the only learnable parameters, the choice of the projection networks Pixt and Ping crucial for aligning the backbones and generalizing across downstream tasks. Table reveals that projecting vision features into the language embedding space consistently underperforms compared to the reverse direction. This suggests that the vision embedding space is more semantically coherent and continuous, allowing the model to belter approximate the lalent concept manifold, particularly in low-data regimes. Conversely, language embeddings appear to encode non-semantic, visually irrelevant information (e.g., syntactic features), which are difficult to predict solely from visual inputs. Similarly, introducing projection layers atop both backbones and thereby removing the fixed reference space leads to reduced performance and signs of overfitting.
Table 7: Architectural choices of projection network. Generalization is best with sufficient transforma- live power (i.e., MLP layers) on top of the language backbone and direct projection into the vision space.
Poxll
P Ping
IN1k
IN-R
IN-A
Pets
Cars
Aircraft
23.8
23.7
33.4
10.0
2.8
2.0
22.4
22.4
29.1
2
52.8
74.1
65.2
33.8
7.2
8.2
4
5-1.5
74.7
65.9
36.0
7.5
6.7
55.1
74.0
65.8
35.8
8.5
2
48.4
46.8
57.0
20.2
6.6
9.7
4
43.7
41.1
48.6
13.7
7.1
6.6
6 Conclusion
We systematically investigate the visual capabilitics and inherent alignment of unimodal language models. Our analysis demonstrates that general LLM quality, measured by MMLU-Pro, correlates with visual aptitude and that decoder-based models effectively generalize across modalities. By integrating off-the-shelf LLMs into a lightweight CLIP-like architecture, we leverage their large-scale pretraining, intrinsic knowledge of the visual world, and multilingual capabilities, achieving competitive performance with conventional VLMs trained on significantly larger datasets. Our findings offer a deeper understanding of state-of-the-art language models and highlight their potential for broader adoption in vision-centric applications.
11

---

Published in Transactions on Machine Learning Research (01/2026)
7 Acknowledgments
The authors thankfully acknowledge the HPC resources provided by the Erlangen National High Performance Computing Center (NHR@FAU) of the Friedrich-Alexane BayernKI project v115be. BayernKI funding is provided by Bavarian state authorities. We thank SURF for providing GPU cluster access and support during this project. This work was supported in part by the Pioncer Centre for AI, DNRF grant number P1 and TNO's research programs Appl.AI and DSS. We gratefully acknowledge the ELLIS Unit Amsterdam for providing funding for a research visit to Copenhagen.
Universität Brlangen-Nürnberg (FAU) under the
References
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Daniel Ho, Jasminc Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario M Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Jayant Joshi, Ryan C. Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Houn Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego M Reyes, Pierre Sermanct, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, F. Xia, Tod Xiao, Peng Xu, Sichun Xu, and Mengyuan Yan. Do can, not say: Grounding language in robolic affordances, In CoRL, 2022.
Danny Driess, Fei Xia, Mchdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy ZZnng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. 2023.
Wayve, Lingo-1: Exploring natural language for autonomous driving, 2023.
Pratyusha Sharma, Tamar Rott Shaham, Manel Baradad, Stephanic Fu, Adrian Rodriguez-Munoz, Shivam Duggal, Phillip Isola, and Antonio Torralba. A vision check-up for language models. In CVPR, 2024.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. ICML, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Christoph H. Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. CVPR, 2009.
Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Woo Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. In ICLR, 2024.
Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yu Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre-training (clip). In ICML, 2022.
Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip H. S. Torr, Adel Bibi, Samuel Albanic, and Matthias Bethge. No 'zcro-shot' without exponential data: Pretraining concept frequency determines multimodal model performance. 2024.
Shubham Parashar, Zhiqiu Lin, Tian Liu, Xiangjue Dong, Yanan Li, Deva Ramanan, James Caverlee, and Shu Kong. The neglected tails in vision- models. In CVPR, 2024.
Prasanna Mayilvahanan, Roland S. Zimmermann, Thaddäus Wiedemer, Evgenia Rusak, Attila Juhos, Matthias Bethge, and Wieland Brendel. In search of forgotten domain generalization. In ICLR, 2025.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
12

---

Published in Transactions on Machine Learning Research (01/2026)
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021a.
Sam Bowman. Eight things to know about large language models. 2023.
Sarah Pratt, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. ICCV, 2022.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICLR, 2023.
Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in bottle: Language model guided concept bottlenecks for interpretable image classification. CVPR, 2022.
Oindrila Saha, Grant Van Horn, and Subhransu Maji. Improved zero-shot classification by adapting vims with text descriptions. In CVPR, 2024.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. CVPR, 2022.
Zaid Khan and Yun Fu. Contrastive alignment of vision to language through parameter-efficient transfer learning. In The Eleventh International Conference on Learning Representations, 2023.
Antonio Norelli, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodola, and Francesco Locatello. Asif: Coupled data turns unimodal models to multimodal without training. NewrIPS, 2023.
Mayug Maniparambil, Raiymbek Akshulakov, Yasser Abdelaziz Dahou Djilali, Mohamed El Amine Seddik, Sanath Narayan, Karttikeya Mangalam, and Noel E O'Connor. Do vision and language encoders represent the world similarly? In CVPR, 2024.
Le Zhang, Qian Yang, and Aishwarya Agrawal. Assessing and learning alignment of unimodal vision and language models. arXiv preprint: 2412.04616, 2024.
Yongqin Xian, Christoph Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning a comprehensive evaluation of the good, the bad and the ugly. PAMI, 2017.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. Caltech-uesd birds-200-2011. Technical report, California Institute Technology, 2011.
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Maxime Oquab, Témothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szalraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. TML.R, 2023.
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Ny-embed: Improved techniques for training 1lms as generalist. embedding models, 2024.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krishevsky, Πya Sutskever, and Ruslan Salakhuldinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 2014.
LATON AT. Clip benchmark. http:///gittuuu.co/Aottt------
2022.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Zach Nussbaum, John X. Morris, Brandon Duderstadt, and Andriy Mulyar. Nomic embed: Training reproducible long context text embedder. arXiv preprint: 2402.07 002.01613, 2024.
Yinhan Liu, Myle Ou, Naman Goyal, Jinglei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint: 1907.1 1692, 2019.
Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Danicl Cer, and Yinfci Yang. Se Scalable sentence encoders from pre-trained text-to-text models. In ACL, 2022.
Yi Tay, A new open source flan 20b with ul2, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandoy, Abhishck Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, el al. The llama 3 herd of models. aiv preprinl: 2407. 2024.
Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Julictte Love, et al. Gemma: Open models based on gemini research and technology. Xiv preprint: 2403.0822 08295, 2024.
Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan. Repetition improves language model embeddings. aiviv preprint: 2402. 15449, 2024.
Orion Weller, Kathryn Ricci, Mare Marone, Antoine Challin, Dawn Lawrie, and Benjamin Van Durme. Seq vs seq: An open suite of paired encoders and decoders. arxiv preprint: 2507.11412, 2025.
Yubo Wang, Xueguang Ma. Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max KU, Kai Wang, Alex Zhnang, Rongqi Fan, Xiang Yue, and Wenhu Chen, Mmlu-pro: more robust and challenging multi-task language understanding benchmark. In NewrIPS, 2024a.
Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open 1lm leader- board v2. attps://huggingface co/
11m_ lem__eedderbbrdd 2024.
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server, 2015.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, In NewrlPS, 2023.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale imago-text pro-training to recognize long-tail visual concepts. In CVPR, 2021.
Dan Hendryeks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of ont-of-distribatiio generalization. ICCV, 2021b.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021c
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In NeurIPS, 2019.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. The oxford-iiit pet datasct. CVPR, 2012.
Maria- Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. Tn 2008 Siath Indian Conference on Computer Vision, Graphics & Image Processing, 2008
14

---

Published in Transactions on Machine Learning Research (01/2026)
Jonathan Krause, Michael Stark, Jin Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In IEEE TOCV Workshops, 2013.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In CVPR, 2022.
Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. In NeurIPS, 2023.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mchdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. NewrIPS, 2023.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. In ECCV, 2022.
Cheng- Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. In NeurIPS Datasets and Benchmarks Track, 2023.
Cijo Jose, Théo Moutakanni, Dahyun Kang, Federico Baldassarre, Timothée Darcet, Hu Xu, Daniel Li, Marc Szafraniec, Michael Ramamonjisoa, Maxime Oquab, Oriane Sim Sim'eoni, Huy V. Vo, Patrick Labatut, and Piotr Bojanowski. Dinov2 meets text: A unified framework for imago- and pixel-level vision-langunge alignment. Xix prepriat: 2412. 16334, 2024.
Microsoft. Phi-3 technical report: A highly capable language model locally on your phone, 2024.
Peng Wang, Shuai Bai, Sinan Tan, Shijic Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's 's perception of the world at any resolution. arXiv preprint: 2409. 12191, 2024b.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging LLM-as-a- judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
XTuner Contributors. Xtuner: A toolkit for efficiently fine-tuning 1mm. xtuner, 2023.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lueas Beyer. Sigmoid loss for language image pre-training, In ICCV, 2023.
15

---

Published in Transactions on Machine Learning Research (01/2026)
A Reproducibility Statement
We acknowledge and emphasize the importance of reproducibility in our work and take active measures to facilitate reproducibility efforts. Besides providing comprchensive documentation of our methods throughout the main paper, with additional details in the supplementary materials, we will publish source code for the proposed ShareLock model.
Our use of existing models aligns with their intended purpose and is carried out in an academic setting. Any data used follows its original access conditions, and all artifacts produced are intended for research purposes only. Only publicly available resources and scientific artifacts are used.
B Limitations
While we compare a wide range of publicly available encoder- and decoder- based language models, attributing concrete performance differences on mainly architectural differences is not possible due to the different. pretraining objectives, training datasets, and number of parameters used in these models. As the primary focus of this work is on highlighting the visual alignment inherent. to unimodal language models and their incorporation into data- and compute-effiiien vision-language models, our investigation is limited to datasots with up to 12M image-caption pairs. Scaling ShareLock to larger datasets for further performance improvements is left for future work. Although pre-oomputing features only once possible due to locked vision and language backbones, using LLMs with billions of parameters significantly increases computational costs of forward passes compared to smaller conventional encoder architectures.
C Acquisition of Textual Class Representations
Besides the template-basod targets proposed by Radford et al. (2021) that solely substitute the respective class names, we generate more comprehensive auxiliary information about classes (e.g., Wikipedia-style articles) using the instruction-tuned version of the Llama-3-8B model and acquire human-curated information from Wikipedia (details provided in C).
Class representations are essential for facilitating the knowledge transfer between classes in the traditional definition of vero-shot learning. Compared to attribules or other forms of class semantics, language-based class representations are more conveniently accessible at various scales and may come in diverse manifestations. The advent of LLMs adds further possibilities for generating and obtaining such auxiliary information. The following paragraphs specify the respective properties and acquisition process. Here, all LLM-based class representations are generated using the instruct-tuned version of LLama-3 8B.
Class Names. A set of 80 human-engineered prompt templates in the style of "a photo of <class name>" are adopted from Radford et al. (2021).
Wikipedia Page. Being a comprchensive and mostly factually correct source of information, Wikipedia constitutes an interesting source of auxiliary information in the context of zero-shot classification. To obtain class-article correspondences, class names are automatically matched with page names, after which additional manual quality checks are performed. Nonetheless, an ideal match does not always exist due to high class specificity or generality, in which case superordinate articles are considered or template- based fallbacks are employed.
LLsebased Wikipedia Style Articles. Despite being specifically prompted for articles mimicking Wikipedia, the Llama-3-generated texts tend to show significant differences in style compared to their real counterparts.
As the lengthy nature of Wikipedia(-style) articles might dilute the information content captured by the language embeddings, the texts are split into individual sentences, which are used as targets during training. For all types of class representations, predictions are made by aggregating class scores through averaging over all individual class-specific texts.
In In-Depth Analysis of Visual LLM Generalization
As outlined Scction 4, we find that the general capability of language models is strongly correlated with their ability to perform well on visual tasks. While this is also true for members of the Phi-3 family of models, their absolute visual generalization scores are notably lower compared to models of similar size and capability,
16

---

Published in Transactions on Machine Learning Research (01/2026)
CLIP
45
Visual ceneralization Score [%]]
40
35
Sentence- T5-XXL
Model Family
T5-XXL
Llama-3
Phi-3
30
Qwen2
Model Variant
Qwen2.5
Base
gemmaa-2
ModerBERT
Instruct
Falcon3
25
0
10
20
30
40
50
Massive Multitask Language Understanding (MMLU-Pro) Score
Figure
Visual generalization performance relative MMLU-Pro scores. Model capability on
language tasks prodictive of visual transfer performance of LLMs (Pearson-r: 0.768 and 0.523 (excl./incl. Phi-3 models)). Dot size is proportional to the LLM's parameter count.
seen in Figure 7. This discrepancy likely illustrates the effects of the extensive data curation and synthetic data creation utilized in Phi-3, which might remove visual information to favor tokens that promote reasoming abilitics. Thus, a lack of exposure to sufficient factual knowledge about real-world conditions may impede the formation of visually informed represenlations.
As seen in Figure and Table 12, the text encoder taken from the CLIP ViT-B/16 model constitutes a strong baseline that is not yet reached by any of the current state-of-the-at LLMs. However, considering the 400M image-text pairs makes the strong generalization abilitics unsurprising.
explicit vision alignment On the contrary, obtaining score of 44.9 the best performing unimodal model, Gemma-2 9B (Mesnard et al., 2024) is close to CLIP's 46.7 despite no multimodal exposure. This further underscores the remarkable degree of visual aligument inherent to decoder-based language models.
LLMs are often subject to additional task-specific fine-tuning. Table compares different models and their derivatives tuned on instruction and multimodal data. While vision-tuned variants are available for Phi-3 (Microsoft, 2024), Qwen2 (Wang et al., 2024b), and Vicuna v1.5 (Zheng et al., 2023), XTuner's LLa VA model (XTuner Contributors, 2023) constitutes the source for the visual Llama-3 variant. Across all models, with the exception of Llama-3, the impact of fine-tuning is minor, typically shifting performance by only few decimal points in the case of Phi-3 and Vicuna. Interestingly, Llama-3's performance doclines significantly after fine-tuning (-1.7 and 100.4 for instruction and visual tuning), contrasting with the generally stable results of other models. Neither instruction- based nor visual fine-tuning shows a clear and consistent advantage improving overall performance. These insights are also reflected in Figure and Table 12. Ultimately, the base model's architecture and training regime are more significant in determining performance than post-hoc fine-tuning strrategies.
17

---

Published in Transactions on Machine Learning Research (01/2026)
Table 8: Visual generalization ability of different fine-tuning regimes. Fine-tuning has minimal impact on visual alignment of LLM representations. Llama-3 is notable exception with significant performance decrease for instruct- and vision-tuned variants.
Llama-3 SB
Phi-3 Mini
Qwen2 7B Llama-22 Saannaa///icun /iiuua
None
44.4
n/a
40.2
42.5
Instruct
42.7
36.9
41.8
42.3
Visual
34.0
37.0
40.3
42.6
MLP
Figure Text features. We obtain the final text features by processing the last caption token with an MLP. This allows avoid- ing expensive forward passes of the LLM during training by pre- computing and storing the fea- tures (x).
Frozen
LLM
Caption
E Additional Ablations and Comparisons
E.1
Projection Network Architecture
The multi-layer perceptron (MLP) projection networks of ShareLock as introduced in Section 3 are conceivably simple. As these are the only unfrozen and tunable parts of the model architect ure and thus responsible for aligning vision and language inputs, they are of particular significance to aptly process and transform the inputs. Following Zhai et al. (2022), no transformation to the vision inputs is applied for any of the architectures. With hidden size of 4096 and four layers, the MLP processing the language features comprises approximately 53M parameters.
In addition to the straightforward MLP-based networks, also more sophisticated Transformer-based archi- tectures are inspired by recent works. First introduced as part of the BLIP-2 model (Li el al., 2023), the Q- Q-Former is a lightweight Transformer- based model that extracts features from an input modality using cross-attention with learnable query tokens. Similarly, albeit introduced in a different context, NV-Embed (Lee el al., 2024) uses latent atlention layer to pool language tokens and receive a global embedding. Slight adjustments are made to both baseline architectures to better suit late-fusion vision-language modeling, which we will denote as Q- Q--ransformer and NV- Transformer. While features are extracted via last-token-pooling (see Figure 8) when used with MLP projection networks, all tokens of the input sequence are considered for the alternative architectures. The hyperparameters were selected based on the implementation details suggested in the original publications and to approximately match the MLP baseline in learnable parameter count. Both the Q-Transformer and the NV-Transformer projection networks have token dimension of 1024 in the Transformer parts of the models, eight learnable queries (Q- (Q-Transformer), and key/values (NV-Transformer) Whereas the Q- T-Tannsfommer consists of 3 blocks and attention heads, Tr-araosfrrerr comprises a total of four layers with eight cross-attention heads each.
The choice of projection network architecture is compared in Table 9. The evaluated models use DINOv2- ViT-B/1e 14 their vision backbone. While single architecture consistently scores best, the MLP-based ShareLock configuration performs competitively compared to NV- the evaluation dalasets. Additionally, Transformer-baseed architectures entail increased computational complexity due to the more evolved attention mechanism and processing of more tokens, making MLPs an attractive choice from an efficiency perspective as well. These results suggest that the additional information contained across all tokens of an input is not significantly more adjuvant compared solely considering the last token representation as is done with the ML.P.
and Q-Transformer throughout
18

---

Published in Transactions on Machine Learning Research (01/2026)
Table 9: Comparisons of the projection network architectures tuned as part of ShareLock training.
Simple MLPs perform competitively compared to more advanced Transformer- based architectures.
Architecture
IN1k
IN-R
IN-A
Pets
Cars
ESAT
NY-Transformer
44.2
35.8
42.1
25.4
4.7
30.3
Q-Transformer
51.8
61.3
48.1
36.8
36.7
MLP
52.1
4.4
27.9
E.2 Loss Function
The use of the Sigmoid Loss (SigLIP) has unlocked further efficiency and performance enhancements in the standard CLIP regime (Zhai al., 2023). However, no substantial gains are found when using SigLIP in the ShareLock framework as presented in Table 10.
Table 10: Comparison the loss function used in ShareLock training. The vanilla CLIP loss trumps the sigmoidal SigLIP loss.
Loss Function
IN1k
IN-R
IN-A
Pets
Cars
ESAT
SigLIP Loss
49.5
61.6
49.5
30.8
31.4
CLIP Loss
52.1
50.9
43.1
27.9
F Extended Visual Generalization Results
Due limited space, the reported results on visual generalization in Table 1 and Figure 1 of the main paper are averaged accuracies across five seeds and four datasets (AWA2 (Xian al., 2017), CUB (Wah et al., 2011), FGVCAircraft (Maji et al., 2013), and ImageNet*)) For increased transparency, the results are presented without dataset-level aggregation in Tables 11 and 12. Moreover, we include more detailed results corresponding to Figure 3 in Table 13. In addition to last token aggregation for decoder- based models, we also evaluate applying the same mcan pooling across all tokens as utilized for encoders. However, last token pooling results in superior performance.
G Supplementary Qualitative Results
Figure provides additional qualitative insights into the retrieval ability of CLIP, LiT, and ShareLock models trained on CC3M. ShareLock demonstrates visual understanding across a wide array of domains and levels of abstraction.
19

---

Published in Transactions on Machine Learning Research (01/2026)
Table 11: Visual generalization capability of various language models. Decoder-based language models outperform encoder-based architectures across all types of input data. Llama-3 8B is used for LLM generated Wikipedia articles.
Class Repr
Type
Language Model
AWA2
CUB
Aircraft
IN+
Avg
Class Names
Dec
Llama-3 8B
50.1 8.3
42.7
29.7 5.5
36.7
39.8
Llama-3.1 8B
6.3
44.2
33.0 2.9
3.8
41.2
Gemma 7B
6.0
42.4+
31.4
0.6
41.0
Gemma-2 9b
52.0 5.5
45.7
44.6
39.0
42.8
NV-Embed-v2
3.7
4.9
33.8 + 5.3
32.2 0.4
39.4
Enc
T5-3b
7.6
3.7
2..6
27.9
32.8
BERT-Large
26.5 18.5
6.5
8.5
7.8
14.2
Flan-UI:
5.8
3.14
26.1
0.9
37.9
ModernBERT-embee
3..
31.9
25.6
1.6
19.6
28.9
sentence-t5-xxl
6..5
3..4
24.1
2.14
36.6
all-roberta-vl
4.8
31.9 1.
22.8 2.0
2..1
31.0
LLM Wiki Page
Dec
Llama-3 8B
6.6
3.0
3.1
3.7
43.9
Llama-3.1 8B
59.0
3.5
3.17
0.6
43.4
Gemma 7B
9.6
41.7
$1.1
3..
42.8
Gemma-2 9b
6.7
2..1226
3.9
0.7
45.0
NV-Embed-v2
2.1
47.1 1.3
36.4 16.42.8 2.8
33.5 0
42.1
Enc
T5-3b
9.44
3.0
3.14
31.9 1.3
36.1
BERT-Large
2.7
16.7 2.5
13.2 2.3
16.4
Flan-UL2
57.7 12.0
3.0
34.3
35.6
41.0
ModernBERR-emmee
4.9
2..6
20.6
23.9
sentence-t5-xxl
5.6
2.0
32.1
39.1
all-roberta-v1
46.2 6.5
36.9
2.24
25.3 5
34.5
Wikipedia
Dec
Llama-3 8B
7.8
41.5
4.2 3.2
0.5
44.5
Llama-3.1 8B
7.8
4.8
3.0
0.4
42.8
Gemma 7B
4.6
4.3
3.6
3.3
43.5
Gemma-2
8.4
4.2
44.3
0.5
44.5
NV-Embed-v2
5.7
2.
35.6
43.6
Enc
T5-3b
6..4
348
34.2 3.9
29.0
39.6
BERT-Large
47.0 16.3
8.0
27.7
13.1
24.0
Flan-UL2
9.2
4.19
35.9
0.5
44.0
ModemnBERT-embee
4.7
2.65
1.44
18.3
23.2
sentence-t5-xxl
6.74
4..0
6.7
312
42.8
all-roberta-vl
53.3 4 5.0
39.0
32.8 4: 4.6
31.8
39.23
20

---

Published in Transactions on Machine Learning Research (01/2026)
Language Model
AWA2
CUB
Aireraft
IN+
Avg
CLIP-B/16 Text Encoder
63.0 + 5.2
51.5 2.2
37.1 + 3.4
35.3
0.6
46.7
Qwen2-0.5B
40.8
27.2
25.6 3.5
24.1
0.6
29.4
Qwen2-0.5B-Instruct
40.9
2.0
2.3
23.9 0.6
29.8
Qwen2-1.5B
62.8
33.9 1.7
3..
30.6
37.2
Qwen2-1.5B-Instruct
48.1
33.6
6.7
0.8
35.9
Qwen2-7B
52. 5.8
38.6
29.6 5.3
0.7
39.1
Owen2-7BB-Insrrun
3.2
4.9
5.8
35.3
38.8
Qwen2.5-0.5B
4.5
2.2
2.0
1.2
30.6
Qwen2:55-.5BB-Issruue
49.0
31.4
24.2 4.3
24.3
32.2
Qwen2.5-1.5B
5..5
35.9
2.11
2.9
35.8
Qwen225-11.B-Instrree
58.3
37.
2.8
29.3
37.8
Qwen2.5-14B
4.7
47
33.8 1.4
0.5
41.4
Qwen2.5-14BBInnsruue
6.2
46.7
32.5
0.5
40.8
Qwen2.5-82B
55.3
47.8
33.7
0.6
43.7
Owen2.5-33B-Instrruee
4.0
1.3
3.0
3.7
43.4
Qwen2.5-3B
52.4
4.1
3.4
0.7
38.6
Dwen2.5-3B-Insttee
54.8
5.0
3.4
3.5
38.7
Qwen2.5-7B
2.2
41.7
2.6
3.80
40.9
1wen2255-7BInssruue
2.1
3.2
2.49
3.78
40.9
gemma-2-2h
4.2
3.2
5.4
3.83
gemma-2-2bas 2-2bbit
7.6
5.3
4.0
0.6
41.5
gemma-2-9b
3.6
2.6
6.2
0.4
43.5
gemma-2-9b-it
59.6
4..1
1.2
0.5
44.3
Meta--1am-
4..0
31.8
4.5
3.7
35.3
Meta-LJamma332211--nnste
8.3
3..4
2.77
3.71
37.0
Meta-Llama-3.2-3B
4.2
3.89
32.
3.7
37.8
Meta-Llama-332-9B-1mm
13.8:10.99
35.7 3.67
33.0 4.8
3..0
39.1
6.4
4.70
36.8
36.8
0.6
41.8
Meta-Llamaa-31188B
5..1
1.94
3.2
3.72
41.6
2hi-3-medlium1128k-nntttuu
5.5
16.1
3..4
3.41
34.3
Phi-3-mmelllm-4--instrrtttt
38.2
3.5
3.0
0.4
34.5
Phi-3-mini-128kk-nsttnre
6.1
2.8
2.2
1.2
30.1
Phi-3-minni44cinsttrect
35.9
3.5
5.0
1.8
29.9
Phi-3.5-minni-nstruee
37.0
1.
27.1 4.5
2..0
30.3
Falcon3-10B-Base
6.8
1.5
42.7 2.8
3.4
43.0
Falcon3-100-IInttruct
7.4
4..0
32.8 5.7
35.3
43.3
Falcon&-1BB-Base
8.0
1.41
5.8
0.7
33.8
Falcon3-313 3---ease
48.7 5.6
3..9
21.9 9.7
2.81
32.6
Balcon3-3BB-Istrueee
47.2
29.3 3.0
20.9 12.0
2.7
31.7
Falcon3-7B-BB
56.9
44.0 =
35.1 4.2
35.1
42.8
Table 12: Visual generalization performance across language models. Larger and more capable
models within family facilitate better generalization in visual tasks and thus boast increased visual alignment
21

---

Published in Transactions on Machine Learning Research (01/2026)
Table 13: Decoder vs. encoder language models. Using the Ettin model family (Weller et al., 2025), we compare the effects of pretraining objectives on visual alignment with consistent training data and model sizes. In these experiments, decoder-based models consistently outperform encoders.
# Params. [
Language Model
AWA2
CUB
FGVCAircraft
ImageNet*
Average
17
Decoder (Avg)
31.2
20.9
Decoder (Last)
29.7
22.1
Encoder (Avg)
20.6
27.7
16.7
12.4
19.4
32
Decoder (Avg)
26.7
29.4
16.0
14.2
21.6
Decoder (Last.)
36.5
24.3
9.9
14.6
21.3
Encoder (Avg)
20.4
30.9
16.5
13.8
20.4
68
Decoder (Avg)
32.0
27.6
19.9
16.9
24.1
Decoder (Last)
37.5
21.5
20.7
17.8
24.4
Encoder (Avg)
21.1
30.2
18.5
14.7
21.1
150
Decoder (Avg)
38.3
29.0
16.6
19.3
25.8
Decoder (Last)
39.8
26.5
24.4
20.6
27.8
Encoder (Avg)
33.1
30.5
16.3
17.8
24.5
400
Decoder (Avg)
36.9
32.1
21.3
21.7
28.0
Decoder (Last)
37.4
28.2
26.4
24.3
29.1
Encoder (Avg)
35.3
30.0
17.8
19.6
25.7
1000
Decoder (Avg)
48.4
32.8
21.9
25.5
32.1
Decoder (Last))
31.7
20.4
Encoder (Avg)
32.3
29.6
22

---

Published in Transactions on Machine Learning Research (01/2026)
CLIP
LIT
ShareLock
photo of banana.
person horseback
ricing.
Gm
5m:0.4
5mm
Bm. 633
5im
3.1
lighttouss caught heavy
SmIDE 04
0.46
Sm:00.0
SmI 0.64
mm 0.50
3m 3M9
387
3,99
photo BMW
A car parked on the street.
Dmm
3.63
lonely bench quiet park
5m:000
0m:00:
1km: 6.72
0.57
ckmi 0.53
Fm 0.79
Sum: 3.70
cm 2.00
helicopter landing building.
Sm:a0
3.51
A man fishing by a lake with mountains in the background
Two different breeds dogs sitting next to each other.
Smines
5m
0.50
Sam 0.14
5.01
3.24
round object.
SmI
04:040
0m::0:4
DiI 8.55
0.00
Sasc: 3.5L
3:50
photo includes
A vintage-looking photo.
5m::0:4
24T1
Skmi..33
Sin:
3.50
Simc 2.43
photo broken object.A funny-looking hot air balloon.
Gm:053
0m:05
6 Emi0.53 0.53
FIm: 0.53
5mm 0153
FMI 330
5mn 3.32
Fim: 3.33
Figure Qualitative comparison on text-to-image retricval (ImagcNet-Ik).
23


Reviews:
[{"review_id": "QNOYgOx3HL", "reviewer": "Reviewer_7e5z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- This paper explores whether stronger large language models exhibit better alignment with vision representations.\n- They proposed Sharelock, a simple framework that freezes both a pretrained vision encoder and a decoder-only LLM and only trains a projection from text space.\n- With this setup, the authors show that decoder-only LLMs achieve higher visual alignment and zero-shot classification accuracy than the traditional encoder-based text models, even with small paired datasets.\n- The work further established a positive correlation between LLM language proficiency and cross-model generalization performance, suggesting that linguistic capacity transfers to visual grounding.\n- The paper also highlights strong multilingual transfer capabilities, achieving notable improvements on non-English zero-shot classification tasks without additional training.\n- Key strengths include a clean and controlled experimental design, extensive comparisons across text models, and a practical low-compute recipe for vision-language alignment.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Claims\n\n1. Decoder LLMs align better to a frozen vision space than encoder text models.\n2. A small text-side head on frozen backbones yields strong zero-shot results in low-data regimes.\n3. Language proficiency correlated with visual generalization.\n\n- The evidence presented in the paper is clear and convincing.\n- The authors conduct well-controlled experiments that isolate the role of the text encoder by freezing both the vision and language backbones.\n- The positive correlation between language ability and visual alignment is consistently observed across various models and datasets.\n- Ablation on projection depth, data size, and multilingual transfer further support the main claims.\n- However, some aspects are overstated. The reported efficiency (“under one GPU-hour”) excludes the heavy cost of LLM feature extraction.\n- Also, the compositional benchmarks reveal limitations, which they attributed to the architecture choice of LLMs. Despite this, the empirical evidence substantiates the primary findings that decoder-based LLMs provide more transferable visual alignment than prior encoder text models.", "audience_yn": "Yes", "audience_explanation": "- Yes, the paper would be interesting to a broader segment of the TMLR audience, particularly those studying vision-language models, V-L alignment, representation learning, and the integration of the large language models into perception tasks.\n- The finding that decoder LLMs inherently encode more visually grounded semantics than the traditional text encoders offers a simple but powerful insight for people developing data-efficient multimodal systems. \n- The multilingual and low-data generalization results also have practical relevance for resource-constrained and cross-lingual applications.", "requested_changes": "1. Clarification of the compute claims, either abstract or methodology, can be revised to report the full cost, including LLM feature precomputation, rather than only the projection training time.\n2. Need for stronger compositional evaluation, include other compositional benchmarks like ARO, SugarCrepe. To support the claims about the visual-semantic alignment and explain the reasoning limitations over compositionality.\n3. Details about how multilingual label translations were normalized.\n4. Include encoder-only models matched in parameter count and context window to ensure that the decoder outperforms encoders; the conclusion is not confounded.\n5. Add a detailed discussion on why decoder-based language knowledge fails on compositional reasoning tasks.", "broader_impact_concerns": "- The work poses minimal ethical risks in itself, as it trains lightweight alignment layers on frozen, publicly available models.\n- Improved multilingual alignment could amplify existing biases in LLM text representations when projected into vision space.\n- The author could strengthen the broader impact section by briefly analyzing fairness implications and potential downstream misuse.", "additional_comments": "This is a clean, empirical paper with thoughtful controls and clear takeaways. It solidifies an emerging trend that high-capacity LLMs, even without fine-tuning, possess semantic benefits for visual grounding. The paper's simplicity is both its strength and its limitations. The compositional and reasoning gaps are not explored. With improved transparency around compute and additional diagnostics, and with sound reasoning, it will be beneficial.", "extra_scores": {}, "extra_text": {}}, {"review_id": "VyGDp0cSys", "reviewer": "Reviewer_hAtV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Contributions**\n1. The authors evaluate the visual alignment inherent to language models using probing.\n2. The authors find that decoder-based LLMs are effective sources of visual knowledge, better than encoder-based models.\n3. The authors propose ShareLock, a frame work to integrate LLMs into lightweight VLM and achieve robust generalization results.\n\n**Strength**\n1. The paper structure is clear and easy to follow.\n2. The experiments and analysis are comprehensive. The design of ShareLock is simple yet effective.\n3. The findings that decoder-based LLMs are more effective than encoder-based models in visual generalization is interesting.\n\n**Weakness**\n1. In experimental setup 3.3, the authors claim that one variant of DINOv2 models are used. However, it is very likely that this visual backbone have been trained on the seen classes data, even though in a self-supervised manner. This contradicts the motivation of \"enforcing disjointness between seen and unseen concepts\" on page 1.\n2. The \"inherent visual alignment\" (section 5, page 6) is not precise. To show the \"inherent\", the ideal way is to illustrate a projection layer on non-LLM text representations perform much worse. Comparing among LLMs does not show there is inherent alignment (table 1, section 4).\n3. For experiments in section 5, the baselines are using different visual backbones from ShareLock (CLIP v.s. DINOv2). At least it is not clear from \"Baselines and comparisons\" section. If the visual backbones are not the same, the comparison is not fair.\n4. Some figures and tables lack clarity. For example, in Figure 1, it is not clear how Visual Generalization score is computed. In Figure 3, there is no LLM but in text the authors describe \"As illustrated in Figure 3, this design places the LLM at the core of the model ...\". In Table 5, it is not clear what BERT-base and Llama-3 8B represents.", "claims_and_evidence_yn": "No", "claims_explanation": "See weakness 1, 2 and 3. The major concern is in experimental settings:\n\n1. It is not clear whether pre-trained DINOv2 has seen *unseen class* data or not.\n2. The \"inherent visual alignment\" is not precise from the resultsof only LLM-based models in Table 1 or Figure 1.\n3. For ShareLock experiments in section 5, it seems the visual backbones are not consistent.", "audience_yn": "Yes", "audience_explanation": "This paper provides insights to the researchers working on vision and language alignment.", "requested_changes": "1. Please clarify or address weaknesses.\n2. It would be better if the authors could provide a computational cost analysis for experiments in section 5. For example, the inference time and FLOPs for different methods.\n\nOther minor suggestions:\nFor \"seen and unseen concepts\" in page1, the reader would easily get confused as there is no explanation.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3wvenlKPoA", "reviewer": "Reviewer_ori9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper aims to investigate how well large language models understand visual data.\n\n However, as explained below, I am unable to fully comprehend the paper because the narrative is convoluted, overly parenthetical, and filled with details that are meaningful only to readers who are already highly knowledgeable in this area (may even only to this solution).", "claims_and_evidence_yn": "No", "claims_explanation": "My feeling is that the paper might be fine, if explained properly. I have not found a detail that is wrong. The topic and findings suggest that they are interesting. \n\nThe problem is that in the current form the paper is very hard to read, until it lost me. While I am not the most suitable reader for this paper, yet I am in the TMLR auditorium, so I should not get lost in detail, parenthesis, cross -references, etc. I should not need to draw additional schematics just to be keep track of which is what! \n\nIn my view, the paper fails to meed the following TMLR criteria: \"clarity of the narrative\".", "audience_yn": "Yes", "audience_explanation": "As said, my feeling is that, if the paper would solve the clarity issue, would be fine. But I would like to really pass an opinion on this matter, after the paper is clear. Emphasize  \"If\" and \"feeling\" not \"sure\" and \"certainty\".", "requested_changes": "The main issues is that the paper is, in general, very difficult to follow. I have not identified incorrect content, but the paper appears to be written for an audience of experts who already understand the core concepts. As a result, it focuses almost exclusively on details. Additionally, the writing style requires significant polishing.\n\nMore specifically:\n\n**Introduction / Section 3**:\n- Page 1: The paragraph beginning with “To ensure a rigorous … in language representation” seems unnecessary here. These details belong in the implementation section, not the introduction.\n- Figure 1:\n   - What is the source of this figure?\n   - What is MMLU-Pro? (I eventually found it, but a non-expert would need to stop reading, look it up, and return.)\n   - What is the “visual generalization score”? (Same issue.)\n   - More importantly, a figure placed in the introduction should motivate the paper, but this one does not clearly communicate the motivation.\n- The statement “We observe non-trivial zero-shot generalization across language model types, indicating latent visual-semantic alignment.” What generalization should the reader be seeing in the figure? And why does this justify the stated conclusion?\n- The sentence “In particular, features from modern generative LLMs outperform classic encoder-based embeddings such as BERT … This advantage of decoder-based models persists …” The first sentence discusses generative LLM features, but the second refers to “the advantage of the decoder.” Where is this decoder-based advantage actually shown? (\"This\" implies that it was described in the previous sentence, but it is not!)\n\n- Include a high-level schematic illustrating the core experiment: how LLMs are used for visual alignment. Provide a simple example.\n- Add one or more schematics explaining key components and design choices:\n   - the LLM,\n   - embedding extraction,\n  - the VLM encoder–decoder setup, etc.\n\n- Explain in more detail where zero-shot classification occurs and how it fits into the workflow.\n- Section 3 (“Method”) begins abruptly with detailed discussion of zero-shot classification and train/test splits. Provide more context before going into these details.\n- Table 1: What is its origin? What should the reader understand from it?\n- Pg3 :\"their similarity is computed as the cosine similarity, given by the dot product of the embeddings\", The similarity is between Z_txt and Z_img?\n- Page 5: The sentence “A new insight resulting from our analysis is the competitiveness of decoder-based language models in representing visual concepts.” This is the first time the paper suggests that evaluating “the competitiveness of decoder-based language models” is a core objective. Please clarify this purpose earlier.\n\n**Experiments**\n- Table 2: Clarify in the caption that accuracy is the evaluation metric.\n- ShareLock: The paper does not explain what this method is. Searching for the term leads to mentions in the introduction, the caption of Figure 3, and the “experimental setup → methodology” section, but nowhere is there a clear description. Provide a dedicated explanation.\n- The sentence “As illustrated in Figure 3, this design places the LLM at the core of the model and necessitates expressive language representations for strong VLM performance.”\n - There is no LLM visible in Figure 3. Perhaps it is labeled as “text”? But given the complexity of the method, the reader should not have to guess. Also, what in Figure 3 demonstrates that expressive language representations are necessary?\n- “We increase the projection network  $p_{text}(⋅) $ to four layers with a hidden dimension of 4096.” The earlier definition of $p_{text}$  is easy to miss, requiring the reader to stop and search for it. The explanation is overburdened with parenthetical details.\n- Then: “The additional parameters … enhance ShareLock’s ability to capture complex, non-linear cross-modal correspondences.”\nWhich parameters? Presumably the weights of the additional MLP layers, but this should be stated clearly. \n-Next: “Additionally, the maximum number of steps is increased to 5000.” Steps of what?  Optimization steps? Again, unclear.\n\n- Clarify what the “conventional CLIP-like methods” used for comparison are, why they are relevant, and how they differ from the proposed approach.\n- In general, the method section is too difficult to follow. It needs a clearer, more structured explanation of the problem, the architecture, and the reasoning behind the design choices.\n\n\n\n\n**Citation**\n-When rewriting, please take into account the journal’s citation style. The manuscript has been obviously composed for short numerical references (e.g., [17]), . Yet the journal uses a full name style that disrupts readability. Adapt the text to the journal’s required format and avoid overloading sentences that present the method  with citations. Place citations in a separate sentence. \n\nI could continue with similar remarks for the **rest of the manuscript**, but ultimately it is not my role to rewrite the entire paper. The presentation needs to be to substantially revised,  to improve clarity, structure, and readability.", "broader_impact_concerns": "Not for the moment. Again, the main problem is that I was not able to understand the details of the paper. Once I manage to do this, the \"Impact concern\" aspect needs to be re-evaluated.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZwHApr720A", "reviewer": "Reviewer_rso2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Main Contributions\nDirect Assessment of Visual Alignment in LLMs\nThe paper provides the first rigorous evaluation of how LLMs' internal text representations correspond to visual concepts, using strict zero-shot image classification tasks. This avoids concept leakage and tests true generalization.\nDecoder-Based LLMs Excel at Visual Tasks\nModern decoder-based LLMs (e.g., Llama-3, Gemma-2) outperform encoder-based models (e.g., BERT) in visual generalization. The ability of an LLM to perform well on language tasks (measured by MMLU-Pro) strongly predicts its visual alignment and transfer performance.\nShareLock: Efficient Vision-Language Model\nThe authors introduce ShareLock, a lightweight VLM that combines frozen LLM and vision model features. ShareLock achieves strong multimodal capabilities, including:\nRobust zero-shot classification\nCross-lingual generalization (e.g., 38.7% accuracy for Chinese vs. CLIP’s 1.4%)\nCompetitive performance with models trained on much larger paired datasets\nReduced Need for Paired Data and Compute\nBy leveraging frozen LLMs, the approach significantly reduces the need for large-scale paired image-text data and compute resources, making VLMs more accessible and adaptable.\nMultilingual and Compositional Reasoning\nShareLock demonstrates strong cross-lingual transfer and above-CLIP performance in compositional reasoning tasks, highlighting the rich semantic content of LLM representations.\n\nKey Findings\nLLMs Encode Visual Knowledge: Even without multimodal training, LLMs contain semantically meaningful representations that can be mapped to visual latent spaces.\nModel Size and Capability Matter: Larger and more capable LLMs show higher visual alignment and generalization.\nMinimal Fine-Tuning Needed: The intrinsic visual alignment of LLMs is robust to fine-tuning, and the base model’s architecture and training regime are more important than post-hoc strategies.\nEfficiency: ShareLock achieves competitive results with orders of magnitude less data and compute compared to conventional VLMs like CLIP.\n\nWhy It Matters\nFor researchers and practitioners: This work shows that advances in LLMs can directly benefit vision-language tasks, even without expensive multimodal training.\nFor multilingual and low-resource settings: The approach enables strong performance in languages and domains with limited paired data.\nFor future VLM development: The findings suggest that leveraging frozen LLMs is a promising direction for building efficient, robust, and generalizable multimodal models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Key Evidences: \n\n1. Zero-Shot Image Classification Benchmarks\nThe authors rigorously evaluate multiple LLMs (both encoder- and decoder-based) on strict zero-shot image classification tasks across four diverse datasets (AWA2, CUB, FGVCAircraft, ImageNet+).\nThey enforce strict disjointness between training and test classes to avoid concept leakage, which is a common pitfall in multimodal evaluation.\n\n2. Comparative Performance Tables\nDetailed tables show that decoder-based LLMs (e.g., Gemma-2, Llama-3) consistently outperform encoder-based models (e.g., BERT, RoBERTa) in visual generalization scores.\nThe correlation between language model capability (MMLU-Pro scores) and visual alignment is quantified (Pearson r = 0.768), supporting the claim that better LLMs yield better visual transfer.\n\n3. ShareLock Model Results\nShareLock, which uses frozen LLM and vision model features, is benchmarked against established models like CLIP and LiT.\nIt achieves competitive or superior performance in zero-shot classification, especially in low-data regimes, and demonstrates strong cross-lingual generalization (e.g., 38.7% accuracy for Chinese vs. CLIP’s 1.4%).\n\n4. Ablation Studies\nThe paper includes ablations on:\nLanguage model choice (decoder vs. encoder)\nVision backbone choice\nProjection network architecture (MLP vs. Transformer)\nLoss function (CLIP loss vs. SigLIP)\nThese studies reinforce the robustness and generality of the main findings.\n\n5. Qualitative Results\nThe authors provide qualitative retrieval examples showing ShareLock’s ability to align text and images across fine-grained and abstract prompts.\n\n6. Limitations and Transparency\nThe paper openly discusses limitations, such as the inability to attribute performance differences solely to architecture due to varying pretraining data and objectives.\nIt also notes the scale of datasets used and computational constraints, which adds credibility to the claims.\n\n\nAreas for Further Validation\nScaling to Larger Datasets: The authors acknowledge that their experiments are limited to datasets with up to 12M image-caption pairs. Further validation on even larger, more diverse datasets would strengthen the generalizability of the findings.\nCompositional Reasoning: While ShareLock improves over CLIP, it still falls short of human-level performance in compositional reasoning tasks, suggesting room for future work.\n\nConclusion\nOverall, the paper provides strong and multi-faceted evidence for its main arguments.\nThe combination of quantitative benchmarks, ablation studies, and qualitative results makes the claims credible and well-supported. The authors are transparent about limitations, which further strengthens the trustworthiness of their conclusions.", "audience_yn": "Yes", "audience_explanation": "suit for low computing resource researchers, since it seems only linear layer probation were involved in the work.", "requested_changes": "In the paper, they claim that decoder-only LLM performs better than encoder-only ones in vision-alignment tasks. However most of encoder-only models listed in the paper seems developed much earlier before decoder-only one, in terms of publish date.  It would be good there is some recent dp with encoder-only model to compare.", "broader_impact_concerns": "Even though I don't think the idea of paper is quite innovative, but I want to emphasize that this paper provide researchers who are interested in LLM and are limited with computing resource some insight about how to do LLM related research within limited capacity.", "additional_comments": "N/A", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)