Frozen Transformers in Language Models Are Effective Visual Encoder Layers
Authors: Ziqi Pang
Ziyang Xie
Yunze Man
Yu-Xiong Wang
Venue: ICLR 2024 spotlight (2024)
Keywords: representation learning
vision-language models
transformer
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: This paper reveals that large language models (LLMs), despite being trained solely on text data, are surprisingly}strong encoders for purely visual tasks in the absence of language. Even more intriguingly, this can be achieved by a simple yet previously overlooked strategy -- employing a frozen transformer block from pre-trained LLMs as a constituent encoder layer to directly process visual tokens. Our work pushes the boundaries of leveraging LLMs for computer vision tasks, significantly departing from conventional practices that typically necessitate a multi-modal vision-language setup with associated language prompts, inputs, or outputs. We demonstrate that our approach consistently enhances performance across a diverse range of tasks} encompassing pure 2D or 3D visual recognition tasks (e.g., image and point cloud classification), temporal modeling tasks (e.g., action recognition), non-semantic tasks (e.g., motion forecasting), and multi-modal tasks (e.g., 2D/3D visual question answering and image-text retrieval). Such improvements are a general phenomenon, applicable to various types of LLMs (e.g., LLaMA and OPT) and different LLM transformer blocks. We additionally propose the information filtering hypothesis to explain the effectiveness of pre-trained LLMs in visual encoding -- the pre-trained LLM transformer blocks discern informative visual tokens and further amplify their effect. This hypothesis is empirically supported by the observation that the feature activation, after training with LLM transformer blocks, exhibits a stronger focus on relevant regions. We hope that our work inspires new perspectives on utilizing LLMs and deepening our understanding of their underlying mechanisms.

Published as a conference paper at ICLR 2024
FROZEN TRANSFORMERS IN LANGUAGE MODELS ARE EFFECTIVE VISUAL ENCODER LAYERS
Ziqi Pang Ziyang Xie Yunze Man"
Yu-Xiong Wang
University of Illinois Stbana-Champaigh
{ziqip2, zi yangg,, yunzem2, yxw) yww.eemm,.x..eeee1111 edu
https: : //g/ thub. cww/ wtw.//iiiaaaggg////ssssstttott. Encoding
ABSTRACT
This paper reveals that large language models (LLMs), despite being trained solely text data, are surprisingly strong encoders for purely visual tasks in the ab- sence of language. Even more intriguingly, this can be achieved by a simple yet previously overlooked strategy employyngg frozen transformer block from pre- trained LLMs as a constituent encoder layer to directly process visual tokens. Our work pushes the boundaries of leveraging LLMs for computer vision tasks, sig- nificantly departing from conventional practices that typically necessitate multi- modal vision- langugge setup with associated language prompts, inputs, or outputs. We demonstrate that our approach consistently enhances performance across di- verse range of tasks, encompassing purely 2D and 3D visual recognition tasks (e.g., image and point cloud classification), temporal modeling tasks (e.g., ac- tion recognition), non-semantic tasks (e.g., motion forecasting), and modai tasks (e.g.. 2D/3D visual question answering and image-text retrieval). Such im- provements are a general phenomenon, applicable to various types of LLMS (e.g.. LLaMA and OPT) and different LLM transformer blocks.
We additionally propose the information filtering hypothesis to explain the ef- fectiveness of pre-trained LLMs in visual encoding the pre-trained LLM trans- former blocks discern informative visual tokens and further amplify their effect. This hypothesis is empirically supported by the observation that the feature acti- vation, after training with LLM transformer blocks, exhibits a stronger focus on relevant regions. We hope that work inspires new perspectives utilizing LLMs and deepening our understanding of their underlying mechanisms.
INTRODUCTION
Large language models (LLMs), trained massive amounts of text data, have recently demon- strated remarkable potential across various tasks, extending beyond their original linguistic domain. For example, in the field of computer vision, LLMs exhibit the ability to interact with visual to- kens and decode them into tokenized output. This is commonly achieved in a multi-modal vision- language framework that incorporates the language modality, as exemplified by either projecting visual tokens to LLMs via linear layers (Koh et al., 2023; Lin et al., 2023; Merullo et al., 2023; Schwettmann et al., 2023) or employing cross-attention mechanisms between visual and language tokens (Alayrac al., 2022; Li et al., 2022; 2023; Wang et al., 2023). As explore the limits of utilizing LLMs for computer vision tasks, an interesting question arises: can LLMs effectively handle tasks that are exclusively visual, without any reliance on language?
This paper provides a positive demonstration of feasibility in addressing this question, by introduc- ing straightforward yet previously overlooked approach: incorporating a frozen transformer block from a pre-trained LLM as a general-purpose visual encoder layer, directly processing the visual tokens. Specifically, as illustrated in Fig. la and Fig. 1b, our design involves the following steps: (1) extract a frozen LLM transformer block and append it on top of the original visual encoder; (2) insert trainable linear layers before and after the added LLM block to align the feature dimensions; and (3) freeze the LLM transformer while optimizing the other modules as usual during training.
Surprisingly, this simple design enhances performance across a wide spectrun of tasks, including 2D and 3D recognition (image and point cloud classification), video understanding (action recognition), and non- non-semantic (motion forecasting) tasks. In addition to these purely visual tasks, our approach
* Equul contribution.

---

Published as a conference paper at ICLR 2024
(a) Architecture
(b) Pseedo-Cooo Code
det 2128 (1027_ Targe, "M g83:
MLP Clositer
Licsur Layer (Dattu
Dittyrl
6827-307 Ercoder(orgt, 1narge)
6827 www.sstfer x Decoder Largg,, CuRTgR)
Frozen Transtormer
"Gt"
LLaggugge Transformen
Linearlayer (Dimar - Dirquud
SE2P- 13 x re:LineueQVAT www.ennnieeee din, LA (1-1358ee_50))
Se2P.12 Lw.Lineer
Tratsforrnae
For paran in se27. pansartenss:: perennnsssino and Palie
Parth Embeodding
def fonvers(aii, Jmgl:
se2 NGT(K)
582P L3Cc)
L2(c)
Visual Tokens
reture
Figure I: Our straightforward method of using a frozen transformer block from pre-trained LLMs as a visual encoder layer. Visualized with example of ViT (Dosovitskiy et al., 2021). (a) Our design simply appends a frozen transformer block (pink) on top of the regular visual encoder (gray). Only two trainable linear layers (green) are added align the feature dimensions. (b) Pytorch-style pseudo-code shows the simplicity of our approach.
also effective in multi-modal tasks (2D/3D visual question answering and image-text retrieval). Notably, such improvements are general across various types of LLMs like LLaMA (Touvron et al., 2023) and OPT (Zhang et al., 2022), as well as different LLM transformer blocks.
Our discovery of using pre-trained LLM transformer block as visual encoder layer is intriguing, because it significantly deviates from the conventional designs of lisoonlanguaae models (VLMs). In particular, our treatment of LLM transformers encoders (1) operates independently of language prompts, inputs, or outputs; (2) allows for training from scratch without the need for pre-trained backbones like CLIP (Radford et al., 2021); and (3) decouples and simplifies the usage of LLMs into separate transformer blocks.
However, one crucial question remains: why are LLMS effective in visual encoding, given that they have been exclusively trained on text and have never encountered visual input? To this end, we pro- pose the information filtering hypothesis: the pre-trained LLM transformer blocks discern informa- tive visual tokens and further amplify their contribution to the latent representation. This hypothesis stems from our observation across multiple tasks, where the feature activation consistently exhibits a stronger focus on relevant regions, after integrating the frozen LLM transformer blocks.
In summary, we have made the following contributions:
We discover that using frozen transformer block from pre-trained LLMS as a visual encoder layer facilitates a diverse range of tasks, by introducing a simple yet under-explored approach.
We demonstrate that the benefits of frozen LLM transformers in visual encoding are a general phenomenon, through our investigation on various LLMs and transformer blocks.
We propose the information filtering hypothesis explain the effectiveness of frozen LLM trans- formers in processing visual tokens: the incorporated LLM blocks distinguish the informative tokens and amplify their effect.
We hope that our work will drawn attention to the intriguing application of employing LLM trans- formers as versatile encoders, not only for visual inputs but potentially also for other modalities. Additionally, we hope to inspire new perspectives on understanding LLMs and VLMs.
2 RELATED WORK
Large language models. Pre-training transformers (Vaswani et al., 2017) with masked token pre- diction facilitates the generalizability of language models (LMs) to various tasks, represented by BERT (Kenton & Toutanova, 2019). Later on. larger models at scale proposed guided by the scaling law (Kaplan et al., 2020), such as GPT (Brown et al., 2020), I.LaMA (Touvron et al., 2023), OPT (Zhang et al., 2022), etc. These large models with tens of billions of parameters unlock the intriguing ability of in-context learning and excellent zero-shot performance on various tasks. Our work highlights the interesting discovery that the transformer blocks in such large language models (LLMs) are able to interact with visual data and enhance a wide spectrum of computer vision tasks.
Language models for visual tasks. LMs are mostly used as text encoders for vision-language models (VLMs) (Dou et al., 2022; Kim et al., 2021) or image-text pre- pre-training (Radford et al., 2021) before the emergence of LLMSs. After the creation of L.Mss, their code generation ability
2

---

Published as a conference paper at ICLR 2024
encourages flexibly combining the vision algorithms for user queries, represented by visual pro- gramming (Gupta & Kembhavi, 2023). In addition, the strong ability of LLMs has also elicited using them as generalizable decoders, i.e., translating the latent feature into output tokens. These frameworks commonly project visual features to the input layer of LLMS directly (Guo et al., 2023; Koh et al., 2023; Lin et al., 2023; Merullo et al., 2023; Chen al., 2023) use the structures of the latent bottleneck (Jaegle et al., 2021) to further encode visual tokens (Alayrac et al., 2022; Hong al., 2023; Li et al., 2022; 2023; Wang al., 2023). Our exploration reveals the potential of con- sidering the transformer blocks in LLMs as general- the previous usages of either pure encoders for text embeddings or decoders for tokenized outputs.
encoders for visual data, as opposed to
Interpreting neural networks. Understanding neural networks begins by visualizing the con- volutional patterns in low-level layers (Erhan et al., 2009). With a deeper interest in semanties, attribution-based methods like Grad-CAM (Selvaraju et al., 2017) further analyze the contribution of neurons for certain class. Network dissection (Bau et al., 2017: Zhou et al., 2018) also discov- ers that neural network units correspond to semantic concepts. For LLMs, researchers find that the knowledge mainly located at the linear layers in feedforward networks (FFN) (Dai et al., 2022; Geva et al., 2020; Meng et al., 2022a;b), and corresponds to visual concepts (Schwettmann et al., 2023), Compared with them, we study a new scenario of why a pre-trained LLM transformer can benefit visual encoding and propose the information filtering hypothesis.
3 METHOD: FROZEN LLM TRANSFORMERS FOR VISUAL ENCODING
Framework design. We formally introduce using pre-trained LLM transformer as a visual en- coder layer shown in Fig. 1a. Without loss of generality, we consider neural network that maps input to latent representation z and predicts labels y with encoder F& and decoder Fp.
Fx - FD( y.
(1)
Then a single pre-trained transformer block from an LLM like LLaMA (Touvron et al., 2023), denoted as FIM. is inserted between the encoder Fg and decoder FD. As the feature dimensions are different between the encoder Fg and the language transformer FLM, we employ two linear layers F). and F7, before and after FLM to align the dimensionality. These modify the neural network into
F (x) 2 FI. F.,M)))())) +2, Fn(2')
(2)
In the training stage, the pre-trained transformer FIM remains frozen, in the pseudo-code of Fig. 1b, while all the other modules are trained normally, including FL and F?.
Comparison with vision-language models. Our approach appears similar to recent vision- language models (VLMs) at the first glance, such as Lin et al. (2023), FROMAGe (Koh et al., 2023), and LiMBeR (Merullo et al., 2023), where linear layers directly project visual features to the input space of LLMs. However, our approach is different, because the linear layer F) does not necessarily align the visual representation into the language space. Concretely, this is reflected in three aspects: (1) Independence of visual pre-training. Our paradigm supports training-frow from-- scratch without relying on pre-trained visual encoders like CLIP (Radford al., 2021). (2) Inde- pendence of language. Our framework can function without language- based input or prompts, and is applicable for general visual representation learning instead of only vision- language tasks. (3) Independence of transformer blocks. Previous VLMs treat an entire LLM as coherent module, while our framework separates each transformer block as an independent layer for visual encoding.
Comparison with LLMs. We substantially change the behaviors of LLM transformers, due to the distinct formats between visual and text data. (1) Attention mask. LLMs commonly utilize auto- regressive masks to mimic the order of text generation. However, the tokens in visual data come all at once, such as the image tokens of the cat (Fig. la). So we abandon auto-regressive attention masks and only use attention masks to indicate the padded tokens. (2) Positional embedding. The positional embedding in LLMs, e.g., rotary positional embedding (Su et al., 2021) in LLaMA, not common option for visual encoders. Therefore, we remove the positional embeddings of LLMSs for simplicity and consistency with the original visual backbones. Considering the importance of attention masks and positional embeddings in transformers, it is surprising in hindsight that our framework has a positive influence on visual tasks.
4 APPLICABILITY OF LLM TRANSFORMERS FOR VISUAL TASKS We instantiate our framework to various tasks and observe the wide applicability of pre-trained LLM transformers. Our experiments cover 2D (image classification) and 3D (point cloud classification),
3

---

Published as a conference paper at ICLR 2024
Model
ImageNet
ImageNer-C ImageNet-A
ImageNer-SK ImageNet-R
VIT-T
72.1
43.9
7.7
19.6
32.3
+LLaMA
73.2
45.8
8.7
20.6
33.8
VIT-S
80.1
57.2
20.5
28.9
42.1
+LLaMA
80.7
58.7
22.7
30.5
42.8
VIT-B
80.6
60.5
23.4
31.9
43.5
+LLaMA
81.7
33.2
44.3
Table I: Incorporating single transformer block from LLaMA to ViT models consistently improves both accuracy (ImageNet) and robustness (rtgeNNett(,A,SS..RR))
framee and multi-framee (action recognition), semantics and motion (motion forecasting), and tasks involving linguistic input or not (2D and 3D vision-language tasks). By default, we adopt the last transformer block from LLaMA-7B (Touvron al., 2023). Our framework achieves consis- tent and significant improvements across these tasks following the standards of prior work. More analysis our designs is in Sec. 5.
4.1 IMAGE CLASSIFICATION
Image classification is the most common challenge for representation learning. We conduct experi- ments on ImageNetlk (Deng et al., 2009), and additionally evaluate on robustness benchmarks: cor- rupted images from ImageNet-C (Hendrycks & Dietterich, 2018), natural adversarial images from ImageNet-A (Hendrycks et al., 2021b), and out-of dittof-distribution images from ImageNet-SK (Wang et al., 2019) and ImageNet-R (Hendrycks et al., 2021a).
Without loss of generality, we select VIT (Dosovitskiy et al., 2021) due to its wide use and native support for transformers. Following the notation in Eqn. 2, the encoder Fg is the set of self-attention transformer blocks and the decoder FD denotes linear classifier. An intuitive illustration is in Fig. la. We train both the baseline ViT models and ViT+LLaMA from scratch following the same configuration of DeiT (Touvron et al., 2021). More details are in Sec. C.1.
The accuracy of ViT models consistently improves after incorporating the frozen LLaMA trans- former block in Table including both the accuracy on clean ImageNet images and the robust- ness on corrupted or adversarial images. Our further experiments validate that the improvement is closely related to the LLM transformer instead of the sole consequence of increased model capacity. Please refer to Sec. 5.1 and Sec. B.2 for details.
4.2 POINT CLOUD CLASSIFICATION
Point cloud classification handles fundamentally different modality compared with images. The models predict labels by processing unordered 3D points and understanding the geom- etry. Our experiments cover two common datasets: ScanOb- jectNN (Uy et al., 2019) and ModelNet40 (Goyal al., 2021). ScanObjectNN contains three splits: background (BG), fore- ground (OBJ), and clipped (T50) points. For ModelNet40, we experiment with different densities (Ik, 4k, 8k) of points.
ScanOhjeetNN OBJ
Model
BG
T50
Point-BERT
87.4
88.1
83.1
+LLaMA
88.0
88.5
83.8
ModeINet40 4k
Model
Point-BERT
92.67 92.91
93.19
+LL.MA
92.42
92.82 93.56
We adopt Point-BERT (Yu et al., 2021) and load its pre-trained parameters on ShapeNet (Chang et al., 2015). Then we ap- pend the LLaMA transformer after its final attention block before fine-tuning on point cloud classification datasets. Details are in Sec. C.2.
Table 2: LLM transformer im- proves point cloud classifica- tion.
As shown in Table 2, our approach improves the accuracy for point cloud classification, further sup- porting the applicability of using frozen LLM transformer the accuracy slightly drops on ModelNet40 with 1 and 4k points, due to the saturation and ~~0.2% variance performance on ModelNet40 which also analyzed in Ma al. (2022). However, with an increased number of points (8k), the improvement of the L.LMMA transformer is noticeable on MedelNet40. More importantly, on the more challenging ScanObjectNN, our approach improves the accuracy consistently and significantly. This experiment also shows that our framework is com- patible with fine-tuning setups, in addition to vaining-frome from-scratch scenarios Sec. 4.1.
visual encoding layer. Note that
4.3 ACTION RECOGNITION
For the video modality, we apply the pre-trained LLM transformer block to action recognition, where the algorithm predicts the action labels of video clips. We choose the benchmark of "Something-
4

---

Published as a conference paper at ICLR 2024
(a) 2D VQA and Image Retrieval
(b) 3D VQA
VQAv2 (Tes-dev)
Flickr30k (Val)
Methods
EM@ 1 EM@ 10
Model
Overall Yes/No Number Other
IR IR@5 IR 10
ScanQA
46.58
85.97
METER
69.60
85.08
47.82
61.37
49.66
80.86 89.48
SQA3D
47.20
86.82
+ I.L.MA
70.23
85.70
48.98
61.89
50.22 82.26 90.08
SQA3D-LLaMA
48.09
89.03
Table 5: Frozen LLaMA transformer enhances both 2D (a) and 3D (b) vision-language models.
something-v2" (SSv2) (Goyal et al., 2017b) for evaluation, because highlights the challenge of understanding fross-frame movement, instead of relying
single-frame semantics.
We follow VideoMAE (Tong et al., 2022) and adopt the simple yet effective VIT backbones. Different from patches of tokens in 2D im- ages, the video tokens are cubes spanning both spatially and tempo- rally. Identical to Fig. la, we place the LLaMA transformer behind the last self-attention block in ViT. Our training setup also adopts the two-step practice in VideoMAB: (1) initialize VIT transformers from MAE (He et al., 2022) pre-trainiin vreitrainnng (2) add the LLM transformer and then fine-tune on the SSv2 dataset using the same configuration of VideoMAE. More details are in Sec. C.3.
Model
Acc1
Acc5
VIT-S +LLaMA
64.71
89.15
65.89
89.93
VIT-B
64.97
89,50
+IJ aMA
66.03
90.25
Table 3: Pre-trained LLM transformer improves action recognition on SSv2.
The LLaMA transformer enhances the accuracy for both ViT-S and ViT-B in Table 3, supporting the applicability of our framework for videos. To clarify, our baseline accuracy is lower than that re- ported in VideoMAE's s original paper, because VideoMAE used 32/64 GPUs to enable a larger batch size than our computational resources. Nonetheless, we control the settings identical between ViT and ViT-LLaMA for a fair comparison and indicate the positive effects of using LLM transformers.
4.4 MOTION FORECASTING
We select motion forecasting as an example of a non- semantic task. It is safety-critical for autonomous driving and capitalizes on the understanding of dynamies, agent- agent interaction, and agent-lane relationship. The input commonly includes the historical trajectories of agents and way-points of lane segments, which are both repre- sented in polylines on the bird's-eye view (BEV). The de- sired output is set of K most possible future trajectories. We conduct experiments on Argoverse (Chang et al., 2019). The evaluation metrics are minimum average displacement (ADE), minimum final displacement (FDE), and miss rate (MR), which cal- culate the errors of predictions from different aspects and are better at lower values. We apply the frozen L.M transformer VectorNet (Gao al., 2020) and mnTrrasffrmer (Liu et al., 2021) )). They first convert the agents and lanes into features, and then our LLaMA transformer block processes these agent and lane tokens. Demonstration and details are in Sec. C.4.
Model
ADE]
FDE,
MR.
VectorNet +LLaMA
0.77
1.23
13.2
0.76
1.20
12.7
mm Traaasfforer
0.72
1.10
10.7
+LLaMA
0.71
10.5
Table 4: LLM transformer layer is ben- eficial for motion forecasting.
According to Table 4, the models with LLaMA forecast better trajectories. However, we notice that the improvement is less significant compared with semantic tasks, which reflects the preference of LLM transformers for encoding rich semantics over object movements.
4.5
VISIAN-LANGUAGE TASKS
2D vision-language tasks. The benefits of frozen LLM transformers for visual encoding are not limited to purely visual tasks. We experiment with 2D lasiunnlanguage (VL) tasks, including vi- sual question answering (VQA) on VQAv2 (Goyal et al., 2017c) and zero-shot image retrieval on Flickr30k (Plummer et al., 2015). We adopt the widely-used METER (Dou et al., 2022) as our baseline. It extracts uni-modal features for images and text, fuses cross-modal features, and de- codes the output from the cross-modal features. We insert the LLM transformer block after the crosl-modal fusion. An intuitive illustration is in Fig. H. During training, our setup follows Shi et al. (2023): initialize the image encoder with CLIP- BBP2 (Radford et al., 2021) and the text encoder with RoBERTa (Liu et al., 2019), and then fine-tune on VQAv2 or Flickr30k. Details are in Sec. C.5. As in Table 5a, both of the 2D VI. tasks significantly enhanced with the LLaMA transformer. This evidence supports the potential a frozen I.LM transformer for multi-modal tasks.
3D visual question answering. We extend our proposed idea into 3D VQA, which requires com- prehending an input 3D scene represented by a point cloud or multi-view images and then answering questions. 3D VQA challenges the ability to ground language in 3D context. We conduct exper-
5

---

Published as a conference paper at ICLR 2024
iments the SQA3D (Ma et al., 2023) dataset, comparing with baseline methods and state of the arts (Ma et al., 2023; Azuma et al., 2022) on the exact match (EM) metric. We follow the baseline SQA3D to process the textual imput with LSTM (Hochreiter & Schmidhuber, 1997) and 3D point clouds with VoteNet (Qi et al., 2019). Here, we add the L.M block after the VI. fusion, which is consistent with our 2D VI. design (Sec. 4.5). More details are in Sec. C.6. According to Ta- ble 5b, adding a frozen LLM transformer effectively enhances the QA ability of models. The full comparison with detailed breakdown metrics is in Table
ANALYSIS ON LLM TRANSFORMERS FOR VISUAL TASKS
We justify our design choices (Sec. 5.1) and illustrate the wide applicability of our framework to various LLMs and transformer layers (Sec. 5.2). Our investigation also discovers that sufficiently large LLMs are the premise of benefiting visual encoding with frozen transformer (Sec. B.3) and discusses the place to insert LLM blocks (Sec. B.4).
5.1 ABLATION STUDY ON DESIGN CHOICES
Model capacity. Regarding the wide applicability of frozen LLM transformers, question if the improvement mainly comes from the increased capacity of the linear layers F/, and F2, instead of
Model
Acc
ViT-S
80.1
ViT-S-LLaMA
80.7
the treitra weights in LLM transformers FLM- To analyze model capacity, we create ViT-S-MLP, which has identical train- able parameters compared with ViT-S-LLaMA. Concretely, ViT- S-MLP removes the LLM block F,M. and then inserts GeLU ac- tivation (Hendrycks & Gimpel, 2016) and layer normalization (Ba et al., 2016) between F) and F7. It also adopts the identical train- ing procedure as VIT and ViT-LLaMA in Sec. 4.1 for fair comparison. The results are summarized in Table 6: the ViT-S-MLP has better performance than ViT-S due to its increased capacity, but the improvement only about half of ViT-S-LLaMA. Therefore, the LLM transformer weights are cru- cial for the improvement and the observed benefits are not mere consequences of increased model capacity. Investigation with more tasks and baselines are Sec. B.2.
ViT-S-MLP
80.4
ViT-S-LLaMA-FT
78.9
Table 6: Ablation study
model capacity and fine- tuning.
Fine-tuning. We further verify whether fine-tuning the language transformer WITTS-LLLMA-FII is better than freezing it. As in Table 6, fine-tuning decreases the performance compared with ViT-S-LLaMA, We analyze this phenomenon by visualizing the loss curves in Fig. F and training under an additional setting of 100 epochs. Although fine-tuning improves performance under short training (100 epochs in Table A), it hurts the accuracy when trained sufficiently: in Fig. F. ViT-S-LLaMA-FT shows lower training loss but relatively larger validation loss, which indicates overfitting. Thus, our observation demonstrates the challenges of training large transform- ers, and we accordingly freeze the LLM transformers in our design because of its simplicity and effectiveness.
Accuracy with LaMMA Layers
76.
75:87 75.84
74.28
Accuracy with OPT Layers
76S
76.20
76
25/97
15.5
75.25
75,00
7455
74.31
74
VITSS
L.5
L16
1.24
132
Figure 2: Various LLM transformer lay- ers improve the accuracy.
5.2 VARYING LLM TRANSFORMER LAYERS
We discover that different LLM transformers influence visual representation learning significantly within our framework, even though they have identical capacity. Specifically, we use transformer blocks from diverse depths of LLaMA-7B (Touvron et al., 2023) and OPT (Zhang al., 2022) onto ViT-S. The models are trained in the ablation study setting of 100 epochs. More details are in Sec. C.7. As shown in Fig. 2, the layer type significantly changes the performance. These experi- ments also validate that our framework is applicable to various L.MS and transformer layers, and highlight the importance of selecting proper transformer layers. We additionally observe that the last LLM layers consistently improve the performance although they might not be optimal.
6 INFORMATION FILTERING HYPOTHESIS
This section aims to explain how a pre-trained and frozen LLM transformer benefits visual tasks. Intuitively, hypothesis can be stated as:

---

Published as a conference paper at ICLR 2024
Feature Activation
(b) Attention Scores
WT-LLaMA
(Fg)
(Fa)
(Rie)
LF2)
Alt (Fdc)
PRN EF
Linosr LF
Mead (Fa)
Attention Scures
VIT Atrention Scores
Magnitude
Prequency
tude
Frequncy
Figure 3: (a) Feature activation regarding both magnitudes and frequencies of features. We highlight that ViT-LLaMA demonstrates the emergent tendency of object segmentation compared with ViT, indicating its ability to select informative tokens. (b) Attention scores between the CLS and visual tokens. The attention from ViT commonly noisy (left). Though ViT-LLaMA improves the con- centration on few heads, most of the attention heads are still noisy. Both good and bad attention from ViT-LLaMA are sampled for demonstration purpose.
Information filtering hypothesis. A pre-trained LLM transformer functions as "filter" that distinguishes the informative tokens and amplifies their contribution for the prediction, in the form of enlarged magnitudes or frequencies in the feature activation.
We first derive this hypothesis in the context of image classification (Sec. 6.1), then provide quan- titative investigation (Sec. 6.2), and discuss the observation on other tasks (Sec. 6.3). Due to space limits, we include more details in Sec. A and discuss limitations in Sec. A.4.
6.1 QUALITATIVE DERIVATION OF INFORMATION FILTERING HYPOTHESIS
Emergent concentration on informative tokens. Our hypothesis originates from the emergent behavior that the feature activation highlights the informative tokens after adding pre-trained LLM transformer. In the analysis, we extract the activation of features after each layer as Fig. 3a, including the original ViT g, the attention layer FL and feedforward network Ff in the L.M transformer, and the linear layers FL and FŽ. Notably, the feature activation calculated regarding both magni- tudes (1.2- norm after centering) and frequencies (L.2-norm of angles after Fourier transformation). The different layers in Fig. 3a indeed show diverse preferences over magnitudes or frequencies.
As clearly demonstrated in Fig. 3a, the token activation better captures the regions of target objects after adding the LLM transformer, especially the magnitudes of F?, and frequencies of F Their tendency of segmentation is surprising discovery, because emergent segmentation is only observed in self-supervised learned (Caron al., 2021) ppecially-designe (Darcet et al., 2024; Shi et al., 2023; Yang et al., 2024) ViTs. More importantly, the activation ' concentration on the target object directly supports our hypothesis evidence of selecting the informative tokens.
Noisy attention scores. In contrast to the feature activation, attention scores struggle to capture the relevant visual tokens for prediction. We investigate the attention scores between the CLS to- ken and visual tokens the last transformer block, which are the last self-attention block in Fg for VIT and the transformer FIM for ViT-LLaMA, respectively. Ideal attention scores that distinguish the target object should exhibit object segmentation patterns like DINO (Caron et al., 2021). How- ever, supervised ViT models commonly have noisy attention scores (left part in Fig. 3b). Although ViT-LLaMA illustrates the ability of emergent segmentation in few attention heads, most of the attention scores also suffer from scattering and noisiness. These observations contrast the feature activation and indicate that the benefits of LLM transformers cannot be simply attributed to attention scores, since attention scores fail to reliably contribute correct visual tokens.
Deriving the amplification of informative tokens. According to our visualization in Fig. 3a, the frozen LLM transformer distinguishes the informative tokens. Intuitively, such tokens naturally
Details of calculation Sec. A.5.

---

Published as a conference paper at ICLR 2024
benefit the downstream decoding, but this is straightforward only when the decoder directly takes the visual tokens as input. As a counterexample, ViT utilizes CLS token for classification, and the visual tokens output by Fž is not the imput to the decoder and always receives zero gradients during training. To bridge the gap in CLS token scenarios, the second half of our hypothesis is necessary: the frozen LLM transformer amplifies the contribution of informative tokens.
Concretely, the calculation of the CLS token is,
CLSS FF FIM
Σ w. =|11
where denotes visual tokens, and wy denotes the weight of visual token v. Eqn. describes the process of (1) aggregating visual tokens z/,[v] guided by the attention scores wy; and (2) sequentially flowing through the subsequent layers, including the LLM transformer's attention head FiM. feed- forward network Ff.y- and the second linear layer F7. To concentrate explaining visual tokens, Eqn. 3 also slightly simplifies self-attention by removing the CLS token from the right-hand side.
In Eqn. 3, the useful visual tokens are not reliably attributed to the CLS token, because the attention scores w, are observed to be noisy. Therefore, our objective is to connect the visual tokens in Eqn. 4,
4vl = F2 FLM (2((v)), where v VEV,
(4)
which are informative (as in Fig. 3a), to the final feature representation =][CLS]. This pursuit makes us notice that attention scores are noisy while token features z][r]] are informative. Such a con- tradiction indicates that amplification of the informative tokens is a more plausible explanation, compared with better attention scores. We express the hypothesis below in formal way, which is simple change to Eqn. 3:
[LSS
Σ PEV
Wy (FZ
FFZ.FEN
FLA
F F (+)())
[Hypothesis]
Eqn. 5 holds equal under the special case of F7. F.M FIM being linear transformation. This equation explains how the informative tokens in Fig. 3a are implicitly supervised in the CLS token.
As a brief remark, our derivation builds upon two observed pieces of evidence: (1) visual tokens concentrating informative regions; and (2) noisy attention scores. These lead to the first half of our hypothesis and explain the benefits when visual tokens are direct input to decoders. By further connecting visual tokens the CLS token, we propose that the inserted LLM transformer amplifies the effects of informative tokens. A more thorough version of the derivation is in Sec. A.
6.2 QUANTITATIVE EVIDENCE
The qualitative observation in Sec. is further sup- ported with quantitative evidence. Specifically, we use the ImageNet-S (Gao al., 2022) dataset to provide the ground truth of "informative regions" from its annota- tion of semantic segmentation masks. To assess the fi- delity of feature activation and attention scores, we first generate pseudo-masks highlighting their concentrating regions, i.e., the tokens with larger activation or atten- tion scores than the others on the same image. Then the quality of features and attention scores are reflected by the mIoU (mean intersection ceenn-vve--uneeee union) between their pseudo-masks and ground truth segmentation masks. Im- plementation details are in Sec. A.3.
VIT-S--L.MA
mloU
95
Fix
PIN
PE
VIT-B-LLaMA
20.3
Finally, we summarize the mloU statistics for feature activation and attention scores in Fig. 4. As demon- strated, both ViT-S-LLaMA and ViT-B-LLaMe have bet- ter mloU of pseudo-mant masks than attention scores. This di- rectly supports hypothesis that the features [v]]vv]ve contribute more reliably than attention scores (w.lveV). We additionally notice that the pseudo-mask masks from ViT-LLaMA generally have larger mloU compared with VIT, which reflects the benefits of training VITs with frozen LLM
Att:
Fi
F1-
Fl F:
WFB
Figure 4: LLaMA's features (F)) have larger
Pseudo-masks from ViT-
mIoU than attention scores and VIT.
8

---

Published as a conference paper at ICLR 2024
Model
law
VIT-S
High
Low
VTT--LLLMA
High
Figure 5: Token activation in action recognition. Video tokens are activated jointly in all the frames, and every video token is a cube with shape 2 x 16 x 16. After adding the LLM transformer. the model better concentrates on the relevant objects and hands ("low threshold") and more accurately focuses on frames with hand-object interaction ("high threshold").
transformer. The advantage of the feature in the first linear layer FL also reveals that training with our framework is beneficial to even earlier stages of features. However, we would like to clarify that the pseudo-masks from either magnitude or frequency activation are intuitive but lossy mea- sures to quantify feature quality, because neural networks can encode information in other formats. Therefore, better measurements to analyze network layers will be meaningful for future work.
6.3 INFORMATION FILTERING HYPOTHESIS ON OTHER TASKS
The previous sections mainly discuss our information filtering hypothesis in terms of image classi- fication. Meanwhile, we also discover supportive evidence of our hypothesis on various other tasks. This section investigates action recognition as an example, and Sec. A.2 covers additional tasks including point cloud classification, 2D VQA, and 3D VQA.
We mainly analyze the information filtering hypothesis in action recognition qualitatively, because the ground truth of "relevant regions" is difficult to quantify for this task. In practice, we follow similar procedure in Sec. 6.1 and visualize the activation of video tokens in Fig. 5, and display the most highly activated tokens according to either low or high thresholds for better clarity. At a low activation threshold, we notice that the video tokens from ViT-S-LLaMA better capture the foreground areas of hands and manipulated objects than ViT-S. With the video tokens VideoMAE activated both spatially and temporally, we further increase the threshold to demonstrate its ability to select informative frames. As the "high threshold" row of Fig. 5, ViT-S-LLaMA more accurately focuses the middle frames with actual human-object interaction. Therefore, we conclude that the informative video tokens are indeed distinguished and augmented in action recognition, which aligns with the information filtering hypothesis.
CONCLUSION
In this work, we explore the unexpected capability of large language models (LLMs) as encoders for visual tasks. significant departure from their conventional text-based applications. By seamlessly integrating a frozen transformer block from pre-trained LLMs into visual encoders, we observe con- sistent performance enhancements across diverse visual challenges, including 2D image and video classification, 3D point cloud classification, motion forecasting, and 2D and 3D vision-language tasks. This phenomenon, underpinned by our proposed information filtering hypothesis, highlights the inherent adaptability and versatility of LLMs for more general representation learning. We hope that our insights will catalyze further exploration into the uncharted fields of LLM applications and foster innovative strategies to harness their potential in novel ways.
Discussion and limitations. We have validated the capability of pre- pre-trained, frozen language trans- formers across wide spectrum of visual tasks. is important to note that our goal is to methodically explore this under-investigated problem. Therefore, our experiments are designed to maximize the diversity of tasks under fair comparisons with well-established or competitive baselines, rather than striving for state-of-the-art performance for all tasks, which is also constrained by our computational resources. We leave scaling up the experiments to state-of-the-an levels for all tasks interesting future work. Meanwhile, we also notice that our information filtering hypothesis has not covered several intriguing questions, e.g., how quantify the functions of different layers and analyze how the training process facilitates the visual token features to cooperate with the language transformer, which are also meaningful directions.

---

Published as a conference paper at ICLR 2024
Acknowledgement. We thank Baifeng Shi and Shoufa Chen for their valuable discussion and suggestions on implementation and model training. This work was supported in part by NSF Grant 2106825, NIFA Award 2020-67021-322 1327999, the Jump ARCHES endowment through the Health Care Engineering Systems Center, and the IBM-IIlinois Discovery Accelerator Institute. This work used NVIDIA GPUs NCSA Delta through allocation CIS220014 and CIS230012 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by NSF Grants #2138259, #2138286, #2138307, #2137603, and #21
REFERENCES
lean-Baptise Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Bart, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadch, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning. In NeurlPS, 2022.
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe, ScanQA: 3D question answering for spatial scene understanding. In CVPR, 2022.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1rXhv:1607.06450 2016.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. CVPR, 2017.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. 2015.
ShapeNet: An information-rich 3D model repository. arXiv preprint
Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hart- nett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, and James Hays. Argoverse: 3D track- ing and forecasting with rich maps. In CVPR, 2019.
Guo Chen, Yin-Dong Zheng, Jiahao Wang. Jilan Xu, Yifei Huang, Junting Pan, Yi Wang, Yali Wang, Yu Qiao, Tong Lu, and Limin Wang. VideoLLM: Modeling video sequence with large language models. arXiv preprint arXiv:2305. 1329,, 2023.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In ACL, 2022.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In ICLR, 2024.
Alexandros Delitzas, Maria Parelli, Nikolas Hars, Georgios Vlassis, Sotirios Anagnostidis, Gregor Bachmann, and Thomas Hofmann. question answering tasks in 3D scenes. In BMVC, 2023.
Contrastive languaeee pre-training for
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
10

---

Published as a conference paper at ICLR 2024
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, Zicheng Liu, and Michael Zeng. An empirical study of training end-to-ens lision-and-lange
transformers. In CVPR, 2022.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1, 2009.
Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. VectorNet: Encoding HD maps and agent dynamics from vectorized representation. In CVPR, 2020.
Shanghua Gao, Zhong-Yo Yu Li, Ming-Hsuan Yang, Ming-Ming Cheng, Junwei Han, and Philip Torr. Large-scale unsupervised semantic segmentation. TPAMI, 45(6):7457-7476, 2022.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012. 149.1333,.. 2020.
Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, and Jia Deng. Revisiting point cloud shape classification with simple and effective baseline. ICML, 2021.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in hour. arXiv preprint arXiv: 706770667777771 2017a.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzyíska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The" something something" video database for learning and evaluating visual common sense. In ICCV, 2017b.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruy Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017c.
Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. From images to textual prompts: Zero-shot visual question answering with frozen large language models. In CVPR, 2023.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning
without training. CVPR, 2023.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. In ICLR, 2018
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GeLUs). arXiv preprint 16t1::660608475. 2016.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distributiin generalization. In ICCV, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021b.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9()): 1735-1780, 1997.
11

---

Published as a conference paper at ICLR 2024
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3D-LLM: Injecting the 3D world into large language models. In NeurlPS, 2023.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In ICML, 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. 2020.
Jacob Devlin Ming- bidirectional transformers for language understanding. In NAACL, 2019.
Ming-Wei Wei Chang Kenton and Lee Kristina Toutanova. BERT: Pre-training of deep
Wonjae Kim, Bokyung Son, and IIdoo Kim. ViLT: Vision-and-language transformer without con- volution or region supervision. In ICML, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014.
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. In ICML, 2023.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 123:32-73, 2017.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In ICML, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BIIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
Xudong Lin, Simran Tiwari, Shiyuan Huang, Manling Li, Mike Zheng Shou, Heng Ji, and Shih- Fu Chang. Towards fast adaptation of pretrained contrastive models for multi-channel video- language retrieval. In CVPR, 2023.
Yicheng Liu, Jinghuai Zhang, Liangji Fang, Qinhong Jiang, and Bolei Zhou. Multimodal motion prediction with stacked transformers. In CVPR, 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: /60800380, 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. SQA3D: Situated question answering in 3D scenes. ICLR, 2023.
Xu Ma. Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local geometry in point cloud: simple residual mlp framework. In ICLR, 2022.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. associations in GPT. In NeurIPS, 2022a.
Locating and editing factual
Kevin Meng. Arnb Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass editing memory in transformer. arXiv preprint arXiv:2210.07229) 2022b.
12

---

Published as a conference paper at ICLR 2024
Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. In ICLR, 2023.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2Text: Describing images using million captioned photographs. In NeurIPS, 2011.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k Entities: Collecting region-to to-phrase correspondences for richer image- to-sentence models. In ICCV, 2015.
Charles R Qi. Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3D object detection in point clouds. In ICCV, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Sarah Schwettmann, Neil Chowdhury, and Antonio Torralba. Multimodal neurons in pretrained text-only transformers. arXiv preprint arXiv:2308.01544, 2023.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual explanations from deep networks via gradient-based local- ization. In 2017.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual Captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACI. 2018.
Baifeng Shi, Trevor Darrell, and Xin Wang. Top-down visual attention from analysis by synthesis. In CVPR, 2023.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu, RoFormer: En- hanced transformer with rotary position embedding. arXiv preprint 204.::::004099000 2021.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data-
efficient learners for self-supervised video pre-training. In NeurlPS, 2022.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers and distillation through attention. In ICML, 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. arXiv preprint urXiv:2302.13977 2023.
Mikaela Angelina Uy, Quang- Hieu Pham, Binh-Son Hua, Due Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real- world data. In ICCV, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 2017.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. In NeurIPS, 2019.
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai. VisionLLM: Large language model is also an open-ended decoder for vision-centric tasks. arXiv preprint 2rXiv:2305.11175. 2023.
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. In ICLR, 2020.
Jiawei Yang, Katie Z Luo, Jiefeng Li, Kilian Q Weinberger, Yonglong Tian, and Yue Wang. Denois- ing vision transformers. arXiv preprint arXiv:2401.02955, 2024.
13

---

Published as a conference paper at ICLR 2024
Xumin Yu. Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-BERT: Pre- training 3D point cloud transformers with masked point modeling. In CVPR, 2021.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: Open pre-trained transformer language models. arXiv preprint 2r55v22005.01068, 2022.
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Interpretable basis decomposition for visual explanation. In ECCV, 2018.
14

---

Published as a conference paper at ICLR 2024
A THOROUGH DISCUSSION ON THE INFORMATION FILTERING HYPOTHESIS
A.1 DETAILED DERIVATION OF THE HYPOTHESIS
We expand the details of several steps in our derivation (Sec. 6.1) for better clarity, Beginning from the formula of the CLS token below,
cLS FŽ FFEM FiM
Σ w. =2(v)
(A)
which is identical to Eqn. 3 in the main paper, further separate the visual tokens into the sub- set of informative tokens Vú and uninformative tokens Vn, Intuitively, the tokens corresponding to the foreground objects are informative, and the background ones are uninformative. This changes Eqn. A into,
य. [CLS] = Fi. F.M
FIM
Σ wi JEV
+ Σ REV,
W/ 2t!u]]
(B)
Using the same notation, our observation on the feature activation can be stated as: the attention weights for informative tokens (w;,ieV; are still noisy after incorporating the frozen LLM trans- former, while the final visual tokens shown as below have emergent concentration target regions:
111 F2
FLM
FLM (z)))), where i E V/
(C)
express our hypothesis in terms of the connection between visual and CLS tokens with Eqn. below, where the added modules F2 F FLM augment the informative tokens V/ and lead to better prediction:
Combining Eqn. B and Eqn. C inspires
2 [CL]
Σ Wi F..
F.M (=h(t)
+
Σ NEV.
Wu (F.. FF.. (+
[Hypothesis] (D)
2
This is more thorough expression of our hypothesis in the main paper (Eqn. 5) to better differentiate the role informative tokens our hypothesis.
A.2 INFORMATION FILTERING HYPOTHESIS ON OTHER TASKS
This section provides the evidence for our information filtering hypothesis in other tasks, supple- menting the discussion on image classification (Sec. 6.1) and action recognition (Sec. 6.3). Specifi- cally, we observe that the frozen LLM transformer selects and amplifies information tokens in point cloud classification, 2D visual question answering (VQA), and 3D VQA. The tasks of motion fore- casting and image-text retrieval their "informative" tokens. For the investigated tasks, we mainly analyze qualitatively because the ground truth for relevant regions is ambiguous on such tasks, unlike the segmentation masks for image classification (Sec. 6.2).
not illustrated, because it is more abstract to intuitively define
Point cloud classification. We visualize the activation of the point tokens in point cloud classifi- cation before and after adding the frozen LLM transformer Fig. A. In the examples of chairs and desks, we observe that "POinBEERT-LLAM""" concentrates less on the background (chairs, indicated with red arrows) and more on the actual object surfaces (desks, indicated with red arrows). This demonstrates that the frozen LLM transformer learns focus on the informative points, which is consistent with our hypothesis.
2D VQA. We investigate the activation of visual tokens in the 2D VQA task in Fig. B. With the METER (Dou et al., 2022) framework initializing the visual backbone from CLIP (Radford et al., 2021) weights, the quality of feature activation is reasonable and it mostly concentrates on the target regions for both the baseline METER and our METTR--LLMa. that feature activation better aligns with the images and questions than the noisy attention heads. Furthermore, the activation of "METER BESTEE-LLANNE -LLMAA is also slightly advantageous over METER with less scattering, such as better concentrating on the regions of light and leaves high thresholds.
However, we can still witness
3D VQA. We analyze the effect of a frozen LLM transformer for 3D VQA and further confirm hypothesis. As in Fig. C, compare the activation of visual tokens, which seed points in
15

---

Published as a conference paper at ICLR 2024
PolntBERT
+LLaMA
PointBERT
+LLaMA
Figure A: Visualization of feature activation for point cloud classification. Brighter colors indicate higher activation values. To highlight the most salient activation, we apply threshold to filter out points with low activation values. This visualization demonstrates that the model learns to focus on the most discriminative foreground object for classifying the point cloud after adding the frozen LLM transformer. For visual clarity, we use red arrows indicate the key regions to observe.
Attn
METER
Attn
METER
Scores
Threshold
LLaMA
Scores
Threshold
LLaMA
the light
many
High
High
Figure B: Visualization of attention scores and feature activation for 2D VQA. We are able to visu- alize attention scores, because METER uses the token. Both low and high thresholds for feature activation are displayed to illustrate the concentration relevant regions.
VoteNet (Qi et al., 2019), before and after incorporating the LLaMA transformer. To provide the context, the scenes in SQA3D (Ma et al., 2023) are projected onto the bird's sird's-eye-view (BEV). From the visualizations, we clearly observe that the feature activation exhibits sharper concentration on the directions guided by language after adding LLaMA, such as the "table behind me" areas in the left figure and "to my left side" areas in the right figure. Therefore, these indicate that the added LLM transformer selects the informative points and augments them for downstream question answering.
A.3 QUANTITATIVE EVIDENCE
In Sec. 6.2, we generate the pseudo-mant meaus from feature activation or attention scores and then eval- uate their quality with mloU. This section describes the details.
SQA3D
SQA3D+LLaMA
SQA3D
SQA3D-LLaMA
Situation: lam standing front of the table and facing trash cun. Question: How many thairs are the table behind me?
Situation: lm standing between the toilet on my right and sink on my left. Question: the door open or dosed to my left side?
Figure C: Analysis of feature activation for 3D VQA. The scenes are viewed from BEV. The green arrow marks the location and facing direction. The colors points indicate their activation: lighter color (yellow) represents larger magnitude than a dark color (blue and green). We observe that "SQA3D+LLaMA" has sharper activation that is better related the questions. Thus, supports our information filtering hypothesis. (Best viewed in color and zooming in.)
16

---

Published as a conference paper at ICLR 2024
Dataset. We leverage ImageNet-S (Gao et al., 2022) because it provides semantic segmentation masks for ImageNet (Deng et al., 2009) images. Specifically, we adopt the ImageNet-S version with 50 categories and run evaluation on its validation set, to avoid data leakage from the training set.
Definition ToU. Our loU calculates the alignment between the highly activated tokens and the ground truth mask. As tokens are sparse and in low resolution, we slightly change the calculation of IoU for our purpose. Specifically, we first project the ground truth mask to the resolution of tokens to acquire binary mask of tokens, indicating whether they are related to the target object (with value 1) or not (with value O), denoted as Mg. Then we compute the IoU between the pseudo-mask Mp generated from feature activation with Mg follows:
TP sum(M,
sum(( M2 pp), sum MM Mp)),
(E)
Mg. = TP +P
(F)
generate pseudo-maske masud from feature activation, we are motivated by Fig. 3a and treat the highly-activated regions as the final result. To generate pseudo-masks with attention scores, we first sum the scores from all the attention heads and follow similar procedure of treating highly-scored regions pseudo-masks. Concretely, given feature activation gener- as straightforward as Mz=(z>1), where / threshold between and 1. signifi- cantly affects the quality of pseudo-masks. masks. Therefore, we always automatically choose the threshold maximizing the mloU between the pseudo-masks and ground truth masks avoid threshold tuning and enable fair comparison. The algorithm of choosing the best threshold for the activation or aftention scores for each image is below,
Pseudo-mask generation.
ating pseudo-mask
Although the process is natural, notice that selecting the thresholds for activation
[MU z) = max [M M)) where M, z and (0.11,02,,033......0333
(G)
Finally, our mIoU in Sec. 6.2 is the mean IoU on all the images.
Full results with both magnitude and frequency activation. Our comparison of mIoU in Fig. only visualizes the larger mloU from the activation of magnitude or frequency for clarity. To sup- plement the comprehension,
display the complete statistics in Fig. D. As illustrated, both ViT-S- LLaMA and ViT-B-LLaMA have better pseudo-mask quality from feature activation than attention scores, which directly supports our hypothesis. With the new statisties from both activation types, we additionally notice that the neural network layers have varied preferences over magnitudes or frequencies. However, the ViT-LLaMA features still have better fidelity compared with attention scores and features from VIT. As stated in Sec. 6.2, either magnitude or frequency is intuitive but lossy way to understand feature representation. Thus, future work is needed to further investigate the advantages and properties of individual layers.
VIT-S-LLaMA
VIT-B-LLLMA
508 50.5
50
17.8
41.4
mlou (66)
40
35.7
30331
35.5
37.8
31.0
30
30
20
20
10
Attn
F1
FLN
FLM
F
VIT-S
Attm
F!
FiM
F
VYT-B
Magnitude
Frequency
Figure D: Visualization of mloU between the ground truth masks and pseudo-masks generated from attention scores/feature activation. This figure supplements Fig. 4 by providing the mloU for both magnitude and frequency activation, where Fig. 4 selects the better one for the clarity of illustration.
A.4 DISCUSSION AND LIMITATIONS OF THE HYPOTHESIS
Perspective of usable information. We supplement the opinions from Xu et al. (2020) that greatly inspire our investigation the information filtering hypothesis. Xu et al. (2020) propose that a well- trained neural network layer can be considered decipher adding usable information into the
17

---

Published as a conference paper at ICLR 2024
features and enabling subsequent modules better infer the latent information. In our informa- tion filtering hypothesis, the incorporated modules are indeed acting as deciphers that enlarge the contribution of usable tokens and benefit downstream predictions.
Though our information filtering hypothesis explains how the performance improves with frozen LLM transformers, we notice that several intriguing phenomena are not yet covered. First, the current hypothesis unable to analyze the utilities of different layers separately. Second, the hypothesis does not explain how the training dynamics facilitate the visual token features to cooperate with the frozen language transformer, which is interesting future work.
Limitations.
A.5 IMPLEMENTATION DETAILS IN DERIVING THE HYPOTIIESIS
Magnitude and frequeney activation Our visualization in Sec. 6 and Fig. 3a calculates the feature activation magnitude or frequency domains to reflect their concentration on target objects. We illustrate the Pytorch-style pseudo-code for our operations in Fig. For magnitude, we simply compute the L2 norm of features after centering them. Similarly, the activation in the frequency domain is the norm of the difference between the angle of a token feature vector and the average angle vector of all the tokens within the image after Fourier transformation. Finally, the activation values are normalized to and for visualization and quantitative analysis.
def activations visual_tokees):
visual tokens: tensor with shape [H, W, c]
magnitude calculation
avg tokettokee_eeeeee to__tteeefeeeuree feature visual_ meanndinne, keepdin=True)
activation (visual_tokenn
avg_token, _wg_token_featuree... norm
mag_min, mag_max max activation.min(), activation.maxx) mag mag_activation (activation mag_min) / (mag_max max mag_min)
frequency calculation
torch.fft. ttt.oo.ftt.ff(((feeeeee fffft.ff((feat.aangg
avg weeffergoooel taken freq Soken.mean(din=e,, keepdin=True) activation (frea_token avg_ avg_freq_toiee token). norm freq_min, freq_nax activation. activation.man max()) treg_activation (activation freq_min) (freq_max freq_ain) return aegaiiiia,, freq_activation
Figure E: Pytorch-style pseudo-code for calculating the activation of features on magnitude and frequency domains.
B ADDITIONAL ANALYTICAL RESULTS
B. ABLATION STUDY ON DESIGN CHOICES
This section provides supplementary results for the analysis on design choices in Sec. 5.1.
Results with 100 epochs of training. We conduct the ablation studies mostly with 100 epochs to balance the computation and fi-
Model
Acc
100 Epocks
delity of conclusions. In addition to the experiments lasting 300 epochs Sec. 5.1, we supplement them with experiments lasting 100 epochs, summarized in Table A. According to the numbers, adding a frozen LLM transformer
VIT-S
75.3
VIT-S-LLaMA
75.8
VIT--MILD
75.5
visual encoder layer is still effective, improving the accuracy of the baseline VIT. In addition, we highlight that fine-tuning beneficial under insufficient train- ing (1 (100 epochs), but it hurts the accuracy when trained longer due to overfitting, which is analyzed in the next paragraph. In conclu- sion, both experiments validate design choices and the effec- tiveness of using pre-trained LLM transformer blocks as encoder layers.
VIT-S-LLaMA-FF
76.8
Epochs
VIT-S
80.1
ViT-S-LLaMA
80.7
ViT-S-MLP
80.4
VIT FL
78.9
Table A: Ablation studies
model capacity and fine- tuning.
Loss curves for fine-tuning. We analyze the design choice fine-tuning the LLM transformer in Sec. 5.1. Fig. FF shows the loss curves for both training and validation sets during the training process. The training loss is much larger than the validation loss, because their loss functions different: the training loss is the label-smoothing cross-entropy, while the validation loss the regular cross-entropy loss. With the trend Fig. F, we conclude that jointly fine-tuning the pre-trained L.M transformer might not benefit the performance, yet making
18

---

Published as a conference paper at ICLR 2024
the training process more complicated. Therefore, experiments adopt the simple solution of freezing the LLM transformer.
VITIS-LLaMA (Train)
VIT-S-LLaMA (Val)
VIT-S-LLMMA-FT (Train) WISS-LLMM-PFT (Val)
Loss Values
0
50
100
150
200
250
300
Epochs
Figure F: Loss curves on the training and validation sets for fine-tuning the LLaMA transformer (VT----LLMMA-FT) not (ViT-S-LLaMA). We solid lines denote training losses and dashed lines to denote validation losses. Though fine-tuning shows advantages in the beginning, it finally hurts the performance due overfitting (a larger loss the validation compared with not fine- tuning).
B.2 ABLATION STUDY ON NETWORK CAPACITY ACROSS DIVERSE TASKS
This section analyzes whether the improvement of our approach is the consequence of the increased capacity and supplements Sec. 5.1. Specifically, we experiment with two sets of additional base- lines: (1) using additional MLPs, aligning the number of trainable parameters; (2) using randomly initialized LLM transformer blocks, aligning the total number parameters, to compare with our approach. Please note that the randomly initialized LLM transformer blocks are trained end-to-end with the visual encoders.
We conduct experiments across all the tasks covered in Sec. 4, as shown in Table B. The results show several important findings validating that our performance gains stem from our method rather than the increased network capacity. More importantly, simply adding MLPs not consistently beneficial strategy for all the visual tasks and can be detrimental, resulting in inferior performance some tasks even compared with the plain baselines. This is because naively adding large MLPs transformer blocks may lead to challenges in optimization and suffers from a small training dataset compared with LLM pre-training.
B.3 VARYING LLM TRANSFORMER SCALES
This section analyzes the influence of the scales of language trans- formers with OPT (Zhang al., 2022). Our experiments incor- porate the final transformer layers from OPT-(125M, 350M, 1.3B, 2.7B, 6.7BJ. into ViT-S for image classification. Our experiment setting builds upon DeiT (Touvron et al., 2021) and trains for 100 epochs. Additionally, our experiments with small-scale OPT (OPT- (125M,300M) even yield the loss values of NAN when trained with the original DeiT learning rate, so we decrease their learning rate by 1/5 for stable training. This reflects the importance of the scales of LLMs for stabilizing the training.
Model
Ace
VIT-S
75.25
OPT-
71.63
OPT-350M
71.56
OPT-1.3B
75.62
+ OPT-2.7B
75.74
+ OPT-6.7B
76.29
Table C: Accuracy improves with a larger transformer.
As indicated by the results in Table C, the benefits of frozen language transformer grow with increasing capacity of OPT transformers. The added transformers enhance the performance only with sufficient sizes (1.3B, 2.7B, 6.7B) and hurt the accuracy when the sizes are small (125M, 350M). Therefore, the phenomenon of LLM transformers enhancing visual tasks only "emerges" sufficient scales.
19

---

Published as a conference paper at ICLR 2024
(a) Image Classification (ImageNet)
(b) Point Cloud Recognition (ScanObjectNN)
Methods
Acc
Methods
BG OBJ T50
VIT-S
80.1
PointBert
87.4 88.1 83.1
+ I.LaMA (Ours)
80.7
.LLaMA (Ours)
88.0 88.5 83.8
MLP
MLP
86.5 87.3 83.4
Random LLM
76.9
Random LLM
87.2 88.0 82.6
(c) Action Recognition (SSv2)
(d) Motion Forecasting (Argoverse)
Methods
Accl Acc5
Methods
ADE| FDE. MRJ
VIE-S
89.2
mm Transformer
0.72
1.10
10.7
LLLMA (Ours)
65.9 89,9
+LLaMA (Ours)
0.71
1.08
10,5
ML.P
63.8 88,9
+ +MIP
0.74
1.16
11.8
Random LLM
64.1 88.8
RRnddm LLM
0.74
1.15
11.5
(c) 2D Retrieval (Flickr30k)
(f) 3D VQA (SQA3D)
Methods
EMI
EM5 EM10
Methods
EMI
EM10
METER
49.66 80.86 89.48
SQA3D
47.20 86.82
+ .LLaMA (Ours)
50.22 82.26 90.08
+ I.LaMA (Ours)
48.09 89.03
ML.P
49.48 81.12 89.58
MLP
47.14 88.08
RRandom LLM
49.80 81.62 89.72
Random LLM
47.26 88.46
Table B: Addition comparisons for model capacity. We compare our approach of adding the frozen
LLM transformer with adding randomly initialized MLP ("+MLPP") LLM blocks (*+Random LLM")) and training end-to-end. The results on diverse tasks uniformly support that our improvement not merely the result of a larger model capacity. Details are Sec. B.2.
Model
Configuration
Acc Top
VIT-S
Baseline VIT-S
75.32
+ LLaMA at Tail (Ours)
Single LLaMA block at the end of VIT
75.84
+ LLaMA at Middle
Single LLaMA block at the middle of VIT
75.55
+ LLaMA at Head
Single LLaMA block at the head of VIT
72.66
+ LLaMA Blocks
2 ILLMA blocks the end of ViT
77.10
Table D: Varying the position and number of LLaMA blocks.
B.4 LAYERS TO INSERT THE FROZEN LLM BLOCK(S)
Table D presents an ablation study on different architecture variants, including different places to insert the LLM blocks and the number of LLM blocks. Without loss of generality, we leverage the image classification with VIT-S for our ablation study. From the experiments, we mainly have the following discoveries:
Inserting the frozen LLaMA block at the beginning or the middle of the visual encoder performs worse than our strategy of inserting LLaMA at the end of the encoder. This supports our design choice and verifies the intuition that L.M blocks are more suitable for high-level semantics instead of low-level visual patterns.
LLM block location.
LLM block number. We also experiment with inserting the last 2 blocks from LLaMA, and find it to be better than our default strategy of using single LLaMA block. Due to computation constraints, we are unable to extend this to diverse computer vision tasks as in Sec. 4, but this avenue presents interesting direction and broadening understanding of LLMs.
B.5 BREAKDOWN METRICS 3D VQA
We additionally provide the full metrics on SQA3D in Table E, supplementing Table 5b. As shown the table, adding a frozen transformer improves the performance on the main metric and most of the analytical metrics.
C IMPLEMENTATION DETAILS
C.I IMAGE CLASSIFICATION
We follow DeiT (Touvron et al., 2021) in training the models of VIT-(T,S,B) and VIT-(T,S,B)- LLaMA in Sec. 4.1. Each visual token is 16x16 patch 224x224 images. We adopt the
20

---

Published as a conference paper at ICLR 2024
Methods
EM@1
EM@ 10
What
Is
How
Can
Which
Others
ScanQA (Azuma al., 2022)
46.58
85.97
31.64
63.80
46.02
69.53
43.87
45.34
Multi-LLPP (Delitzas et al., 2023)
48.02
SQA3D (Ma et al., 2023)
47.20
86.82
33.48
66.10 10
42.37
69.53
43.02
46.40
SQA3D-LLaMA
48.09
89.03
34.27
67.05
48.17
68.34
43.87
45.66
Table E: Performance of SQA3D and adding language transformers on 3D question answering (QA).
Adding LLaMA achieves the best performance. EM@) I and EM@ 10 means Top-1 and Top-10 Exact Match (Accuracy) metric. "What," "Is," "How," "Can," "Which," and "Others" are detailed breakdown of question types reported in EM@1.
identical procedure for ViT-T and VIT-S for a fair comparison. The most important configurations include a total of 300 epochs, a base learning rate of 5e-4, a cosine annealing learning rate sched- ule (Loshchilov & Hutter, 2016), and an AdamW optimizer (Kingma & Ba, 2014; Loshchilov & Hutter, 2017). The total time for training lasts 4-6 days on AxA100 GPUs. The only change we adopt the warm-up length of 20 epochs, compared with the original warm-up of 10 epochs in DeïT. A longer warm-up stabilizes the training of ViT models and also enables us to slightly outperform the original ViT-T and ViT-S performance in Table 1.
C.2 POINT CLOUD CLASSIFICATION
In this section, we describe the implementation details of the point cloud classification method presented in Sec. 4.2. For optimization, & Hutter, 2017) optimizer and a cosine annealing learning rate schedule (Loshchilov & Hutter, 2016). To map the dimension between PointBERT and LLaMa transformer tokens, we add two linear layers with a learning rate of 5e-5. The PointBRRT backbone has a learning rate of 5e-4, consistent with the original setting in Yu al. (2021). We fine-tune our model for 300 epochs on both the ScanObjectNN (Uy et al., 2019) 9) and ModelNet40 (Goyal et al., 2021) datasets. The training takes around 6-10 hours on A10000 GPUs
the AdamW (Kingma & Ba, 2014; Loshchilov
C.3 ACTION RECOGNITION
We investigate action recognition in See. 4.3 and provide more details of implementation here. Our setup strictly follows VideoMAE (Tong et al., 2022), where a VIT model is (1) pre-trained by masked auto-encoding (He et al., 2022); then (2) fine-tuned for additional epochs. As stated in Sec. 4.3, we directly begin from the second step and inherit the parameters of self-attention blocks in VIT from pre-trained VideoMAE models. During the training process, VideoMAE trains ViT-S for 40 epochs (5 epochs of warm-up) and ViT-B for 30 epochs (5 epochs of warm-up), where we adopt the same length of training. The optimizer is AdamW (Kingma & Ba, 2014; Loshchilov & Hutter, 2017) with a cosine annealing learning rate schedule (Loshchilov & Hutter, 2016).
In Sec. 4.3 and Table 3, our ViT-S and ViT-B performance is lower than the reported numbers in VideOMAE. This is because VideoMAE uses 32~64 GPUs during the fine-tuning stage and supports a much larger batch size compared with ours, though we scale the learning rate according to the batch size as Goyal al. (2017a). Concretely, VideoMAE adopts the batch size of 384 video clips, while our computational resource only supports batch size of 24 clips and 12 clips for ViT-S and ViT-B, respectively. However, we control the setup between ViT and ViT-LI for fair comparison. Finally, the ViT-S-LLaMA and ViT-B-LLaME experiments take around 3-4 days 4 A10000 GPUs.
C.4 MOTION FORECASTING
We evaluate the effects of frozen LLM transformers in Sec. 4.4 with motion forecasting. For clarity, we intu- itively demonstrate its problem setting and modular ar- chitecture in Fig. G, where VectorNet (Gao et al., 2020) or mm Traasffoorer (Liu et al., 2021) encodes each lane agent trajectory into token embedding, and then LLM blocks process these tokens and feed them into the regression decoder.
Model
ADF((k=1)
FDE((K=1)
Paper
1.66
3.67
Ours
1.60
3.60
Table F: Our implementation of Vector-
Net is better than their paper. Please note that this table uses the same single-modas modal setting (k=1) as Vector- Net for a fair comparison.
21

---

Published as a conference paper at ICLR 2024
Since VectorNet and mmTransformer have not released their training code, we reproduce their results on our own and achieve better or similar results as reported in their papers. As in Table F and Table G, the baselines used in our paper (Table 4) have comparable or even better performance compared with their original performance the papers, which is critical for a fair and meaningful comparison.
During the training time, we separately train VectorNet or mnTransformer. VectorNet chitecture, so its training lasts 60 epochs, with a cosine annealing learning rate schedule (Loshchilov & Hutter, 2016). We use the AdamW optimizer (Kingma & Ba, 2014; Loshchilov
relatively simple ar-
Model
ADE]
FDE MR.
Paper
0.71
1.15
10.6
Ours
0.72
1.10
10.7
Hutter, 2017) with learning rate of 5e-4 and batch size equal to 32 samples. For mm'rrans- former, we train it with the same learning rate, batch size, and optimizer VectorNet. The training lasts 32 epochs,
Table G: Our implementation of mm- Transformer is comparable to the per- formance in their paper, with large ad- vantages on the main metric of FDE. where we drop the learning rate by 1/4 on epochs 20, 24, and 28. The training time for both models is around 2 days one A100 GPU.
Lane
Tokens
Lanes
Agents
VectorNet
Fl.
FLM
Fi
Regression Decader
mnTranaformer
Agent
Tokens
Figure G: Illustration of the typical motion forecasting design and our implementation. Motion forecasting models the trajectories of agents and lanes as polylines. Exiting models (Gao et al., 2020; Liu et al., 2021) use MLPs or transformers to convert the lanes and agent trajectories into token embeddings, and then employ decoder to regress the future trajectories. In our design, we treat either VectorNet (Gao et al., 2020) or mm Tnnrrnsfoormrr (Liu et al., 2021) as general encoder, and then insert the frozen LLM blocks to process their embeddings.
C.5 2D VISION LANGUAGE
This section provides more details on implementation and designs to supplement our discussion on 2D vision- -anguale models in Sec. 4.5. Our experiments adopt the widely-used METER (Dou et al., 2022) as the baseline and incorporate pre-trained LLM transformers after its vision-language fusion module. In Fig. H. intuitively illustrate the modular design of METER and the specific place to insert our frozen LLM transformer and linear layers.
Conventionally, METER follows two-stage training strategy: (1) pre-training the whole vision- language model (VLM) on a large combination of vision-language datasets, including COCO (Lin al., 2014), Conceptual Captions (Sharma al., 2018), SBU Captions (Ordonez al., 201 I), and Visual Genome (Krishna et al., 2017); (2) fine-tuning on downstream tasks like visual question answering (VQA) or image-text retrieval. However, the first step of pre- pre-training computationally extensive, so we adopt the setup in Shi et al. (2023) by skipping the pre-training step and directly training on the target task. Specifically, we initialize the image encoder from CLIP--/32 (Radford et al., 2021) and text encoder from RoBERTa (Liu et al., 2019), and then fine-tune all the modules jointly expect for the LLM transformer FLM. Because of the initialization from CLIP and RoBERTa, model is capable of predicting reasonably.
During the training stage, we strictly follow the same hyper- parameters and configurations on VQAv2 (Goyal et al., 2017c) and Flickr30k (Plummer al., 2015) provided by METER. The most critical detail is that METER assigns different learning rates for each module. For example, the cross-modal fusion module and the decoder have larger learning rates compared with the pre-trained
22

---

batch size of Published as a conference paper at ICLR 2024
image and text encoders. Similarly, experiments set the learning rates of linear layers (F), and F2) 10x the learning rate of the image encoder, because they randomly initialized. Finally, each training on VQAv2 and Flickr30k lasts for 10 epochs and around day on 4xA100 GPUs.
Image Encoder
Modal Fi
Decoder
What the woman doing in the photo?
Text Encoder
FR
Figure H: Illustration of the METER (Dou et al., 2022) framework and how to insert the frozen language transformer (pink FIM) and linear layers (green F), and F2) process the visual tokens after vision-language fusion.
C.6 3D VISION LANGUAGE
This section provides more details of the dataset and training configurations of the 3D vision- language task. We conduct our experiments on the SQA3D dataset (Ma et al., 2023), which contains 33.4k questions in 650 unique ScanNet scenes. In addition question answering (QA), the bench- mark also requires the model to understand its situation (position, orientation, etc.) the 3D scene described by text. Hence is called situated question answering (SQA). We 32 during our model training, and Adam as our optimizer. The hidden size for each embedding token 768. We train all parameters from scratch for 30 epochs, and decrease the learning rate by 10 times at 10, 15, and 20-th epoch. The model is trained on single A100 GPU.
C.7 DEPTHS OF LLM LAYERS
When varying the depth of transformer blocks in Sec. 5.2 and Fig. 2, we adopt the ablation setup of training for 100 epochs, compared with the full training of 300 epochs. The experiments are based on ViT-S/16 (Dosovitskiy et al., 2021) in DeiT (Touvron et al., 2021) with a batch size of 1,024, which is also used in other ablation experiments. The whole training process involves 20 epochs of warm-up and the remaining 80 epochs adopt a cosine annealing learning rate schedule (Loshchilov & Hutter, 2016). Each experiment takes around 2 days on 4x A100 GPUs.
23


Reviews:
[{"review_id": "XjTXCeqrIl", "reviewer": "Reviewer_SKT3", "summary": "Previous work has shown that large language models (LLMs) that are trained only on text can be effectively sutured with visual models via simple linear layers or more complicated cross-attention modules, so as to build multi-modal vision-language frameworks.\n\nThis paper studies an interesting problem and tries to answer whether LLMs can deal with single-modal visual tasks without the help of language inputs.\n\nA major finding of this work is that a frozen transformer block from pre-trained LLMs can be used as an effective block in a visual encoder to process visual tokens. \n\nThe authors have verified this finding via comprehensive experiences with a variety of 2D and 3D vision tasks.", "questions": "In addition to the above weaknesses, here are more questions:\n\n- Why only insert one frozen LLM block for visual encoding? can adding more blocks bring more performance again?\n\n- Why choose the last transformer block from LLaMA-7B for experiments? Can other blocks also work well? Is there any guidance to choose which block?\n\n- Is there any other way to insert the frozen LLM block in visual encoders? How about adding it at the beginning of visual transformers, or in the middle of visual transformers?\n\n- The authors removed the attention mask and positional embedding for the frozen LLM block, is there any experimental study for these designs?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- ***Scope and relevance***: Considering the growing research interest in LLMs, this paper is exceptionally timely as it broadens their application towards visual perception tasks.\n\n- ***Significance of contributions***:  This paper proposed a straightforward yet effective approach to incorporate a frozen transformer block\nfrom a pre-trained LLM as a general-purpose visual encoder layer that can directly process the visual tokens. \n\n- ***Experimental results***: The experiments in this paper are enough to prove the effectiveness of the proposed method and revealed findings.\n\n- ***Clarity***: The main body of the paper is written very well.", "weaknesses": "- ***Proposed hypothesis***:  The authors proposed the information filtering hypothesis to explain why pre-trained LLMs can be used for visual encoding. To verify this hypothesis, feature activations regarding both magnitudes and frequencies of features are visualized for comparison. However, visualizing feature activations or attention maps cannot explain their finding well, as transformers (both in ViTs and LLMs) are built with the self-attention mechanism. \n\n- ***Technical details***: Although the inserted LLM block is frozen, is still not clear whether the performance gains truly stem from it. How about adding a random initialized LLM block for comparison?  In Table 2, about the point cloud recognition task, the performance gains are marginal, especially on ModelNet40 (degraded performances for 1k and 4k). \n\n- ***Theoretical justification***: Despite not being compulsory, can the author provide some theoretical justifications rather than experimental observations to explain why the language-trained LLM block can work for visual encoding? Or any potential theory that might lead to a better explanation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PFkLSSBvPW", "reviewer": "Reviewer_sfsm", "summary": "The paper proposes to incorporate frozen transformer layers from large language models (LLMs) into vision models when training the model from scratch. By evaluating this approach on a variety of vision tasks, the authors find that when the LLM layer is included, the resulting models observe a slight boost in performance. Finally, the paper proposes the information filtering hypothesis as an explanation to why this happens. Namely, the pre-trained LLM transformer layer focuses the attention and information flow on object-related tokens similar to how self-supervised models such as DINO learn semantically meaningful attention maps.", "questions": "- Which task and dataset was evaluated for the results in Tab. 6 and Fig. 2?\n- The information filtering hypothesis allows some valuable insights. At the same time, it seems to be most prominent after $F_{LM}^A$ judging by Fig. 3, but then in Fig. 4, $F_{LM}^A$ obtains a low score for ViT-B-LLaMA. Why is this the case? Which model size is evaluated in Fig. 3? Is this observation not consistent across architectures?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Reusing pre-trained layers from LLMs is a novel and intriguing idea that could spark further research in the direction of knowledge transfer from foundation models to more specific use cases.\n- The experimental evaluation is extensive covering image classification, point cloud detection, video action recognition, motion forecasting, VQA, image retrieval. The proposed method achieves a small boost in performance across the board. The ablation study highlights some important design choices, e.g., which LLM layer to transfer.\n- The information filtering hypothesis allows some insight and inspection into the reason for why this method works.\n- Good writing style and presentation making the paper an easy read.", "weaknesses": "- The experimental improvements are most of the time rather marginal with a few cases showing practical improvements. One could argue that ViT-S-MLP from Tab. 6 would be a fairer comparison for all experiments as it more closely matches the parameter count. In this case, one would expect the gap to further close.\n- Some additional ablations might have been of interest.\n    - Why only transfer a single LLM layer? Does it also work with more?\n    - Why is the layer inserted at the end of the ViT? How does the outcome change when the layer is inserted earlier?\n    - How would the results change if we instead transfer a pre-trained vision layer? (see next point)\n- The transfer from a language model to a vision model is interesting and most surprising that it works. However, to generalize the approach, one could have also tested to transfer layers from other types of foundation models, e.g. vision-language models (CLIP) or self-supervised vision models (DINO). If there is a considerable gap between LLM and vision layers, one could conclude that, while it works, LLM layers might not be optimal for the domain.\n- It remains an open question if this approach scales with large-scale models and training. One could expect diminishing returns as data and model allow for learning equally capable layers. This might still make this approach attractive for smaller scales so it would be interesting to know about this potential limitation. While this is difficult to evaluate, are there any indications how it might behave?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K3ytcJTLvi", "reviewer": "Reviewer_ksvc", "summary": "This paper explores the use of a transformer layer from a pre-trained large-language model as part of visual encoders. A frozen transformer block from a pre-trained LLM, typically the last block, is inserted towards the end of standard ViT based visual encoders and this layer is kept frozen throughout the visual encoder training. Experimental results across a range of vision and vision-language benchmarks are presented demonstrating the superiority of the proposed LLM-block enhanced visual encoder against the baseline visual encoder. In explanation to this phenomenon, the paper hypothesizes that LLM-blocks discern important visual information and amplify their effect, and  finally presents qualitative and quantitative results supporting such hypothesis.", "questions": "What happens to the results in Tables 1, 2, 3, 4 and 5, if we don't use the LLM weights and just use learnable random weights?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper proposes a simple approach to enhance the capabilities of many existing visual encoders for multiple tasks by just adding frozen LLM blocks with some projection layers into the training process. Experiments are presented on a breadth of image, video, 3D, visual-question answering tasks covering multiple vision problems. Hence, the approach has a broad applicability to the community.\n\nThis paper also analyzes why their method works and presents a promising hypothesis: the frozen-LLM selectively amplifies the informative tokens. Fig. 3, 4 and 5 offer evidence to this hypothesis. \n\nThe paper is well-written and easy to follow.", "weaknesses": "The main weakness of this paper is the choice of the baseline. \n- The proposed approach adds an additional (frozen) LLM block and (learnable) projection layers to the baseline network. So, it's unclear whether the performance improvements in Tables 1, 2, 3, 4 and 5 are just because of the increased network capacity.\n- The ablation in Table 6 already demonstrates that matching the MLP parameter count already bridges some gap between the proposed method and baseline, revealing a clear inferiority of the baseline used in rest of the tables.\n- More importantly, the main (implicit) claim in this paper is that the (frozen) LLM weights have some transferrable knowledge to vision tasks and they can be exploited using the proposed methodology. So, an ideal baseline to support this claim should be the same model with (learnable) weights that are randomly initialized instead of LLM weights.\n\nWithout a good baseline, the strong claims across multiple tasks are not well supported.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2H2fUgoSY3", "reviewer": "Reviewer_8b8L", "summary": "This paper suggests a new paradigm for training ViTs and ViT-based multimodal Transformers, which indicates that the performance of ViTs and ViT-based multimodal Transformers can be enhanced by appending a frozen Transformer layer inherited from LLMs behind the original ViTs. Furthermore, this paper develops an information filtering hypothesis to explain how the frozen LLM transformer layer can benefit visual tasks by distinguishing the informative tokens for scenarios directly using all visual tokens and amplifying informative tokens' contributions to [CLS] token for scenarios only using the [CLS] token.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is generally well-written and easy to follow.\n- This paper provides detailed investigations of how and why the proposed paradigm works.\n- The experimental results verified the proposed paradigm that training ViTs with frozen transformer layers inherited from LLMs is simple yet effective.", "weaknesses": "- Method\n  - The statement \"we place the LLaMA transformer behind the last self-attention block in ViT\" is inconsistent with Figure 1. It is unclear why the LLaMA transformer should be placed behind the last self-attention block instead of the last layer (i.e., behind the last FFN block) in ViT. Could the authors provide some insights?\n\n- Experiments\n  - Some baselines are missing. \n    - For example, it is not persuasive enough to only compare with \"ViT-MLP\" in Table 6, whose number of trainable parameters remains unchanged, but the total number of the parameters is significantly reduced, especially when layers of LLMs have notably more parameters than layers of ViT -Base/-Small. Therefore, an important baseline that should be added is to append frozen ViT layers behind the original ViT to keep the total number of parameters the same as \"ViT-S-LLaMA.\"\n    - Moreover, the total number of parameters, FLOPs, and throughput/latency of ViTs with the LLM layer inserted should be reported to investigate whether the proposed paradigm is more competitive than directly scaling up vanilla ViTs.\n  - Some ablation studies are missing.\n    - For example, append the frozen LLM layer behind prior layers instead of the last layer in ViT.\n    - How does the number of the appended frozen LLM layers affect the model performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)