Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs
Authors: Mustafa Shukor
Matthieu Cord
Venue: NeurIPS 2024 poster (2024)
Keywords: Large multimodal models
LLMs
cross-modal alignment
efficiency
safety
interpretability
Primary area: machine_vision
TL;DR: We analyse frozen LLMs when exposed to multimodal inputs. Inside LLMs, multimodal tokens are significantly different than textual ones, yet they are implicitly aligned. Our findings led to implications on performance, safety and efficiency.

Abstract: Large Language Models (LLMs) have demonstrated impressive performance on multimodal tasks, without any multimodal finetuning. They are the de facto building block for Large Multimodal Models (LMMs), yet, we still lack a proper understanding of their success. In this work, we expose frozen LLMs to image, video, audio and text inputs and analyse their internal representation with the attempt to understand their generalization beyond textual inputs. Our work provides the following **findings.** Perceptual tokens (1) are easily distinguishable from textual ones inside LLMs, with significantly different representations (e.g. live in different narrow cones), and complete translation to textual tokens does not exists. Yet, (2) both perceptual and textual tokens activate similar LLM weights. Despite their differences, (3) perceptual tokens are implicitly aligned to textual tokens inside LLMs, we call this the implicit multimodal alignment effect (IMA), and argue that this is linked to architectural design, helping LLMs to generalize. This provide more evidence to believe that the generalization of LLMs to multimodal inputs is mainly due to their architecture.  These findings lead to several **implications.** This work provides several implications. (1) We find a positive correlation between the implicit alignment score and the task performance, suggesting that this could act as a proxy metric for model evaluation and selection. (2) A negative correlation exists regarding hallucinations (e.g. describing non-existing objects in images), revealing that this problem is mainly due to misalignment between the internal perceptual and textual representations. (3) Perceptual tokens change slightly throughout the model, thus, we propose different approaches to skip computations (e.g. in FFN layers), and significantly reduce the inference cost. (4) Due to the slowly changing embeddings across layers, and the high overlap between textual and multimodal activated weights, we compress LLMs by keeping only 1 subnetwork (called alpha-SubNet) that works well across a wide range of multimodal tasks. The code is available here: https://github.com/mshukor/ima-lmms.

Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs
Mustafa Shukor
Matthieu Cord
"Sorbonne University, Valeo.ii
Abstract
Large Language Models (LLMS) have demonstrated impressive performance on multimodal tasks, without any multimodal finetuning. They the de facto building block for Large Multimodal Models (LMMs), yet, we still lack a proper understanding of their success. In this work, we expose frozen LLMs to image, video, audio and text inputs and analyse their internal representation aiming to understand their generalization beyond textual inputs. Our work provides the fol- lowing findings. Perceptual tokens (1) are easily distinguishable from textual ones inside LLMs. with significantly different representations (e.g. live in different nar- row cones), and complete translation to textual tokens does not exist. Yet, (2) both perceptual and textual tokens activate similar LLM weights. Despite being different, (3) perceptual tokens are implicitly aligned to textual tokens inside LLMs, we call this the implicit multimodal alignment effect (IMA), and argue that this is linked architectural design, helping LLMs to generalize. This provide more evidence to believe that the generalization of LLMs to multimodal inputs is mainly due to their architecture. These findings lead to several implications. (1) We find a positive correlation between the implicit alignment score and task performance, sug- gesting that this could act proxy metric for model evaluation and selection. (2) A negative correlation exists regarding hallucinations (e.g. describing non-existing objects in images), revealing that this problem is mainly due misalignment between the internal perceptual and textual representations. (3) Perceptual tokens change slightly throughout the model, thus, we propose different approaches skip computations (e.g. in FFN layers), and significantly reduce the inference cost. (4) Due to the slowly changing embeddings across layers, and the high overlap between textual and multimodal activated weights, we compress LLMs by keeping only subnetwork (called a-SubNet) that works well across a wide range of multimodal tasks. The code is available here:
Introduction
Large Language Models (LLMs) [1, 2, 3, 4, represent a noteworthy advancement in recent AI developments. Building upon the success of LLMs, the next stride this field involves extending beyond the textual modality, giving rise to Large Multimodal Models (LMMs) [6, 7, 8, 4]. A notable line research involves connecting LLMs to visual encoders, while keeping them frozen and only training connector with modest number of parameters [9, 10, 11, 12, 13, 14, 15, 16, 17]. These methods yield comparable performance [10] to large-scale multimodal models with significantly reduced computational and data budget.
Keeping all pretrained unimodal models frozen and only training couple of millions of parameters [9, 10, 12] is an interesting phenomenon to understand, with limited research trying to decipher it. To
"Contact: ffrssmame.llsttamme esorbonne-uuiversite.ff
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Analyzing LMMs
Implications: performance, safety efficiency
Framework
Multimodal Narrow Cones
Parformance and Safety
LMM
steering blocks
Ha Muldinafione
ACe
IMA score
IMA 900269
Computational Efficiency
Implicit Muttimodal Alignment (IMA)
LLM
DB0
PPN
LLM P/a
ODTONA D0
compression
Skipping Computations
Summary of the work. We start by analysing multimodal tokens inside LLMs, and find that they live in different spaces (e.g., multimodal cones). Yet they are implicitly aligned (i.e. IMA), allowing us to see LLMs residual streams with steering blocks. This lead to implications on performance, safety and efficiency.
Figure I:
explain why frozen LLMs can generalize beyond textual inputs, several hypotheses can be isolated: (1) perceptual tokens are transformed to textual ones, can be simply considered as foreign language [18], and thus the LLM sees text-only tokens. (2) LLMs are able to digest non-textual tokens that are processed by (2a) mopeeit subnetworks or (2b) the same LLM weights that can generalize due to other reasons.
In this study, we expose LLMs to different multimodal inputs, such as image, video, audio and text, and analyse their internal representations. We focus on frozen LLMs and consider two representative setups: single-task (ST) and multitask (MT) finetuning. The former is considered by parameter and data-efficient approaches [11, 12, 9, 10] and consists of training a mapping module for each dataset. The MT setup considered by recent multimodal assistant models [19, 14, 20, 21], and consists of training the same mapping module on several datasets/tasks.
Our study shows (1) that perceptual and textual tokens still live in significantly different representation spaces inside LLMs (Sec. 3.1): live in different narrow cones, and have different norms, rate of change and vocabulary distributions. (2) We notice high similarity between the weights activated by textual and perceptual tokens (Sec. 3.2), allowing to swap these activated subnetworks between different tasks and modalities. Despite their differences. (3) perceptual tokens are implicitly aligned to textual ones across different stages (Sec. 4.1): during training of the mapping module, and during inference across LLM layers, especially inside each LLM block (e.g. after the self-attention layer). As there is no explicit objective to align these representations, we call it the Implicit Multimodal Alignment effect (IMA) (i.e., increasing similarity between textual and perceptual token distributions). We find that this effect is mostly linked to architectural design (e.g. residual stream with refinement blocks acting as steering blocks Sec. 4.2). This provides more evidence to believe the architecture of LLMs is one of the main factors to generalize to multimodal representations.
We shed light on several practical implications (Sec. 5). (1) We find a positive correlation between the implicit multimodal alignment score and the task performance, suggesting that this score could act as a proxy metric. On the other hand, (2) we find a negative correlation with hallucinations, revealing that the main factor leading to this problem is the lack of alignment between the internal representation of textual and perceptual inputs. (3) The perceptual tokens slightly change inside LLMs, thus, we propose to skip their computations (e.g. inside FFN layers). (4) Due to the slowly changing embedding across layers, and the high overlap between weights activated by different modalities, we compress the LLM by keeping one task-agnostic subnetwork that works well across all modalities. To summarize (Fig. 1), we analyse the internal representations of LLMs when exposed to multimodal inputs, leading to the following findings:
Perceptual and textual tokens live in different representation spaces inside LLMs.
They activate similar LLM weights.
They are implicitly aligned (IMA) inside LLMs, during training and during inference.

---

The architectural design of LLMs can be perceived as residual stream with steering blocks. We argue that this one of the main factors allowing LLMs to: digest very different tokens, drive the implicit multimodal alignment effect, and thus generalize to different modalities.
These findings have several practical implications such as:
The IMA score proxy metric candidate for task performance and hallucinations.
Hallucinations as a result of lack of sufficient multimodal alignment.
Skipping computations for visual tokens, leading to efficient inference.
LLMs compression by keeping only subnetwork that generalizes to multimodal tasks.
2 Framework for analysing preceptually augmented LLMs
General framework We focus a general family of models that consists of: frozen language model LLM with L layers, a trainable mapping module C. and a frozen perceptual encoder E& for different modalities M (e.g. image (I), video (V) and audio (A)). The LLM input X consists of the concatenation of [pi.....PI PN,,] multimodaal/prrcettuua tokens (referred to as prompt) with T= l... NN.] textual tokens. The prompt P is obtained after encoding the modality-specific input XM with the corresponding EM and using C' to project it to the LLM input space. T is obtained from the embedding layer ET applied to the tokenized input text XT. This can be expressed as follows:
P= C(M(EMXXM), T T-ET(X,, O= D=LLM(PM ((P.T).
(1)
The & NN + N2) output tokens O jo.,... on] are obtained after a normalization, followed by Our focus is on the internal
the unembedding layer Wont (or LLM head, i.e. o, = W representation of LLMs (i.e. tokens) at different stages, in particular across the L LLM blocks/layers (referred to as B). The mechanism inside the / + LLM transformer block can be expressed as follows:
(2) where FC1, FC2, are the up and down projections and activation inside the FFN, LN1/2 are the layer norms and SAA the self-attention.
X = XSA + FCZIg((
XSA= X' + SA(LNI(X')),
Perceptually augmented LLM baselines. For the single-task (ST) setup, we train many models across different datasets that span image, video and audio-text modalities. Each mapping module trained on a specific dataset, similar to previous works [10, 12, 9]. Inspired by previous studies [10, 12], we use wigght transformer consisting of self-attention to attend to perceptual tokens. In this setup, P refer to perceptual tokens from image. video and audio modalities. For the multitask devise different variants of the LLLVA-1.5 [19] model that differ from the original model as follows: ILL (LLMM kept frozen), I.LaVA-1.533 (LL.M kept frozen, without pretraining) and V.L (LLM kept frozen, without pretraining and with transformer mapping module similar to the ST setup instead of MLP)). In this setup, P refers to image tokens from different it most similar to the ST setup, and analyse other variants in App. E. For analysis (i.e. Sec. 3), we focus on Vicuna-v1.5. 1.5-BB [22] as it is shared by both setups. For the ST, we use unimodal encoders, such as ViT [23], Tirmerormer [24] and AST [25] that are not aligned with text. More implementation details, and experiments with other backbones can be found in App. and App. E. We report the similarity after averaging the tokens SimAvg (Eq. (3)) and details other measures in App. E.
(MT) setup,
datasets. In the paper, we focus on LLa VA-15--5
Analysis tools. We are interested cross-modal or multimodal alignment, and define the alignment in terms of the cosine similarity; the higher the score, the more the vector representations are pointing in similar directions. This could also indicates how much the two token distributions or vectors are close, in terms of L2 distance (assuming the vectors are normalized and in a narrow cones). In other words, alignment and similarity terms can be used interchangeably in the paper. In addition to cosine similarity we also study their norm, decoded vocabulary distributions and which LLM weights they activate. In the paper, we focus on the global representation per example, by analysing their average across the sequence. More finegrained analysis on the token level with different similarity and norm
3

---

measures gives similar observations and are detailed in App. E. For instance, we compute the cosine similarity between perceptual (P) (e.g,, tokens corresponding to image patches) and textual (T)
tokens (e.g., tokens corresponding to the image caption), after the block 1 as follows:
Pt T || ||||||
E Ni
Sim pt T
pi=
T
(3)
N
3 LLMs indeed generalize to non-textual tokens
We investigate the generalization of LLMs multimodal inputs, by studying the perceptual and textual tokens inside LLMs. We investigate if all tokens are projected to textual ones, or rather they are still different and how so (results with other models and similarity measures in App. E).
Tay
(a) ST setup. Tokens from multimodal datasets.
(b) MT setup. Tokens image-text datasets.
Figure 2: Multimodal narrow cones. The cosine similarity after LLM blocks (B) between: perceptual tokens
(P vs P). textual tokens (T' vs T), perceptual and textual tokens (P vs T). p vs p and vs refer to the intra similarity within the same dataset. We also visualize the t-SNE of tokens (at layer 24) showing they stay separated inside the model. V (Video), (Image), A (Audio).
3.1 How perceptual tokens differ from textual ones?
Multimodal cones: different narrow cones for different modalities (Fig. 2). Previous works [26, 27, 28, 29, 30, 31] have found the representation of contextualized embeddings in language models to be anisotropic: embeddings of different inputs exhibit high cosine similarity, shaping a narrow cone, where all embeddings point in the same narrow direction. In the multimodal domain, the cone effect is also observed [32] in contrastive models (CLIP [33]). In this section, we investigate if textual and multimodal tokens live in narrow cones inside L.Mss, and if these cones distinct. We compute the tokens cosine similarity at different layers. In particular, the unimodal similarity: text-only (I' 7) and perceptual-only (P P), and the cross-modal similarity (P vs 7') between perceptual and textual tokens. Note that for the ST setup, /' vs P covers the similarity between image, video and audio tokens, while for the MT ones cover image tokens from different datasets. Fig. 2 shows clear narrow cone effect for textual and perceptual tokens. Different perceptual modalities seem to live different narrow cones, as shown by the low vs P score for the ST setup. Interestingly. the cross modal similarity between textual and perceptual tokens (P vs T) is significantly lower, suggesting that textual and perceptual tokens also live in different narrow cones. We also visualize the t-SNE of the tokens embeddings showing they stay separated inside the L.M.
211
31
213
Figure 3: Tokens norm and evolution across LLM layers. The tokenwise cosine similarity between consecutive blocks (c.g. x and X'), and the median token L.2 norm after each block (X') for the ST (left) and MT (right) setups. Textual and visual tokens evolve differently inside L.L.ss.
The

---

Different token norms and evolution across layers (Fig, 3), We compute the median of the token L2 norms after each LLM block. This shows that textual and perceptual tokens have different norms across layers. Perceptual tokens have significantly higher norm (at the beginning for MT and across all layers for ST), and they change significantly less. When looking at other norm measures, we found perceptual tokens with massive norms, similarly for textual ones [34], especially for the ST setup. We discuss massive tokens more in App. E.2. In addition, we compute the cosine similarity between tokens at block and block ( + showing that textual and perceptual tokens have different change rates. Textual ones change drastically at the beginning of the LLM, while perceptual ones changes significantly less across all layers.
Histogram (1:0)
KI -iveerggen (x x x!)
KI -ilieeggenne
Distribution entropy
Vi000d3777
Figure 4: Tokens vocabulary distribution inside LLMs. The LLM (Vicuna-v1.5) unembedding
layer is used to map each token at different LLM layer, to a probability distribution over the vocabulary. Multimodal tokens exhibit different vocabulary distributions across layers
Different token vocabulary distributions across layers (Fig. 4). For each token, unembedding (i.e. LLM head) to decode the latent representation to a probability distribution over the vocabulary. This approach have shown to work well for LLMs at different layers. not just the last one [35, 36, 37, 38J. We show the histogram of this distribution at the first LLM layer for both textual and perceptual tokens. The histograms show clear differences with some overlap between textual and perceptual prompts. In addition, we compute the Kl.-distance showing that the distributions diverge from each other across LLMs layers. We also notice that the distributions of textual tokens evolve significantly, compared to multimodal ones. This is shown by computing the KL-distance between consecutive blocks and the distribution entropy.
the LLM
Finding 1. Textual and perceptual tokens live in significantly different representation spaces inside L.MMs.
3.2 Do perceptual tokens traverse different paths inside LLMs?
For each trained model, we extract the LLM (frozen) subnetwork activated by each dataset/mmodality. We study these subnetworks (we refer to as pruning masks) by computing their similarity. We leverage the recent SoTT pruning approach (Wanda [39]), that prune models based on both the weights and the activation norms. Specifically, we use a handful (e.g. 256) of calibrated examples coming from different modalities, and keep only p% (1 sparsity) of weights with the highest Wanda score, at different sparsity levels (30 % and 50 %). Note that after removing more than 50% of weights we observe a severe degradation of performance.
Similar activated weights across modalities, in the first and deeper layers (Fig. 5). Each subnetwork is represented as a binary mask indicate which weights are activated. To compute the similarity between these networks, we consider the intersection over union (IoU). Results show an interesting high similarity between subnetworks activated by to different modalities. This high overlap is more seen for the ST setup. for example, the IoU between GQA and VQAv2 is 0.69, similar to GQA vs Audiocaps (0.67) or COCO-TexI (0.65). When looking at the IoU across layers, we notice an interesting high score at first layers. It seems that first layers encode general features that are common for all modalities. This similarity increases as we go deeper in the LLM, moving to more abstract and less modality-specinf representations, closer to the textual output.
Transfer of multimodal subnetworks across tasks and modalities (Fig. 6). To further validate the previous section, we study if we can simply interchange pruning masks between different tasks.
5

---

Figure 5: IoUs of multimodal subnetworks. loU of the subnetworks activated by different tasks and modalitics, for the ST (left) and MT (right) sctups. We show the evolution of IoU across LLM layers and across different multimodal tasks. Different modalities activate similar LLM weights (Fig. 22 for clearer version of the figure).
Modality:
Spersiyy 0.30
Medaliyy video IVIT Spardiyy 0.30
Medaltyy sudio (VTE Sparctyy 0.3)
117.36
59:23
65.19
29.30
(10)
5000 VO00770 3001
40.41
37.20
37.14
63.98
24.50
46.44
Tel SENVT: CEpOlrteea a00l 024 10oo
116.45
59.60
44.08
30.47
50.04
37.30
Grteooo Oats
24.38
138.17
59.67
03.71
49.47
36.60
64.06
24.61
Clicki
(VT. Sparstyy 0.23
IVT. Sparsty::000m 0.30
IVT.. Sparsity: 0.3)
123
67.13
116.26
18.90
110
65.66
115.8 83
17.27
:13
65.00
115.9 935.93
47.62
332
64.80
121.21
49:92
65.60
116.34
46
63.88
116.99
40.30
20
61.00 61.09
120.16
51.00
67.40
117.19
47.70
63.79
116.50
49.16
66 66.20
120.15
49
66
64.81
115.43
47
50.06
95
06
41.63
Figure 6: Transfer of multimodal subnetworks across tasks and modalities. The subnetwork activated by given task is used for other tasks Vicuna-v1.5. From left to right, transfer across: image, video and audio tasks. In each figure, the row corresponds to the subnetwork source dataset the column to the target dataset. hottom:
transfer across modalities for (from left to right): OPT, Llama 2, Vicuna-v1.5.
Specifically, for a given model trained to solve a particular task, we find the pruning mask using calibration data corresponding to different tasks/datasets. The sparsity is set 30%, which is often used to maintain reasonable performance. Fig. shows that the pruning masks transfer very well across tasks within the same modality (e.g. slight degradation by ~1 point CIDEr for captioning with a mask coming from OK VKA))). Similarly, we interchange masks across modalities. We fix the task (captioning) and also consider the text modalities (captions without images). In general, similar transfer with slight performance degradation, especially for OPT and Llama 2. We show similar observations with higher sparsity and with other encoders (e.g., CLIP and MAB) in App. E.4.
observe
Modality-speciiff subnetworks? The experiments suggest high overlap between weights acti- vated by different modalities. However, this does not exclude the possibility of finding weights that are generally activated when seeing a particular modality, even if there are small amount of them. More discussion about this can be found in App. E.4.
Finding 2.
LLM weights activated by perceptual and textual tokens overlap significantly.
4 What helps LLMs to generalize to multimodal tokens?
Textual and perceptual tokens have very different representations inside LLMs,, yet, LLMs are still able to process and generalize to these non-textual tokens. In this section, we try to investigate why this is possible, in particular, we identify which factors facilitate this generalization.

---

Exleddra
9.100
Pis Tap
6,25
V15
V15
1.0
Goans Simtare
3.20
Pwr Tpp
Suatheryy
E.20
1
0.8
1
8.6
3675
0.16
A
6ds0
0.4
C0.0
8.00
0.00
Lagers
15
20
23
30
Layere
Figure 7: Multimodal tokens similarity across LLM layers. The cosine similarity between the textual and multimodal tokens across: training epochs i.c., O, 1, for Vicuna-v1.5 (first), and across LLMs layers: Vicuna-v1. 1.5 (second) and LLa (last). Textual and multimodal tokens are implicitly aligned during training. and during inference across LLM blocks.
4.1 Observation: the Implicit Multimodal Alignment Effect (IMA)
Implicit alignment during training the mapping module (Fig. 7). We compute the cosine similarity between the perceptual tokens at the output of the mapping module, and textual tokens at different L.M blocks. Results show that this similarity increases all layers. This reveals that the mapping module role, is not just adapt the dimension of the visual tokens, but also project the visual tokens to be semantically, as similar as possible to the textual ones.
Implicit alignment during inference, across LLM blocks (Fig. 7). We compute the cosine similarity between perceptual and textual tokens after each LLM block. Here we compute the max of tokenwise similarity (MaxSim App. 1. for each pair of token sequences coming from one example (e.g. image prompt + caption), we take the maximum similarity, then we average across all examples.
The tokenwise similarity between perceptual and textual tokens significantly increases, especially in middle blocks, where the alignment is the highest in deep layers.
EXT 11
Falk
17
Token nomms
Consecative blocks similoriy
(c) Similuity inside each blook.
Figure 8: Multimodal tokens norms and similarity inside LLM blocks. Token norms (left), tokens cosine similarity between consecutive blocks (middle) and between perceptual and textual tokens (last). The tokens are inside Vicuna-v1.5 blocks (and outside the residual stream): after the self-attention (SA), and FFNs (FC1/2) and layer norms (LN1/2). Multimodal tokens are Implicit alignment inside LLM blocks.
Implicit alignment during inference, inside LLM blocks (outside the residual stream) (Fig. 8). We look deeper to investigate the source of this alignment, and focus on tokens inside the LLM block, which consists mainly of self-attention (SA), FFN (FC1/2) and layer norms (LN1/2) layers. Interestingly, the similarity between textual and multimodal tokens is the highest after the SA layers.
An implicit multimodal alignment emerges pull the textual and perceptual tokens closer inside LLMs, during training and inference.
Finding 3.

---

4.2 Explanation: the architectural inductive bias hypothesis
Residual stream with refinement blocks. We notice different observations between the tokens inside and outside the residual stream. In the residual stream, the perceptual and textual tokens exhibit significant representation differences (Sec. 3.1), while outside the residual stream, they are more aligned. Each block contributes slightly to the residual stream (small token norms inside the blocks Fig. 8a), with significantly different contributions (cosine similarity between consecutive blocks close zero, e.g, after the FC1/2 Fig. 8b). This allows us to view the model series of refinement blocks that try to gradually refine the input signals. As the original signals are significantly different, they stay different in the residual stream throughout the model. We argue that this provide a flexibility to handle too different imputs. Moreover, previous works [40] have shown that transformers contain both elements with high and low complexity biases, which helps to build general purpose architectures [41] that are able to generalize. These works support further our findings.
Refinement blocks as steering blocks. Inside the transformer block, notice that the layer normalization play an important role in having comparable norms for both textual and perceptual tokens (Fig. 8a). Perceptual token norms become smaller and closer textual ones several layers in the block. In terms of cross-modal similarity, we notice the highest similarity after the SA, then after the FC2 and LNI. Note that this similarity higher inside the block, than in the residual stream (e.g. 0.45 vs 0.1 for Vicuna-v 1.5 and 0.58 vs 0.15 for LLaVA-1.5-44 in the residual stream Fig. 2). After each block the cross-modal alignment increases, and hence the narrow cones are steered to each other. This suggests that all layers play an important role in steering the textual and perceptual narrows cones be aligned, with the most contributions coming from the SA.
traverse
Finding 4. An LLM can be seen as a residual stream with refinement blocks acting as steering blocks. This architecture design plays important role in generalizing to very different tokens, and hence other modalities.
Implications: performance, safety and efficiency
2.30
80
DOS
TherVOA
2.5
DOA Thetiio
2.10
2.30
Pi Trop 156c
2.00
Fa Trop 9000
3,00
Txip
20
Licers
201
Figure 9: Implicit alignment proxy metric for task performance. Left: different checkpoints of LLaVA-1.5-4. Right: different variants of the LLaVA-1.5 model. We show the cross-modal token cosine similarity across layers, and the task performance across different benchmarks.
Implicit alignment as a proxy metric for task performance? (Fig. 9) We compute the cosine similarity between perceptual tokens at the LLM input and the textual tokens across LLM layers. The similarity increases during training. Interestingly, we notice a clear and positive correlation with the task performance on several multimodal benchmarks. In addition, we find that this correlation exists across different models, as shown for different LLaVA-1.5 variants.
Implicit alignment as proxy metric for hallucination? (Fig, 10) Previous works have shown that LMMs suffer from severe hallucinations [42, 43, 44], and generally try to tackle this problem by training on better datasets [45], using RLHF or RLAIF [46, 47] or post-training heuristics [48, 49]. Here we highlight one of the main causes of hallucinations: which the lack of internal alignment between textual and perceptual representations. we show the cosine similarity between textual and perceptual tokens after each L.MM block, and report the hallucinations on POPE [50] and COCO [51] benchmarks. The curves show clear correlation between the implicit alignment and the hallucinations.
8

---

0.
s0p
CIAAZA
FOOD CIMATT
TLLkoet 52
stg 6003
Pvr mop 7503
0.3
0.3
Lagurs
Figure 10: Implicit alignment as a proxy metric for hallucinations. Left: different checkpoints of LLaVA-1.5-4, Right: different variants of the LLaVA-1.5 model. We show the cross-modal token cosine similarity across layers, and the hallucinations across different benchmarks.
3OO C 3O
PTV Sicorthe
Dag 190 tarerthel
Sig FFN Sareenic
Plom
Skipping computations visual tokens. Skipping (Skip ratio)% of tokens the FFN layers. sl: skipping start layer. (V): visual tokens. (T): textual tokens. Results on the MT (with LLaVA-115) setup.
Figure 11:
Skipping computations for visual tokens (Fig, 11). In Sec. 3.1 show that perceptual tokens change significantly less across layers, compared to textual ones. Sec. 4 highlights the importance of SA layers for cross-modal alignment. In this section, we leverage these observations reduce the LLM computation overhead by skipping the computations of visual tokens. Specifically, starting from given start layer (sl), we reduce computations in FFN layers, which accounts for almost 2/3 of model weights, by skipping p% (Skip ratio) of visual tokens. Fig. 11 shows that skipping the visual tokens leads only to slight decrease in performance, while reducing significantly the amount of compute. We provide additional results with the ST setup, and ablation study in App. F.3.
LLM
C1022
A6C1500
MARL (N)
125
45.5
DEVALM 11
131.2)
11.11
-5.55
95.70
1131
132.5
08:33
30-3
66.34
5.0
0.31
U00
Persestual
Figure 12; o-SubNet: modality-agnoste subnetwork. Left: illustration of how obtain the o-SubNet. Right: different methods to compress multimodal LLMS (OPT). Table 2.
a-SubNet: one LLM Subnetwork for all multimodal tasks (Fig. 12) Despite their differences,
multimodal tokens share an important property: slowly changing embeddings across layers (Sec. 3.1). This suggests the possibility of compressing the model while retaining reasonable performance. In addition, textual and multimodal tokens are pulled closer inside the LLM (Sec. 3.2). and processed by almost the same LLM weights (Sec. 4), especially for the ST setup. This suggests the possibility of finding a common subnetwork (a-SubNet) that works well for all multimodal tasks. Thus, we focus on the ST setup with the OPT and CLIP encoders that are currently used by previous works. We consider two representative tasks: COCO image captioning and VQAv2 and provide similar results for other tasks, and modalities in App. F.4. First use Wanda for task task-specific pruning

---

and show in Fig. 12 that we can obtain scores close to the original ones while removing 50% of the weights. To find the task and modality agmostic a-SubNet, we first extract many pruning masks (e.g. at 30% sparsity) for different modalities, then take the intersection of all these masks (e.g. leading to global mask at ~ 50% sparsity). This approach is significantly better than other baselines such as magnitude pruning random mask, and leads to comparable performance compared to the task-specific Wanda pruning, especially for VQAv2.
Discussion
The paper focuses open-source and frozen LLMs up to 7B parameters, LMMs that concatenate perceptual tokens the LLM input and are relatively efficient. The generalization of our findings, to larger and more powerful models, with different architectures, including proprietary ones remains to be seen. Detailed discussion in App. B.
Limitations.
Conclusion. We propose the first study of the internal representation of frozen LLMs when exposed multimodal inputs. We find very different representations for perceptual and textual tokens, yet LLMs are still able to generalize to these non-textual tokens. The implicit multimodal alignment (IMA) effect, linked mostly to architectural design, facilitates this generalization by bringing mul- timodal tokens closer inside the LLM. Our findings have several implications, such as as reducing the computation resources at inference time, understanding better the performance as well as safety- related problems such as hallucinations. We hope that this study will have positive impact, pushing for more works to understand multimodal LLMs, and pave the way to devise more powerful models that are better aligned to human preferences, while targeting safety-related issues.
7 Acknowledgments
The authors would like to thank Arnaud Dapogny and Edouard Yvinec for fruitful discussions, and Damien Teney and Alexandre Ramé for their helpful feedback the paper. This work was partly supported by ANR grant VISA DEEP (ANR-20-CHH) 20-CHIA-0022), and HPC resources IDRIS under the allocation 2024-[ADO1 1013415R1)) made by GENCI.
References
[1] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022. 1, 24
[2] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01005 2022. 24, 26
[3] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. 1, 24
[4] OpenAL. Gpt-4 technical report. arXiv, 2023. 1, 24, 25
[5] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023. 1, 24, 26
[6] Jean- nann-Batiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems (NeurIPS), 23523376--23766 2022. 1, 24, 25, 26
[7] Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Tay, et al. Pali-x: On scaling up a multilingual vision and language model. arXiv preprint arXtv:2305.18565 2023. 1, 24, 25
10

---

[8] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2333.03.: 2023. 1,
[9] Mustafa Shukor, Corentin Dancette, and Matthieu Cord. ep-alm: Efficient perceptual aug- mentation of language models. In Pruceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pages 22056-22069, October 2023. 1, 2, 3, 9, 24, 37
[10] Théophane Vallaeys, Mustafa Shukor, Matthieu Cord, and Jakob Verbeek. Improved baselines for data-efficient perceptual augmentation of Ilms. arXiv preprint arXiv:2403. 2024. I, 2, 3, 9, 24, 26, 37
[11] Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. arXiv preprint arXiv:2209. AXIV:2209.15162, 2022. 1, 2, 24
[12] Oscar Mañas, Pau Rodriguez, Saba Ahmadi, Aida Nematzadeh, Yash Goyal, and Aishwarya Agrawal. Mapl: Parameter-effiieet adaptation of unimodal pre-trained models for vision- language few-shot prompting. arXiv preprint arXiv:2210.07177 2022. 1, 2, 3, 9, 24, 26, 37
[13] Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304. 2023.
[14] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purge purposs vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500 2023. 1, 2, 24
[15] Meghal Dani, Isabel Rio-Torto, Stephan Alaniz, and Zeynep Akata. Devil: Decoding vision features into language. arXiv preprint axXiv:2399.0661,, 2023. I
[16] Ron Mokady, Amir Hertz, and Amit H Bermano. Clipeap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734 2021.
[17] Antoine Yang. Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. In NeurIPS 2022-36th Conference on Neural Information Processing Systems, 2022. 1, 24
[18] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorek. Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208. 10442, 2022. 2
[19] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023. 2, 3, 24, 26
[20] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
[21] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model unified interface for linguaelanguage multi-task learning. arXiv preprint MxXiv:2310.09411 2023. 2
[22] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
[23] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer. Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. International Conference on Learning Representations (ICLR), 2021.
3, 26
11

---

[24] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space- timee-timeee attention all you need for video understanding? International Conference on Machine Learning (ICML), volume page 4, 2021. 3, 26
[25] Yuan Gong, Yu-An Chung, and James Glass. AST: Audio Spectrogram Transformer. In Proc. Interspeech 2021, pages 571-575, 2021. 3, 26
[26] Kawin Ethayarajh. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Kentaro Inui, Jing Jiang. Vincent Ng. and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55-65, Hong Kong, China, November 2019, A Association for Computational Linguistics. 4, 24
[27] Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. Representation degeneration problem in training natural language generation models. In International Conference on Learning Representations, 2018. 4, 24
[28] Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9119-9130, 2020. 4, 24
[29] Sara Rajace and Mohammad Taher Pilehvar. How does fine-tuning affect the geometry of emhedding space: A case study on isotropy. In Findings of the Association for Computaaional Linguistics: EMNLP 202/, pages
2021. 4, 24
[30] Sara Rajace and Mohammad Taher Pilehvar. A cluster-basee based approach for improving isotropy in contextual embedding space. In Proceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 1 1th International Joint Conference on Natural Language Processing ( (Volume 2: Short Papers), pages 575-584, 2021. 4, 24
[31] William Rudman and Carsten Eickhoff. Stable anisotropic regularization. In The Twelfth International Conference on Learning Representations, 2024. 4
[32] Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 15:12212-17623 2022. 4, 24
[33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning
(ICML), pages 8748- 8763. PMLR, 2021. 4, 26
[34] Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint 4rXiv:240..17762. 2024. 5, 24, 30
[35]
nostalgebraist. interpreting gpt: the logit lens. Less Wrong. 2020. 5,
[36] J Alammar. Ecco: An open source library for the explainability of transformer language models. In Heng Ji, Jong C. Park, and Rui Xia, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 249-257 Online, August 2021. Association for Computational Linguisties. 5, 31
[37] Mor Geva, Avi Caciularu, Guy Dar, Paul Roit, Shoval Sadde, Micah Shlain, Bar Tamir, and Yoay Goldberg. Lm-debugger: An interactive tool for inspection and intervention in transformer-based language models. arXiv preprint arXiv:2204. 12130, 2022. 5, 31
[38] Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.088 12, 2023. 5,31
[39] Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 16695, 2023. 5, 31
12

---

[40] Damien Teney, Armand Nicolicioiu, Valentin Hartmann, and Ehsan Abbasnejad. Neural redshift: Random networks are not random functions. arXiv preprint arXiv:2403.02241, 2024.
8, 24
[41] Micah Goldblum, Mare Finzi, Keefer Rowan, and Andrew Gordon Wilson. The no free lunch theorem, kolmogorov complexity, and the role of inductive biases in machine learning. arXiv preprint arXtv:2304.053366 2023. 8
[42] Mustafa Shukor, Alexandre Rame, Corentin Dancette, and Matthieu Cord. Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context- learning. In The Twelfth International Conference on Learning Representations, 2024. 25
[43] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination large vision-languae models. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing. pages 292-305, 2023. 8
[44] Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18135-18143, 2024. 8, 25
[45] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.1444 14565, 2023.
[46] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang, al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309 14525, 2023. 8, 25
[47] Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Align- ing modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402. 1411, 2024. 8, 25
[48] Yiyang Zhou, Chenhang Cui, Jachong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision- language models. arXiv preprint 1txXix:3110007544 2023. 8, 25
[49] Shukang Yin, Chaoyou Fu, Sirui Zhao. Tong Xu. Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint arXiv:2310. 16045, 2023.
[50] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint 105 2023. 8, 25, 26, 36
[51] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035-4045, 2018. 8, 25, 36
[52] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 33:1877-1901, 2020. 24
[53] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:221 1,00 2022. 24
[54] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. OpenFlaming! An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01333 2023. 24, 26
13

---

[55] Hugo Laurençon, Lucile Saulnier, Leo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang. Siddharth Karamcheti, Alexander M Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. OBELICS: An open web-scale filtered dataset of interleaved image-text documents. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. 24, 25, 26
[56] Xi Chen, Xiao Wang. Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et Pal.: A jointly-scaled multilingual language-image model. arXiv preprint urXiv:2209006990 2022. 24
[57] Peng Wang, An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence- -e-eueeeeeee learning framework. In International Conference on Machine Learning, pages 23318-23344 PMLR, 2022. 24
[58] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. In The Eleventh International Conference on Learning Representations, 2022. 24
[59] Shizhe Diao, Wangchunshu Zhou, Xinsong Zhang, and Jiawei Wang. Write and paint: Generative vision-language models are unified modal learners. In The Eleventh International Conference on Learning Representations, 2023, 24
[60] Mustafa Shukor, Corentin Dancette, Alexandre Rame, and Matthieu Cord. UnIVAL: Unified model for image, video, audio and language tasks. Transactions on Machine Learning Research, 2023. 24, 25
[61] David Mizrahi, Roman Bachmann, Oguzhan Fatih Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Zamir. 4m: Massively multimodal masked modeling. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 24
[62] Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang. Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. arXiv preprint arXiv:2312.17172, 2023. 24
[63] Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models images for multimodal generation. arXiv preprint arXiv:2301. 13823, 2023. 24
[64] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021. 24
[65] Mustafa Shukor, Guillaume Couairon, and Matthieu Cord. Efficient rision-language pre- training with visual concepts and hierarchical alignment. In 33rd British Machine Vision Conference (BMVC), 2022. 24
[66] Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang. Lijuan Wang, Chenguang Zhu, Zicheng Liu, Michael Zeng, et al. An empirical study of training end-to-end vision-and- language transformers. arXiv preprint Stti::2111.023881 2021. 24
[67] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. arXiv preprint arXiv:2201. 12086 2022. 24
[68] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision align- ment model. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15638-15650 2022. 24
[69] Zhanyu Wang, Longyue Wang. Zhen Zhao, Minghao Wu, Chenyang Lyu, Huayang Li, Deng Cai, Luping Zhou, Shuming Shi, and Zhaopeng Tu. Gpt4video: A unified multimodal large language model for Instruction- folloowed understanding and safety-aware generation. arXiv preprint arXiv:2311.16511 2023. 24
14

---

[70] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal 1lm. arXiv preprint arXiv:2309.055: 2023, 24
[71] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Jiao Qiao. ALdMe-rdapter: Efficient fine-tuning of language models with zero-init attention. 2303.16199, 2023. 24
[72] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023. 24
[73] Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafig Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-instructblip: A framework for aligning x-modal anstructionn-ware representations Ilms and emergent cross-modal reasoning. arXiv preprint arXiv:2311.18799, 2023. 24
[74] Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu. Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. Mobilevim: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:3312/1688, 2023. 24
[75] Zhengqing Yuan, Zhaoxu Li, and Lichao Sun. Tinygpt-v: Efficient multimodal large language model via small backbones. arXiv preprint arXiv:2312.1663 2023, 24
[76] Yichen Zhu, Minjie Zhu, Ning Liu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. Llava-phi: Efficient multi-modal assistant with small language model. arXiv preprint arXiv:201..02330 2024. 24
[77] Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402. 14289, 2024. 24
[78] Haoran Wei, Lingyu Kong. Jinyue Chen, Liang Zhao, Zheng Ge, En Yu, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Small language model meets with reinforced vision vocabulary. arXiv preprint arXiv:2401.12500 2024. 24
[79] Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, and Jac-Joon Kim. Sleb: Streamlining Iims through redundancy verification and elimination of transformer blocks. arXiv preprint arXiv:2402.090055, 2024. 24
[80] Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403. 2tt::2443..887777 17887, 2024.
24
[81] Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shri- vastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient Iims inference time. In International Conference on Machine Learning, pages
22137-22176. PMLR, 2023. 24
[82] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers scale. Advances in Neural Information Processing Systems, 35:30318-30332, 2022. 24
[83] Ibrahim Alabdulmohsin. Vinh Q Tran, and Mostafa Dehghani. Fractal patterns may unravel the intelligence in next-token prediction. arXiv preprint arXh::2402.01825, 2024. 24
[84] Sarah Schwettmann, Neil Chowdhury, Samuel Klein, David Bau, and Antonio Torralba. Multimodal neurons in pretrained text-only transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2862-2867, 2023. 24
[85] Haowen Pan, Yixin Cao, Xiaozhi Wang, and Xun Yang. Finding and editing multi-modal neurons in pre-trained transformer. arXiv preprint arXiv:2311.074700 2023. 24
[86] Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lijuan Wang. Yezhou Yang, and Zicheng Liu. Compressing visual-linguistic model via knowledge distillation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1438, 2021. 24
15

---

[87] Kan Wu, Houwen Peng, Zhenghong Zhou, Bin Xiao, Mengchen Liu, Lu Yuan, Hong Xuan, Michael Valenzuela, Xi Stephen Chen, Xinggang Wang, et al. Tinyelip: Clip distillation via affinity mimicking and weight inheritance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21970-21980, 2023. 24
[88] Zhe Gan, Liu, Lijuan Wang, and Zicheng Liu. Playing lottery tickets with vision and language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 652-660, 2022. 24
Chen, Linjie Li, Tianlong Chen, Yu Cheng, Shuohang Wang, Jingjing
[89] Jia Huei Tan, Chee Seng Chan, and Joon Huang Chuah. End-to-end supermask pruning: Learning to prune image captioning models. Patlern Recognition, 122:108366, 2022. 24
[90] Dachuan Shi, Chaofan Tao, Ying Jin, Zhendong Yang, Chun Yuan, and Jiaqi Wang. UPop: Unified and progressive pruning for compressing languge transformers. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Leurning Research, pages 31292-31311. PMLR, 23-29 Jul 2023. 24
[91] Ali Furkan Biten, Lluis Gomez, and Dimosthenis Karatzas. Let there a clock on the beach: Reducing object hallucination in image captioning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1381-1390, 2022. 25
[92] Junyang Wang, Yiyang Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao, Haiyang Xu, Qinghao Ye, Ming Yan, Ji Zhang, Jihua Zhu, et al. Evaluation and analysis of hallucination in large vision-language models. arXiv preprint arXiv:2308. 15/26, 2023. 25
[93] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Li- dong Bing. Mitigating object hallucinations in large iision-language models through visual contrastive decoding. arXiv preprint arXiv:2311.16%% 2023. 25
[94] Wenliang Dai, Zihan Liu, Ziwei Ji, Dan Su, and Pascale Fung. Plausible may not be faithful: Probing object hallucination in vision-language pre-training. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2136-2148, May 2023. 25
[95] Bohan Zhai, Shijia Yang. Xiangchen Zhao, Chenfeng Xu, Sheng Shen, Dongdi Zhao, Kurt Keutzer, Manling Li, Tan Yan, and Xiangjun Fan. Halle-switch: Rethinking and controlling object existence hallucinations large vision language models for detailed caption. arXiv preprint 1rXiv:2310.01779, 2023. 25
[96] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306. 14565, 2023. 25
[97] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Haan-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11800 2023. 25
[98] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12997 2023. 26
Bootstrapping language-
[99] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, al. Judging ilm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024. 26
[100] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning (ICMI.), pages 10347-10357. PMIR, 2021.
26
16

---

[101] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16000-16009, 2022. 26
[102] Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. X-clip: End- to-end multi gualti contrastive learning for video-text retrieval. In Proceedings of the 30th ACM International Conference Multimedia, pages 338-647, 2022. 26
[103] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35: 35:10078-10093, 2022. 26
[104] Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojeiech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. In NeurlPS, 2022. 26
[105] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 6904-6913, 2017. 26
[106] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7700,6709, 2719,2019,.. 26
[107] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 26
[108] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In European Conference on Computer Vision (ECCV), pages 740-755. Springer, 2014. 26
[109] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM International Conference Multimedia, MM '1, page 1645-1653, New York, NY, USA, 2017. Association for Computing Machinery. 26
[110] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2016. 26
[111] Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Gener- ating captions for audios in the wild. In NAACL-HLT, 2019. 26
[112] Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 736-740. IEEE, 2020. 26
[113] Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. Clotho-aqa: A crowdsourced dataset for audio question answering. In 2022 30th European Signal Processing Conference (EUSIPCO), pages 1 1140-1144. IEEE, 2022. 26
[114] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal 1lms with generative comprehension. arXiv preprint 1611: 2023. 26
[115] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEFE//CV conference computer vision and pattern recognition, pages 8317-8326 2019.
26
17

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
For findings Sec. 3 and implications Sec. 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Sec. and more detailed in App.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Details to reproduce the experiments are in Sec. and more detailed in App. D. Also the code will be made public.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification:
The paper uses solely public datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//niss wtwpp.cccpublll./. htcp//blc/gguuds/////eececec //t////ooammseeeeeeeeeea sssisssosicccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The experimental details can be found in Sec. and App. D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper includes different statistical measures report scores such the average, median and the variance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources Question:
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
The paper mostly about analysis, but the considered models are trained by authors with details included in App. D. In addition we report FLOPSs during inference in the implications section Sec. 5.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https //te:r//s.....eeeeeeee httpll//tttissuuuuiue
Answer: [Yes]
Justification:
No violation for the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
App. C
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper cites all used assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23

---

Supplementary material
This supplementary material is organized as follows:
App. A: detailed related work.
App. B: detailed discussion about the work, limitations and other implications.
App. C: the broader impact of the work.
App. D: implementation details, including the trained models, datasets and metries.
App. E: additional experiments analysing LLMs.
App. additional experiments for the implications.
A Detailed related work
Large multimodal models. Motivated by the success of large-scale training of LLMs [52, 1, 3, 53, 2, 5, 4]. the multimodal community has embarked a parallel journey, striving to develop larger and more powerful models capable of processing multiple modalities. Typical Large Multimodal Models (LMMs) constructed either by building upon frozen LLMS [6, 54, 55] by training them end-to-end after initialization [56, 7, 8]. These models have demonstrated success in numerous general and intricate multimodal tasks, achieving performance levels close to human capability, Another important line of research focuses on unified models, where a single model is designed to handle diverse tokenized modalities, such as image-text [57, 58, 59], even beyond two modalities [60, 61, 62].
Efficient large multimodal models. Recently, to mitigate the training cost associated with training large multimodal models, efficient adaptation of unimodal models has emerged as a promising direction. Models like [11, 9, 17, 12, 10, 63] maintain LLMs frozen and train only small subset of adaptation parameters for different multidmodal tasks. These approaches achieve competitive performance compared to end-to-end trained models [64, 65, 66, 67, 68] on image-text tasks and also on audio and video-text tasks [9, 69, 70, 10]. Beyond single-tast taakk tuning, many approaches do relatively light-weight pretraining and/or instruction tuning [71, 72, 14, 73, 19] and achieve good zero- shot generalizaton and instruction following abilities. To make these models more efficient, previous works have trained models with smaller LLMs showing competitive performance [74, 75, 76, 77, 78]
Analyzing LLMs. Previous research has highlighted the highly anisotropic nature of embeddings within language models, characterized by high cosine similarity [26, 27, 28, 29, 30]. Studies focusing on efficiency have shown that textual tokens exhibit small changes across layers [79, 80, 81]. Additionally, LLMs contains outlier features [82] and massive activations [34], which significantly influence model performance. Work by [83] suggests that LLMs may generalize due to the fractal structure of language. Moreover. [40] demonstrate that the building blocks of LLMs implicitly bias towards approximating both complex and simple functions. In the multimodal domain, [32] identify a modality gap in CLIP models attributable to the narrow cone effect. Previous studies have also explored neurons in LLMs that encode multimodal representations [84, 85].
Compression and pruning for multimodal models. Few works have targeted multimodal model compression, focusing mostly on image-text models. Notably, some works have concentrated on distilling knowledge from larger models through attention maps [86] or the affinity matrix in CLIP models [87]. Recent efforts have successfully applied the Lottery Ticket Hypothesis (LTH) to these models, optimizing both the model weights and masks jointly [88, 89]. A unified framework for structured pruning based on iterative training and adaptive sparsity allocation was proposed by [90]. It's worth noting that these approaches, initially designed for relatively small image-text models, encounter scalability challenges when applied to very large models.
24

---

Hallucinations in multimodal models. Hallucinations in multimodal models involve generating text that refers to objects not present in the input image [51, 50]. This pervasive issue affects wide range of multimodal models, varying in architecture, training data, and scale [91, 60, 42, 6]. To gain deeper insights into this phenomenon, numerous studies have proposed evaluation benchmarks to quantify hallucinations across different dimensions [51, 50, 92, 44], shedding light on underlying causes. Specifically, co-occurrences and uncertainty [48], as well as visual uncertainty stemming from lower image resolution [93, 94, 95], have been identified as contributing factors. Additionally, has been demonstrated that multimodal in-context learning exacerbates hallucinations [42]. To address this issue, various techniques have been proposed, including training improved datasets [96, 44], aligning models with reinforcement learning [46, 47], refining training objectives [94], and employing post-training heuristics [48, 93]. Our study highlights the misalignment between internal representations of textual and perceptual tokens as a key cause of hallucinations.
B
Discussion
Study across LLMs and setups. Our investigation primarily centers on Vicuna-v1 1.5 across single- task and multitask setups. We find that our conclusions remain consistent across various LLMs (e.g.. OPT, Llama 2) and different settings (e.g., with and without pretraining, using different mapping modules), as detailed in the appendix. Extending our analysis to encompass other multimodal models, potentially with diverse architectures [6, 55], could offer additional valuable insights.
Study on larger models. While our work primarily focuses frozen LLMs provide insights relevant to future multimodal models, we also present results involving trained LLMs such as in LL..A- These experiments yield observations akin those with frozen LLM variants. However, the applicability of our experiments or the generalizability of our findings to larger LLMs (beyond 7B parameters), larger multimodal models [7, 6], massively-trainee multimodal foundation models like Gemini [97] or GPT4-V [4] remains an open question.
Remaining questions to understand LMMs. While we primarily investigate why and how LLMs generalize to multimodal inputs, and offer insights into issues such as object hallucinations, numerous unanswered questions persist. For instance, further exploration is needed to discern the encoded information in tokens and how LLMs extract information from visual tokens. Deeper inquiries are also required to address salety-related issues in large models, including the inability to abstain from answering, compositionality, and the precise adherence to user instructions [42].
Other implications. Our paper discusses several practical implications with potential benefits. Future extensions of our study could focus on specific aspects, such as enhancing model efficiency during training and inference by reducing redundant computations or model size. Additionally, addressing alignment with human preferences, such as faithfulness and safety, remains significant challenge requiring further investigation. Our study may also inform model architecture design, such as developing mapping modules explicitly aligning multimodal tokens before entering the LLM.
C Broader impacts
The paper aims to enhance our comprehension of LLMs within the realm of multimodal inputs. We contend that a deeper understanding of these models can yield positive societal impacts, which we partially address in this study. For instance, our findings may contribute to mitigating the consumption of large models and their potential societal harms. Moreover, our work may inspire future research endeavors with various impacts, none of which we think must be specifically discussed here.
D Implementation details.
D.1 Perceptually augmented LLM baselines
D.1.1 ST setup
We train many models across different datasets that span image, video and audio-text modalities. We first devise powerful baselines based 3 tenets: (a) having the smallest number of trainable
25

---

parameters, (b) general architecture that span or similar to many existing models and (3) good performance. To this end, and inspired by previous studies showing the effectiveness of using transformer-based mapping module [10, 12, 6, 54, 55, 98], we use light-weight transformer with learnable queries and self-attention to attend to perceptual tokens. This transformer operates in low dimension space (i.e., due to down/up projection layers and the the number of learnable query are limited to 10. We also favor a deeper architecture (5 blocks) compared to a wider one [10]. Our baselines are close to [10], but with significantly less trainable parameters. We train these baselines with different LLMs: OPT-6.7B [2], Llama 2-7B [5] and Vicuna-v1. [99] and different encoders for: image (ViT [23, 100], CLIP[33], MAE[1O1]] D) video (TimesFormer(244 X- X-CLIP[102) VideoMAE[103] and audio (AST[25), AudioMAE[1044
To train these baselines, we use AdamW optimizer with a learning rate of 2e-4 that decreases with a cosine annealing scheduler to minimium of le-5. We train with a total batch size of 16 for captioning and 64 for VQA datasets. The number of epochs is set to 20 to ensure that all models converged, though most of these models converge after only couple of epochs. We select the best checkpoint for evaluation. For example the model for image captioning converged after 4 epochs. All models are trained on V100 GPUs and the training time depends on the task, e,g., for the large VQAv2 dataset each epoch takes ~ 30 mins, for other smaller datasets it takes less time, e.g., 10 mins for Audiocaps and MSVD-QA. DAQ. Unless specified otherwise, we fix the hyperparameters for all baselines to isolate the variations that could results from this.
We refer to all text-aligned models as CLIP, trained for classification as ViT, and self-supervised with MAE objective as MAE.
D.1.2 MT setup
To study the impact of different factors (e.g. pretraining, mapping module) on the internal repre- sentations (e.g. implicit alignment), we devise different variants of the LLaVA-1.5 [19] model that differ from the original model as follows: LLaVA-1.544 (LLM kept frozen), LLaVA-1.553 (LLM kept frozen, without pretraining) and LLaVA-1.5-4 (LLM kept frozen, without pretraining and with transformer mapping module similar. The latter is very similar to the models used in the ST setup, which ensure comparable observatoins. All these models are based on the Vicuna-v1.5-77 LLM.
We follow the same training setup of LLaVA-1.5 [19], including the training data, steps and hyperpa- rameters.
For analysis in the paper (e.g. Sec. 3), we focus on Vicuna-v1.5 as it is shared by both setups. For the ST setup, we use unimodal encoders, such as VIT, Timeerormer and AST that are not aligned with text.
D.2 Datasets and metrics
ST setup. We consider wide range of public multimodal datasets that cover representative tasks:
captioning and question-answering (QA) across image (VQAv2 [105], GQA [106), OKVQA [107],
COCO caption [108)), video (MSVD, MSRVTQA [109], MSRVTT [1 [110), audio (Audiocaps [111], Clotho [1 12], Clotho-AQA [113]) and language tasks. For QA datasets we report the accuracy (in
open-ended generation setup with exact match), and for captioning we report the CIDEr metric.
MT Setup. We also evaluate the MT setup on recent datasets such as SEED [114], TextVQA [115]
and POPE [50].
E LLMs generalize to multimodal inputs: additional experiments
E.1 Tokens evolution across layers
Different VL-1.5 variants. In Fig. 13, we illustrate the differences between perceptual and textual tokens across different LLaVA-1.5 variants. Comparing the two variants with multimodal pretraining, we observe higher cross-modal similarity in LLa LL- compared to LLaVA-1.552, which freezes the L.M. This suggests that training the LLM enhances alignment between representations. For models without pretraining, we note that using an MI.P (1.55VA-1.53)) to connect both models yields better results than using transformer-based pooling (AL.55-))5-4) potentially explaining the
26

---

Lewn
Ligerr
FIXD
Lewn
Par
PM
70a
Figure 13: Textual and multimodal tokens for LLaVA-1.5 variants (MT setup). From top to bottom: (1) the cosine similarity hetween the textual and multimodal tokens across LLM blocks. (2) the cosine similarity between consecutive blocks. (3) token norms, (4) KL-distance between vocabulary distributions decoded from textual and perceptual tokens, (6) cosine similarity between vocabulary distribution at consecutive layers. From left to right: LLaVA-1.5, LLVVA-1.522 LLaVA-1.5-3 LLaVA-1.5-4.
superior scores. Utilizing all visual tokens appears to bolster alignment with textual tokens, with pretraining further enhancing this alignment. Notably, vocabulary distributions undergo significant changes in middle layers, particularly for textual tokens. Similar observations hold across different variants, indicating that our findings generalize to broader setups and that training the LLM does not substantially alter token behavior.
Different similarity measures cross-modal alignment. this section, we compare the following similarity measures to compute the similarity between perceptual (P [p...... PN,,)) and
(PN,...,N,,)
27

---

Figure 14: Different similarity measures. From left to rights: SimAvg, MinSim, AvgSim, MedSim, MaxSim. ST setup (top) and MT setup (bottom).
Pos PvsiT Ww
- Vis
0.0
Chante Simiustt
Cootte Simmitsity
VisT
0.6
P
0.6
TvS
Avs
0.4
VI A-
ET vs A-T
0.0
15
25
30
0.0
30
15
20
30
Layers
Layers
0
Coogre Sintarrty
Coeatre Stnberrty
Lagers
Labes
Lipecs
Figure 15: Different similarity measures and the narrow cone effect. From left to right: SimAvg, MedSim and MaxSim. Vicuna-v1.5 (top), LLaVA-1 1.-44 (bottom).
textual (T [t1,..., (N.) tokens:
X
Sim(X,Y) X
(4)
Σ
Σ
li
SimAvg(P,T) = Sim P, 7)
(5)
N
N₂
MaxSim(P,T)
Sim(pi, tz), max sE(NobeENN.
(6)
min MinSim (P,T) iE(NNNEENN
Sim(pi, tz);
AvgSim(P,7T) Eie (N Sim(p:, tz)
(8)
Np + N₂
MedSim P, Dineie((,
Where [Ny] = [1,..., Np) and [Ne] [1, N2} and Med is the median operation.
Fig. shows the inter (P vs T) and intra (P vs P) similarity. According to measures, except AvgSim and MinSim, we have similar observations: increasing inter similarity and higher intra similarity that increases in last layers. For MinSim and AvgSim for the ST setup, we not see such observations, indicating that not all perceptual tokens are, should be aligned to text.
28

---

Fig. 15 and Fig. 16 show the measures comparison for the narrow cone experiments, Interestingly, the narrow cone effect less seen when looking the median of the token similarities (MedSim), indicating that this effect is not driven by all tokens, and at the token level the representation is not always anisotropic.
In spite of having similar observations between several measures, we focus on SimAvg, as it is much faster compute, especially whey there is large number of tokens (as in LLaVA-1.5).
Vvoc V
Vosi Ven
A
Tos Vos
0.8
Sintlants
V Twl A
Contre Smilanh
AveT
Tw
VT 177
0.5
Vee
Tw
Aw
Oosine
Costin
0.4
VT vs A-
LT vs A.T
0.0
10
15
20
30
Lapers
Lavess
Layers
Figure 16: Narrow cones for image, video, audio and text modalities. From left right: SimAvg. MedSim and MaxSim. Vicuna-v1.5 (top), LLaVA-I VA-1.5-4 1,544 (bottom).
Figure 17: Different similarity measures between tokens at consecutive layers. From left to right: SimAvg, AvgDiagSim, MedDiagSim and MedSim. Vicuna-v1.5 (top), LLaVA-1.544 (bottom).
Different similarity measures for the similarity across consecutive layers. we compare the following similarity measures compute the token similarity between consecutive blocks (e.g. between tokens at block a X = If, xf,,.,, Rx and b x = 2x.....
Xa. X
Sim(X", )) | ||
(10)
SimAvg(X*, x*) -iimmX,, x ΣN F2
(11)
N
AvgDiagSim(X*, x²) EieiN] Sim pp.pp)
(12)
MedDiagSim(X* (X", x = Medigtyy ,!), MedSim(X",,X =
Sim(p? ,p)).
(13)
Where [N] (1.....N) N) and Med is the median operation.
Fig. 17 shows similar observations across different measures when each token is compared with the token at the same position in different layers. However, when taking the median of similarities (MedSim) across all tokens, this similarity is siginficantly smaller, especially for the ST setup. This reveals that tokens can be very different within the same modality example.
29

---

1.00
Figure 18: Different token norm measures. We compute the token L2 norm consecutive blocks (e.g. B'+n and ) for the ST (top) and MT (hottom) sctups. From left to right: NormAvg, MinNorm, MedianNorm and MaxNorm.
Tokens evolution for different modalities. Fig. 19, shows that textual and multimodal tokens evolve differently inside LLMs.
50mm
Figure 19: Textual and multimodal tokens evolve differently inside LLMs. We compute the tokenwise cosine similarity and the median token 1.2 norm at consecutive blocks (e.g. X and X¹) for the ST (left) and MT (right) setups.
E.2 Token norms across layers
Massive token norms. In this section we highlight the presence of tokens with massive norms, this becomes clearer when looking at different norm measures. We compare the following measures to compute the token L2 norms across blocks (e.g. X = 2 21,
M
Norm(X) =
Σ
X?
(14)
NormAvg(X) = Norm
X
(15)
N
MinNorm(Xx min Norm(r;), NE
(16)
MedianNorm(XX (X) = (Nedee())Nom(())
(17)
MaxNorm max Norm(z;),
(18)
(E(N]
Where [N] [1,..., N) and Med is the median operation and M the total number of elements in the tensor. For the ST setup, Fig. 18 shows very high token norms when looking at NormAvg and MaxNorm, compared MinNorm and MedianNorm. These massive norms are present for both textual and perceptual tokens, and they are larger for perceptual ones. When looking closely, we find that these tokens correspond to start or split tokens as seen [34]. For the MT setup, we notice that these massive tokens presents mainly in the system message, which we remove for our study it is common for all examples. Interestingly the perceptual tokens for the MT setup do not seem to have massive norms.
30

---

20
27 trap
1102
Figure 20: L2 token norm increases with training. We compute the token L2 norm during training and across the LLM blocks for the ST setup (Vieuna-vi.5) From left right: NormAvg. MinNorm, MedianNorm and MaxNorm.
Increasing token norm during training. We try to investigate why have very high perceptual token norms. To this end, we compute the norm across different epochs. Fig. 20 shows that during training of the mapping module, the norms increase significantly.
E.3
Vocabulary distribution
For each token, use the LLM unembedding (i.e. LLM head) to decode the latent representation to probability distribution over the vocabulary. This approach have shown to work well for LLMs different layers, not just the last one [35, 36, 37, 38]. In Fig. 21, we show the histogram this distribution at the first LLM layer for both textual and perceptual tokens, the KL-distance between the distributions, Kl.-distance between consecutive layers and the entropy. Here we report additional results for the LLaVA-1. baseline showing similar observations to those of ST reported in the main paper.
Via
VQ D
1m
A9
V9.3
1.
25
10.00
WCC Day
30
Figure 21: Textual and visual tokens have different vocabulary distributions inside LLMs. We the LLM unembedding layer map each token probability distribution over the vocabulary. We then show (from left to right): the histograms at the input of the LLM, the KL divergence between the distributions consecutive layers, the KL divergence between textual and perceptual distributions and the distribution entropy. Top: Vicuna-v1.5, Bottom: LLLVA-115-4.
E.4 Similar activated weights by different modalities
Experimental setup. In this section, we analyse the subnetworks activated by different multimodal inputs. We use the Wanda score [39] to extracted these subnetworks or pruning masks, then compute the loU. For multimodal datasets we consider only the perceptual tokens, for example the visual tokens without the questions for VQAv2. We also use the text in these datasets as source for textual tokens (e.g., COCO-text consider only the captions in the COCO datasct).
Different VA-155 variants. In Fig. 23, we show the overlap between the weights activated by different modalities. Across different LLaVA-1.5 1.5 variants, we find similar observations: high overlap between perceptual textual activated weights (e.g. IoU), which less than the overlap between weights acrtivated by the same modality (e.g. 0.95 for perceptual tokens and 0.87 for textual
31

---

0:11
1.11
6:11
6.91
231
0000 Tm
6:4
1.00
1:93
649
9.99
655
658
0.0
1.00
6:1
3.2
6.71
612
1C.000
3
1.5
3
5
10
033
A00000
030 Tm
NETIT-
Figure 22: ToUs of multimodal subnetworks. ToU of the subnetworks activated by different tasks and modalities, for the ST (left) and MT (right) setups. Different modalities activate similar LLM weights.
LLIVA-15
5 LLM)
LLaVA-1.5 (G. LLM, w/o PT)
LLaVA-155 de Fommer)
LLM. wio PT, Q
LLaVA-15
ELaVA-15
Fanner.)
Ailipw (pmittate
oper buan Ppanco
lager Open Squanap
Figure 23: loUs of activated subnetworks LLaVA-1.5 variants. compute the loU of weights activated by different multimodal tokens. From left to right: LLaVA-1.5, LLaVA-15522 LLaVA- 1.5-3, LLaVA-1.5-4.
ones). We also notice significant decrease IoU the first layers, which might reveal that the first layers encode general features that are shared across modalities.
Different LLMs for the ST setup. In Fig. 24, we show the IoUs of activated weights for different frozen LLMs (i.e., OPT, Llama 2 and Vicuna-v1.5 for the ST setup. We notice similar observations across LLMs, and relatively higher overlap for OPT. We notice similar observation compared to the MT setup, where we have a significant decrease in the IoU in first layers. For this setup, it is clearer that the overlap increase for deeper layers.
Different sparsity levels. In Fig. 25, we study how the overlap between activated weights changes with the size of the extracted subnetworks. This size depends on the sparsity of the final model. We
32

---

OPT (6x0.3)
Liama
Vieuner 1.5 (s=0
X100000
CaaAn X
liger Spe Spandtyy
Jape Oppe Siuaeity 4.51
keym typu 1Speratty
Tosl Tey
Traxstemnr
Tranatorier
Figure 24: ToUs of activated subnetworks for different LLMs. We compute the IoUs for weights activated by differrnt multimodal tokens. From left to right for the ST setup: OPT, Llama 2, Vicuna-v1.5.
laycr types (Sparsity:
Layer types (Sparsity
1
Tvv
A
0.100
100
T
Transtormer Laper
Transfertne Layse
All layer typea (Spursity: 0.3)
All Layer types (Sparsit:
Tvl
0.50
Masko 160
6:30
380
Layer
Transfortire
Figure 25: Overlap between multimodal subnetworks at different sparsity levels.. We compute the IoU of activated weights across layers at and sparsity levels. OPT (top), Llama (bottom).
notice that the lower the sparsity, the higher the overlap, revealing that higher sparsity allows to extract more modality-speciffe activated weights.
Pruning weights by streaming different modalities. In addition to the IoU, we also compare differences between the task performance activated weights. Fro each dataset, we give to the model either the perceptual prompt, the textual prompt both, knowing that each example in the dataset consists perceptual prompt followed by the textual one. Fig. 26 shows slight differences in overall performance when the LLM is pruned by different modalities, with the best performance is
33

---

Image Caption
129
Video Coption (CLIP)
Caption (VI)
Task performance
300
Video Captton (VT))
Audio Caption (V[))
Captica (MAE)
80
Video Caplion (MAB)
Audio Caption (MAE)
0 TP4T TP4T P. TPAT
TPAT P T94T TP4T P. TP4T TP4T
Input for pruning
Figure 26: Similar LLMs weights are activated by different modalities. We report the task performance when keeping only the subnetwork activated by: multimodal prompt (P), text (T). both (P+T) for OPT. Sparsity level: 0.3.
by considering both the textual and the perceptual tokens. This also show the high overlap between the weights used to activate textual and perceptual tokens.
Transfer of pruning masks to other tasks and modalities. To further highlight the overlap between weights, we report the performance when the model is pruned given data from other modalities datasets. We notice similar observations across LLMs, such Llama 2 Fig. 29 and Vicuna-v1.5 1.5 Fig. 30. Interestingly, we find the similar overlap with the unsupervised MAE encoders Fig. 27 compared text aligned ones Fig. 28. We notice a performance degradation when the model is pruned at high sparsity levels (0.5).
Spanity
1:95:3
51.05
19:17
23.03
34.53
28.37
63 65
130.37
12.49
I14
0000 Gn 10a
59.51
52.22
27.56
63.19
130.74
$4.04
66.36
35.00 80
34.65
61 70
28.51
50:30
1:1-42
20.00
$2.51
20.51
62.34
118.10
45.85
111.400
09.36
$2.00
27.44
3M/M
137.75
4b.22
00000
19662
03/W
0000
Medatin
Imace (M4V. Spantty
Mouna E12 100e IMAX, Spondde 0.31
Modultat ands (MAE
DMAZ Spannipy 0.5)
115:37
42.52
30.20
31.55
54.42
27.43
45.97
111.80
110.35
113.68
31.70
32.52
52.53
20.76
40.01
Figure 27: Transfer of multimodal subnetworks across fasks and modalities with OPT and MAE encoders. We use the subnetwork activated by given task/modality to other tasks/modalities and report the task performance. From the captioning task. In each figure, the row corresponds to the source dataset of the subnetwork and the column to the target dataset.
right, transfer across: image lasks, video tasks, audio tasks across modalities for
Modality sppciffc subnetworks? The experiments suggest high overlap between weights ac- tivated by different modalities. For instance, the pruning masks similarity (IoU) between datasets within the same modality is on par with those across modalities for the ST setup. However, this does not exclude the possibility of finding weights that are generally activated when seeing particular modality, even if there are small amount of them. The overlap is smaller with LLaVA-1.5 variants making this possibility more likely for large scale multitask models.
E,5 Implicit multimodal aligument effect
Alignment inside each LLM Block. Fig. reports the tokens similarity and norms for both LLa and Vicuna-v1.5.
34

---

Modality Image ICLPP Spansit: 0.33
Modialiy: video (CLIP Spansity: 0.31
ICLIP Sparsity: 0.31
131.40
63.93
52.99
31.71
120
M000 MORRTTT 3ESITTT
37.26
38.30
39.93
Teet WSST D0cMudtion
125.00
54.36
130
VONE
12.3
116
1300700
84.87
53.06
30.31
58.0
131.58
56.13
38.00
47.3
129.56
64.33
53.73
31.46
129.94
56.66
131.06
64.45
53.55
31.42
57.10
38.10
39.44
40.3
129.35
54.79
VOkvz
Modality: video (CLIP, Sparsity: 0.5)
"het NSSNIT 0300 Aooos.ndocap
Sparsity:
voari coco
128,91
60.97
38.35
23.32
120
MAND NOSIVITOO MOSRVT
54.23
33.80
35.68
118.01
44.73
127.25
61.34
23.48
100
127.88
100
50.67
33.80
37.26
E
127.37
61.26
42.00
24.43
126.15 126.15
32.71
60
127.27
59.35
42.63
24.72
38.17
112.31
39.18
0000
VOAv2
ORVDA
NSBRITOA
00CO
SIBWETT
Figure 28: Transfer of multimodal subnetworks across tasks and modalities with OPT and CLIP encoders. We use subnetwork activated by given task/modality to other tasks/modalities report task performance. From left to right, transfer across: image tasks, video tasks, audio tasks and across modalities for the captioning task. In each figure, the row corresponds to the source dataset of the subnetwork and the column to the target dataset.
Spacoterd
Spanndp
119.96
$5.20
42:85
53.47
18.31
35,90
36
65.00
32,04
49 72
65.98
07.27
114
115.66
$8.15
42.93
96.65
$5.60
115.34
48.94
36,00
38.73
89.877
40.72
11
$7.59
14:25
36.36
67:40
117.29
47.70
1L6.45
07.46
14.34
37.22
02.53
67.01
115.43
$7.09
00000
0000
Medaaltty, vilre (01) Spanniry
Monda Ino vadi
Spunnite 9.51
0800 Ool Toei oot
100.12
0.70
59.06
105.40 44
61.23
108.0
40.07
6,36
42.47
20.33
221-22
30
49.57
3.48
807-92
6.79
42,78
20.40
20.27
56.56
25.7L
42.21
Figure 29: Transfer of multimodal subnetworks across tasks and modalities with Llama 2 and VIT encoders. use subnetwork activated by a given lask/modality to other tasks/modalities performance. From left to right, transfer across: image tasks, video tasks, audio tasks and across modalities for the captioning task. In each figure, the row corresponds to the source dataset of the subnetwork and the column the target dataset.
report task
F Implications on performance, safety and efficiency: additional experiments
F.1 Implicit multimodal alignment as proxy metric for task performance?
IMA score across epochs. Fig. 33 shows an increasing similarity between textual and perceptual tokens during training with OPT and Vicuna-v1.5.
Table IMA score across different encoders. We report the IMA score and the task performance with the ST setup (OPT). A positive correlation exists between IMA score and the performance; the most aligned encdoers (CLIP) have the best accuracy/CIDEr on VQA and captioning tasks.
COCO f
VQAv2
GQA f
LLM
Encoder
IMA Score
CIDEr (test)
Acc (Val)
(Val)
Vicuna-v1.5 Vicuna-v1.5
CLIP-ViT-L
0.130
127.63
63.05
54.34
VIT-L (ImageNet)
0.105
116.76
61.27
51.57
Vicuna-v1.5
MAE4
0.060
76.40
59.57
52.88
IMA score across different encoders. Table
we compare different image encoders and report the the IMA score and the task performance several VQA and image captioning tasks. Encoders
35

---

Image ITIC: Spanate: 0:31
Spenstyy
ratoe Go Yila 0l00
1:7
11
37.20
37.14
603,00
24,50
46 44
65
115.93
17.62
116-43
59.00
30:47
136.90
40.30
37 36
58
46
50.03
44.09
30.92
65.79
136,30
40.16
26
1:8.13
39.89
43.93
50.0M
36,85
64.00
24,81
30.05
M5.0M
0000
60x
02096
HSMV7704
K5Y0
Aecen00
6006
Midaitoo
(VIT.
0.51
Moels Vike Spensly
Sparite 1.31
100 26
66.32
17.72
16,42
56 03
100.05
11:03
107.13
51.79
17:43
18.56
30:67
02
42.28
40
42,03
18.00
23.30
$7.00
18.05
35.60
107
00.79
20.43
18.04
60
100.77 10..76
43.25
106.00
00.67
19.69
17.50
41.43
13.70
19:72
50.90
19,76
41
10
70
36.00
KOLAVE
Figure 30: Transfer of multimodal submetworks across fasks and modalities with Vicuma-v1.5 and ViT encoders. We use the subnetwork activated by a given task/modality to other tasks/modalities and report task performance. From left to right, transfer across: image tasks, video tasks, audio tasks and across modalities for the captioning task. In each figure, the row corresponds to the source dataset of the subnetwork and the column to the target dataset.
kew
Figure 31: High similarity between LLM weights activated by different modalities. We compute the loU of the subnetworks activated by different tasks across modalities, for the ST (left) and MT (right) setups.
that are most aligned to textual tokens inside LLMs (highest IMA egg. CLIP) have also the best task performance.
F.2 Implicit multimodal alignment as proxy metric for hallucination?
We provide more details reagarding the hallucinations metrics. These metrics are supposed to measure multimodal or object hallucination.
CHAIR on COCO captioning [51]. On the COCO image captioning dataset, the model is asked to describe the images. We compute the CHAIR metrics based on the generated captions and the ground truth annotations of all objects in the image. If a caption contains non-existent objects, we classify it as hallucinated caption. The CHAIRs score is the ratio hallucinated captions the total number of captions. Additionally, we calculate the ratio of hallucinated objects to the total number of objects across all captions, which is referred to as CHAIRi. A CHAIR score of indicates no hallucinations. In the paper, we report (1 CHAIR) 100, thus higher score indicates fewer hallucinations.
POPE benchmark [50]. This is question-answerigg task involving questions about the existence of objects in images, The metric used is accuracy; the fewer the hallucinations, the higher the accuracy.
F.3 Skipping computations for visual tokens.
In this section, we propose to skip computations for the visual tokens.
Skip FFN Tokens. We randomly skip a number of tokens (we refer to this amount skip ratio), both the textual and the remaining visual tokens are processed the FFN layers. ??, shows linear relationship between the skip ratio and task performance, the higher the ratio the lower the scores.
36

---

170
Te
(b) Consecuiie blocks similarity
(c) Similaity inside each block.
Figure 32: Implicit alignment inside the LLM blocks. We compute the token norms (left), tokens cosine similarity between consecutive blocks (middle) and across modalities (last). The tokens inside the LLM blocks (and outside the residual stream): after the self-attention (SA), and FFNs (FC1/2) and layer norms (LN). From top down: Vicuna-v1.5, LLaVA-115--
ep
vs T op
vs ee
0.130
Pis T op
eep
0.123
Ps T op
Cosine Similarity
vS Te
Cosine Similarity
Pvs 3
0.100
0,073
0.002
0.23
000
-0003
-0.1
10
25
Layers
Layers
Figure 33: Implicit alignment score epochs. We report the implicit alignment score for OPT (left) and Vicuna-v1.5 right) during training of the mapping module.
F4
a-SubNet
In Table 2, we evaluate our a-SubNett on additional multimodal tasks. Compared to other task- agnostic baselines such as magnitude pruning the scores a-SubNet are significantly higher. This support more that the multimodal tokens activate similar weights inside LLMs.
Table a-SubNet task and modality-agnostic subnetwork. We prune the LLMs using different post-training pruning methods, including our a-Subeet.
Mehsl
Sportyyppssky
Avg
COCO
VQN221
OKVOAT
GQA
MER-VIT
XSEVTE-OA
MSVD-QAT 1
Andsange
Chidao
Cloda-AQA
MAPL(12] (12)
7BCLANANLO
125.2
43.5
18.7/31.
PEALM [J] (10)
111.5
54.0
42.91
35.90
36 40
63.85
Bacite
$0.71
132.83
$3.49
35.01
$5.20
58.23
15.84
35.83
03.24
$2.11
Wasda
4.76/7M0150 50
$1.32 (38033)
128.1
55.25
24.13
42.00
$4.25
35.80
37.17
39.90
31.51
45.41
0000
37

---

6.12
Cosine Similarity
6.10
0.0
604
Tayers
Figure 34: Comparison of implicit multimodal alignment score across layers for different encoders. CLIP models produce features that are most aligned to textual tokens across LLM layers. On the other hand, self-supervised encoders (e.g. MAE) produce the least text-aligned features. However, the relatively low cosine similarity score (closer to O), reveals that the modality gap (e.g. Narrow cones) still exists in LLMs, even for text-aligned encoders.
(Vicunaw2.5)
50
60
Skip Ft1 Tokros 199
Taloos
Thlooss dz
Skip (97
Teloess
PTN Trians
Sctp Thlkws
PTN Triewos
Stcp Thlkes wi
(147)
20
30
40
50
70
80
Figure 35: Skipping computations for visual tokens. Skip Tokens: skip (Skip ratio)% of the tokens in the FFN layers. sl: skipping start layer. (V): visual tokens. (T): textual tokens. Results MT (with LLaVA-1.5) and ST (last column) setups.
Figure 36: t-SNE visualization of tokens inside LLMs. From left to right: layer 0, 1, 24 and 32 for Vicuna-v1.5.
38

---

Figure 37: t-SNE visualization of tokens inside LLM blocks (after SA layers). From left to right:
layer 0, 1, 24 and 32 for Vicuna-v1.5. There is less separation inside the LLM block.
39


Reviews:
[{"review_id": "jHLY0FTvLC", "reviewer": "Reviewer_vaQc", "summary": "This work aims to understand multi-modality representation within MLLMs. It provides some interesting findings about how LLMs generalize to non-textual tokens and what helps LLMs to generalize to multimodal tokens. Additionally, several implications are proposed based on these findings.", "questions": "In the single-task (ST) setting (Fig. 2), the authors use unimodal encoders without text-aligned. Figure 2 illustrates \"Multimodal cones: different narrow cones for different modalities.\" However, I wonder if this phenomenon is related to the unimodal encoders themselves—i.e., different modality features from unimodal encoders might have very different feature distributions, leading to different narrow cones in LLMs. Could the authors conduct new experiments using aligned encoders, such as ImageBind [1], as different modality encoders? This would help determine whether the different narrow cones in LLMs are due to different modalities or different encoders.\n\nRef:\n1. ImageBind: One Embedding Space To Bind Them All. CVPR 2023", "limitations": "As stated by the authors, the generalization of the findings, to larger and more powerful models, with different architectures, including proprietary ones remains to be seen", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The authors present many interesting findings about how LLMs generalize to non-textual tokens. These findings could help in understanding MLLMs and inspire future research.\n2. Based on these findings, the authors propose several implications on performance, safety and efficiency.\n3. The figures are well-illustrated and effectively support the findings.", "weaknesses": "**About experiments on α-SubNet.**\n\n--Figure 12 appears incomplete. What does \"Avg\" mean in this table? \n\n--The comparison between task-specific pruning approaches (Wanda) and the proposed task-agnostic approach is not comprehensive. For instance, what is the performance of Wanda sub-network pruning on COCO across other multimodal tasks? In other words, I'm curious about the generalization of the task-specific methods. Including these results would help in understanding the significance of task and modality-agnostic pruning methods.\n\n--Additionally, the results in Table 1 of the Appendix should be incorporated into the main paper for better readability.\n\n--Overall, I recommend reorganizing the experiments related to α-SubNet.\n\n**Others**.\n\n--line 1192 is unfinished.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors would like to thank the reviewer for the positive feedback and appreciate finding the paper interesting, insightful and well presented. In the following we try to address all the remaining concerns:\n\n## **Weaknesses**\n\n### About experiments on α-SubNet.\n\n- **Clarification about Figure 12**: The \"Avg\" in Figure 12 represents the average score across all multimodal tasks (in this case, VQAv2 and COCO). We will update the figure to reflect this more clearly.\n\n- **Generalization of Task-Specific Methods**: This is an important point. We have included experiments in Figure 6, as well as Figures 26, 27, 28, and 29 in the appendix, that address the generalization of pruned masks/subnetworks to other tasks and modalities. All these results involve task-specific pruning methods (i.e., Wanda). For example, the performance of Wanda sub-network pruning on COCO across other multimodal tasks is as follows (Figure 6): 59.23 (vs 59.69) on VQAv2, 45.19 (vs 44.09) on GQA, and 29.30 (vs 30.89) on OKVQA which indicate very good generalization as the scores are comparable to pruning masks coming from the same datasets (indicated in parenthesis here). The generalization of task-specific pruning to other tasks was a main part of  the motivation for α-SubNet.\n\n- **Incorporation of Table 1 into the Main Paper**: We agree that Table 1 provides a stronger signal regarding the effectiveness of α-SubNet. However, due to space limitations, we included only a subset of the results. We update the revised paper by moving more benchmarks coming from additional modalities.\n\n### Others\n**--line 1192 is unfinished.** Thanks for spotting this. This paragraph is redundant as it is detailed before in line in 1169. We remove this line in the revised paper.\n\n## Questions:\n\n### **Whether the different narrow cones in LLMs are due to different modalities or different encoders?**\nWe can break this question to several sub questions:\n\n1. Does the modality gap still exist for text-aligned encoders?\n\nYes. To demonstrate that the modality gap (or different multimodal \"cones\") persists even for text-aligned features, we conduct a comprehensive comparison between different encoders (new Fig 1 in the uploaded pdf). This includes text-aligned encoders such as CLIP, unaligned encoders like MAE, and encoders trained for classification (e.g., ViT on imagenet). Our findings reveal that: (1) the modality gap persists even for encoders aligned with text; (2) CLIP encoders produce features that are more closely aligned with LLM textual features, while MAE produces the most misaligned features.\n\n2. What causes different narrow cones in LLMs?\n\nPrevious work [1]  has shown that even with the same model architecture and training data, varying the random seed can lead to differences in encoded representations (i.e., different narrow cones). Thus, differences among encoders, or the same encoders trained on different data or modalities, all contribute to having different narrow cones.\n\n3. Does this affect the main paper message?\n\nOur goal is not to show that there is a gap between different multimodal encoders, but rather to highlight that this gap still exists within LLMs even after projecting all modalities into the same LLM input space. Despite this gap between perceptual and textual features, the LLM is still able to generalize effectively. Whether the multimodal aspect is the sole cause of this gap or if there are additional contributing factors, our main conclusions and messages remain valid.\n\n\n[1] Liang, Victor Weixin, et al. \"Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.\" Advances in Neural Information Processing Systems 35 (2022): 17612-17625.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KlKM3QxC0q", "reviewer": "Reviewer_hn2C", "summary": "This paper conducts an in-depth study on the generalization capabilities of LLM when handling multimodal inputs without multimodal fine-tuning. It reveals the implicit multimodal alignment (IMA) effect between perceptual and textual tokens within LLMs and finds that this effect is closely related to the model architecture. The IMA effect contributes to enhanced task performance and reduced inference costs. Additionally, the paper proposes methods for model compression and reducing computational overhead, providing valuable insights for the future design and optimization of multimodal models.", "questions": "I do not fully understand line 247. How is the degree of hallucination measured? What is the relationship between cosine similarity and this measurement?\n\nOut of personal curiosity, how long did these experiments take?", "limitations": "See weakness and question", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Conducted an EXTENSIVE series of experiments to validate four hypotheses, providing insightful understanding of the mechanisms on how the MLLMs perceive multimodal information.\n\nBased on the four findings, the paper offers relevant implications that explain the effects of IMA on tasks and hallucinations.\n\nProposes a novel approach to model compression by retaining a subnetwork.\n\nExperiments are comprehensive and were conducted on various large language models, including OPT, Llama, and Vicuna.", "weaknesses": "The textual descriptions are somewhat difficult to understand, with limited explanation of the figures.\n\nInsufficient explanation of the subnet part. It is unclear whether the similar performance after 50% sparsity is due to the effectiveness of the WANDA method or the extraction of the subnet.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for very positive feedback about the paper and appreciate finding it novel, supported by an extensive series of experiments and insightful messages. This feedback encourages us to push further for similar interesting work. In the following we try to address all the reviewer's remaining concerns.\n\n## Weaknesses:\n\n- **The textual descriptions are somewhat difficult to understand, with limited explanation of the figures.**\n\nWe agree that some figures are not exhaustively explained in the captions. This is due to space limitation, we tried to explain rather in the text or in the appendix. We appreciate it if the reviewer can point to specific figures that need an elaborated explanation. \n\n- **Insufficient explanation of the subnet part. It is unclear whether the similar performance after 50% sparsity is due to the effectiveness of the WANDA method or the extraction of the subnet.**\n\n\nBased on our experiments, the significant drop in performance when exceeding 50% sparsity is attributed to Wanda. To support this, we provide additional scores when pruning the model with Wanda (OPT LLM with a CLIP encoder):\n\n\n| **Sparsity** | **COCO** | **VQAv2** | **GQA** | **OKVQA** |\n|--------------|----------|-----------|---------|-----------|\n| **0.5**  \t| 126.81   | 55.28 \t| 24.72   | 42.00 \t|\n| **0.7**  \t| 84.47\t| 16.41 \t| 10.70   | 1.80  \t|\n\n\nThese results demonstrate a substantial performance decrease with increased sparsity. It is also worth noting that similar observations have been reported in concurrent works [1] (Fig. 4a).\n\n\n[1] Sung, Yi-Lin, Jaehong Yoon, and Mohit Bansal. \"Ecoflap: Efficient coarse-to-fine layer-wise pruning for vision-language models.\" arXiv preprint arXiv:2310.02998 (2023).\n\n\n## Questions:\n\n### **I do not fully understand line 247. How is the degree of hallucination measured? What is the relationship between cosine similarity and this measurement?**\n\nWe follow standard practices for measuring multimodal hallucinations (specifically object hallucinations), which occur when the model describes or refers to objects not present in the input image.\n\n**Hallucination Metrics:**\n- COCO: On the COCO image captioning dataset, the model is asked to describe the images. We compute the CHAIR metrics based on the generated captions and the ground truth annotations of all objects in the image. If a caption contains non-existent objects, we classify it as a hallucinated caption. The CHAIRs score is the ratio of hallucinated captions to the total number of captions. Additionally, we calculate the ratio of hallucinated objects to the total number of objects across all captions, which is referred to as CHAIRi. A CHAIR score of 0 indicates no hallucinations. In the paper, we report (1 - CHAIR) × 100, so a higher score indicates fewer hallucinations.\n\n- POPE: This is a question-answering task involving questions about the existence of objects in images. The metric used is accuracy; the fewer the hallucinations, the higher the accuracy.\n\nWe add these details to the appendix of the revised paper.\n\n**Link to image-text similarity**: In the paper, we hypothesize a link between the degree of cross-modal alignment and hallucinations. Our intuition is that for a textual model to understand the details in images, the image features should be well aligned to the textual domain. The better the alignment (measured here with cosine similarity) the better the model at performing reasoning about it. Misaligned representations can be considered loosely as out-of-distribution/modality samples which cause the model to be less confident and more likely to hallucinate.\n\n### **Out of personal curiosity, how long did these experiments take?** \n\nThe experiments involve both training models and analyzing them. Training LLaVA models typically takes less than one day on 8 GPUs, while ST models require a few hours, depending on the size of the dataset. Computing similarity at the token level also takes a few hours on a single GPU. However, due to the exploration of many different ideas, of which only a portion are included in the paper, the project consumes a significant amount of GPU hours. Generally, such a project can be conducted with an academic-level compute budget (8 GPUs).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4E066TmpPW", "reviewer": "Reviewer_dcgF", "summary": "The paper explores how frozen Large Language Models generalize to multimodal inputs without the need for extensive re-training. It introduces the concept of Implicit Multimodal Alignment, which suggests that despite the distinct representations of perceptual and textual tokens, there exists a form of implicit alignment facilitated by the architecture of LLMs. The study leverages experimental setups across single-task and multitask environments to validate the IMA phenomenon and discusses its implications for computational efficiency and model performance.", "questions": "1. How does the IMA effect influence the overall performance of LLMs on standard multimodal tasks?  Are there performance trade-offs associated with the alignments observed?\n2.  Are there indications that these findings could be applicable to other types of models or architectures not covered in the study?", "limitations": "1. The paper lacks implementation details.\n2. The study focuses primarily on a specific range of LLMs. More domains and tasks should be included.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The analysis spans various setups and modalities, providing a comprehensive look at how LLMs process multimodal inputs.\n2. This study can quickly extend large semantic models to the multimodal domain, effectively reducing computational costs and enhancing the generalizability of LLMs.\n3. The results are meaningful, proving the effectiveness of the method.", "weaknesses": "1. The concept of alignment within neural networks, although well-explored here, does not offer a groundbreaking methodological advance. The novelty lies more in the application context rather than in the development of new techniques or models.\n2. The generalizability of the results is restricted to a subset of model architectures and sizes, potentially limiting the broader applicability of the findings.\n3. The visualization experiments are not clear. Current figures are somewhat generic and do not adequately convey the unique aspects of the IMA effect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the feedback. In the following, we answer the reviewer’s concerns.\n## **Weaknesses:**\n\n- **The concept of alignment within neural networks, although well-explored here, does not offer a groundbreaking methodological advance. The novelty lies more in the application context rather than in the development of new techniques or models.**\n\nAs far as we know, the significant gap between perceptual and textual tokens within LLMs and the ability of LLMs to generalize to non-textual tokens has not been studied before. While we respectfully disagree with the reviewer's categorization, this novelty does not diminish the importance of the messages conveyed by the paper.\n\n- **The generalizability of the results is restricted to a subset of model architectures and sizes, potentially limiting the broader applicability of the findings.**\n\nThe primary motivation for our paper stems from the observation that simply connecting unimodal encoders to LLMs achieves unprecedented performance. This led us to consider two typical setups that use LLMs to tackle multimodal tasks. In the paper, we explore different LLMs (OPT, Vicuna v1.5), visual encoders (ViT, CLIP, MAE), and modalities (image, video, audio and text) as seen for example in Figs. 23, 26, 27, 28, 29, 32. We believe this covers a wide range of multimodal LLM approaches.\n\nRegarding the model size, 7B LLMs are currently widely used and have been shown to outperform many larger LLMs, making them an appealing choice for studying and understanding their capabilities. \n\nHowever, we agree that extending our study to other setups beyond LLMs is an interesting direction for future work, as stated in the discussion in Appendix B.\n\n- **The visualization experiments are not clear. Current figures are somewhat generic and do not adequately convey the unique aspects of the IMA effect.**\n\nWe would appreciate more clarification on what the reviewer means by \"not adequately convey the unique aspects of the IMA effect.\" This comment is too general, and it is unclear to us how to respond.\n\n## **Questions:**\n\n### **How does the IMA effect influence the overall performance of LLMs on standard multimodal tasks? Are there performance trade-offs associated with the alignments observed?**\n\nAs we show in Figure 9, there is a positive correlation between the IMA score and performance on standard multimodal tasks; models with higher performance also have the highest IMA scores, and vice versa. Intuitively, better multimodal alignment leads to better understanding of different modalities. However, we do not claim that the two modalities must be fully aligned, and we leave this investigation for future work.\n\n### **Are there indications that these findings could be applicable to other types of models or architectures not covered in the study?**\n\nOur experiments cover several LLMs with different encoders and connectors, trained on various multimodal datasets. We believe that these observations are generalizable to the broader category of multimodal LLMs (MLLMs). We leave conducting a more in-depth investigation of this for future work.\n\n## **Limitations:**\n- **The paper lacks implementation details.**\n \nDue to space constraints, we included most of the details in the appendix. Could the reviewer point out details that should be considered in the main paper? \n\n- **The study focuses primarily on a specific range of LLMs. More domains and tasks should be included.**\n\nThe current multimodal tasks we cover include question answering and text generation conditioned on images (i.e. image captioning), which cover many multimodal tasks (most of the tasks can be cast as question answering or instruction following). We address four modalities in our work. However, there are other domains of applications, such as 3D vision and speech, that we do not cover, but we believe that our findings should generalize to additional modalities and tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PeLTEbe111", "reviewer": "Reviewer_69kn", "summary": "The authors propose an exploration of the Implicit Multimodal Alignment (IMA) phenomenon in frozen Large Language Models (LLMs): when exposed to perceptual tokens (e.g. from image or audio features), authors show that those tokens are implicitly aligned on text tokens, even so the LLM has only been trained on text previously. The study argues this phenomenon is a fundamental mechanism into LLMs, and that it is relevant to (i) better understand task performance (e.g using that implicit alignment as proxy) and (ii) to better understand model visual faithfulness (i.e absence or presence of hallucinations). Based on their findings, the authors propose as well architecture changes such as skipping specific operations on perceptual tokens in order to decrease inference cost.", "questions": "- Author claim that the alignment between Image <-> Text tokens helps the model performing well on VQA tasks and hallucinations tasks. It would be interesting to know if this is a cause (i.e performance on vision tasks is high thanks to that alignment), or a consequence (i.e alignment is high along performance on vision tasks thanks to the vision-language training). A way to verify that could be to compare alignment among different off-the-shelf models (e.g. LLaVA-1.5, LLaVA-1.6, Qwen-VL-Chat, Phi3-Vision) and see if there are differences in their respective alignment scores. With such population of models and scores, it could be possible to rely on statistical testing to establish it. Do you see a covariation between alignment and performance among that population? Can you control the alignment independently of the training on vision-language itself? E.g. manipulating directly P and T such as the similarity is higher/lower, and still observe that covariation between the alignment and performance variables in that population?\n\n - As well, the authors claim that an LLM can be seen as a residual stream with refinement blocks acting as steering blocks, and that this architecture is what helps the LLM to implicitly having an alignment between text and image tokens. It seems hard to support this claim without contrasting their experiments with another architecture aligning image and text. For instance, using a simple pre-trained CNN to process the image, and a simple pre-trained RNN (e.g. an LSTM or GRU) to process the text input, and then concatenating/pooling the outputs with a simple connector (e.g c-abstractor or just a plain linear projection) could be used to contrast their findings. Is that something planned by the authors?", "limitations": "- One important limitation is the presentation of this work. The majority of the figures are pixelated and not readable. This is often a minor inconvenience in other works, but in this paper all the results are presented through figures. Figure 5 for instance should be removed or importantly reworked. Lacking of space in a limited-page paper is understandable, but not using TKIZ or vectorized figures is not. This is hurting the overall understanding of an interesting work trying to address interesting questions. This work should not be published as is. \n\n- Authors claim that the alignment between Image <-> Text tokens is what helps the model performing well on VQA tasks and hallucinations tasks. Something that is not tested is in which extend this is a cause (i.e performance on vision tasks is high thanks to that alignment), or a consequence (i.e alignment is high along performance on vision tasks thanks to the vision-language training). See \"Questions\" section of this review. \n\n- Authors claim that an LLM can be seen as a residual stream with refinement blocks acting as steering blocks, and that this architecture is what helps the LLM to implicitly having an alignment between text and image tokens. It seems hard to support this claim without contrasting the work with another architecture. See \"Questions\" section of this review.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 4, "strengths": "- This is a very original and in-depth work to understand the inside mechanisms appearing in MLLMs. Many/Most of the current published MLLMs are based on variations a pre-trained LLM + a pre-trained vision encoder with smaller VL connector in between. The insights from this study could be applied on many of those works, especially to better improve the visual faithfulness of those models.", "weaknesses": "- This is not commonly an important weakness but for this work, it might be: the presentation of the results and the overall quality of the layout organization is hurting the understanding. All the figures are pixelated bitmap/jpeg screenshots and are barely readable. This doesn't change the pertinence of the work, but hurt importantly how well its content is conveyed. I urge the authors to revisit this. Many solutions exist today, such as TKIZ or exporting figures in vectorized PDF to better include them in the final document. Lacking of space in a limited-page paper is understandable, but not using TKIZ or vectorized figures is not. Figure 5 for instance shouldn't be as is in this work. I challenge other readers to read/decipher the labels of the x and y axes on the heatmaps. \n\n- In many details, the work could have be improved. See the \"Typos\" list below. As well, some claims are not backed anywhere. For instance, Figure 1 and Figure 12 both claim a 70% sparsity. Where is that introduced in the main text? Consistency is not respected as well: in some contexts, P means Prompt while in other context means Perceptual. Is Prompt == Perceptual? This is not stated in Section 2. Prompt usually refers to the text and image prompts, before preprocessing and tokenization. The formatting is off as well. Sometimes, $P$ is used, sometimes plain-text P. Same for $T$ vs. T. \n\n- Evaluation could be more complete for the Multi Task setup with the use of common benchmarks such as LLaVABench-in-the-Wild, MMVet, etc. As well, Hallucinations benchmarks such as MMHALBench and HallusionBench should be used. Authors quote [46], which is the very work that introduces MMHALBench.\n\nTypos:\n - Line 90: Define k? \n - Line 94: FC1 and FC2 should be $FC1$ and $FC2$. Same for g, LN1, LN2, SA. \n - Line 92: What is (B)? It is not defined in eq (2). \n - Line 101: s/We/we/ \n - L105: Reformulate \"In the paper, we focus no LLaVA-1.5-4 as it is most similar to the ST setup, and analyse other variants in App. E.\"? s/ no / on / ? \n - L106: (e.g. Sec. 3) ? Do you mean (see Sec. 3)? Why e.g.?  \n - Section/Equation/Appendix references are not following the same format. Sec. is used in place of Sec., same for App. and Appendix, but Equation is fully written. What about adopting a consistent terminology? \n- Figure 2 is barely readable, same for Figure 3, Figure 4. Figure 5, as well it is not vectorized. Consider using TKIZ or export your figures in PDF and include those cropped PDF in your tex document. Bitmap/JPEG/PNG used here are leading to poor and pixelated results, hurting the readability. \n- Figure 1: s/Analysing/Analyzing/ \n- Figure 1: s/LMM/LLM/ (twice)\n- Figure 1: s/perceputa/perceptual/ \n- Figure 1: \"Computation efficiency\", do you mean \"Computational efficiency\" or \"Computation and efficiency\"? \n- Figure 1: What is the point of the right block named \"Framework\"? It doesn't seem related to the work or the caption of Figure 1. \n- L136: The claim \"Fig. 2 shows a clear narrow cone effect for textual and perceptual tokens.\" is not verifiable when looking at Figure 2.\n- Figure 10. What is the difference between Cosine Similarity and Sim. Aren't they the same as defined in Equation 3? Why using different terminology?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the detailed feedback on our work. The comprehensive comments reflect a significant effort in reading our paper and providing feedback, which we highly appreciate. We recognize the reviewer's intention to help us further improve our paper and are grateful for their constructive input.\n\n## **Weaknesses:**\n\n- **The presentation of the results and the overall quality**\n\nWe appreciate the reviewer's suggestion to improve the readability of the figures. We have made significant efforts to ensure all numbers in the figures are as clear as possible (sometimes by zooming due to space constraints). To clarify, each figure in our submission is generated by a dedicated Python script that performs model  inference, calculated metrics, creates the figures, and saves them at a very high resolution. Initially, we received initial feedback from several readers about the large PDF file size and slow navigation of the PDF due to the high-resolution figures. Consequently, we compressed the figures to balance clarity and efficient navigation. After inspection, we did not find any figure that could make any claim ambiguous due to clarity.\n\nRegarding Figure 5, we acknowledge that readers may need to zoom in to read it. To address this, we have included a clearer version of Figure 5 in the appendix. \n\nUsing TikZ packages would be an ideal solution for figure clarity, however, it requires significant effort to adapt our codes, rerun the experiments, and reproduce the figures with TikZ. We consider this for future papers.\n\n- **In many details, the work could have be improved (Claims and Typos).**\n\nFigure 1/12: We changed this to 50%. We didn’t put the 70% sparsity results in the paper because the Wanda pruning was not effective at this sparsity regime (see response to hn2C for more details)\n\nPrompt == Perceptual? Yes, we use them interchangeably as mentioned in L85 in Section 2. In our setup he prompt refers to tokens prepended before the textual ones. We unify the formatting to be $T$ and $P$ instead of T and P.\n\n- **Evaluation could be more complete.**\n\nThe benchmarks that we use are typically correlated with other multimodal benchmarks. We agree that adding more recent ones will complete the work and we consider this for future iterations on the paper.\n\n- **Typos:**\n\nWe tried as possible to follow the reviewer's suggestions. We update the paper accordingly and provide answers to raised questions:\n\n**What is the point of the right block named \"Framework\"?**\n\nTo clarify that family of models that we analyze and to show that we focus on the LLM and look at the representations in the residual stream, between and inside the blocks. \n\n**The claim \"Fig. 2 shows a clear narrow cone effect for textual and perceptual tokens.\" is not verifiable when looking at Figure 2.**\n\nWe use the cosine similarity as an indicator of the cone effect. The unimodal tokens have high cosine similarity scores (p vs p) and (T vs T). This score is significantly larger than the cross-modal score (P vs T) or (P vs P in ST across different perceptual modalities). As high scores mean there is a cone effect, this means the narrow cones exist. As the cross-modal cosine similarity is small, this means the multimodal cones are distinct\n\n## **Questions:**\n\n### **Author claim about the alignment between Image <-> Text**\n\n- First, we would like to clarify that our paper discusses positive \"correlation\" and provides initial observations suggesting that the IMA score \"could\" serve as a proxy metric for task performance (framed as questions in the paragraph titles L238 and L243). We don’t claim that alignment leads to better performance or that this alignment should be maximized. Nonetheless this section occupies only half a page.\n\n- Despite being conservative in our claims, we have enough results to consider the IMA score a good candidate metric. We conducted a controlled experimental study by comparing different LLaVA variants (MT setup) and models trained for varying numbers of steps. These experiments showed indeed that the training improves both alignment and performance.\n\n- Additionally, and to support more our hypothesis, we add the following experiments (for the ST setup). The experiment consists of training LMMs with different image encoders; text-aligned (CLIP), self-supervised (MAE) and trained for imagenet (ViT),  We report the task performance and the IMA score (Fig.1 and Tab.1 in the uploaded pdf). Here we also observe that the alignment across correlates with performance: encoders with higher IMA scores (e.g. CLIP) also demonstrates better performance. \n\n- Rigorously studying this alignment across different setups and models is an interesting direction that we leave for future work. We thank the reviewer again for the very interesting suggestion.\n\n### **Claim that an LLM can be seen as a residual stream with refinement blocks acting as steering blocks**\n\n- Contrasting the LLM architecture with other architectures would be necessary if we claimed that this is a unique property of LLM architecture, which we do not. We hypothesize that generalization to non-textual tokens is related to the architecture and provide observations to support this hypothesis.\n\n- The presence of similar architectural inductive biases (e.g., residuals or alternatives to attention modules) in other models does not change the main findings of the paper. Instead, it generalizes and solidifies the proposed hypothesis across a wider range of architectures.\n\n- While studying RNNs might be important, these architectures are not common in the multimodal community. Therefore, we prefer to focus on architectures that are widely adopted.\n\n- We thank the reviewer for this suggestion and leave pursuing this study in future work.\n\n## **Limitations:**\n\n-  Please see answer to weakness 1\n-  Please see answer to question 1\n-  Please see answer to question 2", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
- We would like to thank the reviewers for the very detailed and positive feedback. We appreciate finding our work original (vaQc, hn2C, dcgF, 69kn), provide valuable and interesting insights (vaQc, hn2C, dcgF, 69kn), supported by extensive experimentation (hn2C) and well presented (vaQc). We find the feedback very encouraging with many interesting and relevant suggestions. 

- Based on the reviewers suggestions, we made changes to the paper, some of them are mentioned in this rebuttal and the uploaded PDF, and others are integrated directly to the main paper. The changes mainly cover adding new experiments, clarification and improving the overall presentation (e.g. figures, typos). We detail the changes in our response to each reviewer.

- In the following we try to address all reviewers ' concerns.

Meta-review: This is an original and exciting attempt to understand the underlying mechanics of how LLMs generalize beyond text tokens to process multimodal inputs. All the reviewers liked this work. For examples, Reviewer vaQc noted that "The authors present many interesting findings about how LLMs generalize to non-textual tokens. These findings could help in understanding MLLMs and inspire future research." Reviewer hn2C commented "Conducted an EXTENSIVE series of experiments to validate four hypotheses, providing insightful understanding of the mechanisms on how the MLLMs perceive multimodal information." Reviewer 69kn added that "This is a very original and in-depth work to understand the inside mechanisms appearing in MLLMs. Many/Most of the current published MLLMs are based on variations a pre-trained LLM + a pre-trained vision encoder with smaller VL connector in between. The insights from this study could be applied on many of those works, especially to better improve the visual faithfulness of those models."

Reviewer dcgF who seems most critical about the work, although still positive about the work, is expecting more "methodological advance" from the work, and points out other limitations of the work such as findings specific to model architectures. I think the authors addressed these concerns well in the rebuttal. 

In conclusion, this is an exciting work to advance our understanding about LLMs, which should be of highly interest to NeurIPS audience.
Decision: Accept (poster)