Title: Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large multimodal models

LLMs

cross-modal alignment

efficiency

safety

interpretability
Abstract: Large Language Models (LLMs) have demonstrated impressive performance on multimodal tasks, without any multimodal finetuning. They are the de facto building block for Large Multimodal Models (LMMs), yet, we still lack a proper understanding of their success. In this work, we expose frozen LLMs to image, video, audio and text inputs and analyse their internal representation with the attempt to understand their generalization beyond textual inputs. Our work provides the following **findings.** Perceptual tokens (1) are easily distinguishable from textual ones inside LLMs, with significantly different representations (e.g. live in different narrow cones), and complete translation to textual tokens does not exists. Yet, (2) both perceptual and textual tokens activate similar LLM weights. Despite their differences, (3) perceptual tokens are implicitly aligned to textual tokens inside LLMs, we call this the implicit multimodal alignment effect (IMA), and argue that this is linked to architectural design, helping LLMs to generalize. This provide more evidence to believe that the generalization of LLMs to multimodal inputs is mainly due to their architecture.  These findings lead to several **implications.** This work provides several implications. (1) We find a positive correlation between the implicit alignment score and the task performance, suggesting that this could act as a proxy metric for model evaluation and selection. (2) A negative correlation exists regarding hallucinations (e.g. describing non-existing objects in images), revealing that this problem is mainly due to misalignment between the internal perceptual and textual representations. (3) Perceptual tokens change slightly throughout the model, thus, we propose different approaches to skip computations (e.g. in FFN layers), and significantly reduce the inference cost. (4) Due to the slowly changing embeddings across layers, and the high overlap between textual and multimodal activated weights, we compress LLMs by keeping only 1 subnetwork (called alpha-SubNet) that works well across a wide range of multimodal tasks. The code is available here: https://github.com/mshukor/ima-lmms.

Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs

Abstract

Large Language Models (LLMs) have demonstrated impressive performance on multimodal tasks, without any multimodal finetuning. They are the de facto building block for Large Multimodal Models (LMMs), yet, we still lack a proper understanding of their success. In this work, we expose frozen LLMs to image, video, audio and text inputs and analyse their internal representation aiming to understand their generalization beyond textual inputs. Our work provides the following findings. Perceptual tokens (1) are easily distinguishable from textual ones inside LLMs, with significantly different representations ( e.g. live in different narrow cones), and complete translation to textual tokens does not exist. Yet, (2) both perceptual and textual tokens activate similar LLM weights. Despite being different, (3) perceptual tokens are implicitly aligned to textual tokens inside LLMs, we call this the implicit multimodal alignment effect (IMA), and argue that this is linked to architectural design, helping LLMs to generalize. This provide more evidence to believe that the generalization of LLMs to multimodal inputs is mainly due to their architecture. These findings lead to several implications. (1) We find a positive correlation between the implicit alignment score and the task performance, suggesting that this could act as a proxy metric for model evaluation and selection. (2) A negative correlation exists regarding hallucinations ( e.g. describing non-existing objects in images), revealing that this problem is mainly due to misalignment between the internal perceptual and textual representations. (3) Perceptual tokens change slightly throughout the model, thus, we propose different approaches to skip computations ( e.g. in FFN layers), and significantly reduce the inference cost. (4) Due to the slowly changing embeddings across layers, and the high overlap between textual and multimodal activated weights, we compress LLMs by keeping only 1 subnetwork (called α -SubNet) that works well across a wide range of multimodal tasks. The code is available here: https://github.com/mshukor/ima-lmms.

1 Introduction

Large Language Models (LLMs) [1, 2, 3, 4, 5] represent a noteworthy advancement in recent AI developments. Building upon the success of LLMs, the next stride in this field involves extending beyond the textual modality, giving rise to Large Multimodal Models (LMMs) [6, 7, 8, 4]. A notable line of research involves connecting LLMs to visual encoders, while keeping them frozen and only training a connector with modest number of parameters [9, 10, 11, 12, 13, 14, 15, 16, 17]. These methods yield comparable performance [10] to large-scale multimodal models with significantly reduced computational and data budget.

Keeping all pretrained unimodal models frozen and only training couple of millions of parameters [9, 10, 12] is an interesting phenomenon to understand, with limited research trying to decipher it. To

SA

FFN

Perceptual

Encoder

LLM

IMA score

Acc

Alignment (IMA)

Layers

IMA score

Hallucinations

LLM

(50% sparsity)

LMM compression

Skipping Computations

Multimodal Narrow Cones

Computational Efficiency

Performance and Safety

LMM

Analyzing LMMs

Implications: performance, safety and efficiency

Residual Stream

with steering blocks

L

FFN

SA

FFN

SA

FFN

SA

Framework

L

explain why frozen LLMs can generalize beyond textual inputs, several hypotheses can be isolated: (1) perceptual tokens are transformed to textual ones, can be simply considered as foreign language [18], and thus the LLM sees text-only tokens. (2) LLMs are able to digest non-textual tokens that are processed by (2a) modality-specific subnetworks or (2b) the same LLM weights that can generalize due to other reasons.

In this study, we expose LLMs to different multimodal inputs, such as image, video, audio and text, and analyse their internal representations. We focus on frozen LLMs and consider two representative setups: single-task (ST) and multitask (MT) finetuning. The former is considered by parameter and data-efficient approaches [11, 12, 9, 10] and consists of training a mapping module for each dataset. The MT setup is considered by recent multimodal assistant models [19, 14, 20, 21], and consists of training the same mapping module on several datasets/tasks.

Our study shows (1) that perceptual and textual tokens still live in significantly different representation spaces inside LLMs (Sec. 3.1): live in different narrow cones, and have different norms, rate of change and vocabulary distributions. (2) We notice high similarity between the weights activated by textual and perceptual tokens (Sec. 3.2), allowing to swap these activated subnetworks between different tasks and modalities. Despite their differences, (3) perceptual tokens are implicitly aligned to textual ones across different stages (Sec. 4.1): during training of the mapping module, and during inference across LLM layers, especially inside each LLM block ( e.g. after the self-attention layer). As there is no explicit objective to align these representations, we call it the Implicit Multimodal Alignment effect (IMA) ( i.e. , increasing similarity between textual and perceptual token distributions). We find that this effect is mostly linked to architectural design ( e.g. residual stream with refinement blocks acting as steering blocks Sec. 4.2). This provides more evidence to believe the architecture of LLMs is one of the main factors to generalize to multimodal representations.

We shed light on several practical implications (Sec. 5). (1) We find a positive correlation between the implicit multimodal alignment score and the task performance, suggesting that this score could act as a proxy metric. On the other hand, (2) we find a negative correlation with hallucinations, revealing that the main factor leading to this problem is the lack of alignment between the internal representation of textual and perceptual inputs. (3) The perceptual tokens slightly change inside LLMs, thus, we propose to skip their computations ( e.g. inside FFN layers). (4) Due to the slowly changing embedding across layers, and the high overlap between weights activated by different modalities, we compress the LLM by keeping one task-agnostic subnetwork that works well across all modalities. To summarize (Fig. 1), we analyse the internal representations of LLMs when exposed to multimodal inputs, leading to the following findings :

Perceptual and textual tokens live in different representation spaces inside LLMs.

They activate similar LLM weights.

They are implicitly aligned (IMA) inside LLMs, during training and during inference.

The architectural design of LLMs can be perceived as a residual stream with steering blocks. We argue that this is one of the main factors allowing LLMs to: digest very different tokens, drive the implicit multimodal alignment effect, and thus generalize to different modalities.

These findings have several practical implications such as:

The IMA score as a proxy metric candidate for task performance and hallucinations.

Hallucinations as a result of lack of sufficient multimodal alignment.

Skipping computations for visual tokens, leading to efficient inference.

LLMs compression by keeping only 1 subnetwork that generalizes to all multimodal tasks.

2 Framework for analysing preceptually augmented LLMs

General framework We focus on a general family of models that consists of: a frozen language model LLM with L layers, a trainable mapping module C , and a frozen perceptual encoder E M for different modalities M ( e.g. image (I), video (V) and audio (A)). The LLM input X consists of the concatenation of P = [ p 1 , ..., p N p ] multimodal/perceptual tokens (referred to as prompt) with T = [ t 1 , ..., t N t ] textual tokens. The prompt P is obtained after encoding the modality-specific input XM with the corresponding E M and using C to project it to the LLM input space. T is obtained from the embedding layer E T applied to the tokenized input text XT . This can be expressed as follows:

The k ( k = N p + N t ) output tokens O = [ o i , ..., o k ] are obtained after a normalization, followed by the unembedding layer W out (or LLM head, i.e. o i = W out LN out ( t L i ) ). Our focus is on the internal representation of LLMs ( i.e. tokens) at different stages, in particular across the L LLM blocks/layers (referred to as B). The mechanism inside the l + 1 LLM transformer block can be expressed as follows:

where FC 1 , FC 2 , g are the up and down projections and activation inside the FFN , LN 1 / 2 are the layer norms and SA the self-attention.

Perceptually augmented LLM baselines. For the single-task (ST) setup, we train many models across different datasets that span image, video and audio-text modalities. Each mapping module is trained on a specific dataset, similar to previous works [10, 12, 9]. Inspired by previous studies [10, 12], we use light-weight transformer consisting of a self-attention to attend to perceptual tokens. In this setup, P refer to perceptual tokens from image, video and audio modalities. For the multitask (MT) setup, we devise different variants of the LLaVA-1.5 [19] model that differ from the original model as follows: LLaVA-1.5-2 (LLM kept frozen), LLaVA-1.5-3 (LLM kept frozen, without pretraining) and LLaVA-1.5-4 (LLM kept frozen, without pretraining and with transformer mapping module similar to the ST setup instead of MLP). In this setup, P refers to image tokens from different datasets. In the paper, we focus on LLaVA-1.5-4 as it is most similar to the ST setup, and analyse other variants in App. E. For analysis ( i.e. Sec. 3), we focus on Vicuna-v1.5-7B [22] as it is shared by both setups. For the ST, we use unimodal encoders, such as ViT [23], TimeSformer [24] and AST [25] that are not aligned with text. More implementation details, and experiments with other backbones can be found in App. D and App. E. We report the similarity after averaging the tokens SimAvg (Eq. (3)) and details other measures in App. E.

Analysis tools. We are interested in cross-modal or multimodal alignment, and define the alignment in terms of the cosine similarity; the higher the score, the more the vector representations are pointing in similar directions. This could also indicates how much the two token distributions or vectors are close, in terms of L2 distance (assuming the vectors are normalized and in a narrow cones). In other words, alignment and similarity terms can be used interchangeably in the paper. In addition to cosine similarity we also study their norm, decoded vocabulary distributions and which LLM weights they activate. In the paper, we focus on the global representation per example, by analysing their average across the sequence. More finegrained analysis on the token level with different similarity and norm

measures gives similar observations and are detailed in App. E. For instance, we compute the cosine similarity between perceptual ( P ) ( e.g. , tokens corresponding to image patches) and textual ( T ) tokens ( e.g. , tokens corresponding to the image caption), after the block l as follows:

3 LLMs indeed generalize to non-textual tokens

We investigate the generalization of LLMs to multimodal inputs, by studying the perceptual and textual tokens inside LLMs. We investigate if all tokens are projected to textual ones, or rather they are still different and how so (results with other models and similarity measures in App. E).

V-P

Embedding similarity

I-P

-T

1.0

A-P

Cosine Similarity

8'0

P vs P

0.6

P vs T

T vs T

0.4

p vs p

t vs t

0.2

0.0

0

5

10

Layers

15

20

25

30

I0-P

Embedding similarity

I1-P

-T

12-P

1.0

I3-P

0.6

0.4

P vs P

P vs T

0.2

T vs T

p vs p

0.0

tvst

0

5

10

Layers

15

20

25

30

(a) ST setup. Tokens from multimodal datasets.

(b) MT setup. Tokens image-text datasets.

3.1 How perceptual tokens differ from textual ones?

Multimodal cones: different narrow cones for different modalities (Fig. 2). Previous works [26, 27, 28, 29, 30, 31] have found the representation of contextualized embeddings in language models to be anisotropic: embeddings of different inputs exhibit high cosine similarity, shaping a narrow cone, where all embeddings point in the same narrow direction. In the multimodal domain, the cone effect is also observed [32] in contrastive models (CLIP [33]). In this section, we investigate if textual and multimodal tokens live in narrow cones inside LLMs, and if these cones are distinct. We compute the tokens cosine similarity at different layers. In particular, the unimodal similarity: text-only ( T vs T ) and perceptual-only ( P vs P ), and the cross-modal similarity ( P vs T ) between perceptual and textual tokens. Note that for the ST setup, P vs P covers the similarity between image, video and audio tokens, while for the MT ones cover image tokens from different datasets. Fig. 2 shows a clear narrow cone effect for textual and perceptual tokens. Different perceptual modalities seem to live in different narrow cones, as shown by the low P vs P score for the ST setup. Interestingly, the cross-modal similarity between textual and perceptual tokens ( P vs T ) is significantly lower, suggesting that textual and perceptual tokens also live in different narrow cones. We also visualize the t-SNE of the tokens embeddings showing they stay separated inside the LLM.

0.8

1200

1.0

120

Cosine Similarity

0.6

0.7

1000

0.8

100

0.5

Tokens Norm

800

Cosine Similarity

0.6

Tokens Norm

80

0.4

600

60

0.3

0.4

-P-(n-1)-

400

40

0.2

0.1

0.0

0

5

10

Layers

15

20

25

-T (n 8)

-T (n 1)

-P (n 8)

30

200

D

01

Layers

15

20

25

P（n1)

-T(n1)

30

0.2

0.0

0

5

10

Layers

15

20

25

-P (n 1)

-T (n 1)

-P (n 8)

-T (n 8)

30

20

0

5

10

Layers

15

20

25

-P (n 1)

-T (n 1)

30

Different token norms and evolution across layers (Fig. 3). We compute the median of the token L2 norms after each LLM block. This shows that textual and perceptual tokens have different norms across layers. Perceptual tokens have significantly higher norm (at the beginning for MT and across all layers for ST), and they change significantly less. When looking at other norm measures, we found perceptual tokens with massive norms, similarly for textual ones [34], especially for the ST setup. We discuss massive tokens more in App. E.2. In addition, we compute the cosine similarity between tokens at block l and block l + n , showing that textual and perceptual tokens have different change rates. Textual ones change drastically at the beginning of the LLM, while perceptual ones changes significantly less across all layers.

Histogram (l=0)

KL-divergence (

X

l

+

n

vs

X

l

)

KL-divergence

Distribution entropy

700

10

1.2

T (n 1)

P (n 1)

P (n 8)

16

14

-P vs P

-P vs T

10

T

P

100

T (n 8)

Distance

8

Distance

12

10

Entropy

100

200

100

0005

10000

Vocabulary

15000

20000

25000

30000

10

Layers

15

20

SZ

10

Layers

15

20

25

30

10

Layers

15

20

25

30

Different token vocabulary distributions across layers (Fig. 4). For each token, we use the LLM unembedding ( i.e. LLM head) to decode the latent representation to a probability distribution over the vocabulary. This approach have shown to work well for LLMs at different layers, not just the last one [35, 36, 37, 38]. We show the histogram of this distribution at the first LLM layer for both textual and perceptual tokens. The histograms show clear differences with some overlap between textual and perceptual prompts. In addition, we compute the KL-distance showing that the distributions diverge from each other across LLMs layers. We also notice that the distributions of textual tokens evolve significantly, compared to multimodal ones. This is shown by computing the KL-distance between consecutive blocks and the distribution entropy.

Finding 1. Textual and perceptual tokens live in significantly different representation spaces inside LLMs.

3.2 Do perceptual tokens traverse different paths inside LLMs?

For each trained model, we extract the LLM (frozen) subnetwork activated by each dataset/modality. Westudy these subnetworks (we refer to as pruning masks) by computing their similarity. We leverage the recent SoTA pruning approach (Wanda [39]), that prune models based on both the weights and the activation norms. Specifically, we use a handful ( e.g. 256) of calibrated examples coming from different modalities, and keep only p% (1 - sparsity) of weights with the highest Wanda score, at different sparsity levels (30 % and 50 %). Note that after removing more than 50% of weights we observe a severe degradation of performance.

Similar activated weights across modalities, in the first and deeper layers (Fig. 5). Each subnetwork is represented as a binary mask to indicate which weights are activated. To compute the similarity between these networks, we consider the intersection over union (IoU). Results show an interesting high similarity between subnetworks activated by to different modalities. This high overlap is more seen for the ST setup, for example, the IoU between GQA and VQAv2 is 0.69, similar to GQA vs Audiocaps (0.67) or COCO-Text (0.65). When looking at the IoU across layers, we notice an interesting high score at first layers. It seems that first layers encode general features that are common for all modalities. This similarity increases as we go deeper in the LLM, moving to more abstract and less modality-specific representations, closer to the textual output.

Transfer of multimodal subnetworks across tasks and modalities (Fig. 6). To further validate the previous section, we study if we can simply interchange pruning masks between different tasks.

()AOKVQA

1.00

0.99

0.99

0.99

0.63

0.64

0.09

All layer types (Sparsity: 0.5)

9.99

All layer types (Sparsity: 0.5)

(00000

0.99

1.00

0.99

098

0.63

0.64

0.70

0.80

I vs T

V vs T

1.00

0.90

0.68

I vs T

0.99

0.99

1.09

0.99

0.63

0.64

0.75

A vs T

(A) Ce0e-4Q4

9

0.85

0.66

8.99

0.98

4.99

1.00

0.63

0.64

0.69

6.70

0.85

Masks IoU

0000

@G04

0.72

0.6

0.69

0.65

0.72

0.67

0.74

0.67

1.00

0.80

Masks IoU

0.62

0.64

(T) A-0KVQASet

8.63

0.63

0.63

6.63

1.00

0.71

9.66

0.71

0.86

0.65

0.60

(MMS5TOA

()00V0A

DVOA

0.11

0.73

0.71

0.69

0.68

O.49

9.71

0.71

B6

0U65

1.00

0.75

0.60

0.58

95'0

(T-coco-Sxst

(T)-0945ss

0.64

0.43

0.64

0.69

0.64

0.63

6.64

0.71

1.64

1.00

0.66

60

0.55

Transformer Layer

10

15

20

25

0.72

0.70

0.54

Transformer Layer

19

15

20

L6S

VQAv2COCO

117.36

Modality: image (ViT, Sparsity: 0.3)

59.23

45.19

29.30

49.41

Modality: video (ViT,Sparsity: 0.3)

37.20

37.14

50

48

Audiocaps

63.98

Modality: audio (ViT,Sparsity: 0.3)

24.50

46.44

60

100

55

116.45

59.69

44.08

30.47

80

50.04

37.30

37.26

44

46

Clotho

63.52

24.58

44.84

45

50

117.05

59.83

44.09

30.92

60

42

ClothoAQA

40

40

35

118.17

59.87

43.71

30.89

40

49.47

36.60

36.65

64.06

24.61

44.37

30

COCO

VQAv2

GQA

OKVQA

MSRVTT

MSRVTTQA

MSVD

Audiocaps

Clotho

ClothoAQA

25

TextMSRVTT COCOAudiocaps

(ViT, Sparsity:0.3)

120

Text MSRVTT COCOAudiocaps

(ViT,Sparsity:0.3)

TextMSRVTT COCOAudiocaps

(ViT,Sparsity:0.3)

67.13

116.26

48.90

110

65.68

115.83

47.27

110

65.00

115.93

47.62

110

100

100

100

64.80

121.21

49.92

90

65.60

116.34

46.94

90

65.88

116.99

49.30

90

61.89

66.20

120.16

120.15

51.00

49.66

80

60

70

67.40

64.81

117.19

115.43

47.70

47.39

80

60

70

50.06

63.79

116.50

95.06

49.16

41.63

80

70

60

50

50

50

Audiocaps

COCO

MSRVTT

Audiocaps

COCO

MSRVTT

Audiocaps

COCO

MSRVTT

Specifically, for a given model trained to solve a particular task, we find the pruning mask using calibration data corresponding to different tasks/datasets. The sparsity is set to 30%, which is often used to maintain reasonable performance. Fig. 6 shows that the pruning masks transfer very well across tasks within the same modality ( e.g. slight degradation by ∼ 1 point CIDEr for captioning with a mask coming from OKVQA). Similarly, we interchange masks across modalities. We fix the task (captioning) and also consider the text modalities (captions without images). In general, we observe similar transfer with a slight performance degradation, especially for OPT and Llama 2. We show similar observations with higher sparsity and with other encoders ( e.g. , CLIP and MAE) in App. E.4.

Modality-specific subnetworks? The experiments suggest a high overlap between weights activated by different modalities. However, this does not exclude the possibility of finding weights that are generally activated when seeing a particular modality, even if there are small amount of them. More discussion about this can be found in App. E.4.

4 What helps LLMs to generalize to multimodal tokens?

Textual and perceptual tokens have very different representations inside LLMs, yet, LLMs are still able to process and generalize to these non-textual tokens. In this section, we try to investigate why this is possible, in particular, we identify which factors facilitate this generalization.

Embedding similarity

Embedding similarity

0.150

PvsTep0

Pvs T ep 1

0.25

V vs A

Vvs I

1.0

Simil

0.125

0.100

Similarit

0.20

I vs A

VvsT

IvsT

Cosine Similarity

0.8

0.6

Cosine

S

0.075

575

Cosine

0.15

A vs T

0.050

0.4

0.10

0.025

0.2

Pvs P

0.000

0.05

P vs T

0

5

10

Layers

15

20

25

30

0

5

10

Layers

15

20

25

30

0.0

0

5

10

Layers

15

20

25

30

4.1 Observation: the Implicit Multimodal Alignment Effect (IMA)

Implicit alignment during training of the mapping module (Fig. 7). We compute the cosine similarity between the perceptual tokens at the output of the mapping module, and textual tokens at different LLM blocks. Results show that this similarity increases at all layers. This reveals that the mapping module role, is not just to adapt the dimension of the visual tokens, but also to project the visual tokens to be semantically, as similar as possible to the textual ones.

Implicit alignment during inference, across LLM blocks (Fig. 7). We compute the cosine similarity between perceptual and textual tokens after each LLM block. Here we compute the max of tokenwise similarity (MaxSim App. E.1): for each pair of token sequences coming from one example ( e.g. image prompt + caption), we take the maximum similarity, then we average across all examples. The tokenwise similarity between perceptual and textual tokens significantly increases, especially in the middle blocks, where the alignment is the highest in deep layers.

(a) Token norms

(b) Consecutive blocks similarity

(c) Similarity inside each block.

0.8

175

0.8

150

0.6

125

0.6

Tokens Norm

Cosine Similarity

0.4

100

75

ILN1|I-P (n 1)

ILN1|I-T (n 1)

0.2

I|LN1|-P (n 1)

IILN1|I-T (n 1)

I|LN1|I-Pvs P

IISAII-P (n 1)

IISA|I-P (n 1)

IILNI|I-PVST

IISAII-T (n 1)

IISAII-T (n 1).

IISAIl-P.vsP

50

IILN21|-P (n 1)

0.2

IISAI]-PvsT

IILN21-T(n1)

0.0

LN2]]-T(n1)

I|LN21-PvsP:

25

FC11-P(n-1)

FOI|-T(n1)

HPC21·P(n-1)

IIFC2|I-T (n 1)

0.2

I|FC1||-T (n 1)

IIFC2||-P (n 1)

0'0

FCI-PvsP

1LN21PVST

IIFC1|I-P vs T

IIFC2|I-P vs P

5

10

Layers

15

20

25

30

10

Layers

15

20

25

30

0

U

10

Layers

15

20

25

30

Implicit alignment during inference, inside LLM blocks (outside the residual stream) (Fig. 8). We look deeper to investigate the source of this alignment, and focus on tokens inside the LLM block, which consists mainly of a self-attention (SA), FFN (FC1/2) and layer norms (LN1/2) layers. Interestingly, the similarity between textual and multimodal tokens is the highest after the SA layers.

Finding 3. An implicit multimodal alignment emerges to pull the textual and perceptual tokens closer inside LLMs, during training and inference.

4.2 Explanation: the architectural inductive bias hypothesis

Residual stream with refinement blocks. We notice different observations between the tokens inside and outside the residual stream. In the residual stream, the perceptual and textual tokens exhibit significant representation differences (Sec. 3.1), while outside the residual stream, they are more aligned. Each block contributes slightly to the residual stream (small token norms inside the blocks Fig. 8a), with significantly different contributions (cosine similarity between consecutive blocks close to zero, e.g , after the FC1/2 Fig. 8b). This allows us to view the model as a series of refinement blocks that try to gradually refine the input signals. As the original signals are significantly different, they stay different in the residual stream throughout the model. We argue that this provide a flexibility to handle too different inputs. Moreover, previous works [40] have shown that transformers contain both elements with high and low complexity biases, which helps to build general-purpose architectures [41] that are able to generalize. These works support further our findings.

Refinement blocks as steering blocks. Inside the transformer block, we notice that the layer normalization play an important role in having comparable norms for both textual and perceptual tokens (Fig. 8a). Perceptual token norms become smaller and closer to textual ones as we traverse several layers in the block. In terms of cross-modal similarity, we notice the highest similarity after the SA, then after the FC2 and LN1. Note that this similarity is higher inside the block, than in the residual stream ( e.g. 0.45 vs 0.1 for Vicuna-v1.5 and 0.58 vs 0.15 for LLaVA-1.5-4 in the residual stream Fig. 2). After each block the cross-modal alignment increases, and hence the narrow cones are steered to each other. This suggests that all layers play an important role in steering the textual and perceptual narrows cones to be aligned, with the most contributions coming from the SA.

Finding 4. An LLM can be seen as a residual stream with refinement blocks acting as steering blocks. This architecture design plays an important role in generalizing to very different tokens, and hence other modalities.

5 Implications: performance, safety and efficiency

65

VQAv2

GQA

0.15

60

TextVQA

Cosine Similarity

Task Performance

55

SEED-IMG

0.10

0.05

P vs T step 1500

P vs T step 3000

P vs T step 4500

35

0.00

P vs T step 6000

P vs T step 7500

30

0

01

15

30

0.07

0.08

0.09

0.10

0.11

0.30

P vs T LLaVA-v1.5-4

80

VQAv2

Similarity

0.25

0.20

P vs T LLaVA-v1.5-1

PvsT LLaVA-v1.5-3

P vsT LLaVA-v1.5-2

TaskPerformance

70

GQA

TextVQA

SEED-IMG

0.15

60

Cosine

S

0.10

50

0.05

40

0.00

0.05

30

0

5

10

Layers

15

20

25

30

0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16

Layers

Sim(P, T)

Sim(P, T)

Implicit alignment as a proxy metric for task performance? (Fig. 9) We compute the cosine similarity between perceptual tokens at the LLM input and the textual tokens across LLM layers. The similarity increases during training. Interestingly, we notice a clear and positive correlation with the task performance on several multimodal benchmarks. In addition, we find that this correlation exists across different models, as shown for different LLaV A-1.5 variants.

Implicit alignment as a proxy metric for hallucination? (Fig. 10) Previous works have shown that LMMs suffer from severe hallucinations [42, 43, 44], and generally try to tackle this problem by training on better datasets [45], using RLHF or RLAIF [46, 47] or post-training heuristics [48, 49]. Here we highlight one of the main causes of hallucinations: which is the lack of internal alignment between textual and perceptual representations. we show the cosine similarity between textual and perceptual tokens after each LLM block, and report the hallucinations on POPE [50] and COCO [51] benchmarks. The curves show clear correlation between the implicit alignment and the hallucinations.

0.6

P vs T step 1500

P vs T step 3000

90

1 - CHAIRs

P vs T step 4500

80

1 - CHAIRi

P vs T step 6000

Task Performance

P vs T step 7500

70

0.4

60

Cosine

0.3

50

0.2

40

0.1

30

0

5

10

15

20

25

30

0.07

0.08

0.09

0.10

0.11

Layers

Sim(P, T)

Similarity

0.8

0.7

0.6

P vs T LLaVA-v1.5-3

P vs T LLaVA-v1.5-1

P vs T LLaVA-v1.5-4

Pvs T LLaVA-v1.5-2

Task Performance

100

90

1 -CHAIRs

1 - CHAIRi

0.5

80

Cosine

0.4

70

0.3

60

0.2

0.1

50

0.0

0

5

10

15

20

25

30

Layers

Sim(P, T)

SA

FFN

Perceptual Encoder

Mapping Module

Token skipping

X N

Emb

LLaVA-1.5

LLaVA-1.5

60

65

A

*

GQA

TextVQA

SEED-IMG

50

09

40

55

Acc

AcC

30

50

20

45

Skip FFN Tokens (V) sl0

10

0

★

GQA

SEED-IMG

TextVQA

Skip FFN Tokens (V+T)

Skip FFN Tokens (V)

40

A

A

Skip FFN Tokens (V) sl2

Skip FFN Tokens (V) sl4

Skip FFN Tokens (V) sl6

Skip FFN Tokens (V) sl8

5.5

6.0

6.5

7.0

FLOPS

7.5

8.0

8.5

9.0

9.5

5

9

FLOPS

7

8

6

Skipping computations for visual tokens (Fig. 11). In Sec. 3.1 we show that perceptual tokens change significantly less across layers, compared to textual ones. Sec. 4 highlights the importance of SA layers for cross-modal alignment. In this section, we leverage these observations to reduce the LLM computation overhead by skipping the computations of visual tokens. Specifically, starting from a given start layer (sl), we reduce computations in FFN layers, which accounts for almost 2/3 of model weights, by skipping p% (Skip ratio) of visual tokens. Fig. 11 shows that skipping the visual tokens leads only to slight decrease in performance, while reducing significantly the amount of compute. We provide additional results with the ST setup, and ablation study in App. F.3.

Perceptual

Encoder

Mapping Module

LLM

(50% sparsity)

Method

#P/#TP/Sparsity

Avg

COCO

↑

VQAv2

↑

MSR-VTT

↑

MSVD-QA

↑

Audiocaps

↑

CIDEr (test)

Acc (Val)

CIDEr (test)

Acc (test)

CIDEr (test)

MAPL [12]

7B/3.4M/0.00

-

125.2

43.5

-

-

-

6.7B/4M/0.00

-

111.6

54.9

48.79

38.40

61.86

DePALM [10]

7B/18.1M/0.00

-

131.29

70.11

49.88

-

69.70

Baseline

6.7B/7M/0.00

57.71

132.83

63.49

58.23

38.83

68.24

6.7B/7M/0.50

51.32

126.81

55.28

54.23

37.17

58.99

Random mask

6.7B/7M/0.47

0.00

0.00

0.00

0.00

0.00

0.05

α

-SubNet (s=0.3)

6.7B/7M/0.47

39.34

106.77

51.77

38.37

31.19

23.15

a

A

α -SubNet: one LLM Subnetwork for all multimodal tasks (Fig. 12) . Despite their differences, multimodal tokens share an important property: slowly changing embeddings across layers (Sec. 3.1). This suggests the possibility of compressing the model while retaining reasonable performance. In addition, textual and multimodal tokens are pulled closer inside the LLM (Sec. 3.2), and processed by almost the same LLM weights (Sec. 4), especially for the ST setup. This suggests the possibility of finding a common subnetwork ( α -SubNet) that works well for all multimodal tasks. Thus, we focus on the ST setup with the OPT and CLIP encoders that are currently used by previous works. We consider two representative tasks: COCO image captioning and VQAv2 and provide similar results for other tasks, and modalities in App. F.4. First we use Wanda for task-specific pruning

and show in Fig. 12 that we can obtain scores close to the original ones while removing 50% of the weights. To find the task and modality agnostic α -SubNet, we first extract many pruning masks ( e.g. at 30% sparsity) for different modalities, then take the intersection of all these masks ( e.g. leading to a global mask at ∼ 50% sparsity). This approach is significantly better than other baselines such as magnitude pruning or a random mask, and leads to comparable performance compared to the task-specific Wanda pruning, especially for VQAv2.

6 Discussion

Limitations. The paper focuses on open-source and frozen LLMs up to 7B parameters, LMMs that concatenate perceptual tokens at the LLM input and are relatively efficient. The generalization of our findings, to larger and more powerful models, with different architectures, including proprietary ones remains to be seen. Detailed discussion in App. B.

Conclusion. We propose the first study of the internal representation of frozen LLMs when exposed to multimodal inputs. We find very different representations for perceptual and textual tokens, yet LLMs are still able to generalize to these non-textual tokens. The implicit multimodal alignment (IMA) effect, linked mostly to architectural design, facilitates this generalization by bringing multimodal tokens closer inside the LLM. Our findings have several implications, such as as reducing the computation resources at inference time, understanding better the performance as well as safetyrelated problems such as hallucinations. We hope that this study will have positive impact, pushing for more works to understand multimodal LLMs, and pave the way to devise more powerful models that are better aligned to human preferences, while targeting safety-related issues.

7 Acknowledgments

The authors would like to thank Arnaud Dapogny and Edouard Yvinec for fruitful discussions, and Damien Teney and Alexandre Ramé for their helpful feedback on the paper. This work was partly supported by ANR grant VISA DEEP (ANR-20-CHIA-0022), and HPC resources of IDRIS under the allocation 2024-[AD011013415R2] made by GENCI.

References

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022. 1, 24

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022. 1, 24, 26

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022. 1, 24

OpenAI. Gpt-4 technical report. arXiv , 2023. 1, 24, 25

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023. 1, 24, 26

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems (NeurIPS) , 35:23716-23736, 2022. 1, 24, 25, 26

Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up a multilingual vision and language model. arXiv preprint arXiv:2305.18565 , 2023. 1, 24, 25

Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 , 2023. 1, 24

Mustafa Shukor, Corentin Dancette, and Matthieu Cord. ep-alm: Efficient perceptual augmentation of language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 22056-22069, October 2023. 1, 2, 3, 9, 24, 37

Théophane Vallaeys, Mustafa Shukor, Matthieu Cord, and Jakob Verbeek. Improved baselines for data-efficient perceptual augmentation of llms. arXiv preprint arXiv:2403.13499 , 2024. 1, 2, 3, 9, 24, 26, 37

Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. arXiv preprint arXiv:2209.15162 , 2022. 1, 2, 24

Oscar Mañas, Pau Rodriguez, Saba Ahmadi, Aida Nematzadeh, Yash Goyal, and Aishwarya Agrawal. Mapl: Parameter-efficient adaptation of unimodal pre-trained models for visionlanguage few-shot prompting. arXiv preprint arXiv:2210.07179 , 2022. 1, 2, 3, 9, 24, 26, 37

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485 , 2023. 1

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500 , 2023. 1, 2, 24

Meghal Dani, Isabel Rio-Torto, Stephan Alaniz, and Zeynep Akata. Devil: Decoding vision features into language. arXiv preprint arXiv:2309.01617 , 2023. 1

Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734 , 2021. 1

Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. In NeurIPS 2022-36th Conference on Neural Information Processing Systems , 2022. 1, 24

Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442 , 2022. 2

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 , 2023. 2, 3, 24, 26

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966 , 2023. 2

Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478 , 2023. 2

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36, 2024. 3

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR) , 2021. 3, 26

Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In International Conference on Machine Learning (ICML) , volume 2, page 4, 2021. 3, 26

Yuan Gong, Yu-An Chung, and James Glass. AST: Audio Spectrogram Transformer. In Proc. Interspeech 2021 , pages 571-575, 2021. 3, 26

Kawin Ethayarajh. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 55-65, Hong Kong, China, November 2019. Association for Computational Linguistics. 4, 24

Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. Representation degeneration problem in training natural language generation models. In International Conference on Learning Representations , 2018. 4, 24

Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 9119-9130, 2020. 4, 24

Sara Rajaee and Mohammad Taher Pilehvar. How does fine-tuning affect the geometry of embedding space: A case study on isotropy. In Findings of the Association for Computational Linguistics: EMNLP 2021 , pages 3042-3049, 2021. 4, 24

Sara Rajaee and Mohammad Taher Pilehvar. A cluster-based approach for improving isotropy in contextual embedding space. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) , pages 575-584, 2021. 4, 24

William Rudman and Carsten Eickhoff. Stable anisotropic regularization. In The Twelfth International Conference on Learning Representations , 2024. 4

Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems , 35:17612-17625, 2022. 4, 24

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML) , pages 8748-8763. PMLR, 2021. 4, 26

Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint arXiv:2402.17762 , 2024. 5, 24, 30

nostalgebraist. interpreting gpt: the logit lens. Less Wrong , 2020. 5, 31

J Alammar. Ecco: An open source library for the explainability of transformer language models. In Heng Ji, Jong C. Park, and Rui Xia, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations , pages 249-257, Online, August 2021. Association for Computational Linguistics. 5, 31

Mor Geva, Avi Caciularu, Guy Dar, Paul Roit, Shoval Sadde, Micah Shlain, Bar Tamir, and Yoav Goldberg. Lm-debugger: An interactive tool for inspection and intervention in transformer-based language models. arXiv preprint arXiv:2204.12130 , 2022. 5, 31

Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112 , 2023. 5, 31

Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695 , 2023. 5, 31

Damien Teney, Armand Nicolicioiu, Valentin Hartmann, and Ehsan Abbasnejad. Neural redshift: Random networks are not random functions. arXiv preprint arXiv:2403.02241 , 2024. 8, 24

Micah Goldblum, Marc Finzi, Keefer Rowan, and Andrew Gordon Wilson. The no free lunch theorem, kolmogorov complexity, and the role of inductive biases in machine learning. arXiv preprint arXiv:2304.05366 , 2023. 8

Mustafa Shukor, Alexandre Rame, Corentin Dancette, and Matthieu Cord. Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-contextlearning. In The Twelfth International Conference on Learning Representations , 2024. 8, 25

Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 292-305, 2023. 8

Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 18135-18143, 2024. 8, 25

Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.14565 , 2023. 8

Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525 , 2023. 8, 25

Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.11411 , 2024. 8, 25

Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large visionlanguage models. arXiv preprint arXiv:2310.00754 , 2023. 8, 25

Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint arXiv:2310.16045 , 2023. 8

Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355 , 2023. 8, 25, 26, 36

Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 4035-4045, 2018. 8, 25, 36

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS) , 33:1877-1901, 2020. 24

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´ c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 , 2022. 24

Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. OpenFlamingo: An opensource framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390 , 2023. 24, 26

Hugo Laurençon, Lucile Saulnier, Leo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. OBELICS: An open web-scale filtered dataset of interleaved image-text documents. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2023. 24, 25, 26

Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. PaLI: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794 , 2022. 24

Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International Conference on Machine Learning , pages 23318-23340. PMLR, 2022. 24

Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. In The Eleventh International Conference on Learning Representations , 2022. 24

Shizhe Diao, Wangchunshu Zhou, Xinsong Zhang, and Jiawei Wang. Write and paint: Generative vision-language models are unified modal learners. In The Eleventh International Conference on Learning Representations , 2023. 24

Mustafa Shukor, Corentin Dancette, Alexandre Rame, and Matthieu Cord. UnIVAL: Unified model for image, video, audio and language tasks. Transactions on Machine Learning Research , 2023. 24, 25

David Mizrahi, Roman Bachmann, Oguzhan Fatih Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Zamir. 4m: Massively multimodal masked modeling. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. 24

Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. arXiv preprint arXiv:2312.17172 , 2023. 24

Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal generation. arXiv preprint arXiv:2301.13823 , 2023. 24

Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems (NeurIPS) , 34, 2021. 24

Mustafa Shukor, Guillaume Couairon, and Matthieu Cord. Efficient vision-language pretraining with visual concepts and hierarchical alignment. In 33rd British Machine Vision Conference (BMVC) , 2022. 24

Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Zicheng Liu, Michael Zeng, et al. An empirical study of training end-to-end vision-andlanguage transformers. arXiv preprint arXiv:2111.02387 , 2021. 24

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping languageimage pre-training for unified vision-language understanding and generation. arXiv preprint arXiv:2201.12086 , 2022. 24

Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 15638-15650, 2022. 24

Zhanyu Wang, Longyue Wang, Zhen Zhao, Minghao Wu, Chenyang Lyu, Huayang Li, Deng Cai, Luping Zhou, Shuming Shi, and Zhaopeng Tu. Gpt4video: A unified multimodal large language model for lnstruction-followed understanding and safety-aware generation. arXiv preprint arXiv:2311.16511 , 2023. 24

Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519 , 2023. 24

Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Jiao Qiao. LLaMA-Adapter: Efficient fine-tuning of language models with zero-init attention. 2303.16199, 2023. 24

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 , 2023. 24

Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning. arXiv preprint arXiv:2311.18799 , 2023. 24

Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886 , 2023. 24

Zhengqing Yuan, Zhaoxu Li, and Lichao Sun. Tinygpt-v: Efficient multimodal large language model via small backbones. arXiv preprint arXiv:2312.16862 , 2023. 24

Yichen Zhu, Minjie Zhu, Ning Liu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. Llava-phi: Efficient multi-modal assistant with small language model. arXiv preprint arXiv:2401.02330 , 2024. 24

Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402.14289 , 2024. 24

Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, En Yu, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Small language model meets with reinforced vision vocabulary. arXiv preprint arXiv:2401.12503 , 2024. 24

Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, and Jae-Joon Kim. Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks. arXiv preprint arXiv:2402.09025 , 2024. 24

Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403.17887 , 2024. 24

Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient llms at inference time. In International Conference on Machine Learning , pages 22137-22176. PMLR, 2023. 24

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems , 35:30318-30332, 2022. 24

Ibrahim Alabdulmohsin, Vinh Q Tran, and Mostafa Dehghani. Fractal patterns may unravel the intelligence in next-token prediction. arXiv preprint arXiv:2402.01825 , 2024. 24

Sarah Schwettmann, Neil Chowdhury, Samuel Klein, David Bau, and Antonio Torralba. Multimodal neurons in pretrained text-only transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2862-2867, 2023. 24

Haowen Pan, Yixin Cao, Xiaozhi Wang, and Xun Yang. Finding and editing multi-modal neurons in pre-trained transformer. arXiv preprint arXiv:2311.07470 , 2023. 24

Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lijuan Wang, Yezhou Yang, and Zicheng Liu. Compressing visual-linguistic model via knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1428-1438, 2021. 24

Kan Wu, Houwen Peng, Zhenghong Zhou, Bin Xiao, Mengchen Liu, Lu Yuan, Hong Xuan, Michael Valenzuela, Xi Stephen Chen, Xinggang Wang, et al. Tinyclip: Clip distillation via affinity mimicking and weight inheritance. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 21970-21980, 2023. 24

Zhe Gan, Yen-Chun Chen, Linjie Li, Tianlong Chen, Yu Cheng, Shuohang Wang, Jingjing Liu, Lijuan Wang, and Zicheng Liu. Playing lottery tickets with vision and language. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 652-660, 2022. 24

Jia Huei Tan, Chee Seng Chan, and Joon Huang Chuah. End-to-end supermask pruning: Learning to prune image captioning models. Pattern Recognition , 122:108366, 2022. 24

Dachuan Shi, Chaofan Tao, Ying Jin, Zhendong Yang, Chun Yuan, and Jiaqi Wang. UPop: Unified and progressive pruning for compressing vision-language transformers. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 31292-31311. PMLR, 23-29 Jul 2023. 24

Ali Furkan Biten, Lluis Gomez, and Dimosthenis Karatzas. Let there be a clock on the beach: Reducing object hallucination in image captioning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1381-1390, 2022. 25

Junyang Wang, Yiyang Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao, Haiyang Xu, Qinghao Ye, Ming Yan, Ji Zhang, Jihua Zhu, et al. Evaluation and analysis of hallucination in large vision-language models. arXiv preprint arXiv:2308.15126 , 2023. 25

Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. arXiv preprint arXiv:2311.16922 , 2023. 25

Wenliang Dai, Zihan Liu, Ziwei Ji, Dan Su, and Pascale Fung. Plausible may not be faithful: Probing object hallucination in vision-language pre-training. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pages 2136-2148, May 2023. 25

Bohan Zhai, Shijia Yang, Xiangchen Zhao, Chenfeng Xu, Sheng Shen, Dongdi Zhao, Kurt Keutzer, Manling Li, Tan Yan, and Xiangjun Fan. Halle-switch: Rethinking and controlling object existence hallucinations in large vision language models for detailed caption. arXiv preprint arXiv:2310.01779 , 2023. 25

Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.14565 , 2023. 25

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023. 25

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping languageimage pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 , 2023. 26

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36, 2024. 26

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning (ICML) , pages 10347-10357. PMLR, 2021. 26

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 16000-16009, 2022. 26

Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. X-clip: Endto-end multi-grained contrastive learning for video-text retrieval. In Proceedings of the 30th ACM International Conference on Multimedia , pages 638-647, 2022. 26

Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems , 35:10078-10093, 2022. 26

Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. In NeurIPS , 2022. 26

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6904-6913, 2017. 26

Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6700-6709, 2019. 26

Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2019. 26

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV) , pages 740-755. Springer, 2014. 26

Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM International Conference on Multimedia , MM '17, page 1645-1653, New York, NY, USA, 2017. Association for Computing Machinery. 26

Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016. 26

Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In NAACL-HLT , 2019. 26

Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 736-740. IEEE, 2020. 26

Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. Clotho-aqa: A crowdsourced dataset for audio question answering. In 2022 30th European Signal Processing Conference (EUSIPCO) , pages 1140-1144. IEEE, 2022. 26

Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seedbench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125 , 2023. 26

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8317-8326, 2019. 26

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: For findings Sec. 3 and implications Sec. 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Sec. 6 and more detailed in App. B

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Details to reproduce the experiments are in Sec. 2 and more detailed in App. D. Also the code will be made public.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The paper uses solely public datasets.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental details can be found in Sec. 2 and App. D

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper includes different statistical measures to report scores such as the average, median and the variance.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper mostly about analysis, but the considered models are trained by authors with details included in App. D. In addition we report the FLOPs during inference in the implications section Sec. 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: No violation for the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: App. C

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper cites all used assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Supplementary material

This supplementary material is organized as follows:

App. A: detailed related work.

App. B: detailed discussion about the work, limitations and other implications.

App. C: the broader impact of the work.

App. D: implementation details, including the trained models, datasets and metrics.

App. E: additional experiments analysing LLMs.

App. F: additional experiments for the implications.

A Detailed related work

Large multimodal models. Motivated by the success of large-scale training of LLMs [52, 1, 3, 53, 2, 5, 4], the multimodal community has embarked on a parallel journey, striving to develop larger and more powerful models capable of processing multiple modalities. Typical Large Multimodal Models (LMMs) are constructed either by building upon frozen LLMs [6, 54, 55] or by training them end-to-end after initialization [56, 7, 8]. These models have demonstrated success in numerous general and intricate multimodal tasks, achieving performance levels close to human capability. Another important line of research focuses on unified models, where a single model is designed to handle diverse tokenized modalities, such as image-text [57, 58, 59], or even beyond two modalities [60, 61, 62].

Efficient large multimodal models. Recently, to mitigate the training cost associated with training large multimodal models, efficient adaptation of unimodal models has emerged as a promising direction. Models like [11, 9, 17, 12, 10, 63] maintain LLMs frozen and train only a small subset of adaptation parameters for different multidmodal tasks. These approaches achieve competitive performance compared to end-to-end trained models [64, 65, 66, 67, 68] on image-text tasks and also on audio and video-text tasks [9, 69, 70, 10]. Beyond single-task tuning, many approaches do relatively light-weight pretraining and/or instruction tuning [71, 72, 14, 73, 19] and achieve good zeroshot generalizaton and instruction following abilities. To make these models more efficient, previous works have trained models with smaller LLMs showing competitive performance [74, 75, 76, 77, 78]

Analyzing LLMs. Previous research has highlighted the highly anisotropic nature of embeddings within language models, characterized by high cosine similarity [26, 27, 28, 29, 30]. Studies focusing on efficiency have shown that textual tokens exhibit small changes across layers [79, 80, 81]. Additionally, LLMs contains outlier features [82] and massive activations [34], which significantly influence model performance. Work by [83] suggests that LLMs may generalize due to the fractal structure of language. Moreover, [40] demonstrate that the building blocks of LLMs implicitly bias towards approximating both complex and simple functions. In the multimodal domain, [32] identify a modality gap in CLIP models attributable to the narrow cone effect. Previous studies have also explored neurons in LLMs that encode multimodal representations [84, 85].

Compression and pruning for multimodal models. Few works have targeted multimodal model compression, focusing mostly on image-text models. Notably, some works have concentrated on distilling knowledge from larger models through attention maps [86] or the affinity matrix in CLIP models [87]. Recent efforts have successfully applied the Lottery Ticket Hypothesis (LTH) to these models, optimizing both the model weights and masks jointly [88, 89]. A unified framework for structured pruning based on iterative training and adaptive sparsity allocation was proposed by [90]. It's worth noting that these approaches, initially designed for relatively small image-text models, encounter scalability challenges when applied to very large models.

Hallucinations in multimodal models. Hallucinations in multimodal models involve generating text that refers to objects not present in the input image [51, 50]. This pervasive issue affects a wide range of multimodal models, varying in architecture, training data, and scale [91, 60, 42, 6]. To gain deeper insights into this phenomenon, numerous studies have proposed evaluation benchmarks to quantify hallucinations across different dimensions [51, 50, 92, 44], shedding light on underlying causes. Specifically, co-occurrences and uncertainty [48], as well as visual uncertainty stemming from lower image resolution [93, 94, 95], have been identified as contributing factors. Additionally, it has been demonstrated that multimodal in-context learning exacerbates hallucinations [42]. To address this issue, various techniques have been proposed, including training on improved datasets [96, 44], aligning models with reinforcement learning [46, 47], refining training objectives [94], and employing post-training heuristics [48, 93]. Our study highlights the misalignment between internal representations of textual and perceptual tokens as a key cause of hallucinations.

B Discussion

Study across LLMs and setups. Our investigation primarily centers on Vicuna-v1.5 across singletask and multitask setups. We find that our conclusions remain consistent across various LLMs (e.g., OPT, Llama 2) and different settings (e.g., with and without pretraining, using different mapping modules), as detailed in the appendix. Extending our analysis to encompass other multimodal models, potentially with diverse architectures [6, 55], could offer additional valuable insights.

Study on larger models. While our work primarily focuses on frozen LLMs to provide insights relevant to future multimodal models, we also present results involving trained LLMs such as in LLaVA-1.5. These experiments yield observations akin to those with frozen LLM variants. However, the applicability of our experiments or the generalizability of our findings to larger LLMs (beyond 7B parameters), larger multimodal models [7, 6], or massively-trained multimodal foundation models like Gemini [97] or GPT4-V [4] remains an open question.

Remaining questions to understand LMMs. While we primarily investigate why and how LLMs generalize to multimodal inputs, and offer insights into issues such as object hallucinations, numerous unanswered questions persist. For instance, further exploration is needed to discern the encoded information in tokens and how LLMs extract information from visual tokens. Deeper inquiries are also required to address safety-related issues in large models, including the inability to abstain from answering, compositionality, and the precise adherence to user instructions [42].

Other implications. Our paper discusses several practical implications with potential benefits. Future extensions of our study could focus on specific aspects, such as enhancing model efficiency during training and inference by reducing redundant computations or model size. Additionally, addressing alignment with human preferences, such as faithfulness and safety, remains a significant challenge requiring further investigation. Our study may also inform model architecture design, such as developing mapping modules explicitly aligning multimodal tokens before entering the LLM.

C Broader impacts

The paper aims to enhance our comprehension of LLMs within the realm of multimodal inputs. We contend that a deeper understanding of these models can yield positive societal impacts, which we partially address in this study. For instance, our findings may contribute to mitigating the consumption of large models and their potential societal harms. Moreover, our work may inspire future research endeavors with various impacts, none of which we think must be specifically discussed here.

D Implementation details.

D.1 Perceptually augmented LLM baselines

D.1.1 ST setup

We train many models across different datasets that span image, video and audio-text modalities. We first devise powerful baselines based on 3 tenets: (a) having the smallest number of trainable

parameters, (b) general architecture that span or similar to many existing models and (3) good performance. To this end, and inspired by previous studies showing the effectiveness of using transformer-based mapping module [10, 12, 6, 54, 55, 98], we use light-weight transformer with learnable queries and self-attention to attend to perceptual tokens. This transformer operates in low dimension space ( i.e. , due to down/up projection layers and the the number of learnable query are limited to 10. We also favor a deeper architecture (5 blocks) compared to a wider one [10]. Our baselines are close to [10], but with significantly less trainable parameters. We train these baselines with different LLMs: OPT-6.7B [2], Llama 2-7B [5] and Vicuna-v1.5-7B [99] and different encoders for: image (ViT [23, 100], CLIP[33], MAE[101]), video (TimesFormer[24], X-CLIP[102], VideoMAE[103]) and audio (AST[25], AudioMAE[104]).

To train these baselines, we use AdamW optimizer with a learning rate of 2e-4 that decreases with a cosine annealing scheduler to a minimium of 1e-5. We train with a total batch size of 16 for captioning and 64 for VQA datasets. The number of epochs is set to 20 to ensure that all models converged, though most of these models converge after only couple of epochs. We select the best checkpoint for evaluation. For example the model for image captioning converged after ∼ 4 epochs. All models are trained on 8 V100 GPUs and the training time depends on the task, e.g. , for the large VQAv2 dataset each epoch takes ∼ 30 mins, for other smaller datasets it takes less time, e.g. , ∼ 10 mins for Audiocaps and MSVD-QA. Unless specified otherwise, we fix the hyperparameters for all baselines to isolate the variations that could results from this.

We refer to all text-aligned models as CLIP, trained for classification as ViT, and self-supervised with MAE objective as MAE.

D.1.2 MTsetup

To study the impact of different factors ( e.g. pretraining, mapping module) on the internal representations ( e.g. implicit alignment), we devise different variants of the LLaV A-1.5 [19] model that differ from the original model as follows: LLaVA-1.5-2 (LLM kept frozen), LLaVA-1.5-3 (LLM kept frozen, without pretraining) and LLaVA-1.5-4 (LLM kept frozen, without pretraining and with transformer mapping module similar. The latter is very similar to the models used in the ST setup, which ensure comparable observatoins. All these models are based on the Vicuna-v1.5-7B LLM.

We follow the same training setup of LLaVA-1.5 [19], including the training data, steps and hyperparameters.

For analysis in the paper ( e.g. Sec. 3), we focus on Vicuna-v1.5 as it is shared by both setups. For the ST setup, we use unimodal encoders, such as ViT, TimeSformer and AST that are not aligned with text.

D.2 Datasets and metrics

ST setup. We consider a wide range of public multimodal datasets that cover 2 representative tasks: captioning and question-answering (QA) across image (VQAv2 [105], GQA [106], OKVQA [107], COCO caption [108]), video (MSVD, MSRVTQA [109], MSRVTT [110]), audio (Audiocaps [111], Clotho [112], Clotho-AQA [113]) and language tasks. For QA datasets we report the accuracy (in open-ended generation setup with exact match), and for captioning we report the CIDEr metric.

MTSetup. We also evaluate the MT setup on recent datasets such as SEED [114], TextVQA [115] and POPE [50].

E LLMs generalize to multimodal inputs: additional experiments

E.1 Tokens evolution across layers

Different LLaVA-1.5 variants. In Fig. 13, we illustrate the differences between perceptual and textual tokens across different LLaVA-1.5 variants. Comparing the two variants with multimodal pretraining, we observe higher cross-modal similarity in LLaV A-1.5 compared to LLaV A-1.5-2, which freezes the LLM. This suggests that training the LLM enhances alignment between representations. For models without pretraining, we note that using an MLP (LLaVA-1.5-3) to connect both models yields better results than using transformer-based pooling (LLaV A-1.5-4), potentially explaining the

Embedding similarity

Embedding similarity

P vs P

1.0

P vs T

0.8

Cosine Similarity

P vs P

P vs T

0.2

P vs P

P vs P

P vs T

P vs T

0.0

Layers

Layers

0

Layers

0

25

5

10

Layers

15

20

25

30

1.0

0.8

0.6

0.4

-P (n 1)

Layers

*

-T(n8)

T(n1)

-P (n 1)

-P (n 8)

Layers

+

-T (n 8)

-P (n 8)

-T (n 1)

-P (n 1)

Layers

"*

-T(n 1)

-P (n 8)

-T(n 8)

-P (n 1)

0.0

0.2

0

5

10

Layers

15

20

*

25

-T (n 1)

-P (n 8)

-T (n 8)

30

s

-P (n 1)

-P (n 1)

+

-P (n 1)

+

-P (n 1)

-T (n 1)

-T (n 1)

** -T (n 1)

*=-T (n 1)

Layers

Layers

Layers

Layers

PvsP

P vs P

-P(n 1)

14

-P vs P

P vs T

P vs T

+-

+-

T (n 1)

P (n 8)

12

-P vs T

T (n 8)

10

8

6

2

Layers

20

Layers

15

Layers

15

5

10

Layers

15

20

25

+- P (n 1)

T (n 1)

P (n 8)

T (n 8)

-P(n 1)

T (n 8)

T (n 1)

P (n 8)

-P(n 1)

T (n 1)

P (n 8)

T (n 8)

12

10

--P (n 1)

-+-.

h-.

T (n 1)

P (n 8)

T (n 8)

Distance

Dis

4

2

Layers

Layers

5

10

Layers

15

20

25

superior scores. Utilizing all visual tokens appears to bolster alignment with textual tokens, with pretraining further enhancing this alignment. Notably, vocabulary distributions undergo significant changes in middle layers, particularly for textual tokens. Similar observations hold across different variants, indicating that our findings generalize to broader setups and that training the LLM does not substantially alter token behavior.

Different similarity measures for cross-modal alignment. In this section, we compare the following similarity measures to compute the similarity between perceptual ( P = [ p 1 , ..., p N p ] ) and

P vs P

P vs T

Embedding similarity

P vs P

P vs T

Embedding similarity

P vs P

P vs T

9.10

Embedding similerity

d SA d

P vs T

9225

9.200

Embedding similarity

P vs P

P vs T

Cosine Similarity

9.20

Aueus

0.15

9.150

9.04

auso

.125

.10

0.82

0.875

Embedding similarity

Layers

15

Embeding similarty.

19

Layers

15

EmbedMing similarity

Layers

15

10

Enbedding similarty.

Layers

15

Embedding similarity

10

Layers

15

1.0

P vs P

1.0

Pvs T

Cosine Similarity

0.8

Cosine Similarity

Cosine Similarity

Cosine Similarity

0.8

0.4

P vs P

P vs T

P vs P

P vs T

Cosine

P vs P

P vs T

0.4

0.6

0.2

0.2

P vs P

0.0

5

10

Layers

15

20

25

00

Layers

15.

10

Layers

15

10

Layers

15

0.0

10

Layers

15

20

25

P vs T

00

Embedding similarity

Embedding similarity

Embedding similarity

1.0

0.5

-PvsP

1.0

9999499959

Vvs I

Pvs T

Vvs A

0.8

0.4

TvsT

0.8

IvsA

Cosine Similarity

CosineSimilarity

Cosine Similarity

Vvs T

0.6

P vs P

0.3

0.6

I vs T

PvsT

A vs T

0.4

T vs T

0.2

0.4

V-T vs I-T

V-T vs A-T

I-T vs A-T

0.2

0.1

0.2

0.0

0

5

10

Layers

15

20

25

30

0°0

10

Layers

15

20

25

30

0.0

0

5

10

Layers

15

20

25

30

Embedding similarity

Embedding similarity

Embedding similarity

1.0

0.8

1.0

0.8

Cosine Similarity

0.8

Cosine Similarity

0.6

Similarity

0.6

PvsP

Pvs P

0.6

Pvs P

0.4

P vs T

PvsT

VS T

T vs T

T vs T

0.4

0.2

0.2

0.2

10

Layers

15

20

25

10

Layers

15

20

25

30

0.0

10

Layers

15

20

25

textual ( T = [ t 1 , ..., t N t ] ) tokens:

Where [ N p ] = { 1 , ..., N p } and [ N t ] = { 1 , ..., N t } and Med is the median operation.

Fig. 14 shows the inter ( P vs T ) and intra ( P vs P ) similarity. According to all measures, except AvgSim and MinSim, we have similar observations: increasing inter similarity and higher intra similarity that increases in last layers. For MinSim and AvgSim for the ST setup, we do not see such observations, indicating that not all perceptual tokens are, or should be aligned to text.

Fig. 15 and Fig. 16 show the measures comparison for the narrow cone experiments. Interestingly, the narrow cone effect is less seen when looking at the median of the token similarities (MedSim), indicating that this effect is not driven by all tokens, and at the token level the representation is not always anisotropic.

In spite of having similar observations between several measures, we focus on SimAvg, as it is much faster to compute, especially whey there is large number of tokens (as in LLaV A-1.5).

Embedding similarity

Embedding similarity

Embedding similarity

1.0

Vvs I

0.5

V vs I

1.0

VvsI

VvsA

V vs A

VvsA

Cosine Similarity

0.8

I vS A

V vs T

CosineSimilarity

0.4

I vS A

Vvs T

Cosine Similarity

0.8

I vS A

V vs T

0.6

I vs T

A vs T

V-T vs I-T

0.3

I vs T

A vs T

V-T vs I-T

0.6

I vs T

A vs T

0.4

I-T vs A-T

V-T vs A-T

0.2

I-Tvs A-T

V-T vS A-T

0.4

V-T vs I-T

V-T vs A-T

I-T vS A-T

Z'0

0.1

0.2

0.0

0

5

10

Layers

15

20

25

30

0.0

5

10

Layers

15

20

25

30

0.0

0

5

10

Layers

15

20

25

30

0.8

V-P (n 1)

Cosine Similarity

0.6

*- I-T (n 1)

+-I-P (n 1)

+- I-P (n 8)

+- V-P (n 8)

+- VT (n 8)

VP (n 1)

V-T (n 1)

0.4

I-P(n1)

1-T(n-1)

I.T(n8)

V-P (n 8)

1-P(n8)

V-T (n 1)

V-T (n 8)

I-T(n1)

1-P(n8)

V-T (n 1)

V-P (n 8)

V-T (n 8)

I-P (n 1)

V-P (n 1)

IP(n1)

T（m）

V-T-(n 1)

V-P(n8)

V-T(n8)

I-P(n8)

V-P (n 1)

A-P(n 1)

*- I-T (n 8)

A-T (n 1)

A-P (n 8)

A-T (n 8)

0.2

0.0

A-P (n 8)

A-P (n 1)

A-T (n 1)

A-T (n 8)

IT(n8)

A-P (n 1)

A-T (n 8)

A-T (n 1)

A-P (n 8)

1-T (n.8)

A-P (n 1)

A-P(n8)

A-T (n 1)

A-T (n 8)

Layers

13

0

5

10

Layers

15

20

25

30

Layers

13

Layers

15

1.0

1.0

1.0

0.8

0.8

8°0

0.6

Similarity

0.2

0.0

10

Layers

15

20

+-.

25

-P (n 1)

-T (n 1)

-P (n 8)

-T (n 8)

30

0.2

0.0

0

5

10

Layers

15

20

25

-P-(n-1)

-T (n 1)

-P (n 8)

-T (n 8)

30

0.2

0.0

10

Layers

15

20

25

+-

-T (n 1)

-P-(n-1)--

-P (n 8)

-T (n 8)

Layers

15

-P（n1)

-T (n 1)

-P (n 8)

-T (n 8)

Different similarity measures for the similarity across consecutive layers. we compare the following similarity measures to compute the token similarity between consecutive blocks ( e.g. between tokens at block a X a = x a 1 , ..., x a N and b X b = x b 1 , ..., x b N ):

Where [ N ] = { 1 , ..., N } and Med is the median operation.

Fig. 17 shows similar observations across all different measures when each token is compared with the token at the same position in different layers. However, when taking the median of similarities (MedSim) across all tokens, this similarity is siginficantly smaller, especially for the ST setup. This reveals that tokens can be very different within the same modality or example.

1400

1200

1000

ULION

S

n

800

600

V-P (n 1)

V-P(n 1)

V-P (n 1)

400

V-T (n 1)

V-P (nI)

Layers

3

1-T(n1)

A-T (n 1)

VT (n 1)

A-P(n1)

9

I-P(n1)

ATT

I-T (n 1)

V-T(n1)

A-P (n 1)

200

5

10

A-T(F1)*

Layers

15

I-P (n 1)

I-T (n 1)

A-P (n 1)

20

25

30

Layers

15

1-P (n 1)

-T(1）*

A-T (n 1)

V-T (n 1)

A-P (n 1)

120

100

N S

S

80

Tokens Norm

60

40

20

Layers

SL

-P (n 1)

-T (n 1)

Layers

15

-T (n 1)

-P (n 1)

5

10

Layers

15

20

25

-T (n 1)

-P (n 1)

30

Layers

-T (n 1)

-P (n 1)

Tokens evolution for different modalities. Fig. 19, shows that textual and multimodal tokens evolve differently inside LLMs.

1400

1.0

120

0.8

1200

0.8

100

Cosine Similarity

0.6

0.4

I-P(n1)

I-T(n1)

1-P(n8)

I-T(n8)

V-P (n 1)

V-T (n 1)

V-P (n 8)

V-T (n 8)

Tokens Norm

1000

400

800

600

VP(n 1)

V-T (n 1)

Cosine Similarity

0.6

0.4

Tokens Norm

60

80

40

0.2

A-T (n 1)

A-P (n 1)

200

I-P (n 1)

I-T (n 1)

0.2

-T (n 1)

-P (n 1)

20

A-P (n 8)

A-P (n 1)

-P (n 8)

-P (n 1)

0.0

0

5

10

Layers

A-T (n 8)

15

25

30

0

5

10

A-T(nT)*

Layers

15

20

25

30

0.0

0

5

10

Layers

15

20

25

-T (n 8)

30

0

5

10

Layers

15

20

25

-T (n 1)

30

E.2 Token norms across layers

Massive token norms. In this section we highlight the presence of tokens with massive norms, this becomes clearer when looking at different norm measures. We compare the following measures to compute the token L2 norms across blocks ( e.g. X = x 1 , ..., x N ):

Where [ N ] = { 1 , ..., N } and Med is the median operation and M is the total number of elements in the tensor. For the ST setup, Fig. 18 shows a very high token norms when looking at NormAvg and MaxNorm, compared to MinNorm and MedianNorm. These massive norms are present for both textual and perceptual tokens, and they are larger for perceptual ones. When looking closely, we find that these tokens correspond to start or split tokens as seen in [34]. For the MT setup, we notice that these massive tokens presents mainly in the system message, which we remove for our study as it is common for all examples. Interestingly the perceptual tokens for the MT setup do not seem to have massive norms.

4000

1200

2500

14000

1000

12000

Cosine Similarity

3000

2000

V-T (n 1)ep-1

V-P (n 1) ep -1

V.P（n1）ep0

Cosine Similarity

800

600

V-T（n 1）ep-1

V.P(n 1)ep-1

V.P (n 1) ep 0

Cosine Similarit

2000

1500

V-T(n 1)ep-1

V-P (n 1) ep 0

V-P (n 1) ep-1

Cosine Similarity

10000

8000

6000

V-P (n 1) ep -1

V-T (n 1)ep-1

V.P(n1)ep0

1000

V-T （n 1)ep 1

V-T (n 1) ep 0

V.P (n 1) ep 1

V.P(n 1)ep2

200

400

V-T(n 1)ep.0

V-P(n 1)ep 1

V.T (n 1)ep 1

VP（n1）ep2

1000

500

V-P（n 1)ep1

V-T (n 1)ep 1

V-T (n 1)ep 0

V-P (n 1) ep 2

4000

2000

V-T (n 1) ep 0

V-P (n 1)ep 1

V-T(n1)ep=

V.P(n 1)ep 2-

10

Layers

15

20

V-T (n 1)ep 2

25

30

10

Layers

15

VT(n-1)ep2

亿

30

10

Layers

15

20

VT (n1) ep2

25

30

10

Layers

15

V-T（n 1)ep2

25

30

Increasing token norm during training. We try to investigate why we have very high perceptual token norms. To this end, we compute the norm across different epochs. Fig. 20 shows that during training of the mapping module, the norms increase significantly.

E.3 Vocabulary distribution

For each token, we use the LLM unembedding ( i.e. LLM head) to decode the latent representation to a probability distribution over the vocabulary. This approach have shown to work well for LLMs at different layers, not just the last one [35, 36, 37, 38]. In Fig. 21, we show the histogram of this distribution at the first LLM layer for both textual and perceptual tokens, the KL-distance between the 2 distributions, KL-distance between consecutive layers and the entropy. Here we report additional results for the LLaV A-1.5 baseline showing similar observations to those of ST reported in the main paper.

14

12

V (n 1)

V (n 1)

17.5

+=:

V-P vs T

V-P vs P

10

Distance

10

8

I (n 1)

I (n 1)

A (n 1)

15.0

12.5

A-P vs P

I-P vs P

I-P vs T

Entropy

8

6

Vocabulary

6

4

N

0

5

10

Layers

15

20

I (n 8)

A (n 1)

V(n8)

V(n8)

I (n 8)

A (n 8)

A (n 8)

25

Dist

7.5

5.0

2.5

0

A-P vs T

5

10

Layers

15

20

25

30

2

0

4

0

--.

=@=1

V-P

V-T

I-P

I-T

A-P

A-T

5

10

Layers

15

20

25

30

T

P

12

10

-+-.

T (n 1)

P (n 1)

P (n 8)

14

12

"=#=.

-P vs P

-P vs T

10

8

T (n 8)

10

Distance

Distance

8

6

6

4

4

4

2

2

2

-P

Vocabulary

0

5

10

Layers

15

20

25

0

0

5

10

Layers

15

20

25

30

0

0

-T

5

10

15

20

25

30

Layers

E.4 Similar activated weights by different modalities

Experimental setup. In this section, we analyse the subnetworks activated by different multimodal inputs. We use the Wanda score [39] to extracted these subnetworks or pruning masks, then compute the IoU. For multimodal datasets we consider only the perceptual tokens, for example the visual tokens without the questions for VQAv2. We also use the text in these datasets as a source for textual tokens ( e.g. , COCO-text consider only the captions in the COCO dataset).

Different LLaVA-1.5 variants. In Fig. 23, we show the overlap between the weights activated by different modalities. Across different LLaV A-1.5 variants, we find similar observations: high overlap between perceptual and textual activated weights ( e.g. 0.6 IoU), which is less than the overlap between weights acrtivated by the same modality ( e.g. 0.95 for perceptual tokens and 0.87 for textual

1.00

1.00

AudiocapsText-

1.00

0.84

0.90

0.71

0.70

0.72

0.67

0.71

0.71

0.71

0.72

(I) A-OKVQA-

1.00

0.99

0.99

0.99

0.63

0.64

0.69

0.70

COCO--Text-

0.84

1.00

0.83

0.69

0.68

0.69

0.65

0.69

0.69

0.68

0.70

0.95

0.95

(I)COCO

0.99

1.00

0.99

0.98

0.63

0.64

0.69

0.70

MSRVTT--Text

0.90

0.83

1.00

0.71

0.70

0.72

0.67

0.70

0.71

0.70

0.72

0.90

0.90

(A) Audiocaps

0.71

0.69

0.71

1.00

0.70

0.74

0.67

0.69

0.73

0.71

0.71

(I) GQA-

0.99

0.99

1.00

0.99

0.63

0.64

0.69

0.70

(A) Clotho-AQA

0.70

0.68

0.70

0.70

1.00

0.70

0.66

0.69

0.70

0.71

0.69

0.85

(I)VQAv2-

0.99

0.98

660

1.00

0.63

0.64

0.69

0.70

0.85

0000(1)

0.72

0.69

0.72

0.74

0.70

1.00

0.67

0.69

0.73

0.70

0.73

0.80

(T) A-OKVQA-Text-

0.63

0.63

0.63

0.63

1.00

0.71

0.66

0.71

0.80

(I)GQA

0.67

0.65

0.67

0.67

0.66

0.67

1.00

0.65

0.69

0.67

0.66

OAO())

0.71

0.69

0.70

0.69

0.69

0.69

0.65

1.00

0.68

0.69

0.71

(T) COCO-Text

0.64

0.64

0.64

0.64

0.71

1.00

0.66

0.69

0.75

0.75

(I)VQA

0.71

0.69

0.71

0.73

0.70

0.73

0.69

0.68

1.00

0.70

0.70

(T) GQA-Text-

0.69

0.69

0.69

0.69

0.66

0.66

1.00

0.87

0.70

(V) MSRVTQA

0.71

0.68

0.70

0.71

0.71

0.70

0.67

0.69

0.70

1.00

0.70

0.70

(V) MSRVTT

0.72

0.70

0.72

0.71

0.69

0.73

0.66

0.71

0.70

0.70

1.00

(T) VQAv2-Text-

0.70

0.70

0.70

0.70

0.71

0.69

0.87

1.00

0.65

Audiocaps--Text

MSRVTTText-

()

()COCO

V09 (1)

OAXO()

VOA (I)

(

(V) MSRVTT.

0.65

())

(I)COCO

00 (1)

x-V-V ()

-0000 ())

x-0 (1)

-ZAVA ()

LLaVA-1.5

LLaVA-1.5 (fr. LLM)

LLaVA-1.5 (fr. LLM, w/o PT)

LLaVA-1.5

(fr.

LLM, w/o

PT,

Q-

Former.)

LLaVA-1.5

LLaVA-1.5 (fr. LLM)

LLaVA-1.5 (fr. LLM, w/o PT)

LLaVA-1.5

(fr.

LLM, w/o

PT,

Q-

Former.)

.00

.00

(I)A-0KVQA

1.00

0.95

0.95

0.55

0.66

0.67

0.68

0.69

(I)A-0KVQA

1.00

0.94

0.94

0.94

0.67

0.67

0.68

0.79

(I):A-0KVQA

1.00

0.98

0.98

0.98

0.68

0.69

0.73

0.74

vb/olov(0

1.93

0.99

0.99

9.99

9.63

0.64

0.69

0.95

1.00

0.95

0.95

0.66

0.67

0.68

0.69

0.95

0)COCo

0.94

1.00

0.94

0.94

0.67

0.67

0.68

0.79

0.95

()COCo

0.9

1.00

.5

0.598

0.68

0.69

0.73

0.74

0.95

$6'0

0)C0CO

0.99

1.09

0.99

0.98

9.63

0.64

0.69

(I)-GQA

0.95

9.95

1.00

0.95

0.66

0.67

0.68

0.69

0.90

(I)GQA

0.94

0.94

1.00

0.94

0.67

0.67

0.68

0.79

0.90

(I)GQA

0.99

0.98

1.00

0.98

0.68

0.69

0.73

0.74

0.90

0.90

0.99

6.99

1.00

0.99

9.63

0.64

0.69

0.70

0.95

4.95

0.95

1.00

0.66

0.67

0.68

0.69

0.85

0.94

0.94

0.94

1.00

0.67

0.67

0.68

0.79

0.85

0.98

0.98

0.96

1.00

9.68

0.69

0.73

0.74

0.85

()VQt2

0.99

0.98

0.99

1.00

0.63

0.64

0.69

0.7

0.85

(M) A0IDVQATet

0.66

9.66

0.66

0.66

1.00

0.71

0.66

12'0

0.80

M)A0IVQATct

c90

49'0

0.67

0.67

1.00

0.71

0.66

120

0.80

m)A-0IVQATet

0.68

0.68

0.68

0.68

1.00

0.71

0.66

0.80

(T).A-0KVQATest

0.63

0.63

0.63

0.63

1.00

0.71

0.66

0.71

0.80

(M) C000-Tat

0.67

0.67

0.67

0.67

0.71

1.90

0.66

0.69

0.75

a[-0000 (μ

0.67

0.67

0.67

0.67

0.71

1.00

0.66

0.69

0.75

[-0000 ()

0.69

0.69

0.69

0.69

0.71

1.00

0.66

0.69

0.75

m) 0000-Test

0.64

0.64

0.64

9.64

9.71

1.00

0.66

0.69

0.75

(T) GQ/Tet

0.68

0.68

0.66

0.68

0.66

0.66

1.00

0.87

0.79

(T) GQ/-Test

0.68

0.68

0.66

0.68

0.66

0.66

1.00

0.87

0.79

(T) GQA-Test

0.73

8.73

8.73

0.73

0.66

0.66

1.00

0.79

mT) GQATest

0.69

0.69

0.69

0.69

0.66

0.66

1.00

0.83

8c'0

(T) VQμ2-Tet

0.69

0.69

1000

0.69

0.69

0.71

0.69

0.87

1.00

(T) VQW3-Text

0.70

0.70

0.70

00

0.70

0.71

0.69

0.87

1.00

(M) VQw2·Test

0.74

0.74

0.74

0.74

0.71

0.69

0.87

1.00

(T) VQ/w2·Tcst

0.709

VOASO-Y

8.70

6.70

0.70

0.71

0.69

0.87

1.00

0.65

[L]

E

E

E

0.775

All layer types (Sparsity: 0.5)

I vs T

All layer types (Sparsity: 0.5)

I vs T

0.775

All layer types (Sparsity: 0.5)

I vs T

All layer types (Sparsity: 0.5)

0.750

0.750

0.750

0.68

I vs T

Masks IoU

0.725

0.650

0.700

0.675

IoU

0.725

0.700

0.675

0.650

0.725

0.675

Masks I

0.62

0.66

0.64

0.650

0.625

0.625

0.60

0.600

0.600

0.625

0.58

0.575

0.575

0.600

0.56

Transformer Layer

10

15

Transformer Layer

10

15

29

30

0.575

D

Transformer Layer

10

15

20

25

30

0.54

D

5

10

15

20

25

30

Transformer Layer

ones). We also notice a significant decrease in IoU at the first layers, which might reveal that the first layers encode general features that are shared across modalities.

Different LLMs for the ST setup. In Fig. 24, we show the IoUs of activated weights for different frozen LLMs ( i.e. , OPT, Llama 2 and Vicuna-v1.5) for the ST setup. We notice similar observations across LLMs, and relatively higher overlap for OPT. We notice similar observation compared to the MT setup, where we have a significant decrease in the IoU in first layers. For this setup, it is clearer that the overlap increase for deeper layers.

Different sparsity levels. In Fig. 25, we study how the overlap between activated weights changes with the size of the extracted subnetworks. This size depends on the sparsity of the final model. We

OPT (s=0.3)

Llama 2 (s=0.5)

apsText

1.00

0.87

0.87

0.76

0.780.78

0.740.76

0.78

0.760.75

1.00

capsText

1.00

0.89

0.91

0.720.710.730.710.69

1.00

0.84

0.90

0.710.70

0.72

0.67

0.710.710.71

0.72

COCO~Text-

0.87

1.00

0.83

0.74

0.750.75

0.75

0.730.73

0.95

COCO~Text

1.00

0.88

0.71

0.69

0.71

0.69

0.67

0.710.700.70

S60

COCOText

0.84

1.00

0.83

0.69

0.68

0.69

0.65

0.69

0.69

0.68

0.70

96°0

MSRVTTText

80

1.00

0.790.79

0.75

0.77

0.79

0.77

MSRVTTText

0.91

1.00

0.72

0.71

0.73

0.70

0.68

0.72

20.720.71

MSRVTT-Text

0.90

1.00

02010

0.72

0.67

0.70

0.71

0L0

0.72

sdeoo(pny (V)

0.76

0.74

0.77

0.790.78

0.74

0.77

0.78

0.78

0.77

0.90

(A) Audiocaps

0.72

0.710.72

1.00

0.700.71

0.69

0.67

0.730.720.71

0.90

(A) Audiocaps

0.71

0.69

0.71

1.00

0.70

0.74

0.67

0.69

0.73

0.71

0.71

06°0

(A) Clotho-AQA

0.78

0.75

0.76

0.82

0.80

0.77

(A) Clotho-AQA

0.71

0.710.70

1.00

0.740.720.750.74

0.710.72

0.85

(A) Clotho-AQA

0.70

0.68

0.70

0.70

1.00

0.70

0.66

0.69

0.70

0.71

0.69

S8'0

(1)coCo

0.78

0.75

0.790.78

0.81

0.75

0.76

0303 (1)

0.73

0.71

0.730.71

0.74

0.70

0.72

0.77

0.730.73

(1) COCO

0.72

0.69

0.72

0.74

1.00

0.67

0.69

0.73

06°0

0.73

0.85

(I) GQA

0.74

0.72

0.750.74

0.76

0.740.73

(I)GQA

0.71

0.69

0.70

0.69

0.72

0.71

0.69

0.68

0.80

(D) GQA

0.67

0.65

0.670.670.66

1.00

0.65

0.69

(I) OKVQA

0.76

0.73

0.770.77

0.75

08'0

(I) OKVQA

0.69

0.67

0.67

0.69

()OKVQA

0.71

0.69

0.70

0.690.69

0.69

0.65

S°0

(1) VQA

0.78

0.75

0.79

0.82

(I) VQA

0.73

0.71

0.73

0.75

(I) VQA

0.71

0.69

0.71

0.73

0.70

0.73

(V) MSRVTQA

0.76

0.73

0.77

0.75

(V) MSRVTQA

0.72

0.70

0.71

0.68

0.70

0.710.71

0.70

0.67

69'0

(V) MSVD

0.75

(V) MSVI

0.72

0.72

0.71

0.69

All layer types (Sparsity: 0.5)

All layer types (Sparsity: 0.5)

All layer types (Sparsity: 0.5)

0.775

0.775

-TvsI

TvsV

I vs T

V vs T

0.750

T vs A

A vs T

0.725

IoU

Masks

099'0

0.650

0.625

T vs I

0.625

T vs V

0.600

15

15

10

15

20

Transformer Layer

Transformer Layer

Transformer Layer

All layer types (Sparsity: 0.3)

T vs I

0.775

TvsV

T vs A

0.750

MasksIoU

0.725

0.700

0.675

0.650

0.625

5

TransformerLayer

10

15

25

All layer types (Sparsity: 0.3)

0.88

TvsV

T vs A

0.86

MasksIoU

0.84

0.82

08°0

0.78

0

10

15

20

25

30

All layer types (Sparsity: 0.5)

0.775

0.750

0.725

MasksIoU

0.700

0.675

0.650

0.625

T vs I

T vsV

0.600

TvSA

TransformerLayer

10

15

20

25

30

All layer types (Sparsity: 0.5)

0.775

+

TvsI

T vs V

0.750

T vS A

0.725

Masks IoU

0.700

0.675

0.650

0.625

0.600

0

TransformerLayer

10

15

20

25

30

TransformerLayer

notice that the lower the sparsity, the higher the overlap, revealing that higher sparsity allows to extract more modality-specific activated weights.

Pruning weights by streaming different modalities. In addition to the IoU, we also compare the differences between the task performance of activated weights. Fro each dataset, we give to the model either the perceptual prompt, the textual prompt or both, knowing that each example in the dataset consists of a perceptual prompt followed by the textual one. Fig. 26 shows slight differences in overall performance when the LLM is pruned by different modalities, with the best performance is

ImageCaption(CLIP)

120

Video Caption (CLIP)

ImageCaption(ViT)

Task performance

VideoCaption(ViT)

100

ImageCaption(MAE)

80

Video Caption(MAE)

AudioCaption(MAE)

60

40

20

0

PTP+TPTP+TPTP+TPTP+TPTP+TPTP+TPTP+TPTP+T

Input for pruning

by considering both the textual and the perceptual tokens. This also show the high overlap between the weights used to activate textual and perceptual tokens.

Transfer of pruning masks to other tasks and modalities. To further highlight the overlap between weights, we report the performance when the model is pruned given data from other modalities or datasets. We notice similar observations across LLMs, such Llama 2 Fig. 29 and Vicuna-v1.5 Fig. 30. Interestingly, we find the similar overlap with the unsupervised MAE encoders Fig. 27 compared to text aligned ones Fig. 28. We notice a performance degradation when the model is pruned at high sparsity levels (0.5).

VQAv2COCO

119.51

Modality: image (MAE,Sparsity: 0.3)

58.85

49.17

25.63

100

Modality: video (MAE, Sparsity: 0.3)

45.77

35.90

34.53

46

44

Audiocaps

Modality: audio (MAE,Sparsity:0.3)

64.22

29.37

50.94

60

55

Text MSRVTT COCOAudiocaps

61.65

(MAE, Sparsity: 0.3)

114.37

42.99

110

100

118.36

118.42

59.51

59.06

52.51

52.22

27.56

26.61

80

60

46.36

35.90

34.65

42

40

Clotho

61.30

28.51

50.38

50 4540

63.99

62.34

118.74

118.10

45.83

44.84

06

80

70

118.49

59.16

52.49

27.44

40

45.01

35.90

34.65

90

62.91

28.69

52.06

35

59.95

117.73

45.22

60

OS

COCO

VQAv2

GQA

OKVQA

MSRVTT

MSRVTTQA

MSVD

Audiocaps

Clotho

ClothoAQA

30

Audiocaps

COCO

MSRVTT

VQAv2COCO

115.37

Modality: image (MAE, Sparsity: 0.5)

55.65

36.85

18.54

100

Modality: video (MAE, Sparsity: 0.5)

42.32

30.20

31.55

42

40

sdesopny

Modality: audio (MAE, Sparsity: 0.5)

54.42

27.43

45.97

Text MSRVTT COCOAudiocaps

54.78

(MAE, Sparsity:0.5)

106.84

34.30

110

100

111.86

110.35

53.63

53.74

45.77

46.76

19.27

19.81

80

60

41.56

32.00

32.52

80

90

Clotho

54.49

28.27

49.25

32

41.59

51.38

113.88

112.59

39.66

42.20

06

80

OL

113.48

COCO

52.47

VQAv2

46.43

GQA

21.58

OKVQA

20

40

MSRVTT

38.23

MSRVTTQA

31.70

32.52

MSVD

34

32

Audiocaps

52.53

20.76

Clotho

ClothoAQA

49.81

30

25

Audiocaps

48.81

58.03

COCO

MSRVTT

37.63

60

50

40

Modality-specific subnetworks? The experiments suggest a high overlap between weights activated by different modalities. For instance, the pruning masks similarity (IoU) between datasets within the same modality is on par with those across modalities for the ST setup. However, this does not exclude the possibility of finding weights that are generally activated when seeing a particular modality, even if there are small amount of them. The overlap is smaller with LLaVA-1.5 variants making this possibility more likely for large scale multitask models.

E.5 Implicit multimodal alignment effect

Alignment inside each LLM Block. Fig. 32 reports the tokens similarity and norms for both LLaVA-1.5 and Vicuna-v1.5.

VQAv2COCO

131.40

Modality:image (CLIP, Sparsity:0.3)

63.93

52.99

31.71

120

Modality:video (CLIP, Sparsity:0.3)

57.26

38.30

39.93

55.0

Text MSRVTT COCOAudiocaps

125.00

(CLIP,Sparsity:0.3)

54.36

130

120

130.70

64.67

53.86

30.51

100

52.5

50.0

131.58

56.33

110

100

OKVQA

129.56

131.06

64.45

64.33

53.55

53.73

31.46

31.42

80

60

40

MSVD

57.10

57.20

38.00

38.10

39.14

39.44

47.5

45.0

42.5

40.0

129.35

129.94

56.66

54.79

90

80

70

60

COCO

VQAv2

GQA

OKVQA

MSRVTT

MSRVTTQA

MSVD

COCO

MSRVTT

Modality:image (CLIP, Sparsity:0.5)

Modality:video (CLIP, Sparsity: 0.5)

(CLIP,Sparsity:0.5)

VQAv2COCO

128.91

60.97

38.35

23.32

120

54.23

33.80

35.68

118.01

44.73

120

50

127.25

61.34

43.66

23.48

100

127.88

50.32

100

80

50.67

33.80

37.26

45

127.37

61.26

42.00

24.43

60

126.15

52.71

80

40

OKVQA

127.27

59.35

42.63

24.72

40

MSVD

50.02

34.10

38.17

112.31

39.18

60

COCO

VQAv2

GQA

OKVQA

MSRVTT

MSRVTTQA

MSVD

35

COCO

MSRVTT

40

VQAv2COCO

116.56

Modality: image (ViT, Sparsity: 0.3)

58.20

42.85

35.47

100

48.11

Modality: video (ViT, Sparsity: 0.3)

36.90

36.89

46

48

64.64

Modality: audio (ViT, Sparsity: 0.3)

32.84

49.72

65

60

Audiocaps

65.68

(VIT, Sparsity: 0.3)

115.83

47.27

110

100

115.66

116.63

58.15

57.99

42.91

44.25

36.65

36.36

08

60

47.97

36.90

36.71

44

42

40

Clotho

ClothoAQA

66.67

32.48

49.72

55

50

45

MSRVTT

COCO

65.60

67.40

116.34

117.19

46.94

47.70

06

80

70

116.46

57.46

44.34

37.22

47.58

36.50

35.92

63.76

32.53

49.81

40

Text

64.81

115.43

47.39

09

COCO

VQAv2

GQA

OKVQA

40

MSRVTT

MSRVTTQA

MSVD

36

Audiocaps

Clotho

ClothoAQA

35

Audiocaps

COCO

MSRVTT

50

VQAv2COCO

108.11

Modality: image (ViT, Sparsity: 0.5)

47.58

0.70

16.71

100

42.78

Modality: video (ViT, Sparsity: 0.5)

19.10

18.63

40

Audiocaps

57.12

Modality: audio (ViT, Sparsity: 0.5)

28.31

41.46

55

Audiocaps

59.66

(VIT, Sparsity: 0.5)

105.44

41.23

100

108.97

108.68

49.67

49.57

5.68

6.56

17.28

19.91

80

09

40

42.14

25.30

20.33

35

25

Clotho

58.18

27.82

36.30

45

40

50

35

MSRVTT

COCO

54.03

50.96

107.74

109.98

42.47

42.35

06

80

70

60

107.92

50.23

6.79

20.47

20

42.78

20.40

20.27

56.56

25.71

42.21

30

Text

47.81

95.26

37.29

50

20

40

COCO

VQAv2

GQA

OKVQA

MSRVTT

MSRVTTQA

MSVD

Audiocaps

Clotho

ClothoAQA

Audiocaps

COCO

MSRVTT

F Implications on performance, safety and efficiency: additional experiments

F.1 Implicit multimodal alignment as proxy metric for task performance?

IMA score across epochs. Fig. 33 shows an increasing similarity between textual and perceptual tokens during training with OPT and Vicuna-v1.5.

IMA score across different encoders. In Table 1 we compare different image encoders and report the the IMA score and the task performance on several VQA and image captioning tasks. Encoders

VQAv2COCO

117.36

Modality: image (ViT, Sparsity: 0.3)

59.23

45.19

29.30

100

49.41

Modality: video (ViT, Sparsity: 0.3)

37.20

37.14

48

50

Audiocaps

63.98

Modality: audio (ViT, Sparsity: 0.3)

24.50

46.44

5504540

60

Text MSRVTT COCOAudiocaps

65.00

(ViT, Sparsity: 0.3)

115.93

47.62

100

110

116.45

59.69

44.08

30.47

80

50.04

37.30

37.26

46

44

Clotho

63.52

24.58

44.84

65.88

116.99

49.30

06

08

117.05

118.17

59.83

59.87

44.09

43.71

30.92

30.89

60

40

49.47

36.60

36.65

38

42

40

64.06

24.61

44.37

30

35

63.79

50.06

116.50

95.06

49.16

41.63

50

60

COCO

VQAv2

GQA

OKVQA

MSRVTT

MSRVTTQA

MSVD

Audiocaps

Clotho

ClothoAQA

25

Audiocaps

COCO

MSRVTT

VQAv2COCO

107.26

Modality: image (ViT, Sparsity: 0.5)

48.22

17.72

16.42

100

43.25

Modality: video (ViT, Sparsity: 0.5)

11.00

14.62

40

sdesopny

58.89

Modality: audio (ViT, Sparsity: 0.5)

16.84

28.70

Text MSRVTT COCOAudiocaps

58.89

(ViT, Sparsity: 0.5)

104.05

41.61

100

107.11

107.14

51.79

50.79

17.43

20.43

18.96

18.04

80

60

42.03

18.00

23.30

25

35

30

Clotho

57.69

18.08

35.65

50

40

55.87

60.53

109.02

103.76

42.28

43.25

08

60

VOAXO

106.56

51.87

19.69

17.59

20

40

41.63

13.70

19.72

·15

20

ClothoAQA

55.90

19.76

41.09

20

30

36.34

56.70

16.50

40

20

COCO

VQAv2

GQA

OKVQA

MSRVTT

MSRVTTQA

MSVD

Audiocaps

Clotho

ClothoAQA

Audiocaps

COCO

MSRVTT

①DA0KVQA

1.90

0.99

9.99

99

9.63

0.64

0.80

All layer types (Sparsity: 0.5)

I vs T

V vs T

0.95

0.90

0.68

All layer types (Sparsity: 0.5)

I vs T

00000

(I0QA

0.29

0.99

1.00

0.99

0.95

0.99

0.63

0.63

0.64

0.64

0.75

A vs T

A)CQA

3.85

0.66

(I)VqA2

0.99

0.98

0.99

1.00

0.63

0.64

0.70

0.85

0.70

RCOCO

)004

0.72

0.67

0.69

0.63

0.72

9.67

D6T

0.74

0.06

1.00

0.67

1.00

0.65

LT3

0.80

0.64

T A-0KVQA-Des

0.63

0.63

0.63

6.63

1.03

0.71

dro

0.65

000V0A

6.71

0.69

0.70

0.75

0.60

0.64

0.64

0.64

0.64

1.00

S4'0

0.60

BV0A

0.71

0.T1

0.69

0.48

9.71

0.73

14.0

0.70

E.T3

0.61

1.00

0.58

0.56

(T)-GQAT

0.69

H

0.55

Transformer Layer

10

15

20

0.72

0.70

LF

Transformer Layer

19

15

20

25

30

that are most aligned to textual tokens inside LLMs (highest IMA e.g. CLIP) have also the best task performance.

F.2 Implicit multimodal alignment as proxy metric for hallucination?

We provide more details reagarding the hallucinations metrics. These metrics are supposed to measure multimodal or object hallucination.

CHAIR on COCO captioning [51]. On the COCO image captioning dataset, the model is asked to describe the images. We compute the CHAIR metrics based on the generated captions and the ground truth annotations of all objects in the image. If a caption contains non-existent objects, we classify it as a hallucinated caption. The CHAIRs score is the ratio of hallucinated captions to the total number of captions. Additionally, we calculate the ratio of hallucinated objects to the total number of objects across all captions, which is referred to as CHAIRi. A CHAIR score of 0 indicates no hallucinations. In the paper, we report (1 - CHAIR) × 100, thus a higher score indicates fewer hallucinations.

POPE benchmark [50]. This is a question-answering task involving questions about the existence of objects in images. The metric used is accuracy; the fewer the hallucinations, the higher the accuracy.

F.3 Skipping computations for visual tokens.

In this section, we propose to skip computations for the visual tokens.

Skip FFN Tokens. We randomly skip a number of tokens (we refer to this amount as skip ratio), both the textual and the remaining visual tokens are processed in the FFN layers. ?? , shows a linear relationship between the skip ratio and task performance, the higher the ratio the lower the scores.

(a) Token norms

(b) Consecutive blocks similarity

(c) Similarity inside each block.

175

0.8

8°0

150

0.6

125

0.4

IJLN1|I-P (n 1)

ILN1|1-P (n 1)

0.4

75

IILN1|1-T (n1)

IISA|I-P (n 1)

0.2

IILN1|I-T (n 1)

IISA|I-P (n 1)

ILN1I-P SP

IISA||-T (n 1)

IISA|-T (n 1)

ISAl-PvSP

50

25

FC1-P(n-)

JF-T(n1)

PC21P（n1)

IILN21-T(n1)

IFC2|1-T (n 1)

IILN2|I-P (n 1)

0.2

0.0

--

LN21-P(n1)

LN2]-T(m-1)

IJFC111-T (n 1)

I|FC21|1-P (n 1)

IJFC2|I-T (n 1)

IIFC1|-P (n 1)

0.2

0°0

+-

IIFC211-P vs T

ISAIl-PvST

IFC11-PvsT

LN2I/PySP:

LN2|-PVST

1[FCI]-P vs P

IIFC2|I-P vs P

10

Layers

15

20

25

30

10

Layers

15

20

25

30

10

15

Layers

20

25

1.0

1.0

60

50

0.8

80

Similarity

0.6

9°0

20

10

5

10

Layers

15

20

25

UFC10-T(n.1)-

HSAI-T(n1)

ILN111-P (n 1)

HLN21-P (n1)

ILN1-T (n)

1FC10-P(n1)

FC2|1-P(n 1)

IIFC2|I-T (n 1)

IISA|-P (n 1)

ILN2-T(n1)

30

Cosine

0.2

0.4

0.2

0.0

10

Layers

15

20

ILN11-P(n1)

IFC1-P (n1)

IISAI-T(n1)

HLN21P(n1)

IJLN1|1-T (n)

[SAI-P (n 1)

LN2)T(n 1)

IJFC21|1-P (n 1)

IJFC2|1-T (n 1)

IIFC1|-T(n1)

25

30

0.2

0.4

0.2

0.0

0

10

Layers

15

20

25

SAII-PVSP

SAIL-PVS.T..

[LN1}-P vs P

ILN1-P vST

[ILN2|I-P vs P

LN2]P vs T

IIFC1|I-P vs P

IIFC1|I-P vs T

IFC21I-P vs P

I|FC2|I-P vs T

30

PvsTep 0

P vs T ep 0

0.3

Pvs T ep 1

0.150

P vs T ep 1

PvsTep 2

0.125

P vs T ep 2

PvsTep3

P vs T ep 3

Cosine Similarity

0.2

Cosine Similarit

0.100

0.075

0.1

0.050

0.025

0.0

0.000

0.025

-0.1

10

15

20

25

30

10

15

20

25

30

Layers

Layers

F.4 α -SubNet

In Table 2, we evaluate our α -SubNet on additional multimodal tasks. Compared to other taskagnostic baselines such as magnitude pruning the scores of α -SubNet are significantly higher. This support more that the multimodal tokens activate similar weights inside LLMs.

Embeddingsimilarity

0.16

0.14

0.12

Similarity

0.10

Cosine

0.08

0.06

0.04

CLIP P vST

ViT P vs T

0.02

MAE P vST

0

5

10

15

20

25

30

Layers

LLaVA-1.5

LLaVA-1.5

VQAv2(Vicuna-v1.5)

65

VOAIxaL

SEED-IMG

65

★

GQA

TextVQA

SEED-IMG

60

60

60

50

55

55

40

AcC

AcC

AcC

50

50

30

(V) sl0

(V) sl2

45

Skip FFN Tokens(V) sl0

Skip FFN Tokens (V) sl2

45

Skip FFN Tokens (V) sl0

Skip FFN Tokens (V) sl2

20

(V) sl4

(V) sl6

(V+T) sl0

40

Skip FFN Tokens (V) sl4

Skip FFN Tokens (V) sl6

Skip FFN Tokens (V) sl8

40

Skip FFN Tokens (V) sl8

Skip FFN Tokens (V) sl4

Skip FFN Tokens (V) sl6

10

(V+T) sl2

(V+T) sl4

20

30

40

50

60

70

80

5

6

FLOPS

7

8

9

0

220

240

260

280

FLOPs

300

320

340

360

380

V-P

-T

V-P

-T

V-P

-T

V-P

-T

I-P

I-P

I-P

I-P

A-P

A-P

A-P

A-P

V-P

V-P

V-P

-T

-T

-T

I-P

I-P

I-P

A-P

A-P

A-P

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an exploration of the Implicit Multimodal Alignment (IMA) phenomenon in frozen Large Language Models (LLMs): when exposed to perceptual tokens (e.g. from image or audio features), authors show that those tokens are implicitly aligned on text tokens, even so the LLM has only been trained on text previously. The study argues this phenomenon is a fundamental mechanism into LLMs, and that it is relevant to (i) better understand task performance (e.g using that implicit alignment as proxy) and (ii) to better understand model visual faithfulness (i.e absence or presence of hallucinations). Based on their findings, the authors propose as well architecture changes such as skipping specific operations on perceptual tokens in order to decrease inference cost.\n\nSTRENGTHS:\n- This is a very original and in-depth work to understand the inside mechanisms appearing in MLLMs. Many/Most of the current published MLLMs are based on variations a pre-trained LLM + a pre-trained vision encoder with smaller VL connector in between. The insights from this study could be applied on many of those works, especially to better improve the visual faithfulness of those models.\n\nWEAKNESSES:\n- This is not commonly an important weakness but for this work, it might be: the presentation of the results and the overall quality of the layout organization is hurting the understanding. All the figures are pixelated bitmap/jpeg screenshots and are barely readable. This doesn't change the pertinence of the work, but hurt importantly how well its content is conveyed. I urge the authors to revisit this. Many solutions exist today, such as TKIZ or exporting figures in vectorized PDF to better include them in the final document. Lacking of space in a limited-page paper is understandable, but not using TKIZ or vectorized figures is not. Figure 5 for instance shouldn't be as is in this work. I challenge other readers to read/decipher the labels of the x and y axes on the heatmaps. \n\n- In many details, the work could have be improved. See the \"Typos\" list below. As well, some claims are not backed anywhere. For instance, Figure 1 and Figure 12 both claim a 70% sparsity. Where is that introduced in the main text? Consistency is not respected as well: in some contexts, P means Prompt while in other context means Perceptual. Is Prompt == Perceptual? This is not stated in Section 2. Prompt usually refers to the text and image prompts, before preprocessing and tokenization. The formatting is off as well. Sometimes, $P$ is used, sometimes plain-text P. Same for $T$ vs. T. \n\n- Evaluation could be more complete for the Multi Task setup with the use of common benchmarks such as LLaVABench-in-the-Wild, MMVet, etc. As well, Hallucinations benchmarks such as MMHALBench and HallusionBench should be used. Authors quote [46], which is the very work that introduces MMHALBench.\n\nTypos:\n - Line 90: Define k? \n - Line 94: FC1 and FC2 should be $FC1$ and $FC2$. Same for g, LN1, LN2, SA. \n - Line 92: What is (B)? It is not defined in eq (2). \n - Line 101: s/We/we/ \n - L105: Reformulate \"In the paper, we focus no LLaVA-1.5-4 as it is most similar to the ST setup, and analyse other variants in App. E.\"? s/ no / on / ? \n - L106: (e.g. Sec. 3) ? Do you mean (see Sec. 3)? Why e.g.?  \n - Section/Equation/Appendix references are not following the same format. Sec. is used in place of Sec., same for App. and Appendix, but Equation is fully written. What about adopting a consistent terminology? \n- Figure 2 is barely readable, same for Figure 3, Figure 4. Figure 5, as well it is not vectorized. Consider using TKIZ or export your figures in PDF and include those cropped PDF in your tex document. Bitmap/JPEG/PNG used here are leading to poor and pixelated results, hurting the readability. \n- Figure 1: s/Analysing/Analyzing/ \n- Figure 1: s/LMM/LLM/ (twice)\n- Figure 1: s/perceputa/perceptual/ \n- Figure 1: \"Computation efficiency\", do you mean \"Computational efficiency\" or \"Computation and efficiency\"? \n- Figure 1: What is the point of the right block named \"Framework\"? It doesn't seem related to the work or the caption of Figure 1. \n- L136: The claim \"Fig. 2 shows a clear narrow cone effect for textual and perceptual tokens.\" is not verifiable when looking at Figure 2.\n- Figure 10. What is the difference between Cosine Similarity and Sim. Aren't they the same as defined in Equation 3? Why using different terminology?\n\nQUESTIONS:\n- Author claim that the alignment between Image <-> Text tokens helps the model performing well on VQA tasks and hallucinations tasks. It would be interesting to know if this is a cause (i.e performance on vision tasks is high thanks to that alignment), or a consequence (i.e alignment is high along performance on vision tasks thanks to the vision-language training). A way to verify that could be to compare alignment among different off-the-shelf models (e.g. LLaVA-1.5, LLaVA-1.6, Qwen-VL-Chat, Phi3-Vision) and see if there are differences in their respective alignment scores. With such population of models and scores, it could be possible to rely on statistical testing to establish it. Do you see a covariation between alignment and performance among that population? Can you control the alignment independently of the training on vision-language itself? E.g. manipulating directly P and T such as the similarity is higher/lower, and still observe that covariation between the alignment and performance variables in that population?\n\n - As well, the authors claim that an LLM can be seen as a residual stream with refinement blocks acting as steering blocks, and that this architecture is what helps the LLM to implicitly having an alignment between text and image tokens. It seems hard to support this claim without contrasting their experiments with another architecture aligning image and text. For instance, using a simple pre-trained CNN to process the image, and a simple pre-trained RNN (e.g. an LSTM or GRU) to process the text input, and then concatenating/pooling the outputs with a simple connector (e.g c-abstractor or just a plain linear projection) could be used to contrast their findings. Is that something planned by the authors?\n\nLIMITATIONS:\n- One important limitation is the presentation of this work. The majority of the figures are pixelated and not readable. This is often a minor inconvenience in other works, but in this paper all the results are presented through figures. Figure 5 for instance should be removed or importantly reworked. Lacking of space in a limited-page paper is understandable, but not using TKIZ or vectorized figures is not. This is hurting the overall understanding of an interesting work trying to address interesting questions. This work should not be published as is. \n\n- Authors claim that the alignment between Image <-> Text tokens is what helps the model performing well on VQA tasks and hallucinations tasks. Something that is not tested is in which extend this is a cause (i.e performance on vision tasks is high thanks to that alignment), or a consequence (i.e alignment is high along performance on vision tasks thanks to the vision-language training). See \"Questions\" section of this review. \n\n- Authors claim that an LLM can be seen as a residual stream with refinement blocks acting as steering blocks, and that this architecture is what helps the LLM to implicitly having an alignment between text and image tokens. It seems hard to support this claim without contrasting the work with another architecture. See \"Questions\" section of this review.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores how frozen Large Language Models generalize to multimodal inputs without the need for extensive re-training. It introduces the concept of Implicit Multimodal Alignment, which suggests that despite the distinct representations of perceptual and textual tokens, there exists a form of implicit alignment facilitated by the architecture of LLMs. The study leverages experimental setups across single-task and multitask environments to validate the IMA phenomenon and discusses its implications for computational efficiency and model performance.\n\nSTRENGTHS:\n1. The analysis spans various setups and modalities, providing a comprehensive look at how LLMs process multimodal inputs.\n2. This study can quickly extend large semantic models to the multimodal domain, effectively reducing computational costs and enhancing the generalizability of LLMs.\n3. The results are meaningful, proving the effectiveness of the method.\n\nWEAKNESSES:\n1. The concept of alignment within neural networks, although well-explored here, does not offer a groundbreaking methodological advance. The novelty lies more in the application context rather than in the development of new techniques or models.\n2. The generalizability of the results is restricted to a subset of model architectures and sizes, potentially limiting the broader applicability of the findings.\n3. The visualization experiments are not clear. Current figures are somewhat generic and do not adequately convey the unique aspects of the IMA effect.\n\nQUESTIONS:\n1. How does the IMA effect influence the overall performance of LLMs on standard multimodal tasks?  Are there performance trade-offs associated with the alignments observed?\n2.  Are there indications that these findings could be applicable to other types of models or architectures not covered in the study?\n\nLIMITATIONS:\n1. The paper lacks implementation details.\n2. The study focuses primarily on a specific range of LLMs. More domains and tasks should be included.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

comments: SUMMARY:
This paper conducts an in-depth study on the generalization capabilities of LLM when handling multimodal inputs without multimodal fine-tuning. It reveals the implicit multimodal alignment (IMA) effect between perceptual and textual tokens within LLMs and finds that this effect is closely related to the model architecture. The IMA effect contributes to enhanced task performance and reduced inference costs. Additionally, the paper proposes methods for model compression and reducing computational overhead, providing valuable insights for the future design and optimization of multimodal models.

STRENGTHS:
Conducted an EXTENSIVE series of experiments to validate four hypotheses, providing insightful understanding of the mechanisms on how the MLLMs perceive multimodal information.

Based on the four findings, the paper offers relevant implications that explain the effects of IMA on tasks and hallucinations.

Proposes a novel approach to model compression by retaining a subnetwork.

Experiments are comprehensive and were conducted on various large language models, including OPT, Llama, and Vicuna.

WEAKNESSES:
The textual descriptions are somewhat difficult to understand, with limited explanation of the figures.

Insufficient explanation of the subnet part. It is unclear whether the similar performance after 50% sparsity is due to the effectiveness of the WANDA method or the extraction of the subnet.

QUESTIONS:
I do not fully understand line 247. How is the degree of hallucination measured? What is the relationship between cosine similarity and this measurement?

Out of personal curiosity, how long did these experiments take?

LIMITATIONS:
See weakness and question
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to understand multi-modality representation within MLLMs. It provides some interesting findings about how LLMs generalize to non-textual tokens and what helps LLMs to generalize to multimodal tokens. Additionally, several implications are proposed based on these findings.\n\nSTRENGTHS:\n1. The authors present many interesting findings about how LLMs generalize to non-textual tokens. These findings could help in understanding MLLMs and inspire future research.\n2. Based on these findings, the authors propose several implications on performance, safety and efficiency.\n3. The figures are well-illustrated and effectively support the findings.\n\nWEAKNESSES:\n**About experiments on α-SubNet.**\n\n--Figure 12 appears incomplete. What does \"Avg\" mean in this table? \n\n--The comparison between task-specific pruning approaches (Wanda) and the proposed task-agnostic approach is not comprehensive. For instance, what is the performance of Wanda sub-network pruning on COCO across other multimodal tasks? In other words, I'm curious about the generalization of the task-specific methods. Including these results would help in understanding the significance of task and modality-agnostic pruning methods.\n\n--Additionally, the results in Table 1 of the Appendix should be incorporated into the main paper for better readability.\n\n--Overall, I recommend reorganizing the experiments related to α-SubNet.\n\n**Others**.\n\n--line 1192 is unfinished.\n\nQUESTIONS:\nIn the single-task (ST) setting (Fig. 2), the authors use unimodal encoders without text-aligned. Figure 2 illustrates \"Multimodal cones: different narrow cones for different modalities.\" However, I wonder if this phenomenon is related to the unimodal encoders themselves—i.e., different modality features from unimodal encoders might have very different feature distributions, leading to different narrow cones in LLMs. Could the authors conduct new experiments using aligned encoders, such as ImageBind [1], as different modality encoders? This would help determine whether the different narrow cones in LLMs are due to different modalities or different encoders.\n\nRef:\n1. ImageBind: One Embedding Space To Bind Them All. CVPR 2023\n\nLIMITATIONS:\nAs stated by the authors, the generalization of the findings, to larger and more powerful models, with different architectures, including proprietary ones remains to be seen", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-09"}