Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs

Authors: mustafa  shukor
matthieu  cord
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-24 16:55:16

Abstract: Large Language Models (LLMs) have demonstrated impressive performance on multimodal tasks, without any multimodal finetuning. They are the de facto building block for Large Multimodal Models (LMMs), yet, we still lack a proper understanding of their success. In this work, we expose frozen LLMs to image, video, audio and text inputs and analyse their internal representation with the attempt to understand their generalization beyond textual inputs. Our work provides the following **findings.** Perceptual tokens (1) are easily distinguishable from textual ones inside LLMs, with significantly different representations (e.g. live in different narrow cones), and complete translation to textual tokens does not exists. Yet, (2) both perceptual and textual tokens activate similar LLM weights. Despite their differences, (3) perceptual tokens are implicitly aligned to textual tokens inside LLMs, we call this the implicit multimodal alignment effect (IMA), and argue that this is linked to architectural design, helping LLMs to generalize. This provide more evidence to believe that the generalization of LLMs to multimodal inputs is mainly due to their architecture.  These findings lead to several **implications.** This work provides several implications. (1) We find a positive correlation between the implicit alignment score and the task performance, suggesting that this could act as a proxy metric for model evaluation and selection. (2) A negative correlation exists regarding hallucinations (e.g. describing non-existing objects in images), revealing that this problem is mainly due to misalignment between the internal perceptual and textual representations. (3) Perceptual tokens change slightly throughout the model, thus, we propose different approaches to skip computations (e.g. in FFN layers), and significantly reduce the inference cost. (4) Due to the slowly changing embeddings across layers, and the high overlap between textual and multimodal activated weights, we compress LLMs by keeping only 1 subnetwork (called alpha-SubNet) that works well across a wide range of multimodal tasks. The code is available here: https://github.com/mshukor/ima-lmms.

Main paper: 
 introduction: 
 Large Language Models (LLMs) [1, 2, 3, 4, 5] represent a noteworthy advancement in recent AI developments. Building upon the success of LLMs, the next stride in this field involves extending beyond the textual modality, giving rise to Large Multimodal Models (LMMs) [6, 7, 8, 4] . A notable line of research involves connecting LLMs to visual encoders, while keeping them frozen and only training a connector with modest number of parameters [9, 10, 11, 12, 13, 14, 15, 16, 17] . These methods yield comparable performance [10] to large-scale multimodal models with significantly reduced computational and data budget. Keeping all pretrained unimodal models frozen and only training couple of millions of parameters [9, 10, 12] is an interesting phenomenon to understand, with limited research trying to decipher it. To explain why frozen LLMs can generalize beyond textual inputs, several hypotheses can be isolated: (1) perceptual tokens are transformed to textual ones, can be simply considered as foreign language [18] , and thus the LLM sees text-only tokens. (2) LLMs are able to digest non-textual tokens that are processed by (2a) modality-specific subnetworks or (2b) the same LLM weights that can generalize due to other reasons.In this study, we expose LLMs to different multimodal inputs, such as image, video, audio and text, and analyse their internal representations. We focus on frozen LLMs and consider two representative setups: single-task (ST) and multitask (MT) finetuning. The former is considered by parameter and data-efficient approaches [11, 12, 9, 10] and consists of training a mapping module for each dataset. The MT setup is considered by recent multimodal assistant models [19, 14, 20, 21] , and consists of training the same mapping module on several datasets/tasks.Our study shows (1) that perceptual and textual tokens still live in significantly different representation spaces inside LLMs (Sec. 3.1): live in different narrow cones, and have different norms, rate of change and vocabulary distributions. (2) We notice high similarity between the weights activated by textual and perceptual tokens (Sec. 3.2), allowing to swap these activated subnetworks between different tasks and modalities. Despite their differences, (3) perceptual tokens are implicitly aligned to textual ones across different stages (Sec. 4.1): during training of the mapping module, and during inference across LLM layers, especially inside each LLM block (e.g. after the self-attention layer). As there is no explicit objective to align these representations, we call it the Implicit Multimodal Alignment effect (IMA) (i.e., increasing similarity between textual and perceptual token distributions). We find that this effect is mostly linked to architectural design (e.g. residual stream with refinement blocks acting as steering blocks Sec. 4.2) . This provides more evidence to believe the architecture of LLMs is one of the main factors to generalize to multimodal representations.We shed light on several practical implications (Sec. 5). (1) We find a positive correlation between the implicit multimodal alignment score and the task performance, suggesting that this score could act as a proxy metric. On the other hand, (2) we find a negative correlation with hallucinations, revealing that the main factor leading to this problem is the lack of alignment between the internal representation of textual and perceptual inputs. (3) The perceptual tokens slightly change inside LLMs, thus, we propose to skip their computations (e.g. inside FFN layers). (4) Due to the slowly changing embedding across layers, and the high overlap between weights activated by different modalities, we compress the LLM by keeping one task-agnostic subnetwork that works well across all modalities. To summarize (Fig. 1 ), we analyse the internal representations of LLMs when exposed to multimodal inputs, leading to the following findings:• Perceptual and textual tokens live in different representation spaces inside LLMs.• They activate similar LLM weights.• They are implicitly aligned (IMA) inside LLMs, during training and during inference.• The architectural design of LLMs can be perceived as a residual stream with steering blocks.We argue that this is one of the main factors allowing LLMs to: digest very different tokens, drive the implicit multimodal alignment effect, and thus generalize to different modalities.These findings have several practical implications such as:• The IMA score as a proxy metric candidate for task performance and hallucinations.• Hallucinations as a result of lack of sufficient multimodal alignment.• Skipping computations for visual tokens, leading to efficient inference.• LLMs compression by keeping only 1 subnetwork that generalizes to all multimodal tasks.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: For findings Sec. 3 and implications Sec. 5.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
framework for analysing preceptually augmented llms: 
 General framework We focus on a general family of models that consists of: a frozen language model LLM with L layers, a trainable mapping module C, and a frozen perceptual encoder E M for different modalities M (e.g. image (I), video (V) and audio (A)). The LLM input X consists of the concatenation of P = [p 1 , ..., p Np ] multimodal/perceptual tokens (referred to as prompt) with T = [t 1 , ..., t Nt ] textual tokens. The prompt P is obtained after encoding the modality-specific input XM with the corresponding E M and using C to project it to the LLM input space. T is obtained from the embedding layer E T applied to the tokenized input text XT . This can be expressed as follows:EQUATIONThe k (k = N p + N t ) output tokens O = [o i , ..., o k ] are obtained after a normalization, followed by the unembedding layer W out (or LLM head, i.e. o i = W out LN out (t L i )). Our focus is on the internal representation of LLMs (i.e. tokens) at different stages, in particular across the L LLM blocks/layers (referred to as B). The mechanism inside the l + 1 LLM transformer block can be expressed as follows:EQUATIONwhere F C1, F C2, g are the up and down projections and activation inside the F F N , LN 1/2 are the layer norms and SA the self-attention.Perceptually augmented LLM baselines. For the single-task (ST) setup, we train many models across different datasets that span image, video and audio-text modalities. Each mapping module is trained on a specific dataset, similar to previous works [10, 12, 9] . Inspired by previous studies [10, 12] , we use light-weight transformer consisting of a self-attention to attend to perceptual tokens. In this setup, P refer to perceptual tokens from image, video and audio modalities. For the multitask (MT) setup, we devise different variants of the LLaVA-1.5 [19] model that differ from the original model as follows: LLaVA-1.5-2 (LLM kept frozen), LLaVA-1.5-3 (LLM kept frozen, without pretraining) and LLaVA-1.5-4 (LLM kept frozen, without pretraining and with transformer mapping module similar to the ST setup instead of MLP). In this setup, P refers to image tokens from different datasets. In the paper, we focus on LLaVA-1.5-4 as it is most similar to the ST setup, and analyse other variants in App. E. For analysis (i.e. Sec. 3), we focus on Vicuna-v1.5-7B [22] as it is shared by both setups. For the ST, we use unimodal encoders, such as ViT [23] , TimeSformer [24] and AST [25] that are not aligned with text. More implementation details, and experiments with other backbones can be found in App. D and App. E. We report the similarity after averaging the tokens SimAvg (Eq. ( 3)) and details other measures in App. E.Analysis tools. We are interested in cross-modal or multimodal alignment, and define the alignment in terms of the cosine similarity; the higher the score, the more the vector representations are pointing in similar directions. This could also indicates how much the two token distributions or vectors are close, in terms of L2 distance (assuming the vectors are normalized and in a narrow cones). In other words, alignment and similarity terms can be used interchangeably in the paper. In addition to cosine similarity we also study their norm, decoded vocabulary distributions and which LLM weights they activate. In the paper, we focus on the global representation per example, by analysing their average across the sequence. More finegrained analysis on the token level with different similarity and norm measures gives similar observations and are detailed in App. E. For instance, we compute the cosine similarity between perceptual (P ) (e.g., tokens corresponding to image patches) and textual (T ) tokens (e.g., tokens corresponding to the image caption), after the block l as follows:EQUATION3 LLMs indeed generalize to non-textual tokensWe investigate the generalization of LLMs to multimodal inputs, by studying the perceptual and textual tokens inside LLMs. We investigate if all tokens are projected to textual ones, or rather they are still different and how so (results with other models and similarity measures in App. E). Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Sec. 6 and more detailed in App. B Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
how perceptual tokens differ from textual ones?: 
 Multimodal cones: different narrow cones for different modalities (Fig. 2 ). Previous works [26, 27, 28, 29, 30, 31] have found the representation of contextualized embeddings in language models to be anisotropic: embeddings of different inputs exhibit high cosine similarity, shaping a narrow cone, where all embeddings point in the same narrow direction. In the multimodal domain, the cone effect is also observed [32] in contrastive models (CLIP [33] ). In this section, we investigate if textual and multimodal tokens live in narrow cones inside LLMs, and if these cones are distinct. We compute the tokens cosine similarity at different layers. In particular, the unimodal similarity: text-only (T vs T ) and perceptual-only (P vs P ), and the cross-modal similarity (P vs T ) between perceptual and textual tokens. Note that for the ST setup, P vs P covers the similarity between image, video and audio tokens, while for the MT ones cover image tokens from different datasets. Fig. 2 shows a clear narrow cone effect for textual and perceptual tokens. Different perceptual modalities seem to live in different narrow cones, as shown by the low P vs P score for the ST setup. Interestingly, the cross-modal similarity between textual and perceptual tokens (P vs T ) is significantly lower, suggesting that textual and perceptual tokens also live in different narrow cones.We also visualize the t-SNE of the tokens embeddings showing they stay separated inside the LLM. Different token norms and evolution across layers (Fig. 3 ). We compute the median of the token L2 norms after each LLM block. This shows that textual and perceptual tokens have different norms across layers. Perceptual tokens have significantly higher norm (at the beginning for MT and across all layers for ST), and they change significantly less. When looking at other norm measures, we found perceptual tokens with massive norms, similarly for textual ones [34] , especially for the ST setup. We discuss massive tokens more in App. E.2. In addition, we compute the cosine similarity between tokens at block l and block l + n, showing that textual and perceptual tokens have different change rates. Textual ones change drastically at the beginning of the LLM, while perceptual ones changes significantly less across all layers.Histogram (l=0) KL-divergence (X l+n vs X l ) KL-divergence Distribution entropy Different token vocabulary distributions across layers (Fig. 4 ). For each token, we use the LLM unembedding (i.e. LLM head) to decode the latent representation to a probability distribution over the vocabulary. This approach have shown to work well for LLMs at different layers, not just the last one [35, 36, 37, 38] . We show the histogram of this distribution at the first LLM layer for both textual and perceptual tokens. The histograms show clear differences with some overlap between textual and perceptual prompts. In addition, we compute the KL-distance showing that the distributions diverge from each other across LLMs layers. We also notice that the distributions of textual tokens evolve significantly, compared to multimodal ones. This is shown by computing the KL-distance between consecutive blocks and the distribution entropy.Finding 1. Textual and perceptual tokens live in significantly different representation spaces inside LLMs.For each trained model, we extract the LLM (frozen) subnetwork activated by each dataset/modality. We study these subnetworks (we refer to as pruning masks) by computing their similarity. We leverage the recent SoTA pruning approach (Wanda [39] ), that prune models based on both the weights and the activation norms. Specifically, we use a handful (e.g. 256) of calibrated examples coming from different modalities, and keep only p% (1 -sparsity) of weights with the highest Wanda score, at different sparsity levels (30 % and 50 %). Note that after removing more than 50% of weights we observe a severe degradation of performance.Similar activated weights across modalities, in the first and deeper layers (Fig. 5 ). Each subnetwork is represented as a binary mask to indicate which weights are activated. To compute the similarity between these networks, we consider the intersection over union (IoU). Results show an interesting high similarity between subnetworks activated by to different modalities. This high overlap is more seen for the ST setup, for example, the IoU between GQA and VQAv2 is 0.69, similar to GQA vs Audiocaps (0.67) or COCO-Text (0.65). When looking at the IoU across layers, we notice an interesting high score at first layers. It seems that first layers encode general features that are common for all modalities. This similarity increases as we go deeper in the LLM, moving to more abstract and less modality-specific representations, closer to the textual output.Transfer of multimodal subnetworks across tasks and modalities (Fig. 6 ). To further validate the previous section, we study if we can simply interchange pruning masks between different tasks. Specifically, for a given model trained to solve a particular task, we find the pruning mask using calibration data corresponding to different tasks/datasets. The sparsity is set to 30%, which is often used to maintain reasonable performance. Fig. 6 shows that the pruning masks transfer very well across tasks within the same modality (e.g. slight degradation by ∼1 point CIDEr for captioning with a mask coming from OKVQA). Similarly, we interchange masks across modalities. We fix the task (captioning) and also consider the text modalities (captions without images). In general, we observe similar transfer with a slight performance degradation, especially for OPT and Llama 2. We show similar observations with higher sparsity and with other encoders (e.g., CLIP and MAE) in App. E.4.The experiments suggest a high overlap between weights activated by different modalities. However, this does not exclude the possibility of finding weights that are generally activated when seeing a particular modality, even if there are small amount of them. More discussion about this can be found in App. E.4.Finding 2. LLM weights activated by perceptual and textual tokens overlap significantly.4 What helps LLMs to generalize to multimodal tokens?Textual and perceptual tokens have very different representations inside LLMs, yet, LLMs are still able to process and generalize to these non-textual tokens. In this section, we try to investigate why this is possible, in particular, we identify which factors facilitate this generalization. Implicit alignment during training of the mapping module (Fig. 7 ). We compute the cosine similarity between the perceptual tokens at the output of the mapping module, and textual tokens at different LLM blocks. Results show that this similarity increases at all layers. This reveals that the mapping module role, is not just to adapt the dimension of the visual tokens, but also to project the visual tokens to be semantically, as similar as possible to the textual ones.Implicit alignment during inference, across LLM blocks (Fig. 7 ). We compute the cosine similarity between perceptual and textual tokens after each LLM block. Here we compute the max of tokenwise similarity (MaxSim App. E.1): for each pair of token sequences coming from one example (e.g. image prompt + caption), we take the maximum similarity, then we average across all examples. The tokenwise similarity between perceptual and textual tokens significantly increases, especially in the middle blocks, where the alignment is the highest in deep layers. Implicit alignment during inference, inside LLM blocks (outside the residual stream) (Fig. 8 ).We look deeper to investigate the source of this alignment, and focus on tokens inside the LLM block, which consists mainly of a self-attention (SA), FFN (FC1/2) and layer norms (LN1/2) layers. Interestingly, the similarity between textual and multimodal tokens is the highest after the SA layers.Finding 3. An implicit multimodal alignment emerges to pull the textual and perceptual tokens closer inside LLMs, during training and inference.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
explanation: the architectural inductive bias hypothesis: 
 Residual stream with refinement blocks. We notice different observations between the tokens inside and outside the residual stream. In the residual stream, the perceptual and textual tokens exhibit significant representation differences (Sec. 3.1), while outside the residual stream, they are more aligned. Each block contributes slightly to the residual stream (small token norms inside the blocks Fig. 8a ), with significantly different contributions (cosine similarity between consecutive blocks close to zero, e.g, after the FC1/2 Fig. 8b ). This allows us to view the model as a series of refinement blocks that try to gradually refine the input signals. As the original signals are significantly different, they stay different in the residual stream throughout the model. We argue that this provide a flexibility to handle too different inputs. Moreover, previous works [40] have shown that transformers contain both elements with high and low complexity biases, which helps to build general-purpose architectures [41] that are able to generalize. These works support further our findings.Refinement blocks as steering blocks. Inside the transformer block, we notice that the layer normalization play an important role in having comparable norms for both textual and perceptual tokens (Fig. 8a ). Perceptual token norms become smaller and closer to textual ones as we traverse several layers in the block. In terms of cross-modal similarity, we notice the highest similarity after the SA, then after the FC2 and LN1. Note that this similarity is higher inside the block, than in the residual stream (e.g. 0.45 vs 0.1 for Vicuna-v1.5 and 0.58 vs 0.15 for LLaVA-1.5-4 in the residual stream Fig. 2 ). After each block the cross-modal alignment increases, and hence the narrow cones are steered to each other. This suggests that all layers play an important role in steering the textual and perceptual narrows cones to be aligned, with the most contributions coming from the SA.Finding 4. An LLM can be seen as a residual stream with refinement blocks acting as steering blocks. This architecture design plays an important role in generalizing to very different tokens, and hence other modalities.5 Implications: performance, safety and efficiency Implicit alignment as a proxy metric for task performance? (Fig. 9 ) We compute the cosine similarity between perceptual tokens at the LLM input and the textual tokens across LLM layers. The similarity increases during training. Interestingly, we notice a clear and positive correlation with the task performance on several multimodal benchmarks. In addition, we find that this correlation exists across different models, as shown for different LLaVA-1.5 variants.Implicit alignment as a proxy metric for hallucination? (Fig. 10 ) Previous works have shown that LMMs suffer from severe hallucinations [42, 43, 44] , and generally try to tackle this problem by training on better datasets [45] , using RLHF or RLAIF [46, 47] or post-training heuristics [48, 49] .Here we highlight one of the main causes of hallucinations: which is the lack of internal alignment between textual and perceptual representations. we show the cosine similarity between textual and perceptual tokens after each LLM block, and report the hallucinations on POPE [50] and COCO [51] benchmarks. The curves show clear correlation between the implicit alignment and the hallucinations. Skipping computations for visual tokens (Fig. 11 ). In Sec. 3.1 we show that perceptual tokens change significantly less across layers, compared to textual ones. Sec. 4 highlights the importance of SA layers for cross-modal alignment. In this section, we leverage these observations to reduce the LLM computation overhead by skipping the computations of visual tokens. Specifically, starting from a given start layer (sl), we reduce computations in FFN layers, which accounts for almost 2/3 of model weights, by skipping p% (Skip ratio) of visual tokens. Fig. 11 shows that skipping the visual tokens leads only to slight decrease in performance, while reducing significantly the amount of compute. We provide additional results with the ST setup, and ablation study in App. F. 2 .α-SubNet: one LLM Subnetwork for all multimodal tasks (Fig. 12 ) . Despite their differences, multimodal tokens share an important property: slowly changing embeddings across layers (Sec. 3.1). This suggests the possibility of compressing the model while retaining reasonable performance. In addition, textual and multimodal tokens are pulled closer inside the LLM (Sec. 3.2), and processed by almost the same LLM weights (Sec. 4), especially for the ST setup. This suggests the possibility of finding a common subnetwork (α-SubNet) that works well for all multimodal tasks. Thus, we focus on the ST setup with the OPT and CLIP encoders that are currently used by previous works. We consider two representative tasks: COCO image captioning and VQAv2 and provide similar results for other tasks, and modalities in App. F.4. First we use Wanda for task-specific pruning and show in Fig. 12 that we can obtain scores close to the original ones while removing 50% of the weights. To find the task and modality agnostic α-SubNet, we first extract many pruning masks (e.g. at 30% sparsity) for different modalities, then take the intersection of all these masks (e.g. leading to a global mask at ∼ 50% sparsity). This approach is significantly better than other baselines such as magnitude pruning or a random mask, and leads to comparable performance compared to the task-specific Wanda pruning, especially for VQAv2.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Details to reproduce the experiments are in Sec. 2 and more detailed in App. D. Also the code will be made public.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
open access to data and code: 
 Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: The paper uses solely public datasets.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
discussion: 
 Limitations. The paper focuses on open-source and frozen LLMs up to 7B parameters, LMMs that concatenate perceptual tokens at the LLM input and are relatively efficient. The generalization of our findings, to larger and more powerful models, with different architectures, including proprietary ones remains to be seen. Detailed discussion in App. B.We propose the first study of the internal representation of frozen LLMs when exposed to multimodal inputs. We find very different representations for perceptual and textual tokens, yet LLMs are still able to generalize to these non-textual tokens. The implicit multimodal alignment (IMA) effect, linked mostly to architectural design, facilitates this generalization by bringing multimodal tokens closer inside the LLM. Our findings have several implications, such as as reducing the computation resources at inference time, understanding better the performance as well as safetyrelated problems such as hallucinations. We hope that this study will have positive impact, pushing for more works to understand multimodal LLMs, and pave the way to devise more powerful models that are better aligned to human preferences, while targeting safety-related issues.NeurIPS Paper ChecklistQuestion: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The experimental details can be found in Sec. 2 and App. D Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: The paper includes different statistical measures to report scores such as the average, median and the variance.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 
licenses for existing assets: 
 Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: The paper cites all used assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13 This supplementary material is organized as follows:• App. A: detailed related work.• App. B: detailed discussion about the work, limitations and other implications.• App. C: the broader impact of the work.• App. D: implementation details, including the trained models, datasets and metrics.• App. E: additional experiments analysing LLMs.• App. F: additional experiments for the implications.Large multimodal models. Motivated by the success of large-scale training of LLMs [52, 1, 3, 53, 2, 5, 4] , the multimodal community has embarked on a parallel journey, striving to develop larger and more powerful models capable of processing multiple modalities. Typical Large Multimodal Models (LMMs) are constructed either by building upon frozen LLMs [6, 54, 55] or by training them end-to-end after initialization [56, 7, 8] . These models have demonstrated success in numerous general and intricate multimodal tasks, achieving performance levels close to human capability. Another important line of research focuses on unified models, where a single model is designed to handle diverse tokenized modalities, such as image-text [57, 58, 59] , or even beyond two modalities [60, 61, 62] .Efficient large multimodal models. Recently, to mitigate the training cost associated with training large multimodal models, efficient adaptation of unimodal models has emerged as a promising direction. Models like [11, 9, 17, 12, 10, 63] maintain LLMs frozen and train only a small subset of adaptation parameters for different multidmodal tasks. These approaches achieve competitive performance compared to end-to-end trained models [64, 65, 66, 67, 68] on image-text tasks and also on audio and video-text tasks [9, 69, 70, 10] . Beyond single-task tuning, many approaches do relatively light-weight pretraining and/or instruction tuning [71, 72, 14, 73, 19] and achieve good zeroshot generalizaton and instruction following abilities. To make these models more efficient, previous works have trained models with smaller LLMs showing competitive performance [74, 75, 76, 77, 78] Analyzing LLMs. Previous research has highlighted the highly anisotropic nature of embeddings within language models, characterized by high cosine similarity [26, 27, 28, 29, 30] . Studies focusing on efficiency have shown that textual tokens exhibit small changes across layers [79, 80, 81] . Additionally, LLMs contains outlier features [82] and massive activations [34] , which significantly influence model performance. Work by [83] suggests that LLMs may generalize due to the fractal structure of language. Moreover, [40] demonstrate that the building blocks of LLMs implicitly bias towards approximating both complex and simple functions. In the multimodal domain, [32] identify a modality gap in CLIP models attributable to the narrow cone effect. Previous studies have also explored neurons in LLMs that encode multimodal representations [84, 85] .Compression and pruning for multimodal models. Few works have targeted multimodal model compression, focusing mostly on image-text models. Notably, some works have concentrated on distilling knowledge from larger models through attention maps [86] or the affinity matrix in CLIP models [87] . Recent efforts have successfully applied the Lottery Ticket Hypothesis (LTH) to these models, optimizing both the model weights and masks jointly [88, 89] . A unified framework for structured pruning based on iterative training and adaptive sparsity allocation was proposed by [90] . It's worth noting that these approaches, initially designed for relatively small image-text models, encounter scalability challenges when applied to very large models.Hallucinations in multimodal models. Hallucinations in multimodal models involve generating text that refers to objects not present in the input image [51, 50] . This pervasive issue affects a wide range of multimodal models, varying in architecture, training data, and scale [91, 60, 42, 6] . To gain deeper insights into this phenomenon, numerous studies have proposed evaluation benchmarks to quantify hallucinations across different dimensions [51, 50, 92, 44] , shedding light on underlying causes. Specifically, co-occurrences and uncertainty [48] , as well as visual uncertainty stemming from lower image resolution [93, 94, 95] , have been identified as contributing factors. Additionally, it has been demonstrated that multimodal in-context learning exacerbates hallucinations [42] . To address this issue, various techniques have been proposed, including training on improved datasets [96, 44] , aligning models with reinforcement learning [46, 47] , refining training objectives [94] , and employing post-training heuristics [48, 93] . Our study highlights the misalignment between internal representations of textual and perceptual tokens as a key cause of hallucinations.Study across LLMs and setups. Our investigation primarily centers on Vicuna-v1.5 across singletask and multitask setups. We find that our conclusions remain consistent across various LLMs (e.g., OPT, Llama 2) and different settings (e.g., with and without pretraining, using different mapping modules), as detailed in the appendix. Extending our analysis to encompass other multimodal models, potentially with diverse architectures [6, 55] , could offer additional valuable insights.Study on larger models. While our work primarily focuses on frozen LLMs to provide insights relevant to future multimodal models, we also present results involving trained LLMs such as in LLaVA-1.5. These experiments yield observations akin to those with frozen LLM variants. However, the applicability of our experiments or the generalizability of our findings to larger LLMs (beyond 7B parameters), larger multimodal models [7, 6] , or massively-trained multimodal foundation models like Gemini [97] or GPT4-V [4] remains an open question.Remaining questions to understand LMMs. While we primarily investigate why and how LLMs generalize to multimodal inputs, and offer insights into issues such as object hallucinations, numerous unanswered questions persist. For instance, further exploration is needed to discern the encoded information in tokens and how LLMs extract information from visual tokens. Deeper inquiries are also required to address safety-related issues in large models, including the inability to abstain from answering, compositionality, and the precise adherence to user instructions [42] .Other implications. Our paper discusses several practical implications with potential benefits. Future extensions of our study could focus on specific aspects, such as enhancing model efficiency during training and inference by reducing redundant computations or model size. Additionally, addressing alignment with human preferences, such as faithfulness and safety, remains a significant challenge requiring further investigation. Our study may also inform model architecture design, such as developing mapping modules explicitly aligning multimodal tokens before entering the LLM.The paper aims to enhance our comprehension of LLMs within the realm of multimodal inputs. We contend that a deeper understanding of these models can yield positive societal impacts, which we partially address in this study. For instance, our findings may contribute to mitigating the consumption of large models and their potential societal harms. Moreover, our work may inspire future research endeavors with various impacts, none of which we think must be specifically discussed here.D.1 Perceptually augmented LLM baselinesWe train many models across different datasets that span image, video and audio-text modalities. We first devise powerful baselines based on 3 tenets: (a) having the smallest number of trainable parameters, (b) general architecture that span or similar to many existing models and (3) good performance. To this end, and inspired by previous studies showing the effectiveness of using transformer-based mapping module [10, 12, 6, 54, 55, 98] , we use light-weight transformer with learnable queries and self-attention to attend to perceptual tokens. This transformer operates in low dimension space (i.e., due to down/up projection layers and the the number of learnable query are limited to 10. We also favor a deeper architecture (5 blocks) compared to a wider one [10] .Our baselines are close to [10] , but with significantly less trainable parameters. We train these baselines with different LLMs: OPT-6.7B [2] , Llama 2-7B [5] and Vicuna-v1.5-7B [99] and different encoders for: image (ViT [23, 100] , CLIP [33] , MAE [101] ), video (TimesFormer [24] , X-CLIP [102] , VideoMAE [103] ) and audio (AST [25] , AudioMAE [104] ).To train these baselines, we use AdamW optimizer with a learning rate of 2e-4 that decreases with a cosine annealing scheduler to a minimium of 1e-5. We train with a total batch size of 16 for captioning and 64 for VQA datasets. The number of epochs is set to 20 to ensure that all models converged, though most of these models converge after only couple of epochs. We select the best checkpoint for evaluation. For example the model for image captioning converged after ∼ 4 epochs. All models are trained on 8 V100 GPUs and the training time depends on the task, e.g., for the large VQAv2 dataset each epoch takes ∼ 30 mins, for other smaller datasets it takes less time, e.g., ∼ 10 mins for Audiocaps and MSVD-QA. Unless specified otherwise, we fix the hyperparameters for all baselines to isolate the variations that could results from this.We refer to all text-aligned models as CLIP, trained for classification as ViT, and self-supervised with MAE objective as MAE.To study the impact of different factors (e.g. pretraining, mapping module) on the internal representations (e.g. implicit alignment), we devise different variants of the LLaVA-1.5 [19] model that differ from the original model as follows: LLaVA-1.5-2 (LLM kept frozen), LLaVA-1.5-3 (LLM kept frozen, without pretraining) and LLaVA-1.5-4 (LLM kept frozen, without pretraining and with transformer mapping module similar. The latter is very similar to the models used in the ST setup, which ensure comparable observatoins. All these models are based on the Vicuna-v1.5-7B LLM.We follow the same training setup of LLaVA-1.5 [19] , including the training data, steps and hyperparameters.For analysis in the paper (e.g. Sec.3), we focus on Vicuna-v1.5 as it is shared by both setups. For the ST setup, we use unimodal encoders, such as ViT, TimeSformer and AST that are not aligned with text.ST setup. We consider a wide range of public multimodal datasets that cover 2 representative tasks: captioning and question-answering (QA) across image (VQAv2 [105] , GQA [106] , OKVQA [107] , COCO caption [108] ), video (MSVD, MSRVTQA [109] , MSRVTT [110] ), audio (Audiocaps [111] , Clotho [112] , Clotho-AQA [113] ) and language tasks. For QA datasets we report the accuracy (in open-ended generation setup with exact match), and for captioning we report the CIDEr metric.MT Setup. We also evaluate the MT setup on recent datasets such as SEED [114] , TextVQA [115] and POPE [50] .E LLMs generalize to multimodal inputs: additional experiments E.1 Tokens evolution across layers Different LLaVA-1.5 variants. In Fig. 13 , we illustrate the differences between perceptual and textual tokens across different LLaVA-1.5 variants. Comparing the two variants with multimodal pretraining, we observe higher cross-modal similarity in LLaVA-1.5 compared to LLaVA-1.5-2, which freezes the LLM. This suggests that training the LLM enhances alignment between representations.For models without pretraining, we note that using an MLP (LLaVA-1.5-3) to connect both models yields better results than using transformer-based pooling (LLaVA-1.5-4), potentially explaining the superior scores. Utilizing all visual tokens appears to bolster alignment with textual tokens, with pretraining further enhancing this alignment. Notably, vocabulary distributions undergo significant changes in middle layers, particularly for textual tokens. Similar observations hold across different variants, indicating that our findings generalize to broader setups and that training the LLM does not substantially alter token behavior.Different similarity measures for cross-modal alignment. In this section, we compare the following similarity measures to compute the similarity between perceptual (P = [p 1 , ..., p Np ]) and textual (T = [t 1 , ..., t Nt ]) tokens:EQUATIONSimAvg(P, T ) = Sim( P , T ), P =EQUATIONEQUATIONMinSim(P, T ) = mini∈[Np]j∈[Nt]Sim(p i , t j ),EQUATIONEQUATIONWhere [N p ] = {1, ..., N p } and [N t ] = {1, ..., N t } and Med is the median operation.Fig. 14 shows the inter (P vs T ) and intra (P vs P ) similarity. According to all measures, except AvgSim and MinSim, we have similar observations: increasing inter similarity and higher intra similarity that increases in last layers. For MinSim and AvgSim for the ST setup, we do not see such observations, indicating that not all perceptual tokens are, or should be aligned to text.Fig. 15 and Fig. 16 show the measures comparison for the narrow cone experiments. Interestingly, the narrow cone effect is less seen when looking at the median of the token similarities (MedSim), indicating that this effect is not driven by all tokens, and at the token level the representation is not always anisotropic.In spite of having similar observations between several measures, we focus on SimAvg, as it is much faster to compute, especially whey there is large number of tokens (as in LLaVA-1.5). Different similarity measures for the similarity across consecutive layers. we compare the following similarity measures to compute the token similarity between consecutive blocks (e.g. between tokens at block a X a = x a 1 , ..., x a N and bEQUATIONEQUATIONEQUATIONEQUATIONWhere [N ] = {1, ..., N } and Med is the median operation.Fig. 17 shows similar observations across all different measures when each token is compared with the token at the same position in different layers. However, when taking the median of similarities (MedSim) across all tokens, this similarity is siginficantly smaller, especially for the ST setup. This reveals that tokens can be very different within the same modality or example. Tokens evolution for different modalities. Fig. 19 , shows that textual and multimodal tokens evolve differently inside LLMs.Figure 19 : Textual and multimodal tokens evolve differently inside LLMs. We compute the tokenwise cosine similarity and the median token L2 norm at consecutive blocks (e.g. X l+n and X l ) for the ST (left) and MT (right) setups.Massive token norms. In this section we highlight the presence of tokens with massive norms, this becomes clearer when looking at different norm measures. We compare the following measures to compute the token L2 norms across blocks (e.g. X = x 1 , ..., x N ):EQUATIONEQUATIONMinNorm(X) = minEQUATIONEQUATIONEQUATIONWhere [N ] = {1, ..., N } and Med is the median operation and M is the total number of elements in the tensor. For the ST setup, Fig. 18 shows a very high token norms when looking at NormAvg and MaxNorm, compared to MinNorm and MedianNorm. These massive norms are present for both textual and perceptual tokens, and they are larger for perceptual ones. When looking closely, we find that these tokens correspond to start or split tokens as seen in [34] . For the MT setup, we notice that these massive tokens presents mainly in the system message, which we remove for our study as it is common for all examples. Interestingly the perceptual tokens for the MT setup do not seem to have massive norms. Increasing token norm during training. We try to investigate why we have very high perceptual token norms. To this end, we compute the norm across different epochs. Fig. 20 shows that during training of the mapping module, the norms increase significantly.For each token, we use the LLM unembedding (i.e. LLM head) to decode the latent representation to a probability distribution over the vocabulary. This approach have shown to work well for LLMs at different layers, not just the last one [35, 36, 37, 38] . In Fig. 21 , we show the histogram of this distribution at the first LLM layer for both textual and perceptual tokens, the KL-distance between the 2 distributions, KL-distance between consecutive layers and the entropy. Here we report additional results for the LLaVA-1.5 baseline showing similar observations to those of ST reported in the main paper. We then show (from left to right): the histograms at the input of the LLM, the KL divergence between the distributions at consecutive layers, the KL divergence between textual and perceptual distributions and the distribution entropy. Top: Vicuna-v1.5, Bottom: LLaVA-1.5-4.Experimental setup. In this section, we analyse the subnetworks activated by different multimodal inputs. We use the Wanda score [39] to extracted these subnetworks or pruning masks, then compute the IoU. For multimodal datasets we consider only the perceptual tokens, for example the visual tokens without the questions for VQAv2. We also use the text in these datasets as a source for textual tokens (e.g., COCO-text consider only the captions in the COCO dataset).Different LLaVA-1.5 variants. In Fig. 23 , we show the overlap between the weights activated by different modalities. Across different LLaVA-1.5 variants, we find similar observations: high overlap between perceptual and textual activated weights (e.g. 0.6 IoU), which is less than the overlap between weights acrtivated by the same modality (e.g. 0.95 for perceptual tokens and 0.87 for textual ones). We also notice a significant decrease in IoU at the first layers, which might reveal that the first layers encode general features that are shared across modalities.Different LLMs for the ST setup. In Fig. 24 , we show the IoUs of activated weights for different frozen LLMs (i.e., OPT, Llama 2 and Vicuna-v1.5) for the ST setup. We notice similar observations across LLMs, and relatively higher overlap for OPT. We notice similar observation compared to the MT setup, where we have a significant decrease in the IoU in first layers. For this setup, it is clearer that the overlap increase for deeper layers.Different sparsity levels. In Fig. 25 , we study how the overlap between activated weights changes with the size of the extracted subnetworks. This size depends on the sparsity of the final model. We notice that the lower the sparsity, the higher the overlap, revealing that higher sparsity allows to extract more modality-specific activated weights.Pruning weights by streaming different modalities. In addition to the IoU, we also compare the differences between the task performance of activated weights. Fro each dataset, we give to the model either the perceptual prompt, the textual prompt or both, knowing that each example in the dataset consists of a perceptual prompt followed by the textual one. Fig. 26 shows slight differences in overall performance when the LLM is pruned by different modalities, with the best performance is by considering both the textual and the perceptual tokens. This also show the high overlap between the weights used to activate textual and perceptual tokens.Transfer of pruning masks to other tasks and modalities. To further highlight the overlap between weights, we report the performance when the model is pruned given data from other modalities or datasets. We notice similar observations across LLMs, such Llama 2 Fig. 29 and Vicuna-v1.5 Fig. 30 . Interestingly, we find the similar overlap with the unsupervised MAE encoders Fig. 27 compared to text aligned ones Fig. 28 . We notice a performance degradation when the model is pruned at high sparsity levels (0.5). Modality-specific subnetworks? The experiments suggest a high overlap between weights activated by different modalities. For instance, the pruning masks similarity (IoU) between datasets within the same modality is on par with those across modalities for the ST setup. However, this does not exclude the possibility of finding weights that are generally activated when seeing a particular modality, even if there are small amount of them. The overlap is smaller with LLaVA-1.5 variants making this possibility more likely for large scale multitask models.Alignment inside each LLM Block. Fig. 32 reports the tokens similarity and norms for both LLaVA-1.5 and Vicuna-v1.5. F Implications on performance, safety and efficiency: additional experiments F.1 Implicit multimodal alignment as proxy metric for task performance?IMA score across epochs. Fig. 33 shows an increasing similarity between textual and perceptual tokens during training with OPT and Vicuna-v1.5. IMA score across different encoders. In Table 1 we compare different image encoders and report the the IMA score and the task performance on several VQA and image captioning tasks. Encoders that are most aligned to textual tokens inside LLMs (highest IMA e.g. CLIP) have also the best task performance.F.2 Implicit multimodal alignment as proxy metric for hallucination?We provide more details reagarding the hallucinations metrics. These metrics are supposed to measure multimodal or object hallucination.CHAIR on COCO captioning [51] . On the COCO image captioning dataset, the model is asked to describe the images. We compute the CHAIR metrics based on the generated captions and the ground truth annotations of all objects in the image. If a caption contains non-existent objects, we classify it as a hallucinated caption. The CHAIRs score is the ratio of hallucinated captions to the total number of captions. Additionally, we calculate the ratio of hallucinated objects to the total number of objects across all captions, which is referred to as CHAIRi. A CHAIR score of 0 indicates no hallucinations.In the paper, we report (1 -CHAIR) × 100, thus a higher score indicates fewer hallucinations.POPE benchmark [50] . This is a question-answering task involving questions about the existence of objects in images. The metric used is accuracy; the fewer the hallucinations, the higher the accuracy.F.3 Skipping computations for visual tokens.In this section, we propose to skip computations for the visual tokens.Skip FFN Tokens. We randomly skip a number of tokens (we refer to this amount as skip ratio), both the textual and the remaining visual tokens are processed in the FFN layers. ??, shows a linear relationship between the skip ratio and task performance, the higher the ratio the lower the scores.  
 
 Appendix:  


Decision: This is an original and exciting attempt to understand the underlying mechanics of how LLMs generalize beyond text tokens to process multimodal inputs. All the reviewers liked this work. For examples, Reviewer vaQc noted that "The authors present many interesting findings about how LLMs generalize to non-textual tokens. These findings could help in understanding MLLMs and inspire future research." Reviewer hn2C commented "Conducted an EXTENSIVE series of experiments to validate four hypotheses, providing insightful understanding of the mechanisms on how the MLLMs perceive multimodal information." Reviewer 69kn added that "This is a very original and in-depth work to understand the inside mechanisms appearing in MLLMs. Many/Most of the current published MLLMs are based on variations a pre-trained LLM + a pre-trained vision encoder with smaller VL connector in between. The insights from this study could be applied on many of those works, especially to better improve the visual faithfulness of those models."

Reviewer dcgF who seems most critical about the work, although still positive about the work, is expecting more "methodological advance" from the work, and points out other limitations of the work such as findings specific to model architectures. I think the authors addressed these concerns well in the rebuttal. 

In conclusion, this is an exciting work to advance our understanding about LLMs, which should be of highly interest to NeurIPS audience.

Reviews: review: paper_summary: paper_summary: This work aims to understand multi-modality representation within MLLMs. It provides some interesting findings about how LLMs generalize to non-textual tokens and what helps LLMs to generalize to multimodal tokens. Additionally, several implications are proposed based on these findings.
strength_weakness: strength_weakness: **About experiments on α-SubNet.**

--Figure 12 appears incomplete. What does "Avg" mean in this table? 

--The comparison between task-specific pruning approaches (Wanda) and the proposed task-agnostic approach is not comprehensive. For instance, what is the performance of Wanda sub-network pruning on COCO across other multimodal tasks? In other words, I'm curious about the generalization of the task-specific methods. Including these results would help in understanding the significance of task and modality-agnostic pruning methods.

--Additionally, the results in Table 1 of the Appendix should be incorporated into the main paper for better readability.

--Overall, I recommend reorganizing the experiments related to α-SubNet.

**Others**.

--line 1192 is unfinished.
questions: questions: In the single-task (ST) setting (Fig. 2), the authors use unimodal encoders without text-aligned. Figure 2 illustrates "Multimodal cones: different narrow cones for different modalities." However, I wonder if this phenomenon is related to the unimodal encoders themselves—i.e., different modality features from unimodal encoders might have very different feature distributions, leading to different narrow cones in LLMs. Could the authors conduct new experiments using aligned encoders, such as ImageBind [1], as different modality encoders? This would help determine whether the different narrow cones in LLMs are due to different modalities or different encoders.

Ref:
1. ImageBind: One Embedding Space To Bind Them All. CVPR 2023
limitations: limitations: As stated by the authors, the generalization of the findings, to larger and more powerful models, with different architectures, including proprietary ones remains to be seen
paper_summary: paper_summary: This paper conducts an in-depth study on the generalization capabilities of LLM when handling multimodal inputs without multimodal fine-tuning. It reveals the implicit multimodal alignment (IMA) effect between perceptual and textual tokens within LLMs and finds that this effect is closely related to the model architecture. The IMA effect contributes to enhanced task performance and reduced inference costs. Additionally, the paper proposes methods for model compression and reducing computational overhead, providing valuable insights for the future design and optimization of multimodal models.
strength_weakness: strength_weakness: The textual descriptions are somewhat difficult to understand, with limited explanation of the figures.

Insufficient explanation of the subnet part. It is unclear whether the similar performance after 50% sparsity is due to the effectiveness of the WANDA method or the extraction of the subnet.
questions: questions: I do not fully understand line 247. How is the degree of hallucination measured? What is the relationship between cosine similarity and this measurement?

Out of personal curiosity, how long did these experiments take?
limitations: limitations: See weakness and question
paper_summary: paper_summary: The paper explores how frozen Large Language Models generalize to multimodal inputs without the need for extensive re-training. It introduces the concept of Implicit Multimodal Alignment, which suggests that despite the distinct representations of perceptual and textual tokens, there exists a form of implicit alignment facilitated by the architecture of LLMs. The study leverages experimental setups across single-task and multitask environments to validate the IMA phenomenon and discusses its implications for computational efficiency and model performance.
strength_weakness: strength_weakness: 1. The concept of alignment within neural networks, although well-explored here, does not offer a groundbreaking methodological advance. The novelty lies more in the application context rather than in the development of new techniques or models.
2. The generalizability of the results is restricted to a subset of model architectures and sizes, potentially limiting the broader applicability of the findings.
3. The visualization experiments are not clear. Current figures are somewhat generic and do not adequately convey the unique aspects of the IMA effect.
questions: questions: 1. How does the IMA effect influence the overall performance of LLMs on standard multimodal tasks?  Are there performance trade-offs associated with the alignments observed?
2.  Are there indications that these findings could be applicable to other types of models or architectures not covered in the study?
limitations: limitations: 1. The paper lacks implementation details.
2. The study focuses primarily on a specific range of LLMs. More domains and tasks should be included.
paper_summary: paper_summary: The authors propose an exploration of the Implicit Multimodal Alignment (IMA) phenomenon in frozen Large Language Models (LLMs): when exposed to perceptual tokens (e.g. from image or audio features), authors show that those tokens are implicitly aligned on text tokens, even so the LLM has only been trained on text previously. The study argues this phenomenon is a fundamental mechanism into LLMs, and that it is relevant to (i) better understand task performance (e.g using that implicit alignment as proxy) and (ii) to better understand model visual faithfulness (i.e absence or presence of hallucinations). Based on their findings, the authors propose as well architecture changes such as skipping specific operations on perceptual tokens in order to decrease inference cost.
strength_weakness: strength_weakness: - This is not commonly an important weakness but for this work, it might be: the presentation of the results and the overall quality of the layout organization is hurting the understanding. All the figures are pixelated bitmap/jpeg screenshots and are barely readable. This doesn't change the pertinence of the work, but hurt importantly how well its content is conveyed. I urge the authors to revisit this. Many solutions exist today, such as TKIZ or exporting figures in vectorized PDF to better include them in the final document. Lacking of space in a limited-page paper is understandable, but not using TKIZ or vectorized figures is not. Figure 5 for instance shouldn't be as is in this work. I challenge other readers to read/decipher the labels of the x and y axes on the heatmaps. 

- In many details, the work could have be improved. See the "Typos" list below. As well, some claims are not backed anywhere. For instance, Figure 1 and Figure 12 both claim a 70% sparsity. Where is that introduced in the main text? Consistency is not respected as well: in some contexts, P means Prompt while in other context means Perceptual. Is Prompt == Perceptual? This is not stated in Section 2. Prompt usually refers to the text and image prompts, before preprocessing and tokenization. The formatting is off as well. Sometimes, $P$ is used, sometimes plain-text P. Same for $T$ vs. T. 

- Evaluation could be more complete for the Multi Task setup with the use of common benchmarks such as LLaVABench-in-the-Wild, MMVet, etc. As well, Hallucinations benchmarks such as MMHALBench and HallusionBench should be used. Authors quote [46], which is the very work that introduces MMHALBench.

Typos:
 - Line 90: Define k? 
 - Line 94: FC1 and FC2 should be $FC1$ and $FC2$. Same for g, LN1, LN2, SA. 
 - Line 92: What is (B)? It is not defined in eq (2). 
 - Line 101: s/We/we/ 
 - L105: Reformulate "In the paper, we focus no LLaVA-1.5-4 as it is most similar to the ST setup, and analyse other variants in App. E."? s/ no / on / ? 
 - L106: (e.g. Sec. 3) ? Do you mean (see Sec. 3)? Why e.g.?  
 - Section/Equation/Appendix references are not following the same format. Sec. is used in place of Sec., same for App. and Appendix, but Equation is fully written. What about adopting a consistent terminology? 
- Figure 2 is barely readable, same for Figure 3, Figure 4. Figure 5, as well it is not vectorized. Consider using TKIZ or export your figures in PDF and include those cropped PDF in your tex document. Bitmap/JPEG/PNG used here are leading to poor and pixelated results, hurting the readability. 
- Figure 1: s/Analysing/Analyzing/ 
- Figure 1: s/LMM/LLM/ (twice)
- Figure 1: s/perceputa/perceptual/ 
- Figure 1: "Computation efficiency", do you mean "Computational efficiency" or "Computation and efficiency"? 
- Figure 1: What is the point of the right block named "Framework"? It doesn't seem related to the work or the caption of Figure 1. 
- L136: The claim "Fig. 2 shows a clear narrow cone effect for textual and perceptual tokens." is not verifiable when looking at Figure 2.
- Figure 10. What is the difference between Cosine Similarity and Sim. Aren't they the same as defined in Equation 3? Why using different terminology?
questions: questions: - Author claim that the alignment between Image <-> Text tokens helps the model performing well on VQA tasks and hallucinations tasks. It would be interesting to know if this is a cause (i.e performance on vision tasks is high thanks to that alignment), or a consequence (i.e alignment is high along performance on vision tasks thanks to the vision-language training). A way to verify that could be to compare alignment among different off-the-shelf models (e.g. LLaVA-1.5, LLaVA-1.6, Qwen-VL-Chat, Phi3-Vision) and see if there are differences in their respective alignment scores. With such population of models and scores, it could be possible to rely on statistical testing to establish it. Do you see a covariation between alignment and performance among that population? Can you control the alignment independently of the training on vision-language itself? E.g. manipulating directly P and T such as the similarity is higher/lower, and still observe that covariation between the alignment and performance variables in that population?

 - As well, the authors claim that an LLM can be seen as a residual stream with refinement blocks acting as steering blocks, and that this architecture is what helps the LLM to implicitly having an alignment between text and image tokens. It seems hard to support this claim without contrasting their experiments with another architecture aligning image and text. For instance, using a simple pre-trained CNN to process the image, and a simple pre-trained RNN (e.g. an LSTM or GRU) to process the text input, and then concatenating/pooling the outputs with a simple connector (e.g c-abstractor or just a plain linear projection) could be used to contrast their findings. Is that something planned by the authors?
limitations: limitations: - One important limitation is the presentation of this work. The majority of the figures are pixelated and not readable. This is often a minor inconvenience in other works, but in this paper all the results are presented through figures. Figure 5 for instance should be removed or importantly reworked. Lacking of space in a limited-page paper is understandable, but not using TKIZ or vectorized figures is not. This is hurting the overall understanding of an interesting work trying to address interesting questions. This work should not be published as is. 

- Authors claim that the alignment between Image <-> Text tokens is what helps the model performing well on VQA tasks and hallucinations tasks. Something that is not tested is in which extend this is a cause (i.e performance on vision tasks is high thanks to that alignment), or a consequence (i.e alignment is high along performance on vision tasks thanks to the vision-language training). See "Questions" section of this review. 

- Authors claim that an LLM can be seen as a residual stream with refinement blocks acting as steering blocks, and that this architecture is what helps the LLM to implicitly having an alignment between text and image tokens. It seems hard to support this claim without contrasting the work with another architecture. See "Questions" section of this review.

Comments: comment: I would like to thank the authors for their response. It is unclear why producing TikZ figures would require to rerun experiments as mentioned in the rebuttal. I assume the authors have kept all the outputs of their experiments in a readable format (csv, json, or even better wandb) for future consultation. My other concerns have been addressed. I keep my ratings unchanged.
Dear reviewer, we hope our clarifications can address your concerns, and we sincerely hope that you can reconsider our work in light of these clarifications. If you have any further comments, please do not hesitate to contact us. We greatly appreciate your contributions to the community.
Thank the authors for the detailed response. All my concerns have been addressed. I'd like to keep my rating.