Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning
Authors: Bingchen Zhao
Haoqin Tu
Chen Wei
Jieru Mei
Cihang Xie
Venue: ICLR 2024 spotlight (2024)
Keywords: multi-modality; large language models; generation; model efficiency;
Primary area: generative models

Abstract: This paper introduces an efficient strategy to transform Large Language Models (LLMs) into Multi-Modal Large Language Models. 
By conceptualizing this transformation as a domain adaptation process, \ie, transitioning from text understanding to embracing multiple modalities, we intriguingly note that, within each attention block, tuning LayerNorm suffices to yield strong performance. 
Moreover, when benchmarked against other tuning approaches like full parameter finetuning or LoRA, its benefits on efficiency are substantial.
For example, when compared to LoRA on a 13B model scale, performance can be enhanced by an average of over 20\% across five multi-modal tasks, and meanwhile, 
results in a significant reduction of trainable parameters by 41.9\% and a decrease in GPU memory usage by 17.6\%. On top of this LayerNorm strategy, we showcase that selectively tuning only with conversational data can improve efficiency further. 
Beyond these empirical outcomes, we provide a comprehensive analysis to explore the role of LayerNorm in adapting LLMs to the multi-modal domain and improving the expressive power of the model.

Published as a conference paper at ICLR 2024
TUNING LAYERNORM IN ATTENTION: TOWARDS EFFI- CIENT MULTI-MODAL LLM FINETUNING
Bingchen Zhao
Haoqin Tu
Chen Wei
Jieru Mei
Cihang Xie*
*cqul contribution
University of Edinburgh
2 University of Chinese Academy of Sciences
3 Johns Hopkins University
UC Santa Cruz
ABSTRACT
This paper introduces an efficient strategy to transform Large Language Mod- els (LLMs) into Multi-Modal Large Language Models. By conceptualizing this transformation as a domain adaptation process. i.e., transitioning from text under- standing to embracing multiple modalities, we intriguingly note that, within each attention block, tuning LayerNorm suffices to yield strong performance. Moreover, when benchmarked against other tuning approaches like full parameter finetuning or LoRA, its benefits on efficiency are substantial. For example, when compared to be enhanced by average of over
LoRA 13B model scale, performance 20% across five multi-modal tasks, and meanwhile, results in a significant reduc- tion of trainable parameters by 41.9% and a decrease in GPU memory usage by 17.6%. On top of this LayerNorm strategy, we showcase that selectively tuning only with conversational data can improve efficiency further. Beyond these empirical outcomes, we provide a comprehensive analysis to explore the role of LayerNor in adapting LLMs to the multi- multi-modal domain and improving the expressive power of the model.
INTRODUCTION
Large Language Models (LLMMs) have had many application scenarios since their debut. In particular, extending LLMs to handle multiple modalities has gathered much interest from both academia and industry. Such models, termed Multi-modal Large Language Models (MLLMs), are typically derived by finetuning pretrained LLM on multi-modal data (Liu et al., 2023; Ye et al., 2023). However, this process typically poses a substantial computational challenge (Liu et al., 2023), particularly for exceptionally large-scale models. While Su et al. (2023); Zhang et al. (2023) employ low-rank adapters (LoRA) (Hu et al., 2022) or soft prompts (Li & Liang, 2021a) 12) for more parameter-efficient tuning, this often comes at the cost of compromised performance multi-modal tasks. This challenge prompts the pivotal question: how can we make this process more efficient?
In response to this challenge, we introduce simple and effective strategy for MLLM finetuning: as illustrated in Figure ((a), within each attention block, we adjust only the weights of the Layer- Norm (Ba al., 2016). This strategy is underpinned by the understanding that the evolution from LLMs to MLLMs can be conceptualized as a domain adaptation process, i.e., transitioning from text- centric to multi-modal understanding. Adjusting normalization layers, as suggested by prior research, emerges as a particularly effective technique in such domain shifts (Li et al., 2016). Empirically, this straightforward technique can surprisingly yield comparable or even better performance than the strong baseline of finetuning all parameters offer about 10x more parameter efficiency than LoRA.
By delving deeper, we note that the process can be further simplified by designating LayerNorm as the sole trainable component within the entire model. This means, in contrast to the typical configurations depicted in Figure I (c)- we now freeze the standardly activated elements, including the vision- language connector, word embedding, and the output head. We term it LayerNorm-simple Impressively, despite constituting a mere 0.004% of trainable parameters. this configuration surpasses the performance of LoRA, registering average enhancement of 4.3% across five benchmarks.

---

Published as a conference paper at ICLR 2024
Head
Layerhorm
Layerloom
Layerfoom
Layentorr
Lokk
Athention
Allantion
Attention
Trainable Parameters (%)
Layerfiorm
Layerfiam
Leyerfior
Word Erib.
Connector
VL Connector
Vel Connector
( LayerNorm
(b) QV Proj./ MLP
(c) LoRA
GPU Memory (GB)
Figure (left) Different tuning methods for MLLMs. Trainable components are in blue, while frozen parameters are in gray. Within the attention blocks, (a) only activates LayerNorm parameters. Note that vision-language connector, word embedding, and output head paramters are by default activated for all three options. (right) Comparison on trainable parameters and GPU memory. Tuning LayerNorm achieves significant reductions in trainable parameters and GPU memory usages.
On top of this LayerNorm strategy, we further improve the finetuning efficiency from the data perspective. Specifically, we assess the performance implications of different types of finetuning data, including conversational data, detailed description data, and complex reasoning data. Our results offer a crucial insight: not all data are created equal for the task of MILM finetuning. Remarkably, find that MILMss finetuned conversational data consistently outperform those finetuned on other data types. Specifically, conversational data improves the model performance by an average of 50% compared other data types. This observation interestingly opens up avenues for more targeted data collection and curation strategies, thereby further optimizing the efficiency of MLLMs finetuning. Furthermore, by combining the LayerNorm strategy and this data perspective, we can achieve on average 10.0% performance improvement over full parameter finetuning on traditional VQA benchmarks with an I.LAMA2 13B model while using significantly less parameters and data.
Beyond the empirical outcomes above, we conduct an investigation into the expressive power of LayerNorm tuning. Our analysis reveals that LayerNorm-ture MILMS exhibit lower cross-layer similarity compared to models all of which parameters are finetuned. This lowered similarity is indicative of a more expressive model, since the model incorporates anisotropic layer presentations can capture wider range of learning patterns (Pires et al., 2023). It stands reason that this amplified expressiveness is a key factor underpinning the efficiency and superior performance we noted, granting the model enhanced adaptability novel multi-modal datasets.
In essence, our findings illuminate the profound influence of LayerNorm tuning, suggesting its potential to adeptly harness the intrinsic properties of LLMs. We hope that this study will catalyze subsequent research endeavors focused on efficient multi-modal finetuning.
2 RELATED WORKS
Multi-Modal and Large Language Models. Multi-modality has been extensively studied in the literature. Starting from learning aligned representation across image and text modalities like CLIP (Radford al., 2021), many works have proposed more similar techniques (Yu al., 2022; Mu et al., 2022; Zhao et al., 2023) and try to explain the internal mechanism of models using multiple and efficient representations (Chen et al., 2023; 2024). Given the interest in developing instruction- tuned LLMs (Ouyang et al., 2022), the studies of multi-modal models have also shifted the focus to instruction-tunt -tuned MLLMs. For example, LLAVA (Liu et al., 2023) pioneers the development of iustruction-tuna MLLMs by designing instruction-tunin data with the help of GPT-4. The concurrent work MINIGPT4 (Zhu et al., 2023) is built using QFormers (Li et al., 2023a) and VICUNA (Zheng et al., 2023), but with only linear layer activated for tuning. Similar to MINIGPT4,
2

---

Published as a conference paper at ICLR 2024
Su et al. (2023) devises PANDAGPT with more advanced vision encoder and LoRA-tuned LLM as its base models. MPLUG-OWL (Ye et al., 2023) mixes text-only and multi- instruction data for finetuning LLAMA model. INSTRUCTBLIP (Dai et al., 2023) builds on the BLIP2 model (Li et al., 2023a) with additional finetuning on instruction tuning datasets.
Parameter-Efficient Finetuning. The parameter-efffcient finetuning (PEFT) technique has been widely applied and studied because of huge resource consumption of larger and larger deep learning models. Adapter (Houlsby et al., 2019) additionally adds trainable adapter components in the LM, which proved to achieve comparable results in NLP tasks with less than 10% trainable parameters. Prefix tuning (Li & Liang, 2021b) only inserts trainable parameters to the attention head in LMs LoRA (Hu et al., 2022), as the most widely employed PEFT method recently, injects trainable low rank decomposition matrices into model to reduce the number of training parameters. Following the same line, QLoRA (Dettmers et al., 2023) achieves further reduction in the memory usage for finetuning LLMs with quantized 4-bits parameters. These techniques have been widely utilized for the tuning of LLMs (He et al., 2022; Tu et al., 2022), MLLMs (Zhu et al., 2023; Tu et al., 2023) and other applications (Dutt et al., 2023; Xu et al., 2023). In this paper, we show that tuning LayerNorm in the LLM of an MLLM can achieve better results than tuning other components in the model and LoRA tuning, while requiring less resource for computation.
The Normalization Studies. The normalization layer in neural networks has long been subject for debate and study. The foundational work on batch normalization (Toffe & Szegedy, 2015) first introduces normalization as an important part of neural network architectures, and argues the effectiveness of normalization comes from alleviating the internal covariant shifting problem. Later works have proposed many variants of normalization, such as InstanceNorm (Ulyanov et al., 2016), GroupNorm (Wu & He, 2018), and LayerNorm (Ba et al., 2016). LayerNorm has been the design choice of LLMs for normalization, and its effectiveness in L.M pretraining has also been explored and discussed (Xu et al., 2019). In this work, we explore the effectiveness of finetuning LayerNorm in MLLMs as well as the reason behind the effectiveness.
TUNING AND EVALUATION SETTINGS
In this section, we first introduce the comment structure of MILMMs different tuning strategies of MLLMs, and then present the evaluation benchmarks employed in the paper.
Architecture of MILMs. A typical MLLM usually contains three parts: 1) A vision encoder for extracting visual features; 2) An LLM decoder for generating plausible texts from both text instruc- tions and visual information; 3) A laiionla connector for bridging between the vision encoder and the LLM. We follow Liu et al. (2023) to set up the model, where the vision encoder is CLIP- pretrained ViT-L (Radford et al., 2021) and the vision-language connector is a simple linear projector.
We experiment with a range of LLMs for the language decoder. Specifically, we choose three types of LLM with 7B and 13B scales: VICUNA-7BB (v1.1) (Zheng et al., 2023), LLAMA2-7B& 13B, and LLAMA2-CHAT-7BB. 13B (Touvron et al., 2023).
Baseline Models. Apart from our trained MILMs, we showcase the performances of three publicly available models: MPLUG-OWL (Ye al., 2023), MINIGPT4 (Zhu al., 2023), and LLAVA- vo (Liu et al., 2023), The LLAVA-VO here represents the initial release of LLAVA. These baseline results are obtained from existing literature (Fu et al., 2023; Li et al., 2023b) or tested using their released checkpoints.
Tuning Modules. To analyze the effects of different tuning components in the MILMS, we employ five different tuning paradigms on the same training corpus. (1) finetune: activates all the parameters in LLM for MLLMs tuning: (2) LoRA: inserts LoRA component (Hu et al., 2022) with rank 32 between all linear structure in the LLM; (3) Attn. QV Proj.: activates Q and V linear projection in attention of the LLM, as they are proved to be especially effective for tuning LLMs (Hu et al., 2022); (4) Attn. MLP: activates MLP layers in attention of the LLM; (5) LayerNorm: both input and post LayerNorm in attention blocks of the LLM. Note that all tuning methods activate vision- Langugge connector for training.
3

---

Published as a conference paper at ICLR 2024
Table Model performance on five multi-modal benchmarks with different components tuned in the LLM. We mark the best results with bold and the second best scores with underline. means the model cannot follow the required output format on captioning tasks.
Models
VQAv2
MSCOCO f
F1 Plickr30k
POPR
BASELINE MODELS
MPLUG-OWI.
967 967.4/276.1
21.38
70.70
41.78
50.9/54.0/50.7
MINIGPT4
866.6/292.1
17.30
69.7/79.7/66 7/65.2
LLAVA-VO
502.8/214.6
15.06
58.89
23.02
70.5/74.6/66.0
MM-VICUNA-7B
Finetune
625.2/270.7
15.40
67.50
34.61
73.8/76.5/66.5
LoRA
552.3/11.5
15.00
63.93
0...4/1.6/50..
Attn. QV Proj.
678.0/277.5
15.51
72.63
32.24
72.0/77.1/65.3
Attn. MLP
637.3/268.2
15.37
65.22
37.47
60.0/68 2/56.6
LayerNorm
723.2/253.2
17.06
80.89
48.01
76.1/81.1/70.8
LayerNorm-sime
720.9/251.8
23.46
79.75
46.18
61.1/72.3/58.5
MM-LLAMA2-7B
Finetune
661.3/237.1
16.09
65.08
31.64
56.3/65.0/55.
LoRA
395.0/200.0
14.87
61.97
26.17
51.9/54. 7/51..
Attn. QV Proj.
584.0/222.9
16.39
76.05
42.93
$5.7/63.0/50
Attn. MLP
413.1/203.6
15.29
58.35
29.04
53.7/59.6/57
LayerNorm
583.2/200.7
16.78
88.85
49.24
66.6/68.5/64.9
LayerNorm-simp.
542.6/205.0
14.98
65.10
46.88
51.6/52.5/51.1
RMM-LAMA2-CHAATEE
Finetune
805.4/234.6
15.29
66.33
26.70
60.3/69 0336698//57.9
LoRA
709.8/228.6
15.28
$7.27
25.49
59.2/65.9/56.2
Attn. QV Proj.
926.5/220.7
15.88
58.49
31.10
68.5/77.3/65.
Attn. MLP
840.0/240.0
15.20
$4.42
24.89
56.9/67.3/56.
LayerNorm
651.3/219.3
16.60
75.34
43.75
71.3/72.4167.8
LayerNorm-simp.
372.0/169.3
18.42
59,99
41.63
52.0/54.6/52.33
MM-LLAMA2-1 13BB
Finetune
402.3/199.3
18.33
73.88
45.33
51.6/51.1/52.2
LoRA
400.8/189.3
16.08
68,83
43.70
50.5/51.2/50.5
Attn. QV Proj.
489.0/200.4
15.12
63.07
32.81
51.1/52. 9/52.55
Attn. MLP
387.5/167.1
25.19
64.19
44.06
50.7/52.2/511
LayerNorm
526.0/177.5
15.31
82,92
48.42
60.0/69.1/58.1
LayerNorm-simp
403.3/185.4
18.62
68.28
43.04
55.3/58.0/57.2
MALLLAMA--CHA---AA
Finetune
623.3/221.4
15.17
64.19
41.82
67.6/64.8/64
LoRA
516.7/214.3
14.39
66.33
43.09
66.9/64. 1/63.8
Attn. QV Proj.
624.5/250.4
14.91
60.96
34.90
66.3/66.0/61.8
Attn. MLP
456.7/211.4
14.67
62.19
40.39
06.8/56.9/56.5
LayerNorm
929.3/254.3
16.10
74.96
42.79
78.9/83.9/74.3
LayerNorm-snnp.
824.3/221.1
13.29
52.70
40.20
73.3/76.0/69
Training Details. We pre-train the lision-language connector for 3 epochs on CC3M (Changpinyo al., 2021), and conduct the finetuning stage on 80K filtered image-text pairs collected by Liu et al. (2023) for epoch. For the first stage, set the learning rate to 2e-3 for all variants. During the second stage, we search the learning rate from 2c-3 to le-7 with 11 options for all tuning strategies and pick the best learning rate based on their performances on Flickr30k task. We set the weight decay (Loshchilov & Hutter, 2019) to and warmup ratio to 0.03 with the cosine learning rate scheduler (Loshchilov & Hutter, 2017). Moreover, employ the gradient checkpointing (Chen et al., 2016), DeepSpeed technique (Rajbhandari et al., 2020), and data precision of TensorFloat32 for models training. We conduct all of our experiments 4 80G A100 GPUs the same node.
4

---

Published as a conference paper at ICLR 2024
Table 2: Memory consumption and percentages of trainable parameters tested on single A100 GPU.
7B SCALE
13B SCALE
Model
Mem. (GB)
#param.
Mem. (GB)
#param.
Finetune
OOM
95.70%
OOM
97.72%
LoRA
29.4
5.92%
46.5
4.30%
Attn. QV Proj.
57.0
19.02%
OOM
18.24%
Attn. MLP
0OM
65.21%
OOM
66.24%
LayerNorm
24.2
3.78%
38.3
2.50%
LayerNorm-simp.
18.9
0.004%
31.7
0.003%
Multi-Modal Benchmarks. We test the visual- instrliisttuuttion tuned models on recent multi-modal evaluation benchmarks, where five multi-modal benchmarks are deployed: MME (Fu et al., 2023) consists two evaluation aspects, i.e., cognition (CS) and perception (PS) with total 14 VQA tasks; VOAv2 (Antol et al., 2015), MSCOCO (Lin et al., 2014) and Flickr300 (Young al., 2014) captioning tasks are commonly used benchmarks in the field of VQA and captioning. The former two benchmarks based MSCOCO-2017 dataset (Lin al., 2014). For the latter captioning tasks, we report the zero-shot CIDEr (Vedantam et al., 2015) scores (with three text-only QA examples) on the test set from Karpathy & Fei-Fei (2015). POPE (Li et al., 2023b) is used evaluate the level of object hallucinations in MLLMs, which consists of three versions of balanced yes/no VQA tasks (i.e., Ropular/Random//Aveessttt considering objects in the given image.
4
EXPERIMENTAL RESULTS
4.1 TUNING LAVERNORM
Tuning LayerNorm in Attention Blocks. In table it is noteworthy that activating only the LayerNorm yields the least activated parameters, yet the model performances are surprisingly impressive when compared to tuming other modules. Specifically, in two captioning tasks, the VOAv2 task, and the challenging hallucination benchmark POPE, models with only the LayerNorm activated consistently outperform all other competitors by at least 8.2%. On the comprehensively evaluated benchmark MME, while tuning LayerNorm outperforms finetuning the intact language model by an average of 6.6% on the Perception aspect, lags behind finetuning by an average of 6.3% on the Cognition score. It is vital to note, however, that the LayerNorm only accounts for approximately 2.5% of the training parameters in the whole model.
In addition to tuning modules, another observation is that MLLMs incorporating human-aligned LLMs (such as LLAMA2-CHAT) exhibit superior performance in complex and demanding tasks such POPE and MME compared to their unaligned counterparts. This underscores the importance of utilizing aligned LLMs to construct a more powerful MI.MMs.
Tuning LayerNorm and Only LayerNorm. As the above LayerNorm method finetunes (1) vision- language connector, (2) word embedding, (3) output head, and (4) LayerNorm component in the LLM simultaneously. a pertinent question arises: Is it possible for (4) LayerNorm alone to generalize effectively in training MLLMs? To address this query, we take step further and solely finetune LayerNorm in MLLMs, which is denoted as LayerNorm-simp. in table 1. The results are intriguing, demonstrating that even with mere 0.004% parameter finetuning the whole model, LayerNorm- simp. surpasses full parameter finetuning on three conventional -aniuan tasks (i.e., two captioning and one VQA tasks) by 10%, and only lags behind full finetuning by 7.9 7.9% on the MME. benchmark. This intriguing discovery suggests that the transition from LLM to MILMS probably involves a domain adaptation process as the LayerNorm takes the most credits in tuning a well- behaved MLLMs. The LayerNorm alone may be also capable of integrating vision information with language tokens seamlessly.
Memory Consumption and Parameter Efficiency. In table 2, we present the total memory con- sumption and the percentage of trainable parameters of each MLLMs finetuning method across 7B and 13B scales. Methods like full parameter finetuning and finetuning MLPs in attention modules face out-of-memon (OOM) issue even high-capacity 80GB A100 GPU, while LayerNorm based
5

---

Published as a conference paper at ICLR 2024
65.0
Randem 200
Rancer DBC
60.0
60.0
200
57.5
57.5
55.0
52.5
52.5
50.0
50.0
47.5
400
1.00
502
800
1000
Raniee Convonttrr
Gonveratins
80.0
57.5
52.5
59.9
10
12
12
MSCDCO (CED)))
VI000CO (0
Figure 2: Performances of models that are finetuned different datasets four multi-modal benchmarks. The MME score is the sum of both Cognition and Perception scores on the benchmark.
methods stand out for their efficiency. Specifically, LayerNorm tuning requires only 24.2 GB and 38.3 GB memory 7B and 13B scales respectively. Remarkably, LayerNorm-simp. further reduces the memory to 18.9 GB and 31.7 GB. In terms of trainable parameters. LayerNorm based methods also show remarkable efficiency, LayerNorm utilizes only 3.78% and 2.50% of the total parameters at the 7B and 13B scales, and LayerNorm-sine simm. takes efficiency to an extreme, involving only 0.004% and 0.003% of the parameters at these scales. These results demonstrate the efficiency advantage of LayerNorm tuning, compared with existing methods like LoRA or full parameter finetuning.
4.2 'LESS MORE' ON BOTH DATA AND PARAMETER SIDES
Efficiency in training can also be improved by considering the data used in LLMs and MILMMs (Zhou al., 2023; Wei et al., 2023). To this end, we conducted experiments using LLAMA2-7D and LLAMA2-7B-CIAT where we divided the training data into three categories, each comprising 20K data points: mmage-grounded conversation, image detail descriptions, and mmage-based complex reasoning, as previously deployed in Liu et al. (2023). Based on the results presented in fig. 2, we observe that the image-grounded conversation data the most effective enhancing the multi-modal capabilities of the model, with an average improvement of over 50% compared to other data types. This highlights the potential benefits of a targeted approach that leverages the strengths of specific data types facilitate more nuanced and effective multi-modal tuning for MLLMs.
To validate 'Less is More' on both data and parameter sides, we present results of MLLMs with LayerNorm activated in LLM and tuned on 20k conversational data in table 3. Our experimental results indicate that even with a smaller dataset and the use of LayerNorm tuning, the model outperforms the full parameter finetuning approach the full 80K dataset by 18.4% two captioning tasks, and only falls short in MME by a tolerable 2.5%. It is noteworthy that NorerNorm with 20K data is only 7.6% and 7.4% behind LayerNorm the full 80K dataset for two captioning tasks and MME task, respectively. These findings demonstrate that 'Less is More' for both the parameter and data perspectives beyond language domain Zhou et al. (2023), but for multi-modal tuning.
5 INTUITIONS BEHIND LAYERNORM TUNING
In this section, driven by the empirical success of LayerNorm tuning, we explore the intuitions behind LayerNorm from three perspectives, domain adaptation, expressive power, and gradient variance.

---

Published as a conference paper at ICLR 2024
Table 3: Model performance on different data types. Methods with 80K and Conv.20K suffix are tuned on the full 80K data and the 20K conversational data, respectively.
Method
MME
VOAv2
MSCOCO
Flickr30k
POPE
MM- VICUNA-7B
Finetune-8OK
625.2/270.7
15.40
67.50
34.61
73.8/76.5/66.5
LayerNorm-80r
723.2/253.2
17.06
80.89
48.01
76.1/81.1/70.8
LayerNorm-Conv. 20K
777.1/231.4
15.39
67.30
40.33
75.2/79.2/68.1
MM-LLAMA2-78
Finetune-80E
661.3/237.11
16.09
65.08
31.64
56.3/65.0/55 00/55.455544
LayerNorm-80k
583.2/200.7
16.78
88.85
49.24
66.6/68.5/64.9
LayerNorm-Com 20K
376.2/157.5
16.19
86,80
44.88
50.5/50.7/50.3
MM-ILAMA2-CHITT-77
Finetune-8KK
805.4/234466
15.29
57.40
26.70
60.3/69.8/57.9
LayerNorm-80 -OKK
651.3/219.3
16.60
75.34
43.75
71.3/72.4/67.8
LayerNorm- Conv. 20K
482.9/172.1
13.88
66.85
41.95
62.7/71.7/61.3
MM--LAMMA--13B
Finetune-80EB
402.3/199.3
18.33
73.88
45.33
51.6/51.1/52.2
LayerNorm-S88
526.0/177.5
15.31
82.92
48.42
60.0/69. 1/58.9
LayerNorm-Coo 20K
646.0/242.9
16.01
76.50
44.86
70.0/76.9/68.6
MM-LLAMA2-CHAT--12
Finetune-8OK
623.3/221.4
15.17
64.19
41.82
67.6/64.8/644
LayerNorm-80K
929.3/254.3
16.10
74.96
42.79
78.9/83.9/74.3
LayerNorm-Con Conv. 20K
769.7/227.5
15.57
73.30
43.08
68.2/72.8/65.3
Table 4: Results of models with LayerNorm and/or vision- lssion-langugge Connector activated.
Method
MME
VOAv2
MSCOCO
Flickr30k
POPE
MM LLAMA2-7B
LayerNorm Connector
583.2/200.7
16.78
88.85
49.24
66. 66.6/6
Connector
311.1/105.4
12.72
60.43
35.91
67.9/73.7/6699
LayerNorm
395.0/191.4
18.18
80.13
41.68
50.3/51.3/50.2
MM-LLAMA2-1331
LayerNorm Connector Connector
526.0/177.5
15.31
82.92
48.42
60.0/69.1/58.9
507.0/187.9
15.22
62.60
25.13
60.9/66.8/60.1
LayerNorm
405.0/188.6
16.51
70.41
39.86
50.9/52.7/51.0
5.1 LAYERNORM TUNING ADAPTS LLMs TO MULTI-MODAL
Influence of the Vision-Langusee Connector The vision-language connector to project features from the vision encoder to the LLM domain. In our previous experiments, we focused on finetuning the LLM component of the MILMs while keeping the vision-language connector activated by default. To determine which component plays a more important role for domain adaptation of LIM to multi-modal domain, we performed an ablation study by activating the two components separately. Results are presented in table 4, tuning LayerNorm in attention blocks without activating the vision-language connector resulted in only a 4.2% and 5.4% decrease in performance on three traditional multi-modal tasks and the MME benchmark, respectively. This decrease significantly lower than the 15.6% and 9.2% downgrade observed when only activating the Connector the same tasks. This observation highlights the vital role LayerNorm plays in transforming knowledge from the vision domain to language, indicating LayerNorm as strong domain adaptor for the I.LM architecture.
the converter
7

---

Published as a conference paper at ICLR 2024
Finetuning
(b) LayerNorm Tuning
Figure 3: Layer similarities between different L.M layers in (a) Finetuned and (b) LayerNorm-tunnd MM- MM-VICUNA-7B. The average layer similarity of two models are 0.624 and 0.585, respectively.
Table 5: Results of models with LLAMA2 Finetuned/LayerNom--une with VIT pre-trained on ImageNet (Deng et al., 2009), which have not been aligned with the language domain.
MME
VOAv2
MSCOCO
Flickr30k
POPE
Finctune-7B LayerNorm-P7 7B
406.79/182.5
15.05
47.75
18.97
50.0/51.6/50.1
301.51/127.14
15.48
66.22
31.73
50.0/50.1/50.
Finetune- 13B LayerNorm- 13B
375.41/171.79
25.38
51.26
25.96
50.3/51.1/51.0
445.98/150.0
15.59
64.63
32.17
51.2/53.0/50.8
Switching Visual Features. We employ the VIT encoder from CLIP (Radford et al., 2021) by default in our previous experiments. CLIP (Radford et al., 2021) models are trained with image-text contrastive loss, thus its feature space is already aligned with language. Since LayerNorm has shown its effectiveness as a domain adaptor, we are interested in testing whether or LayerNorm tuning can adapt a LLM to image features that are not pretrained to align with language. The vision encoder switched ViT model that was pretrained on ImageNet (Dosovitskiy al., 2021: Deng et al., 2009). Results in table 5 demonstrate that both LayerNorm and finetuning approaches can yield high performance. Interestingly, we observe that by LayerNorm tuning with Ima geNet trained ViT, which has not been aligned with language, the model is able to achieve comparable performance to full parameter finetuning i.c., results show that LayerNorm tuning outperforms finetuning by 12.0% captioning tasks, but performs slightly worse by 5.0% the MME benchmark. These results again indicates the domain adaptor role of the LayerNorm, hinting the reason behind the empircal success of LayerNorm tuning. Furthermore, it is worth noting that the performance of MLLMs incorporating ViT pretrained on ImageNet is generally inferior to that of CLIP's vision encoder. This observation provides compelling evidence that, despite differences in tokenizer and training paradigm between CLIP's text encoder and LLAMA's, ViT from CLIP has the capacity to learn general patterns of language formulation during pre-training. Thus, significantly enhance MLLM abilities.
5.2 LAYERNORM TUNING IMPROVES THE EXPRESSIVE POWER
is shown in Pires al. (2023) that a Transformer model incorporating anisotropic layer representa- tion can capture wider range of learning patterns. By computing the cosine similarities between all layers in the LLM of a finetuned MLLM, we aim to investigate whether the improved efficiency is the results of the improved expressive power In table 6, we present the average layer similarity of three 7B scale MLLMs, and in fig. 3 we present the visualization per layer similarity scores of MM-V VICUNA--B. Our analysis reveals that the transformer layers in the MLLMs with LayerNorm tuning exhibit a clear distinction from one another (i.e., an average 10.6% lower layer similarities comparing finetuning), indicating superior generalization ability and expressive power compared to
8

---

Published as a conference paper at ICLR 2024
0135
0.02
0803
006
-10
Stxa
(a) Finetuning
LayerNorm Tuning
Figure 4: Gradients of the input LayerNorm in the 11th layer of the MM- LICVICUNA training proceeds. LayerNorm-tuned model has lower gradient variance than full parameter finetuning.
finetuning. This finding sheds light on why tuning LayerNorm is effective for multi-modal LLM training. For additional visualizations, please refer to the Appendix A.2.1.
Table 6: Layer representation similarity of LayerNorm and finetuning methods on three 7B MLLMs. Lower the similarity is, the better expressive power a model possesses.
Model
I LayerNorm Sim.
Finetuning Sim.
MM- VICUNA MM-LLAMA2
0.585
0.624
0.504
0.591
MM-LLAMA2-CHAT
0,550
0.617
5.3 LAYERNORM TUNING HAS SMALLER GRADIENT VARIANCE
well accepted view about LayerNorm is that, the neural network goes deeper, the mean of LayerNorm gradients should goes to zero as the LayerNorm itself is designed to normalize all training parameters. In the meantime, the variance of LayerNorm gradients should be small to ensure better generalization ability of the model (Xu et al., 2019) (See the proof in Appendix A.2.2). As we presented in fig. 4, MILMM with LayerNorm tuning method has more concentrated LayerNorm gra- dients than fine-tuning during the training process. This result gives another view on the effectiveness of LayerNorm from the optimization perspective. More visualizations are listed in Appendix A.2.2.
CONCLUSION AND DISCUSSIONS
LayerNorm is effective and sufficient built upon MILM pre-training, MLLM training typically involves pre-training on image-text pairs followed by finetuning on visual instruction data. While the second stage of training receives more attention, it is worth noting that the function of the first stage pre-training is non-negligible for training a competent MILM. We have presented in the paper only a small portion of parameter activation is sufficient to tune well-behaved MLLM. However, other models such as INSTRUCTBLIP (Dai et al., 2023) and MINIGPT4 (Zhu et al., 2023) only tune the languaae connector, leaving the LLM untouched during the second stage of training. These models have yielded strong performances when given a large-scale finetuning dataset. In Sec. 5. we demonstrate that tuning LayerNorm may be more effective means for the second stage training, especially when compared to existing parameter-efficient methods for training MLLMS.
Limitations. One shortcoming of these parameter-efficieet finetuning methods is that they are more sensitive to trainable parameters of LayerNorm is small, the model performance of LayerNorm method also varies when twitching the training hyper-parameters. This drawback calls for potential future investigations on the LayerNorm tuning method. In the Appendix A. we give hint for the grid search range of learning rate on both 7B and 13B scaled models using LayerNorm tuning based on our experimental results.
(e.g., learning rate, training epoch) than finetuning. Since the number of
Conclusion, Our studies demonstrate LayerNorm tuning as a simple yet effective tuning method for adapting L.LSs comprehend multi-modal content across various model variants. Compared LoRA

---

Published as a conference paper at ICLR 2024
tuning or full parameter finetuning, LayerNorm tuning reduces the trainable parameters by signifi- cant 41.9%, enabling efficient finetuning MLLMs on mansuer--graee GPUs. Moreover, we demon- strate that MLLMs can achieve exceptional performance with minimal "right" data and parameters, showcasing the potential of LayerNorm tuning method in real-world applications. apppicarrapiiiiatioonn. Given the empirical success of LayerNorm tuning, revisited the MILMM finetuning from domain adaptation perspec- tive and showed that LayerNorm plays a critical role in adapting LLMs to the multi-modal domain. Ad- ditionally, our research illustrates the expressive power and optimization potential of LayerNorm tun- ing from layer similarities and the gradient variance. We hope that our work could inspire future works on designing improved PEFT methods that enable more diverse application scenarios for MLLMs.
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015. 5
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization. arXiv:1607.06450. 2016. 1, 3
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021. 4
Ruoyu Chen, Jingzhi Li, Hua Zhang, Changchong Sheng, Li Liu, and Xiaochun Cao. Sim2word: Explaining similarity with representative attribute words via counterfactual explanations. ACM Transactions on Multimedia Computing, Communications and Applications, pp. 1-22, 2023.
Ruoyu Chen, Hua Zhang, Siyuan Liang, Jingzhi Li, and Xiaochun Cao. Less is more: Fewer interpretable region via submodular subset selection. arXiv preprint arXiv:240.009164, 2024. 2
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint 1rXf4:1604.06174, 2016. 4
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards ceneral-purpose vision-language models with instruction tuning. NeurIPS, 2023. 3,9
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009. 8
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lls. arXiv preprint arXiv:2305. 2023. 3
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthia Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16 words: Transformers for image recognition at scale. ICLR, 2021. 8
Raman Dutt, Ondrej Bohdal, Sotirios A Tsaftaris, and Timothy Hospedales. Fairtune: Opti- mizing parameter efficient fine tuning for faimess in medical image analysis. arXiv preprins S.t:::2310000055. 2023. 3
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023. 3,5
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efffcient transfer learning. ICLR, 2022, 3
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In ICML, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. ICLR, 2022. 1,3
10

---

Published as a conference paper at ICLR 2024
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In JCML, 2015.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In CVPR, 2015. 5
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ICML, 2023a. 2, 3
Xiang Lisa Li and Perey Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACI,, 2021a.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. ACL, 2021b.
Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation. arXiv preprint 1603.04779 2016.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 20236. 3,5
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurlPS, 2023. 1, 4,
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. ICLR, 2017.
lly Loshchilov and Frank Hutter. Decoupled weight decay regularization, ICLR, 2019.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. ECCV, 2022.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. NeurIPS, 2022. 2
Telmo Pessoa Pires, António V Lopes, Yannick Assogba, and Hendra Setiawan. One wide feedforward is all you need. arXiv preprint arXiv:2309.01826 2023. 2, 8, 13
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 2, 8
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC, 2020. 4
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305. 16355, 2023. 1,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. 2023. 3
Haoqin Tu, Zhongliang Yang, Jinshuai Yang, and Yongfeng Huang. Adavae: Exploring adaptive gpt-2s variational auto-encoders for language modeling. arXiv preprint wwxivv2205.05862. 2022.
Haoqin Tu, Bingchen Zhao, Chen Wei, and Cihang Xie. Sight beyond text: Multi-modal training enhances Ilms in truthfulness and ethics. arXiv preprint rrXiv:230.007120 2023.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.0000 2016.

---

Published as a conference paper at ICLR 2024
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In CVPR, 2015.
Lai Wei, Zihao Jiang, Weiran Huang, and Lichao Sun. Instructiongpt-4: A 200-instruction paradigm for fine-tuning minigpt-4. arXiv preprint arXiv:2308. /2067, 2023.
Yuxin Wu and Kaiming He. Group normalization. ECCV, 2018.
Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. NeurlPS, 2019. 3, 9, 13
Zumnan Xu, Zhihong Chen, Yong Zhang. Yibing Song, Xiang Wan, and Guanbin Li. Bridging vision and language encoders: Parameter-efficient tuning for referring image segmentation. In ICCV, 2023. 3
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint urXiv:2304. 14/78, 2023. 1,
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference event descriptions. TACL, 2014.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. TMLR, 2022. 2
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint /6199999033.6199 2023.
Bingchen Zhao, Quan Cui, Hao Wu, Osamu Yoshie, Cheng Yang, and Oisin Mac Aodha. Vision learners meet web image-text pairs. arXiv preprint arXiv:2301.07088: 2023. 2
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Bric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. 2.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et Lima: Less more for alignment. arXiv preprint arXiv:2305. 11206, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: En- hancing rision-languag understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023, 2,
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.I TRAINING DETAILS
For the first stage, we the learning rate 2e-3 for all variants. During the second stage, we search learning the learning rate from [2e-3, le-3, 6e-4, 3e-4, le-4, 5e-5, 2e-5, le-5, 6e-6, le-6, le-7] for all models and pick the best learning rate based on their performances on the CIDEr score on the Flickr 30k task.
According to our tryouts based on Flickr30k results in Table AI, the recommended learning rate for 7B scale is between 6c-4 to 2c-3, while on the 13B, the learning rate should be searched in the range of 3e-6 6e-5.
Table A1: Performance of MLLMs (LayerNorm-simp.) trained with different learning rates and scales the Plickr30t task.
Learning Rate
3e-6
le-5
3e-5
6e-5
MM-LLAMA2 7B
21.42
32.45
43.04
28.24
Learning Rate
6e-4
1e-3
2e-3
MM-LLAMA2 13B
37.35
46.88
44.15
A.2 INSIGHTS OF LAYERNORM TUNING
A.2.1 VISUALIZATION EXAMPLES OF LAYER SIMILARITIES
Lower similarities between different layers of the transformer indicates more expressive power (Pires et al., 2023). In section 5.2, we have shown the computed cosine similarity between layers on Vieuna model, here we show the layer similarities between layers on LLAMA2 and LLAMA2 CHAT models in fig. Al and fig. A2. It is clear that, LayerNorm tuning again allows the model to learn dissimilar layer representations, improving the expressive power the model.
A.2.2 GRADIENTS OF LAYERNORM
Visualization examples of LayerNorm gradients. fig. A3 and fig. A4, we present the gradients of the LayerNorm parameters during the training process. Similar to the one have shown in the main text, LayerNorm tuning demonstrates smaller gradient variance which is important for converging better local minimum (Xu al., 2019).
Proof of smaller variance in LayerNorm As stated in Sec. 5.3, deeper the network is, the variance of LayerNorm in the model should be naturally smaller (Xu et al., 2019). We first let
y (y1, //2 .... UN) be the normalized vector, meaning the mean and variance of y is 0 and 1, respectively. We can then formulate the standard LayerNorm as follow:
N
X
(x, -1)²
N
N 1=1
where x = T2, ...... and standard deviation xf. x.
..., n is the input vector and N is the dimension of x. and a are the mean
We first define 1N = (1,1, T)T, For calculating the gradients of the normalized vector we first
simulate the backward propagation regarding the loss &
Dp dy
do dy
de Fy
yy
1
N
(2)
Ox
0x
8x Oμ
8x do
0
N
N
13

---

Published as a conference paper at ICLR 2024
(a) Finetuning
(b) LayerNorm Tuning
Figure Al: Layer similarities between different LLM layers in (a) Finetuned and (b) LayerNorm- tuned MM-LLAMA2-7B.
(01, 02 .... aN) with mean à and standard deviation Das and ac (bi, be...., bx) with mean and standard deviation Dr- We set W1 = can verify that:
Here we define
Øé
äx
by
yy N
1x b we
W, 1/ (1- 1w1 + N
1 (1N
IN N
Ty N yT)
1N 1N 0
IN
1
(3)
Therefore, we can easily proof that Na x 1, W = 0, which means the mean of de should be zero
8x
Then we dive into proofing the variance of LayerNorm gradients should be small when the number of network parameters N becomes large.
D.
Σ 1=1
(a: a)
ag/N
1=1 (a), 02 .aN /N
[1 (b1, ,2,...,,N) /N
Wi (bI -b,b2 b, ...., . bN 6) + wib IN
Wi (91 b.92 b....... 9N 1] /N
S W? Σ (b, b)2/N
Since the projection matrix Wi is idempotent, we have W = W1. That is to say, when N is large enough, there stands D, S (7 yy + 1x EM (b, 2)2/N x 1/N2, consequence, when the network parameter N is large, the gradient variance of LayerNorm should be small.
N
14

---

Published as a conference paper at ICLR 2024
(a) Finetuning
(h) LayerNorm Tuming
Figure A2: Layer similarities between different L.L layers (a) Finetuned and (b) LayerNorm-
tuned MM-LLAMA2-7B CHAT.
Finetuning
462
LayerNorn Tuning
0.62
(a) Layer 21, Post LayerNorm
(b) Layer 21, Input LayerNorm
Figure A3: The gradients of both input and post LayerNorm in 21st layer of the MM- VICUNA the training proceeds.
0.54
Fineturing
A.M
100
1.6
LayerNorm Tuning
2.2
321
1.55
Layer 11, Post LayerNorm
(b) Layer 11, Input LayerNorm
Figure A4: The gradients of both input and post LayerNorm 11th layer of the VICUNA as the training proceeds.
15


Reviews:
[{"review_id": "2iegoKq8zE", "reviewer": "Reviewer_rJin", "summary": "Transforming LLMs, to Multi-Modal LLMs requires further finetuning on multi-modal data which can be pretty expensive.\n\nThere exist parameter efficient alternatives to fully finetuning models, and most popular ones are prefix tuning and LoRA. These methods a small amount of trainable parameters to the model.\n\nThe paper presents an alternative, which is not only parameter efficient, but also doesn't add new parameters to the model. The method simply trains the layer norm weights of the model associated with attention. The authors also demonstrate through multiple experiments that this is a strong alternatives that perform pretty well for multi-modal setup. There are two alternatives discussed: 1) only LayerNorm, and 2) LayerNorm along with vision language connector, word embeddings and output head. While #1 is more efficient, #2 is more performant.", "questions": "Do we have any results comparing it with LoRA for non-multimodal tasks?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Strong results while being much more parameter efficient compared to other similar techniques like LoRA, Attn QV Proj, and Attn MLP.\n- The two variants present a tradeoff between parameter efficiency and quality. This could also be useful when the finetuning dataset is very small and using LayerNorm only variant could provide a less overfitting alternative.\n- While LoRA and other methods add additional parameters (adapters), this method does not need it and hence does not require handling them separately during training/inference to merge them with the original parameters.", "weaknesses": "- The method is claimed to be useful only for MultiModal tuning, and might not be comparable to LoRA or other methods for LLM only finetuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XQU84K0Jjl", "reviewer": "Reviewer_qMBg", "summary": "This paper introduces a strategy for transforming Large Language Models (LLMs) into Multi-Modal Large Language Models (MMLMs) by tuning LayerNorm in attention. The authors argue that this approach can improve the performance and efficiency of multi-modal tasks, such as image captioning and visual question answering. The paper presents empirical evidence to support this claim, showing that the proposed method outperforms existing approaches on several benchmark datasets. The authors also discuss the role of LayerNorm in adapting LLMs to the multi-modal domain and improving the expressive power of the model. Overall, the paper provides a novel and effective solution to the challenge of multi-modal learning, which has important implications for natural language processing and computer vision.", "questions": "See above.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces a novel method to transform large language models into multi-modal language models by tuning LayerNorm only, which yields strong performance. The idea is straightforward but effective.\n2. The experiments are solid and the results support the idea of the authors.\n3. The paper is well-written and well-organized, and the comprehensive analysis is interesting and attractive.", "weaknesses": "1. It is better to detail the LayerNorm tuning way in the paper and compare it with other methods.\n2. Although existing experiments have demonstrated the effectiveness of the method, I suggest evaluating the model's capabilities comprehensively on more data, such as VSR, HM...", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lwYc7OBZAe", "reviewer": "Reviewer_itTB", "summary": "The paper introduces an efficient approach to finetuning Large Language Models (LLMs) for multi-modal tasks, resulting in Multi-modal Large Language Models (MLLMs). By finetuning only the LayerNorm's weights within attention blocks and making LayerNorm the sole trainable component, the model achieves superior performance, especially when finetuned with conversational data. This LayerNorm-focused strategy, combined with the right data type, leads to significant performance improvements on benchmarks while using fewer resources, highlighting the potential of LayerNorm tuning for multi-modal tasks. The experiments are thorough and well-executed, yielding insightful results.", "questions": "1. In the experiments, there seems to be a performance variance between different LLMs, especially between the human-aligned 7B and 13B models. Could the authors elaborate on the potential reasons for this inconsistency?\n2. While the paper provides empirical and intuitive explanations, could the authors shed light on any theoretical insights or hypotheses they might have about why tuning only the LayerNorm yields such significant results?\n3. Beyond LoRA, are there other recent advancements or techniques that have shown promise in multi-modal learning or the finetuning of Large Language Models? It would be beneficial to compare your approach with these newer baselines to provide a more current context.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper introduces an efficient way for transforming LLMs into MLLMs by only finetuning the LayerNorm within each attention block.\n2. The paper provides empirical results showing that the proposed LayerNorm tuning method can yield competitive performance, often surpassing other tuning approaches like full parameter finetuning or LoRA.\n3. The LayerNorm tuning method offers substantial benefits in terms of efficiency, reducing trainable parameters and GPU memory usage.\n4. Beyond empirical results, the paper explores deep into understanding the role of LayerNorm in adapting LLMs to the multi-modal domain, enhancing the model's expressive power.", "weaknesses": "1. Inconsistency Across Different LLMs: The experimental results appear to vary depending on the specific LLM used. While the authors highlight that human-aligned LLMs tend to yield better results, there seems to be inconsistency, especially when comparing the human-aligned 7B model with the 13B one. This raises questions about the robustness of the proposed method across different model sizes and configurations, making the claim less convincing.\n2. Lack of Exploration of Combined Training Strategies: The paper primarily focuses on the LayerNorm tuning strategy. A more comprehensive exploration, combining this with other training strategies, would provide a richer understanding of the method's potential and its interaction with other techniques. Such an exploration could offer more actionable insights for practitioners looking to adopt the proposed method.\n3. Empirical and Intuitive Explanations: The paper's explanations for the observed results lean heavily on empirical evidence and intuitive reasoning. A deeper theoretical analysis or a more rigorous exploration of the underlying mechanisms would strengthen the paper's claims and provide a more solid foundation for the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "82yNHi0Gu8", "reviewer": "Reviewer_6WJ1", "summary": "This paper proposes an interesting idea of tuning LayerNorm for PEFT, yielding impressive results compared with LoRA and tuning other parts in LLMs. Five multi-modal tasks were tested to show the effectiveness of their proposed method. 20% better performances, 41.9% reduction of trainable parameters and 17.6% decreasing of GPU memory usage were reported. The contributions of this paper include, (1) tuning LayerNorm with a simple and efficient direction, and (2) significantly better results and resource/cost reduction on 5 tasks.", "questions": "1. any further detailed results on combining LayerNorm tuning with other types of PEFT? such as those listed in https://github.com/huggingface/peft?\n2. so how exactly do you select the hypermeters during tuning? learning rate and training epoch, any detailed hints on applying your method to new tasks and new datasets?\n3. can you show some examples that tent to be better or worse when applying LayerNorm?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. simple and efficient tuning of LayerNorm for LLMs;\n2. strong results reported for 5 multi-modal tasks.", "weaknesses": "1. simple layernorm finetuning is reported to be including less trainable parameters and sensitive to a list of hyperparameters such as learning rate and training epoch, making it difficult to find the optimal performances in a quick way;\n2. not clear yet of how other researchers combine LayerNorm with other PEFT methods - are the results better or not?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)