An Empirical Study of Multimodal Model Merging
Authors: Yi-Lin Sung
Linjie Li
Kevin Lin
Zhe Gan
Mohit Bansal
Lijuan Wang
Venue: EMNLP 2023 Findings (2023)
Keywords: model merging; vision-and-language

Abstract: Model merging (e.g., via interpolation or task arithmetic) fuses multiple models trained on different tasks to generate a multi-task solution. The technique has been proven successful in previous studies, where the models are trained on similar tasks and with the same initialization. In this paper, we expand on this concept to a multimodal setup by merging transformers trained on different modalities. Furthermore, we conduct our study for a novel goal where we can merge vision, language, and cross-modal transformers of a modality-specific architecture to create a parameter-efficient modality-agnostic architecture. Through comprehensive experiments, we systematically investigate the key factors impacting model performance after merging, including initialization, merging mechanisms, and model architectures. We also propose two metrics that assess the distance between weights to be merged and can serve as an indicator of the merging outcomes. Our analysis leads to an effective training recipe for matching the performance of the modality-agnostic baseline (i.e., pre-trained from scratch) via model merging. Our method also outperforms naive merging significantly on various tasks, with improvements of 3% on VQA, 7% on COCO retrieval, 25% on NLVR2, 14% on Flickr30k and 3% on ADE20k.

An Empirical Study of Multimodal Model Merging
Yi-Lin Sung', Linjie Li", Kevin Lin", Zhe Gan Mohit Bansal', Lijuan Wang"
UNC Chapel Hill, "Microsoft, "Apple AI/ML
(ylsung, mbansal fecnsal@ess... unc. edu (Lindsey.Li, keli, Lijuuanweemicrooofft.cmm zhe. ganeapple. com
enables generate the multi-task solution with- out synchronous training and is especially useful arithmetic) fuses multiple models trained on different tasks to generate multi-task solution. The technique has been proven successful in previous studies, where the models are trained similar tasks and with the same initializa- tion. In this paper, expand on this concept to a multimodal setup by merging transformers trained different modalities. Furthermore, we conduct our study for novel goal where we can merge vision, language, and cross-modal transformers of modality-specie specific architec- ture to create a parameter-efficient modality- agnostic architecture. Through comprehensive experiments, we systematically investigate the key factors impacting model performance af- ter merging, including initialization, merging mechanisms. and model architectures. We also propose two metries that assess the distance between weights to be merged and can serve an indicator of the merging outcomes. Our analysis leads to an effective training recipe for matching the performance of the modality- agnostic baseline (i.., pre-trained from scratch) via model merging. Our method also outper- forms naive merging significantly tasks, with improvements 3% on VQA. 7% COCO retrieval, 25% NLVR2, 14% Flickr30k and 3% on ADE20k.
Abstract
Model merging (e.g., via interpolation task
when the models for merging are already trained and available online. However, current literature has only applied model merging to models trained on similar tasks even the same dataset, which inspires us to explore whether model merging will also work when the models are trained different modalities such language versus vision.
We study this challenging problem with novel goal of obtaining an effective and parameter- efficient modality model by merging the different modality transformers of a modality- specific model. vision-language (VL) do- mains (Driess et al., 2023; OpenAl, 2023), many mrodality-specifi models (Tan and Bansal, 2019; Chen et al., 2020; Radford al., 2021; Li et al.. 2021; Dou et al., 2022) employ dedicated trans- former encoders encode vision/language inputs independently, and on top of which additional transformer layers are used for multimodal fu- sion. On the other hand, the convergence using transformer-bast architectures for various single- modality tasks has prompted researchers to adopt a single modality- transformer (Akbari et al., 2021; Wang et al., 2021; Kim et al., 2021; Lu et al.. 2022; Jaegle al., 2022) encode different modal- ity inputs and learn cross-modal interactions simul- tancously. While modality-specifff architectures usually perform better, modality- ones are simpler and more parameter-efficient. Both ar- chitecture designs offer their unique benefits and have distinct use cases. However, obtaining the benefits of both these setups can be costly as it requires independent training. Therefore, in this paper, we explore how leverage well-trained modality-specific model and develop modality- agnostic model from it via model merging, with a goal of achieving similar performance to the modality aggossic baseline which is pre-trained
various
1 Introduction
Model merging (Singh and Jaggi, 2020; Matena and Raffel, 2022: Ainsworth al., 2022; Jordan al., 2022), investigates the technique of merging (e.g., via linear interpolation, task arithmetic (II- harco et al., 2022), RegMean (Jin et al., 2023)) two models trained on different tasks while preserving their original capabilities or even outperforming multi-task training (Li et al., 2022). This technique
'Work done at Microsoft
OOur code is available at https: //gi thub. cotp.o/oc///l/ggg
v1 vl-merging

---

Seed Pre-training
Pre-training (for Modality-peciffcc Architecture)
Pine- tuning
Moddality-agnooii transfrrmer
transfermrr
(N-M
K
Mx
VI. transformur
Kvi
tranaforner
19
We!
Kyt.
Architectures
We
Merging
(N-M)N
No
Castom
Custom
Custom
Merging Methods
LN
Interpolation
W
Modality vectors
al
FFN
Interpelation
Modality Amithmetic
$1
Ar'
W
RepKeon
Attn
Alte
Altin
Attn
GTTT sp
Figure 1: Overview of framework for multimodal merging. The boxes with red boundaries illustrate the factors (seed pre-training, merging methods, architectures) that are crucial for the performance of merging. We use blue-filled, green-fille and orange- fille- colors to denote vision, language, and VL modalities, respectively.
from scratch. 2 This approach eliminates the need for training the latter architecture.
3% on ADE20k), to match the performance of the modality-agnostic model obtained by independent training, demonstrating a novel way to construct universal architecture that can process different modalities simultaneously. We also demonstrate the merged weight maintains the performance on unimodal tasks (we focus on vision tasks), and the proposed approach generalizes broadly to another backbone model (ViT-Tiny/16).
To this end, we conduct systematic study of model merging on VL models, analyzing the key factors impacting the merging results, including ini- tialization, merging mechanisms, and model archi- tectures. Through comprehensive experiments, we find that initialization is crucial for model merging, and using pre-trained weights on VL data is help- ful. We benchmark three merging methods: inter- polation, modality arithmetic, and RegMean, and we find that interpolation performs competitively while being simple and efficient among compared merging methods. Lastly, we examine the merging different modality-specific architecture variants, including saree-weeight architectures with custom attention layers, layer normalization layers. We find that the ar- chitecture without sharing performs the best be- fore merging and attains on-par performance the modality-agnostie baseline after merging.
Related Work
Different Types of VL Architectures. VL mod- els aim generate outputs based reasoning over both text and visual inputs. Because the model takes two sources of inputs, there are different de- sign choices to fuse the information, and hence yielding several mainstream designs in VL mod- eling: (1) modality-specifie transformer (Tan and Bansal, 2019; Li et al.. 2023; Driess al., 2023), (2) modality-agnoste transformer (Akbari et al., 2021: Wang et al., 2021; Kim et al., 2021; Agha- janyan al., 2022), and (3) dual transformer (Rad- ford et al., 2021). Modality-specifff transformers use language transformer and a visual encoder to extract modality-specific representations and feed them into cross-modal transformer to learn image- text fusion information. Some modern architec- tures (Driess et al., 2023; Sung et al., 2022b; Cho et al., 2021; Yu et al., 2022; Wang al., 2022b: Sung et al., 2022a) the language transformer for both language and cross-modal modeling. The dual transformer a special case of modality-speciffi ar- chitecture, where the cross-mi modal fusion is realized
networks (FFN), or
Moreover, we investigate the correlation be- tween the distance of two weights to be merged and the merging results, and propose two distance measures that can potentially indicate the merging results before training. Overall, our findings serve as a training recipe which can significantly improve the simple merging of the poodality-speciffe model (by around 3% on VQA, 7% on COCO image-text retrieval, 25% on 2 14% on Flickr30k and
'When using model merging, we sometimes refer to the -peccic model as the model before merging, and the modality-agnostic one as the model after merging.

---

via simple dot product, hence more efficient for
2021; Lu et al., 2019; Li et al., 2019; Su et al..
fast retrieval. aggosit transformers use one single transformer to process multi-modal data. These methods can be viewed as early fusion mod- els, and the transformer learns modality-specific and cross-modal information simultancously. The design of -agnossic architectures is rela- tively simple, and they are more memory-efficiue because they only have one backbone.
2020). We denote their weights for layer as W W (i 1,2,.... 12) and W.Sd (i 11, 12). Please refer to Figure 2(a) for illustration. This design is inspired by VLMo (Wang et al., 2022a, 2023), which has different routes for different tasks in for- warding, For visual question answering and other classification tasks, we use the first (N M) lay- of vision and language transformers to extract the image and text features and feed the concate- nation of them to the M multimodal fusion layers to get the final representation. We demonstrate this archtectuuremmechnnisn in the upper part of Figure and the right part of Figure 2(b). For image-text retrieval, we directly use the whole N layers of vision and language transformers to ex- tract image and text representations for matching (left part of Figure 2(b)). Only using unimodal en- coders to extract features allows us to compute the dot product between features efficiently because it eliminates the need to forward the input pairs to cross-modal layers. Please refer to VI.Mo for more details. For casier merging, we initialize all the transformers from the same initialization' regard- less of the modalities they used for, and we use W to denote the initial weight for the ith layer. Modality-
Model Merging. The de facto approach in recent model development centers around the "pre-train then-fine-tund paradigm. Due to the substantial resources required to train robust pre-trained mod- els, the available repertoire is relatively limited, and many subsequent models are fine-tuned from this fi- nite selection. Recent research has discovered that models derived from the same pre-trained weights can be successfully merged into single model. This merge can either improve single-task perfor- mance when the models are designed for the same task (Wortsman et al., 2022a) or create a multi-task solution when they target different tasks (Ilharco et al., 2022; Wortsman et al., 2022b; Don- Yehiya et al., 2022). Various strategies for merging mod- els have been proposed, including linear interpo- lation, task arithmetic (Ilharco et al., 2022), and RegMean (Jin et al., 2023). However, if models do not share the same pre-trained weights, weight permutation might be required before merging (En- tezari et al., 2021; Ainsworth et al., 2022; Jordan et al., 2022). Our work leverages model merging to- wards the innovative goal of generating efficient modality-agnostic model from modality-specific models, thus eliminating the need for re-training.
transformer adopts one set of parameters to jointly learn vision, language, and cross-modal information, that is, W W for i 1, 2,.,... 12 and WT' W W.ad for 11 and 12. The forward mechanism of this model is the same as the modality-speciite architecture. Shared-weight modality-specific transformer is in-between the two aforementioned extreme types of architecture, i.e., with partial modality and modality-specific modules. Specifically, we ex- plore three variants of the shired-weight architec- tures with: (1) custom attention layers, (2) custom FFN, and (3) custom layer normalization layers. The illustration of them is shown in Figure 1.
3 VL Transformers
A VL model takes an image-text pair as input and is designed to tackle various VL tasks with mini- mal architecture changes. The input image is first divided into patches, flattened into sequence, and then fed into the vision embedding layer to obtain visual features. The input text is processed by the language embedding layer to obtain a sequence of textual features. The features are then jointly processed by the VL model, for which we explore different architectures detailed below.
Model Merging for VL Architectures
Our objective is to combine various transform- ers within the a single modality-agnosse
architectures into transformer. We do not merge the embedding layers and keep them modality-ppecific as they have different architec-
-sdelity-specific transformer contains two N- layer (N=12 in this paper) transformers, special- ized for encoding image and text inputs, respec- tively, and an additional M (=2 in this paper) trans- former layers for multimodal fusion (Akbari et al.,
We have explored merging the model that uses BERT to initialize the language transformer and BEiT for the other transformers, but the average of IR and TR is only 49.9 on COCO even if we permute the weights (80.6 before merging).

---

(a) Our modality-specific architecture
Usage for different tasks
tained by subtracting the initial weight from the tuned weight (illustrated in Figure 1). Then, the merged weight containing multi-task information is computed by adding all task vectors to the initial weight. In this paper, we extend this idea to learn- ing such vectors for different modalities, dubbed as modality vectors. We define the modality vec- tors the difference between the weight for a specific modality and the initial weight, for exam- ple, T. W W and the merged weight is WO + + +mm T." for the first (N M) lay- and is W + + >=Eme(udan T." for the top M layers, and À is the scaling parameter.
VOMTTN
M
Image text
Image test
image text
VI.
Viston: langeage, and Vesonslangung trandfrmer
Figure 2: VLMMo-like -ppelit architecture. It has an N-layer vision, N-layer language, and M M-layer cross-modal transformer. For classification tasks and ITM in pre-training. uses cross-modal layers to jointly process the image and text imput. For retrieval tasks and ITC in pre-training, it only forwards inputs through vision and language encoders. For modality-agma aggnosicc architecture, the V, I., and VI. transformer layers share a single set of parameters while keeping the forward mechanism same.
RegMean. Different from the previous two meth- ods, RegMean (Jin et al., 2023) finds closed- form solutions for the weights of linear layers and evenly interpolates other weights (layer nor- malization, bias terms). Assume X!" is the in- put for the 7th layer the transformer for m modality, RegMean finds the merged weight W, that minimizes the distance of features before and after merging, that is, Eme(u..n)) IW; X!" The closed-fon form solution of
tures and dimensions. For shared-weight architec- tures, only merge the weights of custom layers since the other parts are already modality-agnosti In modality-speciffic VL model, the transformers receive inputs from different modalities and work together to accomplish the target task. We merge such transformers to allow a single transformer that can handle different modalities concurrently. This situation deviates from previous studies on model merging (Li et al., 2022; Matena and Raffel, 2022; Ainsworth et al., 2022), where the transformers (in these previous works) be merged take input from the same modality. In the following, we describe our process for merging the VI. architectures.
W xm 12. W; is (Emc(ud)oo) - Emm((udod (GTT (GT" where G.Th = >x:" X.M. Morcover, Jin et al. (2023) pointed out that introducing a scalar y to decrease the non non-diagonal elements of G (G G + (1 - diag (GT")) and replace G by Gi"' in the above solution form leading to stabler results.
4.2 Evaluation Pipeline
Unlike previous model merging literature that merges models fine-tuned on downstream tasks, 4 we merge the transformer weights for different modalities after VL pre-triniing (Gan al., 2022) to explore more effective moddllityagnoosie pre- trained model for fine-tuning. We follow the same pipeline to evaluate different merging methods as shown in the upper part (gray-filled boxes) of Figure 1. First, we pre-train the modality- specific models for KYL steps on VL corpra in an end-to-end manner. Next, we merge the weights of meddality-speciff transformers to construct a modality- model via the methods described in Section 4.1. Lastly, we fine-tune the merged model on downstream tasks. Note that when re- porting the performance of modality tectures, weight merging is not needed. Hence. we
4.1 Merging Methods
We explain the details of the three merging meth- ods, and their illustrations are shown in Figure 1, Interpolation. For simple interpolation, the vi- sion, language, and cross-modal transformer layers are element-wise weighted-averagel with given ra- tios. We control the ratio a between the vision and language transformer and set the ratio of the cross modal transformer constant value ( for the layers that have three modalities). The merged weights are (1-+aa for the first (N-M) layers while they are for the top M layers. Modality Arithmetic. The second method is inspired by the concept of task vectors (Ilharco al., 2022), which indicate the directions to improve the tasks' performance from the initial weight. For a given task, the task vector is ob-
(
archi-
*We find that directly merging a fine-tuned modality- specific model performs poorly: the average of IR and TR is 50.5 on COCO (80.5 without merging).

---

VQA
MSCOCO
and referred the seed phase. We then use this weight initialize different modality transformers of the modality-specific model before the VL pre- training step and evaluate the model performance as described in Section 4.2. For a fair comparison, we ensure that all compared methods go through the same number of total pre-training iterations on the same VL corpora, i.e., Keeed + KvL 200k.
Method
Acc
Arg
Avg
Modality-anoostic architecture
73.94
83,91
72.23
(MAA) VL pre-trained from scratch
Modality-pecciffe architecture
75.14
86.46
74.67
(MSA) VI. pre- trained from scratch
Morging from MSA by interpolation (merging bascline)
70.89
77.38
65.43
Table I: Results on directly merging the modality- specific architecture, without seed pre-training, and the comparison with the modality-annostic baseline. The merging result without seed pre-training has a significant performance gap to the other approaches.
Note that VL pre-training can be conducted on different model architectures, including both modality- models and modality-agnostic models. Also, seed pre-training for a modality- specific model is essentially equivalent to VL pre- training on modality -agiostic model. We use different terms for seed pre-training and VL pre- training to distantiate the different training stages for training the modality-specifie model.
directly finetune the models after pre-training.
We pre-train our models on the standard 10M image-text corpus from COCO (Lin et al., 2014), Visual Genome (Jin et al., 2022), SBU (Ordonez et al., 201 I), and Conceptual Captions (Sharma et al., 2018) with three popular objectives: image- text contrastive (ITC) learning (Radford et al., 2021), image-text matching (ITM) (Li et al., 2021), and masked language modeling (MLM) (Devlin et al., 2019). During finetuning, we evaluate sev- eral VI. and vision downstream tasks. The details of the experimental setup are shown in Section 5.
Experimental Details
We describe the experimental setup for our models and training in this section and more details about hyper-parameters for fine-tuning, computational usage, and the number of parameters of our models are included in Appendix A.
Architectures. We use ViT-B/16 (Dosovitskiy et al., 2021) (which has a similar architecture BERT (Devlin al., 2019)) throughout our experi- ments. The model has 12 transformer layers, and the dimension of the model is 768.
4.3 Seed Pre-training
For our first pilot investigation, we follow the above evaluation pipeline to pre-train a modality-peecific model initialized from BEiT (Bao et al., 2022) for 200k iterations, then perform merging with interpolation (a 0.5), and lastly fine-tune on downstream tasks. We name this simple method as the merging baseline. Table shows that the performance drop from merging is quite signifi- cant, and the merged model lags behind our goal of matching the modality-agnostt baseline, which is pre-trained from scratch. We hypothesize that the model weights of language and cross-modal transformers after VL pre-training are not located in the same basin (Entezari et al., 2021) as the vi- sion transformer, because the initial BETT weight only contains visual information.
VL Pre-training. As described in Section 4.2, we adopt the standard 10M image-text corpus (4M images) and utilize ITC, ITM, and MLM for VL pre-training. The overall pre-training objective is the weighted combination of the three aforemen- tioned loss functions. ITC maximizes the pair-wise positive matching and minimizes the matching be- tween any negative pairs within the training batch. Similar to ITC, ITM predicts whether the image- text pair is positive negative, by modeling it binary classification. For MIM, we mask out 15% of text tokens, and the model required to recon- struct the missing tokens with multimodal context. The overall pre-training objective is the weighted combination of the three aforementioned loss func- tions, and we find that using a smaller weight (0.25) for MLM empirically leads to better results. The forward mechanism of ITC is the same the trieval task, and of ITM and MLM is the same as VQA (Figure 2(b)).
Hence, in order to ensure that different trans- formers remain situated within common basin and avoid significant deviation from one another after VL pre-training, next turn to use a sin- gle pre-trained weight containing vision, language. and cross-modal information for initialization, by pre-training a single modality-agnostii transformer (initialized from BEiT) on VL corpora for K Kyeed steps. This approach is also used by Li et al. (2022)
The image input size is set as 224 x 224 and the

---

75:24
74.77
ICOCO) Average of 136
50:36
80:49
74-
73.00
80.0
72.74
72.84
73.73
[VQA] [VQA] Accuracy
72
70.29
The merging baseline Table
75.0
70-
71.1
The merring baseline Table
70.0
Before merging
Before merging
Interpolation merging la
0.5]
Interpolation 0.5)
Modallty-ggnsste baseline
Modellty-agnoste baseline
65.0
50
100
150
200
100
150
200
Iterations of the seed pre-training
Iterations of the seed pre-training
(a) VQA
(b) COCO retrieval
Figure The effect of seed pre-training's duration on the merging results. Increasing the iterations of seed pre-training improves the merging outcomes, and the optimality happens at 100k iterations as more iterations in seed pre-training reduce the iterations of VL pre-training, and limit the learning the modality speccfff model.
Interpolation
(b) Modality arithmetic
(c) RegMean
for interpolation
VQA
COCO
for modality
VQA
COCO
VQA
COCO
uritimetic
RegMean
Acc
Avg TR
Avg IR
Acc
Avg TR
Avg IR
Acc
Avg TR
Aug IR
(l. weight only)
68.58
15.25
62.64
(no VL PT)
73.44
83.27
70.84
74,14
83.73
72.03
0.25
0.25
72.20
0.5
73.59
83.64
71.77
0.5
73.59
83.64
71.77
0.5
74.13
83,80
72.18
0.75
73.91
83.83
72.30
0.75
72.49
80.22
68.51
0.75
74.07
83.80
72.18
oelg)
82.75
69.49
1.0
74.15
83.71
72.25
Table 2: The ablation of using different hyper-parameters for (a) interpolation, (b) modality arithmetic, and (c)
RegMean. Note that interpolation with a = 0.5 the same as modality arithmetic with 0.5. L, V, and PT are the abbreviations of language, vision, and pre-training. Interpolation works well when 0.5 and 0.75. With additional computations, RegMean can obtain stable performance.
text input length is 40. The text input is encoded by bert-base-uncased (Devlin al., 2019) tok- enizer, which has 30522 vocabularies. During train- ing, we apply RandAngment (Cubuk et al., 2020) on image inputs. For all pre-training experiments, we set the batch size as 1024, the total number of iterations as 200k (if no seed pre-training), with 2500 for warmup. Adam optimizer with the peak learning 2 is adopted for training.
6 Experimental Results
In this section, we investigate the key factors im- pacting the merging performance on VQA and COCO. Recall that our goal is to improve the merg- ing baseline in Table to match the performance of the modality-agnostic model and keep the per- formance of the modality-specific model (before merging) remain similar.
6.1 Effect of Seed Pre-training
Fine-tuning. We use VQA (Goyal et al., 2016),
COCO image-text retrieval (Lin al., 2014).
We first validate the effectiveness seed pre- training by varying Kseed in Figures 3a and 3b. Training more steps in the seed phase naturally leads to worse performance for modality-specific models as fewer training steps are dedicated to the learning of modality-specific information in custom modules. However, the merging perfor- mance is stronger for longer seed pre-training, as the initial weight may absorb more language and cross-moda information, making the weights more likely to locate in the same basin before merg-
NLVR² (visual reasoning) (Suhr al., 2019),
Flickr30k (image retrieval) (Plummer al., 2015).
ImageNet- Ik (image classification) (Deng et al., 2009) and ADE20k (semantic segmentation) (Zhou et al., 2016) as downstream tasks. In evaluation, we report the accuracy for VQA and NLVR², top-1 accuracy for ImageNet, and mIOU for ADE20k. We use the average of top-1, top-5, and top-10 recall (TR), and image recall (IR) for retrieval tasks. We report the results for each method with a single run. Appendix A shows other details of experiments.
Note that the models with Kaeed 0 is the same as what have heen reported in Table 1.

---

74.62
86.56
74
73.01
74,2974.19
24.01
[COCO] Average of and IR (16)
80.0
78.06
75-21
77.053741
77.5
(VOA] Accurary (96)
72
71
10.80
70.0
merging
67.5
merging
Interpolation merging (a 0.75)
interpolation merging (a 0.75)
Modellty--gnose baseline
Modaiity yeggleitt--nniiiii baselne
merging baseline
The ginging basel ine
Custom Everything Custom Atth
Custom FIN
Custom LN
Oustom Everything Custom Altn
Custom FFN
Custom LN
Architecture
Architecture
VQA
(b) COCO retrieval
Figure 4: The effect of weigh architectures the merging results. The modality-seecifie architecture which has entire custom modules performs the best before merging and attains on-par performance with the modality- agnostic baseline tpre-trained from scratch) after merging.
ing. Morcover, the optimality for merging happens when Kseed Kv 100k, which also leads to competitive modality-specifie model that performs comparably the without seed pre-training. Therefore, we adopt this the default recipe for all remaining experiments.
largely retain the performance after merging, how- ever its performance before merging is not com- petitive, only comparable to the aggnstic baseline (whereas ideally, want the performance of the modality-specific model before merging be higher than the ing attention layers or FFN has mixed results on VQA and COCO retrieval, as improves on one task while hurts the other. Notably, the modality- specific architecture with entirely independent modules performs the best without merging and attains on-par performance with the modality- agnostic baseline after interpolation.
model). Shar-
6.2 Comparison between Merging Methods
We compare different merging methods and ab- late their
in Table 2. Table 2c
shows that RegMean performs strongly in both tasks (especially VQA) and is robust y. Table 2b shows that reducing the length of the modality arith- metic leads better downstream performance. In Table 2a, we observe that vision weight is more important for our downstream tasks, as only us- ing vision weight (a = 1) achieves much better results compared to only using language weight (a = 0). Comparing different merging methods, interpolation (with the ratio 0.75) performs as competitively as RegMean without additional computation and storage for the gram matrices GT and is more effective than modality arithmetic. Intuitively, the best performance with an interpo- lation ratio of 0.75 is somewhat similar to using different weights for tasks to mitigate the negative transfer in multi-task learning (Liu et al., 2019).
6.4 Final Recipe
Overall, as an initial study on merging modality- specific models, our empirical results reveal a recommended takeaway recipe: (1) select a pre- trained weight that has been trained on all modali- ties, (2) use interpolation with ratio around 0.75 (or 0.5) effective initial approach, employ RegMean if additional computational resources deemed acceptable, and (3) employing modality- specific architectures with entirely independent modules can attain satisfactory merging outcomes.
6.5 Generalization Diverse Tasks and Architectures
6.3
Ablation on Shiged-weight Architectures
In this section, we further apply our merged model to diverse spectrum of tasks and architectures and compare approach with the modality-Spee Architecture (MSA), Aodality-Agnostic Architec- ture (MAA), and the Merging Baseline (MB) (men- tioned in Table and Section 4.3).
Lastly, explore applying model merging different architectures described in Section 3, in- cluding shared-weight architectures with custom attention layers, FFN, or layer normalization lay- ers. Results are shown in Figures 4a and 4b. The model with custom layer normalization layers can
More Tasks. We include additional evaluation on

---

NLVR
Flick130k
ImageNet- Ik
ADE20K
VQA
NLVR
COCO
Flickr30k
Methods
Methods
Acc
Avg TR
Avg IR
Top-1 Acc
mIOU
Acc
Acc
Avg TR
Avg IR
Avg TR
Avg IR
MSA
79.71
95.10
87.37
83,25
49.85
MSA
68.36
68.42
71.75
59.98
80.73
70.14
MAA
78.39
93.10
84.57
82.78
48.65
MAA
62.25
64.90
63.17
52.08
72.50
62.27
MB
52.55
77.63
68.62
82,90
46.80
MB
55.52
$1.07
51.9%
42.43
17,63
16.09
Ours
77.54
91.53
82.64
83,00
50.14
Ours
60.02
$2.89'
63.40
52.82
68.40
59.53
Table 3: Comparison between our method, Modality- Specific Architecture (MSA), Moddllit-AAgosii Ar- chitecture (MAA), and Merging Baseline (MB) across NLVR², Flickr30k, ImageNet- Ik, and ADE20K. The top results are highlighted in bold for both merged and non- merged model categories. Our method aligns closely with MAA for NLVR and Flickr30k and sur- passes on ImageNet- and ADE20K.
Table 4: Comparison between our approach and base- lines using ViT-Tiny/16 on four VL tasks. Our method demonstrates stronger performance compared to the merging baseline. "We found using an interpolation ratio of 0.5 can boost the performance to 60.94 and cut the gap to MAA significantly.
Our findings demonstrate that our technique significantly outperforms the merging baseline. Furthermore, we observe a larger performance gap between our approach and the mogality-agnostic baseline when employing the ViT-Tiny/16 back- bone, and this phenomenon is aligned with past research (Ainsworth et al., 2022), which shows larger (wider) models exhibit greater resilience to performance drop when merging.
NLVR and Flickr30k well ImageNet-Ik and ADE20K cover more VL and vision tasks. To make direct comparison, we do not apply the inter- mediate fine-tuning (used in VLMo) for Flickr30k, ImageNet- Ik, and ADE20k. For modality-speciffi architectures, we only fine-tune the vision encoder to the vision tasks. The results of our experiments are presented in Table 3. Notably, our approach, which involves seed pre-training and interpolation with ratio of 0.75, exhibits significant improve- ments over the naive merging approach in both 2 and Flickr30k tasks. However, we have observed that the performance gap between our ap- proach and the modality-agnostic architecture is somewhat larger for NLVR" and Flickr30k com- pared to the VQA and COCO tasks, where our results closely align with the baseline performance (Figures 4a and 4b). We hypothesize that the merg- ing outcomes could be better if the domain shift between pre-trained and downstream tasks were re- duced, considering that images of VQA and COCO are used for VL pre-training. In the results of Ima- geNet and ADE20K, our merging approach not only matches but also surpasses the baselines. These results affirm that the merging process does not compromise the representational ca- pacity of the individual modalities.
7 Analysis
In the preceding experiments, we highlight three factors that influence the outcomes of merging. This sparked a curiosity regarding the existence of potential metrics capable of predicting or ex- plaining the quality of these merging results. One intuitive hypothesis is that the distance between the weights to be merged (vision and language weights in case) might play an important role, the performance drop caused by merging should be 0 if the distance between them is O, and vice versa, the drop should be considerable if the distance large (the weights are not in the same basin). To test the hypothesis, we apply various distance measures on the vision and language weights of modality- specific models and compute the correlation be- tween the distance and the performance drop after merging assess the influence on merging results.
Metrics Definition. Four metrics are chosen to measure the distance between vision and language weights of modality-specific models, including the L. distance, cosine dissimilarity (equivalent to one minus cosine similarity), soft sign dissimi- larity (SSD), and truncated soft sign dissimilarity (TSSD), where the last two are inspired by Yadav et al. (2023). To compute the distance, we first flatten all the weight matrices from vision and lan- guage transformers and concatenate them respec- tively to form vision and language weight vectors
More Architectures. We also conduct experiments on another backbone model, ViT-Tiny/16 (Dosovit- skiy et al., 2021). Note that it is not trained by the BEiT unsupervised objective but is trained by the supervised classification objective. In this experi- ment, we pre-train the model for 100k steps instead of 200k steps, and for our merging approach, we use 50k steps for both seed pre-training and VL pre- training. We compare our approach to baselines on four VL datasets and display the results in Table 4.

---

w and w Note that do not include weights that are not mergeable, such as embedding layers. For L2 and cosine dissimilarity, we follow the standard ways to compute them, namely, Ex W
seed
Metrics
Corr.
0
50k
100k
150k
L2
580.6
496.8
388.5
284.6
0.881
Cosine
0.371
0.264
0.163
0.098
0,938
SSD
0.198
0.161
0.125
0.104
0.939
and w"ww" "[lWw
TSSD
0.066
0.037
0.015
0.006
0.971
Yadav et al. (2023) show that model merging might result in information loss due to interfer- ence from redundant parameter values and disagree- ment on the sign of given parameter's s values across weights to be merged. Motivated by this, design two metrics reflect the disagreement between signs and the interference of redundant parameters. Unlike the method used by Yadav et al. (2023), soft sign dissimilarity (SSD) considers the two re- quirements in soft manner, and the mathematical definition is as follows,
Performance drop on COCO retrieval
9.16
4.50
2.80
2.60
We apply four metries to modality-speciffe models trained on different lengths of seed pre-training, The bottom two rows indicate the performance drop on COCO retrieval after model merging, and the right-most column is the Pcarson correlation between these metrics and the performance drop. The result shows that TSSD can best reflect the merging outcomes.
Table 5:
Results. We apply the aforementioned metries to modality-speciffe models using different seed and VL pre--raaiing iterations and show the results along with the performance drop after merging on COCO in Table 5. We observe that the distance in- deed relates to the performance drop as their trends aligned. To investigate what metric best reflects the merging outcomes, we calculate the Pearson correlation (Freedman al., 2007) between the performance drop and each distance metric. The results in Table reveal that the L2 distance is the least correlated to the performance drop, in com- parison to other measures, implying that magnitude may not be a crucial factor for merging. On the other hand, TSSD shows the highest correlation among all metries, suggesting better metric to predict whether two models are mergeable.
SSD(x. -1 Σ Vdl
(1)
where L (= ||||| IlylD is the size of and y. Equation (1) represents one minus the soft sign similarity. The similarity will be one if x; and y; have the same sign, which can quantify the agree- ment of the signs. the case when the values have different signs, the similarity will be the cancel-out value divided by the magnitude: (1) if they have exact opposite values, the similarity will be zero (the numerator is zero); (2) when y; is redundant value, denoting it has the opposite sign of but the magnitude small. The output of the similarity measure will be close to 1, making our approach softly quantifying redundancies.
Truncated soft sign dissimilarity (TSSD), is designed to incorporate the original implementa- tion (Yadav et al., 2023) of the 'redundancy re- moving' stage. To compute this, we zero-out the parameters of and y if their original values among the top-k smallest in x and y respectively, then apply SSD the new inputs. After truncation, the redundant values might be set to zero and will make its similarity with the other value to be one. Therefore, this approach can be viewed as a hard approach to quantify redundancy. Note that the truncation introduces many zeros to both x and and simply do not include the similarity of two zero values in computation because merging two zeros does not affect the merging results and thus should not affect the similarity either. In practice, we set to 50% of the total parameters of vision weights, thus zeroing half the parameters.
Conclusion
We explore applying the model merging tech- nique to transform modality-specifie models into modality- models. With comprehensive experiments, we analyze the key factors impacting the merging results, including initialization, merg- ing mechanisms, and model architectures. With seed pre-training and merging methods as sim- ple as interpolation, we improve the merging re- sults of the modality model significantly (+3% on VQA, +7% on COCO image-text retrieval, +25% NLVR +14% on Flickr30k and +3% on ADE20K). Moreover, we demonstrate the merged weight maintains the performance on unimodal tasks, and the proposed approach generalizes another backbone model. We also discuss an initial recommended recipe for multimodal merging.

---

9 Broader Impact and Limitations
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021. Unifying vision-and-lann aan-lang tasks via text genera- tion. ICMI,, abs/2102.02.
This paper demonstrates the potential to merge a modality-speciffe VL model to obtain a universal architecture that can handle multiple modalities si- multaneously. The findings in this paper address several questions that might be beneficial for fu- ture research. Firstly, what representation do the vision and language transformers learn that the merging performance is better? This helps to un- derstand the fundamental problem of when we can learn universal model. Secondly, can we merge two transformers that are initialized from unimodal pre-trained weights? This is useful as initializing the models from related domain would have per- formance gain. Lastly, can we apply the merging on the models fine-tuned on downstream VL tasks, to construct multi-task agdality model? We hope our study can inspire future work in this direction.
Ekin Dogus Cubuk, Barret Zoph. Jon Shlens, and Quoc Le. 2020. Randaugment: Practical automated data augmentation with a reduced search space. In Ad- vances in Neural Information Processing Systems, volume 33, pages 18613-18624, Curran Associates, Inc.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. 2009, Imagenet: A large-scale hi- erarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. NAACL-HLT.
Shachar Don- Ychiya, Elad Venezian, Colin Raf- fel, Noam Slonim, Yoav Katz, and Leshem Choshen. 2022. Cold fusion: Collaborative de- scent for distributed multitask finetuning. ArXiv, abs/2212.01378.
Though we show seed pre-training is crucial for merging, it shares similar cost issues with other pre-training experiments, leading to high carbon emission and electricity usage that may cause certain degree of environmental damage and energy cost.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference Learning Representations.
References
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuo- hang Wang, Lijuan Wang, Chenguang Zhu, Nanyun Peng. Zicheng Liu, and Michael Zeng. 2022. An empirical study of training end-to-end vision-and language transformers. CVPR, pages 18145-18155.
Armen Aghajanyan, Po-Yao (Bernie) Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, and Luke Zettlemoyer. 2022. Cm3: A causal masked multimodal model of the internet. ArXiv, abs/2201.0755 0775200
Danny Driess, F. Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Ho Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Ser- manet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Peter R. Flo- rence. 2023. Palm-e: An embodied multimodal lan- guage model. ArXiv, abs/2303.03378.
Samuel K. Ainsworth, Jonathan Hayase, and Sid- dhartha S. Srinivasa. 2022. Git re-basin: Merg- ing models modulo permutation symmetries. ArXiv, abs/2209.0488. 04.36.
Hassan Akbari, Linagzhe Yuan, Rui Qian, Wei-Hong Chuang. Shih-Fu Chang. Yin Cui, and Boqing Gong- 2021. Vatt: Transformers for multimodal self- supervised learning from raw video, audio and text. NeurlPS.
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur, 2021, The role of permutation invariance in linear mode connectivity of neural net- works. ArXiv, abs/2110.06296.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2022, BEit: BERT pre-training of image transform- In International Conference Learning Repre- sentations.
David Freedman, Robert Pisani, and Roger Purves. 2007. Statistics (international student edition). Pisani, R. Purves, 4th edn. WW Norton & Company, New York.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Learning universal image- text representations. ECCV.
Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, and Jianfeng Gao. 2022. Vingua-Language Pre-training: Basics, Recent Advances, and Future

---

Trends. In Foundations and Trends Computer Graphics and Vision.
Tsung- Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In ECCV.
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. 2016. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. IJCV, 127:398-414.
Shengchao Liu, Yingyu Liang, and Anthony Gitter. 2019. Loss-balanced task weighting to reduce nega- tive transfer in multi-task learning. In AAAI.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Worts- man, Suchin Gururangan, Ludwig Schmidt, Han- nanch Hajishirzi, and Ali Farhadi. 2022. Editing models with task arithmetic. ArXiv, abs/2212.04089.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguis- tic representations for vision-an ssion-anddlaan aud-language tasks. In NeurlPS.
Andrew Jaegle, Sebastian Borgcaud, Lean-Baptistt Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Andrew Brock, Evan Shelhamer, Olivier J. H' enaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira. 2022. Perceiver io: A general architecture for structured imputs & outputs. ICLR.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi. and Aniruddha Kembhavi. 2022. Unified- io: A unified model for vision, language, and multi- modal tasks. ArXiv, abs/2206.08911
Michael Matena and Colin Raffel, 2022. Merging mod- els with fesher- averaging. NeurIPS.
Xisen Jin, Xiang Ren, Daniel Preotiue-Piente Pietro, and Pengxiang Cheng. 2022. Dataless knowledge fu- sion by merging weights of language models, ArXiv, abs/2212.09849.
OpenAl. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.
Vicente Ordonez, Girish Kulkarni, and Tamara I.. Berg. 201 Im2text: Describing images using million captioned photographs. NIPS.
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietee and Pengxiang Cheng. 2023. Dataless knowledge fusion by merging weights language models. In ICLR.
Bryan A. Plummer, Liwei Wang, Christopher M. Cer- vantes, Juan C. Caicedo, J. Hockenmaier, and Svet- lana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phranl correspondences for richer image- to-sentence models. International Journal of Com- puter Vision, 123:74 - 93.
Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Ein tezari, and Behnam Neyshabur. 2022. Repair: Renor- malizing permuted activations for interpolation repair. ArXiv, abs/2211 1.00403.
Andrej Karpathy and Li Fei-Fei. 2017. Deep visual- semantic alignments for generating image descrip- tions. PAMI, 39:664-676.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In ICML.
Wonjae Kim, Bokyung Son, and Iodoo Kim. 2021. Vilt: Vision-and-language transformer without convolu- tion or region supervision. In ICML.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic im- age captioning. In Proceedings of the 56th Anmual Meeting of the Association for Computational Lin- guistics (Volume Long Pupers). pages 2556-2565. Melbourne, Australia. Association for Computational Linguistics.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2:
Bootstrapping language-iman -iage pre- training with frozen image encoders and large lan- guage models. ArXiv, 22.//2301.1559.
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. 2021. 1. Align before fuse: Vision and language representation learning with momentum distillation. In NeuriPS.
Sidak Pal Singh and Martin Jaggi. 2020. Model fusion via optimal transport. NeurIPS.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A sim- ple and performant baseline for vision and language. ArXiv, abs/1908 003557.
Weijic Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020. VI-bert: Pre-training of generic visual-linguistic representations. ArXiv, abs/1908 998800053..
Margaret Li. Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettle- moyer. 2022. Branch-train-merge: Embarrassingly parallel training of expert language models. ArXiv, abs/2208.03300
Alane Suhr, Stephanie Zhou, Iris Zhang. Huajun Bai. and Yoav Artzi 2019. A corpus for reasoning about natural language grounded in photographs. Annual Meeting of the Association for Computational Lin- guistics (ACL), abs/1811.00499 1.00991.

---

Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. 2022a. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. Neural Information Pro- cessing Systems (NeurIPS).
A Additonal Experimental Details
Datasets split. In training, we use the Karpathy-train split (Karpathy and Fei- Fei, 2017) for VQA and COCO datasets and use the train for other datasets. For evaluation, we use test-dev split for VQA, test-P split for NLVR?, the validation set for ImageNet and ADE20K, Karpathy-val split for COCO and test sct for Flickr30k.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. 2022b. VI-adapter: Parameter-effiieet transfer learning for ssioo-and-language tasks. CVPR, pages 5217-5227.
Hao Tan and Mohit Bansal. 2019. Lxmert: Learning modaltyy encoder representations from trans- formers. In EMNLP.
Jianfeng Wang, Xiaowei Hu, Zhe Gan, Zhengyuan Yang, Xiyang Dai, Zicheng Liu, Yumao Lu, and Lijuan Wang. 2021. Ufo: A unified transformer for lssiun-languag representation learning. ArXiv, 0bs/21111.1022.
Fine-tuning of ImageNet-Ik and ADE20k. We use the source codes of BEIT com/mic comm/mccosoon crosof t/ unilmmmmeet/mmm ttree/matte//bit with our trained checkpoints for fine-tuning.
Hy parraeerrr for fine-tuning. We show the
Peng Wang, Shijie Wang, Junyang Lin, Shuai Bai, Xi- aohuan Zhou, Jingren Zhou, Xinggang Wang, and Chang Zhou. 2023. One-peace: Exploring gen- eral representation model toward unlimited modali- tics. ArXiv, abs/2305.11172
hyper-parameter used in fine-tuning in Table 6.
Dataset
Image Size
Learning Rate
Butch Size
Training Epochs
480 480
3 10
128
COCO
394 x 384
6.25 10-6
640
20
Wenhui Wang, Hangbo Bao, Li Dong, and Furu Wei. 2022a. Vlmo: Unified vision-language pre-training with
NLVR²
384 x 384
10-5
128
10
Flickr30k
384 384
10-7
128
40
NeurlPS.
Table 6: Hyper-parameters used for fine-tuning.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yu- lia Tsvetkov, and Yuan Cao. 2022h. Simvlm: Simple visual language model pretraining with weak super- vision. International Conference on Learning Repre- sentations (ICLR).
Computational Usage. We use 48xV GPUs for seed pre-training and VL pre-training and it takes around 3.5 days for 200k iterations. We use 32x V 100 GPUs for VQA and COCO fine-tuning and they take around 7 hours to finish, while we use 16xV V100 GPUs for NLVR2 and Flickr30k and they take around hours to finish.
Mitchell Wortsman, Gabriel Ilharco. Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Loppe Ari Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Komblith, and Ludwig Schmidt. 2022a. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increas- ing inference time. In JCML.
Number Parameters. We display the number of parameters for BEiT architectures used this work in Table 7.
Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannanch Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. 2022b. Robust fine-tuning of zero-shot models. CVPR, pages 7949-7961.
Musility-ppoiii architectue
Medialky--gnne's architecture
Alm
EFN
Number of parameters
217M
151M
184M
118M
118M
Prateck Yadav, Derek Tam, Leshem Choshen, Colin Raf- fel, and Mohit Bansal. 2023. Resolving interference when merging models. ArXiv, abs/2306.01700
Table 7: Number of parameters for each architecture
based on BEiT base when heing applied on VQA.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Ye-
ung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022. Coca: Contrastive captioners are image-text foun- dation models. Transactions on Machine Learning Research, 2022.
B Licenses
Conceptual Captions: Other
COCO: Creative Commons Attribution 4.0 License
Bolei Zhou, Hang Zhao. Xavier Puig, Sanja Fidler. Adela Barriuso, and Antonio Torralba. 2016. Se- mantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302 - 321.
Visual Genome:
Creative Commons Attribu-
tion 4.0 License

---

VQAv2: Creative Commons Attribution 4.0 License
annotations of NLVRR: CC-BY-4.0 (There no licenses for images as the original authors also do not hold the copyright.)
Flickr30k:
Other
ADE20k:
Creative Commons BSD-3 License Agreement


Reviews:
[{"review_id": "K2xSkixDAH", "reviewer": "Reviewer_KA3D", "summary": "", "questions": "No.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful reviews. We are encouraged that the reviewers found our paper **interesting** [KA3D, v68G, Ms1f], **original** [v68G], and **insightful** [KA3D, v68G]. We are also glad to receive reviewers’ positive comments on our **empirical results** [v68G, Ms1f] and the **analysis of the distance explanation** [v68G, Ms1f]. In the following, we hope our responses can address your concerns, and we will incorporate the updates in the next version.\n\n> *I think the most serious problem of this work is that the evaluation benchmark only includes VQA and COCO. Just considering multi-modal benchmarks, VQA and COCO are not sufficient enough (when we talk about something on multi-modal pretraining). …… As a new strategy implemented based on VLMo, at least the same visual-only tasks (ImageNet and ADE20K) …*\n\nThank you for the feedback. We conducted experiments mainly on VQA and COCO because they are the **most popular and standard tasks adopted in the literature** [1, 2, 3] (also used in VLMo). However, we agree that more evaluation would make the paper stronger, so we also provide **additional experiments on NLVR2 and Flickr30k** as well as **ImageNet-1k and ADE20k** to cover more VL and vision tasks. Due to the short period of the rebuttal phase, we didn’t apply the intermediate fine-tuning (used in the VLMo paper) for Flickr30k, ImageNet-1k, and ADE20k, and only trained ADE20k for 80K steps.\n\nWe compare our merging approach (seed pre-training + interpolation with a ratio of 0.75) with the other three methods, the modality-specific model, the modality-agnostic method, and the naive merging method, and show the results in the table below. We found that **our approach performs much better than the naive merging on NLVR2 and Flickr30k**. However, we found the gap on NLVR2 and Flickr30k between our approach and the modality-agnostic model is a bit larger than the gap on VQA and COCO, where we match the results closely (Figure 4). Since images of VQA and COCO are used in the pre-trained tasks, we hypothesize that the merging results would be better if the domain shift between pre-trained and downstream tasks is smaller.\n\n\nFor the ImageNet and ADE20k results, we found that **our merging approach can match or even outperform the baselines**, and these results show that the merging does not destroy the representative power of the single modality. The investigation of studying the different merging outcomes on VL, vision, and language tasks would be an interesting future work.\n\n\n| Method             | NLVR2 (Acc) | Flickr30k (Avg IR and TR) | ImageNet-1k (Top-1 Acc) |  ADE20k (mIOU)  |\n|--------------------|-------|-----------|-------------| -------- |\n| Modality-Specific  | 79.71 | 91.24     | 83.25       |  48.27   |\n| Modality-Agnostic  | 78.39 | 88.84     | 82.78       |  46.88   |\n| Naive Merging      | 52.55 | 73.13     | 82.90       |  44.94   |\n| Our Merging        | 77.54 | 87.09     | 83.00       |  48.34   |\n\n[1] Li, Junnan, Dongxu Li, Silvio Savarese and Steven C. H. Hoi. “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.” ArXiv abs/2301.12597 (2023)\n\n[2] Tsimpoukelli, Maria, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, Felix Hill and Zacharias Janssen. “Multimodal Few-Shot Learning with Frozen Language Models.” Neural Information Processing Systems (2021).\n\n[3] Wang, Wenhui, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som and Furu Wei. “Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks.” ArXiv abs/2208.10442 (2022)\n\n> *Missing References*\n\nThank you for providing the reference. We will cite them as well as other VL representation learning papers in the next version.\n\n---\n#### ***We also conduct new experiments on another backbone, ViT-Tiny, to respond to the request of the reviewer Ms1f. We found the results are quite encouraging. Please refer to our reply to the reviewer Ms1f for more information.***", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Empirical results about seed pretraining iterations and weight sharing strategy may provide insights to obtain better performance for multimodal pretrained models.\n2. The research question of model merging in multimodal is an interesting topic which can help to reduce parameters and develop unification models.", "reasons_to_reject": "I think the most serious problem of this work is the evaluation benchmark only includes VQA and COCO. Just considering multi-modal benchmarks, VQA and COCO are not sufficient enough (when we talk about something on multi-modal pretraining). Moreover, as this work discuss about merging image and text transformers (or say transformer encoder layers) into multi-modal ones, the pre-training, fine-tuning and evaluation must include text-only and image-only tasks. When we only evaluate (limited) multi-modal tasks but talk about \"multi-modal model merging\" of image & text encoders, the value of this research work will be quite limited. The most referred work VLMo in this paper is also focusing on both multi-modal and visual-only tasks. As a new strategy implemented based on VLMo, at least the same visual-only tasks (ImageNet and ADE20K) should be considered to evaluate whether the merged transformer is actually a \"modality-agnostic\" one, rather than a module becoming incapable to do single-modal tasks after merging.", "questions_to_authors": "No.", "paper_topic_and_main_contributions": "This paper explores the optimal empirical setting (seed pretraining iterations and weight sharing strategy) for multi-modal pretraining and a merging approach to convert modality-specific models into modality-agnostic models. The hyper-param about these settings and different ways of merging are compared on VQA and MSCOCO tasks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Not found."}, "extra_text": {"Missing_References": "(Not all, just what I remember:)\n1. CoCa: Contrastive Captioners are Image-Text Foundation Models\n2. ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities\n3. SimVLM: Simple Visual Language Model Pretraining with Weak Supervision\n4. Unifying Vision-and-Language Tasks via Text Generation\nI recommend the authors to have a deeper paper study on multi-modal pretraining."}}, {"review_id": "U9GgoGhy0Y", "reviewer": "Reviewer_v68G", "summary": "", "questions": "* It's probably worth clarifying the difference between seed pretraining and VL-pretraining. It took me a while to understand the difference (seed pretraining use shared weights for vision and language inputs while VL-pretraining uses separate weights for each type of input? It's confusing partially because the name \"VL-pretraining\" indicates modality agnostic training).\n* It might be a good idea to dig deeper into the analysis part on why merging works at all and when it works better. Having an explanation-guided merging mechanism would definitely strengthen the paper.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful reviews. We are encouraged that the reviewers found our paper **interesting** [KA3D, v68G, Ms1f], **original** [v68G], and **insightful** [KA3D, v68G]. We are also glad to receive reviewers’ positive comments on our **empirical results** [v68G, Ms1f] and the **analysis of the distance explanation** [v68G, Ms1f]. In the following, we hope our responses can address your concerns, and we will incorporate the updates in the next version.\n\n> *Technical-wise seems incremental. Would suggest to study and discuss more on how modality merging differs from previous work on single/multi-task merging. Except the setup difference, are there unique technical challenges or new findings?*\n\nIn our research on merging models for a novel VL setup, we highlighted the **inherent challenges of combining weights from diverse domains in multi-modal models**. This challenge was shown when we observed suboptimal performance with naive merging attempts, as showcased in Table 1. However, a breakthrough was achieved when **utilizing an optimized initial weight derived from seed pre-training**. This led to merged models performing on par with the modality-agnostic models trained from scratch, as demonstrated in Figure 3, effectively closing the performance discrepancy.\n\nSeveral unique insights emerged from our study:\n\n* Our complete comparison of various merging techniques revealed that **RegMean performed robustly to hyperparameter choices** (lines 434 - 435). Additionally, using a slightly **higher interpolation ratio in the vision transformer boosted its performance** (lines 438 - 442).\n* While applying the model merging on shared-weight architectures (specialized to multi-modal setups), **we observed that architectures without any shared components outperformed their counterparts** (lines 466 - 470).\n\n\n> *The result is reasonable, but is not good enough to justify the additional steps (seed pretrain -> pretrain -> merging) needed for this procedure, comparing to just do a model agnostic training. Also, it introduces additional hyperparameters (merging ratios etc).*\n\nIn this paper, we aim to fairly assess the impact of the factors on merging so we don’t assume the weights exist and train the modality-specific and modality-agnostic models from scratch. That is the reason why we need the three steps (seed pretrain -> pretrain -> merging). However, the optimal case for merging is that we can merge the existing models that are already available online (Huggingface hub), so the process will be efficient as we can directly obtain a modality-agnostic model by **only doing the final step to merge an existing pre-trained modality-specific model**. Moreover, as we introduce in our guidelines (Section 6.4), we have also found that using interpolation ratios of 0.5 and 0.75 is good enough for most experiments (Table 2(a)), or that using RegMean is robust to hyperparameters (Table 3(c)). Providing these insights is also one of the values of this paper.\n\n> *It's probably worth clarifying the difference between seed pretraining and VL-pretraining. It took me a while to understand the difference (seed pretraining use shared weights for vision and language inputs while VL-pretraining uses separate weights for each type of input? It's confusing partially because the name \"VL-pretraining\" indicates modality agnostic training).*\n\nThanks for the feedback.  In our paper, **VL pre-training can be conducted on different model architectures, including both modality-specific models and modality-agnostic models**. For example, our experiments include 200k steps of VL pre-training on modality-specific models and modality-agnostic models (the modality-agnostic baseline in Figure 3). The seed pre-training, as you mentioned, is a way to create shared weights for two modalities, and improve the merging results by a large margin. To be more clear, **seed pre-training and VL pre-training are the same for modality-agnostic models, and seed pre-training for a modality-specific model is exactly VL pre-training (or seed pre-training) on a modality-agnostic model**. We use different terms for seed pre-training and VL pre-training to distantiate the different training stages for training the modality-specific model. We understand that the terms can create some confusion, and we will define them more clearly in the next version.\n\n\n\n> *It might be a good idea to dig deeper into the analysis part on why merging works at all and when it works better. Having an explanation-guided merging mechanism would definitely strengthen the paper.*\n\nThe intuition and hypothesis of why merging works are discussed in previous works [1, 2], and the main takeaway is that the models become mergeable when they are located in the same basin (basin means a local minima that has low loss everywhere). While those papers proposed the theory, their setup is somewhat **limited to the same modality or even the same tasks**. Our paper digs deeper into studying **how to apply model merging to multi-modalities and when it works better** in this scenario, as well as an analysis in Section 7 to **predict whether two models are mergeable by computing the distance between them in weight space**. Finally, we also release a **guideline on how to merge a multi-modal model** (Section 6.4). We will add the discussion of the position of our work (compared to previous theoretical ones) to the next version.\n\n[1] Ainsworth, Samuel K., Jonathan Hayase and Siddhartha S. Srinivasa. “Git Re-Basin: Merging Models modulo Permutation Symmetries.” ArXiv abs/2209.04836 (2022)\n\n[2] Garipov, T., Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P. Vetrov and Andrew Gordon Wilson. “Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs.” ArXiv abs/1802.10026 (2018)\n\n---\n#### ***We also conduct new experiments on another backbone, ViT-Tiny, to respond to the request of the reviewer Ms1f and new experiments on NLVR2, Flickr30k, ImageNet-1k, and ADE20k to respond to reviewer KA3D. We found the results are quite encouraging. Please refer to our replies to the reviewer Ms1f and KA3D for more information.***", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The problem on weight merging is interesting, and indeed this seems to be one of the first work to study modality-wise weight merging.\n* Experiments are thorough and the analysis on the distance explanation provides some insights into why/how merging works.", "reasons_to_reject": "* Technical-wise seems incremental. Would suggest to study and discuss more on how modality merging differs from previous work on single/multi-task merging. Except the setup difference, are there unique technical challenges or new findings?\n* The result is reasonable, but is not good enough to justify the additional steps (seed pretrain -> pretrain -> merging) needed for this procedure, comparing to just do a model agnostic training. Also, it introduces additional hyperparameters (merging ratios etc).", "questions_to_authors": "* It's probably worth clarifying the difference between seed pretraining and VL-pretraining. It took me a while to understand the difference (seed pretraining use shared weights for vision and language inputs while VL-pretraining uses separate weights for each type of input? It's confusing partially because the name \"VL-pretraining\" indicates modality agnostic training).\n* It might be a good idea to dig deeper into the analysis part on why merging works at all and when it works better. Having an explanation-guided merging mechanism would definitely strengthen the paper.", "paper_topic_and_main_contributions": "The paper mostly presents an experimental study on weight merging for transformer architectures trained on specific modalities - how it performs multi-modally by merging the weights of language specific arch and vision specific arch. The main contribution is to systematically experiment across 1) effect of initialization 2) merging mechanism and 3) model architecture, to understand how the merged model performs under different combination of these conditions. It arrives at an optimal setup suggestion and shows good performance result.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "y6hObOq3uf", "reviewer": "Reviewer_Ms1f", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful reviews. We are encouraged that the reviewers found our paper **interesting** [KA3D, v68G, Ms1f], **original** [v68G], and **insightful** [KA3D, v68G]. We are also glad to receive reviewers’ positive comments on our **empirical results** [v68G, Ms1f] and the **analysis of the distance explanation** [v68G, Ms1f]. In the following, we hope our responses can address your concerns, and we will incorporate the updates in the next version.\n\n> *The novelty of this work is limited in that it is an empirical study.*\n\nThanks for the feedback. However, we respectfully disagree that an empirical study points to a limited novelty, especially at an \"empirical\" conference (EMNLP). While our paper focuses on empirical findings, we also highlighted the **inherent challenges (the weights are not located in the same basin) and technical variations (model architectures and merging methods)** of combining weights from diverse domains in multi-modal models. Therefore, our study presents **a novel utilization of previous approaches to a novel important application** and can have more contribution given the emergence of large multi-modal models.\n\n> *The target tasks are rather old: VQA (Goyal et al., 2016) and COCO retrieval (Lin 317 et al., 2014). It would be more interesting with VL tasks with more recent benchmarks.*\n\nOur paper focuses on studying how to merge multi-modal models and the downstream performance changes of the chosen models after merging. Therefore, the goal is not to study how the model merging affects the performance of the recent VL benchmarks. VLMo was the state-of-the-art modality-specific model when our project started (still performs in the top 3 on the VQA benchmark), and VQA and COCO are the main tasks VLMo is evaluated on (they are the **most popular and standard tasks adopted in the literature** [1, 2, 3]), so we chose VLMo as the backbone model and VQA and COCO as the target datasets.\n\nThat being said, we have also **expanded our study and conducted two more VL experiments on the popular NLVR2 and Flickr30k datasets and two vision experiments on ImageNet and ADE20k**. Due to the short period of the rebuttal phase, we didn’t apply the intermediate fine-tuning (used in the VLMo paper) for Flickr30k, ImageNet-1k, and ADE20k, and only trained ADE20k for 80K steps. We compare our merging approach (seed pre-training + interpolation with a ratio of 0.75) with the other three methods, the modality-specific model, the modality-agnostic method, and the naive merging method. We found that our approach can **boost the merging results compared to naive merging by a large margin** in most tasks and can closely match the modality-agnostic baseline.\n\n| Method             | NLVR2 (Acc) | Flickr30k (Avg IR and TR) | ImageNet-1k (Top-1 Acc) |  ADE20k (mIOU)  |\n|--------------------|-------|-----------|-------------| -------- |\n| Modality-Specific  | 79.71 | 91.24     | 83.25       |  48.27   |\n| Modality-Agnostic  | 78.39 | 88.84     | 82.78       |  46.88   |\n| Naive Merging      | 52.55 | 73.13     | 82.90       |  44.94   |\n| Our Merging        | 77.54 | 87.09     | 83.00       |  48.34   |\n\n\n\n[1] Li, Junnan, Dongxu Li, Silvio Savarese and Steven C. H. Hoi. “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.” ArXiv abs/2301.12597 (2023)\n\n[2] Tsimpoukelli, Maria, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, Felix Hill and Zacharias Janssen. “Multimodal Few-Shot Learning with Frozen Language Models.” Neural Information Processing Systems (2021).\n\n[3] Wang, Wenhui, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som and Furu Wei. “Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks.” ArXiv abs/2208.10442 (2022)\n\n\n> *Only a single baseline ViT-B/16 is tested.*\n\nAs we explained in the previous reply, we focus on how the merging affects the downstream performance of the state-of-the-art modality-specific models, VLMo. While VLMo has two different backbones, ViT-B/16 and ViT-L/16, we chose ViT-B/16 due to resource constraints. \n\nHowever, as you requested, we conducted **additional experiments on ViT-Tiny/16**. Note that this backbone is not trained by the BEIT unsupervised objective but is trained by the supervised classification objective. Due to the time limit, we only pre-trained the model for 100k steps instead of 200k steps in the paper. For our merging approach, we use 50k steps for both seed pre-training and VL pre-training. We show the results in the following table and **find our technique still performs much better than the naive merging**. The larger gap between our approach and the modality-agnostic baseline with ViT-Tiny/16, compared with the ViT-base model, has been similarly observed in the previous work [4, 5], where they show larger (wider) models are easier to merge.\n\n\n| Method on ViT-Tiny | VQA (Acc)   | COCO (Avg IR and TR)  | NLVR (Acc)  | Flickr (Avg IR and TR) |\n|--------------------|-------|-------|-------|--------|\n| Modality-Specific  | 65.36 | 65.87 | 68.42 | 75.44  |\n| Modality-Agnostic  | 62.25 | 57.63 | 64.9  | 67.38  |\n| Naive Merging      | 55.52 | 47.21 | 51.07 | 16.86  |\n| Our Merging        | 60.02 | 58.11 | 52.89* | 63.96  |\n\n*we found using interpolation ratio of 0.5 can boost the performance to 60.94 and cut the gap to the modality-agnostic model significantly.\n\n\n[4] Ainsworth, Samuel K., Jonathan Hayase and Siddhartha S. Srinivasa. “Git Re-Basin: Merging Models modulo Permutation Symmetries.” ArXiv abs/2209.04836 (2022)\n\n[5] Singh, Sidak Pal and Martin Jaggi. “Model Fusion via Optimal Transport.” ArXiv abs/1910.05653 (2019)\n\n> *The proposed two metrics are largely based on those of Yadav et al. (2023).*\n\nNote that the goal and utilization of the proposed SSD and TSSD metrics are different from the approach of Yadav et al. (2023), and the **definitions of the metrics are also original from our paper**. Yadav studies how to resolve the sign conflicts in the merging while we design the metrics to predict the performance drop of the two models prior to merging (to save the time of training) (lines 574 - 575). Our results show that the **TSSD has the highest correlation with the performance drop**, suggesting that it is the **best indicator to predict the merging outcome**.\n\n\n> *As depicted in Fig.4, the merging results of the modality-specific model increase by +3% on VQA and +7% on COCO image-text retrieval. First, it should justify that these gaps are significant enough, by comparing with other SOTA methods. Second, this is not achieved by any novel idea of this work but by searching for combinations of known simple techniques over key decision factors.*\n\nFor the first question: note that our goal is to match the performance of modality-agnostic models via merging, and it can be viewed as one of the upper bounds in the model merging literature [4,5,6]. **Our approach matches this upper bound** (VQA: 73.91 (ours) vs. 73.94, COCO: 78.06 (ours) vs. 78.07) at the end, so we consider our improvement to be significant. Furthermore, to the best of our knowledge, our study **is the first to explore model merging** in VL domains (also pointed out by reviewer v68G), thus there are no existing works (or other SOTA methods) for us to benchmark against. Instead, our main focus is to evaluate various factors that affect the merging outcomes in our VL setting, such as SOTA merging methods (e.g., Interpolation, Modality arithmetic, RegMean), different architectures, and seed pre-training.\n\nFor the second question, please refer to our reply to your first question.\n\n[6] Wang, Hongyi, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos and Yasaman Khazaeni. “Federated Learning with Matched Averaging.” ArXiv abs/2002.06440 (2020)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work empirically find multiple interesting observations about multimodal margining, as summarized well in section 1. \n\n2. This work merges well-trained modality-specific models into a modality-agnostic model that can achieve similar performance to the one that is pre-trained from scratch.\n\n3. It proposes two simple metrics that assess the distance between weights to be merged and can serve as an indicator of the merging outcomes. \n", "reasons_to_reject": "1. The novelty of this work is limited in that it is an empirical study. \n\n2. The target tasks are rather old: VQA (Goyal et al., 2016) and COCO retrieval (Lin 317 et al., 2014). It would be more interesting with VL tasks with more recent benchmarks. \n\n3. Only a single baseline ViT-B/16 is tested.\n\n4. The proposed two metrics are largely based on those of Yadav 506 et al. (2023). \n\n5. As depicted in Fig.4, the merging results of the modality-specific model increase by +3% on VQA and +7% on COCO image-text retrieval. First, it should justify that these gaps are significant enough, by comparing with other SOTA methods. Second, this is not achieved by any novel idea of this work but by searching for combinations of known simple techniques over key decision factors. \n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "1. This work empirically investigates how to merge modality-specific vision, language, and cross-modal transformers into a modality-agnostic architecture. \n\n2. It tests tree merging methods, interpolation, arithmetic, and RegMean, by varying some key factors of margining, including initialization, merging mechanisms and model architectures.\n\n3. It performs experiments for the tasks of VQA and COCO image-text retrieval.  \n", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides a comprehensive empirical study on multimodal model merging for transformer architectures. Three reviewers provided reviews for this paper and these reviews were in consensus. The reviewers appreciated the paper and found the empirical analysis to be interesting. One major feedback by the reviewers was to increase the number of datasets and tasks for evaluation. The authors responded well to this feedback by providing several new experiments in line with previous work that had already been cited. I really appreciate this diligence and effort put in by the authors. The new experiments solidify the findings and increase the impact of this study. With these improvements and clarifications addressed by the authors, reviewers found the work to be technically sound and complete.
Decision: Accept-Findings