Multimodal Lego: Model Merging and Fine-Tuning Across Topologies and Modalities in Biomedicine
Authors: Konstantin Hemker
Nikola Simidjievski
Mateja Jamnik
Venue: ICLR 2025 Poster (2025)
Keywords: multimodal
deep learning
fusion
biomedicine
model merging
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Multimodal fusion and model merging approach for structurally heterogeneous modalities

Abstract: Learning holistic computational representations in physical, chemical or biological systems requires the ability to process information from different distributions and modalities within the same model. Thus, the demand for multimodal machine learning models has sharply risen for modalities that go beyond vision and language, such as sequences, graphs, time series, or tabular data. While there are many available multimodal fusion and alignment approaches, most of them require end-to-end training, scale quadratically with the number of modalities, cannot handle cases of high modality imbalance in the training set, or are highly topology-specific, making them too restrictive for many biomedical learning tasks. This paper presents Multimodal Lego (MM-Lego), a general-purpose fusion framework to turn any set of encoders into a competitive multimodal model with no or minimal fine-tuning. We achieve this by introducing a wrapper for any unimodal encoder that enforces shape consistency between modality representations. It harmonises these representations by learning features in the frequency domain to enable model merging with little signal interference. We show that MM-Lego 1) can be used as a model merging method which achieves competitive performance with end-to-end fusion models without any fine-tuning, 2) can operate on any unimodal encoder, and 3) is a model fusion method that, with minimal fine-tuning, surpasses all benchmarks in five out of seven datasets.

Published as a conference paper at ICLR 2025
MULTIMODAL LEGO: MODEL MERGING AND FUSION ACROSS TOPOLOGIES AND MODALITIES IN BIOMEDICINE
Konstantin Hemker', Nikola Simidjievski.11 & Mateja Jamnik*
DDpartment of Computer Science & Technology
²PBCI, Department of Oncology University of Cambridge Cambridge, UK (kh701, ns779, mj201 @cam. ac.uk
ABSTRACT
Learning holistic computational representations in physical, chemical or biological systems requires the ability to process information from different distributions and modalities within the same model. Thus, the demand for multimodal machine learning models has sharply risen for modalities that go beyond vision and language, such sequences, graphs, time series, or tabular data. While there many available multimodal fusion and alignment approaches, most of them require end-to-end training, scale quadratically with the number of modalities, cannot handle cases of high modality imbalance in the training set, or are highly topology- specific, making them restrictive for many biomedical learning tasks. This paper presents Multimodal Lego (MM-Lego), turn any set of encoders into a competitive multimodal model with or minimal fine-tuning. We achieve this by introducing a wrapper for any unimodal encoder that enforces shape consistency between modality representations. It harmonises these representations by learning features in the frequency domain to enable model merging with little signal interference. We show that MM-Lego 1) can be used as a model merging method which achieves competitive performance with end-to-end fusion models without any fine-tuning of the original encoder weights, 2) can operate any unimodal encoder, and 3) is model fusion method that, with minimal fine-tuning surpasses all benchmarks in five out of seven datasets.
general-purposs fusion framework
INTRODUCTION
The utility and demand for multimodal machine learning approaches has sharply risen due to their potential to derive holistic representations in various systems, including physics (Zubatiuk & Isayev 2021), chemistry (Belianinov et al. 2018), neuroscience (Alberdi et al. 2016). or biology (Boehm 2022). Multimodal models in the vision language domains leverage the same data distributions, which are represented across different modalities (Yu et al., 2019 Li et al.) 2021b Tu al. 2023), such vision-text pairs of the same concepts. However, in many biomedical domains, modalities represent data at different scales (e.g., cellular, genomic, transcriptomic), cardinalities that are not paired (e.g.. many single-cell reads for single tissue slide per patient), and follow separate distributions. While large foundation models have excelled in tasks confined to individual modalities (Cui et al. 2024, Nguyen et al.) 2024 Chen et al. 2024), training these models across modalities expensive, requires paired modalities, and is still end-to-end process.
Multimodal fusion methods (Zadeh et al., 2017 Liu et al. 2018 Nagrani et al. 2021 Li et al. 202 attempt derive common representation from different data structures and distributions whilst preserving its salient signals (Baltrusaitis et al., 2019). 19.. However, there are several shortcomings of existing fusion methods that relate their utility, scalability and underlying data assumptions. First, many fusion methods require end-to-end training of the multimodal model, and even in scenarios with existing unimodal models, the fusion operation still has be trained for downstream task,

---

Published as a conference paper at ICLR 2025
Wrap Unimodal Encoder
Merge without finetuning
OR
Fusion with minimal finetuning 1 1/0h , c came
Encoder Modalit
Imaging
Multiomice
Long
0f-)
Imput Iatent Mosialyynggro ontic latent
X14
X
X
X
X
Lig Rend
Spatial domair
Fi2
FF
Stuck
Weave Lg
Gupoed suuse swarr)
Ul
L
Frequanccy domwin > Fusion layar
Fourier Block
Harnerc Mean
Final pass
Tosk head
L
Task head
Task head
Tosk hoad
Logits
Lagss
Learns modality frequency representation
Performant merge into multimodil model "for free"
Cross-modal contextualisation
State-of-the thtt--a-- performance with little fine-tuning
No paired data required
Outperforms unimodal models
The Multimodal Lego workflow to turn a set of encoders into a performant multimodal model. LegoBlock (1) makes unimodal encoders compatible with model merging techniques by learning a latent representation in the frequency-doman aggregation. Any set of LegoBlocks can be merged into multimodal model without any fine-tuning (LegoMerge (2a)) or with minimal fine-tuning to achieve state-of-the-art performance (LegoFuuse (2h)).
Figure I:
to prevent signal interference effects upon
typically in supervised manner. This leads to redundant computational overhead and inability to extend the model with additional modalities after it has been trained. Second, many commonly
used fusion methods either scale quadratically (with the number of modalities) (Zadeh al. 2017
Chen et al. 2022) or are only designed to operate with two modalities (Chen et al.) [021] Xu & Chen 2023). Third, many of these methods follow a monolithic design, requiring all modalities be available for every sample during training. This means that they are not robust to missing modalities, modality imbalance, or non-overlapping training sets, which is very common challenge variety of real-world settings (Swamy et al. 2023). Finally, many end-to-end fusion architectures are highly topology-specific, making them difficult to extend to other domains.
Some of these challenges can be addressed through model merging (Labonne 2024) knowledge fusion (Jin et al. ()23)), an approach commonly used in the context of multi-task settings and language modelling, which capitalises combining well well-performing unimodal models trained in isolation. Model merging methods attempt to combine two architecturally identical models trained on different distributions through interpolation, arithmetic manipulation, and aggregation of their weights Yadavv et al. 2023] Stoica et al., 2024 [lharco et al.) 2023). or stacking their layers (Akiba While model merging has been extended to some multimodal vision and language tasks (Sung et al. 2202),, its crucial challenges in a multimodal setting are that: a) the merged components are still trained in isolation, and b) we cannot assume topological equivalence between two models for separate modalities due to their separate input shapes.
(also referred to as
et al. 2024), often without additional training/hne-tuning
In this paper, we present Multimodal Lego (MM-Lego) a flexible framework for combining any unimodal models into a multimodal model with no or minimal fine- fine-tunig (Figure []. We introduce two approaches within our framework - LegoFuse use and LegoMerge, enabling performant multimodal models given a set of unimodal encoders with either little (LegoFuse) or no (LegoMerge) fine-tuning. We show that MM-Lego satisfies multiple desirable properties in a range of real-world multimodal applications combining imaging, tabular, and time series modalities. We demonstrate the utility of MM-Lego on seven medical datasets across three separate downstream tasks, showing that is:
1. Performant without end-to-end training: LegoMerge is highly computationally efficient and achieves competitive performance wrt. state-of-the-art without any fine-tuning, while LegoFuse exceeds the state-of-the-ar in some tasks with only a few epochs of fine-tuning.
Scalable: Both variants of MM-Lego scale linearly with the number of modalities whilst outper- forming methods with quadratic time complexity.
2

---

Published as a conference paper at ICLR 2025
3. Topology agnostic:
Unlike most model merging approaches, LegoMerge does not require equiva- lent architectures between the merged models, allowing users to take advantage of the plethora of open-source models for multimodal learning.
4. Handling modality imbalance & non-overlapping sets: MM-Lego is robust in cases of missing modalities and strong modality imbalance, a common problem in medical domains. MM-Lego can be used even if each modality was trained on unpaired (non-overlapping) training samples.
2 BACKGROUND & RELATED WORK
Preliminaries. Let X(M) UMEM m be multimodal dataset where M B....... 2} represents the set of modalities m such as images (A), tabular data (B), time series (C), etc. Let x ( correspond to the element in the dataset for modality A at sample i, column j. and channel assuming A E IR xxXxX where 1 S N, N.N. J. < k < K. Each sample in X has a set of discriminative task labels y(7) = ULeT y where T = (Ti, ,,, is the set of possible tasks such that y(z)) = {ur' 1//2 1 .... UN are the scalar target values for task T2 for N samples.
Fusion methods. Given multiple data inputs or latent representations, fusion methods construct a single learning representation that can be used for downstream tasks, often whilst reducing dimension- ality. Many fusion methods (Alberdi et al. [006;CChen
al. 2022) first learn a set of modality-speciiff encoders G = {9m m - h assuming a single task label y. This results in a set of latent representations H { (mz) m MM, which are combined with fusion operator to obtain the final fused representation z = (H) and its final prediction y = f()). Following this problem setup, fusion methods can be diffcrentiated by: 1) the choice of the fusion operator eb(:); ); 2) the fusion stage in the pipeline of when (-) is applied; and 3) the fusion order in which the fusion operations are ap- plied (i.e., sequential vs. parallel). The fusion operator can be either static (e.g., concatenation (Jaegle et al. 2021), Kronecker product (Zadeh et al., (017)) or learnable (e.g., low-rank tensor fusion (Liu
Chen et
al. 2018), cross-attention mechanisms (Nagrani al. 2021 Li et al. 202 Xu & Chen 2023).
mixture of experts (Mustala et al.] 2022, Han el al. (024)) The fusion stage is typically characterised as early, intermediate or late fusion. Early fusion methods apply a static fusion operator (-) to the
raw data while only applying this after passing each modality through G. Intermediate fusion methods often do not apply a static aggregation but rather learn a fusion function (i.e., a small netwook or neural layer) in the latent space as part of its end-to-end training (Baltrusaitis et al. 2019).
shorteoming of existing fusion methods for many real-world applications is the fusion order- most fusion methods use monolithic task setup, where all modalities are required during training and inference to calculate the set of latent representations H. This often leads to noisy fused representations in cases when modality is (partially or fully) missing, as the missing tensor requires imputation. Moreover, (19mmm (), m E .M)) is typically trained end-to-end. This hinders the potential of extending the multimodal model with additional modalities (beyond the ones it has been trained on), without training anew. Rerunning the complete training pipeline just to add one (or more) additional modality can be infeasible or can lead to redundant computational overheads. Finally, many methods are highly domain-specific and are either not designed for [M] > 2 or scale quadratically with the number of modalities (Li et al. 202 Chen al. 2021 Xu & Chen (023).
Model merging. The core idea behind model merging, typically deployed in multi-task settings, is that earlier layers in a network may learn similar features that may be used across tasks (Sundar et al. 2024). Using linear interpolation (Labonne 2024) or arithmetic manipulation (Ilharco et al. 2023) of the task ssse-spec weights, model merging approaches have shown that they can effectively generalise to new tasks without any fine-tuning. Formally, given multiple tasks (7) for the same modality A, they first learn the set of task- functions F {fr) A ((t)) | t E T where w denotes the corresponding model parameters. Assuming the same architecture for each model in F, parameters from pre-trained base model Whase can be used to derive task vectors as V= (T. + Wi:- Whose 1 E T) (Ilharco et al. 2023). Given these task vectors, a multi-task model
can be constructed by updating the weights of the base model w/ = Whose + +X Tr. This idea has been extended by the TIES (Yadav et al. 2023) and DARE (Yu et al.) 2024) that merge models through sparsifying and resolving sign conflicts in the task vectors. Another popular approach is spherical linear interpolation (SLERP), a method used to smoothly interpolate between two vectors while respecting the geometric properties of the vector space (Labonne 2024). More specifically,
3

---

Published as a conference paper at ICLR 2025
given model parameters way and wry E RR, derived from two models with identical architectures, the merged multi-task model parameters can be calculated as w/ wT, sin(0-(1 11))) sén(O) + wr, sin(O)
sin(0-p)
where 0 is the radian between the two vectors WT, and wTy (Shoemake) 1985). J. The underlying
assumption of the above model merging approaches is that the models should have equivalent network topology, ensuring that the dimensions IRd match up between tasks. However, while this is an
acceptable constraint for multi-task learning, it infeasible for multimodal models where modality shapes and the corresponding network topologies vary greatly.
3 MULTIMODAL LEGO
MM-Lego introduces two novel approaches for multimodal model merging (LegoMerge) and mul- timodal model fusion (LegoFuse). It addresses a number of limitations of existing model merging (topological equivalence) and fusion methods (such as scalability for |M| > missing modality handling, end-to-end training, paired data, etc.). The core component of MM-Lego is a LegoBlock wrapper for any modality-speciift encoder that imposes several constraints on the latent feature space and structure. These constraints are necessary condition for LegoMerge, which aggregates the latent encodings with minimal signal interference between modalities to perform multimodal model merge. Moreover, using LegoFuss Fuse, MM-Lego allows us to fine-tune the combined blocks, ensuring that cross-modal dependencies and mutual context can be learned.
Architecture. Rather than learning single fusion operator (²H) that applies to all latent representa- tions at once, we learn a set of latent update functions for each modality, in the form of
{v (9m.(X (m)) Lmm)
[(m) E m E M},
where L, L.(") E our target latent representation for each modality that we will later use in the merge and fusion, and S is the number of update steps. Using the iterative update architecture with latent state passing in Equation [ has number of advantages. First, iterative attention architectures have been shown to be highly generalisable across modalities (Jaegle al. (022), and effective at dealing with missing individual modalities (Swamy et al. 2002) Hemker et al. (022).. Second, since all modalities are encoded into self-defined latent representation, we can impose dimensionality
constraint such that each latent has the same dimensions (e.g., [(A), I (B)), I R. d for latent channels and dimensions c and d). Third, we can do latent state passing between elements B, which allows to "stack" the update functions top of each other (hence the name) to sequentially encode each modality's 's signal into the same latent representation.
LegoBlocks. Each element in B represents LegoBlock (Figure 2). which learns the latent update function thm for any given encoder Im- acknowledging that different data modalities and structures require different inductive biases effectively encode each modality's information (gm)). LegoBlock acts wrapper to accurately encode hm, into L (m)) The benefit of training each modality update function separately instead of end-to-end is that we can train on entirely separate samples for the
(()
g(-)
h (4)
F(-)
Ln
Reconstruct state and inverse FFT ((L))+
L
A(
F(-)
(L.5
Cross-attn
(L Track with imaginary component
Latent representatenn
Figure 2: Frequency-domain state passing in LegoBlock. The latent bottleneck Lo is randomly initialized as a model parameter at the start of training and iteratively updated by each pass through the Llocks The real components of the FFT (LF)) and (()( are used in the cross-attention update, and the imaginary component (IT)) )" is used for reconstruction.
4

---

Published as a conference paper at ICLR 2025
same tasks. For example, in many medical domains, may have single-cell data for one subset of patients and bulk sequencing data for a different subset, while having the same task labels for the entire set. To address this, we use latent representations L that effectively encode signal across modalities, and are robust or invariant to transformations (shifts, rotations, etc.), noise and signal interference. Alleviating signal interference is particularly important for model merging, as it is undesirable to apply an aggregate function on the learned modality latents L that can cancel out each other's signal. Beyond preventing signal interference, Fourier features have also been shown to be effective as mixing mechanisms [Lee-Thhop et al., 2021), positional encodings [Lee-Thorp et al. 202T]. and to be naturally emerging invariant networks (Marchetti et al.) 2023).
This motivated us to design MM-Lego for learning latent representations in the frequency domain, taking advantage of number of desirable properties for multimodal merging and fusion. In particular, siqquencydommall representations are: 1) signal-preserving as frequency features are less prone to signal interference upon aggregation (see Appendix F): 2) distance-preservvin
as the Euclidean distance between two signals remains unchanged after the Fourier Transform (following from Parseval's Theorem (Parseval 1806),, making them suitable for distance-based loss functions; 3) invertible the spatial/temporal domain can be reconstructed, allowing for the iterative updates outlined in Equation [ and 4) efficient, as the Fast Fourier Transform (FFT) has a time complexity of O(n lo)( making scalable to very large datasets (Lee-Thorp al., 2021). Further detail on why the Fourier transform exhibits desirable properties for model mergin can be found in Appendix B
Figure 2 depicts single update of LegoBlock that operates over frequency-domain latent of modality ((A), Starting with the latent representation in the spatial domain, we first apply a discrete FFT F(-) (Nussbaumer representation:
1982) along each dimension of the 2D Tensor to yield a frequency domain
c-1d-1 (u,
(2)
==
where j.j denote the spatial-domain indices, and u, v denote the requeencydoomaie indices. This results in a complex rrequency-domaii representation from which we separate the real (symmetrical) and imaginary (asymmetrical) components of the FFT ((T.")² and (L) (Smith et al. 1997]. We update the real component using a standard cross- attention layer (Vaswani et al. 2017). where we aim learn the weight matrices Wq. for the update query (L )" and W W for the keys and values (h(4)) resulting in the latent update:
( (F(6(A))(.)))
(Le+1)" softmax
(F(K(A)y<WM))
(3)
Vdh
In contrast to other Fourier-based architectures (Lee- Thorp et al. 2021), which only use the real component of the transform, we keep track of the mmginnrry component (L,")² as well. This allows us to reconstruct the complex representation, and subsequently apply the inverse transform. We found this to be critical for our iterative architecture, as otherwise the signal gets distorted and we lose phase information (encoded in the imaginary component) at each update pass. Once we reconstruct the complex representation, we apply the inverse transform to recover the spatial representation in preparation of the next pass L++1 = = ((L,((( + i(L))'). Finally, the last task-specific heads of each block are fully- cally-connected layer after applying layer normalisation. We omit the inverse transform after the last update such that each head is trained in the frequency domain. This ensures that we can apply aggregations with low signal interference on L during LegoMerge.
MegoMerge. With the architectural assumptions imposed on each modality encoder in G through LegoBlocks B3, we can apply model merging techniques in multimodal setting. With L C Rexd and each element in L being in the frequency domain, we can use aggregation functions (f-), which are less prone to cancelling out signal. For example, let ((4) and ((B) be the final frequency domain latent representations for modalities A and B, then we can calculate merged multimodal representation as:
| [B) (AA)), (B)) | + [
(2(B)
where the real component the harmonic mean of the magnitudes ()- D. and the imaginary component is the arithmetic mean of the phases (2) of L (A) and (B)) We take the harmonic mean since it is less prone to outliers (Smith al. 1997). that is, the merged representation is less likely to be strongly
5

---

Published as a conference paper at ICLR 2025
skewed towards either modality by very large frequency components. With the cross-modal combined representation T,(M) we need to combine the task heads of each block, where we apply spherical linear interpolation (SLERP) (Shoemake] 1985) for the set of task heads y from each element B. As- suming two task heads with weight vectors Wyl and Wy,. where Wyl WA, 22 C WB, we calculate the merged weights as Wyly2 = Wy1 sin(0.
(0 (Shoemake 1985), where 0 the an-
()n())
+ Wy y2" (i
gle (in radians) between both vectors and js is a binary variable indicating whether Wyl or Wy2 is used.
LegoFuse. LegoMerge is designed to construct a performant multimodal model without any additional training. Nevertheless, its key limitation is that each element in B is trained in isolation. To allow for modalities to mutually contextualise each other, limited amount of fine-tuning is beneficial. To avoid fine-tuning a potentially noised signal emerging from the merged latent I,(M), rather than directly fine-tuning the merged model (at the parameter-level) LegoFuse use operates at the layer level by sequentially passing through all layers in B. Specifically, the shape consistency introduced by L C Rex allows the stacked model to pass the Fourrir-transformmed latent states either between blocks (stacking) or different layers between blocks (weaving). as illustrated in Figure I We then fine-tune the stacked or the weaved model for a few epochs with all (paired) modalities, such that the state updates are conditioned on all modalities' updates. This, in turn, becomes the query for the cross-attention layer. Note that both the stacked and weaved variants of LegoFuse allow for encoders.
fine-tuning all model parameters, including the ones of the initial
4 EXPERIMENTAL SETUP
Datasets. We evaluate MM-Lego (LegoMerge and LegoFuse) and its components (LegoBlock) on seven multimodal medical datasets from three separate studies: The Cancer Genome Atlas (TCGA) (Institute 2006), Medical Information Mart for Intensive Care (MIMIC) (Johnson et al. 2016) and the International Skin Imaging Collaboration (ISIC) (Collaboration 2020). The TCGA study includes data from four cancer subtypes (BLCA, BRCA, KIRP, UCEC) across four different modalities: digitalised pathology (whole slide image data), gene expression (continuous variables, tabular data), copy number variations (categorical variables, tabular data), and mutations (binary variables, tabular data). The MIMIC study includes intensive care data captured in two distinct modalities of continuous and time-series data. The ISIC study comprises image and time-series data pertaining to clinical data from melanoma patients. The seven tasks shown in our results correspond to survival analysis tasks on four TCGA sites (BLCA, BRCA, KIRP, UCEC), classification tasks on two variants of MIMIC (disease classification (ICD9) and patient in-hospital mortality (MORT)), and predicting melanoma for the ISIC patients. Full details on the datasets and their pre-processing steps can be found in Appendix C and details on task losses and evaluation metrics are in Appendix D
Baselines. For all experiments, we compare LegoMerge and LegoFuse to several uni- and multimodal baselines to evaluate their performance. For all tabular modalities, we use a self-normalising network (Klambauer et al. 2017) due to its performance and regularisation mechanisms suitable for high- dimensional tabular data. For the image and time series modalities, we Instance Learning (AMIL) (Shao et al. 202T). Across all modalities, we benchmark two related iterative-learnngg architectures: MultiModN (Swamy et al.! (023) and Perceiver (Jaegle et al. 2002). which generally show strong performance across a wide range of unimodal tasks. In terms of specific multimodal baselines, we use two late fusion combinations of SNN+AMIL, namely concatenation of their final latent representation and bi-linear fusion (Li et al.) 2022). For the Perceiver, we use the same multimodal setup as suggested in the original paper, that is, concatenation of modalities before passing them into the model. We use two additional domain-specific multimodal baselines: the Hybrid Early-
attention- based Multiple
Attention Learning Network (HEALNet) (Hemker et al., 2024), which is using an end-to-end trained iterative cross-attention architecture, and the Multimodal Co-Attention Transformer (MCAT) (Chen et al. 202T), which is using the tabular (1D) modality as context for the imaging (2D) modality.
Implementation Details. For each experiment and dataset, we perform a 5-fold repeated random sub-sampling with a 70-15-15 train-estt-allidation split. We re-ran all of the baseline models in this paper using their open-soure code to ensure that no performance differences are caused by different task setups, losses, training splits. We ran brief Bayesian Hyperparameter search (Biases) (024) for key parameters of each model (learning rate, decay, schedulers. dropout, layer dimensions). The experiments were run single Nvidia A 100 80GB GPU Ubuntu 22,04 virtual machine. The

---

Published as a conference paper at ICLR 2025
code implementation for MM-Lego is available at mm -ego
5 RESULTS
Table I: Mean and standard deviation of unimodal and multimodal task performance, showing the concordance Index (survival analysis tasks) and AUC (classification tasks) on random sub- sampling folds with the best and second-best models highlighted. LegoMerge achieves competitive performance with multimodal baselines across all tasks without end-to-end training or any fine-tuning. LegoFuse achieves top-2 performance for all datasets and has the highest performance amongst all benchmarked models in five out of seven datasets. CC and BL denote the monolithic fusion operators (-) of concatenation and bilinear fusion respectively. Modalities: img=image; ts=time series; mul=mutations (binary); env=copy number variations (categorical); ma=gene expressions (continuous); tab=other tabular
BLCA
BRCA
KIRP
UCEC
ICD9
MORT
ISIC
Somples
n-32616 tab, Is
Modalities
ing. mut,
img. mul,
img. mut.
img, mul.
tab, Is
lab, img
I-Index
Unimodal (Tahular)
MultiMeNNN
0.500 0.500=0000
0.500
0.525 20.140
0.5 2.500
0.500 1.50002..00
0.500
0.500
Ponceiver
0.686
0.557
0.836
0.615
0.629 jioccs
0.658
0.840
LogoBlock
0.645_jaut 1aurT
0.619guazs
1411
Unimodal (Inaag//iiee-srrie)))
0.610
0.741 =2
0.558
0.614
2.601
0.500
MulnMMoNN
p.564
0.500
0.544
0.500
Penceiver
0.532
2.604
0.716
2.534
0.700
0.7 0.715 2.055
0.719
LegoBlock
10 009
n.533
0.630
0.565
0.643
0.701
0.706
Multimodal
LegoMerge
0.701
0.601 ganess
0.825 gn
0.625 1,0000
0.684_20055
0.751
0.721
Uptil (Merge va, best Block)
2.9%
1.7%
-1.8%
1.6%
5.7%
5.3%
2.1%
SNN ABMIL (CC, Late)
0.561
0,841
0.601
0.628_10.o0
0.617
0,661
(BL,
0.557
+0.10
0.666-20a
0.500++ax00
0.5000-
0.501 Hoxe
Perceiver (CC, Early)
0.547-areo
0.561 +saas
0.692 a.000
0.548.+uon
0.733
0.723
0.721 0.721 H..19
MultiMeNN (Inter)
0.524
2.500
0.602
0.512
0,500
20000
0.500_0000
MCAT (Inter)
0.702-a181
1.564+
+uuns
4
0.500++ax00
0.500
0.627
HE Het (Inter))
0.714
0.842-uses
0.594
0.767
0.748
0.639 cxm
LegoFuse, w/ spochs
0.734-
0.626
0.863
0.634:
0.771
0.759
0.701
The results across the three prediction tasks (survival analysis, multi-class and binary classification) are summarised in [ showing the mean and standard deviation of the task-relevant performance metric across the random sub-sampling folds. We compare our baselines to: LegoMerge, which is taking two LegoBlocks, trained unimodally, and merges them without fine-tuning, and 2) LegoFuse,
BLCA
BRCA
KIRP
UCEC
ICD9
MORT
ISIC
1.0
C-Index/AlIC
0.8
0.6
Uplift: 2.0%
Uplift: 1.0%
Uplift: -1.5%
Uplift: 1.0%
Upllft: 3.9%
Uplift: 4.0%
Uplift: 1.5%
Figure 3: Mean task performance (concordance Index/AUC) of LegoBlock (Tabular), LegoBlock (Image/Time Series) and LegoMerge, showing the increase in task performance by applying multimodal model merge without any fine-tuning. Our proposed multimodal model merge shows a positive performance improvement on 6 out of 7 datasets.
7

---

Published as a conference paper at ICLR 2025
MIMIC-ICDD
MIMIC MorialMMrt
0.80
0,75
0.75
0.75
AUC
0.89
0.08
0.05
0.63
0.55
SNN
AMIL
Ensembie Model
LegoMerge
LegoFuse
SNN
AMIL
Ensamble Model
LegoMerge
LegoRuse
Figure 4: AUC performance the MIMIC dataset when merging existing encoders (SNN for tabular, AMIL for Time Series) using LegoMerge and LegoFuse. Our multimodal model merge shows much better performance than using ensemble, exhibiting the performance gains, at additional costs, through the merge even prior to fine-tuning in LegoFuse.
which is taking the same blocks and fine-tuning them for two epochs. Note that, we did not see sigmificant performance difference between the "stack" and "weave" variants of LegoFuse, therefore the reported results in Table [ correspond to stacked blocks. Across all datasets, LegoFuse is within the top 2 performers of all uni- and multimodal datasets with only two epochs of fine-tuning.
Both LegoFus use and LegoMerge generalise well across domains (pathology, clinical care, dermatology), which some unimodal and multimodal baselines struggle with and heavily overfit (AUC of 0.5). Despite never seeing a single multimodal training step. LegoMerge achieves strong results, closely matching the performance of the best baselines in six out of seven datasets. We note that the performance stability across folds is in line with or better than the multimodal baselines for all datasets except TCGA-KIRP where we have significantly larger standard deviations. We believe that this instability caused by the relatively low sample size of this dataset, and posit that this could be alleviated by pre-training LegoBlocks on other TCGA datasets before fine-tuning them KIRP. The SIMM-ISIC dataset exhibits a case of multimodal collapse (also referred to as dominance), where all multimodal models struggle to effectively take advantage of both modalities.
We assess the gains afforded by LegoMerge in Figure 3 where we use LegoMerge to merge LegoBlock for tabular data with either LegoBlock for imaging or LegoBlock for times series data (depending on the dataset). We compare the performance of the unimodal LegoBlocks with the resulting multimodal LegoMerge and find a performance uplift gained from the merge. Namely, the harmonic mean (Equation 4 of each block's latent states coupled with spherical linear interpolation of the weights and biases in the task heads leads better performance than for either unimodal block in 6 out of 7 datasets. Again, the outlier to this trend is the dataset with the lowest sample size (KIRP) where LegoMerge generally struggles for stability. As outlined in Figure II the blocks can either be trained as models from scratch or can be used as wrapper for a unimodal encoder. This is demonstrated in Figure 4 which shows the performance of the SNN and AMIL,, compared to three multimodal combinations of the two: 1) naive ensemble that is taking the average logits of each encoder, 2) LegoMerge applied after wrapping each encoder in LegoBlock, and 3) LegoFuse with limited fine-tuning. For both MIMIC tasks,
see that LegoMerge beats the ensemble by 7.1 % and 7.3% on the MIMIC disease classification (ICD9) and mortality prediction (MORT) respectively. all other models by further 1-3%.
Moreover, LegoFuse improves the performance
6 DISCUSSION
In this work, we introduce two novel approaches for both multimodal model merging (LegoMerge) and fusion (LegoFuse) to address some common limitations of existing methods for multimodal modelling. We outline five desirable criteria of multimodal models for biomedical data Tabel 2 to highlight scenarios in which we believe introduced methods to be highly beneficial.
Performance without end-to-end training. With the increasing volume, complexity and diversity of collected biomedical data, (re)training multimodal models from scratch becomes more expensive, unsustainable, and even infeasible in the long run. Similar to trends seen in large language models, providing access fine-tuning and merging frameworks can help to make established state-of-the-nn art unimodal models more accessible to tailor for highly specific applications, as is commonly a requirement in medicine. The results in Table that we can achieve state-of-the-art performance
8

---

Published as a conference paper at ICLR 2025
by combining existing, pre-trained unimodal models and fine-tuning them accordingly: LegoFuse outperforms end-to-end trained baselines with as little as 2 epochs of fine-tuning. We anticipate that LegoMerge, where no multimodal supervised data is required, and LegoFuse, where only limited number of supervised paired samples are required, can aid in fields where paired observations are searce. These scenarios are frequent and very realistic, from studies on rare diseases data gathered from clinies that do not have access to sophisticated data acquisition technologies. In such cases, training large encoders wrapped in LegoBlocks on similar domains, and then fine-tuning them on small amount of supervised paired samples may present feasible path for domain adaptation for multimodal models. A further advantage of our modular approach is that additional data (from either the same or novel modalities) can be added at different points in time. For example, if large bimodal model was already trained and a new modality becomes available later on, MM-Lego can be readily applied to extend such model without training a new model, saving time, costs and energy.
Missing modalities and modality imbalance. The results in Section[Jsuppon our hypothesis that we can build performant multimodal models without having perfectly paired training data. This is evident by the highly competitive performance of LegoMerge in comparison with existing multimodal base- lines (Table I) Paired data requirements can be problematic when modalities of the samples, leading to a data-performance trade-off: one can either attempt to use all the available data and impute the missing values, which introduces noise, or take the intersection of samples with available data modalities, which may dramatically reduce the sample size available for training.
missing for some
MM-Lego overcomes this data-performance trade-off as it can be effectively trained on non- overlapping training sets. This is further supported by our experiments (see Appendix (G), where we found that the performance of MM-Lego remains stable even when trained with completely non-overlapping (symmetrically different) sets of samples. In contrast, such a training scenario infeasible for any other current end-to-end model setup. Moreover, MM-Lego can effectively handle missing modalities. During training, value imputation is not an issue for MM-Lego, since we can train LegoBlocks independently, and subsequently combine them into a performant model, as shown in Figure 4 During inference, we can easily skip missing modalities, which is another benefit of making each block compatible with an iterative attention architecture (Equation [T]. That is, while monolithic fusion operator v²(-) requires the entire set of unimodal encodings H to be present, MM-Lego's s architecture can just query the elements in B for the available modalities. Fi- nally, MM-Lego's modular design allows for handling high modality imbalance and me-to--mayy cardinalities ( ((X(A))= o (A)).). An example of such a case would be dataset where A are the histopathology slides (one per patient) and is the associated single-cell data (many per patient).
Architecture agnostic. A key limitation of existing model merging literature in multi-task learning is the assumption that the majority of the network topology between tasks is equivalent. While this is a feasible assumption for merging in multi-task learning, the data heterogeneity limits its application in multimodal settings. Therefore, the design of LegoBlock is sufficiently permissive to use any unimodal encoder as part of this framework, while complying with the necessary architectural assumptions required for model merging. Our results in Table I and Figure that any unimodal encoder (SNNs and AMIL in this example) can be wrapped in LegoBlock without any practical loss in performance, whilst making them capable for further merging and/or fusion.
this by showing
Table 2: Comparison of desirable requirements of multimodal systems for clinical practice. LegoMerge and LegoFuse present two alternatives to existing, end-to-end trained, multimodal fusion approaches (early, intermediate, and late fusion) and multi-task merge methods applied to multimodal settings. v: meets requirement, (V): some approaches meet requirement, X: fails requirement.
Fusion
Criteria/ethhd
Multi-task Merge
LegoMerge
LegoFuse
Early
Intermediate
Late
Learns cross-modal interactions
Architecture agnostic
Handles strong modality imbalance
X
5
X
V
/
Add modalities without re-training
9

---

Published as a conference paper at ICLR 2025
To the best of our knowledge, MM-Lego is the first purppsse model merging framework for multimodal data outside of the vision & language domain.
Low computational requirements. Scaling to more than two modalities is increasingly important for multimodal models as more modalities are captured at scale: this was one of the main motivations for the modular design of MM-Lego. Many intermediate fusion approaches (Chen et al., 2021] Xu & Chen 2023) Sundar et al. 2024, Zhang et al., (024) are natively designed for two modalities, and centred around a cross-attention layer between the modalities with time complexity of ( Scaling to more than two modalities requires calculating the modality-guidee cross-attention for all unique pairwise combinations (2)) M(M-1) which is ((M²), leading to total upper bound time complexity of ((M2 MM-Leeg improves this wrt. both M and N. The time complexity of LegoBlock is bound by the cross-attentis unit, which reduces the quadratic time complexity by using the latent I. E Rexd the query to (cdN) for latent dimensions d. The sequential design in Equation T ensures linear scaling wrt. the number of modalities, leading to a final time complexity of O NN, ). Beyond time complexity, another key benefit of MM-Lego is the number of training steps required at the given complexity. For the end-to-end baselines, we typically observe loss convergence around 10 training epochs, while LegoMerge and LegoFuse, by design, require zero or as little as two training epochs, respectively. This results in very low total train time, as shown in Table
Table 3: Train time per epochs and total wall time for training on the TCGA-UCEC dataset on single A100 80GB GPU. For encoders wrapped with LegoBlock during training, LegoMerge achieves competitive performance despite requiring no additional training.
Model
Time/epoch (s)
Train Time (s)
SNN+ABMIL (BL)
5.7
102.6
SNN+ABMIL(CC)
4.7
84.6
Perceiver
9.6
172.8
MultiModN
3.2
57.6
MCAT
4.8
86.4
HEALNet
10.1
181.8
LegoMerge
0
LegoFuse
8.1
16.2
Limitations. We believe that our MM-Lego approach would benefit from further research of parameter-efficient ways to design the fusion layer within each LegoBlock (Equation 33. that is, efficiently encoding (A4) into the updated latent representation (L.+))". Whilst this is suitable in both our solutions and has proven to effective other work (Jaegle et al., 2022 Carreira et al.. 2022, Hemker et al.] 2024). finding more parameter-efficien solution is desirable. Note that, employing the LegoBlock adapters requires minimal fitting. This can be achieved by either training together with the modality-specific encoder or retrospectively, when applied to pre-trained encoder (e.g., a large foundation model). In the latter case, this still requires a small amount of training (supervised) samples for LegoMerge to be effective. As such, MM-Lego would benefit from further research on self-supervised fitting of the adapters. Finally, while in this paper we limit our focus to multimodal problems from biomedical domains, MM-Lego is designed to be general-purpose and applicable to any multimodal tasks. Therefore, we leave it to future work to further demonstrate these properties for more (and other) data modalities and domains, including vision & text tasks.
CONCLUSION
We present MM-Lego, a general-purpor and modular learning framework to build performant multimodal models with minimal fine-tuning. To achieve this, we introduce three novelties. First, we introduce a wrapper for unimodal encoders (LegoBlock) that a) enforces shape consistency between modality representations, and b) harmonises the latent representations in the frequency-domain to enable model merging with little signal interference. Second, we introduce the first multimodal model merge framework (LegoMerge) that goes beyond vision & language modalities and outperforms many unimodal baselines without seeing single multimodal supervised training sample. Third, we show that these building blocks can be combined to construct a fusion model that achieves state-of-the- oh-the-art performance with only few epochs of fine-tuning (LegoFuse).
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors would like thank Philip Schouten for his insightful clinical feedback. KH acknowl- edges support from the Gates Cambridge Trust via the Gates Cambridge Scholarship. NS and MJ acknowledge the support of the U.S. Army Medical Research and Development Command of the Department of Defense; through the FY22 Breast Cancer Research Program of the Congression- ally Directed Medical Research Programs, Clinical Research Extension Award GR TRRANT1376971S 13. Opinions, interpretations, conclusions, and recommendations are those of the authors and are not necessarily endorsed by the Department of Defense.
REFERENCES
Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary Optimization of Model Merging Recipes, March 2024.
Ane Alberdi, Asier Aztiria, and Adrian Basarab. On the early diagnosis of Alzheimer's Disease from multimodal signals: A survey. Artificial Intelligence Medicine, 71:1-29, July 2016. ISSN 0933-3657. doi: 10.101 200
Tadas Baltrusaitis, Chaitanya Ahuja, and Louis Philippe Morency. Multimodal Machine Learning: A Survey and Taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41)) 423 443, February 2019. ISSN 19393539. doi: 10. 10.100//PAMM.2218.2272999 27798677
Alex Belianinov, Anton V Ievlev, Matthias Lorenz, Nikolay Borodinov, Benjamin Doughty, Sergei V Kalinin, Facundo M Fernández, and Olga S Ovchinnikova. Correlated materials characterization via multimodal chemical and functional imaging. ACS nano, 12(12))11798-11 1818, 2018.
Weights & Biases. Sweeps
Sweep configurations, 2024. URL https: sestps://docs.oo wando ai/ ai/guides///////
Kevin M. Bochm, Pegah Khosravi, Rami Vanguri, Jianjiong Gao, and Sohrab P. Shah. Harnessing multimodal data integration to advance precision oncology. Nature Reviews Cancer, 22(2)1144-126.
February 2022. ISSN 1474-1768. doi: 10108//4118-001--000000
Joao Carreira, Skanda Koppula. Daniel Zoran, Adria Recasens, Catalin Ionescu, Olivier Henaff, Evan Shelhamer, Relja Arandjelovic, Matt Botvinick, Oriol Vinyals, Karen Simonyan, Andrew Zisserman, and Andrew Jaegle. HiP: Hierarchical Perceiver, November 2022.
Richard J. Chen, Ming Y. Lu, Wei-Hung Weng, Tiffany Y. Chen, Drew Fk. Williamson, Trevor Manz, Maha Shady, and Faisal Mahmood. Multimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. doi: 10. 109/CCCV4992.202 000998.
Montreal, QC, Canada, October 2021. IEEE. ISBN
978-1-66542-81244
Richard J. Chen, Ming Y. Lu, Drew F.K. Williamson, Tiffany Y. Chen, Jana Lipkova, Zahra Noor, Muhammad Shaban, Maha Shady, Mane Williams, Bumjin Joo, and Faisal Mahmood. Pan- cancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell, 40(8): 865-878.e6, August 2022. ISSN 1535-6108, doi:
Richard J Chen, Tong Ding, Ming Lu, Drew FK Williamson, Guillaume Jaume, Andrew Song, Bowen Chen, Andrew Zhang, Daniel Shao, Muhammad Shaban, et al. Towards general-purpose foundation model for computational pathology. Nature Medicine, 30(3):850-886 2024.
International Skin Imaging Collaboration. SIMM-IIC melanoma classification. 2020. URL https
7/ kaggle. cwm http://oopeetiiiioooo si im -eliccmmelaannn--claaaee
datal
Zhanbei Cui, Tongda Xu, Jia Wang, Yu Liao, and Yan Wang. Geneformer: Learned gene compression using transformer-based context modeling. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 0035- IEEE, 2024.
Xing Han, Huy Nguyen, Carl Harris, Nhat Ho, and Suchi Saria. FuseMoE: Mixture-of Cetture-of-Expeeee of-Experts Transformers for Fleximodal Fusion, February 2024.

---

Published as a conference paper at ICLR 2025
Konstantin Hemker, Nikola Smidjievski, and Mateja Jamnik. Healnet-hybree multi-modal fusion for heterogeneous biomedical data. NeurIPS 2024, 2024.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing Models with Task Arithmetic, March 2023.
NIH National Cancer Institute. The Cancer Genome Atlas Program (TCGA), 2006. URL https
cancer. //wwwcancer..c www.nnccerrov/eeooooooooooo
7/www.
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver: General Perception with Iterative Attention. ICML, March 2021. doi: 10.48550/arxiv. 2103.03206.
Andrew Jaegle, Sebastian Borgeaud, Jean- Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver IO: A General Archi- tecture for Structured Inputs & Outputs. ICLR, July 2022. doi: (0.48550/mmxx.21177.4477
Xisen Jin, Xiang Ren, Daniel Preotiuc-Piette and Pengxiang Cheng. Dataless Knowledge Fusion by Merging Weights of Language Models, October 2023.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-Normalizing
Neural Networks, September 2017.
Maxime Labonne.
Merge Large Language http://gginggg.ooooooooo///////n models 2024.
Models
with
mergekit.
James Lee- Leerph Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms. arXiv preprint arXiv:2105.0388 2021.
Jiao Li, Xing Xu, Wei Yu, Fumin Shen, Zuo Cao. Kai Zuo, and Heng Tao Shen. Hybrid Fusion Image-Recipe
with Intra- and Cross- Modaaity Attention for Image Recipe Retrieval. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, pp. 244-254, New York, NY, USA, July 2021a. Association for Computing Machinery. ISBN 978-1 4503-8037-9
doi: 10. 145533404835.4462966
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before Fuse: Vision and Language Representation Learning with Momentum Distillation. In Advances in Neural Information Processing Systems, volume 34, pp. 9694-9705. Curran Associates, Inc., 2021b.
Ruiqing Li, Xingqi Wu, Ao Li, and Minghui Wang. Hfbsurv: hierarchical multimodal fusion with factorized bilinear models for cancer survival prediction. Bioinformatics, 38(9):55772254, 2022.
Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimban, Paul Pu Liang, AmirAli Bagher Zadeh, and Louis-Philippe Morency. Efficient Low-rank Multimodal Fusion With Modality-Speciffe Factors. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2247-2256, Melbourne, Australia, July 2018. Association for Computational Linguisties. doi: 10. 18.18653/v 1/P18- 1209..
Giovanni Luca Marchetti, Christopher Hillar, Danica Kragic, and Sophia Sanborn. Harmonics of Learning: Universal Fourier Features Emerge in Invariant Networks, December 2023.
Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Multimodal Contrastive Learning with LIMoE: The Language- Image Mixture of Experts, June 2022.
Arsha Nagrani, Shan Yang, Anurag Amnb, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention Bottlenecks for Multimodal Fusion. In Advances in Neural Information Processing Systems, volume 34, pp. 12200-14213 Curran Associates, Inc., 2021.
12

---

Published as a conference paper at ICLR 2025
Eric Nguyen, Michael Poli, Matthew G Durrant, Armin W Thomas, Brian Kang, Jeremy Sullivan, Madelena Y Ng. Ashley Lewis, Aman Patel, Aaron Lou, et al. Sequence modeling and design from molecular to genome scale with evo. bioRxiv, pp- 2024-022200 02, 2024.
Henri J Nussbaumer. Fast Fourier Transform and Convolution Algorithms. Springer, 1982.
Marc-Antoine Parseval. Mémoire sur les séries et sur l'intégration complète d' d'une équation aux différences partielles linéaires du second ordre. à coefficients constants. Mém. prés. par divers savants, Acad. des Sciences, Paris,(1). :263--648):2,, 1806.
Johnathan Pocock, Simon Graham, Quoc Dang Vu, Mostafa Jahanifar, Srijay Deshpande, Giorgos Hadjigeorghiou, Adam Shephard, Raja Muhammad Saad Bashir, Mohsin Bilal, Wenqi Lu, al. Tiatoolbox as an end-to-end library for advanced tissue image analytics. Communications medicine, 2(1):120, 2022.
Zhucheng Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, and Yongbing Zhang. TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classication. In Neural Information Processing Systems, June 2021.
Ken Shoemake. Animating rotation with quaternion curves. In Proceedings of the 12th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '85, pp. 245-22-222 254, New York, NY, USA, July 1985. Association for Computing Machinery. ISBN 978-0- 897919-697-615 doi: 10.1145/92533..325222
Steven W Smith et al. The complex fourier transforms, ch. 31 in the scientist and engineer's guide to digital signal processing, 1997.
George Stoica, Daniel Bolya, Jakob Bjorner, Pratik Ramesh, Taylor Hearn, and Judy Hoffman. Ziplt! Merging Models from Different Tasks without Training. In ICLR 2024. arXiv, March 2024.
Anirudh S. Sundar, Chao-Han Huck Yang, David M. Chan, Shalini Ghosh, Venkatesh Ravichandran, and Phani Sankar Nidadavolu. Multimodal Attention Merging for Improved Speech Recognition and Audio Event Classification, February 2024.
Yi-Lin Sung, Linjie Li, Kevin Lin, Zhe Gan, Mohit Bansal, and Lijuan Wang. Empirical Study of Multimodal Model Merging, October 2023.
Vinitra Swamy, Malika Satayeva, Jibril Frej, Thierry Bossy, Thijs Vogels, Martin Jaggi, Tanja Käser, and Mary-Anne Hartley. MultiModN- Multimodal, Multi-Task Task, Interpretable Modular Networks, November 2023.
Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno, Ira Ktena, Basil Mustafa, Aakanksha Chowdhery, Yun Liu, Simon Kornblith, David Fleet, Philip Mansfield, Sushant Prakash, Renee Wong, Sunny Virmani, Christopher Semturs, S. Sara Mahdavi, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Joelle Barral, Dale Webster, Greg S. Corrado, Yossi Matias, Karan Singhal, Pete Florence, Alan Karthikesalingam, and Vivek Natarajan. Towards Generalist Biomedical AI, July 2023.
Ashish Vaswani, Google Brain, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. Advances in Neural Information Processing Systems, 30, 2017.
Yingxue Xu and Hao Chen. Multimodal Optimal Transport- Co-Attention Transformer with Global Structure Consistency for Survival Prediction. In 2023 IEEE/CVF International Confer- ence on Computer Vision (ICCV), pp. 21184-21 194, Paris, France, October 2023. IEEE. ISBN 9798350307184 184. doi: 10. 10011999111100000220.00099
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit Bansal. TIES-Merging: Resolving Interference When Merging Models. Advances in Neural Information Processing Systems, 36:7093-7115, December 2023.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch, February 2024.
13

---

Published as a conference paper at ICLR 2025
Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep Modular Co-Attention Networks for Visual Question Answering. In 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6274-6283, Long Beach, CA, USA, June 2019. IEEE. ISBN 978-1- 72813-299 12813-293-8. doi: 201.1/9//V....9.......
Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Tensor Fusion Network for Multimodal Sentiment Analysis. Martha Palmer. Rebecca Hwa, and Sebastian Riedel (eds.). Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1103-1111 1114,, 14, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 1053////11/DD11-113
Shekoufeh Gorgi Zadeh and Matthias Schmid. Bias in Tross-Entropy-Bat Based Training of Deep Survival Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9): 3126-3137, Septemher 2021. ISSN 1939-3539. doi: 10.1 111109TPAMII.000 (000//PAMM.22202299994555
Yilan Zhang, Yingxue Xu, Jianqi Chen, Fengying Xie, and Hao Chen. Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Prediction, February 2024.
Tetiana Zubatiuk and Olexandr Isayev. Development of multimodal machine learning potentials: toward a physics-aware artificial intelligence. Accounts of Chemical Research, 1477):1575-1585 2021.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A NOTATION
Objects.
X matrix corresponding to modality A
();; vector in ((A) (e.g., sample of modality A)
x (A)
i.j. elements of matrix (A) at row i, column j. channel k, assuming ( E RRIJxx
where v i S SJ S J, s k
( = UMEM X(m) multimodal dataset
y E U y()); of task labels for all available tasks T
y (Ti); task labels for task Ti
Sets.
M: set modalities
T: set of tasks
V: set of task task-specific heads
g (gm m h(m) m MM: set of mopality-specific encoders
Hy {gm (m, (m.y)m MM} set of task- and modality sppecifcc embeddings
B {t (9m (X(m)) (())
I (m)
st1
88 E S.m. m E .M}: set of LegoBlocks
Functions and Operators.
gm (+): modality-speciff encoder
(+): fusion operator (monolithic)
Vm ()) modality-specifie latent update
F: Fourier transform
Inverse Fourier transform
15

---

Published as a conference paper at ICLR 2025
B PROPERTIES OF FREQUENCY-DOMAIN REPRESENTATIONS
In this section, we further elaborate on the desirable properties of frequency-domain representations that motivate MM-Lego design.
The general motivation for eegnal-preservnng aggregation for multimodal machine learning has been extensively studied in the context of fusion methods. For example, bilinear pooling of two latent vectors hy E RR h2 E IRm and resulting tensor 412 IRd (y = h Ah1 b) with learnable parameter A E Raxmxd follows the same motivation. Several related works in different domains and modalities [1, 2, 3] emphasise the importance of fusion aggregation methods that avoid signal interference, where meaningful patterns or complementary information from individual modalities may be attenuated, distorted, or lost due naive aggregation approaches such as averaging summation. With the same motivation in mind, we take advantage of the Fourier transform's 's orthogonal properties, as after the transform each frequency component (represented by sine and cosine waves) is orthogonal to the others, such that f_ sin sin(wi ha (cos(w2)) )dh = 0 for angular frequencies W1 F W2 [4]. This means that the contribution of each frequency is independent of others and there is no overlap between them. It follows that if take the harmonic mean two fourier-transformed latents (w) and (2)) as H(w) = 2.5 (a((())
Signal preserving:
(1) the harmonic mean aggregates each w in a localised manner. This means that each frequency component in Fiw)) interacts only with the corresponding frequency component in F2 i.e., without interference from other frequencies.
Distance preserving: Parseval's Theorem shows that the Fourier transform operator, meaning that the sum or integral of the square of function is equal to the sum or square of its transform [4,5]. As such, the distances between two signals are the same between the transformed and untransformed representations. Concretely, the theorem states that the energy of signal is preserved in both the time domain and frequency domain, where its energy is measured as the integral of the function.
unitary
Formally 1 [f S (w) de for latent signal fh) and its fourier transform F.
The Euclidean distance between two signals fi (h) and (2(h) in the spatial domain is:
11/1 f2ll V Ifi (h) [2h)
- The Fuclidean distance between the Fourier transforms of two signals is ([F(1)
F(J2)lI - - [F())(w) - F()
From Parseval's Theorem, it follows that Ilfr ( -
TThis prrtenne- capability is beneficial in designing loss functions in multi- modal setting.
Invertible: The Fourier transform not an idempotent function but periodic with period of 4, F')) f. This would prevent the iterative architecture outlined in Section from working since repeat application of the transform would lead to different representations. Meanwhile, the Fourier Inversion Theorem [6] shows that we can invert the frequeney- domain representation to its original function without the 4x repeat application, making it suitable for the chosen iterative architecture.
Efficient: The Fast Fourier Transform (FFT) has time complexity of ((nlog(n) making it scalable to very large datasets.
16

---

Published as a conference paper at ICLR 2025
C DATASETS
We evaluate MM-Lego (LegoMerge and LegoFuse) and its components (LegoBlock) on seven multi- modal medical datasets covering three separate modalities (images, tabular, time series) from three separate sources: histopathology (The Cancer Genome Atlas (TCGA)) Inntitute (2006), intensive care data (Medical Information Mart for Intensive Care (MIMIC) Johnson et al. (2016), and skin imaging (Society for Imaging Informatics in Medicine & International Skin Imaging Collaboration (SIM--SSIC) Collaboration (2020).
TCGA: Some of the results shown in this paper here are based upon data generated by the TCGA Research Network: Intpp:: /www. cancer. gov/t The Cancer Genome Atlas (TCGA) is open-source genomics program run by the United State National Cancer Institute (NCI) and National Human Genome Research Institute, containing a total of 2.5 petabyts of genomic, epigenomic, transcriptomic, and proteomic data. We predict survival of right-censored patients based on the high-resolution histopathology slides 80, 000 x 80, 000 pixels) and multi-omic data (gene expressions, copy number variations and gene mutations) captured from bulk sequencing in tabular format. We train on four separate cancer cohorts with multimodal data available: Urorethelial Bladder Carcinoma (BLCA, 436), Breast Invasive Carcinoma (BRCA, 1021), Kidney Renal Papillary Cell Carcinoma (KIRP, 284), and Uterine Corpus Endometrical Carcinoma (UCEC, 538).
We use the following encoders for each modality in the TCGA datasets:
WSI Encoder
Sample Input: Raw image converted to a bag of patches, dwst Nputches X 256 X 256 x 3
Encoded input using ResNet50, pre-trained on Kather100k: hwss Apatchee X 2048
Mutation Encoder
Non-variable mutation genes were filtered (i.e., every sample contains mutation or none contains mutation)
Input:
Raw input mutation data vector for cross-attention (Figure 2)
Encoder: none the vector is already relatively small after ETL: dmmat hmoat 21
.
Copy Number Variation Encoder
Non-variable copy number genes were filtered
Input:
Copy number variations for each gene (categorical variable ranging -2 (deep deletion) to +2 (strong oncogenic amplification): dens 1333
Encoder: SNN (1333 + 512): hove 512
Gene Expression Encoder
Low variable genes were filtered
Input: loglp transformed bulk RNAseq gene expression: drna- 1558
Encoder: SNN(1558 + 512): hyna = 512
We train models on two separate tasks: patient mortality (multi-class classification) and disease classification (ICD-9 codes), which we formulate as a binary classification task. We use both clinical variables and small time series data on various vital signs measured at 24 time steps. Both tasks have 32616 and the same feature set for different task labels.
MIMIC-11I:
Stems from the Society for Imaging Informatics in Medicine & International Skin Imaging Collaboration (SIM-ISII) melanoma classification Kaggle challenge Collaboration (2020). which contains both tabular data and images of skin leisures to be classified for melanoma patients. To account for class imbalance, we randomly downsampled the majority class to 5:1 ratio for the class of interest (melanoma) to sample size of n 2875. All images were patched and encoded using the Saanet50-kahher 100k for TCGA (ResNet pre-trained on a large histopathology patch collection) Pocock et al.) (2022) and regular ImageNet v2 pre-trae pre-trained ResNet for the pictures of skin leisures. Both images (patch encodings) and times series were represented as 2D tensors, and the tabular clinical and multi-omic data as 1D tensors to pass into the modality-specific encoders g(-).
SHIM-ISIC:
17

---

Published as a conference paper at ICLR 2025
D LOSSES AND METRICS
The results report the (unseen) test set performance, by evaluating the concordance Index (c-Index) in the case of TCGA, AUC in the for MIMIC-II-ICDD, As indicated in Figure thhe output of each task head in y are the logits with predictions for each class given the final is survival prediction task with right- ceeht-cd data, we have divided the survival period into four non-overlapping bins and use the logits of these bins to calculate the hazard (3/1 Ie and survival
MIMIC-IIIICDD and ISIC, and Macro-AUC ("one-vs-rest")
latent state y = f(Lf) ). Since TCGA
(y. - -I y/ ) respectively for k bins. Given the hazards, censorship, and ground truth bins, can calculate the negative log-likelihood loss from a proportional hazards model Zadeh & Schmid (202T) which is used as the survival loss. We evaluate the performance using the Concordance Index (c-Index), for which we determine the fraction of paired samples in which the prediction outcomes are concordant with the ground truth. As MIMIC and ISIC relate classification tasks, employ categorical cross-entropy loss for training. Note that both AUC and the e-Index have similar interpretations, therefore the values range between [0.5 1].
18

---

Published as a conference paper at ICLR 2025
E HYPERPARAMETERS
Scope
Parameter
Value
Learning Rate
0.003
Epochs
40
Shared
Early Stopping Patience
L1 Regularization
0.0002
Batch size
512
Optimizer
Adam
LR Scheduler
Reducel..ROnPlateuu
Tune Epochs
Fuse Method
Stack Harmonic SLERP
Merge method
MM-Lego
Head method Alpha
0.5
Track imaginary
True True 17 x 126
Normalise
Latent dims
Depth
Attention Dropout
0.45
FCNN Dropout
0.36
Latent dims
I000
MultiModN
Error penalty
State change penalty
Layer dims Depth
512, 256, 128, 64
2
HEALNet
Lafent dims
17x126
Attn Head dims
64
Attention Dropout
0.4
FCNN Dropout
0,27
Layer sizes
1024, 512, 256
AMIL
Dropout
0
Attention Dropout
0.25
Layer dims
256
SNN
Depth
Dropout
0.25
Depth
2
Latent dims
17x126
Perceiver
Attn Head dims
64
Attention Dropout
0.4
FCNN Dropout
0.36
AMII layers
1024, 3T2, 256
AMIL dropout
0
MCAT
Attention Dropout
0.25
SNN Layer dims
256
SNN Depth
SNN Dropout
0.25
19

---

Published as a conference paper at ICLR 2025
F
SIGNAL INTERFERENCE ON LATENT VARIABLES
Mod A (normal distribution)
FFT(Mod
60
50
Magnitude
40
200
400
600
800
1000
0.0
0.1
0.2
0.3
0.4
0.5
Sample
Frequency
Mod (normal distribution)
FFT(Mod
Megnitude
Value
200
400
600
800
1000
0.0
0.2
0.3
0.4
0.5
Sample Index
Frequency
Mean(A,
Harmonic (Fan(FFT(A), FFT(B))
60
Magnitude
Value
40
800
800
Sample Index
Prequency
Figure 5: Example of signal interference on a random normal latent variable and its additive inverse variable with some added noise, showcasing severe case signal interference where nearly all signal cancels out. We can see that the we apply the harmonic mean. This is key reason for the choice model merging architecture.
data does not suffer this problem when
20

---

Published as a conference paper at ICLR 2025
Mod (scuare mave, f-5, rundom amplitude)
FFTIMod
y
202
100
1:M N. And 1 1n Aoddin Teah 1o Ma Rnnd
moquercy
Mod F.5.5. amplitude)
FFT(Mod
2.0
200
200
150
09
100
-0.5
5
50
ROQUETA FFTM--B)
vapWhd:
1100
Inoqurrey
MeanUAl, 1912
16
Figure 6: The argument against Fig. 5 would be use absolute only positive values. This example shows that this logic can also be flawed. We demonstrate this using a squarewave function with a frequency offset beteen M a and Modg and scaled amplitude by a normal distribution. We can see that the mean of the regular and the absolute values suffers some signal interference while the FFT aggregation does not.
21

---

Published as a conference paper at ICLR 2025
G TRAINING ON UNPAIRED DATA
MIMIC-ICD9 Symmetric Difference vs AUC
0.800
legomerge
0.775
ensemble
0.750
C-index Mean
0.725
0.700
0.675
0.650
0.625
0.600
0.0
0.2
0.4
0.6
0.8
1.0
Symmetric Difference
Figure 7: Test performance of LegoMerge (SNN+AMIL) compared to the SNN-AMIL ensemble when training on different levels of overlapping samples between the modalities. A symmetric difference of means no overlap between the samples, 0 being perfect overlap. We selected N=10,000 MIMIC examples for this experiment.
22


Reviews:
[{"review_id": "eiIZh1gbes", "reviewer": "Reviewer_rH9P", "summary": "The authors study the multimodal learning problem and propose a novel method that can merge encoders from different modalities freely. This is achieved through their newly proposed Lego block, which iteratively applies a discrete FFT to modality-specific dataset X^m and modality-specific latent representation L^m. The authors further propose LegoMerge, which can merge Lego blocks from different modalities without any additional training, and LegoFuse, which merge Lego blocks by stacking all the blocks and fine-tuning.", "questions": "1. I suggest to carry out the same type of experiments on CheXpert (https://arxiv.org/abs/1901.07031), where the downstream task can be disease classification. The experiments can be done only on the subset of CheXpert given the limited time during rebuttal.\n2. If possible, I'd like to see the performance on datasets with >=3 modalities.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The idea of the LegoBlocks is novel and interesting.\n2. Both LegoMerge and LegoFuse show promising results.\n3. The wrapper nature of LegoBlocks allows its application to wider range of tasks.", "weaknesses": "1. The experiments do not include the merge between image and textual data, which might be the most common/important scenario. \n2. The authors state one of the advantage of the proposed method is the ability to merge more than 2 modalities, yet all experiments are conducted with only 2 modalities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ivC15tFbWz", "reviewer": "Reviewer_Qibw", "summary": "The presented work introduces multimodal Lego, a novel strategy to fuse latent embeddings from different domains, such as representations learned from images, tabular or time series. The authors hypothesize that processing and fusing embeddings in the frequency domain allows them to be agnostic towards the topology of the representation space and avoid signal interference when merging embeddings. To this end, the authors propose a so-called \"LegoBlock\" that converts multimodal embeddings via a Fourier transform before iteratively updating them via an attention mechanism. Multiple of these modules can be combined by calculating the harmonic mean between the embeddings (\"LegoMerge\") or by training a cascade of modules that sequentially update the latent embedding based on information from the various domains (\"LegoFuse\"). The utility of the method is demonstrated on three medical datasets that each combine two data modalities. LegoMerge is shown to outperform unimodal methods, while LegoFuse outperforms various recent multimodal baselines.", "questions": "Questions and suggestions regarding the work's motivation:\n\n- The authors motivate processing the embeddings in frequency space by citing the advantageous properties of Fourier space with regard to its compactness, robustness against noise and robustness against signal interference. However, several of the referenced works are rather unspecific with regard to these claims and the authors do not provide detailed proof. I suggest that the authors expand the corresponding section and include more detailed arguments for this design choice.\n\n- I believe that the statement that \"LegoMerge does not require any finetuning\" is slightly misleading. If I understand the method correctly the LegoBlocks are a wrapper around already existing embeddings that have to be trained themselves in order to tune (at least) the weights of the attention mechanism, $W^q_m$, $W^k_m$, and $W^v_m$. Compared to simply merging the embeddings, this does require additional fine-tuning steps. Accordingly, I do not believe that the statement that LegoMerge does not require any training time (cf. Table 3) is accurate. Consequently, I believe the paper should be adjusted to clearly reflect this.\n\n\nQuestions and suggestions regarding the proposed methodology:\n\n- The paper is ambiguous on whether the raw input embeddings $h^{(A)}$ or their Fourier transform $\\mathcal{F}(h^{(A)})$ are used in the attention mechanism of the LegoBlocks and the calculation of the harmonic mean during LegoMerge. Equations 3 and 4 indicate the former, while Figure 1 depicts the latter case. Please clarify this and clean up the notation.\n\n- Also, in the right-most panel of Figure 1, why is one of the inputs to the LegoBlocks $L^{(A)}$?\n\n- In the LegoBlocks, how is $L_0$ initialized, and is it updated during tuning? \n\n- How is the number of update steps $S$ tuned?\n\n- One of the main claims of the work is its robustness against different topologies of embeddings. What happens if the initial data embeddings, $X^{(A)}$, $X^{(B)}$, $X^{(C)}$, have different dimensionalities and consequently require Fourier transforms of different dimensionalities? It is not apparent to me, why embeddings of time series and tabular information should be two-dimensional as suggested by the paper.\n\n- Why do the authors apply an inverse Fourier transform at the end of each LegoBlock only to convert it back into frequency domain for a subsequent block or the final task head? Do any artifacts occur because of this repeated conversion?\n\n\nQuestions and suggestions regarding the conducted experiments:\n\n- Even after reading Appendix C and D, many aspects about the conducted experiments are unclear to me and I do not believe that it is possible for a reader to reimplement them based on the current description. For example, how were the 80,000 $\\times$ 80,000 histopathology images of the TCIA dataset encoded? Which mortality was assessed during the experiments with the MIMIC-III dataset? Which model was used to encode the time series data? The authors should aim to provide much more detail with regard to the conducted experiments.\n\n- In light of the considerable effort required to fine-tune each LegoBlock (see earlier comments), I am missing a comparable effort to tune a neural network that processes the concatenated multimodal embeddings. Analogously to the proposed method, one could employ a multi-layer perceptron with S layers and an attention mechanism at each layer. If this approach was adequately tuned, it would function as meaningful, simple and potentially powerful baseline, while simultaenously directly ablating the design choice of converting embeddings to Fourier space.\n\n- The authors should include experiments in which three or more different modalities are included. Currently, none of the conducted experiments supports their main claim that their proposed method scales effectively beyond two modalities.\n\n- While the authors have conducted basic experiments demonstrating their method's ability to handle missing data, the conducted experiments are rather simplistic and only included in the supplementary material. Considering they pertain to a main claim of the work, the authors should expand these experiments and work them into the main paper.\n\n- Have the authors conducted experiments whether the order of fusion during LegoFuse influces its performance?\n\n- What is the added benefit of including both Tables 1 and 4 instead of simply merging them?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper researches a very timely topic, the processing and fusion of multimodal medical data. The authors compellingly motivate this in a well written introduction and background section.\n\n- The proposed method of adapting and fusing multimodal embeddings in the frequency domain code appears to be interesting, novel, and have some theoretical grounding.\n\n- The conducted experiments include three public datasets spanning various data domains, as well as several relevant baseline methods.", "weaknesses": "- The authors only provide limited theoretical motivation for their design choice of converting and processing embeddings in frequency space.\n\n- Some parts of the proposed method are not clearly described. In particular, several aspects with regard to the architecture, functioning, training and tuning of the LegoBlocks are ambiguous or not described at all.\n\n- Similarly, the description of the experimental setup is missing key information so that it is probably not possible to reimplement the benchmarking experiments.\n\n- Several of the core claims made by the authors are not backed up by experimental evidence. In particular the method's ability to handle more than two modalities and its computational efficiency are not adequately evaluated.\n\n- Finally, a meaningful ablation study is currently missing.\n\n(All of the these perceived weaknesses are described in much greater detail in form of specific questions and suggestions in the next section.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "32OyHDQKyr", "reviewer": "Reviewer_Ph1F", "summary": "This paper introduces Multimodal Lego (MM-Lego), a flexible framework for transforming any set of encoders into a multimodal model with minimal or no fine-tuning. By using a wrapper mechanism to ensure shape consistency among different modality representations, MM-Lego enables efficient fusion of representations in the frequency domain, reducing signal interference across modalities. Results on seven tasks demonstrate MM-Lego’s strong performance.", "questions": "1. Could additional ablation studies be conducted to establish the necessity of frequency-domain representations in both LegoBlock and LegoMerge? For example, how would performance change if fusion occurred in the spatial domain?\n\n2. Could the performance of LegoBlock be included in Table 1, not just in Table 4, to facilitate a more direct comparison in the main text? Additionally, certain phenomena, such as LegoBlock’s occasional underperformance compared to individual encoders like SNN or ABMIL, could benefit from further discussion.\n\n3. A case study involving the fusion of more than two modalities would strengthen the paper’s claims about MM-Lego’s multimodal capabilities.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. MM-Lego offers a versatile solution to integrate various encoders into a single multimodal model with limited fine-tuning, bringing novelty with its wrapper and frequency-domain fusion approach.\n\n2. The paper provides an extensive comparison with existing multimodal fusion methods, addressing multiple advanced criteria and establishing MM-Lego’s relative advantages.\n\n3. Comprehensive experiments validate MM-Lego’s performance, particularly in the biomedical domain.", "weaknesses": "1. A primary concern is the lack of rigorous evidence supporting the claim that frequency-domain merging avoids signal interference. Unlike data types like images or time series, frequency representations of high-dimensional embeddings lack intuitive clarity, making the necessity of this approach uncertain.\n\n2. While the authors highlight MM-Lego’s ability to handle multiple modalities, the paper lacks experiments involving more than two modalities. This limitation leaves the model’s robustness in such scenarios untested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AcsZPOCDqX", "reviewer": "Reviewer_fqpF", "summary": "This paper presents MM-Lego, a general-purpose and modular learning framework to build performant multimodal models with minimal fine-tuning.  They introduce a wrapper for unimodal encoders to enforce shape consistency between modality representations and also harmonize the latent representations in the frequency domain to enable model merging with little signal interference. This paper conduct experiments on biomedical datasets and shows good results.", "questions": "Please see the weakness.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The study problem is interesting and useful. How to fuse the feature/prediction from multimodal data better.\n2. The proposed framework demonstrate good results on the datasets.", "weaknesses": "In general, the presentation of this paper needs improvement, and the experimental section should be enhanced. \n\n1. The motivation for some key parts is not clearly articulated. For example, the authors mention \"topological equivalence\" and \"topology agnostic.\" However, it is unclear why these properties are necessary for the problem being addressed. \n\n2. The proposed framework falls under the category of \"model merging.\" However, I am uncertain if this classification is accurate. Essentially, the authors aim to explore how to enhance the fusion of different modalities. \n\n3. The key technical advancement appears to be the feature fusion conducted in the frequency domain. Again, the motivation for this approach is not well presented. Why is this method necessary? \n\n4. The authors claim that the proposed framework is scalable, particularly regarding the number of modalities. However, in most of the experiments, they only utilize two modalities. This claim requires more experimental results to substantiate it. \n\n5. It is unclear how the proposed framework addresses the issue of missing modality data, which is an important challenge in the field. \n\n6. The authors should elaborate more on the experimental settings and the methods used for comparison. For instance, it would be helpful to know if they employed the same feature backbone across experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)