Multimodal Lego: Model Merging and Fine-Tuning Across Topologies and Modalities in Biomedicine

## Abstract

Abstract Learning holistic computational representations in physical, chemical or biological systems requires the ability to process information from different distributions and modalities within the same model.
Thus, the demand for multimodal machine learning models has sharply risen for modalities that go beyond vision and language, such as sequences, graphs, time series, or tabular data. While there are many available multimodal fusion and alignment approaches, most of them require end-to-end training, scale quadratically with the number of modalities, cannot handle cases of high modality imbalance in the training set, or are highly topology-specific, making them too restrictive for many biomedical learning tasks. This paper presents Multimodal Lego (MM-Lego), a modular and general-purpose fusion and model merging framework to turn any set of encoders into a competitive multimodal model with no or minimal fine-tuning. We achieve this by introducing a wrapper for unimodal encoders that enforces lightweight dimensionality assumptions between modalities and harmonises their representations by learning features in the frequency domain to enable model merging with little signal interference. We show that MM-Lego 1) can be used as a model merging method which achieves competitive performance with end-to-end fusion models without any fine-tuning , 2) can operate on any unimodal encoder, and 3) is a model fusion method that, with minimal fine-tuning, achieves state-of-the-art results on six benchmarked multimodal biomedical tasks.

## 1 Introduction

The utility and demand for multimodal machine learning approaches has sharply risen due to their potential to derive holistic representations in various systems, including physics [], chemistry [], neuroscience [], or biology []. Multimodal models in the vision & language domains leverage the same data distributions, which are represented across different modalities [, , ], such as vision-text pairs of the same concepts. However, in many biomedical domains, modalities represent data at different scales (e.g., cellular, genomic, transcriptomic, etc.), cardinalities that are not paired (e.g., many single-cell reads for a single tissue slide per patient), and follow separate distributions. While large foundation models have excelled in tasks confined to individual modalities [, , ], training these models across modalities is expensive, requires paired modalities, and is still an end-to-end process.

Multimodal *fusion* methods [, , , ] attempt to derive a common representation from different data structures and distributions whilst preserving its salient signals []. However, there are several shortcomings of existing fusion methods that relate to their utility, scalability and underlying data assumptions. First, many fusion methods require end-to-end training of the multimodal model, and even in scenarios with existing unimodal models, the fusion operation still has to be trained for a downstream task, typically in a supervised manner. This leads to redundant computational overhead and an inability to extend the model with additional modalities after it has been trained. Second, many commonly used fusion methods either scale quadratically (with the number of modalities) [, ] or are only designed to operate with two modalities [, ]. Third, many of these methods follow a monolithic design, requiring all modalities to be available for every sample during training. This means that they are not robust to missing modalities, modality imbalance, or non-overlapping training sets, which is a very common challenge in a variety of real-world settings []. Finally, many end-to-end fusion architectures are highly topology-specific, making them difficult to extend to other domains.

Some of these challenges can be addressed through *model merging* [] (also referred to as *knowledge fusion* []), an approach commonly used in the context of multi-task settings and language modelling, which capitalises on combining well-performing unimodal models trained in isolation. Model merging methods attempt to combine two architecturally identical models trained on different distributions through interpolation, arithmetic manipulation, and aggregation of their weights [, , ], or stacking their layers [], often without additional training/fine-tuning. While model merging has been extended to some multimodal vision and language tasks [], its crucial challenges in a multimodal setting are that: a) the merged components are still trained in isolation, and b) we cannot assume topological equivalence between two models for separate modalities due to their separate input shapes.

In this paper, we present Multimodal Lego (*MM-Lego*) – a flexible framework for combining any unimodal models into a multimodal model with no or minimal fine-tuning (Figure 1). We introduce two approaches within our framework – *LegoFuse* and *LegoMerge*, enabling performant multimodal models given a set of unimodal encoders with either little (*LegoFuse*) or no (*LegoMerge*) fine-tuning. We show that MM-Lego satisfies multiple desirable properties in a range of real-world multimodal applications combining imaging, tabular, and time series modalities. We demonstrate the utility of MM-Lego on seven medical datasets across three separate downstream tasks, showing that it is:

- 1.
Performant without end-to-end training: *LegoMerge* is highly computationally efficient and achieves competitive performance wrt. state-of-the-art *without any fine-tuning*, while *LegoFuse* exceeds the state-of-the-art in some tasks with only a few epochs of fine-tuning.
- 2.
Scalable: Both variants of MM-Lego scale linearly with the number of modalities whilst outperforming methods with quadratic time complexity.
- 3.
Topology agnostic: Unlike most model merging approaches, *LegoMerge* does not require equivalent architectures between the merged models, allowing users to take advantage of the plethora of open-source models for multimodal learning.
- 4.
Handling modality imbalance & non-overlapping sets: MM-Lego is robust in cases of missing modalities and strong modality imbalance, a common problem in medical domains. MM-Lego can be used even if each modality was trained on unpaired (non-overlapping) training samples.

## 2 Background & Related Work

Preliminaries.
Let $\mathbf{X}^{(\mathcal{M})}=\bigcup_{m\in\mathcal{M}}{m}$ be a multimodal dataset where $\mathcal{M}=\{A,B,\dots,Z\}$ represents the set of modalities $m$ such as images ($A$), tabular data ($B$), time series ($C$), etc. Let $\mathbf{X}_{i,j,k}^{(A)}$ correspond to the element in the dataset for modality $A$ at sample $i$, column $j$, and channel $k$, assuming ${A}\in{}^{I\times J\times K}$ where $1\leq i\leq N$, $1\leq j\leq J$, $1\leq k\leq K$. Each sample in $\mathbf{X}$ has a set of discriminative task labels $\mathbf{y}^{(\mathcal{T})}=\bigcup_{t\in\mathcal{T}}\mathbf{y}^{(t)}$, where $\mathcal{T}=\{T_{1},T_{2}\dots,T_{c}\}$ is the set of possible tasks such that $\mathbf{y}^{(T_{1})}=\{y_{1}^{T_{1}},y_{2}^{T_{1}},...,y_{N}^{T_{1}}\}$ are the scalar target values for task $T_{1}$ for $N$ samples.

Fusion methods.
Given multiple data inputs or latent representations, fusion methods construct single learning representations that can be used for a downstream task, often whilst reducing dimensionality. Many fusion methods [, ] first learn a set of modality-specific encoders $\mathcal{G}=\{g_{m}:{m}\rightarrow\mathbf{h}^{(m)}\}$ assuming a single task label $\mathbf{y}$. This results in a set of latent representations $\mathcal{H}=\{\mathbf{h}^{(m)},m\in\mathcal{M}\}$, which are combined with a fusion operator to obtain the final fused representation $\mathbf{z}=\psi(\mathcal{H})$ and its final prediction $\hat{\mathbf{y}}=f(\mathbf{z})$. Following this problem setup, fusion methods can be differentiated by: 1) the choice of the *fusion operator* $\psi(\cdot)$; 2) the *fusion stage* in the pipeline of when $\psi(\cdot)$ is applied; and 3) the *fusion order* in which the fusion operations are applied (i.e., sequential vs. parallel). The *fusion operator* can be either static (e.g., concatenation [], Kronecker product []) or learnable (e.g., low-rank tensor fusion [], cross-attention mechanisms [, , ], mixture of experts [, ]). The *fusion stage* is typically characterised as early, intermediate or late fusion. Early fusion methods apply a static fusion operator $\psi(\cdot)$ to the raw data while only applying this after passing each modality through $\mathcal{G}$. Intermediate fusion methods often don’t apply a static aggregation but rather learn a fusion function (i.e., a small sub-network or neural layer) in the latent space as part of its end-to-end training [].

A shortcoming of existing fusion methods for many real-world applications is the *fusion order* – most fusion methods use a monolithic task setup, where all modalities are required during training and inference to calculate the set of latent representations $\mathcal{H}$. This often leads to noisy fused representations in cases when a modality is (partially or fully) missing, as the missing tensor requires imputation. Moreover, $\psi(\{g_{m}({m},\mathbf{y}),m\in\mathcal{M}\})$ is typically trained end-to-end. This hinders the potential of extending the multimodal model with additional modalities (beyond the ones it has been trained on), without training anew. Rerunning the complete training pipeline just to add one (or more) additional modality can be infeasible or can lead to redundant computational overheads. Finally, many methods are highly domain-specific and are either not designed for $|\mathcal{M}|>2$ or scale quadratically with the number of modalities [, , ].

Model merging.
The core idea behind model merging, typically deployed in multi-task settings, is that earlier layers in a network may learn similar features that may be used across tasks []. Using linear interpolation [] or arithmetic manipulation [] of the task-specific weights, model merging approaches have shown that they can effectively generalise to new tasks without any fine-tuning. Formally, given multiple tasks $\mathbf{y}^{(\mathcal{T})}$ for the same modality ${A}$, they first learn the set of task-specific functions $\mathcal{F}=\{f_{t}(\omega_{t}):{A}\rightarrow\hat{\mathbf{y}}^{(t)}\mid t\in
\mathcal{T}\}$ where $\omega$ denotes the corresponding model parameters. Assuming the same architecture for each model in $\mathcal{F}$, parameters from a pre-trained base model $\omega_{base}$ can be used to derive task vectors as $\mathcal{V}=\{\tau_{t}\leftarrow\omega_{t}-\omega_{base}\mid t\in\mathcal{T}\}$ [] . Given these task vectors, a multi-task model can be constructed by updating the weights of the base model $\omega^{\prime}=\omega_{base}+\lambda\sum_{t}^{\mathcal{T}}\tau_{t}$. This idea has been extended by the TIES [] and DARE []
that merge models through sparsifying and resolving sign conflicts in the task vectors. Another popular approach is spherical linear interpolation (SLERP), a method used to smoothly interpolate between two vectors while respecting the geometric properties of the vector space []. More specifically, given model parameters $\omega_{T_{1}}$ and $\omega_{T_{2}}\in\mathbb{R}^{d}$, derived from two models with identical architectures, the merged multi-task model parameters can be calculated as $\omega^{\prime}=\omega_{T_{1}}\frac{sin(\theta\cdot(1-\mu))}{sin(\theta)}+
\omega_{T_{2}}\frac{sin(\theta\cdot\mu)}{sin(\theta)}$ where $\theta$ is the radian between the two vectors $\omega_{T_{1}}$ and $\omega_{T_{2}}$ []. The underlying assumption of the above model merging approaches is that the models should have an equivalent network topology, ensuring that the dimensions $\mathbb{R}^{d}$ match up between tasks. However, while this is an acceptable constraint for multi-task learning, it is infeasible for multimodal models where modality shapes and the corresponding network topologies vary greatly.

## 3 Multimodal Lego

Figure: Figure 2: Frequency-domain state passing in *LegoBlock*.
Refer to caption: extracted/5630120/img/legoblock.png

MM-Lego introduces two novel approaches for multimodal model merging (*LegoMerge*) and multimodal model fusion (*LegoFuse*). It addresses a number of limitations of existing model merging (topological equivalence) and fusion methods (such as scalability for $|M|>2$, missing modality handling, end-to-end training, paired data etc.). The core component of MM-Lego is a *LegoBlock* - a wrapper for any modality-specific encoder that imposes several constraints on the latent feature space and structure. These constraints are a necessary condition for *LegoMerge*, which aggregates the latent encodings with minimal signal interference between modalities to perform a multimodal model merge. Moreover, via *LegoFuse*, it further allows us to fine-tune the combined blocks, ensuring cross-modal dependencies and mutual context can be learned.

Architecture.
Rather than learning a single fusion operator $\psi(\mathcal{H})$ that applies to all latent representations at once, we learn a set of latent update functions for each modality, in the form of

$$ $\mathcal{B}=\{\psi_{m}:(g_{m}(X^{(m)}),L_{s}^{(m)})\rightarrow L_{s+1}^{(m)} \mid s\in S,m\in\mathcal{M}\},$ (1) $$

where $L_{t}^{(m)}\in\mathcal{L}$ is our target latent representation for each modality that we will later use in the merge and fusion, and $S$ is the number of update steps. Using the iterative update architecture with latent state passing in Equation 1 has a number of advantages. First, iterative attention architectures have been shown to be highly generalisable across modalities [], and effective at dealing with missing individual modalities [, ]. Second, since all modalities are encoded into a self-defined latent representation, we can impose a dimensionality constraint such that each latent has the same dimensions (e.g., $L^{(A)},L^{(B)},L^{(C)}\in\mathbf{R}^{c\times d}$ for latent channels and dimensions $c$ and $d$).
Third, we can do latent state passing between elements in $\mathcal{B}$, which allows us to “stack” the update functions on top of each other (hence the name) to sequentially encode each modality’s signal into the same latent representation.

LegoBlocks. Each element in $\mathcal{B}$ represents a *LegoBlock* (Figure 2), which learns the latent update function $\psi_{m}$ for any given encoder $g_{m}$. Acknowledging that different data modalities and structures require different inductive biases to effectively encode each modality’s information ($g_{m}$), *LegoBlock* acts as a wrapper to accurately encode $h_{m}$ into $L^{(m)}$. The benefit of training each modality update function separately instead of end-to-end is that we can train on entirely separate samples for the same tasks. For example, in many medical domains, we may have single-cell data for one subset of patients and bulk sequencing data for a different subset, while having the same task labels for the entire set. To address this, we use a latent representation $L$ that effectively encode signal across modalities, and are robust or invariant to transformations (shifts, rotations, etc.), noise and signal interference. Alleviating signal interference is particularly important for model merging, as it is undesirable to apply an aggregate function on the learned modality latents $\mathcal{L}$ that can cancel out each other’s signal. Beyond preventing signal interference, Fourier features have also been shown to be effective as mixing mechanisms [], positional encodings [], and to be naturally emerging in invariant networks [].

This motivated us to design MM-Lego for learning latent representations in the frequency domain, taking advantage of a number of desirable properties for multimodal merging and fusion. In particular, frequency-domain representations are 1) *signal-preserving* as frequency features are less prone to signal interference upon aggregation (see Appendix E); 2) *distance-preserving*, as the Euclidean distance between two signals remains unchanged after the Fourier Transform (following from Parseval’s Theorem []), making them suitable for distance-based loss functions; 3) *invertible* as the spatial/temporal domain can be reconstructed, allowing for the iterative updates outlined in Equation 1; and 4) *efficient*, as the Fast Fourier Transform (FFT) has a time complexity of $\mathcal{O}$($n$$log(n)$), making it scalable to very large datasets  [].

Figure 2 depicts a single update of *LegoBlock* that operates over frequency-domain latent of modality $X^{(A)}$. Starting with the latent representation in the spatial domain, we first apply a discrete FFT $\mathcal{F}(\cdot)$ [] along each dimension of the 2D Tensor to yield a frequency domain representation:

$$ $L_{t}^{\mathcal{F}}(u,v)=\sum_{i=0}^{c-1}\sum_{j=0}^{d-1}L_{t}(i,j)e^{-2\pi i( \frac{ux}{c}+\frac{vy}{d})},$ (2) $$

where $i,j$ denote the spatial-domain indices, and $u,v$ denote the frequency-domain indices. This results in a complex frequency-domain representation from which we separate the real (symmetrical) and imaginary (asymmetrical) components of the FFT ($(L_{t}^{\mathcal{F}})^{r}$ and $(L_{t}^{\mathcal{F}})^{i}$) []. We update the real component using a standard cross-attention layer [], where we aim to learn the weight matrices $W_{m}^{q}$ for the update query $(L_{t}^{\mathcal{F}})^{r}$, and $W_{m}^{k}$, $W_{m}^{v}$ for the keys and values ($h^{(A)}$) resulting in the latent update:

$$ $(L_{t+1}^{\mathcal{F}})^{r}=\text{softmax}\left(\frac{(L_{t}^{\mathcal{F}})^{r }W_{m}^{q}\cdot(h^{(A)}W_{m}^{k})^{\top}}{\sqrt{d_{k}}}\right)\cdot(h^{(A)}W_{ m}^{v}).$ (3) $$

In contrast to other Fourier-based architectures [], which only use the real component of the transform, we keep track of the imaginary component $(L_{t}^{\mathcal{F}})^{i}$ as well. This allows us to reconstruct the complex representation, and subsequently apply the inverse transform. We found this to be critical for our iterative architecture, as otherwise the signal gets distorted and we lose phase information (encoded in the imaginary component) at each update pass. Once we reconstruct the complex representation, we apply the inverse transform to recover the spatial representation in preparation of the next pass $L_{t+1}=\mathcal{F}^{-1}((L_{t+1}^{\mathcal{F}})^{r}+i(L_{t}^{\mathcal{F}})^{i})$. Finally, the last task-specific heads of each block are a fully-connected layer after applying layer normalisation. We omit the inverse transform after the last update such that each head is trained in the frequency domain. This ensures that we can apply aggregations with low signal interference on $\mathcal{L}$ during *LegoMerge*.

LegoMerge.
With the architectural assumptions imposed on each modality encoder in $\mathcal{G}$ through *LegoBlocks* $\mathcal{B}$, we can apply model merging techniques in a multimodal setting. With $\mathcal{L}\subseteq\mathbb{R}^{c\times d}$ and each element in $\mathcal{L}$ being in the frequency domain, we can use aggregation functions $\psi(\cdot)$, which are less prone to cancelling out signal. For example, let $L^{(A)}$ and $L^{(B)}$ be the final frequency domain latent representations for modalities $A$ and $B$, then we can calculate a merged multimodal representation as:

$$ $\psi(L^{(A)},L^{(B)})=(\frac{2|L^{(A)}|\cdot|L^{(B)}|}{|L^{(B)}|+|L^{(A)}|}) \cdot e^{i\cdot\frac{\angle L^{(A)}+\angle L^{(B)}}{2}},$ (4) $$

where the real component is the harmonic mean of the magnitudes ($|\cdot|$), and the imaginary component is the arithmetic mean of the phases ($\angle$) of $L^{(A)}$ and $L^{(B)}$. We take the harmonic mean since it is less prone to outliers [], that is, the merged representation is less likely to be strongly skewed towards either modality by very large frequency components. With the cross-modal combined representation $L^{(\mathcal{M})}$, we need to combine the task heads of each block, where we apply spherical linear interpolation (SLERP) [] for the set of task heads $\mathcal{Y}$ from each element in $\mathcal{B}$. Assuming two task heads with weight vectors $\mathbf{w}_{y1}$ and $\mathbf{w}_{y2}$, where $\mathbf{w}_{y1}\subset\omega_{A},\mathbf{w}_{y2}\subset\omega_{B}$, we calculate the merged weights as $\mathbf{w}_{y1y2}=\mathbf{w}_{y1}\cdot\frac{sin(\theta\cdot(1-\mu)}{sin(\theta
)}+\mathbf{w}_{y2}\cdot\frac{sin(\theta\cdot\mu)}{sin(\theta)}$ [], where $\theta$ is the angle (in radians) between both vectors and $\mu$ is a binary variable indicating whether $\mathbf{w}_{y1}$ or $\mathbf{w}_{y2}$ is used.

LegoFuse.
*LegoMerge* is designed to construct a performant multimodal model without any additional training. Nevertheless, its key limitation is that each element in $\mathcal{B}$ is trained in isolation. To allow for modalities to mutually contextualise each other, a limited amount of fine-tuning is beneficial. To avoid fine-tuning a potentially noised signal emerging from the merged latent $L^{(\mathcal{M})}$, rather than directly fine-tuning the merged model (at the parameter-level), *LegoFuse* operates at the layer level by sequentially passing through all layers in $\mathcal{B}$. Specifically, the shape consistency introduced by $\mathcal{L}\subseteq\mathbb{R}^{c\times d}$ allows the stacked model to pass the Fourier-transformed latent states either between blocks (stacking) or different layers between blocks (weaving), as illustrated in Figure 1. We then fine-tune the stacked or the weaved model for a few epochs with all (paired) modalities, such that the state updates are conditioned on all modalities’ updates. This, in turn, becomes the query for the cross-attention layer. Note that both the stacked and weaved variants of *LegoFuse* allow for fine-tuning all model parameters, including the ones of the initial modality-specific encoders.

## 4 Experimental Setup

Datasets. We evaluate MM-Lego (*LegoMerge* and *LegoFuse*) and its components (*LegoBlock*) on seven multimodal medical datasets covering three separate modalities (images, tabular, time series) from three separate sources: histopathology (The Cancer Genome Atlas (TCGA)) [], intensive care data (Medical Information Mart for Intensive Care (MIMIC)) [], and skin imaging (International Skin Imaging Collaboration (ISIC)) []. The seven tasks shown in our results correspond to survival analysis tasks on four TCGA sites (BLCA, BRCA, KIRP, UCEC), classification tasks on two variants of MIMIC (disease classification (ICD9) and patient mortality (MORT), and predicting melanoma for the ISIC patients. Full details on the datasets and their pre-processing steps can be found in Appendix C, and details on task losses and evaluation metrics are in Appendix D.

Baselines. For all experiments, we compare *LegoMerge* and *LegoFuse* to several uni- and multimodal baselines to evaluate their performance. For all tabular modalities, we use a self-normalising network [] due to its performance and regularisation mechanisms suitable for high-dimensional tabular data. For the image and time series modalities, we use an attention-based Multiple Instance Learning (AMIL) []. Across all modalities, we benchmark two related iterative-learning architectures: MultiModN [] and Perceiver [] which generally shows strong performance across a wide range of unimodal tasks. In terms of specific multimodal baselines, we use two late fusion combinations of SNN+AMIL, namely concatenation of their final latent representation and bi-linear fusion []. For the Perceiver, we use the same multimodal setup as suggested in the original paper, i.e., concatenation of modalities before passing them into the model. We use two additional domain-specific multimodal baselines: the Hybrid Early-Fusion Attention Learning Network (HEALNet) [] which is using an end-to-end trained iterative cross-attention architecture and the Multimodal Co-Attention Transformer (MCAT) [] which is using the tabular (1D) modality as context for the imaging (2D) modality.

Implementation Details. For each experiment and dataset, we perform a 5-fold repeated random sub-sampling with a 70-15-15 train-test-validation split. We re-ran all of the baseline models in this paper using their open-source code to ensure that no performance differences are caused by different task setups, losses, or training splits. We ran a brief Bayesian Hyperparameter search [] for key parameters of each model (learning rate, decay, schedulers, dropout, layer dimensions). The experiments were run on a single Nvidia A100 80GB GPU on a Ubuntu 22.04 virtual machine. The MM-Lego code and PyTorch module is available at [https://github.com/konst-int-i/mm-lego](https://github.com/konst-int-i/mm-lego).

## 5 Results

Table: Table 1: Mean and standard deviation of unimodal and multimodal task performance, showing the concordance Index (survival analysis tasks) and AUC (classification tasks) on 5 random sub-sampling folds with the best and second-best models highlighted. *LegoMerge* achieves competitive performance with multimodal baselines across all tasks without end-to-end training or any fine-tuning. *LegoFuse* achieves top-2 performance for all datasets and has the highest performance amongst all benchmarked models in 5 out of 7 datasets. CC and BL denote the monolithic fusion operators $\psi(\cdot)$ of concatenation and bilinear fusion respectively.

The results across the three prediction tasks (survival analysis, multi-class, and binary classification) are summarised in Table 1, showing the mean and standard deviation of the task-relevant performance metric across the 5 random sub-sampling folds. We compare our baselines to: 1) *LegoMerge*, which is taking two *LegoBlocks*, trained unimodally, and merges them without fine-tuning, and 2) *LegoFuse*, which is taking the same blocks and fine-tuning them for two epochs. Note that, we did not see a significant performance difference between the “stack” and “weave” variants of *LegoFuse*, therefore the reported results in Table 1 correspond to *stacked* blocks. Across all datasets, *LegoFuse* is within the top 2 performers of all uni- and multimodal datasets with only two epochs of fine-tuning.
Both *LegoFuse* and *LegoMerge* generalise well across domains (pathology, clinical care, dermatology), which some unimodal and multimodal baselines struggle with and heavily overfit (AUC of 0.5).
Despite never seeing a single multimodal training step, *LegoMerge* achieves strong results, closely matching the performance of the best baselines in 6 out of 7 tasks.
We note that the performance stability across folds is in line with or better than the multimodal baselines for all datasets except TCGA-KIRP, where we have significantly larger standard deviations. We believe that this instability is caused by the relatively low sample size of this dataset, and posit that this could be alleviated by pre-training *LegoBlocks* on other TCGA datasets before fine-tuning them on KIRP. The SIIM-ISIC dataset exhibits a case of multimodal collapse (also referred to as dominance), where all multimodal models struggle to effectively take advantage of both modalities.

We assess the gains afforded by *LegoMerge* in Figure 3, where we use *LegoMerge* to merge *LegoBlock* for tabular data with either *LegoBlock* for imaging or *LegoBlock* for times series data (depending on the dataset). We compare the performance of the unimodal *LegoBlocks* with the resulting multimodal *LegoMerge* and find a performance uplift gained from the merge.
Namely, the harmonic mean (Equation 4) of each block’s latent states coupled with spherical linear interpolation of the weights and biases in the task heads leads to a better performance than for either unimodal block in 6 out of 7 datasets. Again, the outlier to this trend is the dataset with the lowest sample size (KIRP) where LegoMerge generally struggled for stability. As outlined in Figure 1, the blocks can either be trained as models from scratch or can be used as a wrapper for a unimodal encoder. This is demonstrated in Figure 4, which shows the performance of the SNN and AMIL, compared to three multimodal combinations of the two: 1) a naive ensemble that is taking the average logits of each encoder, 2) *LegoMerge* applied after wrapping each encoder in *LegoBlock*, and 3) *LegoFuse* with limited fine-tuning. For both MIMIC tasks, we can see that *LegoMerge* beats the ensemble by 7.1% and 7.3% on the MIMIC disease classification (ICD9) and mortality prediction (MORT) respectively. Moreover, *LegoFuse* improves the performance even further over all other models by a further 1-3%.

Figure: Figure 3: Mean task performance (concordance Index/AUC) of LegoBlock (Tabular), LegoBlock (Image/Time Series) and LegoMerge, showing the increase in task performance by applying a multimodal model merge *without any fine-tuning*. Our proposed multimodal model merge shows a positive performance improvement on 6 out of 7 datasets.
Refer to caption: extracted/5630120/img/legomerge_uplift.png

Figure: Figure 4: AUC performance on the MIMIC dataset when merging existing encoders (SNN for tabular, AMIL for Time Series) using LegoMerge and LegoFuse. Our multimodal model merge shows much better performance than using an ensemble, exhibiting the performance gains, at no additional costs, through the merge even prior to fine-tuning in LegoFuse.
Refer to caption: extracted/5630120/img/merge_uplift.png

## 6 Discussion

In this work, we introduce two novel approaches for both multimodal model merging (*LegoMerge*) and fusion (*LegoFuse*) to address some common limitations of existing methods for multimodal modelling. We outline five desirable criteria of multimodal models for biomedical data in Tabel 2 to highlight scenarios in which we believe our introduced methods to be highly beneficial.

Performance without end-to-end training. With the increasing volume, complexity and diversity of collected biomedical data, (re)training multimodal models from scratch becomes more expensive, unsustainable, and even infeasible in the long run. Similar to trends seen in large language models, providing access to fine-tuning and merging frameworks can help to make established state-of-the-art unimodal models more accessible to tailor for highly specific applications, as it is commonly a requirement in medicine. The results in Table 1 show that we can achieve state-of-the-art performance by combining existing, pre-trained unimodal models and fine-tuning them accordingly: *LegoFuse* outperforms end-to-end trained baselines with as little as 2 epochs of fine-tuning. We anticipate that *LegoMerge*, where no multimodal supervised data is required, and *LegoFuse*, where only a limited number of supervised paired samples are required, can aid in fields where paired observations are scarce. These scenarios are frequent and very realistic, from studies on rare diseases to data gathered from clinics that do not have access to sophisticated data acquisition technologies. In such cases, training large encoders wrapped in *LegoBlocks* on similar domains, and then fine-tuning them on a small amount of supervised paired samples may present a feasible path for domain adaptation for multimodal models. A further advantage of our modular approach is that additional data (from either the same or novel modalities) can be added at different points in time. For example, if a large bimodal model was already trained and a new modality becomes available later on, MM-Lego can be readily applied to extend such model without training a new model, saving time, costs and energy.

Missing modalities and modality imbalance. The results in Section 5 support our hypothesis that we can build performant multimodal models without having perfectly paired training data. This is evident by the highly competitive performance of *LegoMerge* in comparison with existing multimodal baselines (Table 1. Paired data requirements can be problematic when modalities are missing for some of the samples, leading to a data-performance trade-off: one can either attempt to use all the available data and impute the missing values, which introduces noise, or can take the intersection of samples with available data modalities, which may dramatically reduce the sample size required for training.

MM-Lego overcomes this data-performance trade-off as it can be effectively trained on non-overlapping training sets. This is further supported by our findings (see Appendix F), where we found that the performance of MM-Lego remains stable even when trained with completely non-overlapping (symmetrically different) sets of samples. In contrast, such a training scenario is infeasible for any other current end-to-end model setup. Moreover, MM-Lego can effectively handle *missing modalities*. During training, value imputation is not an issue for MM-Lego, since we can train *LegoBlocks* independently, and subsequently combine them into a performant model, as shown in Figure 4. During inference, we can easily skip missing modalities, which is another benefit of making each block compatible with an iterative attention architecture (Equation 1). That is, while a monolithic fusion operator $\psi(\cdot)$ requires the entire set of unimodal encodings $\mathcal{H}$ to be present, MM-Lego’s architecture can just query the elements in $\mathcal{B}$ for the available modalities. Finally, MM-Lego’s modular design allows for handling high modality imbalance and one-to-many cardinalities ($|X^{(A)}|\neq|X^{(A)}|$). An example of such a case would be a dataset where $A$ are histopathology slides (one per patient) and $B$ is single-cell data associated (many per patient).

**Table 2: Comparison of desirable requirements of multimodal systems for clinical practice. *LegoMerge* and *LegoFuse* present two alternatives to existing, end-to-end trained, multimodal fusion approaches (early, intermediate, and late fusion) and multi-task merge methods applied to multimodal settings. ✓: meets requirement, (✓): some approaches meet requirement, ✗: fails requirement.**
| Criteria/Method | Late | Intermediate | Early | Multi-task merge | LegoMerge | LegoFuse |
| --- | --- | --- | --- | --- | --- | --- |
| Performant without end-to-end training | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ |
| Learns cross-modal interactions | ✗ | ✓ | (✓) | ✗ | ✗ | ✓ |
| Architecture agnostic | ✓ | (✓) | ✓ | ✗ | ✓ | ✓ |
| Handles strong modality imbalance | ✗ | (✓) | ✗ | ✓ | ✓ | ✓ |
| Add modalities without re-training | ✗ | ✗ | ✗ | ✗ | ✓ | (✓) |

Architecture agnostic. A key limitation of existing model merging literature in multi-task learning is the assumption that the majority of the network topology between tasks is equivalent. While this is a feasible assumption for merging in multi-task learning, the data heterogeneity limits its application in multimodal settings. Therefore, the design of *LegoBlock* is sufficiently permissive to use any unimodal encoder as part of this framework, while complying with the necessary architectural assumptions required for model merging. Our results in Table 1 and Figure 4 support this by showing that any unimodal encoder (SNNs and AMIL in this example) can be wrapped in a *LegoBlock* without any practical loss in performance whilst making them capable for further merging and/or fusion. To the best of our knowledge, MM-Lego is the first general-purpose model merging framework for multimodal data outside of the vision & language domain.

Low computational requirements. Scaling to more than two modalities is increasingly important for multimodal models as more modalities are captured at scale: this was one of the main motivations for the modular design of MM-Lego.
Many intermediate fusion approaches [, , , ] are natively designed for two modalities and centred around a cross-attention layer between the modalities with a time complexity of $\mathcal{O}(N^{2})$.
Scaling to more than two modalities requires calculating the modality-guided cross-attention for all unique pairwise combinations $\binom{M}{2}=\frac{M(M-1)}{2}$ which is $\mathcal{O}(M^{2})$, leading to a total upper bound time complexity of $\mathcal{O}(M^{2}N^{2})$.
MM-Lego improves on this wrt. both $M$ and $N$. The time complexity of *LegoBlock* is bound by the cross-attention unit, which reduces the quadratic time complexity by using the latent $L\in R^{c\times d}$ as the query to $\mathcal{O}(dN)$ for latent dimensions $d$. The sequential design in Equation 1 ensures linear scaling wrt. the number of modalities, leading to a final time complexity of $\mathcal{O}(dMN)$. Beyond time complexity, another key benefit of MM-Lego is the number of training steps required at the given complexity. For the end-to-end baselines, we typically observe loss convergence around $\sim\!10$ training epochs, while *LegoMerge* and *LegoFuse*, by design, require zero or as little as two training epochs, respectively.

Limitations. We believe that our MM-Lego approach would benefit from further research of parameter-efficient ways to design the fusion layer within each *LegoBlock* (Equation 3), that is, efficiently encoding $h^{(A)}$ into the updated latent representation $(L_{t+1}^{\mathcal{F}})^{r}$. While this is suitable in both our solutions and has proven to be effective in other work [, , ], finding a more parameter-efficient solution is desirable. Moreover, while in this paper we limit our focus to multimodal problems from biomedical domains, MM-Lego is designed to be general-purpose and applicable to any multimodal tasks. Therefore, we leave it to future work to further demonstrate these properties for more (and other) data modalities and domains, including vision & text tasks.

Conclusion. We present MM-Lego, a general-purpose and modular learning framework to build performant multimodal models with minimal fine-tuning. To achieve this, we introduce three novelties. First, we introduce a wrapper for unimodal encoders (*LegoBlock*) that a) enforces shape consistency between modality representations, and b) harmonises the latent representations in the frequency-domain to enable model merging with little signal interference. Second, we introduce the first multimodal model merge framework (*LegoMerge*) that goes beyond vision & language modalities and outperforms many unimodal baselines without seeing a single multimodal supervised training sample. Third, we show that these building blocks can be combined to construct a fusion model that achieves state-of-the-art performance with only a few epochs of fine-tuning (*LegoFuse*).

## Acknowledgements

KH acknowledges support from the Gates
Cambridge Trust via the Gates Cambridge Scholarship.

## Broader Impact

We present MM-Lego, a novel and flexible general-purpose framework for constructing performant multimodal models from complex heterogeneous biomedical data with no/minimal training. As learning holistic computational representations requires the ability to process information from different distributions and modalities within the same model, the demand for multimodal machine learning models has significantly soared in various domains. The application of many currently available multimodal fusion and alignment approaches is too restrictive in a biomedical setting, since most of them require end-to-end training, are highly topology-specific and/or scale quadratically (with the number of modalities), and often cannot handle cases of high modality imbalance in data. To address these shortcomings, we introduce two approaches within our MM-Lego framework: *LegoFuse* and *LegoMerge*, enabling performant multimodal models given a set of pre-trained unimodal encoders with either little (*LegoFuse*) or no (*LegoMerge*) fine-tuning. We show that our framework can lead to cost-effective models that exhibit state-of-the-art performance, are easily scalable to many modalities and are robust to data imbalance and noise.

The primary aim and impact of our work is advancing data analysis capabilities in critical domains such as medicine and biology, focusing on complex tasks that require simultaneous modelling of multimodal heterogeneous data. The ability of our framework to readily utilise and combine well-performing pre-trained unimodal models, leading to performant multimodal models, can alleviate a variety of issues pertaining to computational overheads, data storage and security. With the increasing volume, complexity and diversity of collected biomedical data, (re)training multimodal models from scratch becomes increasingly expensive, unsustainable, and often infeasible. With MM-Lego, we can achieve state-of-the-art performance with no (via *LegoMerge*) or minimal fine-tuning (via *LegoFuse*). This significantly decreases the need for lengthy training procedures as well as access to all of the, likely sensitive, training data. Moreover, MM-Lego does not require access to paired data, which further facilitates its application in data scarce scenarios, ranging from applications for rare diseases to applications in institutions that do not have access to sophisticated data acquisition technologies. A further advantage of MM-Lego is that it supports model extensions with additional data (from either the same or novel modalities) at different points in time without training a model anew. All of these benefits can have an immediate and practical impact on many biomedical applications.

To this end, our work has only been evaluated in a strictly research setting. Further applications of our work in scenarios with sensitive data, such as clinical practice, introduce some challenges. As our primary focus is biomedical applications, data privacy must be carefully managed. Furthermore, as the models constructed with MM-Lego are intended to serve as decision-aiding tools, they bear risks of decision bias. Mitigating such risks involves further extensive evaluations, clinical trials, and medical regulation to ensure its reliability and safety before wider adoption in clinical settings.

## Appendix A Notation

Objects.

- •
$X^{(A)}$: matrix corresponding to modality A
- •
$\mathbf{x}^{(A)}$: a vector in $X^{(A)}$ (e.g., a sample of modality A)
- •
$\mathbf{X}_{i,j,k}^{(A)}$: elements of matrix $X^{(A)}$ at row $i$, column $j$, channel $k$, assuming $X^{(A)}\in{}^{I\times J\times K}$ where $1\leq i\leq I$, $1\leq j\leq J$, $1\leq k\leq K$
- •
$\mathbf{X}^{(\mathcal{M})}=\bigcup_{m\in\mathcal{M}}X^{(m)}$: multimodal dataset
- •
$\mathbf{y}\in\mathcal{Y}=\bigcup_{t\in\mathcal{T}}\mathbf{y}^{(t)}$: set of task labels for all available tasks $\mathcal{T}$
- •
$\mathbf{y}^{(T_{1})}$: task labels for task $T_{1}$

Sets.

- •
$\mathcal{M}$: set of modalities
- •
$\mathcal{T}$: set of tasks
- •
$\mathcal{Y}$: set of task-specific heads
- •
$\mathcal{G}=\{g_{m}:{m}\rightarrow\mathbf{h}^{(m)}\mid m\in\mathcal{M}\}$: set of modality-specific encoders
- •
$\mathcal{H}_{\mathbf{y}}=\{g_{m}({m},\mathbf{y})|m\in\mathcal{M}\}$: set of task- and modality-specific embeddings
- •
$\mathcal{B}=\{\psi_{m}:(g_{m}(X^{(m)}),L_{s}^{(m)})\rightarrow L_{s+1}^{(m)}
\mid s\in S,m\in\mathcal{M}\}$: set of *LegoBlocks*

Functions and Operators.

- •
$g_{m}(\cdot)$: modality-specific encoder
- •
$\psi(\cdot)$: fusion operator (monolithic)
- •
$\psi_{m}(\cdot)$: modality-specific latent update
- •
$\mathcal{F}$: Fourier transform
- •
$\mathcal{F}^{-1}$: Inverse Fourier transform

## Appendix B Full Results

Table: Table 3: Task performance of uni- and multimodal models across 7 medical datasets – for each task target metric, we highlight the best and second-best models.

## Appendix C Datasets

We evaluate MM-Lego (*LegoMerge* and *LegoFuse*) and its components (*LegoBlock*) on seven multimodal medical datasets covering three separate modalities (images, tabular, time series) from three separate sources: histopathology (The Cancer Genome Atlas (TCGA)) [], intensive care data (Medical Information Mart for Intensive Care (MIMIC)) [], and skin imaging (Society for Imaging Informatics in Medicine & International Skin Imaging Collaboration (SIIM-ISIC)) [].

TCGA: Some of the results shown in this paper here are based upon data generated by the TCGA Research Network: [https://www.cancer.gov/tcga](https://www.cancer.gov/tcga). The Cancer Genome Atlas (TCGA) is an open-source genomics program run by the United State National Cancer Institute (NCI) and National Human Genome Research Institute, containing a total of 2.5 petabyts of genomic, epigenomic, transcriptomic, and proteomic data. We predict survival of right-censored patients based on the high-resolution histopathology slides ($\sim\!80,000\times 80,000$ pixels) and multi-omic data (gene expressions, copy number variations and gene mutations) captured from bulk sequencing in a tabular format. We train on four separate cancer cohorts with multimodal data available: Urorethelial Bladder Carcinoma (BLCA, $n=436$), Breast Invasive Carcinoma (BRCA, $n=1021$), Kidney Renal Papillary Cell Carcinoma (KIRP, $n=284$), and Uterine Corpus Endometrical Carcinoma (UCEC, $n=538$).

MIMIC-III: We train models on two separate tasks: patient mortality (multi-class classification) and disease classification (ICD-9 codes), which we formulate as a binary classification task. We use both clinical variables and small time series data on various vital signs measured at 24 time steps. Both tasks have $n=32616$ and the same feature set for different task labels.

SIIM-ISIC: Stems from the Society for Imaging Informatics in Medicine & International Skin Imaging Collaboration (SIIM-ISIC) melanoma classification Kaggle challenge [], which contains both tabular data and images of skin leisures to be classified for melanoma patients. To account for class imbalance, we randomly downsampled the majority class to a 5:1 ratio for the class of interest (melanoma) to a sample size of $n=2875$. All images were patched and encoded using the resnet50-kather100k for TCGA (ResNet pre-trained on a large histopathology patch collection) [] and a regular ImageNet v2 pre-trained ResNet for the pictures of skin leisures. Both images (patch encodings) and times series were represented as 2D tensors, and the tabular clinical and multi-omic data as 1D tensors to pass into the modality-specific encoders $g(\cdot)$.

## Appendix D Losses and Metrics

The results report the (unseen) test set performance, by evaluating the concordance Index (c-Index) in the case of TCGA, AUC in the case of MIMIC-III-ICD9 and ISIC, and Macro-AUC (“one-vs-rest”) for MIMIC-III-ICD9. As indicated in Figure 1 the output of each task head in $\mathcal{Y}$ are the logits with predictions for each class given the final Fourier-transformed latent state $y_{l}=f(L_{T}^{\mathcal{F}})$. Since TCGA is a survival prediction task with right-censored data, we have divided the survival period into four non-overlapping bins and use the logits of these bins to calculate the hazard ($y_{h}=\frac{1}{1e^{-y_{l}}}$) and survival ($y_{s}=\prod_{1}^{k}{1-y_{h}}$) respectively for $k$ bins. Given the hazards, censorship, and ground truth bins, we can calculate the negative log-likelihood loss from a proportional hazards model [] which is used as the survival loss. We evaluate the performance using the Concordance Index (c-Index), for which we determine the fraction of paired samples in which the prediction outcomes are concordant with the ground truth. As MIMIC and ISIC relate to classification tasks, we employ categorical cross-entropy loss for training. Note that both AUC and the c-Index have similar interpretations, therefore the values range between $[0.5-1]$.

## Appendix E Signal Interference on Latent Variables

Figure: Figure 5: Example of signal interference on a random normal latent variable and its additive inverse variable with some added noise, showcasing a severe case of signal interference where nearly all signal cancels out. We can see that the fourier-transformed data does not suffer this problem when we apply the harmonic mean. This is a key reason for the choice of model merging architecture.
Refer to caption: extracted/5630120/img/signal_interference.png

Figure: Figure 6: The argument against Fig. 5 would be to use absolute or only positive values. This example shows that this logic can also be flawed. We demonstrate this using a squarewave function with a frequency offset beteen $Mod_{A}$ and $Mod_{B}$ and a scaled amplitude by a normal distribution. We can see that the mean of the regular and the absolute values suffers some signal interference while the FFT aggregation does not.
Refer to caption: extracted/5630120/img/signal_interference_squarewave.png

## Appendix F Training on unpaired data

Figure: Figure 7: Test performance of *LegoMerge* (SNN+AMIL) compared to the SNN-AMIL ensemble when training on different levels of overlapping samples between the modalities. A symmetric difference of 1 means no overlap between the samples, 0 being perfect overlap. We selected N=10,000 MIMIC examples for this experiment.
Refer to caption: extracted/5630120/img/overlap_plot_mimic.png

## Peer Reviews

confidence: 3
contribution: 4
initial_rating: 6
presentation: 3
questions: 1. I suggest to carry out the same type of experiments on CheXpert (https://arxiv.org/abs/1901.07031), where the downstream task can be disease classification. The experiments can be done only on the subset of CheXpert given the limited time during rebuttal.
2. If possible, I'd like to see the performance on datasets with >=3 modalities.
soundness: 3
strengths: 1. The idea of the LegoBlocks is novel and interesting.
2. Both LegoMerge and LegoFuse show promising results.
3. The wrapper nature of LegoBlocks allows its application to wider range of tasks.
summary: The authors study the multimodal learning problem and propose a novel method that can merge encoders from different modalities freely. This is achieved through their newly proposed Lego block, which iteratively applies a discrete FFT to modality-specific dataset X^m and modality-specific latent representation L^m. The authors further propose LegoMerge, which can merge Lego blocks from different modalities without any additional training, and LegoFuse, which merge Lego blocks by stacking all the blocks and fine-tuning.
weaknesses: 1. The experiments do not include the merge between image and textual data, which might be the most common/important scenario. 
2. The authors state one of the advantage of the proposed method is the ability to merge more than 2 modalities, yet all experiments are conducted with only 2 modalities.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Questions and suggestions regarding the work's motivation:

- The authors motivate processing the embeddings in frequency space by citing the advantageous properties of Fourier space with regard to its compactness, robustness against noise and robustness against signal interference. However, several of the referenced works are rather unspecific with regard to these claims and the authors do not provide detailed proof. I suggest that the authors expand the corresponding section and include more detailed arguments for this design choice.

- I believe that the statement that "LegoMerge does not require any finetuning" is slightly misleading. If I understand the method correctly the LegoBlocks are a wrapper around already existing embeddings that have to be trained themselves in order to tune (at least) the weights of the attention mechanism, $W^q_m$, $W^k_m$, and $W^v_m$. Compared to simply merging the embeddings, this does require additional fine-tuning steps. Accordingly, I do not believe that the statement that LegoMerge does not require any training time (cf. Table 3) is accurate. Consequently, I believe the paper should be adjusted to clearly reflect this.


Questions and suggestions regarding the proposed methodology:

- The paper is ambiguous on whether the raw input embeddings $h^{(A)}$ or their Fourier transform $\mathcal{F}(h^{(A)})$ are used in the attention mechanism of the LegoBlocks and the calculation of the harmonic mean during LegoMerge. Equations 3 and 4 indicate the former, while Figure 1 depicts the latter case. Please clarify this and clean up the notation.

- Also, in the right-most panel of Figure 1, why is one of the inputs to the LegoBlocks $L^{(A)}$?

- In the LegoBlocks, how is $L_0$ initialized, and is it updated during tuning? 

- How is the number of update steps $S$ tuned?

- One of the main claims of the work is its robustness against different topologies of embeddings. What happens if the initial data embeddings, $X^{(A)}$, $X^{(B)}$, $X^{(C)}$, have different dimensionalities and consequently require Fourier transforms of different dimensionalities? It is not apparent to me, why embeddings of time series and tabular information should be two-dimensional as suggested by the paper.

- Why do the authors apply an inverse Fourier transform at the end of each LegoBlock only to convert it back into frequency domain for a subsequent block or the final task head? Do any artifacts occur because of this repeated conversion?


Questions and suggestions regarding the conducted experiments:

- Even after reading Appendix C and D, many aspects about the conducted experiments are unclear to me and I do not believe that it is possible for a reader to reimplement them based on the current description. For example, how were the 80,000 $\times$ 80,000 histopathology images of the TCIA dataset encoded? Which mortality was assessed during the experiments with the MIMIC-III dataset? Which model was used to encode the time series data? The authors should aim to provide much more detail with regard to the conducted experiments.

- In light of the considerable effort required to fine-tune each LegoBlock (see earlier comments), I am missing a comparable effort to tune a neural network that processes the concatenated multimodal embeddings. Analogously to the proposed method, one could employ a multi-layer perceptron with S layers and an attention mechanism at each layer. If this approach was adequately tuned, it would function as meaningful, simple and potentially powerful baseline, while simultaenously directly ablating the design choice of converting embeddings to Fourier space.

- The authors should include experiments in which three or more different modalities are included. Currently, none of the conducted experiments supports their main claim that their proposed method scales effectively beyond two modalities.

- While the authors have conducted basic experiments demonstrating their method's ability to handle missing data, the conducted experiments are rather simplistic and only included in the supplementary material. Considering they pertain to a main claim of the work, the authors should expand these experiments and work them into the main paper.

- Have the authors conducted experiments whether the order of fusion during LegoFuse influces its performance?

- What is the added benefit of including both Tables 1 and 4 instead of simply merging them?
soundness: 3
strengths: - The paper researches a very timely topic, the processing and fusion of multimodal medical data. The authors compellingly motivate this in a well written introduction and background section.

- The proposed method of adapting and fusing multimodal embeddings in the frequency domain code appears to be interesting, novel, and have some theoretical grounding.

- The conducted experiments include three public datasets spanning various data domains, as well as several relevant baseline methods.
summary: The presented work introduces multimodal Lego, a novel strategy to fuse latent embeddings from different domains, such as representations learned from images, tabular or time series. The authors hypothesize that processing and fusing embeddings in the frequency domain allows them to be agnostic towards the topology of the representation space and avoid signal interference when merging embeddings. To this end, the authors propose a so-called "LegoBlock" that converts multimodal embeddings via a Fourier transform before iteratively updating them via an attention mechanism. Multiple of these modules can be combined by calculating the harmonic mean between the embeddings ("LegoMerge") or by training a cascade of modules that sequentially update the latent embedding based on information from the various domains ("LegoFuse"). The utility of the method is demonstrated on three medical datasets that each combine two data modalities. LegoMerge is shown to outperform unimodal methods, while LegoFuse outperforms various recent multimodal baselines.
weaknesses: - The authors only provide limited theoretical motivation for their design choice of converting and processing embeddings in frequency space.

- Some parts of the proposed method are not clearly described. In particular, several aspects with regard to the architecture, functioning, training and tuning of the LegoBlocks are ambiguous or not described at all.

- Similarly, the description of the experimental setup is missing key information so that it is probably not possible to reimplement the benchmarking experiments.

- Several of the core claims made by the authors are not backed up by experimental evidence. In particular the method's ability to handle more than two modalities and its computational efficiency are not adequately evaluated.

- Finally, a meaningful ablation study is currently missing.

(All of the these perceived weaknesses are described in much greater detail in form of specific questions and suggestions in the next section.)

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: 1. Could additional ablation studies be conducted to establish the necessity of frequency-domain representations in both LegoBlock and LegoMerge? For example, how would performance change if fusion occurred in the spatial domain?

2. Could the performance of LegoBlock be included in Table 1, not just in Table 4, to facilitate a more direct comparison in the main text? Additionally, certain phenomena, such as LegoBlock’s occasional underperformance compared to individual encoders like SNN or ABMIL, could benefit from further discussion.

3. A case study involving the fusion of more than two modalities would strengthen the paper’s claims about MM-Lego’s multimodal capabilities.
soundness: 3
strengths: 1. MM-Lego offers a versatile solution to integrate various encoders into a single multimodal model with limited fine-tuning, bringing novelty with its wrapper and frequency-domain fusion approach.

2. The paper provides an extensive comparison with existing multimodal fusion methods, addressing multiple advanced criteria and establishing MM-Lego’s relative advantages.

3. Comprehensive experiments validate MM-Lego’s performance, particularly in the biomedical domain.
summary: This paper introduces Multimodal Lego (MM-Lego), a flexible framework for transforming any set of encoders into a multimodal model with minimal or no fine-tuning. By using a wrapper mechanism to ensure shape consistency among different modality representations, MM-Lego enables efficient fusion of representations in the frequency domain, reducing signal interference across modalities. Results on seven tasks demonstrate MM-Lego’s strong performance.
weaknesses: 1. A primary concern is the lack of rigorous evidence supporting the claim that frequency-domain merging avoids signal interference. Unlike data types like images or time series, frequency representations of high-dimensional embeddings lack intuitive clarity, making the necessity of this approach uncertain.

2. While the authors highlight MM-Lego’s ability to handle multiple modalities, the paper lacks experiments involving more than two modalities. This limitation leaves the model’s robustness in such scenarios untested.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see the weakness.
soundness: 3
strengths: 1. The study problem is interesting and useful. How to fuse the feature/prediction from multimodal data better.
2. The proposed framework demonstrate good results on the datasets.
summary: This paper presents MM-Lego, a general-purpose and modular learning framework to build performant multimodal models with minimal fine-tuning.  They introduce a wrapper for unimodal encoders to enforce shape consistency between modality representations and also harmonize the latent representations in the frequency domain to enable model merging with little signal interference. This paper conduct experiments on biomedical datasets and shows good results.
weaknesses: In general, the presentation of this paper needs improvement, and the experimental section should be enhanced. 

1. The motivation for some key parts is not clearly articulated. For example, the authors mention "topological equivalence" and "topology agnostic." However, it is unclear why these properties are necessary for the problem being addressed. 

2. The proposed framework falls under the category of "model merging." However, I am uncertain if this classification is accurate. Essentially, the authors aim to explore how to enhance the fusion of different modalities. 

3. The key technical advancement appears to be the feature fusion conducted in the frequency domain. Again, the motivation for this approach is not well presented. Why is this method necessary? 

4. The authors claim that the proposed framework is scalable, particularly regarding the number of modalities. However, in most of the experiments, they only utilize two modalities. This claim requires more experimental results to substantiate it. 

5. It is unclear how the proposed framework addresses the issue of missing modality data, which is an important challenge in the field. 

6. The authors should elaborate more on the experimental settings and the methods used for comparison. For instance, it would be helpful to know if they employed the same feature backbone across experiments.

Meta-review: Key Strengths:
1. The paper presents a novel approach to multimodal data fusion using frequency-domain processing, with promising results across multiple datasets
2. The "wrapper" nature of LegoBlocks makes it adaptable to various tasks and architectures
3. The paper includes comprehensive comparisons with existing methods and evaluations on public datasets
4. The writing quality is good, particularly in the introduction and background sections

Major Concerns:
1. Limited Experimental Validation.
2. Insufficient theoretical justification for using frequency-domain processing for embedding fusion
3. Experimental setup lacks sufficient detail for reproducibility

Decision: Accept (Poster)