Multimodal Lego: Model Merging and Fine-Tuning Across Topologies and Modalities

Authors: konstantin  hemker
nikola  simidjievski
mateja  jamnik
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-23 21:55:13

Abstract: Learning holistic computational representations in physical, chemical or biological systems requires the ability to process information from different distributions and modalities within the same model. While there are many available multimodal fusion and alignment approaches, most of them require end-to-end training, scale quadratically with the number of modalities, cannot handle cases of high modality imbalance in the training set, or are highly topology-specific, making them too restrictive for many biomedical learning tasks. This paper presents \emph{Multimodal Lego} (MM-Lego), a general-purpose fusion framework to turn any set of encoders into a competitive multimodal model with no or minimal fine-tuning. We achieve this by introducing a wrapper for any unimodal encoders that enforces shape consistency between modality representations and harmonises these representations by learning features in the frequency domain to enable model merging with little signal interference. We show that MM-Lego 1) can be used as a \emph{model merging} method which achieves competitive performance with end-to-end fusion models \emph{without any fine-tuning}, 2) can operate on any unimodal encoder, and 3) is a \emph{model fusion} method that, with minimal fine-tuning, achieves state-of-the-art results on six benchmarked multimodal biomedical tasks.

Main paper: 
 introduction: 
 The utility and demand for multimodal machine learning approaches has sharply risen due to their potential to derive holistic representations in various systems, including physics [1] , chemistry [2] , neuroscience [3] , or biology [4] . Multimodal models in the vision & language domains leverage the same data distributions, which are represented across different modalities [5, 6, 7] , such as vision-text pairs of the same concepts. However, in many biomedical domains, modalities represent data at different scales (e.g., cellular, genomic, transcriptomic, etc.), cardinalities that are not paired (e.g., many singlecell reads for a single tissue slide per patient), and follow separate distributions. While large foundation models have excelled in tasks confined to individual modalities [8, 9, 10] , training these models across modalities is an expensive end-to-end process, that requires paired modalities. One recently emergent solution to these challenges is presented through model merging [11] (also referred to as knowledge fusion [12] ), an approach commonly used in the context of multi-task settings and language modelling, which capitalises on combining well-performing unimodal models trained in isolation. Model merging methods attempt to combine two architecturally identical models trained on different distributions through interpolation, arithmetic manipulation and aggregation of their weights [13, 14, 15] , or stacking their layers [16] , often without additional training/fine-tuning. While model merging has been extended to some multimodal vision and language tasks [17] , its crucial challenges in a multimodal setting are that: a) the merged components are still trained in isolation, and b) we cannot assume topological equivalence between two models for separate modalities due to their separate input shapes.In this paper, we present Multimodal Lego (MM-Lego) -a flexible framework for combining various unimodal models into a multimodal model with no or minimal fine-tuning (Figure 1 ). We introduce two approaches within our framework -LegoFuse and LegoMerge, enabling performant multimodal models given a set of unimodal encoders with either little (LegoFuse) or no (LegoMerge) fine-tuning. We show that MM-Lego satisfies multiple desirable properties in a range of real-world multimodal applications combining imaging, tabular, and time series modalities. We demonstrate the utility of MM-Lego on seven medical datasets across three separate downstream tasks, showing that it is: 1) performant without end-to-end training, 2) topology agnostic, 3) is scalable, and 4) handles modality imbalance and non-overlapping sets.Fusion methods. Given multiple data inputs or latent representations, fusion methods construct single learning representations that can be used for a downstream task, often whilst reducing dimensionality. Many fusion methods [3, 18] first learn a set of modality-specific encoders G = {g m : m → h (m) } assuming a single task label y. This results in a set of latent representations H = {g m (m), m ∈ M}, which are combined with a fusion operator to obtain the final fused representation z = ψ(H) and its final prediction ŷ = f (z). Following this problem setup, fusion methods can be differentiated by: 1) the choice of the fusion operator ψ(•); 2) the fusion stage in the pipeline of when ψ(•) is applied; and 3) the fusion order in which the fusion operations are applied (i.e., sequential vs. parallel). The fusion operator can be either static (e.g., concatenation [19] , Kronecker product [20] ) or learnable (e.g., lowrank tensor fusion [21] , cross-attention mechanisms [22, 23, 24] , mixture of experts [25, 26] ). The fusion stage is typically characterised as early, intermediate or late fusion. Early fusion methods apply a static fusion operator ψ(•) to the raw data while only applying this after passing each modality through G. Intermediate fusion methods often don't apply a static aggregation but rather learn a fusion function (i.e., a small sub-network or neural layer) in the latent space as part of its end-to-end training [27] .Model merging. The core idea behind model merging, typically deployed in multi-task settings, is that earlier layers in a network may learn similar features that may be used across tasks [28] . Using linear interpolation [11] or arithmetic manipulation [15] of the task-specific weights, model merging approaches have shown that they can effectively generalise to new tasks without any fine-tuning.Formally, given multiple tasks y (T ) for the same modality A, they first learn the set of task-specific functions F = {f t (ω t ) : A → ŷ(t) | t ∈ T } where ω denotes the corresponding model parameters. Assuming the same architecture for each model in F, parameters from a pre-trained base model ω base can be used to derive task vectors as V = {τ t ← ω t -ω base | t ∈ T } [15] . Given these task vectors, a multi-task model can be constructed by updating the weights of the base model ω ′ = ω base + λ T t τ t . This idea has been extended by the TIES [13] and DARE [29] that merge models through sparsifying and resolving sign conflicts in the task vectors. Another popular approach is spherical linear interpolation (SLERP), a method used to smoothly interpolate between two vectors while respecting the geometric properties of the vector space [11] . More specifically, given model parameters ω T1 and ω T2 ∈ R d , derived from two models with identical architectures, the merged multi-task model parameters can be calculated asω ′ = ω T1 sin(θ•(1-µ)) sin(θ) + ω T2 sin(θ•µ) sin(θ)where θ is the radian between the two vectors ω T1 and ω T2 [30] . The underlying assumption of the above model merging approaches is that the models should have an equivalent network topology, ensuring that the dimensions R d match up between tasks. However, while this is an acceptable constraint for multi-task learning, it is infeasible for multimodal models where modality shapes and the corresponding network topologies vary greatly. 
multimodal lego: 
 Preliminaries. Let X (M) = m∈M x m be a multimodal dataset where M = {A, B, . . . } represents the set of modalities m such as images (A), tabular data (B), time series (C), etc. Let X (A) i,j,k correspond to the element in the dataset for modality A at sample i, column j, and channel k,assuming A ∈ R I×J×K where 1 ≤ i ≤ N , 1 ≤ j ≤ J, 1 ≤ k ≤ K.Each sample in X has a set of discriminative task labels y (T ) = t∈T y (t) , where T = {T 1 , T 2 . . . , T c } is the set of possible tasks such that y (T1) = {y T1 1 , y T1 2 , ..., y T1 N } are the scalar target values for task T 1 for N samples. Architecture. Rather than learning a single fusion operator ψ(H) that applies to all latent representations at once, we learn a set of latent update functions for each modality, in the form ofEQUATIONwhereL (m) t∈ L is our target latent representation for each modality that we will later use in the merge and fusion, and S ∈ Z + is the number of update steps.Using the iterative update architecture with latent state passing in Equation 1 has a number of advantages. First, iterative attention architectures have been shown to be highly generalisable across modalities [31] , and effective at dealing with missing individual modalities [32, 33] . Second, since all modalities are encoded into a self-defined latent representation, we can impose a dimensionality constraint such that each latent has the same dimensions (e.g., L (A) , L (B) , L (C) ∈ R c×d for latent channels and dimensions c and d). Third, we can do latent state passing between elements in B, which allows us to "stack" the update functions on top of each other (hence the name) to sequentially encode each modality's signal into the same latent representation.LegoBlocks. Each element in B represents a LegoBlock, which learns the latent update function ψ m for any given encoder g m . Acknowledging that different data modalities and structures require different inductive biases to effectively encode each modality's information (g m ), LegoBlock acts as a wrapper to accurately encode its latent h m into L (m) . The benefit of training each modality update function separately instead of end-to-end is that we can train on entirely separate samples for the same tasks. For example, in many medical domains, we may have single-cell data for one subset of patients and bulk sequencing data for a different subset, while having the same task labels for the entire set. To address this, we use a latent representations L that effectively encode signal across modalities, and are robust or invariant to transformations (shifts, rotations, etc.), noise and signal interference.This motivated us to design MM-Lego for learning latent representations in the frequency domain, taking advantage of a number of desirable properties for multimodal merging and fusion. In particular, frequency-domain representations are: 1) signal-preserving as frequency features are less prone to signal interference upon aggregation (see Appendix F); 2) distance-preserving, as the Euclidean distance between two signals remains unchanged after the Fourier Transform (following from Parseval's Theorem [34] ), making them suitable for distance-based loss functions; 3) invertible as the spatial/temporal domain can be reconstructed, allowing for the iterative updates outlined in Equation 1; and 4) efficient, as the Fast Fourier Transform (FFT) has a time complexity of O(n log(n)), making it scalable to very large datasets [35] .Starting with the latent representation in the spatial domain, we first apply a discrete FFT F(•) [36] along each dimension of the 2D Tensor to yield a frequency domain representation.L F t (u, v) = c-1 i=0 d-1 j=0 L t (i, j)e -2πi( ux c + vy d ), where i, j denote the spatial-domain indices, and u, v denote the frequency-domain indices. This results in a complex frequency-domain representation from which we separate the real (symmetrical) and imaginary (asymmetrical) components of the FFT ((L F t ) r and (L F t ) i ) [37] . We update the real component using a standard cross-attention layer [38] , where we aim to learn the weight matrices W q m for the update query (L F t ) r , and W k m , W v m for the keys and values (h (A) ) resulting in the latent update:EQUATIONIn contrast to other Fourier-based architectures [35] , which only use the real component of the transform, we keep track of the imaginary component (L F t ) i as well. This allows us to reconstruct the complex representation, and subsequently apply the inverse transform. We found this to be critical for our iterative architecture, as otherwise the signal gets distorted and we lose phase information (encoded in the imaginary component) at each update pass. Once we reconstruct the complex representation, we apply the inverse transform to recover the spatial representation in preparation of the next passL t+1 = F -1 ((L F t+1 ) r + i(L F t ) i ).Finally, the last task-specific heads of each block are a fully-connected layer after applying layer normalisation. We omit the inverse transform after the last update such that each head is trained in the frequency domain. This ensures that we can apply aggregations with low signal interference on L during LegoMerge.With the architectural assumptions imposed on each modality encoder in G through LegoBlocks B, we can apply model merging techniques in a multimodal setting. With L ⊆ R c×d and each element in L being in the frequency domain, we can use aggregation functions ψ(•), which are less prone to cancelling out signal. For example, let L (A) and L (B) be the final frequency domain latent representations for modalities A and B, then we can calculate a merged multimodal representation as:EQUATIONwhere the real component is the harmonic mean of the magnitudes (| • |), and the imaginary component is the arithmetic mean of the phases (∠) of L (A) and L (B) . We take the harmonic mean since it is less prone to outliers [37] , that is, the merged representation is less likely to be strongly skewed towards either modality by very large frequency components. With the cross-modal combined representation L (M) , we need to combine the task heads of each block, where we apply spherical linear interpolation (SLERP) [30] for the set of task heads Y from each element in B.LegoFuse. LegoFuse overcomes the limitations of LegoMerge of training each element in B in isolation, thus allowing for modalities to mutually contextualise each other. As such it requires a minimal amount of fine-tuning. To avoid fine-tuning a potentially noised signal emerging from the merged latent L (M) , LegoFuse operates at the layer level (by sequentially passing through all layers in B), rather than directly fine-tuning the merged model (at the parameter-level). Specifically, the shape consistency introduced by L ⊆ R c×d allows the stacked model to pass the Fourier-transformed latent states either between blocks (stacking) or different layers between blocks (weaving), as illustrated in Figure 1 . We then fine-tune the stacked/weaved model for a few epochs with all (paired) modalities, such that the state updates are conditioned on all modalities' updates. This, in turn, becomes the query for the cross-attention layer. Note that, both the stacked and weaved variants of LegoFuse allow for fine-tuning all model parameters, including the ones of the initial modality-specific encoders. 
results & discussion: 
 Experiments. We evaluate MM-Lego (LegoMerge and LegoFuse) and its components (LegoBlock) on seven multimodal medical datasets covering three separate modalities (images, tabular, time series) [40] , and skin imaging (International Skin Imaging Collaboration (ISIC)) [41] . The seven tasks shown in our results correspond to survival analysis tasks on four TCGA sites (BLCA, BRCA, KIRP, UCEC), classification tasks on two variants of MIMIC (disease classification (ICD9) and patient mortality (MORT), and predicting melanoma for the ISIC patients. Further details on datasets and task setup can be found in Appendices C and D.Discussion. With the increasing volume, complexity and diversity of collected biomedical data, (re)training multimodal models from scratch becomes more expensive, unsustainable, and even infeasible. Going beyond computational constraints, further desired properties that guided the design of MM-Lego are outlined in Table 1 . Our results in Table 2 provide strong evidence that MM-Lego meets these requirements, efficiently achieving competitive performance.LegoMerge matches end-to-end trained multimodal baselines in most tasks without any additional training, while LegoFuse outperforms strong baselines with as little as 2 epochs of fine-tuning. Notably, LegoMerge does not require a single paired modality training sample whilst still being useful for multimodal inference, outperforming ensemble models (Appendix B). Our results also show that MM-Lego addresses a key limitation in model merging literature which assumes topology equivalence. While this is a feasible assumption for model merging in multi-task learning, different data shapes across modalities limit the application of these methods in multimodal settings. Therefore, the design of LegoBlock is sufficiently permissive to use any unimodal encoder as part of this framework, whilst enforcing the necessary architectural assumptions required for model merging.Our findings (in Figure 2 , Appendix 3) support this, showing that any unimodal encoder (such as SNNs and AMIL) can be wrapped in a LegoBlock without any practical loss in performance.To the best of our knowledge, MM-Lego is the first general-purpose model merging framework for multimodal data outside of the vision & language domains.Objects.• X (A) : matrix corresponding to modality A • x (A) : a vector in X (A) (e.g., a sample of modality A)• X (A)i,j,k : elements of matrix X (A) at row i, column j, channel k, assuming Table 3 : Task performance of uni-and multimodal models across 7 medical datasets -for each task target metric, we highlight the best and second-best models.X (A) ∈ R I×J×K where 1 ≤ i ≤ I, 1 ≤ j ≤ J, 1 ≤ k ≤ K • X (M) = m∈M X (m) : multimodal dataset • y ∈ Y = t∈T y (Figure 2 : AUC performance on the MIMIC dataset when merging existing encoders (SNN for tabular, AMIL for Time Series) using LegoMerge and LegoFuse. Our multimodal model merge shows much better performance than using an ensemble, exhibiting the performance gains, at no additional costs, through the merge even prior to fine-tuning in LegoFuse. We evaluate MM-Lego (LegoMerge and LegoFuse) and its components (LegoBlock) on seven multimodal medical datasets covering three separate modalities (images, tabular, time series) from three separate sources: histopathology (The Cancer Genome Atlas (TCGA)) [39], intensive care data (Medical Information Mart for Intensive Care (MIMIC)) [40] , and skin imaging (Society for Imaging Informatics in Medicine & International Skin Imaging Collaboration (SIIM-ISIC)) [41] . [41] , which contains both tabular data and images of skin leisures to be classified for melanoma patients. To account for class imbalance, we randomly downsampled the majority class to a 5:1 ratio for the class of interest (melanoma) to a sample size of n = 2875. All images were patched and encoded using the resnet50-kather100k for TCGA (ResNet pre-trained on a large histopathology patch collection) [45] and a regular ImageNet v2 pre-trained ResNet for the pictures of skin leisures. Both images (patch encodings) and times series were represented as 2D tensors, and the tabular clinical and multi-omic data as 1D tensors to pass into the modality-specific encoders g(•).The results report the (unseen) test set performance, by evaluating the concordance Index (c-Index) in the case of TCGA, AUC in the case of MIMIC-III-ICD9 and ISIC, and Macro-AUC ("one-vs-rest") for MIMIC-III-ICD9. As indicated in Figure 1 the output of each task head in Y are the logits with predictions for each class given the final Fourier-transformed latent state y l = f (L F T ). Since TCGA is a survival prediction task with right-censored data, we have divided the survival period into four non-overlapping bins and use the logits of these bins to calculate the hazard (y h = 1 1e -y l ) and survival (y s = k 1 1 -y h ) respectively for k bins. Given the hazards, censorship, and ground truth bins, we can calculate the negative log-likelihood loss from a proportional hazards model [46] which is used as the survival loss. We evaluate the performance using the Concordance Index (c-Index), for which we determine the fraction of paired samples in which the prediction outcomes are concordant with the ground truth. As MIMIC and ISIC relate to classification tasks, we employ categorical cross-entropy loss for training. Note that both AUC and the c-Index have similar interpretations, therefore the values range between [0.5 -1].Baselines. For all experiments, we compare LegoMerge and LegoFuse to several uni-and multimodal baselines to evaluate their performance. For all tabular modalities, we use a self-normalising network [42] due to its performance and regularisation mechanisms suitable for high-dimensional tabular data. For the image and time series modalities, we use an attention-based Multiple Instance Learning (AMIL) [43] . Across all modalities, we benchmark two related iterative-learning architectures: MultiModN [32] and Perceiver [31] which generally shows strong performance across a wide range of unimodal tasks. In terms of specific multimodal baselines, we use two late fusion combinations of SNN+AMIL, namely concatenation of their final latent representation and bi-linear fusion [47] . For the Perceiver, we use the same multimodal setup as suggested in the original paper, i.e., concatenation of modalities before passing them into the model. We use two additional domain-specific multimodal baselines: the Hybrid Early-Fusion Attention Learning Network (HEALNet) [33] which is using an end-to-end trained iterative cross-attention architecture and the Multimodal Co-Attention Transformer (MCAT) [44] which is using the tabular (1D) modality as context for the imaging (2D) modality.Validation & Compute. For each experiment and dataset, we perform a 5-fold repeated random sub-sampling with a 70-15-15 train-test-validation split. We re-ran all of the baseline models in this paper using their open-source code to ensure that no performance differences are caused by different task setups, losses, or training splits. We ran a brief Bayesian Hyperparameter search [48] for key parameters of each model (learning rate, decay, schedulers, dropout, layer dimensions). The experiments were run on a single Nvidia A100 80GB GPU on a Ubuntu 22.04 virtual machine. Both the complete MM-Lego experimental code as well as its "lightweight" PyTorch package (installable via the Python Package Index (PyPI)) will be published on GitHub.Figure 4 : Example of signal interference on a random normal latent variable and its additive inverse variable with some added noise, showcasing a severe case of signal interference where nearly all signal cancels out. We can see that the fourier-transformed data does not suffer this problem when we apply the harmonic mean. This is a key reason for the choice of model merging architecture. shows that this logic can also be flawed. We demonstrate this using a squarewave function with a frequency offset beteen M od A and M od B and a scaled amplitude by a normal distribution. We can see that the mean of the regular and the absolute values suffers some signal interference while the FFT aggregation does not. Preprint. 
 
 Appendix:  


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: Official Review by GXpz
main_review: main_review: **Paper Summary** \
The paper introduces a novel framework (Multimodal Lego) for merging unimodal encoders into a multimodal model using model-specific wrappers. Standard modality merging frameworks require paired data for jointly training the encoders end-to-end, which poses a significant challenge in practice. This work, however, enables training encoder-specific wrappers in isolation, requiring no paired data or homogeneous encoder architectures, which is often assumed in the model merging literature. These wrappers, referred to as Lego Blocks, exploit the advantages of the frequency domain, such as robustness to aggregation interference, to generate modality-agnostic and shape-consistent representations, facilitating merging. The paper proposes two variants of merging: one without end-to-end fine-tuning and one with, where the latter achieves state-of-the-art performance in their benchmarks. The use of frequency-domain representations, often overlooked, and the ability to merge encoders with arbitrary architectures are key contributions of this work.

**Strengths** \
Lego Merging is highly flexible and practical, as it requires no paired data and allows the use of arbitrary encoder architectures. The decision to use the frequency domain appears well-grounded, and the merging process is efficient. Additionally, the design of Lego Blocks significantly reduces the engineering effort for aligning different modalities, as they produce compatible representations. Overall, these strengths make the framework a notable contribution to the field.

**Weaknesses** \
The motivation for the iterative updates within each block is not fully explained. It would be helpful to clarify how these iterations contribute to the learning process and whether they affect convergence or model stability. Furthermore, the paper does not adequately discuss the computational costs associated with multiple iterations, which may be a concern.

**Questions & Suggestions**
 - Clarify the motivation behind the iterative updates and how they affect model performance and stability. Also, provide details on the computational overhead associated with these iterations.
 - Consider including additional datasets from non-medical domains (e.g., vision-and-language or audio-visual tasks) to demonstrate the generalizability of the framework across diverse modalities. Replacing some results from the medical domain with results from new domains could broaden the paper's applicability.
 - Given the flexibility of the framework, it would be beneficial to include ablation studies on key design choices, such as the use of frequency domain vs. spatial domain representations, and the effect of stacking vs. weaving layers in the LegoFuse variant. This would significantly strengthen the claims of generalizability and adaptability of the Multimodal Lego framework.
title: title: Multimodal Lego: Combining Unimodal Models in Biomedical Models - Well Motivated but Lacking Clarity on the Claims
main_review: main_review: ## Overview

This paper introduces Multimodal Lego (MM-Lego), a framework for combining unimodal models into performant multimodal models with minimal or no fine-tuning. The method is designed to address challenges in multimodal learning for biomedical tasks, where data from different modalities may have different distributions, scales, and cardinalities. The method is well-designed and has a strong motivation, but could benefit from some clarification.

## Pros

1. **Novel approach**: MM-Lego offers a unique solution to multimodal fusion that doesn't require end-to-end training.

2. **Flexibility**: The framework can work with various unimodal encoders and handle different modalities (images, tabular data, time series).

3. **Comprehensive evaluation**: The authors test their approach on seven multimodal medical datasets.

4. **Clear motivation**: The paper provides a compelling motivation for the biomedical domain application.

5. **Innovative technique**: The authors make innovative use of Fourier transforms for multimodal fusion. Page 3 describing the Fourier motivation, the cross attention, and the merging is well written. 

## Cons

1. **Limited applicability to pre-existing models**: 
   - The method requires training the unimodal Fourier block from scratch for each modality.
   - This requirement may limit its applicability to pre-existing models.
   - The abstract and introduction initially suggest that any pretrained unimodal model can be used without additional training, which does not seem to be the case. Only the specific unimodal lego black can be used without additional training.

2. **Imprecise notation**:
   - In line 47, is the union of $\cup_{m \in \mathcal{M}} m$ just $\mathcal{M}$? Did the authors mean $x^{m}$?
   - The notation $\mathcal{M}$ from A to Z implies precisely 26 modalities.
   - In line 50, is it only $A \in \mathbb{R}^{I \times J \times K}$, or is any modality $m \in \mathbb{R}^{i \times j \times k}$?
   - What is $S$ in equation (1)? Is $S = \mathbb{R}$ or more precisely $S = \mathbb{Z}^+$ (set of positive integers)?
   - In line 60, what is $h_m$? This variable appears to be undefined.

4. **Vague comparisons in Table 1**:
   - The table is neat but somewhat vague/subjective.
   - Are the authors referring to specific previous works?
   - For Late Fusion, can't three separate unimodal models perform without end-to-end training or add modalities without retraining by averaging the output logits or performing simple voting for the output class?
   - The meaning of "architecture agnostic" in the context of this table needs clarification.
   - Don't the Lego methods require that the architecture extracts a latent vector, thus not any architecture can be used for them?
   - Or does this refer to the fact that only the feature extractor $g$ can be implemented with various backbones?

5. **Figure 1 clarification**: 
   - Should the task head in (1) be outside of the Fourier Lego block?
   - If not, in 2A, would the authors be taking the harmonic mean of the output logits?
   - In 2B, would they be passing the output logits into the next block?

6. **Highlight disjoint data experiments**:
   - In line 62, the authors provide a good motivation for single-cell data versus bulk sequencing data and the need to train on different samples for the same task.
   - Did any of the tested datasets have disjoint data like this for different modalities?
   - If so, this should be highlighted as it appears to be the main motivation for medical data applications. It's not clear whether this goal is achieved by the end of the paper.
title: title: Review of "Multimodal Lego: Model Merging and Fine-Tuning Across Topologies and Modalities"
main_review: main_review: This paper presents Multimodal Lego (MM-Lego), a novel framework designed to merge unimodal encoders into a competitive multimodal model with minimal or no fine-tuning. MM-Lego addresses key challenges such as handling modality imbalance, ensuring topology-agnostic fusion, and avoiding signal interference through frequency-domain learning. The paper demonstrates impressive performance across seven medical datasets spanning multiple modalities. However, the complexity of the method’s explanation could benefit from more clarity and better visual representation of some core ideas.
The quality of the research is high, with a well-thought-out approach that addresses several significant challenges in multimodal learning, particularly in medical applications. The use of Fourier transforms to reduce signal interference is an innovative solution, and the experimental results are promising.

Strengths:

The architecture is versatile, capable of combining multiple data modalities without retraining, which is a major advantage in computationally constrained environments.
The performance improvements shown in various datasets demonstrate the method’s applicability across tasks like survival analysis, disease classification, and image-based diagnostics.
The results indicate that LegoMerge achieves comparable performance to end-to-end training models, while LegoFuse outperforms these models with just a few epochs of fine-tuning, highlighting the efficiency of the approach.
Weaknesses:

The mathematical formalism could be explained more clearly, particularly for the transformation between spatial and frequency domains. Readers unfamiliar with Fourier transforms might find these sections challenging to follow.
Although the performance is strong, additional comparisons with other state-of-the-art methods beyond the selected baselines (e.g., transformers specifically designed for multimodal data) could provide a more comprehensive evaluation of the model’s advantages.
Clarity
The paper is generally clear but could improve in certain sections, particularly in explaining some of the more technical details.

Strengths:

The figures are helpful, especially in illustrating the workflow of MM-Lego, and the architecture is well-documented in terms of its components.
The discussion of signal interference and the use of the harmonic mean in the frequency domain is insightful, providing a strong rationale for the design choices.
Weaknesses:

The explanation of LegoBlocks and how they interact with each other could be more intuitive. The reader may benefit from more visual aids or examples of how unimodal models are transformed and fused within the MM-Lego framework.
More explicit examples of real-world scenarios where the model’s flexibility and efficiency would be critical could help contextualize its practical benefits.
Originality
The originality of the approach is evident, particularly in the use of frequency-domain learning for multimodal model merging. This paper stands out by offering a flexible, scalable method that doesn’t require paired modalities for training.

Strengths:

The concept of using Fourier transforms to handle signal interference and ensure smooth model merging is novel and effective.
MM-Lego’s ability to operate on unpaired data and still achieve strong performance is a unique feature that sets it apart from many existing multimodal approaches that rely heavily on paired data.
Weaknesses:

While the approach is novel, some components, such as the fine-tuning with LegoFuse, bear similarities to standard model fusion techniques. More emphasis could be placed on how these elements differ from traditional methods.
Significance
The significance of this work lies in its potential to simplify the process of building multimodal models in fields such as healthcare, where different data modalities are common, but training paired models can be difficult or infeasible.

Strengths:

The ability to merge unimodal models without end-to-end training is a significant breakthrough, especially for large-scale medical datasets where computational resources are often limited.
MM-Lego’s flexibility and scalability make it well-suited for real-world applications, particularly in medical and scientific research, where datasets often include unpaired modalities.
Weaknesses:

The practical implications of the model’s use in real-time clinical settings or other high-stakes applications could be further explored. Demonstrating how the framework handles challenges like real-time inference or highly imbalanced datasets would add to its significance.
Pros
Innovative use of frequency-domain learning to prevent signal interference in multimodal model merging.
Flexible framework that allows for the merging of models without requiring paired modality data or retraining.
Strong performance on multiple datasets across different modalities, demonstrating the framework’s versatility.
Efficient architecture that achieves competitive results with minimal fine-tuning.
Cons
Technical explanations, particularly regarding Fourier transforms and signal interference, could be clearer for a broader audience.
Limited comparisons with more advanced multimodal models could reduce the impact of the results.
The practical benefits in real-world, time-sensitive applications need further exploration.
title: title: Review
main_review: main_review: The paper focuses on the problem of multimodal fusion and alignment, and proposes a method to fuse data from different modalities based on unimodal encoders with no or minimal fine-tuning. The method outperforms baselines in the multimodal biomedical benckmarks.

Strengths:
- The proposed method is able to make good use of unpaired unimodal data and achieves multimodal fusion with little fine-tuning.
- The idea to utilize the representations in the frequency domain is interesting.
- The method performs well in real-world multimodal datasets.

Limitations:
- Are there theoretical guarantees for the effectiveness of such fusion in the frequency domain, especially in the fine-tuning free setting?
title: title: Multimodal Lego: Model Merging and Fine-Tuning Across Topologies and Modalities
main_review: main_review: The authors propose and evaluate a model merging framework, where they use the frequency domain to merge unimodal models trained separately into a well-performing multimodal model. Because they merge latent representations in the frequency domain, the framework can deal with different input shapes. They also evaluate a multimodal model that allows the modalities to interact with each other through minimal fine tuning of a stacked or layer-woven combined model and find that this model outperforms other multimodal (trained end-to-end) models on biomedical multimodal tasks. 

Pros:

- Addresses a challenging problem: combining differently shaped inputs in a multimodal model
- Interesting, well-motivated, and scalable method to merge model representations in the frequency domain

Cons:

- I do not have enough experience in this area to identify any cons, if there are any

Comments: title: Thank you for your reviews
comment: We would like to thank all reviewers for their constructive feedback on our extended abstract submission! Please do not hesitate to come and talk to us during the workshop to discuss some of these comments further!
Thank you very much for your thoughtful, detailed, and constructive feedback. We are very encouraged by your positive view of the paper. We acknowledge the notational clarifications and have addressed these along with other comments in the camera-ready version.

We also would like to answer some of your questions below and encourage you to reach out on the day of the workshop to discuss this further! 

**Comments on Table 1**: 
* Vagueness/subjectivity: We had to make some compromises given the space constraints of the extended abstract format which meant that we briefly cut an introductory section to the late, intermediate, and early fusion benchmarks. We have used a part of the additional page in the camera-ready version for these clarifications. 
* Late fusion: It's true that you could use simple ensembling, but the performance is typically worse than the unimodal model, as we show in Appendix B. Hence the wording of "_performant_ without end-to-end-training"
* Architecture agnostic: this refers to 1) that many model merging methods require equivalent architectures (e.g., in multi-task settings) while MM-Lego does not make that assumption and 2) that you can use it with any modality-specific encoder

**Comments on Figure 1**: 
* Task head: Yes, the task head is outside of the Lego block and we have updated the grey box in Fig. 1.1 to reflect that. This should also answer the remaining questions.

**Experiments and relevance**
* The experiments are still run on paired datasets to allow a fair comparison with the fusion baselines, which require paired data. We agree that this would be an interesting additional experiment which can be conducted with TCGA where we have a lot of disjoint data available for other modalities that we currently do not include. It is worth noting that the fact that we "artificially" disjoin the data for the LegoMerge and LegoFuse experiments, so the experiments shown here are still relevant to its core motivation.
Thank you very much for your thoughtful and constructive feedback. We are very encouraged by your positive view of the paper and address your questions below as well as the camera-ready version: 

**Motivation for iterative updates**: The main motivation behind an iterative architecture is that missing modalities (or "blocks") can be easily skipped for individual samples, which is in line with the original motivation of the paper that little paired data is available. We will make sure that this is clearer in the methods section of the final manuscript. 

**Adding datasets and additional ablations**: We fully agree with this suggestion as no current methodological choices would restrict the method to medical modalities. This is planned for a further iteration of this project.