Multimodal Structure Preservation Learning

## Abstract

Abstract When selecting data to build machine learning models in practical applications, factors such as availability, acquisition cost, and discriminatory power are crucial considerations.
Different data modalities often capture unique aspects of the underlying phenomenon, making their utilities complementary.
On the other hand, some sources of data host structural information that is key to their value.
Hence, the utility of one data type can sometimes be enhanced by matching the structure of another.
We propose Multimodal Structure Preservation Learning (MSPL) as a novel method of learning data representations that leverages the clustering structure provided by one data modality to enhance the utility of data from another modality.
We demonstrate the effectiveness of MSPL in uncovering latent structures in synthetic time series data and recovering clusters from whole genome sequencing and antimicrobial resistance data using mass spectrometry data in support of epidemiology applications. The results show that MSPL can imbue the learned features with external structures and help reap the beneficial synergies occurring across disparate data modalities.

## 1 Introduction

Selecting the appropriate data is critical when deploying machine learning models in real-world applications.
Factors such as availability, acquisition cost, and discriminatory power (reflected by information density, resolution, etc.) are of primary concern.
On the other hand, distinct data modalities may encode different information about the same underlying phenomenon (Xu et al., ; Baltrušaitis et al., ), thus forming a gap in utility.
For instance, in medical diagnostics, imaging data such as X-ray and CT scans reveal structural anomalies, while genomic sequencing offers insights into the molecular mechanisms of diseases (Esteva et al., ).

Traditionally, research in multimodal machine learning attempts to bridge this gap by learning a shared feature space between data modalities (Li et al., ; Liu et al., ; ). Deviating from these feature-level alignment approaches that require complete data in two modalities, we approach the problem from structure-level alignment: In scenarios such as clustering, it is the structure of the data that directly influences the results. Thus, learning representations of one data modality that preserve the structure of another can extend the utility of the former and effectively bridge their gap.

For instance, in hospital outbreak investigations, epidemiologists use the single nucleotide polymorphism (SNP) distance defined on whole genome sequencing (WGS) data to cluster
microbial samples, assess their lineages, and identify outbreaks.
WGS provides the highest discriminatory power and is considered the gold standard in microbial disease epidemiology (Bertelli & Greub, ).
However, the labor, cost, and expertise required for WGS make it
prohibitive to deploy broadly (Rossen et al., ).
On the other hand, due to its low cost and rapid time to generate results, Matrix-Assisted Laser Desorption Ionization–Time of Flight (MALDI-TOF) mass spectrometry rose as a standard tool for microbial species identification in clinical microbiology laboratories (Croxatto et al., ; Clark et al., ).
Though MALDI has weaker discriminatory power than WGS, it is gaining attention as a potential cost-effective alternative to WGS for hospital outbreak detection (Griffin et al., ).
Hence, if MALDI representations that preserve the SNP distance structure can be learned, its utility can extend from species identification to outbreak detection,
making it a viable substitute for WGS in practice.

To achieve the aforementioned goals, we propose a novel machine learning framework called Multimodal Structure Preservation Learning (MSPL) that learns data representations that enhance the utility of one data modality through alignment with the dissimilarity-based clustering structure provided by another data modality. We first demonstrate the effectiveness of MSPL in identifying latent structures on a synthetic time series dataset. We then apply MSPL to epidemiology settings, where we enhance the utility of MALDI mass spectrometry by leveraging the clustering structure in whole genome sequencing (WGS) and antimicrobial resistance (AMR) data, respectively.
Our results demonstrate that MSPL can effectively inject structural information of one modality into the representations of another, improve the clustering performance, bridge the utility gap of two modalities, substantially reduce data acquisition cost, and increase feasibility of learning.

## 2 Related work

Multimodal connectors.
In multimodal machine learning models, connectors are employed to bridge the gap between different modalities in the feature space. In vision-language pre-training, Li et al. () proposed the Querying Transformer (Q-Former), a connector that learns query vectors to extract the visual features most relevant to the text.
The Q-Former architecture has also been adopted to align time series and text features (Cai et al., ).
Llava (Liu et al., ), a pioneering work in visual instruction fine-tuning, introduced a more lightweight connector using just a linear projection that projects image features onto the word embedding space. Liu et al. () later improved Llava’s multimodal capabilities by changing the linear projection connector to a two-layer multilayer perceptron (MLP), which affords more representation power. However, Qi et al. () found that multimodal connectors can be performance bottlenecks for multimodal large language models and may fall short with insufficient training data compared to the amount of pre-training data. They proposed to enhance the connector with retrieval-augmented tag tokens that contain rich object-aware information.

Structure in multimodal self-supervised learning.
Pre-training approaches in multimodal self-supervised learning, e.g., CLIP (Radford et al., ), commonly employ an instance-wise contrastive objective  (Oord et al., ) to learn joint features. However, the contrastive objective ignores the underlying semantic structure across samples (Zellers et al., ; Singh et al., ), and thus may adversely impact model performance. To remedy this issue, Chen et al. () proposed to combine the contrastive objective with a joint multimodal clustering objective to capture the cross-modal semantic similarity structure. Alternatively, Swetha et al. ()
preserved modality-specific relationships in the joint embedding space by learning semantically meaningful “anchors” and representing inter-sample relations with sample-anchor relations.

Deep learning for MALDI spectrometry. Deep learning is widely applied to MALDI spectra analysis. Weis et al. () used an MLP to encode MALDI spectra of bacterial strains and predict their antimicrobial resistance to a range of drugs. Normand et al. () utilized a 1-D CNN to identify a subpopulation from MALDI data of the same species. Abdelmoula et al. () employed a variational autoencoder to learn low-dimensional latent features of MALDI spectra that reveal biologically relevant clusters of tumor regions.

## 3 Methods

Figure: Figure 1: Overview of the MSPL framework.
Refer to caption: x1.png

##### Multimodal Structure Preservation Learning.

Figure 1 presents an overview of the MSPL framework.
MSPL entails three objectives:
(1) reconstruction of the input data as in standard autoencoders for feature extraction,
(2) a pretext task on which the input data has discriminatory power, and
(3) structure preservation through alignment between the clustering structure of two modalities.

Formally, the input data to MSPL is $\bm{x}$ with batch size $N$. An autoencoder consisting of an encoder $\text{Enc}_{x}(\cdot)$ and a decoder $\text{Dec}(\cdot)$ encodes $\bm{x}$ to a latent representation $\bm{h}_{0}$ and uses it to obtain the reconstructed input data $\hat{\bm{x}}$. A further encoding step $\text{Enc}_{h}(\cdot)$ prepares $\bm{h}_{0}$ for the pretext task and structure preservation. In all our experiments, the autoencoder adopts the U-Net (Ronneberger et al., ) architecture on 1-D data, and the pretext task is defined as a classification task, accomplished by the classification head $\text{CLS}(\cdot)$. The mathematical formulation for the MSPL framework is as follows:

$$ $\displaystyle\bm{h}_{0}=\text{Enc}_{x}(\bm{x}),$ (1) $\displaystyle\hat{\bm{x}}=\text{Dec}(\bm{h}_{0}),$ (2) $\displaystyle\bm{h}=\text{Enc}_{h}(\bm{h}_{0}),~{}$ (3) $\displaystyle\bm{z}=\text{CLS}(\bm{h}).~{}$ (4) $$

We then define the loss functions corresponding to the three objectives:

$$ $\displaystyle\mathcal{L}_{\text{recon}}=\frac{1}{N}\|\bm{x}-\hat{\bm{x}}\|_{2} ^{2},$ (5) $\displaystyle\mathcal{L}_{\text{pretext}}=\text{CE}(\bm{z},\bm{y}),$ (6) $\displaystyle\mathcal{L}_{\text{struct}}=f_{\text{struct}}(\text{pdist}(\bm{h} ),\bm{d}).$ (7) $$

Here, $\bm{y}$ represents the labels for the pretext task, $\text{CE}(\cdot)$ stands for the cross-entropy loss, $\text{pdist}(\bm{h})$ computes the
$\ell_{2}$ distance between each pair of row vectors in the learned feature matrix $\bm{h}$, and
$\bm{d}$ refers to the external dissimilarity matrix computed from another modality.
The function $f_{\text{struct}}$ matches these dissimilarities measured in the two modalities and by default is implemented as the mean squared error, but can vary as required by application.

The loss for MSPL is a weighted sum of these three losses:

$$ $\mathcal{L}_{\text{MSPL}}=\mathcal{L}_{\text{recon}}+\lambda_{0}\mathcal{L}_{ \text{pretext}}+\lambda_{1}\mathcal{L}_{\text{struct}},$ (8) $$

where $\lambda_{0}$ and $\lambda_{1}$ are hyperparameters controlling the relative weights of the component losses.

Through MSPL, we aim to learn input data representations $\bm{h}$ that retain discriminatory power on the pretext task while preserving the clustering structure characterized by the external dissimilarity matrix $\bm{d}$. For example, we learn representations of MALDI spectra that readily identify species (a pretext task) while their pairwise distances recover the clusters defined by the dissimilarity among WGS or AMR data.

##### Baseline models.

In comparison with MSPL, we develop two baseline models.
The first model, named “onlyCLS,” constructs an ablation study by removing the structure preservation objective, i.e., $\mathcal{L}_{\text{struct}}$, from the MSPL loss. Hence, onlyCLS can only rely on the feature extraction capabilities of the autoencoder and the discriminatory power of the pretext task to implicitly recover the clustering structure of another modality.

The second model, named “clusCLS,” formulates structure preservation through classification. Specifically, the ground truth cluster labels $\mathcal{C}_{T}$ are derived beforehand using the external dissimilarity matrix $\bm{d}$ on the full dataset. Then, an additional classification head $\text{CLS}_{C}(\cdot)$is used to classify these cluster labels:

$$ $\bm{z}_{c}=\text{CLS}_{C}(\bm{h}\|\bm{z}),$ (9) $$

where “$\|$” stands for the concatenation operation, and $\bm{h}$ and $\bm{z}$ are the learned features and pretext classification logits as computed in Eqs. 3 and 4, respectively. clusCLS also replaces $\mathcal{L}_{\text{struct}}$ with the following cross-entropy loss:

$$ $\mathcal{L}_{\text{structCLS}}=\text{CE}(\bm{z}_{c},\mathcal{C}_{T}).$ (10) $$

##### Cluster evaluation.

We cluster the learned representations $\bm{h}$ and measure the similarity of the generated clusters to those derived from the external dissimilarity matrix $\bm{d}$.
Besides adopting the Adjusted Rand Index (ARI) and Normalized Mutual Information (NMI) as standard evaluation metrics, we propose an alternative cluster similarity metric, cluster F1 score, as described below.

We first define the purity of a cluster assignment with respect to ground truth labels: given a dataset $X$, a set of clusters $\{C_{1},\cdots,C_{p}\}$ and a label set $Y$ on $X$, the purity score for the $j$-th cluster is defined as

$$ $\text{Purity}(C_{j},Y)=\frac{1}{|C_{j}|}\max_{k\in Y}|C_{j}\cap T_{k}|,$ (11) $$

where $T_{k}$ is the set of data points in $X$ with label $k$.

We then extend the purity metric to our cluster evaluation setting.
Specifically, given dataset $X$, predicted clusters $\mathcal{C}_{S}=\{C_{S}^{1},\cdots,C_{S}^{p}\}$ derived from the input data representations $\bm{h}$, and “ground truth” clusters $\mathcal{C}_{T}=\{C_{T}^{1},\cdots,C_{T}^{q}\}$ derived from the external dissimilarity matrix $\bm{d}$, we define the cluster precision, recall, and F1 score of the predicted $\mathcal{C}_{S}$ with respect to $\mathcal{C}_{T}$ as follows:

$$ $\displaystyle\text{Prec}(\mathcal{C}_{S},\mathcal{C}_{T})=\frac{1}{p}\sum_{i=1 }^{p}\text{Purity}(C_{S}^{i},\mathcal{C}_{T}),$ (12) $\displaystyle\text{Rec}(\mathcal{C}_{S},\mathcal{C}_{T})=\frac{1}{q}\sum_{j=1} ^{q}\text{Purity}(C_{T}^{j},\mathcal{C}_{S}),$ (13) $\displaystyle\text{F1}(\mathcal{C}_{S},\mathcal{C}_{T})=2\cdot\frac{\text{Prec }(\mathcal{C}_{S},\mathcal{C}_{T})\cdot\text{Rec}(\mathcal{C}_{S},\mathcal{C}_ {T})}{\text{Prec}(\mathcal{C}_{S},\mathcal{C}_{T})+\text{Rec}(\mathcal{C}_{S}, \mathcal{C}_{T})}.$ (14) $$

To reach high precision, each predicted cluster must contain as few distinct ground truth cluster labels as possible (i.e., be pure). To achieve high recall, the data points with the same ground truth label should be clustered together in the predictions. The defined precision reaches its maximum value $1$ when $p=|X|$ and $|C_{S}^{i}|=1$ for all $i$, while the recall reaches its maximum value $1$ when $p=1$ and $|C_{S}^{1}|=|X|$.
In contrast, a cluster assignment with a high $F_{1}$ score does not fall into either extreme. Nevertheless, to avoid the impact of singleton ground truth clusters ($|C_{T}^{j}|=1$) on the above metrics, we only evaluate them on the subset of the dataset where the ground truth cluster has more than $1$ element, i.e., $\cup_{j:|C_{T}^{j}|\geq 2}C_{T}^{j}\subseteq X$.

## 4 Datasets

To demonstrate the ability of MSPL to learn representations that preserve external clustering structure from another modality, we utilized three datasets: a synthetic time series dataset (Synth-TS), a proprietary dataset of MALDI spectra with paired whole genome sequencing SNP distance profiles, and a public dataset of MALDI spectra paired with AMR profiles, Database of Resistance Information on Antimicrobials and MALDI-TOF Mass Spectra (DRIAMS) (Weis et al., ).

##### Synth-TS.

Figure: Figure 2: Generating the Synth-TS dataset.
Refer to caption: x2.png

We construct a synthetic time series dataset called Synth-TS to demonstrate the ability of MSPL to learn representations that preserve external structure. Each time series in Synth-TS is a superposition of three components: (1) a seasonal component consisting of full-wave rectified sine waves (i.e., the absolute value of a sine wave) or triangle waves, (2) a trend component consisting of a linearly increasing or declining trend, and (3) a Gaussian noise component. Each time series is parameterized by the frequency $f$ of the seasonal component and the slope $k$ of the trend component.
As shown in Figure 2, $(f,k)$ are jointly sampled from a two-dimensional Gaussian distribution.
In Synth-TS, we construct a grid of such two-dimensional Gaussians and randomly generate multiple time series from each Gaussian.
Formally, the dataset Synth-TS$(m,n)$ generates $2n$ samples from each Gaussian in a $m\times m$ grid, with $n$ samples having a sine wave component and $n$ samples having a triangle wave component.

We define the pretext task as the binary classification of seasonal components (sine or triangle waves). The MSPL framework learns representations of the time series such that their pairwise distances match the external pairwise dissimilarity, which we defined as the Euclidean distance between their respective parameters $(f,k)$.
This encourages the time series to cluster according to the Gaussian distributions generating their parameters.
More details about the construction of Synth-TS can be found in Appendix A.

##### Proprietary dataset.

The proprietary dataset consists of $1862$ bacterial samples with MALDI spectra spanning $42$ species with corresponding WGS information from a single hospital. Though the raw WGS data were unavailable, we have access to the pairwise dissimilarity of this data, measured by SNP distance. In our framework, species identification from MALDI spectra forms the pretext classification task, and the SNP distances define the external dissimilarity matrix $\bm{d}$ (Eq. 7).

In bacterial outbreak investigations, SNP distances lower than a pre-defined threshold indicate closely related or nearly identical strains that may form an outbreak cluster (Guerra-Assunção et al., ; Hatherell et al., ). In our experiments, the threshold is set to $15$ as used by epidemiologists (Xiao et al., ).
To derive the ground truth outbreak clusters from the SNP distances,
we apply hierarchical clustering with complete linkage on the full SNP distance matrix, using $15$ as the distance threshold.

While small SNP distances are crucial to outbreak cluster detection, the distances can vary drastically in scale, ranging from less than $10$ to more than $10^{5}$.
Here, we develop a custom loss function for the structure preservation objective that avoids potential overfitting to large SNP distances:

$$ $\displaystyle\mathcal{L}_{\text{struct}}=\frac{1}{N^{2}}\sum_{i,j\in[N]^{2}}f_ {\text{SNP}}(\text{pdist}(\bm{h})_{ij},\bm{d}_{ij},t),$ (15) $\displaystyle f_{\text{SNP}}(x,y,t)=\begin{cases}(x-y)^{2}&y\leq t,\\ (\max\{0,t-x\})^{2}&y>t,\end{cases}$ (16) $$

where $\text{pdist}(\bm{h})$ and $\bm{d}$ are the feature distance and SNP distance matrices, respectively (Eq. 7), $N$ is the batch size, and $t$ is the chosen SNP threshold. Under this custom loss function, no penalty is imposed when both the feature distance and SNP distance exceed the SNP threshold, as they have no impact on the practice of outbreak detection.

##### DRIAMS data.

DRIAMS (Weis et al., ) is a public dataset with paired MALDI spectra and antimicrobial resistance (AMR) profiles. In our experiments, we use the DRIAMS-B and DRIAMS-C subsets, which consist of $10404$ bacterial samples with MALDI spectra spanning $251$ species. As above, we define species identification from MALDI spectra as the pretext classification task.

The AMR profile of each bacterial sample documents its resistance to various antibiotics. We utilize the AMR profiles against $33$ shared drugs recorded in both DRIAMS subsets. We preprocess the AMR profiles to construct the external dissimilarity matrix $\bm{d}$, where each entry is an integer ranging from 0 0 to $33$. Details about the AMR profile preprocessing can be found in Appendix B.

To derive ground truth outbreak clusters from the AMR dissimilarity matrix, we apply hierarchical clustering with varying distance thresholds from $1$ to $33$.
The optimal threshold—chosen to maximize the number of non-singleton clusters—is set to $10$ for ground truth generation.

## 5 Experiments and Results

In our experiments, we adopt two different clustering schemes for the features $\bm{h}$ learned from MSPL and onlyCLS.
The first scheme involves hierarchical clustering with a distance threshold, denoted by the subscript “thr.”
For the proprietary dataset and DRIAMS, we first determine the distance threshold that yields the optimal cluster F1 score on the training data.
Specifically, for the proprietary dataset, the upper bound of the thresholds is set to $20$, an alternative threshold used by epidemiologists for outbreak detection (Szarvas et al., ), which is close to the threshold we used to obtain groud-truth clusters.
For DRIAMS, the upper bound is set to $33$, the maximum dissimilarity between AMR profiles (see Appendix B).
The same threshold is then used for evaluation.
The second scheme also employs hierarchical clustering, but the number of output clusters is set to match that of the ground truth.
We denote this scheme by the subscript “num.”
For Synth-TS, only the second clustering scheme is used since the ground truth clusters are not derived from hierarchical clustering with a distance threshold.

For evaluation, we perform $2$-fold cross-validation on Synth-TS and the proprietary dataset and $5$-fold cross-validation for DRIAMS.
Each cross-validation experiment is repeated over $5$ random trials, using different random seeds for data splitting.
We first average the metrics across the validation folds, then report the mean and the $95\%$ confidence interval (using $t$-distribution) of the averaged metrics across the $5$ random trials.

**Table 1: Model performance on Synth-TS, the proprietary dataset, and DRIAMS.**
| Dataset | Model | ARI | NMI | Precision | Recall | F1 Score | Pretext Accuracy |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Synth-TS$(5,80)$ | $\text{MSPL}_{\text{num}}$ | $0.426\pm 0.011$ | $\mathbf{0.734\pm 0.003}$ | $0.609\pm 0.019$ | $0.611\pm 0.018$ | $0.61\pm 0.019$ | $0.984\pm 0.002$ |
| $\text{onlyCLS}_{\text{num}}$ | $0.175\pm 0.009$ | $0.581\pm 0.008$ | $0.509\pm 0.014$ | $0.584\pm 0.015$ | $0.543\pm 0.013$ | $0.986\pm 0.002$ |  |
| clusCLS | $\mathbf{0.462\pm 0.008}$ | $0.727\pm 0.003$ | $\mathbf{0.667\pm 0.01}$ | $\mathbf{0.662\pm 0.008}$ | $\mathbf{0.664\pm 0.009}$ | $0.976\pm 0.003$ |  |
| Synth-TS$(10,20)$ | $\text{MSPL}_{\text{num}}$ | $\mathbf{0.395\pm 0.007}$ | $\mathbf{0.819\pm 0.001}$ | $0.605\pm 0.005$ | $\mathbf{0.6\pm 0.005}$ | $\mathbf{0.602\pm 0.004}$ | $0.982\pm 0.002$ |
| $\text{onlyCLS}_{\text{num}}$ | $0.169\pm 0.019$ | $0.708\pm 0.012$ | $\mathbf{0.609\pm 0.024}$ | $0.515\pm 0.023$ | $0.558\pm 0.021$ | $0.987\pm 0.002$ |  |
| clusCLS | $0.32\pm 0.014$ | $0.78\pm 0.005$ | $0.561\pm 0.007$ | $0.556\pm 0.008$ | $0.559\pm 0.007$ | $0.982\pm 0.002$ |  |
| Synth-TS$(16,10)$ | $\text{MSPL}_{\text{num}}$ | $\mathbf{0.386\pm 0.005}$ | $\mathbf{0.869\pm 0.001}$ | $0.635\pm 0.007$ | $\mathbf{0.631\pm 0.005}$ | $\mathbf{0.633\pm 0.006}$ | $0.988\pm 0.002$ |
| $\text{onlyCLS}_{\text{num}}$ | $0.179\pm 0.018$ | $0.798\pm 0.007$ | $\mathbf{0.668\pm 0.011}$ | $0.541\pm 0.009$ | $0.598\pm 0.008$ | $0.991\pm 0.001$ |  |
| clusCLS | $0.234\pm 0.004$ | $0.821\pm 0.001$ | $0.552\pm 0.006$ | $0.53\pm 0.006$ | $0.541\pm 0.006$ | $0.979\pm 0.003$ |  |
| Proprietary Dataset | $\text{MSPL}_{\text{thr}}$ | $0.001\pm 0.002$ | $0.137\pm 0.04$ | $0.962\pm 0.021$ | $\mathbf{0.962\pm 0.009}$ | $\mathbf{0.962\pm 0.011}$ | $0.813\pm 0.029$ |
| $\text{MSPL}_{\text{num}}$ | $0.034\pm 0.001$ | $0.884\pm 0.001$ | $0.935\pm 0.007$ | $0.605\pm 0.009$ | $0.734\pm 0.008$ |  |  |
| $\text{onlyCLS}_{\text{thr}}$ | $0.0\textperiodcentered\pm 0.004$ | $\mathbf{0.897\pm 0.01}$ | $\mathbf{0.986\pm 0.004}$ | $0.525\pm 0.012$ | $0.685\pm 0.009$ | $0.864\pm 0.007$ |  |
| $\text{onlyCLS}_{\text{num}}$ | $0.030\pm 0.003$ | $0.88\pm 0.001$ | $0.937\pm 0.007$ | $0.584\pm 0.004$ | $0.719\pm 0.004$ |  |  |
| clusCLS | $\mathbf{0.437\pm 0.049}$ | $0.759\pm 0.015$ | $0.722\pm 0.018$ | $0.621\pm 0.016$ | $0.667\pm 0.006$ | $0.804\pm 0.003$ |  |
| DRIAMS | $\text{MSPL}_{\text{thr}}$ | $0.207\pm 0.074$ | $0.54\pm 0.058$ | $\mathbf{0.984\pm 0.004}$ | $\mathbf{0.896\pm 0.016}$ | $\mathbf{0.937\pm 0.01}$ | $0.912\pm 0.003$ |
| $\text{MSPL}_{\text{num}}$ | $0.005\pm 0.0001$ | $\mathbf{0.736\pm 0.001}$ | $0.966\pm 0.001$ | $0.31\pm 0.022$ | $0.468\pm 0.025$ |  |  |
| $\text{onlyCLS}_{\text{thr}}$ | $0.130\pm 0.028$ | $0.665\pm 0.018$ | $0.976\pm 0.004$ | $0.716\pm 0.072$ | $0.818\pm 0.053$ | $0.966\pm 0.001$ |  |
| $\text{onlyCLS}_{\text{num}}$ | $0.005\pm 0.001$ | $0.729\pm 0.0005$ | $0.970\pm 0.002$ | $0.302\pm 0.012$ | $0.46\pm 0.013$ |  |  |
| clusCLS | $\mathbf{0.574\pm 0.026}$ | $0.647\pm 0.024$ | $0.819\pm 0.017$ | $0.837\pm 0.011$ | $0.826\pm 0.007$ | $0.953\pm 0.001$ |  |

### Observation 1: MSPL effectively preserves external structure

##### Uncovering latent structure in Synth-TS.

We evaluate our models on three versions of the Synth-TS dataset: Synth-TS$(5,80)$, Synth-TS$(10,20)$, and Synth-TS$(16,10)$, containing $80$, $20$, and $10$ samples per type of the seasonal component (sine or triangle) per Gaussian, respectively.

As shown in Table 1, while MSPL falls short of clusCLS in terms of F1 score and ARI on Synth-TS$(5,80)$, it significantly outperforms clusCLS across all clustering metrics on Synth-TS$(10,20)$ and Synth-TS$(16,10)$, suggesting that MSPL has a marked advantage over the classification approach on sparser datasets.

Furthermore, for the clusCLS model, all reported clustering metrics–—except NMI—–exhibit a clear downward trend as the number of samples per class decreases. This is expected, as fewer samples make it more challenging for the cluster label classifier to accurately capture class characteristics.
In contrast, the metrics for MSPL remain consistent despite the decreasing number of samples per Gaussian, suggesting that MSPL is more effective at preserving external clustering structures and is robust to cluster sparsity.

Additionally, MSPL consistently outperforms onlyCLS in ARI, NMI, cluster recall, and cluster F1 score, underscoring the importance of dissimilarity matching in $\mathcal{L}_{\text{struct}}$ for structure preservation.

##### Recovering WGS clusters in the proprietary dataset.

Figure: Figure 3: Bipartite graph of Klebsiella pneumoniae clusters.
Refer to caption: x3.png

We first observe that when the number of predicted clusters is constrained to match the ground truth, $\text{MSPL}_{\text{num}}$ outperforms the other models in both NMI and cluster F1 score. Furthermore, $\text{MSPL}_{\text{thr}}$ vastly outperforms both $\text{onlyCLS}_{\text{thr}}$ and clusCLS in terms of cluster recall and F1 score. This demonstrates the superior ability of MSPL to preserve structure in real-world settings, effectively mimicking the decision-making criterion of epidemiologists during outbreak investigations.

Figure 3 presents the bipartite graph showing the correspondence between ground truth WGS clusters and predicted clusters for one of the species, Klebsiella pneumoniae, with links indicating matched clusters with at least two samples. Our method manages to group data points of the same ground truth cluster together (high recall), though the precision is lower given the two large MALDI clusters, instead of eight smaller WGS clusters.

##### Recovering AMR clusters in DRIAMS.

As shown in Table 1, MSPL outperforms all baseline models in precision, recall, and F1 score.
Additionally, when we constrain the number of output clusters to match the ground truth,
$\text{MSPL}_{\text{num}}$ outperforms other models in NMI. Figure 4 illustrates the ground truth and predicted clusters of data points projected through multidimensional scaling of the predicted dissimilarity matrix ($\text{pdist}(\bm{h})$).
These findings further validate MSPL’s ability to learn representations that preserve external structure, effectively bridging different modalities.

Figure: Figure 4: Multidimensional scaling projection based on predicted distance matrix of DRIAMS data using MSPL model, colored in ground truth clusters (left) and predicted clusters (right) respectively. MSPL recovered most of the clusters in ground truth.
Refer to caption: x4.png

### Observation 2: MSPL thrives in diverse data subsets

Figure: Figure 5: Lift in F1 score for different species in the proprietary dataset and DRIAMS, sorted by the entropy of ground truth clusters or the pretext accuracy. The dot size reflects the number of MALDI samples in that species. Species label descriptions are listed in Appendix C.
Refer to caption: x6.png

After evaluating the model performance on the full datasets, we now investigate the impact of data substructure on MSPL’s performance. In both the proprietary dataset and DRIAMS, the ground truth clusters assigned to MALDI samples within a single bacterial species naturally represent such substructures.
For each species-defined subset, we calculate the lift in cluster F1-score between MSPL and the two baseline models.
The lift is defined as the ratio of MSPL’s cluster F1-score to that of clusCLS or onlyCLS.
To characterize the substructure, we measure the “diversity” of each subset using the mean Shannon entropy of the ground truth cluster assignment on its samples across the validation sets.

We then plot the lift against the Shannon entropy for all species in both the proprietary dataset and DRIAMS.
We find that MSPL achieves outperforms onlyCLS (i.e., $\text{lift}>1$) in all subsets of the proprietary dataset except Stenotrophomonas maltophilia (STEN) (Figure 5(a)) and in the majority of subsets in DRIAMS (Appendix D).
This result underscores the importance of the dissimilarity matching to MSPL’s ability to preserve structure across different subsets of the data.

On the other hand, when compared to clusCLS, MSPL achieves higher lift for species with greater Shannon entropy in both datasets (Figure 5(c,e)).
This finding suggests that, compared to the classification approach, MSPL excels when the data exhibits high diversity in cluster distribution.

### Observation 3: MSPL is robust to the difficulty of its pretext task

We further investigate whether the difficulty of the pretext task affects structure preservation in MSPL.
Again, we evaluate model performance on the species-defined subsets of both the proprietary dataset and DRIAMS.

To quantify the difficulty of the pretext task for each species, we calculate the mean accuracy of predicting that species across the validation sets.
As illustrated in Figure 5(b,d), the relationship between MSPL and baseline F1 scores in the proprietary dataset is agnostic to pretext task accuracy.
Notably, the lift can exceed $1$ for species with both high and low pretext task accuracies.
A similar pattern is observed in DRIAMS (Figure 5(f) and Appendix D). These results suggest that the difficulty of the pretext task does not limit MSPL’s ability to preserve structure.

## 6 Discussion

We introduced Multimodal Structure Preservation Learning (MSPL), a method designed to enhance the utility of a data modality by learning representations that preserve external clustering structures from another modality.
Through empirical evaluation, we demonstrate that MSPL can effectively capture and preserve latent structures in synthetic time series, whole genome sequencing (WGS), and antimicrobial resistance (AMR) data.
Additionally, MSPL offers two advantages: it excels in preserving highly diverse substructures and remains robust to the difficulty of the pretext task. Our approach is a novel addition to the family of multimodal machine learning techniques in that it aggregates information across different forms of its representation, even though the underlying form of data may originally be transactional in both sources. It can be useful in applications where a pattern structure present in some data sources can support models trained on other data sources.

Figure: Figure 6: Distributions of clusters on the proprietary dataset. There are $544$ ground truth clusters while $\text{MSPL}_{\text{thr}}$ predicted only $38$ clusters.
Refer to caption: x7.png

##### Limitations.

In our performance evaluation (Table 1), we observed that MSPL yields low NMI scores on both the proprietary dataset and DRIAMS when clustering the learned features using a distance threshold ($\text{MSPL}_{\text{thr}}$).
However, when constraining the number of clusters ($\text{MSPL}_{\text{num}}$), the NMI score improves and surpasses the baseline, albeit resulting in a drop in the F1 score.
Additionally, for both clustering schemes, the ARI remains consistently low. While NMI can be inadequate when comparing predictions to a large number of clusters (Amelio & Pizzuti, ), and ARI is more suitable for comparisons involving large, equally sized clusters (Romano et al., ), the low NMI and ARI scores indicate a potential limitation of the current MSPL framework in handling imbalanced cluster distributions, as observed in the proprietary dataset (Figure 6). Moreover, in real-world clinical applications (e.g., outbreak detection), the number of clusters is often unknown, making it infeasible to apply the $\text{MSPL}_{\text{num}}$ model.

##### Future work.

Given the limitations of the current MSPL framework, future research will focus on improving it in the face of cluster imbalance.
Specifically, we plan to enhance the structure preservation objective
($\mathcal{L}_{\text{struct}}$) by incorporating additional supervision that encourages MSPL to accurately estimate the number of ground truth clusters as well as their distribution.
We will also experiment with more than one source of structural information (e.g., including WGS and AMR simultaneously in support of learning on MALDI data), and consider other than clustering structures as sources of predictive information for associated tasks.

#### Acknowledgments

This study was partially funded by the National Institute of Allergy and Infectious Diseases (R01AI127472) and the National Science Foundation award (2406231). The authors thank Nicholas Gisolfi and Jiayi Li for helpful discussions.

## Appendix A Constructing Synth-TS

##### Gaussian grid.

We first construct an $N_{\mu}\times N_{\mu}$ grid to define the means of the Gaussians from which we sample $(f,k)$.
For simplicity, we assume the two dimensions of the Gaussians are independent, with standard deviations $\sigma_{f}$ and $\sigma_{k}$ for all Gaussians.

The mean frequency and slopes are defined as follows:

$$ $\displaystyle\mu_{f}\in[\mu_{0},\mu_{0}+2\sqrt{2}\sigma_{f},\cdots,\mu_{0}+2(N _{\mu}-1)\sqrt{2}\sigma_{f}],$ (17) $\displaystyle\mu_{k}\in[-\sqrt{2}N_{\mu}\sigma_{k},(-N_{\mu}+2)\sqrt{2}\sigma_ {k},\cdots,(-N_{\mu}+2(N_{\mu}-1))\sqrt{2}\sigma_{k}].$ (18) $$

In the frequency direction, the distance between two points sampled from the same Gaussian follows a normal distribution $\mathcal{N}(0,2\sigma^{2}_{f})$. Conversely, for two points sampled from distinct Gaussians, their distance follows a distribution $\mathcal{N}(2\sqrt{2}k\sigma_{f},2\sigma^{2}_{f})$ for $k\geq 1$. The results for the slope direction are analogous. In our experiments, we set $\mu_{0}=0.2$, $\sigma_{f}=0.4$, and $\sigma_{k}=0.5$.

##### Seasonal component.

From each Gaussian in the grid, we generate time series of duration $2$ seconds and sampling rate $256$Hz. The sine waves and triangle waves are defined as follows:

$$ $\displaystyle\text{Sine}(f,t)=|\sin(\pi ft)|+b,$ (19) $\displaystyle\text{Triangle}(f,t)=\begin{cases}4fx-1+b&0\leq x<\frac{1}{2},\\ -4fx+2+b&\frac{1}{2}\leq x<1,\\ \end{cases}$ (20) $$

where $x=tf-\lfloor tf\rfloor$ and the offset $b$ is set to $1$.

##### Trend component.

The trend component, parameterized by $k$, is defined as follows:

$$ $\text{Trend}(k,t)=\begin{cases}kt&k\leq 0,\\ k(t-\max(t))&k<0.\end{cases}$ (21) $$

##### Gaussian noise.

For each sampled data point on the time series, we added a Gaussian noise component: $\textit{Noise}\sim\mathcal{N}(0,\sigma_{n}^{2})$. We set $\sigma_{n}=0.02$ in our experiments.

## Appendix B AMR profile Preprocessing in DRIAMS

AMR profiles The DRIAMS dataset uses a mixed labeling scheme to represent AMR.
Each AMR label can be one of ‘S’(susceptible), ‘R’(resistant), ‘I’(Intermediate), ‘1’(susceptible or intermediate), ‘0’(susceptible), or unknown (we labeled as ‘N’).

We first build the following similarity lookup table between labels:

**Table 2: Similarity between AMR labels in DRIAMS.**
|  | S | I | R | 1 | 0 | N |
| --- | --- | --- | --- | --- | --- | --- |
| S | 1 | 0 | 0 | 0 | 1 | 0 |
| I | 0 | 1 | 0 | 1 | 0 | 0 |
| R | 0 | 0 | 1 | 1 | 0 | 0 |
| 1 | 0 | 1 | 1 | 1 | 0 | 0 |
| 0 | 1 | 0 | 0 | 0 | 1 | 0 |
| N | 0 | 0 | 0 | 0 | 0 | 0 |

For a pair of AMR profiles, we look up and sum the similarity for each pair of AMR labels to get the similarity between the AMR profiles. For example, profiles [‘1’, ‘S’, ‘N’] and [‘I’, ‘0’, ‘R’] will have similarity $1+1+0=2$. The dissimilarity between the two profiles is calculated by subtracting the similarity from the length of the AMR profiles, i.e., the number of drugs.

## Appendix C Species Label Descriptions of the Proprietary Dataset

**Table 3: Reference table for species appearing in Figure 5.**
| Label | Species |
| --- | --- |
| ACIN | Acinetobacter baumannii |
| BC | Burkholderia cepaciae |
| CB | Citrobacter |
| EB | Enterobacter cloacae |
| EC | Escherichia coli |
| KLO | Klebsiella oxytoca |
| KLP | Klebsiella pneumoniae |
| MRSA | Staphylococcus aureus |
| PR | Proteus mirabilis |
| PRV | Providencia |
| PSA | Pseudomonas aeruginosa |
| PSB | Pseudomonas |
| SER | Serratia marcescens |
| STEN | Stenotrophomonas maltophilia |
| VRE | Vancomycin-resistant Enterococcus |

## Appendix D Lift in F1 score for MSPL vs onlyCLS on DRIAMS

Figure: Figure 7: Lift in F1 score between MSPL and onlyCLS for different species on DRIAMS, sorted by the entropy of ground truth clusters or the pretext accuracy. The dot size reflects the number of MALDI samples in that species.
Refer to caption: x8.png

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: See above.
soundness: 2
strengths: I think the paper has several strenghs: 
1: It presents a flexible framework that can incorporate different modality as inputs, incorporating various loss functions and clustering objectives
2: It addresses a real-world problem in epidemiology (using MALDI data as a cost-effective alternative to WGS)
3: It introduces a new cluster evaluation metric (cluster F1 score)
summary: This paper presents a multimodal framework called MSPL, which builds upon encoder decoder structure with extra regularizations form prediction task and structure loss.
weaknesses: This paper has several areas that can be improved:
1: lt could benefit from more extensive comparison with other multimodal learning approaches
2: Authors could explore more sophisticated structure preservation objectives The three losses are common objective functions in multimodal and VE/VAE variants. Besides, there is limited discussion of the impact of different encoder architectures
3: Model needs further optimization. Even comparing with its own variants, the proposed model cannot outperform them in most cases. 
4: I am not sure if it can handle a large number of clusters or clusters with imbalanced sizes.

---

confidence: 4
contribution: 1
initial_rating: 1
presentation: 3
questions: The above comments raise some broader questions regarding the method and particularly the evaluation approach. Some more specific questions are listed below:
 
- Regarding the choice of the custom loss function for SNP distance, the choice to impose no penalty when feature distance and SNP distance both exceed 15 seems confusing. The cited source suggests that a distance <= 5 indicates a definite transmission  <= 15 indicates probable transmission. When clustering this data, might it also be useful to have a representation that accurately captures the relationships between samples that are even somewhat less likely to cluster together? If so, it could make sense to either relax this constraint or try a different normalization approach (such as log transforming the SNP distances).

- The model requires the choice of a pretext task, and the authors suggest that the difficulty of the pretext task does not affect MSPL’s ability to preserve structure. What, then, is the effect of the pretext task on the learned representation, and how should a user choose the pretext task for their particular application?  
 
Some minor comments on Figure 5 that did not affect my score:
-  e) and f) are missing species labels.
-  The paper claims that F1 lift and species diversity are correlated based on c) and e) – a regression line or correlation statistic would be helpful to back up this claim.
soundness: 1
strengths: The concept of preserving structure level alignment without need for the entire dataset is interesting, and the proposed approach appears to be novel. The application of multimodal deep representation learning approaches of this kind to mass spectrometry data in the context of epidemiology is particularly original and exciting.
 
The method is very clearly described, as is the evaluation approach and the metrics used. In evaluating, the authors considered extrinsic clustering metrics that went beyond more common approaches such as ARI and NMI, which greatly assist in the interpretation of the results.
 
Additionally, the authors evaluate the method on multiple datasets, including a variety of simulations and two real-world datasets, which are well-described.
summary: This paper presents an approach to multimodal representation learning that leverages an autoencoder along with a combination of 3 loss functions (reconstruction, pretext task performance and structural alignment) in order to learn representations of the data that preserve the structure in one of the modalities (represented by a dissimilarity matrix) without requiring the raw data itself.
 
The authors apply the method in the context of epidemiology, where mass spectrometry data is becoming a potentially valuable tool for outbreak detection but is limited in power compared to whole genome sequencing, which can be a prohibitively labor-intensive and costly approach. They present the method as a way of integrating these modalities. The method is evaluated on a simulated dataset, a public dataset of paired MALDI spectra and antibiotic resistance data, and a proprietary dataset of WGS structural data and MALDI spectra. To evaluate, the authors compare their proposed method (MSPL) to two baseline methods that they construct without the structure alignment loss function, and evaluate clusterings based on the resulting representations using a variety of extrinsic clustering metrics with respect to ground truth.
weaknesses: The paper has several significant weaknesses. 

First, the significance of the method’s real-world impact in the application area is somewhat unclear. The introduction states that the main utility of the learned representation in this context is that it could replace WGS in practice as a more cost-effective alternative; however, the method seems to require SNP distances between each pair of samples (and thus WGS for every sample) as an input in order to learn the representation. As such, it is not clear how such representations would be learned without doing WGS first – thus incurring the same costs as would be necessary to do outbreak detection in the usual way. This somewhat reduces the perceived contribution of the work. 

The evaluation approach is also a major weakness of the paper. The performance of the model is poor in many cases, and the proposed metrics make it very difficult to understand why. Cluster purity, precision, recall and F1 scores for clustering have already been defined in existing literature  – see the chapter on “Evaluation of Clustering” in Information Retrieval by Manning. In order to deal with the challenge of comparing clusters of different sizes and number, precision, recall and F1 score are typically defined with respect to the cluster memberships of sample pairs. However, the paper defines these metrics very differently: with respect to purity, which is easy to achieve when cluster sizes are large, and makes the results very difficult to interpret. For example, while the F1 scores seem generally high, they appear to be driven predominantly by a sharp increase in recall. Figure 6 demonstrates that MSPL learns many fewer clusters than the ground truth – if MSPL is also learning fewer or larger clusters than the baseline models, then this could easily explain the increase in the purity-based recall metric. Although the purity-based precision metric decreases in these cases, it could also be artificially inflated or otherwise biased by cluster size or distribution. Unfortunately, the number of clusters learned in each experiment is not reported, which makes evaluation even more difficult. The NMI and ARI metrics are designed to account for these potential sources of bias, but the authors were not able to demonstrate that MSPL consistently outperforms the baselines according to these metrics in real-world data. Overall, the evaluation approach should be reformulated to be consistent with the literature and the results require much more investigation.
 
The choice of baselines is also a substantially limiting factor. While the authors construct two baselines, the paper does not make any comparison of MSPL to existing methods. While relevant deep learning approaches may be limited, there are many papers on late integration multi-view clustering approaches, which integrate multiple modalities using only clustering labels or dissimilarity matrices and not the original data (see, e.g. “Multi-omic and multi-view clustering algorithms: review and cancer benchmark” by Rappaport et al for a brief review of such approaches). Since the evaluation in the paper is based entirely on the quality of clustering based on the learned representation, this class of methods seem very relevant. Furthermore, there is no attempt to evaluate how well the model performs in comparison to models that leverage the entire dataset rather than just the distance-based structure. This makes it difficult to evaluate the real-world utility of the method.
 
Finally, there is no reproducibility statement and no mention of code or data being made available.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: How does the "seasonal and trend components" in the synthetic datasets related to MALDI-WGS matching?
soundness: 2
strengths: The stated problem is pervasive in biomedical applications and is challenging.
summary: The paper proposed a domain adaptation method that learns the data distribution structure in one modal and transfer it to the other modal. They applied it to the problem of hospital outbreak detection that using MALDI and whole genome sequencing.
weaknesses: 1) This is a typical subset of domain adaptation problems. However, they did not include SOTA domain adaptation methods into the baseline. The baseline methods are weak.
2) Also, from references we see that there are already methods that perform prediction tasks directly based on MALDI, which were not compared.
3) The experiments are carried out only on MALDI-WGS datasets and most are synthetic datasets. Due to the small-sample nature of these problems, the models are vulnerable to short-cut learning and testing on several similar datasets is not reliable. I don't see any reason that the problem should be restricted on MALDI-WGS data. There are lots of two-domain problems with similar character in biomedical fields and the data should be tested on more types of applications.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: It would be good to further discuss how to suitable tune the contribution of each loss term.

How is the approach influenced by initialization conditions?

How does architectural choices influence the model and why is UNETs chosen as the backbone as opposed to other architectures such as transformer based architectures?

Why is the approach not compared to any existing SOTA approaches within the domain or similar domains for instance based on the approaches reviewed in related work?

The results are also not that surprising in that regularizing towards a clustering structure will enhance such learning of the clustering structure. It would in this context be interesting to see if the regularization also improves upon the pretext class and contrast this to other methodologies directly learning the pretext class.
soundness: 3
strengths: The approach is useful and enable to integrate information of multiple (two) modalities taking overall structural information into account from the opposing modality.

The considered problem domain is interesting and the approach’s seem to enhance the learned representations in terms of cluster level structures.

The paper is well written and easy to follow.
summary: The authors propose the Multimodal Structure Preservation Learning (MSPL) approach that learns data representations utilizing clustering structure in one data modality to inform upon the other modality using a regularization approach towards compliance of this clustering structure when learning representations. The approach is applied to synthetic as well as whole genome sequencing (WGS) and antimicrobial resistance datasets. Rather than learning a shared feature space the approach thus relies on gross structural information at the level of groups exploring alignment according to the dissimilarity-based clustering learned by the opposing modality. The approach relies on three tasks, an autoencoder for learning representations, a pretext discriminatory task, and  alignment of the two modalities clustering structure formulated as a multiobjective function reflected in three loss terms with associated relative weights. Apart from conventional ARI and NMI cluster validity metrics the authors further propose a cluster-based F1 score. The approach is compared against two model ablations (baselines) not having the structure preserving loss and classifying the cluster groups respectively as opposed to operating on dissimilarities.
weaknesses: The methodological contribution of the paper is very limited and rather straightforward combining three loss components. As such, the contribution seems rather incremental and limited in scope. 

The contribution of the F1 metric is also straightforward and does not contribute much in terms of novelty.

The comparisons are very limited only considering simple model ablations but not any alternative state-of-the-art methodology for the same problem domain. 

The results are not overly convincing with the approach working better than baselines in some situations and not in other.

Overall I find the contribution of limited novelty and the experimentation not overly convincing - and therefore do not recommend publication at this point.

Meta-review: This paper proposes a multimodal representation learning method called MSPL, which leverages an autoencoder with three loss functions: reconstruction, pretext task performance, and structural alignment. MSPL learns to preserve the structure of one modality, represented as a dissimilarity matrix, without requiring raw data. The method is applied to hospital outbreak detection using MALDI spectra and whole genome sequencing (WGS) data, addressing the limitations of WGS in cost and labor. It is evaluated on simulated, public, and proprietary datasets, comparing MSPL to baselines without structural alignment loss, using extrinsic clustering metrics to assess performance.

As the reviewers pointed out, there is significant room for improvement in the paper, including hyperparameter tuning and comparison to existing methods. Therefore, it is difficult to accept the paper in its current form. I encourage the authors to revise the paper based on the reviewers' comments and resubmit it to a future venue.

Decision: Reject