Multimodal Fusion with Relational Learning for Molecular Property Prediction

## Abstract

Abstract Graph-based molecular representation learning is essential for accurately predicting molecular properties in drug discovery and materials science; however, it faces significant challenges due to the intricate relationships among molecules and the limited chemical knowledge utilized during training. While contrastive learning is often employed to handle molecular relationships, its reliance on binary metrics is insufficient for capturing the complexity of these interactions. Multimodal fusion has gained attention for property reasoning, but previous work has explored only a limited range of modalities, and the optimal stages for fusing different modalities in molecular property tasks remain underexplored. In this paper, we introduce MMFRL (Multimodal Fusion with Relational Learning for Molecular Property Prediction), a novel framework designed to overcome these limitations. Our method enhances embedding initialization through multi-modal pre-training using relational learning. We also conduct a systematic investigation into the impact of modality fusion at different stages—early, intermediate, and late—highlighting their advantages and shortcomings. Extensive experiments on MoleculeNet benchmarks demonstrate that MMFRL significantly outperforms existing methods. Furthermore, MMFRL enables task-specific optimizations. Additionally, the explainability of MMFRL provides valuable chemical insights, emphasizing its potential to enhance real-world drug discovery applications.

## 1 Introduction

Graph representation learning for molecules has gained significant attention in drug discovery and materials science, as it effectively encapsulates molecular structures and enables the effective investigation of structure-activity relationships (Wieder et al., ; Zhang et al., ; Fang et al., ; Wang et al., ). In this paradigm, atoms are treated as nodes and chemical bonds as edges, effectively encapsulating the connectivities that define molecular behavior. However, it poses significant challenges
due to intricate relationships among molecules and the limited chemical knowledge utilized during training.

Figure: Figure 1: Multimodal Fusion with Relational Learning for Molecular Property Prediction (MMFRL). This figure shows our proposed idea about how to transfer the knowledge from other modalities and use fusion to improve the performance further. Unlike the general contrastive learning framework shown in Appendix Figure A.2, MMFRL doesn’t need to define positive or negative pairs and is capable of learning continuous ordering from target similarity.
Refer to caption: extracted/5930035/Figures/Picture1.png

Often, contrastive learning (CL) is employed to study relationships among molecules, but it relies on binary metrics of positive and negative pairs, and tends to oversimplify complex molecular interactions. For example, consider Thalidomide: while the (R)- and (S)-enantiomers share the same topological graph and differ only at a single chiral center, their biological activities are drastically different—the (R)-enantiomer is effective in treating morning sickness, whereas the (S)-enantiomer causes severe birth defects. In other words, the (R)- and (S)-enantiomers are similar in terms of topological stucture but dissimilar in terms of biological activities. Thus, a more sophisticated approach is required to tackle these scenarios. A potential solution would be to use continuous metrics within a multi-view space, enabling a more comprehensive understanding of these complex molecular relationships.

When it comes to multimodal learning for molecules, we often encounter data availability and incompleteness issues. This raises a critical question: how can multimodal information be effectively leveraged for molecular property reasoning when such data is absent in downstream tasks? Recent studies have demonstrated the effectiveness of pretraining molecular Graph Neural Networks (GNNs) by integrating additional knowledge sources (Wang et al., ; ; Liu et al., ; Xu et al., ). Building on this foundation, a promising solution is to pretrain multiple replicas of molecular GNNs, with each replica dedicated to learning from a specific modality. This approach allows downstream tasks to benefit from multimodal data that is not accessible during fine-tuning, ultimately improving representation learning.

Facing these challenges and opportunities, we propose MMFRL (Multimodal Fusion
with Relational Learning for Molecular Property Prediction), a novel framework
features relational learning (RL) and multimodal fusion (MMF). RL utilizes a continuous relation metric to evaluate relationships among instances in the feature space  (Balcan & Blum, ; Wen et al., ). Our major contribution comprises three aspects:
Conceptually: We introduce a modified relational learning metric for molecular graph representation that offers a more comprehensive and continuous perspective on inter-instance relations, effectively capturing both localized and global relationships among instances. To the best of our knowledge, this is the first work to demonstrate such generalized relational learning metric for molecular graph representation.
Methodologically:
Our proposed modified relational metric captures complex relationships by converting pairwise self-similarity into relative similarity, which evaluates how the similarity between two elements compares to the similarity of other pairs in the dataset. In addition, we integrate these metrics into a fused multimodal representation, which has the potential to enhance performance, allowing downstream tasks to leverage modalities that are not directly accessible during fine-tuning.
Empirically:
MMFRL excels in various downstream tasks for Molecular Property Predictions. Last but not least, we demonstrate the explainability of the learned representations through two post-hoc analysis. Notably, we explore minimum positive subgraphs and maximum common subgraphs to gain insights for further drug molecule design.

## 2 Preliminaries

Directed Message Passing Neural Network (DMPNN).
The Message Passing Neural Network (MPNN)  (Gilmer et al., ) is a GNN model that processes an undirected graph $G$ with node (atom) features $x_{v}$ and edge (chemical bond) features $e_{vw}$. It operates through two distinct phases: a message passing phase, facilitating information transmission across the molecule to construct a neural representation, and a readout phase, utilizing the final representation to make predictions regarding properties of interest. The primary distinction between DMPNN and a generic MPNN lies in the message passing phase. While MPNN uses messages associated with nodes, DMPNN crucially differs by employing messages associated with directed edges  (Yang et al., ). This design choice is motivated by the necessity to prevent totters  (Mahé et al., ), eliminating messages passed along paths of the form $v_{1}v_{2}\dots v_{n}$, where $v_{i}=v_{i+2}$ for some $i$, thereby eliminating unnecessary loops in the message passing trajectory.

Relational Learning.
Original Relation Learning  (Zheng et al., ) ensures that different augmented views of the same instance from computer vision tasks share similar features, while allowing for some variability. This approach captures the essential characteristics of the instance, promoting consistency across the views without requiring them to be identical. By doing so, it enhances the model’s ability to generalize and recognize underlying patterns in the data. Suppose $z_{i}$ is the original embdding for the $i-th$ instance. Then $z_{i}^{1}$ is the embedding of first augmented view for $z_{i}$, and $z_{i}^{2}$ is the embedding of second augmented view for $z_{i}$. In this case, the Loss of Relational Learning (RL) is formulated as following:

$$ $s_{ik}^{1}=\frac{\mathbbm{1}_{i\neq k}\cdot\exp(z_{i}^{1}\cdot z_{k}^{2}/\tau) }{\sum_{j=1}^{N}\mathbbm{1}_{i\neq j}\cdot\exp(z_{i}^{1}\cdot z_{j}^{2}/\tau)}$ $$

$$ $s_{ik}^{2}=\frac{\mathbbm{1}_{i\neq k}\cdot\exp(z_{i}^{2}\cdot z_{k}^{2}/\tau_ {m})}{\sum_{j=1}^{N}\mathbbm{1}_{i\neq j}\cdot\exp(z_{i}^{2}\cdot z_{j}^{2}/ \tau_{m})}$ $$

$$ $L_{RL}=-\frac{1}{N}\sum_{i=1}^{N}\sum_{\begin{subarray}{c}k=1\\ k\neq i\end{subarray}}^{N}s_{ik}^{2}\log(s_{ik}^{1}).$ $$

Multi-Modality Fusion.
Multi-Modality Fusion combines diverse heterogeneous data (e.g. text, images, graph) to create a more comprehensive understanding of complex scenarios  (Lahat et al., ; Khaleghi et al., ; Poria et al., ; Ramachandram & Taylor, ; Pawłowski et al., ). This approach leverages the strengths of each modality, potentially improving performance in tasks like sentiment analysis or medical diagnosis. While challenging to implement due to the need to align different data streams, successful fusion can provide insights beyond what’s possible with single modalities, advancing AI and data-driven decision-making. In particular, the way to fuse different modality should also depends on the dominace of each unimodality  (Pawłowski et al., ).

## 3 Methods

We first explain our proposed modified metric in relational learning to facilitate smooth alignment between graph and referred unimodality. Then, we introduce approaches for integrating multi modalities at different stages of the learning process.

### 3.1 Modified Relational Learning in pretraining

We propose a modified relational metric by adapting the softmax function as a pairwise weighting mechanism. Let $|\mathcal{S}|$ denote the size of the instance set. The variable ${s}_{i,j}$ represents the learned similarity distribution where $z_{i}$ is the embedding to be trained. On the other hand, ${t}_{i,j}^{R}$ defines the target similarity distribution that captures the relationship between the pair of instances in the given space or modality $R$, where $z_{i}^{R}$ is a fixed embedding. The detailed formulation for the Loss of Modified Relatioal Learning (MRL) is provided below:

$$ $s_{i,j}=\frac{\exp(sim(z_{i},z_{j}))}{\sum_{k=1}^{|\mathcal{S}|}\exp(sim(z_{i} ,z_{k}))}$ (1) $$

$$ $t_{i,j}^{R}=\frac{\exp(sim(z_{i}^{R},z_{j}^{R}))}{\sum_{j=1}^{|\mathcal{S}|} \exp(sim(z_{i}^{R},z_{k}^{R}))}$ (2) $$

$$ $L_{MRL}=-\frac{1}{|\mathcal{S}|}\sum_{i=1}^{|\mathcal{S}|}\sum_{\begin{ subarray}{c}j=1\end{subarray}}^{|\mathcal{S}|}t_{i,j}^{R}\log(s_{i,j}).$ (3) $$

Notably, unlike other similarity learning approaches  (Wang et al., ; Zhang et al., ), our method does not rely on the categorization of negative and positive pairs for the pair weighting function. Additionally, the use of the softmax function ensures that the generalized target similarity $t_{i,j}$ adheres to the principles of convergence
as following:

###### Theorem 3.1 (Convergence of Modified Relational Learning Metric) .

Let $\mathcal{S}$ be a set of instances with size of $|\mathcal{S}|$, and let $\mathcal{P}$ represent the learnable latent representations of instances in $\mathcal{S}$ such that $|\mathcal{P}|=|\mathcal{S}|$. For any two instances $i,j\in\mathcal{S}$, their respective latent representations are denoted by $\mathcal{P}_{i}$ and $\mathcal{P}_{j}$. Let $t_{i,j}$ represent the target similarity between instances $i$ and $j$ in a given domain, and let $d_{i,j}$ be the similarity between $\mathcal{P}_{i}$ and $\mathcal{P}_{j}$ in the latent space. If $t_{i,j}$ is non-negative and $\{t_{i,j}\}$ satisfies the constraint $\sum_{j=1}^{|\mathcal{S}|}t_{i,j}=1$, consider the loss function for an instance $i$ defined as follows:

$$ $L(i)=-\sum_{j=1}^{|\mathcal{S}|}t_{i,j}\log\left(\frac{e^{d_{i,j}}}{\sum_{k=1} ^{|\mathcal{S}|}e^{d_{i,k}}}\right)$ (4) $$

then when it reaches ideal optimum, the relationship between $t_{i,j}$ and $d_{i,j}$ satisfies:

$$ $\text{softmax}(d_{i,j})=t_{i,j}$ (5) $$

For detailed proof, please refer to Appendix Section  B.1.

###### Theorem 3.1 (Convergence of Modified Relational Learning Metric) .

### 3.2 Fusion of multi-modality information in downstream tasks.

During pre-training, the encoders are initialized with parameters derived from distinct reference modalities. A critical question that arises is how to effectively utilize these pre-trained models during the fine-tuning stage to improve performance on downstream tasks.

#### 3.2.1 Early Stage: Multimodal Multi-Similarity

With a set of known target similarity $\{t^{R}\}$ from various modalities, we can transform themto multimodal space through a fusion function. There are numerous potential designs of the fusion function. For simplicity, we take linear combination as a demonstration. The multimodal generalized multi-similarity $t_{i,j}^{M}$ between $i^{th}$ and $j^{th}$ objects can be defined as follows:

$$ $\displaystyle t_{i,j}^{M}$ $\displaystyle=fusion(\{t^{R}\})$ (6) $\displaystyle=\sum w_{R}\cdot t_{i,j}^{R}$ (7) $$

where $t_{i,j}^{R}$ represents the target similarity between $i^{th}$ and $j^{th}$ instance in unimodal space $R$, $w_{R}$ is the pre-defined weights for the corresponding modal, and $\sum w_{R}=1$. Then we can make $t_{i,j}=t_{i,j}^{R}$ in equation  3. Such that, it still satisfy the requirement of convergence (See proof in Appendix SectionB.2). In this case, the learnt similarity during pretraining will be aligned with this new combined target similarity.

#### 3.2.2 Intermediate Stage: Embedding concatenation and fusion

Intermediate fusion integrates features from various modalities after their individual encoding processes and prior to the decoding/readout stage. Let $\mathbf{f}_{1},\mathbf{f}_{2},\ldots,\mathbf{f}_{n}$ represent the feature vectors obtained from these different modalities. The resulting fused feature vector can be defined as follows:

$$ $\mathbf{f}_{\text{fused}}=\text{MLP}(\text{concat}(\mathbf{f}_{1},\mathbf{f}_{ 2},\ldots,\mathbf{f}_{n}))$ (8) $$

Where concat represents concatenation of the feature vectors. The fused features are then fed into a later readout function or decoder for downstrean tasks prediction or classification. The MLP (Multi-Layer Perceptron) is used to reduce the dimension to be the same as $\mathbf{f}_{i}$.

#### 3.2.3 Late Stage: decision-level

Late fusion (or decision-level fusion) combines the outputs of models trained on different modalities after they have been processed independently. Each modality is first processed separately, and their predictions are combined at a later stage.

Let $p_{1},p_{2},\ldots,p_{n}$ be the predictions (e.g., probabilities) from different modalities. The final prediction $p_{\text{final}}$ can be computed using a weighted sum mechanism:

$$ $w_{i}=T_{i}(\text{f}_{i})$ (9) $$

$$ $p_{i}=\text{readout}_{i}(\text{f}_{i})$ (10) $$

$$ $p_{\text{final}}=\sum_{i=1}^{n}w_{i}p_{i}$ (11) $$

Where $w_{i}$ are the weights assigned to each modality’s prediction, and they can be adjusted based on the importance of each modality. In particular, $w_{i}$ is tunable during the learning process for respective downsteak tasks.

## 4 Experiments

In this section, we begin by presenting the datasets and selected modalities. Subsequently, we showcase the results obtained from MMFRL. Finally, we demonstrate the explainability of the learned molecular representations. (Please refer to the experimental details of pre-training and fine-tuning in the Appendix Section D.)

### 4.1 Dataset

#### 4.1.1 Selected Modalities for Target Similarity Calculation

The following modalities are used for target similarity calculation. For details on training the corresponding encoders to obtain fixed embeddings for these modalities, please refer to Appendix Section  C.1.

Fingerprint: Fingerprints are binary vectors that represent molecular structures, capturing the presence or absence of particular substructures, fragments, or chemical features within a molecule.

SMILES (Simplified Molecular Input Line Entry System): SMILES offers a compact textual representation of chemical structures.

NMR (Nuclear Magnetic Resonance): NMR spectroscopy provides detailed insights into the chemical environment of atoms within a molecule. By analyzing the interactions of atomic nuclei with an applied magnetic field, NMR can reveal information about the structure, dynamics, and interactions of molecules, including the connectivity of atoms, functional groups, and conformational changes. In our experiments, $\text{NMR}_{\text{spectrum}}$ provides the information about the overal information of molecule while $\text{NMR}_{\text{peak}}$ provides the information about the individual atoms in the molecule.

Image: Images (e.g., 2D chemical structures) provide a visual representation of molecular structures.

All of the similarity calculation from the modalities above are listed in Appendix  C.2.

#### 4.1.2 Pre-training

NMRShiftDB-2  (Landrum, ) is a comprehensive database dedicated to nuclear magnetic resonance (NMR) chemical shift data, providing researchers with an extensive collection of expert-annotated NMR data for various organic compounds with molecular structures (SMILES). There are around 25,000 molecules used for pre-training and no overlap with downstream task datasets. And molecular images and graphs are generated via RDkit  ().

#### 4.1.3 Downstream tasks

For Downstream tasks, our model was trained on 11 drug discovery-related benchmarks sourced from MoleculeNet  (Wu et al., ). Eight of these benchmarks were designated for classification downstream tasks, including BBBP, BACE, SIDER, CLINTOX, HIV, MUV, TOX21, and ToxCast, while three were allocated for regression tasks, namely ESOL, Freesolv, and Lipo. The datasets were divided into train/validation/test sets using a ratio of 80%:10%:10%, accomplished through the scaffold splitter  (Halgren, ; Landrum, ) from Chemprop  (Yang et al., ; Heid et al., ), like previous works. The scaffold splitter categorizes molecular data based on substructures, ensuring diverse structures in each set. Molecules are partitioned into bins, with those exceeding half of the test set size assigned to training, promoting scaffold diversity in validation and test sets. Remaining bins are randomly allocated until reaching the desired set sizes, creating multiple scaffold splits for comprehensive evaluation.

### 4.2 Results

#### 4.2.1 The effectiveness of pre-training

We first illustrate the impact of pre-training initialization on performance. As shown in Table 1, the average performance of pre-trained models outperform the non-pre-trained model in all tasks except for Clintox. The results of various downstream tasks indicate that different tasks may prefer different modalities. Notably, the model pre-trained with the NMR modality achieves the highest performance across three classification tasks. Similarly, the model pre-trained with the Image modality excels in three tasks, two of which are regression tasks related to solubility, aligning with findings from prior literature (Xu et al., ). Additionally, the model pre-trained with The fingerprint method achieves the best performance in two tasks, including MUV, which has the largest dataset.

**Table 1: Study on the performances of $\text{MMFRL}_{Unimodality}$. The best results are denoted in bold, and the second-best are indicated with underlining among the five modalities. The first 8 tasks are for classification under evaluation of ROC-AUC, while the last three are for regression with evaluation of RMSE.**
| Data Set | BBBP | bace | Sider | Clintox | HIV | MUV | Tox21 | ToxCast | ESOL | FreeSolv | Lipo |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SMILES | 92.9$\pm$1.5 | 90.9$\pm$3.3 | 64.9$\pm$0.3 | 78.2$\pm$1.9 | 83.3$\pm$1.1 | 80.1$\pm$2.5 | 85.7$\pm$1.2 | 70.5$\pm$2.5 | 0.811$\pm$ 0.109 | 1.623$\pm$ 0.168 | 0.539$\pm$ 0.017 |
| $\text{NMR}_{\text{spectrum}}$ | 91.0$\pm$2.0 | 93.2$\pm$2.7 | 68.1$\pm$1.5 | 87.7$\pm$6.5 | 80.9$\pm$5.0 | 80.9$\pm$5.0 | 85.1$\pm$0.4 | 71.1$\pm$0.8 | 0.844$\pm$ 0.123 | 2.417$\pm$ 0.495 | 0.609$\pm$ 0.031 |
| Image | 93.1$\pm$2.4 | 92.9$\pm$1.8 | 65.3$\pm$1.5 | 86.2$\pm$6.5 | 82.3$\pm$0.6 | 78.7$\pm$1.7 | 86.0$\pm$1.0 | 71.0$\pm$1.6 | 0.761$\pm$ 0.068 | 1.648$\pm$ 0.045 | 0.537$\pm$ 0.005 |
| Fingerprint | 92.9$\pm$2.3 | 91.7$\pm$3.6 | 65.6$\pm$0.7 | 87.5$\pm$6.0 | 81.2$\pm$2.5 | 82.9$\pm$3.1 | 85.3$\pm$1.3 | 70.0$\pm$1.4 | 0.808$\pm$ 0.071 | 1.437$\pm$ 0.134 | 0.565$\pm$ 0.017 |
| $\text{NMR}_{\text{Peak}}$ | 93.4$\pm$2.7 | 89.3$\pm$1.7 | 62.8$\pm$2.1 | 86.1$\pm$5.4 | 82.1$\pm$0.4 | 75.4$\pm$5.2 | 84.9$\pm$1.0 | 70.6$\pm$0.8 | 0.924$\pm$0.083 | 1.707$\pm$0.126 | 0.587$\pm$0.021 |
| Average | 92.8$\pm$1.9 | 91.4$\pm$2.7 | 65.3$\pm$2.0 | 85.0$\pm$5.7 | 81.8$\pm$2.2 | 79.4$\pm$4.0 | 85.4$\pm$0.9 | 70.6$\pm$1.3 | 0.830$\pm$0.094 | 1.766$\pm$0.394 | 0.586$\pm$0.048 |
| No Pre-training | 91.9$\pm$3.0 | 85.2$\pm$0.6 | 57.0$\pm$0.7 | 90.6$\pm$0.6 | 77.1$\pm$0.5 | 78.6$\pm$1.4 | 75.9$\pm$0.7 | 63.7$\pm$0.2 | 1.050$\pm$0.008 | 2.082$\pm$0.082 | 0.683$\pm$0.016 |

#### 4.2.2 Overall performance of MMFRL

As shown in Table  2 and Table  3, MMFRL demonstrates superior performance compared to all baseline models and the average performance of DMPNN pretrained with extra modalities across all 11 tasks evaluated in MoleculeNet. This robust performance highlights the effectiveness of our approach in leveraging multimodal data. In particular, while individual models pre-trained on other modalities for ClinTox fail to outperform the No-pretraining model (DMPNN), the fusion of these pre-trained models leads to improved performance. Besides, apart from Tox21 and Sider, the fusion models significantly enhances overall performance. In particular, the intermediate fusion model stands out by achieving the highest scores in seven distinct tasks, showcasing its ability to effectively combine features at a mid-level abstraction. the late fusion model achieves the top performance in two tasks. These results underscore the advantages of utilizing various fusion strategies in multimodal learning, further validating the efficacy of the MMFRL framework.

**Table 2: Overall performances (ROC-AUC) on classification downstream tasks. The best results are denoted in bold, and the second-best are indicated with underlining. For early fusion of MMFRL, all the predefined weight of each modality are 0.2. (Note: N-Gram is highly time-consuming on ToxCast.)**
| Data Set | BBBP | bace | Sider | Clintox | HIV | MUV | Tox21 | ToxCast |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AttentiveFP | 64.3$\pm$1.8 | 78.4$\pm$2.2 | 60.6$\pm$3.2 | 84.7$\pm$0.3 | 75.7$\pm$1.4 | 76.6$\pm$1.5 | 76.1$\pm$0.5 | 63.7$\pm$0.2 |
| DMPNN | 91.9$\pm$3.0 | 85.2$\pm$0.6 | 57.0$\pm$0.7 | 90.6$\pm$0.6 | 77.1$\pm$0.5 | 78.6$\pm$1.4 | 75.9$\pm$0.7 | 63.7$\pm$0.2 |
| N-Gram | 91.2$\pm$0.3 | 79.1$\pm$1.3 | 63.2$\pm$0.5 | 87.5$\pm$2.7 | 78.7$\pm$0.4 | 76.9$\pm$0.7 | 76.9$\pm$2.7 | - |
| GEM | 72.4$\pm$0.4 | 85.6$\pm$1.1 | 67.2$\pm$0.4 | 90.1$\pm$1.3 | 80.6$\pm$0.9 | 81.7$\pm$0.5 | 78.1$\pm$0.1 | 69.2$\pm$0.4 |
| Uni-Mol | 72.9$\pm$0.6 | 85.7$\pm$0.2 | 65.9$\pm$1.3 | 91.9$\pm$1.8 | 80.8$\pm$0.3 | 82.1$\pm$1.3 | 79.6$\pm$0.5 | 69.6$\pm$0.1 |
| InfoGraph | 69.2$\pm$0.8 | 73.9$\pm$2.5 | 59.2$\pm$0.2 | 75.1$\pm$5.0 | 74.5$\pm$1.8 | 74.0$\pm$1.5 | 73.0$\pm$0.7 | 62.0$\pm$0.3 |
| GraphCL | 67.5$\pm$3.3 | 68.7$\pm$7.8 | 60.1$\pm$1.3 | 78.9$\pm$4.2 | 75.0$\pm$0.4 | 77.1$\pm$1.0 | 75.0$\pm$0.3 | 62.8$\pm$0.2 |
| MolCLR | 73.3$\pm$1.0 | 82.8$\pm$0.7 | 61.2$\pm$3.6 | 89.8$\pm$2.7 | 77.4$\pm$0.6 | 78.9$\pm$2.3 | 74.1$\pm$5.3 | 65.9$\pm$2.1 |
| $\text{MolCLR}_{\text{cmpnn}}$ | 72.4$\pm$0.7 | 85.0$\pm$2.4 | 59.7$\pm$3.4 | 88.0$\pm$4.0 | 77.8$\pm$5.5 | 74.5$\pm$2.1 | 78.4$\pm$2.6 | 69.1$\pm$1.2 |
| GraphMVP | 72.4$\pm$1.6 | 81.2$\pm$9.0 | 63.9$\pm$1.2 | 79.1$\pm$2.8 | 77.0$\pm$1.2 | 77.7$\pm$6.0 | 75.9$\pm$5.0 | 63.1$\pm$0.4 |
| $\text{Unimodality}_{avg}$ | 92.8$\pm$1.9 | 91.4$\pm$2.7 | 65.3$\pm$2.0 | 85.0$\pm$5.7 | 81.8$\pm$2.2 | 79.4$\pm$4.0 | 85.4$\pm$0.9 | 70.6$\pm$1.3 |
| $\text{MMFRL}_{early}$ | 91.6$\pm$5.0 | 94.3$\pm$2.4 | 66.4$\pm$1.9 | 85.3$\pm$6.8 | 82.0$\pm$2.4 | 80.6$\pm$3.2 | 85.2$\pm$0.2 | 69.8$\pm$1.1 |
| $\text{MMFRL}_{intermediate}$ | 95.4$\pm$0.7 | 95.1$\pm$1.0 | 64.3$\pm$1.2 | 93.4$\pm$1.1 | 81.2$\pm$1.3 | 83.5$\pm$1.6 | 85.1$\pm$0.1 | 71.9$\pm$1.1 |
| $\text{MMFRL}_{late}$ | 94.7$\pm$0.6 | 91.6$\pm$2.6 | 64.2$\pm$1.2 | 87.0$\pm$0.4 | 82.9$\pm$0.2 | 82.1$\pm 1.7$ | 77.7$\pm$0.5 | 70.2$\pm$0.3 |

**Table 3: Overall performances (RMSE) on regression downstream tasks. The best results are denoted in bold, and the second-best are indicated with underlining. For early fusion of MMFRL, all the predefined weight of each modality are 0.2.**
| Data Set | ESOL | FreeSolv | Lipo |
| --- | --- | --- | --- |
| AttentiveFP | 0.877$\pm$0.029 | 2.073$\pm$0.183 | 0.721$\pm$0.001 |
| DMPNN | 1.050$\pm$0.008 | 2.082$\pm$0.082 | 0.683$\pm$0.016 |
| $\text{N-Gram}_{\text{RF}}$ | 1.074$\pm$0.107 | 2.688$\pm$0.085 | 0.812$\pm$0.028 |
| $\text{N-Gram}_{\text{XGB}}$ | 1.083$\pm$0.082 | 5.061$\pm$0.744 | 2.072$\pm$0.030 |
| GEM | 0.798$\pm$0.029 | 1.877$\pm$0.094 | 0.660$\pm$0.008 |
| Uni-Mol | 0.788$\pm$0.029 | 1.620$\pm$0.035 | 0.660$\pm$0.008 |
| MolCLR | 1.113$\pm$0.023 | 2.301$\pm$0.247 | 0.789$\pm$0.009 |
| $\text{MolCLR}_{\text{CMPNN}}$ | 0.911$\pm$0.082 | 2.021$\pm$0.133 | 0.875$\pm$0.003 |
| $\text{Unimodality}_{avg}$ | 0.924$\pm$0.083 | 1.707$\pm$0.126 | 0.587$\pm$0.021 |
| $\text{MMFRL}_{early}$ | 1.037$\pm$0.170 | 2.093$\pm$0.090 | 0.607$\pm$0.034 |
| $\text{MMFRL}_{intermediate}$ | 0.730$\pm$0.019 | 1.465$\pm$0.096 | 0.552$\pm$0.014 |
| $\text{MMFRL}_{late}$ | 0.763$\pm$0.035 | 1.741$\pm$0.191 | 0.525$\pm$0.018 |

### 4.3 Analysis of the fusion effect

#### 4.3.1 General Comparison among various Ways of Fusions

Early Fusion is employed during the pretraining phase and is easy to implement, as it aggregates information from different modalities directly. However, its primary limitation lies in the necessity for predefined weights assigned to each modality. These weights may not accurately reflect the relevance of each modality for the specific downstream tasks, potentially leading to suboptimal performance.

Intermediate Fusion is able to capture the interaction between modalities early in the fine-tuning process, allowing for a more dynamic integration of information. This method can be particularly beneficial when different modalities provide complementary information that enhances overall performance. If the modalities effectively compensate for one another’s strengths and weaknesses, Intermediate Fusion may emerge as the most effective approach.

In contrast, Late Fusion enables each modality to be explored independently, maximizing the potential of individual modalities without interference from others. This separation allows for a thorough examination of each modality’s contribution. When certain modalities dominate the performance metrics, Late Fusion can maximize on these strengths, ensuring that the most impactful information is utilized effectively. This approach is especially useful in scenarios where the dominance of specific modalities can be leveraged to enhance overall model performance.

#### 4.3.2 Explainability of Learnt Representations

To demonstrate the interpretability of learnt representations of fusion, we present post-hoc analysis for two tasks, ESOL and Lipo, as demonstration. The results showcase learnt representations can capture task-specific patterns and offer valuable insights for molecular design.

ESOL with Intermediate Fusion. As presented in Table 3, the intermediate fusion method  3.2.2 exhibits superior performance on the ESOL regression task for predicting solubility. To further analyze this performance, we employed t-SNE to reduce the dimensionality of the molecule embeddings from 300 to 2, resulting in a heatmap visualized in Figure 2. The embeddings derived from individual modalities prior to fusion do not display a clear pattern, showing no smooth transition from low to high solubility. In contrast, the embeddings by intermediate fusion reveal a distinct and smooth transition in solubility values: molecules with similar solubility cluster together, forming a gradient that extends from the bottom left (indicating lower solubility) to the upper center (representing higher solubility). This trend underscores the effectiveness of the intermediate fusion approach in accurately capturing the quantitative structure-activity relationships for aqueous solubility.

Additionally, we examined the similarity between the respective embeddings prior to intermediate fusion and the resulting fused embedding, as depicted in Figure 3. Our analysis indicates that the embeddings from each modality exhibit low similarity with the intermediate-fused representation. This observation suggests that the modalities complement each other, collectively enhancing the resulting representation of the intermediate-fused embedding.

Lipo with Late Fusion. As detailed in Table  3, the Late Fusion method (described in Section  3.2.3) demonstrates superior performance on the Lipo regression task for predicting solubility in fats, oils, lipids, and non-polar solvents. According to Equation  11, the final prediction is determined by the respective coefficients ($w_{i}$) and predictions ($p_{i}$) from each modality.

In Figure 4, we present the distribution of values for the coefficients, predictions, and their products for each modality. Notably, the SMILES and Image modalities display a wide range of values, highlighting their potential to significantly influence the final predictions. This observation aligns with the strong performance achieved when pretraining using either of these two modalities, as shown in Table  1. In contrast, the $\text{NMR}_{\text{Peak}}$ values display a narrower range, indicating its role as a modifier for finer adjustments in the predictions. Furthermore, we observe that the contributions from $\text{NMR}_{\text{Spectrum}}$ and Fingerprint modalities are minimal, with their corresponding values approaching zero. This outcome highlights the advantages of the Late Fusion approach in effectively identifying and leveraging dominant modalities, thereby optimizing the overall predictive performance.

Figure: Figure 2: T-SNE visualization depicting the ESOL molecule embeddings for intermediate fusion in Section 3.2.2 alongside molecules within the highlighted region. Each point in the heatmap corresponds to the embeddings of respective molecules in ESOL, with color indicating solubility levels. Red denotes higher solubility, while blue indicates lower solubility. The embeddings derived from individual modalities prior to fusion do not display a clear pattern, the embeddings by intermediate fusion forms a gradient that extends from the bottom left (indicating lower solubility) to the upper center (representing higher solubility).
Refer to caption: extracted/5930035/Figures/ESOL_heatmap.png

Figure: Figure 3: This figure shows the distribution of similarities between each modality and the intermediate fusion embedding for ESOL. In both Cosine Similarity and Dot Product, the embeddings from each modality exhibit low similarity with the intermediate-fused representation.
Refer to caption: extracted/5930035/Figures/ESOL_histograms_similarity.png

Figure: Figure 4: Lipo late fusion contribution analysis reveals that the three primary contributors are SMILES, image, and $\text{NMR}_{\text{peak}}$. In contrast, $\text{NMR}_{\text{spectrum}}$ and fingerprint exhibit negligible contributions.
Refer to caption: extracted/5930035/Figures/lipo_histograms_3_5_adjusted_true.png

## 5 Related work

Contrastive Learning on Molecular Graphs. The primary focus within the domain of contrastive learning applied to molecular graphs centers on 2D-2D graphs comparisons. Noteworthy representative examples: InfoGraph  (Sun et al., ) maximizes the mutual information between the representations of the graph and its substructures to guide the molecular representation learning;
GraphCL  (You et al., ), MoCL  (Sun et al., ), and MolCLR  (Wang et al., ) employs graph augmentation techniques to construct positive pairs; MoLR  (Wang et al., ) establishes positive pairs with reactant-product relationships. In addition to 2D-2D graph contrastive learning, there are also noteworthy efforts exploring 2D-3D and 3D-3D contrastive learning in the field.
3DGCL  (Moon et al., ) is 3D-3D contrastive learning model, establishing positive pairs with conformers from the same molecules. GraphMVP  (Liu et al., ), GeomGCL  (Li et al., ), and 3D Informax  (Stärk et al., ) proposes 2D–3D view contrastive learning approaches. To conclude, 2D-2D and 3D-3D comparisons are intra-modality contratsive leraning, as only one graph encoder is employed in these studies. And these approaches often focus on the motif and graph levels, leaving atom-level contrastive learning less explored.

Similarity Learning. Instance-wise discrimination, a crucial facet of similarity learning, involves evaluating the similarity between instances directly based on their latent representations or features  (Wu et al., ). Naive instance-wise discrimination relies on pairwise similarity, leading to the development of contrastive loss  (Hadsell et al., ). Although there are improved loss functions such as triplet loss  (Hoffer & Ailon, ), quadruplet loss  (Law et al., ), lifted structure loss  (Oh Song et al., ), N-pairs loss  (Sohn, ), and angular loss  (Wang et al., ), these methods still fall short in thoroughly capturing relationships among multiple instances simultaneously  (Wang et al., ). To address this limitation, a joint multi-similarity loss has been proposed, incorporating pair weighting for each pair to enhance instance-wise discrimination  (Wang et al., ; Zhang et al., ). Notably, it is crucial to emphasize that employing these pair weightings requires the manual categorization of negative and positive pairs, as distinct weights are assigned to losses based on their categories.

## 6 Discussion

In summary, we introduce a novel relational learning metric for molecular graph representation that enhances the understanding of inter-instance relationships by capturing both local and global contexts. This is the first implementation of such a generalized metric in molecular graphs.Our method transforms pairwise self-similarity into relative similarity through a weighting function, allowing for complex relational insights. This metric is integrated into a multimodal representation, improving performance by utilizing modalities not directly accessible during fine-tuning. Empirical results show that our approach, MMFRL, excels in various molecular property prediction tasks. We also demonstrate detailed study about the explainability of the learned representations, offering valuable insights for drug molecule design. Despite these accomplishments, further exploration is needed to achieve more effective integration of graph- and node-level similarities. Looking ahead, we are enthusiastic about the prospect of applying our model to additional fields, such as social science, thereby broadening its applicability and impact.

## Accessibility

The code and dataset will be made available upon the date of publication.

## Appendix A Multi-Similarity & Contrastive Learning

### A.1 Multi-Similarities in Contrastive Learning

Two distinct types of similarities, as illustrated in Appendix Figure A.1, can be identified: self-similarity (the pairwise similarity between two objects, typically defined through cosine similarity) and relative similarity (distinctions in self-similarity with other pairs)  (Wang et al., ).

Figure: Figure A.1: Illustration of Different Types of Similarities.
Refer to caption: x1.png

### A.2 Current Molecular Graph Contrastive Learning Approaches

In current molecular graph contrastive learning approaches, positive pairs are commonly formed through either data augmentation  (Sun et al., ; You et al., ), employing techniques such as node deletion, edge perturbation, subgraph extraction, attribute masking, and subgraph substitution, or domain knowledge, as demonstrated by reactant-product pairing  (Wang et al., ) or conformer grouping  (Moon et al., ).

Figure: Figure A.2: General framework of Intra-Modality Graph Contrastive Learning. It relies on definition of positive and negative pairs.
Refer to caption: x2.png

## Appendix B Supplementary Proof

### B.1 Revisiting Theorem of Convergent Similarity Learning

Let $\mathcal{S}$ be a set of instances with size $|\mathcal{S}|$, and let $\mathcal{P}$ represent the tunable latent representations of instances in $\mathcal{S}$ such that $|\mathcal{P}|=|\mathcal{S}|$. For any two instances $i,j\in\mathcal{S}$, their latent representations are denoted by $\mathcal{P}_{i}$ and $\mathcal{P}_{j}$, respectively. Let $t_{i,j}$ represent the target similarity between instances $i$ and $j$ in a given domain, and $d_{i,j}$ be the similarity between $\mathcal{P}_{i}$ and $\mathcal{P}_{j}$ in the latent space.

###### Theorem B.1 (Theorem of Convergent Similarity learning) .

Given $t_{i,j}$ is non-negative and $\{t_{i,j}\}$ satisfies the constraint $\sum_{j=1}^{|\mathcal{S}|}t_{i,j}=1$, consider the loss function for an instance $i$ defined as follows:

$$ $L(i)=-\sum_{j=1}^{|\mathcal{S}|}t_{i,j}\log\left(\frac{e^{d_{i,j}}}{\sum_{k=1} ^{|\mathcal{S}|}e^{d_{i,k}}}\right)$ (B.1) $$

then when it reaches ideal optimum, the relationship between $t_{i,j}$ and $d_{i,j}$ satisfies:

$$ $\text{softmax}(d_{i,j})=t_{i,j}$ (B.2) $$

###### Proof.

In order to optimize the loss $L(i)$, we need to set the following partial derivative to be 0 for each $d_{i,j}$ with $1\leq j\leq|\mathcal{M}|$. Here are the detailed steps:

$$ $\displaystyle\frac{\partial L(i)}{\partial{d_{i,j}}}$ $\displaystyle=\frac{\partial}{\partial d_{i,j}}\underbrace{\left(-t_{i,j}\log \frac{e^{d_{i,j}}}{e^{d_{i,j}}+\sum_{\begin{subarray}{c}k\neq j\end{subarray}} e^{d_{i,k}}}\right)}_{\text{When the numerator includes }e^{d_{i,j}}}+\frac{ \partial}{\partial d_{i,j}}\underbrace{\left(\sum_{\begin{subarray}{c}k\neq j \end{subarray}}-t_{i,k}\log\frac{e^{d_{i,k}}}{e^{d_{i,j}}+\sum_{\begin{ subarray}{c}k\neq j\end{subarray}}e^{d_{i,k}}}\right)}_{\text{When the numerator does not include }e^{d_{i,j}}}$ $\displaystyle=-(t_{i,j}-t_{i,j}\cdot\text{softmax}(d_{i,j}))-\sum_{\begin{ subarray}{c}k\neq j\end{subarray}}t_{i,k}\cdot\text{softmax}(d_{i,j})$ $\displaystyle=-\left(t_{i,j}-\left(t_{i,j}+\sum_{\begin{subarray}{c}k\neq j \end{subarray}}t_{i,k}\right)\cdot\text{softmax}(d_{i,j})\right)$ $$

Since $\sum_{l=1}^{|\mathcal{M}|}t_{i,l}=1$, we can further simplify it as

$$ $\displaystyle\frac{\partial L(i)}{\partial{d_{i,j}}}=-(t_{i,j}-\text{softmax}( d_{i,j}))$ $$

In order to optimize, we need to see the above partial derivative to be 0:

$$ $\displaystyle\frac{\partial L(i)}{\partial{d_{i,j}}}=-(t_{i,j}-\text{softmax}( d_{i,j}))=0$ $$

In addition, the corresponding second partial derivative denoted as $\frac{\partial L(i)}{\partial{d^{2}_{i,j}}}$ manifests as follows:

$$ $\displaystyle\frac{\partial L(i)}{\partial{d^{2}_{i,j}}}=\text{softmax}(d_{i,j })(1-\text{softmax}(d_{i,j}))$ $$

As $\text{softmax}(d_{i,j})$ takes values within the open interval (0,1), it follows that $\frac{\partial L(i)}{\partial{d^{2}_{i,j}}}$ is always positive. Consequently, the global optimum is global minimum.
Furthermore, when it comes to optimum:

$$ $\displaystyle t_{i,j}$ $\displaystyle=\text{softmax}(d_{i,j})$ $\displaystyle d_{i,j}$ $\displaystyle=\log(t_{i,j})+\log\left(\sum_{\begin{subarray}{c}1\leq l\leq| \mathcal{M}|\end{subarray}}e^{d_{i,j}}\right)$ $$

It is easy to show that when it reaches optimum, $d_{i,j}$ is consistent with target similarity metric $t_{i,j}$. Without loss of generosity, suppose $t_{i,j}>t_{i,j^{\prime}}$ :

$$ $\displaystyle d_{i,j}-d_{i,j^{\prime}}$ $\displaystyle=\log(t_{i,j})+\log\left(\sum_{\begin{subarray}{c}1\leq l\leq| \mathcal{M}|\end{subarray}}e^{d_{il}}\right)-\left(\log(t_{i,j^{\prime}})+\log \left(\sum_{\begin{subarray}{c}1\leq l\leq|\mathcal{M}|\end{subarray}}e^{d_{il }}\right)\right)$ $\displaystyle=\log(t_{i,j})-\log(t_{i,j^{\prime}})$ $\displaystyle=\log\left(\frac{t_{i,j}}{t_{i,j^{\prime}}}\right)>0$ $$

∎

###### Theorem B.1 (Theorem of Convergent Similarity learning) .

###### Proof.

### B.2 Guarantee of Sum of Fused Multimodal Similarity

Given sets of uni-modal generalized similarity $\{t^{R}\}$ and $\sum w_{t^{R}}=1$, the sum of fused multimodal similarity also equals 1, as demonstrated below:

$$ $\displaystyle\sum(t_{i,j}^{R})$ $\displaystyle=\sum\sum(w_{R}\cdot t^{R}_{i,j})$ $\displaystyle=\sum(w_{R}\sum t^{R}_{i,j})$ $\displaystyle=\sum w_{R}\cdot 1=1$ $$

## Appendix C Revisiting Target Similarity Settings

### C.1 Encoders & Packages

To derive the target similarities, we need to reply on pre-trained encoders or well-defined packages as follows:

**Table C.1: Encoders and packages used to produce self-similarities**
| Unimodal | Representation | Encoder/Package | Pre-trained Source |
| --- | --- | --- | --- |
| Image | 2D image | CNN | Img2mol  (Clevert et al., ) |
| SMILES | Sequence | Transformer | CReSS  (Yang et al., ) |
| ^13CNMR Spectrum | Sequence | 1D CNN | AutoEncoder  (Costanti et al., ) |
| ^13CNMR peak | Scalar | NMRShiftDB2  (Steinbeck et al., ) | N/A |
| Fingerprint | Sequence | RDKit  (Landrum, ) | N/A |

### C.2 Target Similarity at Graph Level

Fingerprint. The mathematical formula of fingerprint similarity, denoted as $S_{i,j}^{F}$, can be viewed as follows:

$$ $\displaystyle S_{i,j}^{F}$ $\displaystyle=Tanimoto(A,B)=\frac{|A\cap B|}{|A\cup B|}$ (C.1) $$

where $A$ and $B$ are sets of molecular fragments for molecule $i$ and $j$, and $|A\cap B|$ and $|A\cup B|$ denote the size of their intersection and union, respectively.

Image. The self-similarity for Image, denoted as $S_{i,j}^{I}$, can be defined as follows:

$$ $S_{i,j}^{I}=Cos(\mathcal{V}_{i},\mathcal{V}_{j})=\frac{\mathcal{V}_{i}\cdot \mathcal{V}_{j}^{T}}{\|\mathcal{V}_{i}\|\cdot\|\mathcal{V}_{j}\|}$ (C.2) $$

where $\mathcal{V}_{i},\mathcal{V}_{j}$ represents the embedding of Image for two given molecules.

NMR Spectrum. The self-similarity for NMR spectrum, denoted as $S_{i,j}^{C}$, can be defined as follows:

$$ $S_{i,j}^{C}=Cos(\mathcal{V}_{i},\mathcal{V}_{j})=\frac{\mathcal{V}_{i}\cdot \mathcal{V}_{j}^{T}}{\|\mathcal{V}_{i}\|\cdot\|\mathcal{V}_{j}\|}$ (C.3) $$

where $\mathcal{V}_{i},\mathcal{V}_{j}$ represents the embedding of NMR spectra for two given molecules.

Smiles. The self-similarity for Smiles, denoted as $S_{i,j}^{S}$, can be defined as follows:

$$ $S_{i,j}^{S}=Cos(\mathcal{V}_{i},\mathcal{V}_{j})=\frac{\mathcal{V}_{i}\cdot \mathcal{V}_{j}^{T}}{\|\mathcal{V}_{i}\|\cdot\|\mathcal{V}_{j}\|}$ (C.4) $$

where $\mathcal{V}_{i},\mathcal{V}_{j}$ represents the embedding of Smiles for two given molecules.

NMR Peak The similarity among nodes (atoms) is derived from the positions of their signal peaks on ^13C NMR spectra, measured in parts per million (ppm). The ppm values are continuous, typically ranging from 0 to 200 (see more introduction of ppm in Appendix C.3). The self-similarity of NMR peaks $S^{P}_{l,m}$ can be defined as following:

$$ $S^{P}_{l,m}=\frac{\tau_{2}}{|ppm_{l}-ppm_{m}|+\tau_{1}}$ (C.5) $$

where $ppm_{l}$ and $ppm_{m}$ are the positions of NMR peaks for the $l^{th}$, $m^{th}$ Carbon atom, $\tau_{1}$ and $\tau_{2}$ are temperature hyper-parameter.

### C.3 A Brief Introduction to PPM for NMR Peak

In chemistry, ^13C NMR stands out as a common technique for structural analysis by revealing molecular structures by elucidating the chemical environments of carbon atoms and their magnetic responses to external fields  (Gerothanassis et al., ; Lambert et al., ). It quantifies these features in parts per million (ppm) relative to a reference compound, such as tetramethylsilane (TMS), thereby simplifying comparisons across experiments. As a result, the continuous peak positions, measured in parts per million (ppm), offer a robust knowledge span—a natural ordering metric that can be employed to derive measures of similarity  (Xu et al., ).

### C.4 Configuration of Early Fusion

A simple linear combination is used to formulate the multimodal relational similarity $t_{i,j}^{M}$ between the $i^{th}$ and $j^{th}$ molecules, represented as as follows:

$$ $\displaystyle t_{i,j}^{M}=w_{SM}\cdot t^{SM}_{i,j}+w_{{C}}\cdot t^{C}_{i,j}+w_ {{I}}\cdot t^{I}_{i,j}+w_{{F}}\cdot t^{F}_{i,j}+w_{{F}}\cdot t^{F}_{i,j}+w_{{P }}\cdot t^{P}_{i,j}$ (C.6) $$

where $t^{SM}_{i,j}$ denotes the similarity based on SMILES, $t^{C}_{i,j}$ denotes the similarity with respect to ^13C NMR spectrum, $t^{I}_{i,j}$ denotes the similarity regarding images, $F$ denotes the similarity based on fingerprints, and $P$ denotes the similarity based on fingerprints. $w_{SM}$, $w_{C}$, $w_{I}$, and $w_{F}$ are the pre-defined weights for their respective similarity, and $w_{SM}+w_{{C}}+w_{{I}}+w_{{F}}+w_{{P}}=1$.

## Appendix D Experimental Settings

### D.1 Pre-Training Setting

During pretraining, we utilized an Adam optimizer with a learning rate set to 0.001, spanning 200 epochs and employing a batch size of 256. The model was trained on around 25,000 data points. The NMR data were experimental data, extracted from NMRShiftDB2  (Steinbeck et al., ). Other chemical modalities, such as images, fingerprints and graphs, were produced from SMILES by RDKit  (Landrum, ).

### D.2 Fine-Tuning Setting

#### D.2.1 Datasets

For fine-tuning, our model was trained on 11 drug discovery-related benchmarks sourced from MoleculeNet  (Wu et al., ). Eight of these benchmarks were designated for classification downstream tasks, including BBBP, BACE, SIDER, CLINTOX, HIV, MUV, TOX21, and ToxCast, while three were allocated for regression tasks, namely ESOL, Freesolv, and Lipo. The datasets were divided into train/validation/test sets using a ratio of 80%:10%:10%, accomplished through the scaffold splitter  (Halgren, ; Landrum, ) from Chemprop  (Yang et al., ; Heid et al., ), like previous works. The scaffold splitter categorizes molecular data based on substructures, ensuring diverse structures in each set. Molecules are partitioned into bins, with those exceeding half of the test set size assigned to training, promoting scaffold diversity in validation and test sets. Remaining bins are randomly allocated until reaching the desired set sizes, creating multiple scaffold splits for comprehensive evaluation.

#### D.2.2 Baselines

We systematically compared MMFRL’s performance with various state-of-the-art baseline models across different categories. In the realm of supervised models, AttentiveFP  (Xiong et al., ) and DMPNN  (Yang et al., ) stand out by leveraging graph attention networks and node-edge interactive message passing, respectively. The unsupervised learning method N-Gram  (Liu et al., ) employs graph embeddings and short walks for graph representation. Predictive self-supervised learning methods, such as GEM  (Fang et al., ) and Uni-Mol  (Zhou et al., ), are specifically designed for predicting molecular geometric information. Moreover, our evaluation encompasses a range of contrastive learning methods, namely InfoGraph  (Sun et al., ), GraphCL  (You et al., ), MolCLR  (Wang et al., ), and GraphMVP  (Liu et al., ), all serving as essential baselines. The baseline results are collected from recent works  (Fang et al., ; Zhou et al., ; Moon et al., ; Fang et al., ).

#### D.2.3 Evaluation

To assess the effectiveness of our fine-tuned model, we measure the ROC-AUC for classification downstream tasks, and the root mean squared error (RMSE) metric for regression tasks. In order to ensure a fair and robust comparisons, we conduct three independent runs using three different random seeds for scaffold splitting across all datasets. The reported performance metrics are then averaged across these runs, and the standard deviation is computed as prior works.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: -	Does removing a key modality result in a significantly larger performance drop compared to removing other modalities? Ablation studies seem to be needed to better demonstrate claim.
-	The paper suggests that lower cosine similarity between each modality’s embeddings and the fused embeddings indicates a complementary effect. Could additional information clarify this claim, such as comparing similarity in other scenarios?
soundness: 2
strengths: -	The method effectively employs a continuous loss function for multimodal relational learning in molecular property prediction, delivering promising results.

-	The method introduces three distinct multimodal fusion approaches – early, intermediate, and late stage – offering alternative strategies for integrating different modalities and illustrating the strengths and weakness of each.
summary: The paper introduces Multimodal Fusion with Relational Learning (MMFRL), a framework that enhances molecular property prediction through multimodal fusion and relational learning. The authors applied relational learning on molecular prediction by modifying relational learning loss with a continuous similarity metric, to capture nuanced differences in molecular distances through relative positioning among molecules. By utilizing multimodal learning with a pretrained encoder for each modality, they aimed to obtain a unified molecular representation that incorporates information from multiple modalities. To fuse information from these modalities, they proposed three distinct fusion strategies, each demonstrating unique characteristics in representation. This approach led to varied performance patterns across 11 downstream tasks in MoleculeNet dataset, with MMFRL achieving superior performance.
weaknesses: Major comments

-	While the idea of modifying the loss function for efficient relational learning sounds novel, the explanation on the specific modification of the loss function does not effectively support the novelty of the proposed method, as the objective function is a rather standard one widely used in the community.
-	Although the paper presents three approaches to multimodal fusion - early, intermediate, and late-stage fusion - it is not clear which of the three stage fusion strategies can be effectively used for certain tasks. Clearly demonstrating for which tasks each fusion strategy performs better would strengthen the work. For example, it remains unclear to users when would specific modality dominates the learning process, a case where the authors suggest late fusion is more effective.

Minor comments

-	The case studies are intriguing, but some results appear inconsistent. For example, while the analysis on the fusion effect suggests that late fusion performs best when one modality dominates, the case study on the Lipo regression task implies the multiple modalities complement each other for better results.
-	Ablation result regarding pretrained graph encoder does not seem to need to be placed as first result.
-	Additional analysis only performed on regression tasks. 
-	Typos : Section 3.2.1 1st line: ‘themto’,  Section 4.2.2 5th line: ‘No-pretraining’

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: See the weakness above
soundness: 3
strengths: 1.	This paper addresses the important topic of pre-trained molecular representation learning. The authors comprehensively consider multiple modalities such as Fingerprint, SMILES, NMR, and Image, and systematically explore the effects of modality fusion at different stages (early, intermediate, late) to understand their benefits and limitations.              
2.	The authors evaluate the model’s performance across various molecular property prediction datasets.
summary: The paper proposes MMFRL (Multimodal Fusion with Relational Learning for Molecular Property Prediction), a new framework to address challenges in graph-based molecular representation learning for drug discovery and materials science. It enhances embedding initialization through multimodal pre-training with relational learning and systematically explores the effects of modality fusion at different stages (early, intermediate, late) to understand their benefits and limitations. Experiments on MoleculeNet benchmarks show that MMFRL outperforms existing methods and offers explainability that provides chemical insights, highlighting its potential for real-world applications in drug discovery.
weaknesses: 1.The paper states that while contrastive learning is often used to handle molecular relationships, its reliance on binary metrics is insufficient for capturing the complexity of these interactions. However, this research question is not novel in the context of molecular pretraining. Prior work, such as MoleBERT [1], has already discussed this issue in the context of molecular graphs and proposed corresponding solutions. Unfortunately, the authors did not reference this previous work.            
2.The introduction of multimodal contrastive learning to address the limitations of binary metrics has limited novelty.        
3.The baselines used for comparison are not comprehensive. For example, in Table 2, there is no comparison with the current state-of-the-art model, Uni-Mol [2].          
4.The authors only evaluated performance on molecular property prediction tasks, lacking assessments on other tasks such as molecule-molecule interaction prediction and molecule-target affinity prediction.                   
5. There is no code or checkpoint provided for reproducing the results.         

[1] Mole-BERT: Rethinking Pre-training Graph Neural Networks for Molecules (ICLR 2023)              
[2] Uni-Mol: A Universal 3D Molecular Representation Learning Framework (ICLR 2023)

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Given that the paper claims that the proposed method can contribute to drug design, I am particularly interested in its performance on large molecule-size datasets with significant practical value, such as the LBA dataset.
2. I am not very familiar with the concept of relational learning mentioned in the paper. Is relational learning a specialized research area? What is its connection to contrastive learning?
3. I am curious about the generalization ability of the proposed method on out-of-distribution data, particularly when training and testing involve molecules with varying ranges of heavy atom counts. A common challenge in molecular property prediction is the poor generalization ability on out-of-distribution data. I wonder whether the integration of multiple intra-molecular modalities can alleviate this issue to some extent.
soundness: 2
strengths: 1. The writing of the paper is clear and comprehensible. 
2. The visualization design in Figure 2 is meaningful and clear, effectively illustrating the advantages of multimodal fusion.
summary: This paper introduces a new metric for measuring inter-molecular relationships and applies this metric to molecular multimodal fusion. In the context of molecular multimodal fusion, the paper integrates various intra-molecular modalities and conducts ablation studies on the fusion at different stages and the contributions of different modalities. The effectiveness of the proposed method is validated through downstream tests on molecular property prediction tasks using the MoleculeNet dataset.
weaknesses: 1. Regarding the key contribution emphasized in the paper—the measurement of inter-molecular relationships—it is noted that the smooth contrastive learning metric has been previously proposed in various studies. The paper appears to merely transfer this metric to the molecular domain for application.
2. Concerning the work on multimodal fusion, the focus of the paper is on exploring the optimal fusion stage. However, it seems that most existing fusion works also perform fusion at this identified optimal stage. While conducting an ablation study on this empirical approach is somewhat meaningful, it appears somewhat limited as a primary contribution of the paper.
3. The downstream tasks conducted in the paper are rather limited, which does not provide compelling evidence for the effectiveness of the proposed method. The experiments are solely tested on the MoleculeNet dataset, which generally has a small sample size, leading to a degree of randomness in the results. Given that the title of the paper specifically addresses molecular property prediction, it is also essential to test on classic large-scale molecular property prediction datasets, such as PCQM4Mv2 and QM9.
4. The design of the figures and the structure of the paper require improvement. The figures are not presented in a reasonable format, and the font is somewhat unclear, making them difficult to read. Additionally, the overall structure of the paper appears somewhat disorganized, particularly in the experimental section, where a more logical design could be adopted.

---

confidence: 3
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. What's the difference between Relation Learning and Contrastive Learning?

2. The authors should also compare their approach with [1,2] for a more comprehensive evaluation.

3. How do the modified relational learning metric help to capture both localized and global relationships?

[1] Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editing

[2] MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter
soundness: 2
strengths: 1. The paper introduces a relational learning metric that moves beyond binary metrics, allowing a continuous perspective on molecular relationships. 

3. The method fuses on different stages and adapts well to various scenarios. The intermediate fusion method stands out in multiple tasks, as it allows a more dynamic integration of information from different modalities.

3. The model achieves superior performance over several baselines on molecular property prediction tasks across multiple datasets from MoleculeNet.
summary: This paper addresses challenges in graph-based molecular representation learning. They propose MMFRL, which enhances embedding initialization through multimodal pre-training and investigates fusion at various stages. MMFRL aims to improve downstream performance on molecular property prediction tasks by leveraging relational learning and multimodal fusion.
weaknesses: 1. In terms of multimodal fusion, previous works [1,2] have introduced methods for multimodal molecular graph-text pre-training, with one of them [2] also incorporating SMILES strings. These models demonstrate strong performance not only on molecular property prediction tasks but also on a range of multimodal downstream tasks. The authors should also compare their approach with these existing models for a more comprehensive evaluation.

2. Why do models with different stage fusions perform differently across datasets? The authors need to explain this.

3. typos: line 204: themto

Meta-review: The paper introduces a method for relational learning and multimodal fusion in molecular property prediction. However, reviewers raised significant concerns regarding the novelty, experimental design, and completeness of the work. While the modification to the loss function is presented as novel, it does not sufficiently distinguish itself from prior methods, such as MoleBERT. The multimodal fusion strategies are discussed, but the paper lacks clarity on when each fusion stage is most effective. Additionally, comparisons with important state-of-the-art models, like Uni-Mol, and the evaluation on diverse tasks and datasets are missing. The paper also lacks code and checkpoints for reproducibility. Furthermore, the experiments are limited, focusing primarily on small datasets like MoleculeNet, which may lead to inconsistent results. The paper's structure and presentation also need improvement. These issues led to a decision to reject the submission.

Decision: Reject