CL-MFAP: A Contrastive Learning-Based Multimodal Foundation Model for Molecular Property Prediction and Antibiotic Screening
Authors: Gen Zhou
Sugitha Janarthanan
Yutong Lu
Pingzhao Hu
Venue: ICLR 2025 Poster (2025)
Keywords: Contrastive Learning
Multimodal Foundation Model
Antibiotic Property Prediction
Bi-level Routing Attention
Transformer
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Due to the rise in antimicrobial resistance, identifying novel compounds with antibiotic potential is crucial for combatting this global health issue. However, traditional drug development methods are costly and inefficient. Recognizing the pressing need for more effective solutions, researchers have turned to machine learning techniques to streamline the prediction and development of novel antibiotic compounds. While foundation models have shown promise in antibiotic discovery, current mainstream efforts still fall short of fully leveraging the potential of multimodal molecular data. Recent studies suggest that contrastive learning frameworks utilizing multimodal data exhibit excellent performance in representation learning across various domains. Building upon this, we introduce CL-MFAP, an unsupervised contrastive learning (CL)-based multimodal foundation (MF) model specifically tailored for discovering small molecules with potential antibiotic properties (AP) using three types of molecular data. This model employs 1.6 million bioactive molecules with drug-like properties from the ChEMBL dataset to jointly pretrain three encoders: (1) a transformer-based encoder with rotary position embedding for processing SMILES strings; (2) another transformer-based encoder, incorporating a novel bi-level routing attention mechanism to handle molecular graph representations; and (3) a Morgan fingerprint encoder using a multilayer perceptron, to achieve the contrastive learning purpose. The CL-MFAP outperforms baseline models in antibiotic property prediction by effectively utilizing different molecular modalities and demonstrates superior domain-specific performance when fine-tuned for antibiotic-related property prediction tasks.

Published as a conference paper at ICLR 2025
CL-MFAP: A CONTRASTIVE LEARNING-BASED MULTIMODAL FOUNDATION MODEL FOR MOLEC- ULAR PROPERTY PREDICTION AND ANTIBIOTIC SCREENING
Gen Zhou'; Sugitha Janarthanan'; Yutong Lu², Pingzhao Hu Westtrnn University, London, ON, Canada
2 University of Toronto, Toronto, ON, Canada
ABSTRACT
Due to the rise in antimicrobial resistance, identifying novel compounds with an- tibiotic potential is crucial for combatting this global health issue. However, tra- ditional drug development methods are costly and inefficient. Recognizing the pressing need for more effective solutions, researchers have turned machine learning techniques to streamline the prediction and development of novel an- tibiotic compounds. While foundation models have shown promise in antibiotic discovery, current mainstream efforts still fall short of fully leveraging the poten- tial of multimodal molecular data. Recent studies suggest that contrastive learning frameworks utilizing multimodal data exhibit excellent performance in represen- tation learning across various domains. Building upon this, we introduce CL- MFAP, unsupervised contrastive learning (CI.)-based multimodal foundation (MF) model specifically tailored for discovering small molecules with potential antibiotic properties (AP) using three types of molecular data. This model em- ploys 1.6 million bioactive molecules with drug-like properties from the ChEMBL dataset to jointly pretrain three encoders: (1) transformer based encoder with ro- tary position embedding for processing SMILES strings; (2) another transformer- based encoder, incorporating novel bi-level routing attention mechanism to han- dle molecular graph representations; and (3) a Morgan fingerprint encoder using a multilayer perceptron, achieve the contrastive learning purpose. The CL-MFAP outperforms baseline models in antibiotic property prediction by effectively uti- lizing different molccular modalities and demonstrates superior domain- specific performance when fine-tuned for antibiotic-related property prediction tasks.
INTRODUCTION
Bacteria play pivotal role in diverse array of diseases within the human body, serving either the primary cause or a contributing factor. A promising and sometimes sole treatment for these diseases is antibiotics, a specialized class of drugs designed target pathogenic bacteria. Despite advancements, a lack of antibiotics for many pathogenic bacteria persists, and antibiotic resistance allows bacteria to survive once effective treatments. Consequently, there is a pressing demand for the continual development of antibiotics. However, traditional antibiotic discovery faces two major issues: 1) it is extremely costly and 2) it is very cone-consuming. Artificial Intelligence (AI) and Machine Learning (ML) methods can combat these issues and thus, have been employed over the past couple of years to aid antibiotic discovery for wide range of conditions. Deep learning (DL) tools including convolutional, recurrent, and graph neural networks have been leveraged to explore high-dimensional data and design compounds with desired antibiotic properties (Cesaro et al.) 2023).
Large Language Models (LLMs) have increasingly stood out in recent years due to their exceptional performance, garnering widespread attention. As such, they have been implemented and fine-tuned
"Equally Contributed
Contact Author: phu49@uwo.cr

---

Published as a conference paper at ICLR 2025
to target pathogenic bacteria. For an LLM dedicated to the domain of antibiotic discovery, utiliz- ing an extensive general molecular dataset for model training may not be a computationally cost- effective choice. By employing domain-sppcifff training, the model can be taught to learn the unique characteristics, patterns, and nuances relevant to the field. Gu et al.] (021) support this assertion, arguing that for fields like biomedicine, which have a large amount of unlabeled text, pretraining model from scratch yields greater benefits than continual pretraining of general-domain LLM.
Contrastive learning, an effective method for utilizing large amounts of unlabeled data, has made significant progress in the field of ML in recent years. For antibiotic-related property prediction, contrastive learning significantly enhances model performance. Rather than relying on limited la- beled molecular property data, this method leverages the vast amount of unlabeled molecular data available, helping identify patterns that contribute to a compound's properties. The resulting molec- ular representations are thus more robust as they include patterns that may be missed by traditional supervised learning approaches. This leads to more accurate predictions, better generalization to novel chemical spaces, and ultimately increases the success rate of identifying potential antibiotic candidates.
In this study, we introduce novel approach to streamline antibiotic discovery by leveraging con- trastive learning framework with multimodal data to train a domain -secciiiss LLM. We propose CL-MFAP, unsupervised contrastive learning (CL)-based multimodal foundation (MF) model specifically tailored for discovering small molecules with potential antibiotic properties (AP). CL-- integrates transformer-basee encoder with rotary position embedding for SMILES strings, transformer-based encoder using a novel Bi-Level Routing Attention (BRA) mechanism for molec- ular graphs, and multilayer perceptron for Morgan fingerprint embeddings. This model is pre- trained on 1.6 million bioactive molecules with drug-like properties from the Chemical Database of Bioactive Molecules (ChEMBL) (Gaulton et al., 2011), a smaller, domain -specffcc dataset. Our comprehensive evaluation demonstrates that CL-MFAP outperforms baseline models trained on large-scale general datasets for antibiotic property prediction, while also exhibiting superior domain- specific performance when fine-tuned on targeted downstream tasks.
2 RELATED WORK
Transformers. Among the current mainstream LLMs, the most representative architecture is the transformer. A transformer is a DL architecture consisting of encoder and/or decoder components built around multi-head attention mechanisms. Different models in the transformer family use these components distinctively: Bidirectional Encoder Representations from Transformers (BERT) series based solely on the encoder (Devlin et al., 2018). the Generative Pre-trained Transformer (GPT) series based solely on the decoder (Radford & Narasimhan) 2018), and the Text-to- Text Transfer Transformer (T5) series utilizing both the encoder and decoder (Raffel al. 2020). The architec- ture's 's core features are self-attention computation and positional encoding Vaswani et al.) 2017). The former is used to capture the semantic dependencies between the target word and the context and then determine its importance, while the latter understands the syntax and sequence information of the word by recording its position in the sequence. L.MSs based on the transformer architecture have been widely proven to exhibit superior performance in capturing sequence semantics.
LLMS for Molecular Property Prediction. LLMs have recently gained popularity in molecu- lar property prediction due to their enhanced success. MoLFormer is a successful unsupervised rransformer-based LLM that accurately captures sufficient chemical and structural information to predict a diverse range of chemical properties (Ross et al. 2022). ChemBERTa is stack of bidi- rectional encoders that uses representations from transtormers for molecular property prediction (Chithrananda et al. 2020) and is fine-tuned to better predict drug-target interactions (Kang et al.) 2022). MolBERT is self-supervised model, consisting of the bidirectional attention mechanism- based BERT architecture (Fabian et al. (020). It is one of the most efficient pre-trained models for molccular property prediction that can be easily generalized to different molecular property predic- tion tasks via fine-tuning. All these examples of successful LLMs take in the structure of compounds in Simplified Molecular Input Line Entry System (SMILES) format for predictions.
Contrastive Learning Models for Molecular Representation Learning. As the field of drug de- velopment continues to advance, the integration and utilization of multimodal data have become essential for improving the performance of molecular property prediction LLMss. Contrastive learn-
2

---

Published as a conference paper at ICLR 2025
PRETRAINING
DOVNSSREAM PROPERTY
PREDICTION TASKS
Rotary Positional Embedding
Anti-E. coli Activity
Blood-Brain Burrier Penetration
SMILES
(12 luyurs)
Chaones
Aochvaation
million motecules
(6.0.1.0..........00))
Lincr
(9,1,0,9,1,.11,, 1,0,0)
Morgan 2-radius
2048-bit Fingerprints
Muttilayer Perceptron
Contmative Laarning
p-secretase Binding
Bioavallability
Transformer
Grapha
Encoder (12 Layora'
Bi-Level Routting
Cell Humbrrna
Figure 1: Illustration of the proposed approach.
ing can enhance a model's feature extraction capabilities by learning different representations of molecular data the absence of labeled data. For example, MoICLR employs three distinct molec- ular graph augmentations to achieve contrastive learning, significantly improving the model's ability learn molecular representations (Wang al.) 2022). UniCorn combines several pretraining meth- ods: 2D graph masking, 2D-3D contrastive learning, and 3D denoising, to depict molecular views from three different levels, resulting in superior performance compared traditional models (Feng
et al. 2024
PROPOSED APPROACH
3.1 MODEL DEVELOPMENT
We designed multimodal contrastive learning model based on molecular SMILES, Morgan finger- prints, and molecular graphs to comprehensively capture different chemical characteristics. The input data is SMILES representations, which describe the linear form of a molecule, including information about its composition, bond types, and functional groups, used to depict the overall connectivity of the molecular structure. From the SMILES representation, Morgan fingerprints and molecular graphs are constructed. Morgan fingerprints provide a quantitative representation of the molecule's 's features, encoding its structure as high-dimensional binary vector that captures the pres- ence and distribution of various substructures and functional groups. Specifically, a radius size of was determined through ablation studies detailed in Appendix A.1 (Table AT). and 2048-bij bit vec- tor was chosen as it large enough minimize hash collisions (where different structural features map to the same bit) while being computationally efficient. The graph representation of molecule describes its topology through nodes (atoms) and edges (chemical bonds), including details about the atom types, bond characteristics, and overall connectivity. Ablation studies were performed to validate the inclusion of these three modalities, detailed in Appendix A.2 (Table A2). Altogether, the model receives widespread in-depth representation of each compound, allowing it to learn the specificities and patterns of the compounds that influence their antibiotic-relate properties.
Figure illustrates the overall architecture of our model, which learns the three molecular feature modalities mentioned above through different embedding pathways. First, the model employs a munformer-based graph encoder with novel bidirectional relation aggregation (BRA) mechanism
3

---

Published as a conference paper at ICLR 2025
to learn the molecular graph features. Second, the transformer encoder with rotary positional em- bedding is used to learn the SMILES features of the molecule. This self-attention-bart based encoder excels at capturing global information in sequential data and handling complex contextual depen- dencies. Finally, to encode Morgan fingerprints, we use a multilayer perceptron (MIP), a classical feedforward neural network capable of processing high-dimensional data and extracting complex features.
3.1.1 ROTARY POSITIONAL EMBEDDING
Rotary Positional Embedding (RoPE) is an improved positional encoding method used trans- former models (Su et al. 2024). The rotation transformation effectively integrates positional infor- mation into each token and helps the model capture dependencies between distant tokens. Together, this preserves the relative position relationships between elements and improves prediction accuracy. This method is particularly suitable for processing molecular data with complex structural depen- dencies, as improves the model's ability understand sequential structural relationships. In the two-dimensional case, the formula for implementing rotary positional encoding through complex multiplication is as follows:
g In, m n) Re (MV )* e sim-njj
where and n are positions, y is the query, h is the key, W, is the query projection matrix, Wa the key projection matrix, Re[-] is the real part of a complex number, (Wzum) represents the complex conjugate, and (æm, zn.) denotes the representation in two-dimensional coordinate sys- tem. Through this rotation formula, rotational transformation is achieved, generating the rotary positional encoding. The original linear attention formula is expressed as follows, where up-) and ((+) are typically non-negative functions:
Attention Q, V)m E @(am)² (kknm)))(kkk)))) Em=1 ((gm)) )' (km)
where Un is the value of nth token, Im is the query, and kn is the key.
Combining both equations 1 and 2 gives equation 3. RoPE injects positional information through rotation, which keeps the norm of hidden representations unchanged. Thus, RoPE is combined with linear attention by multiplying the rotation matrix with the outputs of the non-negative function as follows:
Attention(4 @2 K, V) C (² (KK)))))(
Σ (4m )r p(k))
where Re is the orthogonal matrix used in RoFormer the process of encoding position information.
(Su et al 2024 that ensures stability during
3.1.2 BI-LEVEL ROUTING ATTENTION
The Bi-level Routing Attention (BRA) mechanism crucial, as it partitions the attention mechanism into two phases: an initial focus on global relationships followed by a more detailed scrutiny of local specifies. In conventional applications within computer vision, the BRA mechanism first identifies critical areas within an image and then focuses on local details. For instance, in an image featuring a dog, the model would initially identify the most prominent features, such as the dog's head, across the entire image, and then subsequently focus on local details such as the eyes and nose within the defined window.
In molecular graphs, diverse structural features are exhibited by different molecules, and these fea- tures significantly influence the functional performance of the molecules. For antibiotic molecules, complex cyclic structures represent a typical characteristic, the importance of which often surpasses other local structures in medicinal functionality, making precise understanding by the model crucial. In our model, through the Window-to- Window Attention mechanism of BRA, the model efficiently identifies and focuses on key structures and functional groups within the molecular graph that are central to functionality, such cyclic structures. Concurrently, for peripheral structures or less likely
4

---

Published as a conference paper at ICLR 2025
node-edge combinations that have minimal impact molecular functionality, the model minimizes their importance or filters them out through a dynamic adjustment mechanism, thereby achieving a clear prioritization in feature learning.
The BRA mechanism has been proven effective in handling long-range dependencies in images within the field of computer vision, and the same theory applies molecular graphs (Dong al.] 2023). Compared to the traditional approach graph transformers which use classical attention, the BRA first filters irrelevant key-value pairs coarse regional level, significantly reducing the number of potential interactions that need to be considered in the subsequent fine-grained token-to- token attention phase. This two-step filtering process ensures that attention is focused on areas most relevant to the query, enhancing the model's ' ability to manage long-range dependencies without the computational overhead attending all token pairs.
Window-to-Window Window Level Routing. This mechanism efficiently computes attention across regions of feature map while considering local context. Beginning with 2D feature map. e H #xxXXX XC a linear transformation is applied to create three tensors: Q (query), K (key), and V (value), as shown in Equation
Q = XW K XW,, V = XW.
where Wg WR. and W. are the learnable projection weights, each of size RCC.
To perform Hindow-to-windee
level routing, the feature map is divided into S x S non-overlapping windows, each containing HW feature vectors, resulting in reshaped Q' K' and V The window 52
size S is set to 7, based ablation studies explained in Appendix A.1 (Table [33]. Within each window, the Q' K", and V' tensors are used to compute the average, resulting in Q" and k which are the window- level representations for each non-overlapping window. These then used to calculate the sorrddo-towwiidor score matrix (containing window-to-window attention scores) shown in Equation 5.
A Q (K")?
In the score matrix, each row contains the indexes of the top-k windows that relevant the corresponding window.
Pixel-to-Pixel Level Attention. For window T, its top-k relevant windows scattered across the feature map. To gather these windows together, we use the following equation to collect K and V9:
K = gather( 1"), V = gather(VV ("))
(6)
K29 and 1 represent the collected Key and Value tensors containing features from the top-k win- dows relevant to the current window 1, respectively. For given pixel within a window the pixel will attend to all pixels in the top-k windows most relevant to window /. This ensures fine-grained attention mechanism, allowing the model to refine feature representations at the individual pixel level.
Algorithm Bi-Level Routing Attention
1: #Graph:
2:
node + Prosesp (mirieeStiii 3: graphNodee Feature concatenateegroppTTTkeee FaakkenFFaaaii Feature, node Feature)
4: node Fadeeeettreaaaee aatii +- creaceNodeFFcatureMatiiss (wrriiigggrrrNNooo ode Feature)
5: #Bi-Level Routing Attention:
6: Srindow-to-Window Level Routing:
7: windows& t Wineoiioooodoons(nodd
Www.ddwwwwddddeeeeeeeeeaaaaaa eature
8: distonces + calclaatDissancseeeeenni Windows( Windows(windous)) wwnddws))
9: top Windws +- weleer/opp
distances, k)
10: #Pixel-to-Pixel Level Attention:
11: InterntiomEEmedlleee + @ateberrpppeeeleeettcttttttttooo
Algorithm 1 presents the basic architecture of the Bi-Level Routing Attention (BRA) algorithm, including the processing of input data and the implementation logic of BRA. To our knowledge, this
5

---

Published as a conference paper at ICLR 2025
is the first time BRA has been introduced into the attention mechanism for processing molecular graphs. We utilize a transformer based graph encoder, equipped with 8 attention heads and 12 encoder layers, configuration particularly suited for analyzing and interpreting complex molecular structures (Ying et al. 2024).
3.1.3 MULTIMODAL CONTRASTIVE LEARNING
The advantage of a multimodal model lies in its ability to integrate information from different modal- ities, thus obtaining a more comprehensive understanding of molecular structure that enhances the robustness and generalization of the model. Contrastive learning is an approach that enhances fea- ture learning by pulling similar pairs closer together while pushing dissimilar pairs apart. This approach significantly improves representation quality as it facilitates learning similarities and asso- ciations across different modalities. also aids the limited data issue commonly associated with antibiotic property discovery by leveraging the unlabeled molecular data available.
Algorithm 2 Multimodal Contrastive Learning
1: function CONTLEAANNIGGMOOE.... Batch, fpBatch, graphBatch) smilesOutput + (mmilleefnccodersssslsssesese fpOutput - FpEncoder((pBaachh) if BiGrraphormerEnncoder with MPNN then
2:
3:
4:
5:
graphOulppul +- MePPNNNEncoderggaaahaBaaaa + BGGaphemeeFFooodeegggahaaaa
6:
else if BiGraphormerrEncodr without MPNN then
7:
graphOutput + heterrpppoommcnnoooooggrraaaeaae
8: else if BiGrapphormerEncoode without Bi-level routing attention then
9: graphOutput + (ara + GraphomeeFEnodder(szppBaua
10: end if
11: return SytiiesSOuuuu,, graphOOppe
12:
end function
function COMPUTELOSS CoxPuTEELosssmilleeouutu.. fpOutput, graphOutput)
14:
/oLoss Function (Initial Weight w1, w2, w3)
lossSmiles F < NT-Xent( smilesOOtput, fpOutput)
16:
LossSmilesGraph + NT-Xent( (aaiitsmiiessust
graphOutput)
17:
LossPPGraphl + NT-Xentí feentfpOOlppt, graphOulput
18:
Losa w1+ LossSmiles.FP + W2 lossSmilesCraph + wy IoroPh
19:
return Loss
20:
end function
Algorithm illustrates the basic architecture and loss computation of the multimodal contrastive learning model. In our model, SMILES, Morgan fingerprints, and molecular graphs encoded using dedicated encoders and the representations are then processed through a contrastive learning framework, using Normalized Temperature-Scater Cross-Entropy (NT-Xent) loss function to compare pairs across modalities (Equation Y) et al,, (020). NT-Xent Loss learns well-distributed feature representations by maximizing the similarity of similar samples (pos- itive pairs) and minimizing the similarity of dissimilar samples (negative pairs). The function takes the concatenated vectors of two modalities for two molecules as input and calculates the loss for each pair of modalities. For example, for SMILES and molecular graphs, we first compute the con- catenated vector of the SMILES embedding and the graph embedding, then compute similarity matrix for all pairwise samples for the two modalities, and finally calculate the loss between two modalities. To enable the use of NT-XXent loss with different modalities, we project the represen- tations from different modalities into the same vector space. In each iteration, different modalities of the same molecule are treated as positive pairs, while representations from different molecules are treated negative pairs. NT-Xent loss is advantageous as it effectively measures the similar- ity between high- high-dimensional embeddings from different modalities, emphasizing the alignment of directions rather than absolute values, which is crucial for robust multimodal learning.
the fundamental
exp( sim(z.a))
Le -log Σ expl sim((x.)))
(7)

---

Published as a conference paper at ICLR 2025
The total loss is defined in Equation 8, where and j represent two different molecules, and and n denote different data modalities. For each modality pair, we assign a weight, and the total loss is calculated the weighted sum of these individual losses.
L=
Σ Wmn Σ (Le + Fin, jim xjn) ) Lo(Tim + Din, o' + ('nn))
(8)
3.2 PRETRAINING PROCESS
The ChEMBLL24 database was downloaded after the removal of salts, charge neutralization, removal of molecules with SMILES strings longer than 100 characters, re- moval of molecules containing any element other than H. B. C, N, O. F, Si, S, CI, Se, Br, and I, and removal of molecules with larger ECFP4 similarity than 0.323 compared to holdout set consisting of marketed drugs (celecoxib, aripiprazole, cobimetinib, osimertinib, troglitazone, ranolazine, thiothixene, albuterol, fexofenadine, mestranol) (Gaulton al., 2011) (Fiscato et al.) 2018). Pre-processing was then applied to the raw molecular data, which included de-duplication, normalization via conversion to canonical SMILES using RDKit (rdk). and removal of entries with over 123 tokens, as these molecules are exceedingly rare in practical applications (Ross et al. [2022). After processing,
Dataset and Pre-processing.
obtained 1,591,020 SMILES for model training. The preprocessed data was divided into 80%- 10% 10% for training, validation, and testing, respectively. Given the input data of SMILES strings, CL-MFAP generates Morgan fingerprints and molecular graphs using RD- Kit (rdk) and all three types of data are subsequently used to train the model. Pretraining CL-MFAP using ChEMBL was validated via ablation studies detailed in Appendix A.I [Table/A4]
Domain-specific. Our target domain contains bioactive molecules with drug-related compounds from ChEMBL, whereas other large-scale databases, such PubChem, typically include much more widely used, commercially available molecules (Lyubishkin et al. (002) (Kim et al. 2016).
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Environment. implementations were conducted on the PyTorch platform using an NVIDIA A100 GPU. All models were trained using learning rate of le-4, over 20 epochs, with batch size and 4 worker processes. The Adam optimizer and gradient clipping were also applied during training. limiting the gradient norm 1.0. For the bi-level routing attention, the window size is 7, the number of top k windows is 4, and there are 16 pixels per window and attention heads.
Pre-trained CL-Models. To analyze the contribution of each component along the molecular graph embedding grath transformer encoder (GTE) and the newly introduced BRA-a as well as to test whether combining this GTE with mnssagge-pssing neural network (MPNN) can further enhance the model's ability to capture global information, we pre-trained five models within the overall framework of multimodal contrastive learning which differ in structural configurations along the graph embedding path. Aside from CL-MFAP, the other four models are labeled as Contrastive Learning Baseline 1-4 (CL-BL1-4). The labels and structures of all the models pre-traines under the multimodal contrastive learning framework are presented in Table [
Table 1: Proposed pre-trained models with different graph embedding paths
Model Name
Structural Configuration
Graph Embedding Description
CL-MFAP
Proposed Model
GTE BRA
CL-BL1
CL-MFAP w/ MPNN
GTE BRA MPNN
CL-BL2
CL-MFAP MPNN w/o BRA
GTE MPNN
CL-BL3
CL-MFAP w/o BRA
GTE
CL-BL4
CL-MFAP w/ MPNN w/o BRA w/o GTE
MPNN
7

---

Published as a conference paper at ICLR 2025
Model Size. Moreover, measured the size of models in terms of Params and FLOPs to further evaluate their performance and cost efficiency. Params refer to the number of trainable parameters in model. This is directly related to the structure of the model, representing each learnable weight, including weights and biases in different layers. As such, it serves as a measure of the model's complexity and storage requirements (Han al. 2024). FLOPs refer to the number of floating-poin operations performed during single forward pass of the model. This metric measures the computational complexity and cost of the model, providing insight beyond just the number of parameters. FLOPs are closely related to the model's inference speed and the computational resources required for its operation (Han et al., 2024).
4.2 DOWNSTREAM PROPERTY PREDICTIONS
Datasets. Six datasets were used for downstream property prediction: MIC activity against E. coli
(E. coli MIC) dataset curated from COADD database (Desselle al., 2017). MIC activity against
H. influenzae (H. influenzae MIC) dataset curated from ChEMBL database Gaultoon et al.) (011),
BACE (Wu al.) (0188). Blood-Braad Barrier Penetration (BBBP) (Wu et al. [0188). Parallel Arti- ficial Membrane Permeability Assay (PAMPA) (Siramshetty et al. 2021). and Bioavailability (Ma)
al. 2008). All datasets were divided into 80% 10% 10% for training. testing and validation,
respectively. More details can be found in Appendix A.1.
Baseline Models. We selected MoLFormer, ChemBERTa-2, MolBERT, MoICLR, and FP-GNN as baselines to evaluate the performance of CL-MFAP. MoL a transformer-basee model, was trained on 1.1 billion molecules from ZINC and PubChem databases (Ross et al., 2022). ChemBERTa-2,
BERTTbassel L.M with 12 encoders, was trained on 77 million PubChem com- pounds (Ahmad et al. 2022). MolBERT, another BERT-based model with encoders, was trained on smaller dataset of 1.6 million ChEMBL molecules (Fabian et al. 2020). MolCLR employs graph neural networks and contrastive learning with molecular augmentations, trained on 10 million PubChem SMILES (Wang al.) 2022). Finally, FP-GNN is multimodal framework that combines molecular graphs and fingerprints for property prediction (Cai et al. 2022))
Mean Reciprocal Rank. To more intuitively evaluate the overall performance of each model across all downstream tasks, we employed the mean reciprocal rank (MRR) method, a statistical approach that synthesizes the rankings of all models on various downstream tasks (Wu et al. (011). This method assigns a corresponding score to each model, with higher scores indicating superior overall performance. We first recorded the rank of each model's ROC-AUC metric in comparison to the other models for each task and then used the ranks to calculate the model's MRR value using the following equation:
Q
1
MRR
(9)
Q
rank;
where denotes the task index and Q represents the total number of tasks.
4.3 RESULTS
The performance of CL-MFAP on downstream property prediction tasks was compared against all baselines. Using Area under the Receiver Operating Characteristic Curve (ROC-AUC) as the eval- uation metric, the experimental results are summarized in Table 2 Notably, CL-MFAP outper- forms all other baseline models on the E. coli MIC dataset AUOC-AUC: 0.85440.037), which particularly relevant for antibiotic drug discovery as it predicts the antibacterial activity of com- pounds against E. coli. In addition, performs second best on the H. influenzae MIC dataset IROC-AUC:0.844+00011 with negligible difference from the best performing model, MoLFormer RROC-AUC:0.576:000017 We noted similar performance for pre-trained chemical language models (CL-MFAP, FoLForrer, MolBERT, and ChemBERTa-2) that outperform models without pretrain- ing (MoICLR and FP-GNN). Together, these results show the ability of CL-MFAP to exceed in antibacterial activity prediction, regardless of sample size. Thus, our model can also predict tibacterial activity for less studied bacterial strains with less data. On the remaining datasets, our model demonstrates consistently strong performance, ranking among the top or 3 models, unlike
8

---

Published as a conference paper at ICLR 2025
other baselines that excel in only 1-2 datasets. This highlights the robustness and generalizability of CL-MFAP across diverse tasks.
When ranked by MRR scores, CL-MFAP significantly outperforms the other models (Figure 2). The elevated MRR scores underscore the model's i superior overall performance, reaffirming its effective- ness and broad applicability.
Table 2: ROC-AUC of CL-MFAP vs. baseline models on downstream property prediction datasets
coli MIC
influenzae MIC
Bioavail- ability
Model
BBBP
PAMPA
BACE
CL-MFAP
0.85+0.04
0.87=0.02
0.93+0.01
0.76#0.03
0.6010.03
0.88+0.01
MoLFormer
0.71++0.01
0.88+0.02
0.93+0.01
0.72+0.03
0.72+0006
0.87+-0.02
MolBERT
0.77+-0.00
0.87+0.03
0.97+0.01
0.73+0.05
0.75+-0.08
0.89+00022
ChemBERTa-2
0.74+10.03
0.86+1002
0.9740.01
0.6710.03
0.70=0.07
0.01
MoICLR
0.71
0.8610.02
0.93:0.01
0.7610.02
0.6310.16
0.8610.01
FP-GNN
0.75++0002
0.87=0.02
0.94+0.01
0.75+0.01
0.75+0.04
0.87+0001
Mean Reciprocal Rank Performance
Models
3.5
3
2.5
2
0.5
CL-MFAP
MoLFomer
MoIBERT
ChemBERTo-2
MoiCLA
FP-GNN
Figure 2: Mean reciprocal rank (MRR) of the average performance for CL-MFAP versus baseline models. CL-MFAP demonstrates superior overall performance.
4.4 ABLATION STUDIES
Overall Performance Ranking of CL-based Models. We compared the performance of five pre- trained CL models verify the effectiveness of different components in the graph embedding path. We evaluated the performance of these pre-trained CL models on the downstream tasks using ROC- AUC (Table A5 and then ranked the performance of each model across all tasks based on these findings. As shown in Table 3 CL-MFAP outperforms the model variations in 5 out of down- stream tasks. To further assess model performance and cost efficiency, an MRR analysis of the overall model rankings was performed. The model size, represented via Params (Figure 33) and FLOPs (Figure BB), was plotted against MRR score. CL-MFAP's top-left position in Figure 3 highlights its superior performance with fewer parameters.
Ablation study on the BRA. We conducted an ablation analysis on the contribution of BRA by comparing CL-MFAP vs. CL-BL3, and CL-BLI vs. CL-BL2. The former compares the impact of BRA in the absence of MPNN, while the latter compares the effect of BRA when MPNN and GTE used together. In both cases, models with BRA consistently outperform their counterparts Figure 3) Therefore, BRA plays a significant role in enhancing model performance.
(Table 3
Ablation study the MPNN. The value of MPNN was also evaluated. As initially hypoth- esized that introducing MPNN could help further capture comprehensive information (Cai et al.) 2023 we introduced MPNN path running parallel to GTE in the graph embedding process.

---

Published as a conference paper at ICLR 2025
Table Overall performance ranking on downstream property prediction datasets for all pre-trained CL models
E. coli
H. influenzae
Bioavai- lability
Model
BBBP
PAMPA
BACE
MIC
MIC
CL-MFAP (GTE, BRA)
1
1
1
5
CL-BLI (GTE, BRA, MPNN)
3
4
2
2
CL-BL2 (GTE, MPNN)
4
2
4
3
1
4
CL-BL3 (GTE)
2
3
5
4
CL-BL4 (MPNN)
5
5
5
4
3
5
A
Mean Rediprocal Rank ROC-AUC vs Param
Mean Recipracal Rank ROC-AUC vs Flops
Moean Ceripracaa Nank ROC-AUS
5.0
Model
MFAP
5.0
Model
4.5
CL-3L1
45
CL-BL1
C1-912
C--112
CL-3L3
CL-0L3
2.0
2.0
1.5
1.5
Param//)
Hlopstts)
Figure 3: Mean reciprocal rank (MRR) of the ROC-AUC rankings for all CL models on downstream property prediction datasets plotted against (3A) Params, and (3B) FLOPs. Models closer the top left corner demonstrate better performance with fewer parameters (3A) and lower FLOPs (3B).
However by comparing the results of CL-MFAP vs. CL-BL1 and CL-BL3 vs. CL-BL2, introducing MPNN weakens the performance of the model and thus was not incorporated (Table Figure 3.
Ablation study on the GTE, We also analyzed whether GTE is replaceable. A comparison between CL-BLL3 and CL-BL4 shows that replacing GTE with MPNN for molecular graph encoding signifi- cantly decreases model performance. Also, when comparing CL--BL2 and CL--BL4, despite MPNN weakening the performance GTE, the combination of GTE and MPNN still outperforms MPNN alone (Table Figure 3 Thus, GTE is essential for encoding molecular graphs in model.
addition, additional ablation analyses (to analyze the effects window size, data modalities, pre- training CL-MFAP, and Morgan fingerprint radius model performance), Representation-Prooee Propertt Relationship Analysis (RePRA), and a case study were performed, detailed Appendix A.1 (Ta- A.4 (Table A6 Figure AT), and A.5 (Table A7 A8). respectively.
ble A1
A4
CONCLUSION
In this work, we present CL-MFAP, novel multimodal contrastive learning framework. The model combines and compares molecular information from three modalities SMILES, molecular graphs and fingerprints to efficiently learn representations of molecules that improve its performance in predicting antibiotic-related properties. We also. for the first time, incorporate the BRA mechanism enhance the quality of molecular representation learning. Experimental results demonstrate that CL-MFAP achieves outstanding performance in predicting drug molecule properties. In the future, aim to integrate this model with other cross-domain potential modules and further refine its multimodal contrastive learning algorithm to enhance its generalization capabilities.
All code be found at wttpt:tt///.mm /ttp:://///tt..bb.t/ com CLMFAP CLMFAP
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research was supported in part by the Canadian Institutes of Health Research (PLL185683, PJT 190272), the Canada Research Chairs Tier II Program (CRC-2021-00482), the Natural Sciences and Engineering Research Council of Canada IRGPIN-202) 100002)) and the Canada Foundation for Innovation (CFI) John R. Evans Leaders Fund (JELF) program (grant #43481).
special thanks to Yan Yi Li and Zihao Jing for their help in this work.
REFERENCES
Rdkit: Open-source cheminformatics. URL https: www.www.i.....
Walid Ahmad, Elana Simon, Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. Chemberta-2: Towards chemical foundation models, 2022. URL
abs/ 01712
Chen Cai, Truong Son Hy, Rose Yu, and Yusu Wang. On the connection between mpnn and graph transformer. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
Hanxuan Cai, Huimin Zhang, Duancheng Zhao, Jingxing Wu, and Ling Wang. FP-GNN: a versatile deep learning architecture for enhanced molecular property prediction. Briefings in Bioinformat- ics, 23(6), 2022. doi: 10. 10.1093/bib/bbac400.
Angela Cesaro, Mojtaba Bagheri, Marcelo Torres, Fangping Wan, and Cesar De La Fuente-Nunez. Deep learning tools to accelerate antibiotic discovery. Expert Opin Drug Discovery, 18(11); 1245 1257, 2023, doi: 10.1080/14460441 2222 222.222177...
Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. Chemberta: Large-scale self- supervised pretraining for molecular property prediction. CoRR, abs/2010.09885, 2020. URL
http:s://aaxxv..ooto abs 200002010.oo 09885
Mathilde R. Desselle, Ruth Neale, Karl Hansford, Johannes Zuegg, Alysha Elliott, Matthew A. Cooper, and Mark A. Blaskovich. Institutional profile: Community for open antimicrobial drug discovery crowdsourcing new antibiotics and antifungals. Future Science OA, 3(2)FFO017,, 2017. doi: 10:4155/isoa-2216-0022
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, 110.0180, 2018. URL
http:://axxit org/ abs 010. 04805
Kun Dong. Jian Xue, Xing Lan, and Ke Lu. Biunet: Towards more effective unet with bi-level routing attention. In 34th British Machine Vision Conference 2023, BMVC 2023, Aberdeen, UK, November 20-24, 2023. BMVA, 2023.
Benedek Fabian, Thomas Edlich, Héléna Gaspar, Marwin H. S. Segler, Joshua Meyers, Marco Fiscato, and Mohamed Ahmed. Molecular representation learning with language models and domain-relevan auxiliary tasks. CoRR, abs/2011.13230, 2020. URL httpt:::////t atttps://axxiv.ory) abs 12201.112211
Shikun Feng, Yuyan Ni, Minghao Li, Yanwen Huang, Zhi-Ming Ma, Wei-Ying Ma, and Yanyan Lan. Unicorn: A unified contrastive learning approach for multi-view molecular representation learning, 2024. [RL/:tts:://aatt
org. abapg//bb/24400 /2405 2201/11150333333333333 10343
Marco Fiscato, Alain C. Vaucher, and Marwin Segler. GuacaMol All SMILES. 11 2018. doi: 10.6084/m9. wwt.:ffssaaaee22222222222222 URL
GuacaMo All -__SSMSSS//2225555
Anna Gaulton, Louisa J. Bellis, A. Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, and John P. Over- ington. Chembl: a large-scale bioactivity database for drug discovery. Nucleic Acids Research, 40:D1 100- 2. doi: 10. 1093//aa//kr7777

---

Published as a conference paper at ICLR 2025
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing, ACM Transactions Computing for Healthcare (HEALTH), 3(1):
1-23, 2021. doi: 10.1145/3458754.
Kai Han, Yunhe Wang, Jianyuan Guo, and Enhua Wu. Parameternet: Parameters are all you need for large-scale visual pretraining of mobile networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 15751-15761, June 2024.
Hyeunseok Kang, Sungwoo Goo, Hyunjung Lee, Jung-woo Chae, Hwi-yeol Yun, and Sangkeun Jung. tine-tuning of bert model to accurately predict drug- target interactions. Pharmaceutics, 14(8):1710, 2022. doi: 00.3300pharmacceutiss 14081710.
Sunghwan Kim, Paul A. Thiessen, Evan E. Bolton, Jie Chen, Gang Fu, Asta Gindulyte, Lianyi Han, Jane He, Siqian He, Benjamin A. Shoemaker, Jiyao Wang, Bo Yu, Jian Zhang, and Stephen H. Bryant. Pubchem substance and compound databases. Nucleic Acids Research, 44(D1):D1202- D1213, 2016. doi: 10. 10.1093/nar/kkv9511
N. R. Lyubishkin, O. V. Kardash, O. V. Klenina, and T. I. Chaban. Virtual databases for drug discovery. 2022.
Chang-Ying Ying Ma, Sheng- Yong Yang, Hui Zhang, Ming-Li Xiang, Qi Huang, and Yu-Quan Wei. Prediction models of human plasma protein binding rate and oral bioavailability derived by using ga-cg-svm method. Journal of Pharmaceutical and Biomedical Analysis, 47(4):677-682, 2008. doi: (..1016/jjjpba.2000 2a0pa.0000.0000....
Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre- training. 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1), 2020.
Jerret Ross, Brian Belgodere, Vijil Chenthamarakshan, Inkit Padhi, Youssef Mroueh, and Payel Das. Large-scale chemical language representations capture molecular structure and properties. Nature Machine Intelligence, 4(12): 1264, 2022. doi: 10. 10.10090642256-022000550--
Vishal Siramshetty, Jordan Williams, c-Trung Nguyn, Jorge Neyra, Noel Southall, Ewy Mathé, Xin Xu, and Pranav Shah. Validating adme qsar models using marketed drugs. SLAS DIS- COVERY: Advancing the Science of Drug Discovery, 26(10):136611336, 2021. doi: 10.1177/
247255522110175200
Jianlin Su, Murtadha Ahmed, Yu La, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomput, 568(C), March 2024. doi:
(.10166).ncucom.2222
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31 Inter- national Conference on Neural Information Processing Systems, NIPS'17, pp. 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 97815 10860966.
Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence, 4(3):279- 287, 2022. doi: 10. (038//42256-022-0044
Yang Wu, Masayuki Mukunoki, Takuya Funatomi, Michihiko Minoh, and Shihong Lao. Optimizing mean reciprocal rank for person re-identification. In 2011 8th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 408-413, 413, 2011. doi: 10.1109/AVSS. 2011 1.6023663
Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learn- ing. Chem. Sei., 9(2)5113-330, 2018. doi: 10. 10.10//77S002264A.
12

---

Published as a conference paper at ICLR 2025
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie- Yan Liu. Do transformers really perform bad for graph representation? Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS '21. Curran Associates Inc., 2024.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20. Curran Associates Inc., 2020.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 ADDITIONAL ABLATION STUDIES
Ablation study Morgan Fingerprint Radius. We performed additional ablation study investigate the effect of the Morgan fingerprint radius size on CL-MFAP's predictive capabilities. CL-MFAP was tested with five fingerprint radius sizes (0, 2, 3, and 5) | As shown in Ta- ble A1 radius of size 2 has the best overall performance, achieving the highest results in 5 of the downstream datasets, proving that is the best radius size for CL-MFAP.
Table A1: ROC-AUC of CL-MFAP models with varying Morgan fingerprint radius sizes down- stream property prediction datasets. Models are named based on their respective fingerprint radius sizes in the format fingerrrii radius size.
Fingerprint
E. coli
H. influenzae
Model
BBBP
PAMPA
Bioavailability
BACE
Radius Size MIC
MIC
MR 0
0.827
0,846
0.905
0.747
0.535
0.886
MR 1
0.843
0.857
0.900
0.721
0.523
0.880
MR _
0.854
0.855
0.928
0.747
0.605
0.882
MR
0.849
0.853
0.913
0.738
0.546
0.880
MR
0.852
0.858
0.900
0.719
0.553
0.868
Ablation study on Data Modalities. We removed each of the three data modalities from CL-MFAP individually and assessed its performance on downstream property prediction tasks to determine their individual impact. As shown in Table A2 We observe that removing either the SMILES or the molecular fingerprints results in a certain degree of performance decline. This suggests that both data modalities contribute approximately equally to the overall model performance, with the impact of removing Fingerprints being slightly greater than removing SMILES. However, when remove the molecular graph modality, the model performance experiences a significant drop. This indicates model's performance is the molecular graph, processed through
that the primary contributor the GTE integrated with the BRA mechanism, which aligns well with our assumptions.
Table A2: ROC-AUC of CL-MFAP models with varying data modalities on downstream property prediction datasets. Models named based on their missing modalities in the format M.missing modality.
Missing Modality
E. coli MIC
H. influenzae MIC
Bioavai- lability
Model
BBBP
PAMPA
BACE
none
NA
0.875
0.855
0.941
0.784
0.559
0.891
M_noSMI
SMILES
0.834
0.741
0.920
0.720
0.568
0.877
M.noFP
Fingerprint
0.784
0.859
0.903
0.725
0.622
0.878
M.noGraph
Graphs
0.541
0.512
0.656
0.633
0.647
0.625
Ablation study on Window Size. We tested several different window sizes in our CL-MFAP to study their impact and determine the most optimal choice. Six distinct window sizes (2, 3. 7, 9, and 11) were evaluated for CL-MFAP, and their performance on downstream property prediction tasks was assessed. As shown in Table A3 a window size of 7, representing a moderate configu- ration, achieved the best performance in 4 out of 5 tasks. In contrast, performance tends to decline when the window size either too large or too small. This was predicted as when the window size too small, the BRA mechanism is confined to focusing highly local regions, overly emphasizing
Due to time constraints, all of these fine-tuning evaluations were performed for epochs, compared to 20 epochs used for our final CL-MPAP model.
14

---

Published as a conference paper at ICLR 2025
fine- fine-grained details, and, some extent, losing the ability to capture long-range dependencies. On the other hand, when the window size is too large, the sparsity of the BRA mechanism becomes excessive, leading to the dilution of some critical local information, which partially undermines the effectiveness of routing and aggregation. As a result, we set the default window size to 7.
Table A3: ROC-AUC of CL-MFAP models with varying window sizes on downstream prop- erty prediction datasets. Models are named based on their respective window sizes the format MW WWW_indow size.
E. coli MIC
H. influenzae MIC
Bioavai- lability
Model
Window Size
BBBP
PAMPA
BACE
MW -22
0.847
0.840
0.913
0.715
0.557
0.856
MW S3
3
0.844
0.849
0.909
0.717
0,564
0.851
MW
0.831
0.841
0.902
0.754
0.507
0.890
MW_ST
7
0.875
0.855
0.941
0.784
0.559
0.891
MW SS9
9
0.830
0.848
0.914
0.731
0,632
0.872
MW-SII
11
0.837
0.845
0.928
0.715
0.526
0.887
Ablation study on Pretraining CL-MFAP. We performed an ablation study to investigate whether pretraining on the larger ChEMBL dataset improves model performance. CL-MFAP with and with- out ChEMBL pretraining was trained/finetuned on all downstream property prediction datasets. As shown in Table A4 in 5 of tasks, dropping the pretraining slightly weakens model performance, although not significantly. This indicates that while pretraining enhances model performance and represents the ideal scenario, our algorithm and novel methodology still able achieve excel- lent results even without pretraining. In scenarios where cost-effectiveneee is prioritized in training resource consumption, the model can handle the intended use cases similar extent without pre- training.
Table A4: ROC-AUC of CL-MFAP with ChEMBL dataset pretraining pretraining down- stream property prediction datasets
CL-MFAP with ChEMBL pretraining
CL-MEAP without ChEMBL prefraining
Dataset
E. coli MIC H. influenzae MIC
0.854
0.824
0.874
0.850
BBBP PAMPA
0.933
0.900
0.759
0.728
Bioavailability BACE
0.599
0.549
0.881
0.882
A.2 DOWNSTREAM PROPERTY PREDICTION DATASETS
The choice of the downstream property prediction datasets was based on the availability of good quality data and biological relevance to antibiotic properties. The most relevant antibiotic property is antibacterial activity, and thus the E. coli and influenzae MIC datasets were curated from COADD (Desselle et al. 2017) ChEMBL (Gaulton et al. 2011). respectively. to analyze CL-MFAP's ability to predict antibacterial activity. The other datasets were obtained from trusted databases (Moleeu- leNet (Wu et al.) 2018) and Therapeutics Data Commons (Huang al. (021)) and are commonly used ML models to benchmark model performance in drug discovery.
E. coli MIC Dataset. This dataset describes compound ability to inhibit Escherichia coli (E. coli). Obtained from COADD (Desselle et al., 2017), each compound has an associated Minimum In- hibitory Concentration (MIC) value, which represents the antibacterial activity against E. coli. The compounds were binarized as active (1) if MIC ug/mL and inactive (0) if MIC ug/mL. Size: ~ 100,000 compounds.
15

---

Published as a conference paper at ICLR 2025
H. inftuenzae MIC Dataset. This dataset describes the ability of compounds to inhibit Haemophilus infiuenzae (H. influenzae). Obtained from ChEMBL (Gaulton et al. 2011), each compound has an associated MIC value, which represents the antibacterial activity against H. influenzae. The compounds were binarized as active (1) if MIC o 4 ug/ml. and inactive (0) if MIC > ug/ml.. Size: 3,341 compounds.
BACE Dataset. This dataset from MoleculeNet (Wu et al. 2018) assesses compounds' binding ability for set of inhibitors for 8-secretase The compound is labeled active (1) if it is potential inhibitor of B-secretase 1, otherwise. Size: 1,512 compounds.
Blood-Brain Barrier Penetration (BBBP) Dataset. This MoleculeNet (Wu et al. 2018) dataset assesses compounds' capacity to traverse the fhood-brain barrier. The compound is labeled "p" if it can penetrate the barrier and "np" if cannot. Size: 2,038 compounds.
Parallel Artificial Membrane Permeability Assay (PAMPA) Dataset. This dataset evaluates compounds' permeability across the cell membrane based on the PAMPA assay. The compound is labeled if it has high permeability, and O if it has low permeability. Size: NCATS set 2,035
compounds; Approved drugs set 142 drugs (Siramshetty et al., 2021).
Bioavailability. This dataset contains the oral bioavailability of different drugs, defined "the rate and extent which the active ingredient or active moiety absorbed from drug product and becomes available at the site of action" (Chen et al., 2001) Size: 640 compounds (Ma et al. 2008
A.3 EVALUATION OF CL-BASED MODELS ON DOWNSTREAM PROPERTY PREDICTION DATASETS
In Table A5 below are the ROC-AUC values for all pre-trained CL models on the downstream property prediction datasets.
Table A5: ROC-AUC of all pre-trained CL models on downstream property prediction datasets
E. coli MIC
H. influenzae MIC
Bioavai- lability
Model
BBBP
Pampa
BACE
CL-MFAP
0.85
0.87+0.02
0.93+0.01
0.76
0.60+0..3
0.88+10.01
CL-BLI
0.80+0001
0.86110002
0.93+00.01
0.75
0.86-10.02 0.6910.05
CL-BL2
0.80+0.01
0.87=0.02
0.92+00.00
0.75+0.02
0.72+0.05
0.86+0.01
CL-BL3
0.82+0.04
0.87+0.02
0.93+0.011
0.72+0.03
0.6510.05
0,87+0.01
CL-BL4
0.78
0.86+10.22
0.92+00.11
0.7310.04
0.67#0.06
0.85
A.4 REPRA RVALUATION OF PRE-TRAINED MODELS
We primarily applied the Representation- Proporry Relationship Analysis (RePRA) method to evalu- CL-MFAP against its model variations and all baselines (Mol.oormer MolBERT, ChemBERTa- 2, MoICLR, and FP-GNN). RePRA, a novel method introduced by Zhang et al. in 2023, draws inspiration from the concepts of Activity Cliffs (ACs) and Scaffold Hopping (SH) (Zhang et al.) 2024). It assesses the quality of molecular representations extracted by pre-trained models and visu- alizes the relationship between these representations and molecular properties. RePRA generalizes context, defining an
ACs and SH from the structure-activity context to the representation- ideal relationship between molecular representations and their properties boundary condition. This condition drives the ACs and SH regions to borderline state without observed data points, allowing for the calculation of ACs and SH thresholds based on these constraints. By using the de- tected ACs and SH, RePRA generates map showing the distances between pairs of representations and molecular properties, thereby evaluating the quality of the representations.
RePRA Map. The RePRA map serves as visualization tool for assessing the quality of molecu- lar representations produced by a pre-trained model. Its x-axis denotes the similarity between the representations of pair of target molecules, while the y-axis indicates the difference between the properties this pair molecules. Typically, a RePRA map is partitioned into four main regions, with shadowed ACS and SH zones that should ideally be avoided by the data points on the map.
16

---

Published as a conference paper at ICLR 2025
Activity Cliffs. This region is delineated by scenarios in which pair of molecules showcases markedly different properties beyond the y-axis threshold of ACs, while their representations ex- hibit noticeable similarity surpassing the x-axis threshold of ACs. A predominance of data points clustered in this area indicates that the model's representations are too similar to adequately capture the diverse range of molecular properties, thus indicating limited ability of the pre-trained model to differentiate between molecular properties.
Scaffold Hopping. This region is characterized by instances where pair of molecules exhibit fairly similar properties beyond the y-axis threshold of SH, yet their representations demonstrate a significant disparity surpassing the x-axis threshold of SH. A prevalence of data points clustered in this zone suggests that the model tends to generate highly various representations that correspond to a narrow range of similar molecular properties, indicative of subpar representation quality from the pre-trained model.
Evaluation Scores. Two evaluation scores, average deviation (SAD) and improvement rate (SIR), are derived from the RePRA Map to assess the performance of the models. SAD quantifies the average deviation by considering the ratio of data points situated in ACS and SH, adjusting for noise points in the remaining ideal regions; a lower SAD value indicates better performance. On the other hand, SIR is computed by comparing the numbers of data points in ACs and SH between a standard baseline (ECFP) and the pre-trained model under evaluation. Again, a lower SIR value signifies superior performance.
Visualization of Cosine Similarities. In addition to the RePRA map, visualization of cosine similarities is also presented to analyze the distribution of similarities using CosineSim as a metric between pairs of molecules. This visualization aids in identifying if there are common substructures shared among most molecular pairs.
Datasets. For the RePRA measurement, employed the Estimated SOLubility (ESOL) dataset, which includes the measured log solubility (mol/L) for 902 compounds (Niwa et al. 2009). The "measured log solubility in mols per liter" data from the ESOL dataset was utilized labels for molecular properties. Initially, the distance between each pair of labels was computed, followed by calculating the distance between each pair of logits. These labels and logits were then collectively inputted into the RePRA algorithm to generate the map.
Results. All models were evaluated using the RePRA test, with the scores presented in Table A6 For the SAD parameter, it can be observed that the CL-MFAP model has the lowest result, indicating fewer noise data points with detected ACs and SH, which suggests better representation praesentation-property relationship. For the SIR parameter, the CL-MFAP model also has the lowest score, demonstrating an improvement representation quality compared to the traditional ECFP method and indicating that CL-MFAP generates better representations compared to the other models. Since lower SAD and SIn scores jointly indicate superior molecular embedding and representation quality, it unsurpris- ing that the CL-MFAP model, enhanced by the BRA, excels in this test. Notably, all CL models utilizing GTE outperformed the baseline models, highlighting the inherent advantage of contrastive learning frameworks trained multimodal data in effectively learning molecular representations. The results of the RePRA map are shown in Figure Al
Table A6: RePRA scores of all trained CL models and bascline models Model
SAD
SIR
CL-MFAP
0.008
1.317
CL-BLI
0.013
1.501
CL-BL2
0.011
1.431
CL-BL3
0.010
1.395
CL-BL4
0.019
1.753
MoLFormer
0.017
1.607
MolBERT
0.016
1.758
ChemBERTa-2
0.020
1.904
MolCLR
0.016
1.267
FP-GNN
0.007
1.434
17

---

Published as a conference paper at ICLR 2025
CL-MFAP
CL-BL1
CL-BL2
CL-BL3
CL-8L4
Sroppi Cistence
Prupery
Propery
Propping
Pepretencation Similantty
Representation Simiarity
Repretentation Siillarty
Representation Simlarity
Representation Simliantty
Propory Disurnc
Property Dutancc
Propury DiuuuryDeeuncce Dierunce
Prosurry Darunco
Pyoperty Drstnne
0 R 0.
1830
809
"oo
Representation Smilarity
Representaiion Simllritty
Rapresentation Similarity
Reprisentation Simiarity
Similnstty
MolFormer
MoIBERT
ChemBERTo-2
MoiCLR
FP-GNN
Pooponn Distanc
Proparty Distaroze
propory Ossaacce
Propert
Sivilarity
of 10
Similarity
Representation Similantty
62
24
Representation Simllanty
14
Ga Roprescmtation Similaitty
Xx
2014
Proseetty Diounce
Propary Dirance
Procerty Disturcs
1900
1338
Proparty
1905
1050
to Representation Simllrtty
Poop up
No C3
C2 80
oscntaation Simflarity
Representaiion Siitantty
Reprvoentation Simiarity
Representation Similantty
Figure Al: RePRA measurement of all pre-trained CL models and baseline models. The shaded areas in the top right and bottom left represent the ACs region and the SH region, respectively.
A.5 Escherichia coli CASE STUDY
Escherichia coli (E. coli) is a gram-negative bacterium commonly found in the gut microbiome of humans that usually harmless. However, can become pathogenic under certain conditions or pathogenic E. coli can be ingested and cause variety of issues in humans. The issues can range from traveler's diarrhea and pneumonia (Mueller & Tainter 2024) to playing part Inflamma- tory Bowel Disease many strains develop antibiotic resistance, thus showcasing the need for new antibiotic compounds effective against E. coli.
Pertinez-Medina & Garcia-Gil|[2114 Although antibiotics exist for E. coli,
In this case study, we employ CL-MFAP to identify novel antibiotic compounds that are highly likely be effective against E. coli.
Model Training. CL-MFAP was finetuned on Minimum Inhibitory Concentration (MIC) data against E.coli (Anti-E. coli Activity) described in Appendix A.2. Obtained from the COADD database, each compound has its associated MIC value, which represents the antibacterial activ- ity, against coli. The compounds were binarized as active (1) MIC 8 ug/mL and inactive (0) if MIC > 8 ug/mL.
Virtual Screening. Based the finetuned CL-MFAP model, virtual screening was performed using the ZINC database. ZINC is free database containing over 230 million commercially available compounds ready-to-dock, 3D formats (Irwin et al. 2020). Due to its massive size, we used the ZINCK250k dataset (Basu 2021), subset of 250,000 compounds from ZINC. From this, 9389 compounds were identified with predicted activity (predicted to be effective inhibiting E. coli) with 100% probability and were chosen for further property testing.
Pharmacokinetic and ADMET Property Predictions. For the 9389 compounds identified via virtual screening, their pharmacokinetic and ADMET (Absorption, Distribution, Metabolism, Ex- cretion, and Toxicity) properties were predicted using ADMET-SAR (Yang et al., 2018). These properties allow us to identify compounds that have necessary molecular properties and are most likely to perform well as antibiotics. From this, we filtered to only include compounds that follow the Lipinski Rule of 5 (molecular weight < 500 Da, logP 5, number of hydrogen bond acceptors
18

---

Published as a conference paper at ICLR 2025
I 10, and number of hydrogen bond donors & 5) with maximum of 1 violation. In addition, their topological surface area had to be between 20-130 A and their aqueous solubility range had to be between -1 and -5, As result, 7358 compounds remained. Then, an ADMET score generated for each remaining compound based on 18 properties related to absorption, toxicity, and metabolism. We followed the ADMET-score method proposed by /uuan et al. (2018).
Similarity to existing E. coli antibiotic compounds. To validate the compounds with predicted anti-E. coli activity and ideal pharmacokinetic and ADMET properties, we compared their simi- larity to existing FDA-approved E. coli antibiotic compounds include Levofloxacin (Drago et al.) 2001]. and Ciprofloxacin (Jakobsen et al. 2020). We first selected the top 1000 compounds with the highest predicted probabilities and ADMET scores and they were split into 4 groups: level 1 (top 1- 250 compounds), level 2 (top 251-500 compounds), level (top 501-750 compounds and level 4 (top 751-1000 compounds). For each group, the number of
scaffolds and the number of Bernis-Murcko scaffolds per compound were evaluated. As shown Table A7 the identified com- pounds show structural diversity, an essential feature in drug discovery to ensure coverage of broad chemical space. Results also show that molecules ranked higher (those with more favorable ADMET properties) have larger diversity than the molecules ranked lower. We also calculated the Tanimoto similarity (also known as Jaccard Index) based on the MACCs and MAP4C fingerprints between the top 1000 selected compounds and known antibiotics, Levofloxacin and Ciprofloxacin. Among these, two candidates were identified to have high MACCs and low MAP4C similarity with exist- ing coli antibiotic compounds: C22H22CINO4 (ZINC ID: ZINC2059 1249) and C25H25CIN402 (ZINCID: ZINC8758881). As shown in Table A8, they have high MACCs similarity scores and low MAP4C similarity scores the existing www.biotcs.MAACO.. keys are well-suited for functional group-based based similarity searching, allowing to identify compounds that share key pharmacophoric features and common medicinal chemistry substructures. MAP4C captures more detailed struc- tural information, such atom types and bonding patterns, which is more relevant for identifying structural similarities between compounds. The high MACCs similarity scores with low MAP4C similarity scores confirm that our identified compounds possess functional similarity to the existing antibiotics while maintaining structural novelty. This outcome not only validates our approach but also suggests potential candidates for further investigation in antibiotic development.
Table A7: Bemis-Murcks Scaffolds results of top 1000 compounds predicted to be active against Escherichia coli using CL-MAP
Compounds Included (By Ranking)
Number of Bemis-Mureko Scaffolds
Number of Bemis-Murcko Scaffolds per Compound
Level
Level 1
1-250
245
0.980
Level 2
251-500
241
0.964
Level 3
501-750
236
0.944
Level 4
751-1000
236
0.944
Table A8: Fingerprint similarity scores of potential Escherichia coli antibiotic compounds with existing Escherichia coli antibiotics Compound
MACCs
MAP4C
Levofloxacin
Ciprofloxacin
Levoffoxacin
Ciproftoxacin
C22H22CINO4 C25H25CIN402
0.739
0.696
0.030
0.032
0.716
0.623
0.023
0.018
19

---

Published as a conference paper at ICLR 2025
REFERENCES
Victor
Basu.
Zinc250k dataset, 2021.
URI.
basu36 9v ctor /it.rr/iiic22.
Mei-Ling Chen, Vinod Shah, Ravi Patnaik, William Adams, Ajaz Hussain, David Conner, Mehul Mehta, Henry Malinowski, James Lazor, and Shiew-Mei Huang. Bioavailability and bioequiv- alence: fda regulatory overview. Pharmaceatical Research, 18(12)1645-1650, 2001. doi: 10.1023/a: 101331 13949988.
Mathilde R. Desselle, Ruth Neale, Karl A. Hansford, Johannes Zuegg. Alysha G. Elliott, Matthew A. Cooper, and Mark A. Blaskovich. Institutional profile: Community for open antimicrobial drug discovery crowdsourcing new antibiotics and antifungals. Fulure Science OA, 3(2):F80171, 2017. doi: 10.4155/fsa-2206--009.
L. Drago, E. De Vecchi, B. Mombelli, L. Nicola, M. Valli, and M. R. Gismondo. Activity of lev- ofloxacin and ciprofloxacin against urinary pathogens. Journal of Antimicrobial Chemotherapy, 48(1):37-45, 2001.
doi: 10.1093/jac/48.1333
Anna Gaulton, Louisa J. Bellis, A. Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGllichhey, David Michalovich, Bissan Al-Lazikani, and John P. Over- ington. Chembl: large-scale bioactivity database for drug discovery. Nucleic Acids Research, 10:D1100-D1107, 2011. doi: 10. 10.1093/nar/gkr777.
Longfei Guan, Hongbin Yang, Yingchun Cai, Lixia Sun, Peiwen Di, Weihua Li, Guixia Liu, and Yun Tang. ADMET-score comprehensive scoring function for evaluation of chemical drug- likeness. MedChemConm, 10(1):148-155) 157, 2018. doi: 10. 10.109/c3md00472bb.
Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovee, Connor W. Co- ley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeuties data commons: Machine learning datasets and tasks for drug discovery and development. Proceedings of Neural Information Pro- cessing Systems, NeurlPS Datasets and Benchmarks, 2021.
John J. Irwin, Ling Tang, Jared Young, Chinzorig Dandarchuluun, Benjamin R. Wong, Munkhzul Khurelbaatar, Yurii S. Moroz, James Mayfield, and Roger A. Sayle. Zinc20 a free ultralarge- scale chemical database for ligand discovery. Journal of Chemical Information and Modeling, 2020. doi: 10. /ac.a.cccci..00067777
Lotte Jakobsen, Carina Vingsbro Lundberg. and Niels Frimodt-Moller. Ciproftoxacin pharma- ookinetis/phammacdyaaaiii against susceptible and low-level resistant escherichia coli isolates in an experimental ascending urinary tract infection model in mice. Antimicrobial Agents and Chemotherapy, 65(1), 2020. doi: 10. 2a1128/acc.0188 0184-20..
Chang- Ying Ma, Sheng- Yong Yang. Hui Zhang, Ming-Li Xiang, Qi Huang, and Yu-Quan Wei. Prediction models of human plasma protein binding rate and oral bioavailability derived by using ga-cg-svm method. Journal of Pharmaceutical and Biomedical Analysis, 47(4):677-682, 2008. doi: 10.1016/j h/tp://ppba..000 03.0023........
Margarita Martinez-Medinn and Librado Jesus Garcia-Gil. Escherichia coli in chronic inflammatory bowel diseases: An update adherent invasive escherichia coli pathogenicity. World Journal of Gastrointestinal Pathophysiology. 5(3):213-227, 2014, doi: 10.429 1...299/wiigp.5533.22
Matthew Mueller and Christopher R. Tainter. Escherichia coli Infection.
StatPearls Publishing,
Treasure Island (FL), 2024.
Tatsuya Niwa, Wen Ying, Katsuyo Saito, WenZhen Jin, Shoji Takada, Takuya Ueda, and Hideki Taguchi. Bimodal protein solubility distribution revealed by an aggregation analysis of the entire ensemble of escherichia coli proteins. Proceedings of the National Academy of Sciences, 106(1 I l): 4201-4206, 2009. doi: 10. 10.1073/pnas.08 1922106.
Vishal Siramshetty, Jordan Williams, c-Trung Nguyn, Jorge Neyra, Noel Southall, Ewy Mathé, Xin Xu. and Pranav Shah. Validating adme qsar models using marketed drugs. SLAS DIS- COVERY: Advancing the Science of Drug Discovery, 26(10):1326133 1336. 2021. doi: 10.1177/ 247255522110177
20

---

Published as a conference paper at ICLR 2025
Zhengin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learn- ing. Chem. Sci., 93):513-5330, 2018. doi: 10. 10.1039/C7SC0264A.
Hongbin Yang, Chaofeng Lou, Lixia Sun, Jie Li, Yingchun Cai, Zhuang Wang. Weihua Li, Guixia Liu, and Yun Tang. admetSAR 2.0: web-service for prediction and optimization of chemical AD- MET properties. Bioinformatics. 35(6):1067-106,, 2018. doi: 10. 00.100//bonnfommaicc/yyy0.
Ziqiao Zhang, Yatao Bian, Ailin Xie, Pengju Han, and Shuigeng Zhou. Can pretrained models really learn better molecular representations for Al-aided drug discovery? Journal of Chemical Information and Modeling, 64(7):2921-2933)) 2024, doi: 10. 10.1021/aa Jcm..3c..
21


Reviews:
[{"review_id": "vl8U2HOAnH", "reviewer": "Reviewer_PXXZ", "summary": "The work introduces CL-MFAP, a model designed to predict the antibiotic properties of small molecules based on multimodal data and contrastive learning.  The model is trained on 1.6 million bioactive molecules from the ChEMBL database and takes three types of molecular data: SMILE strings, binary fingerprints, and graphs. To handle these three different modalities, the model uses a transformer with rotisserie positional embedding, MLP, and graph-based transformer accordingly. The optimization of the model is carried out with contrastive learning loss between the three modalities’ outputs. The models are then compared with several baselines on five downstream datasets and showed good performance on three of them.", "questions": "1. Is it meaningful to perform more ablations to the Window-to-Window Level Routing such as adjusting the window size?\n2. Will you consider adding std to results based on several independently running?\n3. Will you consider adding more published baselines?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The integration of RoPE and BRA enhances the model's ability to handle long-range dependencies and molecular structure in a multimodal contrastive learning setting. Furthermore, the BRA is used jointly alongside MPNN and graph transformer encoder. Ablation studies of the combinations of these sub-models are performed and these combinations are included as baselines.", "weaknesses": "1. The number of baselines is quite limited, out of 7 baselines, 4 of them are variant models from the proposed model. Only three of the baselines are models by other authors. \n2. There’s one mistake in algorithm 2, line 4, and line 6, where the two if conditions are the same. Maybe this is a typo. \n3. Some of the parameter values are not explained such as the value of window size S. The functions in algorithm 2 are not explained, I think one sentence of description can make the algorithm easier to understand.\n4. A graph of the Window-to-Window Level Routing and Pixel-to-Pixel Level Attention could be helpful for this is a major novelty of the paper. A graph for layer structures which include the substructures would be helpful and would make the combination of BRA, GTE, and MPNN easier to understand.\n5. The template is not the latest ICLR 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ynPoadhzIO", "reviewer": "Reviewer_o7pA", "summary": "In this paper, a new method for molecular property prediction is proposed. The model is termed CL-MFAP, and it uses a multi-model approach and contrastive learning to train a molecular property predictor specialized in antibiotic properties. As an input, the model takes three molecular modalities: SMILES strings, fingerprints, and molecular graphs. Each modality is transformed using a different neural network, and the resulting representation vectors are contrasted to create one representation that is useful for predicting various properties, including bioavailability, blood-brain barrier penetration, and cell membrane permeability. The graph neural network that processes molecular graphs is redesigned to use new bi-level routing attention, designed initially by Zhu et al. (2023). The experiments demonstrate the performance of CL-MFAP across five datasets as well as compare different versions of the model in the ablation study.", "questions": "1. What is special about this approach that makes this model specifically designed for antibiotic property prediction? The architecture seems to be rather general, and most of the datasets (besides MIC) used in this study are not specific to antibiotics. \n2. What does it mean that you \"collected bacterial-related compounds targeting the specific domain of antibiotics from the ChEMBL database\"?\n3. Why should the bi-level routing attention mechanism perform well in the context of molecular graphs? If I understand correctly, this layer was designed to help the model focus on the relevant parts of images. Molecular graphs of typical antibiotics (or generally therapeutic small molecules) are rather small and do not share the problems that were identified in the case of large images. Why should this approach work better for capturing long-range dependencies than graph transformers?\n4. What is $l$ in Equation 9? How is $l_i$ from Equation 8 used?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "Originality:\n- There are two main novel contributions in this paper. The first one is the adaptation of bi-level routing attention to molecular graphs. The second one is the exploration of the architecture space within the contrastive learning framework for molecular property prediction.\n\nQuality:\n- Different architectures were tested to select the best model architecture in this study.\n\nClarity:\n- The introduction and related work sections are written very clearly and with clear motivation.\n- Figure 1 effectively explains the model proposed in this study.\n\nSignificance:\n- The problem that the Authors try to solve in this study is very important. As explained in the introduction, we need more models that will help accelerate the discovery of new antibiotics. The proposed model is a step in that direction.\n- The code of the model is shared publicly.", "weaknesses": "Originality:\n- The model described in the paper is a combination of known techniques, including multi-modal molecular representations and contrastive learning.\n- The focus on antibiotics is an interesting direction that is currently underexplored. However, the paper does not emphasize any novel contributions that are specific to this field.\n\nQuality:\n- The experimental results do not include standard deviations or confidence intervals. The proposed method is compared only against three models for molecular property prediction, none of which are multimodal.\n- The data splitting method is not described in the paper.\n- The data collection method is not explained.\n\nClarity:\n- The \"Proposed Approach\" section is difficult to follow. It is impossible to understand the model without reading the original cited works that introduce the methods used in this study. For example, the notation used in any of the equations is not explained, like $m$ and $n$ in Equation 1, $k_n$ in Equation 2, $R_{\\Theta,i}^d$ in Equation 3. Some symbols are introduced but not used, like $\\varphi$ before Equation 2. If these symbols and formulas are not explained in the text, they should be removed, and there should be references to the original works instead.\n- For the BRA mechanism, the equations were also copied from the work of Dong et al. (2023) (or Zhu et al. (2023)) and not properly explained or defined. There are a few typos that make the text difficult to read, e.g. \"feature map $x$, $X\\in\\mathbb{R}^{H\\times W\\times C}$\" (is it $x$ or $X$?). \n- The algorithms included in the text use functions that were not defined. At least parts of these algorithms could be explained more clearly in the text using mathematical formulas. For example, the computeLoss function is unnecessary as it can be explained by the equations in the text. Also, it seems that the definition in Algorithm 2 is different from what Equation 9 represents.\n- The font size in Figures 2 and 3 is too small. Figure 2 would be more readable if it was a 2D barplot.\n\nSignificance:\n- The code repository does not contain any instructions on how to run the code or how to install the code dependencies. The repository should also include a license file that describes the allowed use of the code.\n- The experiments are insufficient to support the claim that CL-MFAP \"combines and compares molecular information [...] to efficiently learn representations of drug molecules **with potential antibiotic properties**.\"\n\nMinor comments:\n- The positive pairs on page 6 should probably include [x1, y1] instead of [x1, y2].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MIiTIjj8dy", "reviewer": "Reviewer_jJeK", "summary": "The paper introduces CL-MFAP, an unsupervised contrastive learning-based multimodal foundation model for predicting antibiotic properties of small molecules, addressing the urgent need for novel antibiotics due to rising antimicrobial resistance. This model employs 1.6 million bioactive molecules with drug-like properties from the ChEMBL dataset to jointly pretrain three encoders:SMILES, graph and fingerprint encoder. The experimental results seem encouraging.", "questions": "1. Could you clarify the interpretation of the RePRA scores presented in the results?\n2. Are you considering including specific case studies where the model performed well or poorly to better illustrate these points?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors conducted multiple down-stream tasks to verify the efficacy of their method.\n\nOverall, the paper is clearly represented.", "weaknesses": "1. The technical novelty is limited.\n1.1.  While the paper mentions contrastive learning as a foundational concept, it lacks a clear explanation of how this framework uniquely applies to antibiotic property prediction compared to existing methodologies. A more detailed discussion would help contextualize its application. The contrastive loss is the vanilla form of classical contrastive learning?\n1.2.  Although the bi-level routing attention mechanism is a critical innovation in the model, the explanation provided is somewhat vague.\n\n2. There are several statements that lack specificity and can be interpreted in multiple ways. For instance, the authors make sweeping claims about model performance without providing adequate context or evidence, which decreases the credibility of the discussion.\n\n3. The use of jargon without sufficient definitions makes it difficult for readers who may not be well-versed in the specifics of deep learning or molecular biology to fully grasp the content. A glossary of terms or more contextual explanations could remedy this.\n\n4. Experiments: \n4.1 It would be better to add some biological justification or visualization of the results.\n4.2 Providing specific case studies or examples would not only enrich the findings but also guide future improvements in the model.\n4.3 Some results lack proper contextualization, particularly in the representation of RePRA scores, which could lead to misunderstandings.\n4.4 Another concern is about label leakage. Is there any overlap between the datasets?\n\n5. While the paper includes anonymous code available on GitHub, it may not be very intuitive for users. ​It is advisable to include straightforward and easy-to-test data alongside the code, or to provide a Jupyter Notebook.​ This would enable readers to quickly and clearly grasp the operational process of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UE9I2J1EB7", "reviewer": "Reviewer_oysA", "summary": "This study presents a novel pre-training strategy for a multimodal representation model for the aiding in the discovery/design of antibiotic drugs. This method learns to map three different modalities describing molecular systems (1. SMILES string, 2. Molecular graph, 3. Molecular hash-based circular fingerprints) into a shared embedding space through contrastive learning.\n\nEach modality has an associated architecture to map the raw data into the shared embedding space, it is divided as follows:\n\n1. SMILES strings are modelled with a transformer architecture with rotatory positional embeddings.\n2. Molecular graphs are modelled with a graph transformer encoder (GTE). The study introduces the bi-level routing attention (BRA) from computer vision models to molecular graphs as an additional component to the graph transformer encoder.\n3. Molecular fingerprints are modelled with a multi-layer perceptron.\n\nThe model is pre-trained in the subset of ChEMBL molecules with known antibiotic activity with over 1.5 M data points.\n\nThe approach is evaluated against 5 datasets relevant for antibiotic discovery and it outperforms the baselines considered (3 molecular language models: MolFormer, MolBERT, and ChemBERTa-2) in 3 of them.", "questions": "1. What is the reason why you think this particular approach is specifically well-suited for representing antibiotic drugs? Have you run a comparison of your method against other benchmarks for a wider class of small molecules (e.g., MoleculeNet [1] or Therapeutics Data Commons [2])\n2. What criteria was used to decide on the downstream datasets? What other datasets were considered and why were they excluded?\n3. Were other fingerprints types or configurations considered for the fingerprint modality? If so, which and what criteria was used to reach the final decision?\n\n\nReferences\n----\n1. Wu Z, Ramsundar B, Feinberg EN, Gomes J, Geniesse C, Pappu AS, Leswing K, Pande V. MoleculeNet: a benchmark for molecular machine learning. Chemical science. 2018;9(2):513-30.\n2. Huang K, Fu T, Gao W, Zhao Y, Roohani Y, Leskovec J, Coley CW, Xiao C, Sun J, Zitnik M. Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development. arXiv preprint arXiv:2102.09548. 2021 Feb 18.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Evaluation.** The experimental results of the model demonstrate a notable improvement with regards to the baselines chosen, the datasets chosen are well studied benchmarks that could be relevant for the purposes of antibiotic discovery/design.\n2. **Molecular graph encoder.** The idea of adapting the bi-level routing attention to graphs is interesting and could be beneficial for the broader domain.", "weaknesses": "Main weaknesses\n----\n\n1. **Relevance.** Antibiotic discovery/design is a societally important problem and new approaches for accelerating or improving pharmaceutical pipelines are sorely needed. From a technical standpoint, however, antibiotic small molecules are not inherently different from any other small molecule drugs. I think that the authors should include justification regarding why this method is specifically well-suited for antibiotics and not for any other class of small molecule (e.g., is there any antibiotic-specific inductive bias embedded in the method?) Conversely, if there is no technical reason why the method could not be applied to other small molecule drugs, then I think that its evaluation should concern a more diverse evaluation including general benchmarks like MoleculeNet [1] or Therapeutic Data Commons [2].\n2. **Baselines.** The baselines considered in Table 2 and Figure 2 correspond to Molecular Language Models. The method herein proposed combines three different modalities (or molecular views), i.e., fingerprints, molecular graph, and SMILES (or molecular language). Therefore, I think that additional baselines  should be included to, at least, consider:\n   - i) A model with fingerprints as input that can be an MLP or any other type of downstream model.\n   - ii) A GNN for the molecular graphs.\n   - iii) A contrastive learning pre-trained model like MolCLR [3].\n3. **Ablation study.** The ablation study conducted with models CL-BL1-4 only concerns the molecular graph and the different components that integrate it. There are missing elements in this ablation that should be considered:\n   - i) the contribution of each modality (does the performance change when the SMILES, the graphs, or the fingerprints are not used?)\n   - ii) 2. is the pre-training helpful or could the same performance be achieved with a model trained from scratch for each dataset?\n   - iii) what would happen if the more general models like MolFormer or ChemBERTa-2 were finetuned to the same pre-training data as the model?\n4. **Significance of results**. The results in Table 2 and Figure 2 do not have an error associated which makes it difficult to compare among approaches as there is no sense on the variability between independent runs. The introduction of error estimations for different runs (3 or 5 would be sufficient) would provide a proxy value that would give an intuitive idea of the statistical significance of the differences between model performances.\n\nMinor weaknesses\n----\n_(These weaknesses do not influence my decision-making, but I think could serve to improve the paper)_\n\n1. There are minor typos throughout the paper.\n2. Please note that the PDF corresponds to the ICLR template for 2024 and not 2025, be careful when submitting the final version.\n3. The description of bi-level routing attention focuses primarily in its original use in computer vision, where context windows are described in terms of a 2D feature map or pixel-to-pixel attention, I would find it helpful if the explanations concern the generalisation of the method to graphs and how the concept of 2D feature map or pixel would apply to graph topology.\n4. There is not a description of what fingerprint was used, even though, is one of the modalities used in the paper (In appendix A.2, ECFP is mentioned as a standard baseline, but no further details regarding the pre-trained model are included). Not only fingerprint type is missing, but also critical parameters as number of bits in the fingerprint (1,024; 2,048; ...) and radius (1, 2, ...). \n\nReferences\n----\n1. Wu Z, Ramsundar B, Feinberg EN, Gomes J, Geniesse C, Pappu AS, Leswing K, Pande V. MoleculeNet: a benchmark for molecular machine learning. Chemical science. 2018;9(2):513-30.\n2. Huang K, Fu T, Gao W, Zhao Y, Roohani Y, Leskovec J, Coley CW, Xiao C, Sun J, Zitnik M. Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development. arXiv preprint arXiv:2102.09548. 2021 Feb 18.\n3. Wang Y, Wang J, Cao Z, Barati Farimani A. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence. 2022 Mar;4(3):279-87.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)