Curriculum-aware Training for Discriminating Molecular Property Prediction Models
Authors: Hansi Yang
Quanming Yao
James Kwok
Venue: ICLR 2025 Poster (2025)
Keywords: molecular property prediction
curriculum learning
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Despite their wide application across various fields, current molecular property prediction models struggle with the challenge of activity cliff, which refers to the situation where molecules with similar chemical structures display remarkable different properties. This phenomenon hinders existing models' ability to learn distinctive representations for molecules with similar chemical structures, and results in inaccurate predictions on molecules with activity cliff. To address this limitation, we first present empirical evidence demonstrating the ineffectiveness of standard training pipelines on molecules with activity cliff. We propose a novel approach that reformulates molecular property prediction as a node classification problem, introducing two innovative tasks at both the node and edge levels to improve learning outcomes for these challenging molecules with activity cliff. Our method is versatile, allowing seamless integration with a variety of base models, whether pre-trained or randomly initialized. Extensive evaluation across different molecular property prediction datasets validate the effectiveness of our approach.

Published as a conference paper at ICLR 2025
CURRICULUM-AWARE TRAINING FOR DISCRIMINATING MOLECULAR PROPERTY PREDICTION MODELS
Hansi Yang
Quamming Yao
Department of Computer Science and Engineering Hong Kong University of Science and Technology Hong Kong, China
Department of Electronic Engineering, State Key Laboratory of Space Network and Communications, Tsinghua University Beijing, China qyaoaa@tsinghua edu. cn
hyangbwecse ust. hk
James Kwok
Department of Computer Science and Engineering Hong Kong University of Science and Technology Hong Kong, China
jame @ cse. ust.hk
ABSTRACT
Molecular property prediction plays a crucial role in various fields such as chemin- formatics and artificial intelligence. Despite its wide applicability. current models still struggle in the presence of activity cliff, in which molecules with similar chem- ical structures display remarkable different properties. This hinders the model's ability to learn distinctive representations for molecules with similar chemical structures, resulting in inaccurate predictions on molecules with activity cliff. In this paper, we first present empirical evidence demonstrating the ineffectiveness of standard training pipelines on these molecules. We then propose a novel approach that reformulates molecular property prediction as a node classification problem, and introduce both node-level and edge-level tasks to improve the learning for these challenging molecules. The proposed method is versatile, and can be seam- lessly integrated into a variety of pre-trained or randomly initialized base models. Extensive evaluation various molecular property prediction datasets validate the effectiveness of our approach.
INTRODUCTION
Molecular property prediction aims to determine the properties of molecules directly from their chemical structures. It plays
crucial role in various fields, including drug discovery (Stokes
CHa
et al.
2020), material science (Chanussot et al. 2021 Tran et al.
OH
2023
and bioinformatics (Narayanan et al.] 2002 Zhou al.
CHa
2023
Despite its broad applicability, recent studies (van Tilborg
CHS
CHg
CH2
al.
2022 Deng et al. 2023) show that current models often
(a) Retinol.
fail to generate sufficiently discriminative molecular representa- tions. sometimes. they can even perform worse than models using fixed representations (c.g., molecular fingerprints). This problem arises as existing machine learning models tend to produce sim- ilar representations for structurally similar molecules. When two such molecules exhibit different properties, accurately predicting their properties becomes challenging due to their indistinguish- able representations. This phenomenon, referred as activity cliff (AC) (Stumpfe et al.) 2019: Tamura al. [023 Dablander et al.)
CH2
CHs
CH2
(b) Retinal.
Figure Examples of two molecules with AC
2023) is prevalent across various molecular property datasets, Fig-
shows example from the Tox2/ / data set (Wu et al. 2018). Here, the molecules have only

---

Published as a conference paper at ICLR 2025
minor differences (the two yellow boxes), but their responses to the ER, ATAD5 and HSE receptors are all different.
While numerous studies (Maggiora 2006 van Tilborg et al. 2022 Graff al. 2023 Deng et al. 2023) have verified that AC causes difficulties for existing molecular property prediction models, their analysis focus only on the inference stage. It remains unclear why these models fail to learn discriminating molecular representation during training. Similar to inference, training a model to differentiate structurally similar molecules with distinct properties inherently presents challenges. Nevertheless, no existing work considers how to address this challenge. Standard training pipelines only lead models that are incapable of distinguishing molecules with AC.
Motivated by the shortcoming of existing training algorithms in obtaining discriminative molecular representations, in this paper, we propose a novel training algorithm enhance learning from molecules with AC. Through empirical analysis, we first demonstrate that standard training algorithms struggle accurately fit molecules with AC during training, and this challenge persists across different model backbones and pre-training tasks. To alleviate this problem, we propose new training algorithm that focuses on improving the model's discriminative power by effectively learning from molecules with AC. We first reformulate molecular property prediction problem on graphs, where each node represents a molecule, and edges are defined by similarities in their chemical structures. We then introduce two tasks at the node and edge levels respectively. For the node-level task, we employ curriculum learning that considers both the loss and AC information the selection of informative molecules for model training. For the edge-level task, introduce novel pairwise modeling task to align the model directly with AC on different molecular properties. The proposed method can be integrated with different base models, pre-trained. or randomly initialized. Empirical results on various molecular property prediction data sets demonstrate effectiveness of the proposed method.
node classification
Our contributions can be summarized as follows:
We are the first to investigate why existing molecular property prediction models fail to produce discriminative molecular representations. Using molecules with AC as representatives, we show that standard training pipelines struggle to accurately fit these molecules, a limitation observed in both randonly-initialized and pre-trained models.
We propose to reformulate molccular property prediction as a node classification problem. We then introduce two novel tasks the node and edge levels, as learn from molecules with AC more effectively and produce models with good discriminative ability.
Empirical results on various molecular property data sets demonstrate that the proposed method improves the performance of both random-inine initialized and pre-trained models.
2
RELATED WORKS
2.1 MOLECULAR PROPERTY PREDICTION WITH GRAPH NEURAL NETWORKS
Molecular property prediction predicts the molecular properties from molecular graph, in which each node is an atom and each edge is a chemical bond between atoms. Naturally, various graph learning architectures can be applied. Pioneering works (Merkwirth Lengauer 2005 Gilmer et al.
2017) use the message-passing graph neural networks (GNN) (Veličković et al. 2018 Xuu et al. 2019).
However, the GNN may be able to capture long-range dependencies (Rampásek Instead, recently. transformer models (Vaswani et al. 2017 are used to model long-range interactions between nodes (Ying et al., |02T Rampášek et al., 2022). For example. EGT (Hussain et al. 2022)
al. 2022
uses global self-attention to update both the node and edge representations for quantum-chemical regression. This allows unconstrained dynamic long-range interactions between nodes, and results in better performance.
Besides using various deep learning architectures, another approach improves performance by using different graph pre-training tasks. Most of these works consider how to effectively use the geometric information contained in the 3D conformers of different molecules (Townshend et al.) 2019 Axelrod (2022). For example, Klicpera et al. (2020) uses the relative 3D information (such as bond length and bond angle) derived from the absolute Cartesian coordinates. GemNet (Gasteiger et al. 2021) further captures information from the dihedral angle to uniquely define all
&
2

---

Published as a conference paper at ICLR 2025
relative atom positions. SphereNet (Liu et al. 2021) network, and designs a spherical message passing mechanism. 3D Infomax (Stärk et al. 2022) proposes to maximize mutual information between the 3D structures and representations from the GNN, enabling the model to produce implicit 3D information that can be useful for the downstream tasks. 3D-PGT (Wang al., 2023b) proposes multi-task 3D pre-training framework that predicts bond length, bond angle and dihedral angle from molecular graphs. UniMol (Zhou et al., (202)) proposes jointly use the 3D position recovery task and masked atom prediction task for pre-training, and achieves state-of-the ahrt performance on various molecular property prediction benchmarks.
proposes a generic framework for the 3D graph
The negative impacts of AC to molecular property prediction have long been investigated (Maggiora 2006; van Tilborg et al.] 2022 Graff et al., 2023: Deng et al. (023). However, they focus on the negative impacts with a novel training algorithm.
inference stage, while we propose to confront
Some other works (Horvath et al. (2016) 66; Iqhal et al. [021] Park et al. [2022] Zhang et al., 2023] Wu
2024) predict whether a given pair of molecules have AC, which differs from our focus on solving
molecular property prediction.
2.2 CURRICULUM LEARNING
Curriculum learning (CL) (Wagg et al. 2022) first trains learning model with easier training samples so that the model can easily obtain a coarse decision boundary. The model is then refined by harder samples later in the training process. As easy-to-use plug-in, curriculum learning has shown improved generalization performance of various models wide range scenarios, including computer vision (Guo al. 2018), natural language processing (Platanios et al. 2019 Liu et al. 2020) and reinforcement learning Narvekar et al. 2017).
Curriculum learning has also been applied to graph learning (Wei et al. 2022 Wang al. 2023a). CLNode (Wei et al., (022) proposes to jointly consider the loss and node labels in the curriculum learning of node classification. MotifNet (Wang et al.! (202a)) uses curriculum learning for motif- based graph learning, and orders the various motifs based on their difficulty levels. CurrMG (Gu et al., 2022) further considers using curriculum learning in molecular property prediction. Nevertheless, their approach only yields limited improvements as they consider the prediction eтror and molecular structure for each molecule separately, while the proposed method considers the pairwise relationship between molecules.
CASE STUDIES ON MOLECULES WITH ACTIVITY CLIFF
To see how existing models suffer from limited abilities in distinguishing molecules with similar chemical structures, we take the set of molecules with AC as an example. Loosely speaking, AC refers to a pair of molecules with similar structures but distinct properties. Its precise definition depends on how we characterize structural similarity. In the following, we build upon the definition of matched molecule pairs (Dablander et al. 2023).
Definition 3.1 (Matched Molecule Pair: Dablander al. (0233) )
matched molecule pair is a pair of molecules that share common structural core (which contains at least twice as many heavy a as in the variable parts) but differ by small variable parts (which contains no more than 13 heavy atoms) from the chemical transformation of bond cutting exocyclic bonds.
The definition of AC can then be given as follows:
Definition 3.2 (Activity Cliff (AC)). Activity cliff refers to a matched molecule pair with different labels with respect given property.
Note that the definition of AC depends on the property being considered. For pair of molecules with similar chemical structures, it may exhibit activity cliff on one property but not another.
While many works have demonstrated the difficulty of making accurate predictions molecules with AC (Maggiora 2006] van Tilborg et al. 2022] Graff et al. 2023 Deng et al.) 2023). it remains unclear why such difficulty arises, and why existing models cannot produce discriminating representations on these molecules. To empirically investigate these issues, we consider four tasks from the Tox2/ data
Heavy atoms other than hydrogen.
3

---

Published as a conference paper at ICLR 2025
set (Wu al. 2018) which predict a molecule's response to different receptors ARR, NR-ER, SR-ARE and SR-MMP). We use two graph neural network models that have been commonly used for molecular property prediction: (i) GIN [Xu et al. 2019) as representative for message-passing neural networks, and (ii) GraphGPS (Rampášek et al. 2022) as a representative for attention- based graph learning models. Besides training the GIN GraphGPS models from scratch, also include two recent state-of-the-ar pre-trained models: 3D-PGT (Wang et al. 2023b) and Uni-Mol (Zhou)
al. 2023). both of which use attention-based graph learning models similar GraphGPS.
Figure shows the proportion of molecules with AC among molecules with the top-n% training loss values. While only about 40% of all samples in the Tox21 data set have activity cliff (Table in Appendix B). molecules with AC make up a significantly higher proportion of large-loss molecules (about 60% in samples with the top-10% loss). In other words, activity cliff is critical source for samples that are not accurately learnt. This also indicates the inability of current models in distinguishing structural similar molecules.
(a) GIN.
(b) GraphGPS.
(c) Pre-trained 3D-PGT. (d) Pre-trained Uni-Mol.
Figure 2: Proportion of molecules with AC among molecules with top-n loss values.
Figure 3 shows the training loss curves for the top 10%-loss molecules with and without AC. We can see that even for these "hard" molecules, molecules with AC have significantly larger training losses than those without AC. Moreover, even at the end of the training process, the average loss on molecules with AC is still much larger than zero, indicating that the four models do not learn these molecules well with standard training pipelines.
(a) GIN.
(b) GraphGPS
(c) 3D-PGT.
(d) Uni-Mol.
Figure 3: Training losses of large-loss molecules with and without activity cliffs in four model
training setups.
In order to accurately distinguish pair of molecules with activity cliff, the model's decision boundary is expected separate them even though they have very similar chemical structures. In this experiment, we learn two models, one uses only molecules with AC (denoted "AC") as training samples, while the other initialized GraphGPS model and pretrained 3D-PGT model are used as base models. Table [able[]showw shows the ROC-AUC values of the two models on various data sets. As expected, using only molecules without AC for training yields worse performance than training on all molecules. Using molecules with AC for training leads to some improvements on the Tox21 and ToxCast data sets. However, the improvements are still limited, as we ignore information from molecules without AC.
only molecules without AC (denoted "non-AC")) The randomly-
Since molecules with AC are both difficult and useful for model training, in the next section, propose a more effective approach to learn from molecules with activity cliff and obtain a molecular property prediction model with discriminating molecular representation.
EFFECTIVE LEARNING FROM SAMPLES WITH ACTIVITY CLIFF
Since molecules with AC are both difficult and useful for model training, in this section, we propose novel training algorithm to effectively learn from molecules with AC for more discriminative
4

---

Published as a conference paper at ICLR 2025
Table I: ROC-AUC different molecular property prediction data sets when only using molecules with/wwthout AC for training.
Method
Tox21
ToxCast
Sider
MUV
Bace
BBBP
ClinTox
HTV
GraphGPS (all samples)
71.5
68.5
56.4
66.9
76,9
67.0
71.1
77.0
GraphGPS (AC only)
71.8
69.2
56.5
68.8
77.6
67.0
67.8
72.2
GraphGPS (non-AC only)
67.8
66.9
56.3
69.2
75.8
66.3
67.4
74.8
3D PGT (all samples)
73.8
69.2
60.6
69.4
80.9
72.1
79.4
69.4
3D PGT (AC only)
74.0
70.1
59.7
67.3
79.9
68.6
68.7
3D PGT (non-AC only)
68.6
68.9
58.6
64.6
79.1
65.7
77.3
69.1
molecular representations. We first reformulate molecular property prediction as node classification problem, with the graph structure induced by the structural similarity in Section In Section[4.2) we propose a novel sample selection method that gradually selects hard molecules with AC training. We further propose novel edge-level task to align the model with AC on different properties in Section
4.1 MOLECULAR PROPERTY PREDICTION AS NODE CLASSIFICATION
Given set of molecules, the definition of matched molecule pairs (Definition) naturally induces graph = (V.E). Each molecule corresponds to a node, whose features correpond to the molecular representation obtained by the pre-trained models. Two nodes (molecules) are connected if they are a matched molecule pair. Figure shows an example subgraph for seven molecules on two property prediction tasks (responses ARE and MMP receptors) from the Tox21 data set. As mentioned in Section 3 matched molecule pair may have activity cliff on one property (dashed edges in Figure 4 but not the other (solid edges). The graph G allows to formulate molecular property prediction as a node classification problem, where the node labels describe the properties of different molecules. This graph formulation is different from those proposed by /Zhuang al. (2023) and /haa al.) (2024). as they do not consider the AC information inside the graph (reflected by the different types of edges in Figure
H&N
My
HO
(a) Responses to ARE receptors.
(b) Responses MMP receptors.
Figure 4: An example graph. Molecules with similar structures (as defined by Definition 3.1 are connected by edges. The edge is dashed (resp. solid) when the two molecules involved have different (resp. the same) properties.
4.2 NODE-LEVEL TASK FOR MOLECULES WITH ACTIVITY CLIFF
we consider the use of curriculum learning [Wang et al. 2022), which first selects easier samples and then harder samples to gradually train a better model. However, from Figure g even for molecules with similar losses, molecules with AC are still more difficull to learn than molecues without AC. As such, propose weighted curriculum learning algorithm that jointly considers AC and the molecule's training loss. Specifically, for given molecule i, we define its weighted loss as (ww (w)),, where (w) is the original loss on molecule (e.g., cross-entropy loss for classification tasks, squared loss for regression
Since molecules with AC are more difficult to learn (Section 3)

---

Published as a conference paper at ICLR 2025
tasks), and pj is the weight molecule i defined as:
Pi C molecule has activity cliff
molecule does not have activity cliff
with p < 1. In other words, molecules with AC have higher weights than those without AC. Thus, when two samples have the same loss values (i.e., equally difficult for the model), we select molecules with AC first. iteration I, let the sampled mini-batch be B. We select subset B of large-loss samples from B:
B(w)= {ili (,((()))) 2 R(t) percentile of loss in 83.
In other words, R(t) controls the percentage of casy molecules that are discarded at iteration t, as we focus more on hard molecules that cannot be learned well. The loss on B, namely. L(w; B(w)) [B(w)]] Bie (wr), is then used to update the model. This allows the model to gradually focus more on the difficult molecules with AC that are more useful for making accurate prediction.
4.3 EDGE-LEVEL TASK FOR ACTIVITY CLIFF PAIRS
While the aforementioned sample selection method can better learn from molecules with AC, it only considers the molecules separately. However, AC is defined for a pair of molecules, and they may affect the predictions of each other. As such, we introduce an edge-level task. Specifically, for each edge eij = (vi, U2) in G, we define the loss:
Lee (w) (1: ()) Ū3 ()())),
(2)
where (resp., Yj) is the binary label for molecule i (resp., j), and (w)) (resp., Ú (w)) is the prediction for molecule (resp.. j) with model parameter w. For a classification task, y/: yj indicates that molecules i and have the same label and do not form an AC pair. On the other hand, when Yi * yj- we have AC and y: - Yj = #1. When y, 1 (which implies Yj = 0), equation (2) minimizes - (is Új), ). which corresponds to maximizing
and minimizing 1/. When yi 0 (which implies yj 1), equation ( minimizes (úi úz), which corresponds to minimizing ús and maximizing ÿj- Similar deduction can be obtained for regression tasks well, and we also draw the predictions of both molecules with activity cliff towards the ground truth. The total edge-level loss over all matched molecule pairs is then:
Co A)
1 [A|
1
EeyeA
Ley,
1 L [A|
E
- 31 (w) (w)),
where A C E is the set of all matched molecule pairs. The following Proposition shows the gradient of the edge-level loss in terms of the 00.(w) for each molecule i.
Proposition 4.1. the
(C.(w)
A Σ. n, 1)
@9.(w)
where n, is the number of AC pairs Bw
involving molecule
Proof is in Appendix In other words, the gradient of L. is weighted sum of 86.1m) The the
weight on each (w.(w) Zur depends the number of AC pairs involving molecule i, which does not change throughout training. However, not all AC pairs are equally important for the learning of discriminative molecular representations. Some pairs can be easily separated, while other pairs may be more difficult to distinguish. Thus, we also employ curriculum learning into this edge-level task, and change the edge loss in (3) to:
1 [ÁI
1
(ow A)
Enseh (e, (w)
ler (e,((w) (w).
B(t)[A)
where Á (eyleey E A, R(t) percentile of loss in A}. Using A instead of A allows to focus more on AC pairs Cij E A with larger loss Lesy, which correspond to less well-separated pairs that are more important for model update.

---

Published as a conference paper at ICLR 2025
Algorithm Learning with Activity Cliff (LAC).
Initialize prediction model f with parameter w (random initialization pre-trained weights);
2: for 0, T do
3: Draw a mini-batch B from molecule data set D:
4: Obtain the set A of molecule pairs in B with activity cliff;
5: Determine R(I);
6: Select R(() x [8] large-loss samples B from B based on network f's predictions;
7: Select R(t) X |A| pairs of molecule with activity pairs and compute Le in (4);
8: Update
17 V (Cww; B) + aw; A));
9: end for
4.4 COMPLETE ALGORITHM
The complete algorithm, which will be called Learning with Activity Cliff (LAC), is shown in Algorithm Compared with standard curriculum learning algorithms (Wang et al. 2022) that may be applied training molecular property prediction models, it has the following two key differences: (i) Algorithm D innoolvss training on two different tasks, combined together with a hyper-paramettr o, while existing works only consider curriculum learning
task (namely the node-level task); (ii) We propose a novel design of the curriculum in Algorithm baseed on AC information, which is unique for molecular property prediction. Note that the proposed method be used with various (randomly- inittalizie or pre-trained) base models. It also introduces a hyper-parametts R(t) to control the number of large-loss samples. effect on model performance will be studied in detail in Section 5.4
Table 2: ROC-AUCs on various molecular property prediction classification data sets. The best performance for each task is marked in bold.
Method Tox21
ToxCast
Sider
MUV
Bace
BBBP
ClinTox
HIV
GIN
74.9
61.6
58.0
71.0
72.6
65.4
58.8
75.3
GIN+LAC
75.6
62.2
58.3
72.4
74.8
65.9
61.6
76.1
GraphGPS
71.5
68.5
56.4
66.9
76.9
67.0
71.1
77.0
GraphGPSLLAC
74.0
73.7
60.4
71.3
82.5
67.7
72.4
77.6
GraphMVP
75.9
63.1
63.9
77.7
81.2
72.4
79.1
77.0
GraphMVP4LAC
76.7
70.1
64.5
78.1
81.6
72.9
80.2
77.8
3D-PGT
73.8
69.2
60.6
69.4
80.9
72.1
79.4
69.4
3D-PGT+LAC
75.2
74.0
61.0
75.1
84.5
72.4
79.6
69.5
UniMol
79.6
69.6
65.9
82,1
85.7
72.9
91.9
80.8
Uni
80.2
72.5
66.2
82.7
86.4
73.6
92.2
80.9
EXPERIMENTS
In this section, we demonstrate the performance of the proposed method on both classification data sets (Section 5.1 popularly used in existing works (Stärk al. 2022 Wang al. 2023b Zhouu et al.) 2023) and regression data sets (Section 5.2p that are more common in real-world application (van Tilborg al. 2022). Section 5.3 presents ablation studies verify the effectiveness of each component in the proposed method. The effect of the parrameasssseeess that define R(t) are studied in Section 5.4 Section further visualizes the loss distribution on molecules. Section 5.6 presents case studies to better understand the proposed method.
5.1 EXPERIMENTS ON CLASSIFICATION DATA SETS
In this section, we perform experiments on eight classification tasks from the MoleculeNet Wu et al., 2018): Tox21. ToxCast, Sider, MUV, Bace, BBBP, ClinTox and HIV. The proposed LAC is combined with the following baseline methods: (i) training from scratch with GIN (Xu et al. 2019) and GraphGPS (Rampášek et al. 2022) models, and (ii) using model initializations from the
7

---

Published as a conference paper at ICLR 2025
following pre-training methods: GraphMVP (Liu al. 2022). 3D-PGT (Wang al. 2023b and UniMol (Zhou et al. 2023). Statistics the data sets used and detailed experimental setups can be found in Appendix
The ROC-AUCs on various molccular property data sets are shown Table 2 LAC improves the performance of all the models considered. With LAC, the pre-trained UniMol model achieves the best performance on all data sets except ToxCast, where 3D-PGT pre-trained model performs the best. Moreover, performance improvement depends partially on the proportion of AC samples in the data set. For example, the improvement on Tox21 is generally larger than that on MUV, and activity cliff more commonly encountered in Tox21, as shown in Table
5.2 EXPERIMENTS ON REGRESSION DATA SETS
While the definition for activity cliff straight- forward for classification tasks, recent works (van Tilborg al. 2022 Deng et al. 2023) also consider activity cliff on regression data sets. Following (van Tilborg et al. 2022), we se-
Table 3: MAE on various molecular property prediction regression data sets. The best performance for each task is marked in bold.
Target ChEMBL ID
5-HTIA
MOR
D3R
FXR 2047
HRH3
214
233
234
264
MLP(ECFP)
0,692
0.845
0.669
0.796
0.672
MLPIECFP)+ACC
0.656
0.827
0.635
0.762
0.657
lect five data sets from the ChEMBL database (Zrrazil et al. 2023), which describe the (continuous)
bioactivity values of molecules to specific target. We train MLP with ECFP molecular finger- prints (Rogers & Hahn 2010) it performs best on these data sets in (van Tilborg et al. 2022), with
more experimental details in Appendix B Table (aale3sssan shows the obtained mean absolute error (MAE). The proposed LAC can also improve model performance regression tasks.
Table 4: Ablation studies on different components the proposed method LAC. The evaluation
metric is ROC-AUC (Larger is better).
base model
node-level curriculum
edge-level
Tox21
ToxCast
Sider
MUV
Bace
BBBP
ClinTox
HIV
71.5
68.5
56.4
66.9
76.9
67.0
71.1
77.0
72.0
69.8
58.6
67.2
79.3
66.6
71.6
77.1
GraphGPS
59.3
69.5
67.1
72.1
77.4
74,0
73.7
60.4
71.3
82.5
67.7
72.4
77.6
73.8
69.2
60.6
69.4
80.9
72.1
79.4
69.4
74.0
70.2
60.2
69.1
81.8
69.4
77.4
68.6
3D PGT
74.6
73.0
61.0
72.2
83.1
72.2
79.6
69.5
61.0
72.4
79.6
69.5
5.3 ABLATION STUDIES
In this experiment, study the effectiveness of curriculum learning in the node-level task (Sec- tion 4.2) and the pairwise loss in the edge-level task (Section 4.3). Experiments are performed on the GraphGPS model (randomly-inittalizze) and 3D-PGT model (pre-trained) as representatives.
Table shows the model performance with different values of in (T). Setting p = corresponds only using the original loss and does not distinguish molecules with/without AC, while setting 0 corresponds to using only molecules with AC for training. As can be seen, using p < 1 usually outperforms the baseline with p = demonstrating the effectiveness of AC information in selecting informative molecules for training. However, setting too small can harm model performance as we neglect the congtributions of molecules without AC In this paper, we set p = 0.5 as it achievs the best overall performance. Table 4 shows the ROC-AUCs obtained with without the pairwise (edge-level) task and curriculum learning on samples (node-level curriculum). Both the pairwise task and curriculum learning on samples can generally improve the model performance, and curriculum learning on samples often has more significant improvements. The only exception is the MUV data set 3D-PGT model, where only using pairwise task achieves slightly worse performances. That
8

---

Published as a conference paper at ICLR 2025
is because MUV data set contains fewer molecules with activity cliff, as is shown in Table @ in Appendix Combining both components achieves the best overall performances across all data sets.
Table 5: Ablation studies the effect of activity cliff weights for curriculum learning samples.
The evaluation metric is ROC-AUC (Larger is better).
base model
Tox21
ToxCast
Sider
MUV
Bace
BBBP
Clin
HIV
1.0
73.5
72.6
60.3
72.9
80.0
65.0
71.1
77.0
0.75
73,8
72.9
60.4
72.3
81.7
67.3
71.9
77.5
GraphGPS
0.5
74.0
73.7
60.4
71.3
82.5
67.7
72.4
77.6
0.25
71.6
70.3
57.9
69.8
67.1
70.1
73.4
67.8
66.9
56.3
69.2
75.8
66.3
67.8
72.2
1.0
74.2
73.0
60.7
72.9
81.5
70.5
79.4
69.4
0.75
74.7
73.7
60.9
74.6
83.8
72.1
79.6
69.2
3D PGT
0.5
75.2
74.0
61.0
75.1
84.5
72.4
79.6
69.5
0.25
72.4
71.9
59.2
70.4
81.3
71.8
73.6
69.1
68.6
68.9
58.6
64.6
79.1
65.7
69.1
68.7
Table 6: ROC-AUC different data sets with different types of R(t) schedules. 3D-PGT pre-trained
model is used.
Schedule
Tox21
ToxCast
Sider
MUV
Bace
BBBP
ClinTox
HIV
linear
75.2
74.0
61.0
75.1
84.5
72.4
79.6
69.5
74.5
73.2
59.5
71.0
83.5
70.0
79.3
69.2
geometric
75.0
73.7
60.5
75.0
85.0
71.2
79.6
69.3
5.4 IMPACTS OF R(t)
In this section, we investigate how different R(t) schedules (in Algo- rithm [ affects the performance of LAC. We consider the follow- ing three schedules: (i) linear:
Table 7: ROC-AUC on Tox21 data set with different and for LAC. 3D-PGT pre-trained model is used.
R(t) A mmin(t/(22),, 1). which
0.1
0.2
0.3
À
A= 0.5
increases the difficulty of training
7 7=0.1
75.1
75.1
73.8
samples uniform rate; (ii) root:
0.2
74.0
75.0
75.1
72.7
74.3
R(t) Amin( vt/(?2),,, 1), which
7-0.3
73.0
75.0
75.0
72.3
72.2
introduces more hard samples in
7-0.4
73.7
73.1
74.1
72.1
72.2
fewer epochs; and (iii) geometric: R(I) = ((2min((((77)))) trains for greater number of epochs on the subset of easy samples. We set = 0.1 and À = 0.2 for all schedules. Ta shows the ROC-AUCs obtained on the various data sets. In general, the linear schedule achieves the best performance, and the root schedule achieves the worst performances.
7-0.5
74.6
72.5
72.3
72.6
73.5
1), which
Using the linear schedule, Table 7 shows the ROC-AUCs on the Tox21 data set with different hyper-parameters and X. The performance is stable on wide range of and values.
5.5 Loss DISTRIBUTIONS FOR MOLECULES WITH ACTIVITY CLIFF
In this section, we compare the training loss distributions on molecules with AC obtained with and without the proposed LAC. Figure (resp. Figure 6) shows the distributions with the randomly- initialized GraphGPS model (resp. pre-trained 3D-PGT model) at the end of the training process. Models trained by the baseline algorithm (blue columns) have inaccurate predictions molecules with AC, while the proposed method LAC (orange columns) can reduce the loss for these samples. LAC improves the performance for both randomly sundomly-initialized -nitiallizz model and pre-trained model.
part of
5.6
CASE STUDIES

---

Published as a conference paper at ICLR 2025
LAC BaneTe
20
04
C4B
700
530 0M 248 240 cal cAo
(a) NR-AhR,
(b) NR-ER.
(c) SR-ARE.
(d) SR-MMP.
Figure 5: Loss distributions obtained by the randomly-initiaizz GraphGPS model on molecules
with AC 4 tasks in Tox21.
LAC
LAC Sanddss
LAC Saudio
LAK
(a) NR-AhR,
(b) NR-ER.
(c) SR-ARF.
(d) SR-MMP.
Figure 6: Loss distributions obtained by 3D-PGT pre-trained GraphGPS model on molecules with
AC on 4 tasks in Tox21.
Finally, we choose some examples to illustrate how the proposed method LAC can improve upon existing molecular property predic- tion model. We choose the UniMol pre-trained model as it achieves the best over- vari- ous data sets. As in Fig- 7(b) without the pro- posed LAC, UniMol cannot correctly classify molecules with AC when the structural differences are very small, even if it can handle cas- ier pairs like in Figure ((a)) With tasks from two levels,
H₂C.
HyC
CH3
HyC
CH₃
HyC.
CH2
all performance
HyC
CH2
(a) Example 1: correctly classified both UniMol and UniMol+LAC
(b) Example 2: wrongly classified by UniMol but correctly classified by UniMol+LAC
Figure 7: Examples of molecules with AC. LAC improves upon isting methods to obtain more accurate predictions on molecules with AC.
LAC further improves the model performance to accurately classify two molecules in Figure 7(b)
CONCLUSION
In this paper, we propose to improve the performance of molecular property prediction models from the perspective of activity cliff (AC). We first empirical results with different tasks and models to demonstrate that standard training pipeline cannot learn molecules with AC well. By reformulating the original problem graph problem, propose novel training algorithm LAC that uses both node-level and edge-level tasks to effectively learn from molecules with AC. Extensive empirical results demonstrate that the proposed method significantly improves the performance of different baseline methods.
ACKNOWLEDGEMENTS
Q. Yao's work is supported by National Natural Science Foundation of China (under Grant No. 92270106) and Beijing Natural Science Foundation (under Grant No. 4242039). This is also supported in part by the Research Grants Council of the Hong Kong Special Administrative Region (Grants 16200021, 16202523 and C7004-22G-1).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Simon Axelrod and Rafael for property prediction and molecular generation. Scientific Data, 9(1):1-14, 2022.
Geom, energy-annotatee molecular conformations
Lowik Chanussot, Abhishek Das, Siddharth Goyal, Thibaut Lavril, Muhammed Shuaibi, Morgane Riviere, Kevin Tran, Javier Heras-Domingo, Caleb Ho, Weihua Hu, Aini Palizhati, Anuroop Sriram, Brandon Wood, Junwoong Yoon, Devi Parikh, C. Lawrence Zitnick, and Zachary Ulissi. Open catalyst 2020 (oc20) dataset and community challenges, ACS Catalysis, 11(10):059-6072, 2021.
Markus Dablander, Thierry Hanser, Renaud I Lambiotte, and Garrett M. Morris. Exploring QSAR models for activity-cliff prediction. Journal of Cheminformatics, 15:47, 2023.
Jianyuan Deng. Zhibo Yang, Hehe Wang, Iwao Ojima, Dimitris Samaras, and Fusheng Wang. A sys- tematic study of key elements underlying molecular property prediction. Nature Communications, 14:6395, 2023.
Johannes Gasteiger, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems, 2021.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263-1272. 2017.
David E. Graff, Edward O. Pyzer-Knapp, Kirk E. Jordan, Eugene Shakhnovich, and Connor W. Coley. Evaluating the roughness of structure structure-property relationships using pretrained molecular representations. Digital Discovery, 2: 2:1452-1460, 2023.
Yaowen Gu, Si Zheng, Zidu Xu, Qijin Yin, Liang Li, and Jiao Li. An efficient curriculum learning- based strategy for molecular graph learning. Briefings in Bioinformatics, 23(3), April 2022.
Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew R. Scott, and Dinglong Huang. Curriculumnet: Weakly supervised learning from large-scale web images, In Computer Vision ECCV 2018, pp. 139-154, 2018.
Dragos Horvath, Gilles Marcou, Alexandre Varnek, Shilva Kayastha, Antonio de Vega de León, and Jürgen Bajorath. Prediction of activity cliffs using condensed graphs of reaction representations, descriptor recombination, support vector machine classification, and support vector regression. Journal of Chemical Information and Modeling, 56(9): 2016.
Md Shamim Hussain, Mohammed J Zaki, and Dharmashankar Subramanian. Global self-attention as a replacement for graph convolution. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022.
Javed Iqbal, Martin Vogt, and Jürgen Bajorath. Prediction of activity cliffs on the basis of images using convolutional neural networks. Journal of Computer-Aided Molecular Design, 35:1157- 1164, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference Learning Representations, 2015.
Johannes Klicpera, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. arXiv preprint arXiv:2003.03: 03, 2020.
Tao Lin, Sebastian U. Stich, Kumar Kshitij Patel, and Martin Jaggi. Don't 't Use Large Mini-Batched Use Local SGD. In International Conference on Learning Representations, 2018.
Tao Lin, Lingjing Kong, Sebastian Stich, and Martin Jaggi. Extrapolation for Large-batch Training in Deep Learning. In Proceedings of the 37th International Conference on Machine Learning, 2020.
Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo. and Jian Tang. Pre- training molecular graph representation with 3d geometry. In ICLR 2022 Workshop on Geometrical and Topological Representation Learning, 2022.
11

---

Published as a conference paper at ICLR 2025
Xuebo Liu, Houtim Lai, Derek F. Wong, and Lidia S. Chao. Norm-based curriculum learning for neural machine translation. In Annual Meeting of the Association for Computational Linguistics, 2020.
Yi Liu, Limei Wang, Meng Liu, Xuan Zhang. Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d graph networks. arXiv preprint arXiv:2102.05013, 2021.
Gerald M. Maggiora. On outliers and activity cliffs why qsar often disappoints. Journal of Chemical Information and Modeling, 46(4):1535-1535 2006.
Christian Merkwirth and Thomas Lengauer. Automatic generation of complementary descriptors with molecular graph networks, Journal of Chemical Information and Modeling, 45-5)1115-11168 2005.
Ajit Narayanan, Edward C Keedwell, and Bjöm Olsson. Artificial intelligence techniques for bioinformatics. Applied bioinformatics, 1:191-222, 2002.
Sanmit Narvekar, Jivko Sinapov, and Peter Stone. Autonomous task sequencing for customized curriculum design in reinforcement learning. International Joint Conference on Artificial Intelligence, 2017.
Junhui Park, Gaeun Sung, SeungHyun Lee, SeungHo Kang, and ChunKyun Park. Acgen: Graph convolutional networks for activity cliff prediction between matched molecular pairs. Journal of Chemical Information and Modeling, 62(10):2341-233 -2351, 2022.
Emmanouil Antonios Platamios, Otilia Stretcu, Graham Neubig, Barnabás Póczos, and Tom Michael Mitchell. Competence-basee curriculum learning for neural machine translation. In North American Chapter of the Association for Computational Linguistics, 2019.
Ladislav Rampášck, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Do- minique Beaini. Recipe for general, powerful, scalable graph transformer. arXiv preprint arXiv:2205. 12.20...2444. 12454, 2022.
David Rogers and Mathew Hahn. Extended-commeectvvty fingerprints. Journal of Chemical Informa- tion and Modeling, 50(5):742-754, 2010.
Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günnemann, and Pietro Liò. 3d infomax improves gnns for molecular property prediction. In International Conference on Machine Learning, pp. 20479-20503 2022.
Jonathan M. Stokes, Kevin Yang. Kyle Swanson, Wengong Jin, Andres Cubillos-Ruiz. Nina M. Donghia, Craig R. MacNair, Shawn French, Lindsey A. Carfrae, Zohar Victoria M. Tran, Anush Chiappino-Pepe, Ahmed H. Badran, lan W. Andrews, Emma J. Chory, George M. Church, Eric D. Brown, Tommi S. Jaakkola, Regina Barzilay, and James J. Collins. A Deep Learning Approach Antibiotic Discovery. Cell, (81(4):775-48, 2020.
Dagmar Stumpfe, Huabin Hu, and Jürgen Bajorath. Evolving concept of activity cliffs. ACS Omega, 4(11):14360-14366 2019.
Shunsuke Tamura, Tomoyuki Miyao, and Jürgen Bajorath. Large-scale prediction of activity cliffs using machine and deep learning methods of increasing complexity. Journal of Cheminformatics, 15:4, 2023.
Raphael Townshend, Rishi Bedi, Patricia Suriana, and Ron Dror. End-to-end learning on 3d protein structure for interface prediction. Advances in Neural Information Processing Systems, 2019.
Richard Tran, Janice Lan, Muhammed Shuaibi, Brandon M. Wood, Siddharth Goyal, Abhishek Das, Javier Heras-Domings Adeesh Kolluru, Ammar Rizvi, Nima Shoghi, Anuroop Sriram, Félix Therrien, Jehad Abed, Oleksandr Voznyy, Edward H. Sargent, Zachary Ulissi, and C. Lawrence Zimick. The open catalyst 2022 (oc22) dataset and challenges for oxide electrocatalysts. ACS Catalysis, 13(5):30663000 2023.
12

---

Published as a conference paper at ICLR 2025
Derek van Tilborg, Alisa Alenicheva, and Francesca Grisoni. Exposing the limitations of molecular machine learning with activity cliffs. Journal of Chemical Information and Modeling, 62(23): 5938-5951, 2022.
Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
Daixin Wang. Zhiqiang Zhang, Yeyu Zhao, Kai Huang, Yulin Kang, and Jun Zhou. Financial default prediction via motif-preserving graph neural network with curriculum learning. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023a.
Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):4555 4576, 2022.
Xu Wang, Huan Zhao, Wei-wei Tu, and Quanming Yao. Automated 3d pre-training for molecular property prediction. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, 2023b.
Xiaowen Wei, Xiuwen Gong, Yibing Zhan, Bo Du, Yong Luo, and Wenbin Hu. Clnode: Curriculum learning for node classification. Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, 2022.
Fang Wu. A semi-supervised molecular learning framework for activity cliff estimation. In Pro- ceedings of the Thirty-Thine International Joint Conference on Artificial Intelligence, IJCAI-24, 2024.
Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg. Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical Science, 9:513-530, 2018.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019.
Chengxuan Ying. Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? 2021.
Barbara Zdrazil, Eloy Felix, Fiona Hunter, Emma J Manners, James Blackshaw, Sybilla Corbett, Marleen de Veij, Harris loannidis, David Mendez Lopez, Juan F Mosquera, Maria Paula Magarinos, Nicolas Bosc, Ricardo Arcila, Tevfik Kizilören, Anna Gaulton, A Patrícia Bento, Melissa F Adasme, Peter Monecke, Gregory A Landrum, and Andrew R Leach. The ChEMBL Database in 2023: drug discovery platform spanning multiple bioactivity data types and time periods, Nucleic Acids Research, D2(D1):D1800DDI 180-DD19.. 2023.
Ziqiao Zhang, Bangyi Zhao, Ailin Xie, Yatao Bian, and Shuigeng Zhou. Activity Cliff Prediction: Dataset and Benchmark, February 2023. arXiv:2302.07541
Bangyi Zhao, Weixia Xu, Jihong Guan, and Shuigeng Zhou Molecular property prediction based on graph structure learning. Bioinformatics, 30t5):btae304, 05 2024.
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: universal 3d molecular representation learning framework. In The Eleventh International Conference on Learning Representations, 2023.
Xiang Zhuang, Qiang Zhang, Bin Wu, Keyan Ding, Yin Fang, and Huajun Chen. Graph sampling- based meta-learning for molecular property prediction. In Proceedings of the Thirty-Secome International Joint Conference Artificial Intelligence, IJCAI-23, 2023.
13

---

Published as a conference paper at ICLR 2025
A
PROOF OF PROPOSITION 4.1
Proof.
tL. dw
Oles
Σ EenEA EA Er
AAl
Oir Ow
dūj dw
Σ En (yi
U;)(
[A|
Regarding yi Vj, since Eij E A indicates that molecule and have activity cliff, consider the following two cases:
y/s 1, then we must have yj = 0, and yi Vj 21/
1/ 0, then we must have Uj = 1, and y/ Vj 21/
Thus, have y: yı 2y always holds and:
(C))
1
Exseged
Σ
-(2y: 1)
0g: Oun
Ou
[A
1
E,
(2y/ 1)
A
O'o
B EXPERIMENTAL DETAILS
All experiments are run on a single NVIDIA RTX A6000 GPU. For all experiments in this work, we use the Adam optimizer (Kingma & Ba, 2015), and follow its default paremetarm:::::: learning rate is set 0.001, first-order momentum weight 31 is set to 0.9, and the second-order momentum weight B2 set to 0.99. The batch size is set to 256 for all data sets.
Unless otherwise specified, we set the R(t) schedule as R(t) Amin(t/(7T).1) with À = 0.2 and y 0.1, and the weight for pairwise loss C. is set to 0.1. For the classification experiments, the data splits of all data sets in our experiments follow the scaffold split in (Wang et al. (2023b). For the regression experiments,
data splits all data sets our experiments are the same as in (van) three-layer MLP model with input dimension 1024 and hidden
Tilborg et al., 2022). and dimension 512 for all hidden layers.
All data sets used in our experiments are released under MIT license. Some statistics data sets used in experiments are in Table (classification) and Table 9
(regression).
Table Summary for the data sets used for classification tasks.
Tox21
ToxCast
Sider
MUV
Bace
BBBP
ClinTox
HIV
# molecules
7831
8521
1427
93087
1513
2039
1477
41127
#
MMPs
3212114 14
3802710
11935
2243595
15894
24105
7080
20740266
# AC pairs
315841
381260
3183
2610
1470
1186
1912
2484912
AC ratio (%)
9.83
10.03
26.67
0.12
9.23
4.92
27.01
11.98
C ADDITIONAL EMPIRICAL RESULTS
C.I ABLATION STUDY ON THE IMPACT OF CURRICULUM LEARNING FOR PAIRWISE TASK
Table 10 compares the model performances whether curriculum learning for pairwise task. We see that using curriculum learning for pairwise task further improves the performance than using the naive pairwise task for most data sets.
14

---

Published as a conference paper at ICLR 2025
Table 9: Summary for the data sets used for regression tasks.
5-HTIA
MOR
D3R
FXR
HRH3
molecules
3317
3142
3657
3097
2862
MMPs
19240
17200
26707
21264
15652
AC pairs
6734
6045
10418
9282
5913
AC ratio (%)
35.00
35.15
39.01
43.65
37.78
Table 10: Ablation studies on the effect of curriculum learning for pairwise task. The evaluation metric is ROC-AUC (Larger is better).
Base model
Pairwise curriculum
Tox21
ToxCast
Sider
MUV
Bace
BBBP
ClinTTox
HIV
GraphGPS
73.0
73.2
59.7
70.6
81.6
67.8
71.9
77.5
V
74.0
73.7
60.4
71.3
82.5
67.7
72.4
77.6
3D PGT
74.0
73.5
60.8
71.0
83.6
70.4
78.9
69.1
75.2
74.0
61.0
75.1
84.5
72.4
79.6
69.5
C.2 EFFECTS OF BATCH SIZE
Table compares the performance with different batch sizes for LAC on both GraphGPS and 3D-PGT model. While we set the batch size to be 256 for all data sets in our experiments, we can see that setting the batch size either too large (1024) or too small (64) may not lead to the best performance. Setting the batch size too small cannot cover enough activity cliff pairs in the edge-level loss of our method, hence cannot utilize this task well and may even leads to performance worse than the standard training pipeline (e.g., the Tox21 1 data). While setting the batch size larger leads to some improvement on large data sets like MUV or ToxCast, it leads to even worse performance for other data sets with limited molecules like Sider BBBP. Such observation agrees with existing theoretical works on stochastic optimization for neural networks (Lin et al. 2018 2020), as they demonstrate that large batch sizes can lead worse generalization performance. Therefore, although setting the batch size to be larger can include more activity cliff pairs in single batch, may still not lead to better performance all data sets.
Table 11: Ablation studies on the effect of batch size. The evaluation metric is ROC-AUC (Larger is better).
Method
Batch size
Tox21
ToxCast
Sider
MUV
Bace
BBBP
ClinTox
HIV
64
72.9
72.1
59.7
70.7
81.9
67.1
72.2
77.3
GraphGPS+LAC
256
74.0
73.7
60.4
71.3
82.5
67.7
72.4
77.6
1024
73.9
73.8
58.2
71.6
81.5
66.4
71.7
77.4
64
74.9
73.8
60.5
73.9
83.8
72.1
79.3
69.1
PGT+LAC
256
75.2
74.0
61.0
75.1
84.5
72.4
79.6
69.5
1024
75.0
74.0
60.1
75.2
81.3
71.8
76.6
69.2
C.3 TIME COST ON ACTIVITY CLIFF DETECTION
Table compares the total time cost in fine-tuning for the standard training pipeline and our proposed method LAC. Note that compared to standard training. LAC involves an additional process of finding all activity cliff pairs, therefore we show its time cost in two parts parenthesis, where the first number represents the time cost of finding all activity cliff pairs and the second number represents the time cost fine-tuning in Algorithm We can see that the time cost for our method is almost the same as the standard training pipeline. In other words, the node and leve-level tasks do not incur much additional time cost. Also, the time cost of finding all activity cliff pairs is generally limited compared to fine-tuning.
15

---

Published as a conference paper at ICLR 2025
Table 12: CPU time cost (in minutes) of standard training pipeline and the proposed method LAC when fine-tuning 3D-PGT/UniMol model.
Data sets
Tox21
Sider
Bace
BBBP
3D-PGT
156
54
62
69
3D-PGT+LAC
196 (37+159)
58 (3+55)
70 (5+65)
73 (4+69)
UniMol
208
77
83
91
UniMol+LAC
247 (37+210)
80 (3+77)
88 (5+83)
97 (4+93)
C.4 ENLARGED FIGURES IN SECTION B AND ADDITIONAL MOTIVATION RESULTS
Figure X shows the average training losses for molecules with AC and molecules without AC. As can be seen, for all the four setups, molecules with AC have significantly larger training losses than molecules without AC. This demonstrates that molecules with AC are more difficult to learn due to their similar structures yet different properties. Moreover, from Figures [(c) and [(d) we can see that this phenomenon also exists for the 3D-PGT and Uni-Mol pre- trained models. In other words, molecules with AC are still more difficult to learn during fine-tuning of pre-trained models.
NR-AAR (AC)
0.7
NINANR
(AC)
NR. ER Inon-ACI
0.6
NR. ER Inon-AC)
SR ARE (ACI
SR ARE (ACI
0.7
0.5
0.5
SR-MMP (neRwAC)
SA-MMP Incn-AC)
0.4
0.5
0.3
0,4
0.2
0.1
20
40
60
80
100
120
140
20
40
80
BC
100
120
140
Epochs
Epochs
(a) GIN.
(b) GraphGPS
NI-AHR (AC)
MR-ABR (AG)
NR-AHR (non-AC)
NR.ARR (ron-AC)
NR ER (AC)
NR ER (ACI
NR
0.5
SR-ARE (nee-AC)
0.4
SRARRE (nen-AC)
5AHMPP (AC)
SR-MMP (AC)
SR-MMP (non-AC)
SR-MMP (non-AC)
0.4
0.3
8.1
140
20
Epoch
(c) Pre-trained 3D-PGT.
(d) Pre-trained Uni-Mol.
Figure& 8: Training losses of molecules with and without activity cliffs in four model training setups.
Since a pair of molecules with activity cliff have large difference in their properties, they may have larger influence on the prediction of each other during training. To demonstrate this, F the average difference training losses (*loss gap") between molecules with activity cliff. As can be seen, AC leads to loss gaps between two molecules. which also indicates that all these models fail to accurately classify both molecules, as in such cases the loss gap should be small (both with small loss). Instead, current models make the same prediction for these two molecules with AC. Only one
16

---

Published as a conference paper at ICLR 2025
molecule is correctly classified with small loss, while another molecule has large loss that leads to the large loss gap.
3.0
NR-AhR
NR ARR
22
SAWARE
2.5
SA-MRE
56-MMP
SA.01M
2.0
1.8
1.6
1.5
1.0
1.2
M
0.0
0.0
20
40
80
100
120
140
20
40
90
100
120
140
Epochs
Epochs
(a) GIN.
(b) GraphGPS.
NRWAhi
2.4
KilWhii
MR-EN
Kik-E
SR.ARE
22
SR-RR
Www
18.
936
1.5
Loss
1.0
14:
0.5
0.8
140
Epochs
(c) pre-trained 3D-PGT.
(d) pre-trained Uni-Mol.
Figure 9: Loss gaps of molecules with AC for different tasks during model training.
17

---

Published as a conference paper at ICLR 2025
MA-En
NIR-ER
Repart fand sarples with AC
SR,ARE
0.9
SR.ARE
MMM
of with
0.8
0.7
0.6
0.4
Quantile large samples
large samples
(a) GIN.
(b) GraphGPS.
NR-AhR
1.0
NIF-Ahi
SR-ARE
SR-MMM
Proporton samplos wit AC
0.9
SR-MMP
troorbnn simpls wite
0.8
0.8
0.7
0.6
3.6
0.5
34
34
6.0
0.2
0.4
0.6
0.8
1.0
0.0
6.2
0.4
0.6
0.8
100
Quantite of large lose samples
Ouantle of targe-loss samples
(c) Pre-trained 3D-PGT.
(d) Pre-trained Uni-Mol.
Figure 10: (Larger version of Figure Proportion of molecules with AC among molecules with % loss values.
18

---

Published as a conference paper at ICLR 2025
NRAAR
1.8
NIJAHR (nunAAC)
NIAAN (HonACI)
NI. ER (AC)
NALER IAC)
(AC)
1.6
SR-RRE tron AC
SR-ARE (nOT-ACI
SRAIMP (AC)
SRAMP (AC]
SR-MMP (NERWAE)
SI-MM (non-AC)
1.0
0.6
(a) GIN.
(b) GraphGPS.
NR-AIR (AC)
NR-AER (ACI
(non
16-
(AC)
NR-ER (norAC)
NR-ER, (non-AC)
SR-ARE JAC)
14-
SRARE (ACT
S.RAALE (nor-AC)
SWARE (nor-AC)
SH-HPP
SRIMHP (non-AC)
ACO
605
1.0
1.0
0.6
0.
20
100
120
140
20
60
80
100
120
140
Epochs
Epochs
(c) pre-trained 3D-PGT.
(d) pre-trained Uni-Mol.
Figure 11: (Larger version of Figure 3 Training losses of large-loss molecules with and without activity cliffs in four model training setups.
19


Reviews:
[{"review_id": "ZaVbeo1prm", "reviewer": "Reviewer_gRFn", "summary": "The paper proposes to use curriculum training to address the activity cliff (AC) problem in molecular property prediction. The proposed method re-formulates property prediction task as node classification where the molecules are considered as nodes, and the edges are constructed based on whether the molecules are AC pairs. The edge-level task also helps curriculum training. Extensive experiments are conducted on MoleculeNet dataset by adding the proposed method to various baselines to evaluate the model performance.", "questions": "See weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles a very important task in molecular property prediction, and is from the training perspective. \n- The adaptation of curriculum training here is rational. \n- The paper has performed a series of analyses to show the findings.", "weaknesses": "- Activity cliff is a critical concept in chemistry, primarily based on differences in molecular structure. Any method addressing this task should be motivated by chemical intuition related to these structural variations somehow. However, the proposed method lacks insights that would specifically address activity cliffs. It is more like tackling a hard sample problem rather than focusing explicitly on the activity cliff challenge.\n\n- How is the \"training loss for the top 10%-loss molecules with and without AC\" calculated? Are the molecules involved in the matched pairs removed from this calculation? If the training involves only AC vs. non-AC molecules, the results seem fairly intuitive. Additional clarifications would help make these experiments easier to understand.\n\n- In Fig. 4, are the dashed and solid edges used to denote different labels or categories, like being used differntly? Or are they simply shown to illustrate how AC pairs vary?\n\n- The selection of molecules is based on loss differences, but this approach does not necessarily correlate with AC performance. For instance, an AC might result in high loss, but a high loss does not necessarily indicate an activity cliff.\n\n- Each task seems to require a separate graph, as molecules can behave differently depending on the properties being evaluated. This approach might incur significant computational costs. Table 9 shows the results of AC pairs obtained in each dataset, which helps clarify the data size. However, the number of pairs seems quite large, making the computational complexity and scalability other concerns.\n\n- The backbone models seem not very new. There are many recent SOTA models and the authors should evaluate the performance of adding the proposed component to these models. \n\n- For some datasets, the method shows only marginal improvement. Have the authors explored the underlying reasons for this?\n\n- The experiments seem only run once, which might not robust. The authors could try cross validation or run several times to report the standard deviation. \n\n- There are already proposed AC datasets [1, 2], why the authors do not evaluate the proposed method on them? \n\n[1] Van Tilborg, Derek, Alisa Alenicheva, and Francesca Grisoni. \"Exposing the limitations of molecular machine learning with activity cliffs.\" Journal of chemical information and modeling 62, no. 23 (2022): 5938-5951.\n[2] Zhang, Ziqiao, Bangyi Zhao, Ailin Xie, Yatao Bian, and Shuigeng Zhou. \"Activity cliff prediction: Dataset and benchmark.\" arXiv preprint arXiv:2302.07541 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MvtvvIHaDB", "reviewer": "Reviewer_B7eX", "summary": "This paper focused on the molecular property prediction task, in particular accounting for properties exhibiting activity cliffs, which are defined as minor structural changes conferring significant changes in activity.  A new method based on curriculum learning is proposed, where property prediction is formulated as a node classification problem on a graph where nodes are molecules and edges encode molecular similarity. The proposed approach is evaluated using several classification and regression datasets and different molecular encoders.", "questions": "See weaknesses. In particular, The authors should 1) better clarify the claims, 2) reference previous work focused on AC prediction 3) better clarify the novelty of the proposed approach, 4) include more baselines.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Except for a few points, the method is clearly described and the paper is relatively easy to follow.\n- Figures help understand the intuition for the method and the method itself.\n- The initial analyses are well-conducted and help familiarize with the challenge tackled in this paper.", "weaknesses": "- The main claims of this work appear to be not well supported by results. In particular\n    - Line 70: \"We are the first to investigate why...\". I could not see results indicating why molecular property prediction models struggle in these casese. Instead, the work include some empirical evidence that only reinforces the (known) observation that generalizing to AC is challenging.\n     - Additionally, many works investigated activity cliff in the context of property prediction. See for example \"Zhang et al., Activity Cliff Prediction: Dataset and Benchmark, 2023\", or \"Wu et al., A Semi-Supervised Molecular Learning Framework for Activity Cliff Estimation, 2024\". These works are not cited or compared.\n    - Line 75: \"We propose to re-formulate molecular property prediction as a node classification problem.\". This does not appear completely novel, see for example \"Zhuang et al., Graph Sampling-based Meta-Learning for Molecular Property Prediction, 2023\" or \"Zhao et al., Molecular Property Prediction Based on Graph Structure Learning, 2023\", which are not cited. In general, previous works on this direction are not accounted for. \n- Novelty. The novelty of the work appears limited. As stated in line 324, the methodological novelty is the extension from node to node+edge curriculum learning. However, the definition of the edge-level loss (Eq. 2) is based on the same node-level loss. This work largely appears as an application of standard curriculum learning.\n- Lack of baselines. The authors only compare the proposed method to the baseline network, i.e., no other methods of any kind are taken into account. The authors should compare the proposed method to existing methods. These include works focused on AC prediction (see points above for some examples), but also methods broadly focused on representation robustness (e.g., based on adversarial perturbations or mixup) and domain generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kpKmDeepvc", "reviewer": "Reviewer_YTtN", "summary": "------- Update -------\n\nThank you to the authors for addressing the comments and questions I had on the paper, I'd like to say again what a pleasure it was to read and that I really appreciate both the content clarifications and updates to the presentation. \n\nOn reviewing your answers to my questions and reading the paper again, I'm very happy to raise my score to an 8. \nWell done on the paper, and thank you again for all of your hard work. \n\n\n\n-----------\nThank you for a really interesting read. I found both the analysis of model performance on the molecules with activity cliffs a strong addition to the literature, to take something commonly acknowledged and quantify it is vital work, and the novel exploration of the problem very good to see. \n\nSummary:\n* The authors present a novel approach to tackling the problem of molecules exhibiting an activity cliff. This problem plagues computational chemists as many molecules share large portions of their structure, differing only for a small percentage of the molecule, and still have very different properties. Such properties are intuitive to human experts, but hard to handle using many deep learning methods. \n* The authors take this anecdotal knowledge and make strong baseline measurements of the discrepancy found in many SOTA models in cases where activity cliffs are present. \n* The authors then reformulate the problem of molecular property prediction into a node classification problem as part of a graph structure representing molecules with similar structures.\n* They present this worm as Learning with Activity Cliff (LAC) - and use the concept of curriculum learning to slowly introduce harder to separate AC molecules - this work shows improvements across a range of benchmark datasets and conduct extensive ablation studies to investigate which properties are contributing.", "questions": "Questions / Suggestions:\n* What is the impact of the batch size on training here? In line 3 of the algorithm (line 312) the mini batch is chosen, then pairs with the activity cliff found. This means the second term in the loss L_e is dependant on how many samples are found, did you study the impact of the batch size on the training? As a reader I would want to know - my dataset has X% of possible activity cliff molecules, what batch size Y do I need to see an improvement of magnitude Z using this method? Otherwise I would suspect the impact to only be a small reguarlising term? Is this a correct analysis - and could some more detail be given on this point? \n* Do you have any analysis at how effectively the LAC process picks out the molecules which have an activity cliff? It selects the ones with high loss and then forms edges based on the relative labels - but do you know how much of the time these pairs do form AC pairs? \n\n\nFormatting - lower priority but would be good to aesthetically improve\n* Figure text sizes - the text on many of the figures is unreadable at a normal zoom level, try adjusting the matplotlib params to make these more readable. You can always move some into the appendix and leave one or two examples from each figure in the main text. Specially Fig 2, 3 and 6\n* Caption of Fig 7 is a bit unwieldy \n* Line 194 has a typo “even they” -> “even though they” ? \n\n\nThank you again for the work, I found the paper really enjoyable to read and showed strong scientific process. \nSome clarifying on a few points and tidying up of some of the figures are my main concerns, otherwise I find the work very solid.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The analysis of the well known but poorly quantified impact  of molecules with activity cliffs is a really valuable contribution to the field \n* The novel problem construction looks like a genuinely different way to approach training models for molecular property prediction. \n* The extensive ablation study is really strong showing which components of the restructuring are contributing to the improvements in score - this means as a reader I have the ability to start replicating / incorporating this method into new work. \n* The combination of new reformatting of the problem with the curriculum learning approach to slowly introduce more complex examples seems like a very promising path.", "weaknesses": "* The degree of improvement over the baselines in each case was perhaps hard to quantify - I really appreciate the comparison (Table 2, 3) with the baseline models and + LAC - but felt these tables perhaps lacked context. From the tables alone it’s hard to evaluate if the degree of improvement is significant or not. If a couple of reference models could be added to these tables to show other work that would help calibrate my perception of the improvement of the LAC method. \n* The comment about the lack of baseline against which to judge the contributions applies to tables 4 and 5 and 6 as well. (However I recognise that adding / evaluating baseline models for all cases can be expensive / difficult to conduct.) \n* The way the initial node features are generated feels unclear to me - on line 228 “In this graph, each molecule corresponds to a node, and the molecule’s chemical structure can be stored as node features” - How exactly are these features chosen? Are these features the readout from the baseline models (GraphGPS, UniMol ?) or are these simply generated from something like RDKit. My interpretation from the text is the former, but some more clarity here in the text would be good. \n* I find it slightly unclear what the exact message is - my take away is the LAC is a really good way to improve the abilities of an already performed model - but I'm left concluding the paper a bit unsure exactly how strong the case is.\n* The loss distributions in Fig 6 are very hard to read, please use the same binning scheme for both histograms and show them either with low alpha or as lines only so I can see how they change. Also the text is too small, try adjusting figure sizes for these results?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5GSwRak9Ia", "reviewer": "Reviewer_nmrm", "summary": "This paper proposes a curriculum learning-based training method (LAC) for molecular graph learning on property prediction tasks. Empirical evidences are provided to expose the limitation of current molecular graph learning is the insufficient learning for activity cliff molecules. Further, an elaborate definition is designed to convert the general graph learning to node-level and edge-level tasks with activity cliff being considered, followed by a designed integrated loss function. The experiments on several general molecular property prediction tasks show improvements of LAC than ordinary training method.", "questions": "- What are the criteria used for the detection of activity cliffs? Since it is a concept mainly for binding affinity, people may be more interested about how the authors transfer and expand such concept to molecular property tasks (especially regression tasks and some non-affinity property tasks such as BBBP) with rationales behind. Please provide more details about the clear criteria and other necessary descriptions about this. It could be the most important experimental setting and basis of the study to ensure accurate definition for AC is applied.\n\n- How is the alpha ratio for pairwise loss determined?\n\n- Some typos exist. Such as \"ChemBL\" should be \"ChEMBL\".", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The empirical analysis results effectively highlight the current limitations in molecular property prediction, especially concerning the activity cliff issue.\n\n- The definitions of node-level and edge-level tasks, considering activity cliffs, are clearly elaborated.", "weaknesses": "- It’s unclear why only an MLP model was applied for regression tasks but other models (GraphGPS, GraphMVP, etc.) were excluded. Additionally, the choice of specific ChEMBL assay data (please include the ChEMBL ID) over commonly used molecular property regression benchmarks (e.g., FreeSolv, ESOL) is not explained.\n\n- A study on curriculum learning for molecular graph learning and property prediction should be referenced to strengthen the related works section:\n\nRef:\nGu Y, Zheng S, Xu Z, et al. An efficient curriculum learning-based strategy for molecular graph learning[J]. Briefings in Bioinformatics, 2022, 23(3): bbac099.\n\n- A comparison of computation times with standard random-sampling training would provide additional context for performance evaluation regarding the time cost issue.\n\n- LAC demonstrates performance improvements, it would be helpful to clarify how these gains are achieved—does LAC contribute more to activity cliff (AC) data, non-AC data, or both? A table or figure result may be helpful to understand that.\n\n- Since the loss functions for standard training and curriculum learning differ in equations and terms, directly comparing their values may be misleading. An alternative approach would be to show the proportions of large-loss instances for AC data across each training strategy throughout the process (e.g., similar to Figure 3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)