Instructor-inspired Machine Learning for Robust Molecular Property Prediction
Authors: Fang Wu
Shuting Jin
Siyuan Li
Stan Z. Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Molecular Representations
Semi-supervised Learning
Primary area: machine_learning_for_healthcare

Abstract: Machine learning catalyzes a revolution in chemical and biological science. However, its efficacy is heavily dependent on the availability of labeled data, and annotating biochemical data is extremely laborious. To surmount this data sparsity challenge, we present an instructive learning algorithm named InstructMol to measure pseudo-labels' reliability and help the target model leverage large-scale unlabeled data. InstructMol does not require transferring knowledge between multiple domains, which avoids the potential gap between the pretraining and fine-tuning stages. We demonstrated the high accuracy of InstructMol on several real-world molecular datasets and out-of-distribution (OOD) benchmarks.

Instructor-inspired Machine Learning for Robust Molecular Property Prediction
Fang Wu's Shuting Jin', Siyuan Li3, Stan Z. Li
Computer Science Department, Stanford University School of Computer Science and Technology, Wuhan University of Science and Technology School of Engineering, Westlake University
Abstract
Machine learning catalyzes a revolution in chemical and biological science. How- ever, its efficacy heavily depends the availability of labeled data, and annotating biochemical data is extremely laborious. To surmount this data sparsity challenge, we present an instructive learning algorithm named InstructMol to measure pseudo- labels' reliability and help the target model leverage large-scale unlabeled data. InstructMol does not require transferring knowledge between multiple domains, which avoids the potential gap between the pretraining and fine-tuning stages. We demonstrated the high accuracy of InstructMol on several real-world molecular datasets and out-of outtoo-ddisribution (OOD) benchmarks.
Introduction
An enduring obstacle in applied chemical and biological sciences is identifying and making chemical compounds or materials that have optimal properties for given purpose 21] However, the vast majority of progress in these areas is still achieved primarily through time-consuming and costly tril--and-error experiments. Machine learning (ML) has undergone an unparalleled technological advancement, opening up myriad of applications across various domains. Its potential is particularly notable expediting the discovery and development of novel materials, pharmaceuticals, and chemical processes 3. However, ML models' efficacy heavily relies on the availability of labeled data and consistency of prediction targets. Meanwhile, the cost of generating new data labels through wet experiments is prohibitively high. Consequently, the size of labeled data in this field is several orders of magnitude lower than the one that can inspire breakthroughs other MI. fields ( This data scarcity severely hampers ML in addressing scientific challenges within this realm, impeding its ability to generalize to new molecules
Biochemists have noticed this problem and propose several strategies to overcome the low data limitation (view Fig. [T. Firstly, inspired by the remarkable success of self-supervised learning from NLP 8 and CV [9] [0], researchers apply the pretrain and finetune paradigm [11 to molecule modeling. They boost the performance of various molecular models by pretraining them on massive unlabeled data. However, this benefit can be negligible when large gap exists between the sample distributions of pretraining and downstream tasks [1]. An alternative option to use active learning [16], which iteratively selects new data points to annotate according to the current model's predictions. However, auxiliary labor is still required to enrich the original database. Thirdly, domain knowledge is incorporated to enhance molecular representations, such providing more high-quality hand-crafted features [7]. constructing motif-based hierarchical molecular graphs [18). and leveraging knowledge graphs [19]. However, domain knowledge can be biased and is difficult to integrate into different training techniques universally.
"Corresponding Author, email: fangwu37@stanford. edu.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Unisbeled Datb
Congenent
Deselding
Experizante
13.88
Maccmito grezzet
Landing
C corning
Autoomooling
Labsted Daz
Protraining
B. Aediveee Learning
Kowwledge
Auggnesbdiz
Arteyonen
Parudo-laehie
Mitap-hased Methed
Stpptement
Horralex
Teacher
Besaffuil
Consitoncy Cost
C. Domain Knowledge
Db. Sann--rprrviel Fand Learning
Figure 1: Four mainstream paradigms ameliorate the scarcity of labeled biochemical data. (A) Self-supervised pretraining tasks include masked component modeling, contrastive learning, and auto-encoding. (B) Active learning involves the iterative selection of the most informative data, in which the molecular models are the most uncertain. These samples are then subjected to laboratory testing determine their labels. This process repeated with newly labeled data added to the training set. (C) Knowledge graphs are introduced to provide structured relations among multiple drugs and unstructured semantic relations associated with different drug molecules.
(D) In SSL,, the unlabeled data is used to create a smooth decision boundary between different classes or to estimate the distribution the input data, while the labeled data is used to provide specific examples of the correct output.
In this study, we develop InstructMol, flexible semi-supervised learning (SSL) approach excavate the abundant unlabeled biochemical data for robust molecular property prediction. It differs from prior studies in two aspects: (1) utilizes an additional instructor model to predict the confidence of the predicted label, measure its reliability, and generate pseudo-label information for unannotated data; (2) with the help of pseudo-label information guiding the model, the target molecular model reliably utilize unlabeled data without the need to transfer knowledge between multiple domains, which perfectly avoids the potential gap between pre- pre-training and fine-tumin tne-tuning stages. We demonstrate that InstructMol outpasses existing SSL approaches and achieves tfte-off-hhe-nt art performance in MoleculeNet and several OOD benchmark datasets. Besides, via InstructMol, we accurately predict the properties of all newly discovered drug molecules in the latest patent (ZAA20203668..) A).. Extensive experiments showcase the effectiveness of our model in surmounting the challenge of data scarcity, propelling advancements in the chemistry and biology domains.
2 Related Work
How to exploit large-scale unlabeled molecular data becomes an essential topic in the MI. community alleviate the scarcity of labeled data and improve OOD generalization, where pretraining and SSL are the two major fast-growing lines. The former traditionally employ unsupervised techniques to
pretrain ML models, such as autoencoder 20 autoregressive modeling [13 211. masked component context-prediction context- prediction 24 contrastive learning (25), and multi modti-modality 266. Despite several progress claims, the benefits of self-super pretraining can negligible in many cases (15). Recent years have witnessed rising interest in developing SSL to reduce the amount of required labeled data [27] Several hypotheses have been discussed the literature to support specific SSL design decisions [28], such as the smoothness and manifold assumptions. Existing SSL algorithms can roughly separated into three sorts: consistency regularization, proxy-label methods, and generative models. The consistency regularization is based the simple concept that randomness
modeling 22 23

---

Phase Assignmmen Pseudo-label
Phase 2: Instructive Learning
Properties Prodiction
Labeled Duta
0.74
Pscudo-lube lubels
28
OOD Property Prediction Train
3,9
POLO
Target Molecular Model
Instructor Model
Datasets
Tusst thebe
Repoccs
Prediction Label
Label Clussifeer
Pretraining
doooo
Welli tralined Molecular Model
Labeled
Untebeted Cuta
Loso
Preudo-
3.05
1.24
Pretraining
Modill
Instructor Model
Logg Htaoneightingg
Lose'
Datasets
Performance
Improve
Figure 2: The outline of InstructMol. We utilize pre-trained target molecular model forecast the properties of unlabeled examples as pseudo-labels. Then, an instructor model predicts the confidence of those secudo-annotations, which leveraged to guide the target molecular model to distribute different attention in inferring different data points.
within the neural network (NNs) such as dropout) or data augmentation transformations should not modify model predictions given the same input and impose an auxiliary loss. This line of research includes m-model [29], temporal ensembling [29). and mean teachers [30 The proxy-label methods
regard proxy labels on unlabeled data as targets and consist of two groups: self-training (31) where the model itself produces the proxy labels, and multi-view learning [32), where proxy labels produced by models trained on different views of the data. The generative models rely on VAE [33
and GAN 34
to capture the joint distribution more accurately.
Preliminaries and Background
Task formulation. Suppose we have access to some labeled molecular data D {xx, Ms) > a; can be in any kind of formats such as ID sequences, 2D graphs, and 3D structures. yi can be any discrete (e.g., toxicity, and drug reaction) or continuous properties (e.g., water solubility, and free energy). Here we consider continuous property for simple illustration, but our approach can be easily extended to binary- be any category of architectures, including Transformers, GNNs, and geometric NNs. There are also some unseen data Dtest evaluate the performance of the learned model. Traditionally, D is divided No into the train and validation sets as Dirain = (atrrin (raain)) N. and Deal = {(x!") y/!"l) ==1"
multi-labod cireumstances. The target molecular model f X - ) can
Besides, we can obtain some unlabeled data points, denoted as D { where the number of unlabeled data M is usually orders of magnitude larger than that of labeled data N.
Challenge of proxy-labeling. Labeled data D and unlabeled data D sometimes follow significantly different data distributions, ie.,, [P( 31) o (= ), However, many SSL algorithms 29 335 36
assume no distributional shift between D and D* They are likely only reinforce the consistent information in the labeled data D to unlabeled examples D* instead of mining auxiliary information from D*, without exception for proxy labeling [37]. More importantly, despite the versatility and modality- of proxy labeling, it achieves relatively poor performance compared to recent SSL approaches (38]. This arises because some pseudo-labels o can be severely incorrect during training due to the poor generalization ability of classic DL models 33 40) If we directly utilize pseudo- labels from previously learned model for subsequent training, the conformance-biased information in the proceeding epochs could increase confidence in erroneous predictions and eventually lead to a vicious circle of error accumulation 41 42 The situation can be even worse when labeled data D contain noises because of unavoidable experimental errors. Accordingly, it becomes essential to

---

grasp the quality and dependability of pseudo -anootainnt and intelligently select a subset of them to reduce the hidden noise.
Motivation of Mol.ructMol. reduces pseudo-label error rates, NNs' poor calibration renders this solution insufficient. Explicitly, incorrect predictions in poorly calibrated NNs can also have high confidence (i.e., Df + 0 or 9 + 1) (38] More importantly, prior studies such as UPS (38 resort the target model's output 9: as the confidence indicator and produce hard labels by yI
Though confidence is crucial for pseudo-label selection and this selection
[ 2 71] or y = 1 [ S 72). where "1 and 1/2 are pre-defined thresholds. Nonetheless, this selection mechanism becomes inapplicable ) is continuous label space, as networks no longer output class probabilities. For regression tasks, of discloses no confidential information, and numerous biochemical problems are eegression-basees including molecular property prediction 43 451, 3D structure prediction [46, and binding affinity prediction 47). This brings a challenge for probabilistic output-based proxy-labeling algorithms (38]. Thus, instead of depending
U to judge proxy labels' reliability, we accompany the target molecular model f with an additional instructor model g. which plays the role of a critic and predicts label observability, i.e., whether the label is true or fake, g disentangles the confidence prediction and the property prediction, reducing the noise introduced by the pseudo-labeling process.
4
Method
The Overall Instructive Learning Framework. We separate the integral workflow of InstructMol into two phases. In the first step, we retain pseudo-labels { MM = f (1)))=)) There are several approaches to creating proxy labels, such as label propagation via neighborhood graphs [48] Here, we require the molecular model f to directly annotate samples in the unlabeled dataset D Thhe, in the following step, we construct new dataset with both labeled and pseudo-labeled samples as
D' = U {( úi J== and proceed training both the target molecular model f and the instructor model based on this new set. These two procedures are iteratively repeated until f reaches the optimal performance on the validation set Dev!
M
To be specific, the instructor model g (X' xX' H.y) + P forecasts the confidence measure (0 S I 1) of whether the given label y/ belongs to the ground-truth label set } or the pseudo-label set M It digests three items: the data sample with its label (ar vf) D' and an additional loss term (fx)), vf). where H. is traditionally selected as
-------a-uuuredeeeeeee (RMSE) loss or a mean- -bsuuubboouteerrror (MAE) loss for regression tasks and cross-entrope (CE) loss for classification problems. Here we regard Htr() as the ingredient of g input to provide more information about the main molecular property prediction task. At last, the instructor model g is supervised via binary CE loss (BCE) as:
Cg (D', (MANN
Σ BC (LWD)DD
Σ BCE gll,,,l6 (Bl. (l..))).)) (1)
(N/D)ED
where e; E [0, 1] is an integer and represents the observability mask. It indicates whether ye is pseudo-labeled (c: = 0) or not (c: = 1). However, since the number of unlabeled data M is much larger than the number of labeled data N, it is proper to shift the loss function from BCE to a focal loss 44 (FI.) for unbalanced classes as: C, (D', ( EeeAWED FL =
Ee -11 (1) log(r)), where 7 2 0 is a tunable focusing parameter.
Meanwhile, the target molecular model ff receives the discriminative information [ N+M from the instructor model g and uses it to reweight the importance of different samples in backpropagating its gradient. In other words, the instructor model g guides the target model f to deliver different attention different labels that correct labels are attached more importance while erroneous labels are ignored. This can be realized by specially designing the loss of the target model f as:
C (D', (prNNtM) = a Σ (ELD)ED
(z))) +
(2p; - 1) Hy (f (x²) 03), (2) (N),0))DD
Σ
where 0 S a S 1 is a hyper-parameter to balance the dominance of labeled and unlabeled data sets. (.) transforms the original main task into cost-sensitive learning problem [50] by imposing

---

Algorithm InstructMol Algorithm
1: Input: target model f. instructor model g. labeled data D, unlabeled data D*, pseudo-lahel update frequency k. loss weight
2: Initialize and pretrain a target model fo and an instructor model go
3: for epochs
0, 2,... do
4: in n mod k = then
5:
With no gradient:
6:
5r - f(T),
Va E D
> Iteratively assign pseudo- labels every epochs
7:
end if
8:
D' + DU (U((()t M
> Construct the hybrid database
9:
8. < f(x() Va!
E ED
10:
D: + g( Ve Hy(,), Y(r,
E D
1 Predict the confidence scores
(D', 10 N+M t Equation
11:
12:
(D', (M -- Equation
13: Update the parameters of and based on L.(.) and Ly(,)
14:
end for
a group of soft-labeling weights based on the predicted confidence of data labels. That is, for pseudo-labeled samples (i.e., xzj D*), the soft-labeling weight becomes 2p, 1.
This loss format in Equation 2 induces different behaviors on the loss (-) of labeled and pseudo- labeled instances, where the judgment N produced by the instructor model g is leveraged to differentiate their informativeness. Notably, if the instructor model regards a pseudo-label jž to be unreliable (i.e., 0.5 > P3 > 0), the loss (-(-) chooses to enlarge the gap with the pseudo-label 0j. Meanwhile, once the instructor g fully trusts the pseudo-label Új, InstructMol forces the target molecular model f to give more effort to inferring this proxy- proxy-labeled sample. Generally, the more likely pseudo-label f7 is considered reliable by the instructor model (i.e. Pj - 1), the stronger it drives the target molecular model / to make further improvement on inferring this label jij. Otherwise, InstructMol will push / to overturn its previous belief if p; O. Noticeably, a soft-labeling weight for samples with true labels (i.e., o E DD. For instance, a weight factor of can navigate f to concentrate more on samples that are more trusted by g. But we practically discover no significant refinement with this design on labeled data and leave it for future exploration. The whole pseudo-code of InstructMol depicted in Algorithm !
can also impose
Loss Selection for InstructMol. We compare some relevant methods from the literature under the proxy-labeling SSL algorithms in Appendix Tab. 6 containing the vanilla proxy-labeling (PL), curriculum learning for PL (CPL) [51), UPS, and self-interested coalitional learning (SCL) [52]. is structured in three main columns that describe the selection for unlabeled samples, loss function, and fitness for regression problems, Except for SCL and InstructMol, all approaches adopt a subset of unannotated instances for training rather than utilizing the entire unlabeled datasets. SCL can take advantage of all data points from D", but its main limitation is its improper or even severely wrong loss function design. As the confidence score close to SCL assigns a strong negative multiplier factor as T-p; + -00 and forces the target model f to disregard this label, which is actually reliable. On the contrary, when the instructor model g doubts the reliability of 07, the ratio 1 I-P,
becomes a, driving the target model move towards
Guidelines for InstructMol. Before executing InstructMol, it is natural to first obtain well-trained molecular target model through regular supervised learning on D and then initialize an instructor 9o by discriminating pseudo-labels that are generated by fo- This is empirically proven to achieve higher training stability and robustness. Moreover, pseudo-labels are assigned every ic epoch and a proper setting of & is critical to the success of InstructMol. If pscudo-labels are updated too frequently, the training procedure will be volatile the very beginning. While too-large would significantly increase the training complexity. Here we adopt an adaptive decay strategy: with an initial value ko. the update frequency decreases by a factor of 0.5 until it reaches the minimum threshold kmiin
Analysis of InstructMol. After the curation of D', there are two distinct learning tasks during the second stage of InstructMol. Specifically, the target model follows the regular routine to predict the molecular properties. Mcanwhile, the instructor model strives to differentiate whether the label
5

---

Table 1: Performance of three distinct ML models with different SSL methods nine molecular property prediction tasks. For classification tasks, we calculate the ROC-AUC, while for regression tasks, use RMSE as the evaluation metric. The number in the bracket is the standard deviation of three runs.
Classification (ROC-AUC %. higher is better
Regression (RMSE, lower bemerr
Ditisett Molecules # Tasks
BBBР 20:3
BACE
ClinTox 1478
Tux21 7X51
ToxCast 8575
SIDER 1427
ESOL
FroeSolv 6x2
Lipo 4200
1513
1128
1
2
12
617
27
GIN
65. 2)
78.2(0.5)
76.1/0.5)
74.2(0.4)
61.1(0.1)
(0000
(.955(0.028)
0.897 (.100)
(..10)
Semi-GAN
66 (6..)
76.8088)
76.4:0.7)
74.3/0.6)
02 0(0.2)
59: 7111
1 (0088)
D. 0.881/0.011)
0.7255000225
-n-model 29
66:9)0.3)
(0.7
76.5/0.7)
74.7
62. 02.3(0.2)
59 8(100)
1 (0.000))000)
0.881 (0.0155
0.710(U.022)
UPS [8
67.060.4;
78. (0.8)
77.3:0.6)
75.2(0.7)
66. (0.88
62 62.4(1.6)
InstructMol
70.5(0.5)
83.3
86.2(0.6)
76.6(0.3)
67.0(1.1)
64.21 1.2)
0.839 0.75)
018)
0 0.632 0-40)
GAT
64.500.1)
77 (7.3)
03.3/0.3)
72:2/0.4 4)
61.7(0.1)
55 9(0.6
2.008(0.01))
U. 0.806 (08)
0. (222))
Semi-GAN
64.900.2)
(0.1
69.6:0.4)
72.3/0.4)
61.9(0.3)
2(0.7)
2.017(0:034)
0.852(0.017 01
0.802(0.1125)
---modal 20
63.3(0.2)
78.500.4)
68. (0.6)
72.6 71.6
02 (
56 5(0.8)
1.956(0.006)
0.817(0.021)
0.780(0.029)
-PS 58
67:200.6
81.4(0.5)
74,9 (2)
(1.1
66. 61.7
30.3(1.44)
InstructMol
68.1(0.4)
77.3(100) 11.1
70.8(0.8)
(6.74(17)
60.8(1.5)
1.862(0.017)
0.825
0. 0.738(0 028)
GCN
62.4(0.1)
73.8(0.4)
76.3:0.2)
75.6(0.1)
61.5(0.7)
2(0.6)
2.285(0.011)
0.812(00011)
0. 0.756(0.015)
Semi--aAN 34
02.6)0
74:0(0.6)
76.6/0.3)
74.000.2)
64.8(0.8)
600.9
2.198(0.015)
0.744/0.018 (.11)
---mndel 29
62.710.2)
74.4(0.5)
76.6(0.4)
74.3/0.1)
65.0(1.0)
61.4(0.8)
(0026)))020)
0.833(00023)
(.737(
UPS [S
65.8(0.4)
79.0(0.8)
82.2:008)
75.1(1.1)
66.5(1.9)
63 21.11
InstructMol
67.6(0.3
80.1(1.0)
84.0(0.6)
76.7(0.9) 9:
67.5(1.5)
64.7(1.9)
1.975(0.018;
0.811 020)
0. 00701700017)
Table 2: In-domain and OOD performance on the GOOD benchmark All numerical results are
averages across 3 random runs.
GOOD HIV
GOOD PCDD-PCBAT
scaffold
size
scaffold
size
ID
OOD
ID
0OD
OOD
OOD
ERM
82.79
69.58
83.72
59.94
33.45
16.89
34.31
17,86
IRM 57]
81.35
67.97
81.33
59.00
33.56
16.90
34.28
18.05
VREs 58
82.11
70.77
83.47
58.53
33.88
Gm
59
82,60
70.64
81.79
58.98
13.81
16.98
31.95
17.59
DANN
81.18
70.63
83,90
58.68
33,63
16,90
34.17
17,86
Deep Coral 61
82.53
68.61
84.70
60.11
33.47
16.93
34.49
17.94
Q-SAVI 62
82.73
70.66
84.58
59.92
33.90
16.88
34.76
17.99
Mixup
82,29
68.88
83.16
59.03
30.22
16.59
30.63
17.06
DIR [64
82.54
67.47
80.46
57.11
32.55
14.98
32.89
16.61
InstructMol
84.16
72.10
85.44
63.97
36.02
18.55
35.91
19.20
real. Notably, some prior works embody a similar idea of jointly learning multiple tasks. For instance, the multi-objective optimization (MOO) methods [53] exploit the shared information and the underlying commonalities between two tasks and solve the problem by minimizing an augmented loss. In addition, GAN (54) makes generator and discriminator constantly compete against each other. Nevertheless, MOO cannot handle possible contradictions among different tasks in certain settings, where jointly minimizing the augmented loss may impede both tasks from attaining the global optimal (55). While GAN has been praised for generating high-quality data, it is notoriously difficult to train and requires large amount of training data. More essentially, making predictions in an adversarial manner for unlabeled data deviates from our primary goal of pisuudollabling but is a mature technology for domain adaptation [56].
5 Experiments
We carry out wide scope of experiments in all contexts. Section 5.1 in predicting molecular properties compared with various SSL learning algorithms. Section 5.2 veri- fies the superiority of InstructMol in lowering the predictive error over existing OOD generalization algorithms. Section 5.3 investigates the possibility of marring InstructMol with cutting-edge pretrain- ing methods and demonstrates its potency by setting a state-of-the-ar performance on MoleculeNet. Section 5.4 analyzes pseudo-labels' accuracy, the instructor's behavior, and ablation studies.
shows the benefits of InstructMol
5.1 Molecular Property Prediciton
Data and Setups. We first investigate the effectiveness of InstructMol on three sorts of backbone including GCN 65), GAT [66,, and GIN (67) and report their performance on the standard Molecu- leNet (3).
Datasets are divided using scaffold splitting into training, validation, and test sets with

---

Table 3: Performance on molecular property prediction tasks, where GEM+InstructMol achieves
the best result.
Classificution (ROC-ALIC S. higher hetter
Regression (RMSE, lower better
Datasets Molecules
BBBP 2039
BACE
ClinTox 1478
Tex2]
ToxCast 8575
SIDER 1427
ESOL
FioeSotv
Lipo
1513
7821
1128
642
4200
Tasks
617
27
w.o, pretraining
D D-MPNN [63]
71.0(0.3)
S0.9[0.6)
90.6(0.6)
75.9(0.7)
65.5(0.3)
57.0(0.7)
2 2.082(0.082) 1.000000008)
0.683(0.016)
Attentive HP [9
64.3(1.8)
84.7(0.3) 78.4(0.12)
(0.1(
63-7(0.2)
60-6(3.2)
0.877(0.029)
2. 1.073(0.183)
(0.00))
MGCN 70
65.000.5)
73.4(0.8)
90.5(0.4)
74.100.6
58.7(1.9)
w. pretraining
N-Giramer
(3.7(0
77.9(1.6)
77.6(4.0)
74-3(0.4)
(6.7
(4074(0107)
2 2.68%(0000
0.812/0
N-GramxGB
(0.8)
79.1 11:3)
87.5(2.7)
(0.9(0.9)
(0.70.7)
1.083(0.882)
5.061 (0.744)
2.072(0.080)
PretrainGNN
72
61.3
81.5
72.6(1.5)
(0.1
65.7(0.6)
(2.7(0.8)
1.100(0.006)
2: 2.764(0.002)
0. 0.739K000003 0.003)
InfoGraph
t9
69.2(0.8)
2..9
(5.1
73.0(0.7)
62.0/0.3)
(0.2)))2)
GPT-GNN
64.5[1.1)
7..0
57.8(3.1)
T0.3(0.5)
62.2(0.1)
57.M4.2)
GROVER 74
70.0(0.1)
82.6(0.7)
81.2(3.0)
74.3(0.1)
65.4(0.4)
64.8(0.6)
0.983/0.090)
2. 2.178(0.052)
0.817(0.008)
GROVER
74
60.5(0.1)
81.0(1.4)
76.2(3.7)
73.1(0.1)
65.3(0.5)
65.4(0
(.801(0.011)
2. 2.272(0.061)
(.823(0.011)
3D-Hufomax
69.1(1.1)
79.9
39.433.)
71.5(0.7)
(1.99
33.37(3
0.89:/0.0223
(.3377
0.73%00.009)
GraphMVP 26
(2.4
81.2
79.1
75.9(0.5)
63.1(0.4)
(1.9(1.2)
L.029/0.633)
0.681(0.010)
MoiCLR 25
72.2(2.1)
82.4(0.9)
91.2(3.5)
75.0(0.2)
18.9(1.4)
(0.41(
2. 2.584(0.249)
(0.0000.004))
Uni-Mal
72.9(0.6)
85.7(0.2)
91.((18)
79.6(0.5)
69.6(0.1)
65.9(1.3)
0.788(0.029)
1.620(0.033)
0.603/0.010)
GEM [4]
72.4(0.4)
85.6(1.1)
90.1(1.3)
78.1(0.1)
69.2(0.4)
67:2(0:4)
0.798/0.08 (29)
1.877(0.084)
0.6GINO DOS)
GEM + InstructMol 73.3(0.8)
85.9(1.3)
82.5(2.1)
79.9.0.6)
70.8;0.4)
67. (
0.761(0.045)
1.604(0.043)
0.582(0 (.00)
ratio of 8:1:1. We report the mean and standard deviation of the results for three random seeds. More experimental details are put in Appendix A.2
Baselines. Several SSL baselines are chosen including consistency regularization, proxy-label methods, and generative models. T-model [29] applies different stochastic transformations (i.e., dropout) to the networks instead of the input graphs. Semi-GAN (34] introduces a discriminator to classify whether the input is labeled or not. Uncertainty-awar pseudo-label selection (UPS) (38 leverages the prediction uncertainty to guide the pseudo-label selection procedure but is merely applicable to classification problems.
Results. Tab. 1 shows that InstructMol significantly improves various ML architectures and outper- forms all SSL baselines. For GIN, GAT, and GCN, it leads to an average increase in ROC-AUC of 8.6%, 7.1%, and 6.6%, respectively, for six classification tasks and an average decrease in RMSE of 9.3%, 8.0%, and 7.7% separately for three regression tasks. These statistics demonstrate our approach effectively boosts existing ML models in low-data circumstances for molecular scaffold property prediction, as most datasets in MoleculeNet have only thousands of labeled samples. Besides that, more up-to-date ML models like GIN enjoy stronger benefits of our InstructMol than primitive ones like GCN. It is also worth mentioning that InstructMol overcomes UPS's drawbacks and can be utilized for regression tasks. All evidence clarifies that InstructMol is more advanced proxy labeling algorithm than the existing SSL mechanisms with stronger virtual screening capacity and broader applications.
5.2 OOD Generalization
Data and Setsups. Measuring OOD generalization is particularly relevant in molecular property prediction, where distributional shifts can be large and difficult to handle for ML models. Different molecular datasets obtained by distinct pharmaceutical companies and research groups often contain compounds from vastly different areas of chemical space that exhibit high structural heterogeneity. Towards this goal, we leverage the Graph Out-of-Distribnn HIV is small-scale dataset for HIV replication inhibition and GOOD-PCBA includes 128 bioassays and forms 128 binary classification tasks. They are divided into a training set, an in-domain (ID) validation set, an ID test set, and OOD test sets by covariate and concept shift splits.
(GOOD) benchmark [77). where GOOD-
The empirical risk minimization (ERM) and several OOD algorithms are considered. IRM (57] searches for data representations that perform well across all environments by penalizing feature distributions with different optimal linear classifiers for each environment. VREx [58] targets both covariate robustness and the invariant prediction. GroupDRO 59 tackles the problem that the
Baselines.
distribution minority lacks sufficient training. DANN (60] adversarially trains the regular classifier and a domain classifier to make features indistigguishable. Deep Coral 61 encourages features in different domains to be similar by minimizing the deviation of covariant matrices from different
7

---

Train Size:
Train Size: 73
Train Sioe: 146
Tran Size 730
Tiain Sioo 1328
KMAS
FRYOL 2.5
PRIDE 4.44
Philetan
-5
-10
20
RYSE 6.25
IM62
-5
-5
5.0
2.5
0.0
0.0
0.
-25
-60
RVE -50
RMS5-004
+50
RVS5
MAIC 0.00
Head
Heal
Vatues
Haal Vilaes
Meal Vaataae
Figure 3: The scatter plot of the distributions of LogP predictions for unlabeled data with and without InstructMol. The first row includes predictions before instructive learning, and the second row includes predictions after instructive learning.
domains. Q-SAVI 62 encodes explicit prior knowledge of the Gata-generating process into a prior distribution over functions. DIR [64] and Mixup 63 two graph-specific OOD methods.
Results. Tab, documents the statistics, where most OOD algorithms have comparable performance with ERM. The risk interpolation methods like GroupDRO and the risk extrapolation mechanisms like VREx perform favorably against others on multiple shift splits. In contrast, InstructMol significantly exceeds ERM and other OOD baselines in all cireumstances. Specifically, InstructMol brings improvements of 2.40% 2.05%, 7.68%, and 4.69% for GOOD-HIV and GOOD-PCBA's different II) splits, and 3.62%, 6.72%, 9.83%, and 7.51% % for GOOD-HIV and ECBA's different OOD splits. It can be seen that gains for OOD are greater than benefits for ID, indicating the promise of pseudo-labeling to tackle OOD generalization.
5.3 SSL with Self-supervised Learning
Setups and Background. Pretraining and SSL are not mutually exclusive but can collaborate for more robust scientific investigations (78). So we design a two-step workflow: (1) In the pretraining stages, unlabeled data is first used in a task-agnostic way, and attain more general molecular representations. Then, those general representations are adapted for a specific task for fine-tuning. (2) In the instructive learning stage, unlabeled data is used again in task-specific way via InstructMol. We combine GEM [4] and InstructMol and examine their joint effectiveness on MoleculeNet.
Baselines. Multiple baselines selected for thorough comparison. D-MPNN 68 MGCN 70
and AttentiveFP (69 are supervised GNN methods. N-gram (71). PretrainGNN 722. InfoGraph 73
GPT-GNN (22), GROVER [74), 3D-Infomax (75), GraphMVP 26). MolCLR 25 and Uni-Mol
76
are pretraining methods. We adopt the same scaffold splitting strategy as GEM and Uni-Mol with three repeated runs.
Results. The overall performance of InstructMol based on GEM and other baseline methods is summarized in Tab. InstructMol achieves new SOTA results all MoleculeNet tasks and brings an average improvement of 4.35%. Notably, its benefits are much stronger for regression tasks with mean decrease of 9.98% in RMSE. Another set of results on MoleculeNet with different splitting method is in Appendix Tab. 5 where InstructMol also outperforms all baselines. This highlights necessity and importance leveraging unlabeled examples to refine and transfer specifc knowledge after pretraining through instructive learning. It also implies that InstructMol is not incompatible with existing pre-training MI. models but can effectively supplement them with an additional instructor model.
8

---

famatior
Berabona
Paratora
terabora
Roall abve
Roal Label
Rool Label
Roxl
Roxl
Label
Pukn Lubul
Fale Lubel
Pule Labul
Lubel
4Kite/stions
txmafore
loradions
Foration
13K ttrstions
Reall abel
Reil Libi
Rnol Lokel
Real Label
Rooi Lawel
Label
Pubc Lubd
Pale
Pake
Lubsl
8.00
0.26
1.00
0.00
0.25
0.75
1.00
0.00
0.00
0.75
1.00 0.00
0.26
0.50
0.75
1.00
0.25
0.00
0.75
1.00
Contidenca Booms
Contidence Scores
Gont dence Scconns
Coriieenc Scores
Corl Contitance Scores
Figure 5: The distributions of confidence scores given by the instructor model during the training process.
5.4 Discussion, Ablation, and Other Applications
Empirical evidence of pseudo-labels' accuracy. Pseudo-labels' accuracy is crucial, as it reflects InstructMol's ability to generalize to unseen molecules. However, no ground-trutt -trth annotations exist for unlabeled data across the mentioned tasks. To address this issue, we adopt values (LogP) as the target. key predictor of sruuglikenesss featured in the famous "Lipinski's 's rule of five." LogP' can quickly screen large libraries of potential therapeutics using computational tools like XLogP. We utilize a public Kaggle dataset of 14.6k molecules with associated LogP values for training and evaluate the predictions on the much larger ZINCI5, whose LogP is computed using RDKit. Furthermore, measurements often number the tens rather than thousands in the low-data regime of drug discovery. To assess InstructMol's limits, we significantly reduce the training size and investigate its efficacy with only 0.1%, 0.5%, 1%. 5%, and 10% of the entire training samples, resulting in scaffold-spli training sets of 14, 73, 146, 730, and 1,326 molecules, respectively.
Fig. compares the LogP prediction distributions before and after instructive learning. can be observed that DL models trained solely on labeled data perform poorly in estimating unseen molecules' LogP with an average RMSE of 5.63. Contrarily, InstructMol enables pretty accurate prediction of LogP with an average RMSE of 0.63 even with very limited number of training examples, verifying the robustness of pseudo-labels.
Ablation Studies. We further investigate the effects of the number of unannotated molecules on the performance of InstructMol. As shown in Figure 4 the enrichment of unlabeled data consistently brings benefits to various downstream tasks.
90.0
88.9
36.3
86.2
86.0
84.4
85
Real-world Drug Discovery. We retrieved the lat- est patent ZA202303678A targeting the 5-HTIA recep- tor to examine IntructMol in real-world applications.
AUC ROC
80-
79.5
78.3
76.2
75.7
76.6
74.2
BBBP
71.8
72.6
BACE
ZA202303678A is published after 2023, and the prop- erty test standards of new compounds in ZA202303678A are consistent with those recorded in the CHEMBL214 Ki dataset 799. All new small molecule drugs are marked as good binders by InstructMol. Since the patent ZA202303678A provides accurate ground truth values of K, through wet experiments, we compare the predicted values with the real ones as shown in Appendix(2 [7] It can be observed that the errors are, at most, within four times, and most predicted results are close to real values. Among them, predicted K; of (S)-5-PPT (Ground truth: K, = 4, Prediction: K; = 3.71), (S)-5-NaTT (Ground truth: K; = 64, Prediction: K;=6i 61.86), (S)-5-CPPT (Ground truth: K; = 0.6, Prediction: K, = 0.93), and (S)-5-FPyT (Ground truth: K; = 1.3, Prediction:K, = 1.29) is almost equal to actual K; obtained from biological experiments.
68.0
ClinTox
55.9
Tox21
65
250K
10M
Unlabeled Data Size
Figure 4: data size on four tasks.
The influence of unlabeled

---

tenatora
2K hural ore
Real
Roal
Rooll Libel
Soall Label
Pake Labu
Lobel
Labal
Pabk Labed
Lubel
82
7K telaions
aK horel
OK tensoor
10K Borstins
0.4
RAal LiM
RANl Lavel
Small mt
Raal aad
Riall Jhhel
Pake
Label
Labul
cabed
Lubel
0.3
02
100
-..0 Predictons
-2.5
-7.5
-5.0
-7.6
-6.0
-10.0
-7.5
-5.0
-2.
-1..0
-7.5
-6.0
-2.
irediction
Pradiction
Predetons
Predekore
Figure 6: The distribution of the predictions of labeled data and unlabeled data generated by the target model during different training stages.
Instructor model's Behavior. The instructor estimates the target model's uncertainty. Therefore, thoroughly evaluating its judgment and contribution is beneficial. Here, we analyze its output value distribution on labeled and unlabeled data in different training stages of activity cliff estimation task (CHEMBL214 (i). The plots in Fig. demonstrate the transition of an instructor model. Notably, since we pretrain the instructor before SSL, it performs well in distinguishing fake labels but remains confused with real ones, From the first iterations to 10K iterations, the instructor gradually gains a stronger capacity to discriminate true labels (confidence score > 1.0) and fake ones (confidence score - 0.0). Taking a step further, we draw the distribution of the target model's s output in Fig. 6 and show a significant overlap between predictions of labeled and unlabeled data. This undoubtedly excludes the hypothesis that the instructor discriminates labeled and unlabeled data simply based on distinct distributions of their predictions. Even though predictions of labeled and unlabeled data are highly similar, the instructor still succeeds in comprehending their uncertainty and guides the target model to leverage pseudo-labels more cautiously. The instructor model's 's interpretability is a byproduct of our InstructMol and can be useful for many real-world biochemical problems. Here we give the relevant appendix figures cited in the manuscript.
6 Conlcusion
This paper presents InstructMol, novel instructive learning framework, to alleviate the difficulty of experimentally obtaining the ground truth properties of molecular data and to overcome the limitation of the small number of labeled biochemical data points. InstructMol sufficiently cutting-edge pretraining methods for molecular representation, and addresses essential real-world problems.
Limitations. Despite the great progress of InstructMol in achieving enhanced molecular learning capacity, there are still minor limitations that require future exploration. For instance, our combination of InstructMol with self-supervised mechanisms is based on existing methodologies such GEM, Uni-Mol, and GROVE. It interesting to develop a more suitable self-supervised learning algorithm that can be better aligned with InstructMol.
Acknowledgments. This work was supported by the National Natural Science Foundation of China (No. (No.62402351). the Hubei Provincial Natural Science Foundation of China (No.2024AFB2275, and the Scientific Research Project of Education Department of Hubei Province (No.Q20231
References
[1] John Keith, Valentin Vassilev-Galindo, Bingqing Cheng, Stefan Chmiela, Michael Gastegger, Klaus-Robert Muller, and Alexandre Tkatchenko. Combining machine learning and compu- tational chemistry for predictive insights into chemical systems. Chemical reviews, 121(16): 9816-9872, 2021.
10

---

[2] Xiangru Tang. Howard Dai, Elizabeth Knight, Fang Wu, Yunyang Li, Tianxiao Li, and Mark Gerstein. A survey of generative ai for de novo drug design: new frontiers in molecule and protein generation. Briefings in Bioinformatics, 2544)bbbae338, 2024.
[3] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. International conference on machine learning, pages 1263-1272. PMLR, 2017.
[4] Fang Wu and Stan Z Li. Diffmd: geometric diffusion model for molecular dynamics simu- lations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 5321-5329, 2023.
[5] Houtim Lai, Longyue Wang, Ruiyuan Qian, Geyan Ye, Juhong Huang, Fandi Wu, Fang Wu, Xiangxiang Zeng. and Wei Liu. Interformer: An interaction-awant model for protein-ligand docking and affinity prediction. 2024.
[6] Pedro Hermosilla and Timo Ropinski. Contrastive representation learning for 3d protein structures. arXiv preprint arXiv:2205.15675. 2022.
[7] Fang Wu, Nicolas Courty, Zhang Qiang, Ziqing Li, et al. Metric learning-enhancee optimal transport for biochemical regression domain adaptation. arXiv preprint arXiv:2202.06200 2022.
[8] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:11100004805 2018.
(9] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for ssion-and-language tasks. Advances in neural information processing systems, 32, 2019.
[10] Siyuan Li, Weiyang Jin, Zedong Wang, Fang Wu, Zicheng Liu, Cheng Tan, and Stan Z. Li. Semireward: A general reward model for semi-supervised learning. arXiv preprint arXiv:2310.03011, 2023.
[11] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Out, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. PNAS, 2019. doi: 10.1101/62280,, URL https: :/wp:..///ww... http://wwwwwwwww.//////////t//tt biorxiv. org/ content/i00 1ttp//000..100/22233304
[12] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang. Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. Prottrans: Toward understanding the language of life through self-supervised learning. IEEE transactions on pattern analysis and machine intelligence, (4(10))7112-7122 2021.
[13] Fang Wu, Shuting Jin, Yinghui Jiang, Xurui Jin, Bowen Tang, Zhangming Niu, Xiangrong Liu, Qiang Zhang, Xiangxiang Zeng, and Stan Z Li. Pre-training of equivariant graph matching networks with conformation flexibility for drug binding, Advanced Science, 9(33):2203796, 2022.
[14] Shengjie Luo, Tianlang Chen, Yixian Xu, Shuxin Zheng, Tie- Yan Liu, Liwei Wang, and Di He. One transformer can understand both 2d & 3d molecular data, arXiv preprint arXiv:2210.01765, 2022.
[15] Ruoxi Sun. Does gnn pretraining help molecular representation? 2XXX::2277.00110 2022.
arXiv preprint
[16] Justin S Smith, Ben Nebgen, Nicholas Lubbers, Olexandr Isayev, and Adrian E Roitherg. Less more: Sampling chemical space with active learning. The Journal of chemical physics, 148 (24):241733, 2018.
[17] Lauri Himanen, Marc OJ Jäger, Eiaki V Morooka, Filippo Federici Canova, Yashasvi S Ranawat, David Z. Gao, Patrick Rinke, and Adam S Foster. Dscribe: Library of descriptors for machine learning in materials science. Computer Physics Communications, 247: 106949, 2020.
11

---

[18] Fang Wu, Dragomir Radev, and Stan Z Li. Molformer: motif-based transformer on 3d heteroge- neous molecular graphs. Rn, 1:1, 2023.
[19] Xuan Lin, Zhe Quan, Zhi-Jie Wang, Tengfei Ma, and Xiangxiang Zeng. Kgnn: Knowledge graph neural network for drug-drug interaction prediction. In IJCAI, volume 380, pages 2745, 2020.
[20] Fang Wu and Stan Li. Surface-vqmae: Vector-quantized masked auto-encoders on molecular surfaces, In International Conference on Machine Learning, pages 53619-53634, PMLR, 2024.
[21] Mengying Sun, Jing Xing, Huijun Wang, Bin Chen, and Jiayu Zhou. Mocl: Data-driven molecular fingerprint via knowledge-aware contrastive learning from molecular graph. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 3585-3594, 2021.
[22] Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai- Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1857-1867, 2020.
[23] Fang Wu and Stan Z Li. A hierarchical training paradigm for antibody structure-sequence co-design. Advances in Neural Information Processing Systems, 36, 2024.
[24] Xiaomin Fang, Lihang Liu, Jicqiong Lei, Donglong Hc, Shanzhuo Zhang, Jingbo Zhou, Fan Wang, Hua Wu, and Haifeng Wang. Geomeety-eenannced molecular representation learning for property prediction. Nature Machine Intelligence, 12)-1177,34, 2022.
[25] Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence, 4(3): 279-287, 2022.
[26] Shengchao Liu, Hanchen Wang. Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre-training molecular graph representation with 3d geometry. arXiv preprint arXiv:21 2021.
[27] Yassine Ouali, Céline Hudelot, and Myriam Tami. An overview of deep semi-supervised learning. arXiv preprint arXiv:2006.05278, 2020.
[28] Olivier Chapelle, Bernhard Scholkopf. and Alexander Zien. Semi-supervised learning (chapelle, et al., eds.: 2006)[book reviews]. IEEE Transactions on Neural Networks, 20(3):542-542, 2009.
[29] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint 1.X.00661. 2016.
[30] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-aweraged con- sistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
[31] David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics, pages 189-196, 1995.
[32] Jing Zhao, Xijiong Xie, Xin Xu, and Shiliang Sun. Multi-view learning overview: Recent progress and new challenges. Information Fusion, 38:43-54 54, 2017.
[33] M Ehsan Abbasnejad, Anthony Dick, and Anton van den Hengel. Infinite variational autoencoder for semi-supervised learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5888-5897, 2017.
[34] Augustus Odena. Semi-supervised learning with generative adversarial networks, arXiv preprint arXiv:1606.01588 01583, 2016.
[35] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608 2020.
12

---

[36] Qizhe Xie, Minh- Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687-10698, 2020.
Minh-Thang
[37] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3. page 896, 2013.
[38] Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty-awarr pseudo-label selection framework for semi-supervised learning. arXiv preprint arXiv:2101.06329, 2021.
[39] Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribttion generalization: A survey. arXiv preprint arXiv:2108. 13624, 2021.
[40] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: critical analysis of out-of-distribution generalization. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 8340-8349, 2021.
[41] Xiao Cai, Feiping Nie, Weidong Cai, and Heng Huang. Heterogeneous image features integra- tion via multi-modal sepe-super learning model. In Proceedings of the IEEE International Conference on Computer Vision, pages 1737-1744, 2013.
[42] Eric Arazo, Diego Ortego, Paul Albert, Noel O' Connor, and Kevin McGuinness. Pseudo- labeling and confirmation bias in deep semi-super learning. In 2020 International Joint Conference on Neural Networks (IJCNN, pages 1-8. IEFE, 2020.
[43] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9():513-5330, 2018
[44] Fang Wu, Qiang Zhang, Dragomir Radev, Jiyu Cui, Wen Zhang, Huabin Xing, Ningyu Zhang, and Huajun Chen. 3d-transformer: Molecular representation with transformer in 3d space. 2021.
[45] Fang Wu. A semi-supervised molecular learning framework for activity cliff estimation.
[46] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek. Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-589 2021.
[47] Renxiao Wang, Xueliang Fang, Yipin Lu, Chao- Yie Yang, and Shaomeng Wang. The pubbind database: methodologies and updates. Journal of medicinal chemistry, 48(12):4111-411, 2005.
[48] Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi-supervised learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 5070-5079, 2019.
[49] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[50] Bianca Zadrozny, John Langford, and Naoki Abe. Cost-sensitive learning by cost-proportionate example weighting. In Third IEEE international conference on data mining, pages 435-443 IEEE, 2003.
[51] Paola Cascante- Rascante-Bonilla, Fuwen Tan, Yanjun Qi, and Vicente Ordonez. Curriculum labeling: Revisiting useudo-labeling for semi-supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6912 6920, 2021.
[52] Huiling Qin, Xianyuan Zhan, Haoran Xu, et Enhancing semi-supervised learning via self-interested coalitional learning.
13

---

[53] Kalyanmoy Deb and Kalyanmoy Deb. Multi -ujeei-beeeetive optimization. In Search methodologies: Introductory nutorials optimization and decision support techniques, pages 403-449. Springer. 2013.
[54] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11))139-14 2020.
[55] Huiling Qin, Xianyuan Zhan, Yuanxun Li, Xiaodu Yang, and Yu Zheng. Network-wide traffic states imputation using self-interested coalitional learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1370-1378, 2021.
[56] Fang Wu, Nicolas Courty, Shuting Jin, and Stan Z Li. Improving molecular representation learning with metric learning-enhanced optimal transport. Patterns, 4(4), 2023.
[57] Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez- Paz. Invariant risk mini- mization. arXiv preprint arXiv: 02899,,, 2019.
[58] David Krueger, Ethan Caballero, Joern- Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. olation (rex). In International Conference on Machine Learning, pages 5815-5826. PMLR, 2021.
generalization via risk extrap-
[59] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprini arXiv: 19189911110111111111111111111111 2019.
[60] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Journal of machine learning research, 17(59):1-35, 2016.
training of neural networks.
[61] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14, pages 443-450. Springer, 2016.
[62] Leo Klarner, Tim GJ Rudner, Michael Reutlinger, Torsten Schindler, Garrett M Morris, Charlotte Deane, and Yee Whye Teh. Drug discovery under covariate shift with domain-informed prior distributions over functions. In International Conference on Machine Learning, pages 17176- 17197. PMLR, 2023.
[63] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 12.009442, 2017.
[64] Ying-Xin Wu, Xiang Wang, An Zhang, Xiangnan He, and Tat-Seng Chua. Discovering invariant rationales for graph neural networks. arXiv preprint arXiv:2201.12872. 2022.
[65] Thomas N Kipf and Max Welling. supervueed classification with graph convolutional networks. arXiv preprint
2016.
[66] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10003, 2017.
[67] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.0082. 2018.
[68] Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman-Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, et Analyzing learned molec- ular representations for property prediction. Journal of chemical information and modeling, 59 (8):3370-3388, 2019.
[69] Zhaoping Xiong, Dingyan Wang, Xiaohong Liu, Feisheng Zhong, Xiaozhe Wan, Xutong Li, Zhaojun Li, Xiaomin Luo, Kaixian Chen, Hualiang Jiang, et al. Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism. Journal of medicinal chemistry, (3(16))8749-876) 2019.
14

---

[70] Chengqiang Lu, Qi Liu, Chao Wang, Zhenya Huang, Peize Lin, and Lixin He. Molecular property prediction: A multilevel quantum interactions modeling perspective. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1052-1060, 2019.
[71] Shengchao Liu, Mehmet F Demirel, and Yingyu Liang. N-gram graph: Simple unsupervised representation for graphs. with applications to molecules. Advances in neural information processing systems, 32, 2019.
[72] Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks, arXiv preprint arXiv: 1905. 12265, 2019.
[73] Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised
graphh graph-level representation learning via mutual information maximization. arXiv preprint arXiv: Tet::/908. /908. 01000, 2019.
[74] Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang. and Junzhou Huang. Self-supervised graph transformer large-scale molecular data. Advances in Neural Information Processing Systems, 33: 12559-12257,, 2020.
[75] Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günnemann, and Pietro Liò. 3d infomax improves gnns for molecular property prediction. In International Conference on Machine Learning, pages
PMLR, 2022.
[76] Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. 2022.
[77] Shurui Gui, Xiner Li, Limei Wang, and Shuiwang Ji. Good: A graph out-of-distribttion benchmark. Advances in Neural Information Processing Systems, 35:2059-2073 2022.
[78] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi semi-superes supervised learners. Advances in neural information processing systems, 33:22243 -22255, 2020.
[79] Derek van Tilborg, Alisa Alenicheva, and Francesca Grisoni. Exposing the limitations of molecular machine learning with activity cliffs. Journal of Chemical Information and Modeling, 62(23))5938-5951, 2022.
[80] Teague Sterling and John J Irwin. Zinc 15-ligand discovery for everyone. Journal of chemical information and modeling, 55(11):2324 00::242422333 2337, 2015.
[81] Bharath Ramsundar, Peter Eastman, Patrick Walters, Vijay Pande, Karl Leswing, and Zhenqin Wu. Deep Learning for the Life Sciences. O' O'Reilly Media, 2019. https:///ww/ amazon. com/ Deep-Learning Life-
[82] Greg Landrum et al. Rdkit: A software suite for cheminformatics, computational chemistry, and predictive modeling. Greg Landrum, 8, 2013.
[83] Anna Gaulton, Louisa J Bellis, A Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, et al. Chembl: a large-scale bioactivity database for drug discovery. Nucleic acids research, 10(0 D1107, 2012.
[84] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14Xiv:1412. 1e... 6980, 2014.
15

---

Appendices
Appendix A
Experimental Setups
A.1
Unlabeled Data
We use the ZINCIS [80] database to collect unlabeled molecular data, which can be downloaded from DeepChem 8T. There are four different data sizes supported by ZINC15: 250K, 1M, 10M, and 270M. For MoleculeNet, we utilize the 1M molecules as the unlabeled dataset. Those unlabeled SMILES are then converted by RDKit [2] into 2D graphs. We expect future work to enrich the unlabeled corpus by leveraging other resources such as ChemBL 83 Chembridge, and Chemdiv.
A.2 Molecular Property Predictions
Dataset Description. We conduct experiments on the MoleculeNet + to examine the effica- ciousness of our algorithm for molecular property prediction. It is widely used benchmark and we include datasets the main text, which described follows:
BBBP. The blood-brain barrier penetration (BBBP) dataset contains binary labels of blood- brain barrier penetration (permeability).
BACE. The BACE dataset provides quantitative (1C50) and qualitative (binary label) binding results for set of inhibitors of human 8-secretase (BACE-1).
ClinTox. The ClinTox dataset compares drugs approved by the FDA and those that have failed clinical trials for toxicity reasons.
Tox21. The "Toxicology in the st Century" (Tox21) initiative created public database, which contains qualitative toxicity measurements on 12 biological targets, including nuclear receptors and stress response pathways.
Toxcast. ToxCast is another data collection (from the same initiative as Tox21) providing toxicology data for large library of compounds based on in vitro high tig-thhoughhutt screening. including experiments on over 600 tasks.
SIDER. The Side Effect Resource (SIDER) is a database of marketed drugs and adverse drug reactions (ADR), grouped into 27 system organ classes.
ESOL. ESOL is a small dataset consisting of water solubility data (log solubility in mols per liter) for common organic small molecules.
FreeSolv. The Free Solvation Database (FreeSolv) provides experimental and calculated hydration-free free energy of small molecules in water.
Lipo. Lipophilicity is an important feature of drug molecules, which affects both membrane permeability and solubility. This dataset provides experimental results of octanol/water distribution coefficient The Free Solvation Database (FreeSolv) provides(logD at pH 7.4).
Data Split. In our experiment, we follow the previous work GEM 24 and Uni-Mol (76 and adopt the scaffold splitting to divide different datasets into training, validation, and test sets with ratio of 80%, 10%, and 10%. It has been widely acknowledged that scaffold splitting more challenging than random splitting because the scaffold sets of molecules in different subsets do not intersect (76). This splitting tests the model's generalization ability and reflects the realistic cases 44) and (76) find that whether or not chirality is considered when generating the scaffold using RDKit has significant impact on the division results. The performance of different methods Table T all follows the same scaffold splitting, where MoICIR is reproduced. In all experiments, we choose the checkpoint with the best validation loss and document the results on the test set run by that checkpoint.
Implementation Details. In our experiments for molecular property prediction, we utilize 4 A100 GPUs and an Adam Optimizer 84 with weight decay of le- for all GNN models, i.e., GCN, GAT, and GIN. A Reducel_ROnnlateer ROnPlateau scheduler is employed to automatically adjust the learning rate with a patience of epochs. Before the SSL stage, we first pretrain the target molecular model via supervised learning for 100 epochs and then pretrain the instructor model for 50 epochs, where an
16

---

early stopping mechanism is utilized with patience of 5 epochs. Similar to GEM [24]. we normalize the property label by subtracting the mean and dividing the standard deviation of the training set. As for the performance of baselines, we copy all available results from GEM 24 Uni-Mol (76 and 3ID-Infomax 75
As for the reproduction of three SSL methods, Semi-GAN is modified from /ttpp://gttiiiiii com/ ppetross//annsspersssseerrrrdddd r--odel transformed from simple Tensorflow-based UPS is directly modified from its official GitHub
version at at http:p/s//gattub....m/m/a/aaeee
Hyperparameter Search Space. Referring to prior studies, adopt grid search to find the best combination of hyperparameters for the molecular property prediction task. To reduce the time cost, set smaller search space for the large datasets (e.g., ToxCast). We report the details of the hyperparameter setup of InstructMol in Table
Table 4: Hyperparameters setup for InstructMol in molecular property prediction.
Hyperparameters Search Space
Symbol Value
Training Setup
Epochs
[100, 200, 300]
Batch size
[32, 64, 128]
Learning rate
[le-4, 5e-5, 1e-6, 5e-6]
Warmup ratio
[0.0, 0.05, 0.1]
The initial update frequency
kp
[5, 10, 20)
Unlabeled data size
[IK, 10K, 100K, 250K]
Balance weight for Uulabeled data and labeled data
[0.01, 0.05, 0.1, 0.2, 0.3, 0.5]
GNN Architecture
Dropout rate
[0.2, 0.4] [2, 3, 4, 6]
Number of GNN layers
The hidden dimension of node representations
[32, 64, 128, 256, 512]
The hidden dimension of edge representations
[64, 128, 256]
Number of heads in GMT
[4, 8 ,112
Hidden dimension in GMT
[64, 128, 256,
Number of fully-connected layers
[1, 2]
A.3 LogP Value Prediction
The Kaggle dataset for LogP prediction is downloaded from https://ww..kagat kag.lw com/ datasets/mattheewe httttttwwttttteee///a cttpicchemcaalcccccc wwwssstuuuuurneeeeeeee aac.:a/-ppppprreooreeeomm....
Appendix B Additional Experiments
B.1
Performance for Random Scaffold Splitting
We additionally execute experiments using the random scaffold splitting on the classification datasets following the same experimental setting used GROVE 77), which is much easier than the standard scaffold splitting used in our main text. As shown in Table our findings notice that GEM blended with InstructMol also achieves stronger results than all baselines. The baseline results are copied from GROVE and GEM.
Appendix C Comparison among Proxy-labeling Algorithms
Here we provide a clear and simple comparison between InstructMol and existing lroxy-labeling approaches in Table
17

---

Table 5: Comparison of performance on the molecular property prediction tasks, where random
scaffold splitting is adopted.
Classification (ROC-AUC Se higher is hetter )
Datasets Tasks
BBBP 2039
BACE 1513 1
ClinTox
Tox21 12
ToxCast 8575
SIDER
Avg.
2
617
27
pretraining
D-MPNN
91.9(3.0)
85.2(5.3)
89.7(4.0)
82.6(2.3)
718(11)
63.2(2.3)
80.7
Attentive
90.8(5.0)
86.3(1.5)
93.3(2.0)
80.7(2.0)
37.9(0.1)
60.5(6.0)
78.3
pretraining
N-Gramxxa
91.2(1.3)
87.6(3.5)
85.5(3.7)
76.9(2.7)
63.2(0.5)
PretruinGNN
91.5(4.0)
85.1(2.7)
(6.2(5.8)
81.1(1.5)
71.4(1.)
61.4(0.6)
GROVERASS
83.6(0.8)
87.8(1.6)
(2.5
81.1(1.5)
72.((1.0)
65.6(0.6)
82.3
GROVER PROVERtaage fange
94.0(1.9)
89.7(2.8)
94.4(2.1)
81.9(2.0)
72.3(1.0
63.8(2.3)
83,4
GEM
95.3(0.7)
92.5(1.0)
97.7(1.9)
83.1(2.5)
73.7(1.0)
66.3(1.4)
GEM InstructMol
95.8(1.4)
93.2(1.6)
97.8(2.0)
83.7(3.0)
74. 74.2(1.3
67.0(1.8)
85.3
Table 6: Comparison over different literature under the proxy-labeling framework.
Method
Unlabeled Sample Selection
Loss Function
Reg.
Vanilla
1 2 ml
1 72]
H, (f (ar))
2
M [c]]
Hy (f (ar)
UPS
(.00)) S (o]
10
2
2m
+1 (x( S Mal 00 s mal
16 (J (177) -.0))
No
InstructMol
All Samples from D
(2p, 1) 9yy (xg).07)
Yes
C.1 Discovery of Drug-like Molecules
should be noted that these new small molecules are highly similar in structures, but IntructMol successfully and confidently differentiates their bioactivity based on their geometries. In addition to that, most experienced medicinal chemists also believe that the two sets of data within 10 times are manipulable and can be recognized accurate prediction results. The error range of InstructMol's results for the prediction of small molecule properties of new compounds further proves the potential of our model in predicting essential properties of activity cliffs molecular and can effectively get aware of subtle changes in the presence of bioactivity cliffs.
18

---

Ki Pre: Ki 25.06
True: Ki 4 Pre: Ki = 3.71
True: Pre: Ki 6.04
(-)-5PPT @NC)C@@/1co2ejose. 22coo03)C1
(S)-5-FPT
(S)-5-CPT
ON(CYCEHI C0220oac2--222o00001111
True: Ki = 24
True: Ki = 64
Ki
Pre: 20.1
Pre: Ki 22.6
(S)-5-PyT CNCICCH) 2ecocaannccceeeooo-----eoe1111
(S)-5-PPT True: CC coc2ejcccz c2cecoc2yC1
5-)-5NNTT
True: Ki = 240
Pre: 0.93
Pre: Ki 1.29
Pre: Ki 64.06
(S)-5-CH) (S)-5-CPPT
(S)-5-FPyT Fetteooef-cttoo22e1C0000 hwt.e//cteeooooccoooooo/oooooooo
(S)-AS-19 http://DDn.etteo0eee000000o0Ooo
Figure 7: The K; value prediction results of the 9 newly discovered small molecules of 5-HTIA receptor.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction have claborated that our study is located in the molecular representation learning field. The main contribution our paper is to propose an instructive learning technique to leverage the massive unlabeled molecular data, especially when the labeled points are searce.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the Appendix, we claim the potential limitation of our work.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: [NA]
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
At the beginning of the Appendix, we have stated clearly the experimental setting, including the datasets used in the main text, the split strategy, and the training details. We also elaborated how we tuned the hyperparameters to achieve the optimal performance.
Justification:
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Notably, all data used in the paper are public datasets, which are easily accessibly from the Internet. Regarding the code for reproducing the results, we are very pleased to release it once our paper is accepted by the conference.
Justification:
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: At the beginning of the Appendix, have stated clearly the experimental setting, including the datasets used in the main text, the split strategy, and the training details. We also elaborated on how we tuned the hyperparameters achieve the optimal performance.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: the experiments, we have repeated the trials multiple times and reported the mean and standard deviations at all tables and relevant figures.
20

---

8. Experiments Compute Resourees
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: As mentioned in the Appendix, all experiments were implemented on A100 GPUs with memory of 80G. Notably, it took around week to complete the training process.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics hcttps///eerrise..cppbbbbieiiiie
Answer: [Yes]
Justification: We have strictly adhered to the NeurIPS Code of Ethics. We promise that we did not reveal any kind of non-anonymous information in the submission.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [NA]
Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: [NA]
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: We have described clearly which version data that we used for experiments and cited them correctly.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether IRB approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
21


Reviews:
[{"review_id": "SruJhGLVOr", "reviewer": "Reviewer_73Xa", "summary": "The authors present \"InstructMol\" which does not require transferring knowledge between multiple domains, which avoids the potential gap between the pretraining and fine-tuning stages. and demonstrate it on real-world molecular datasets and out-of-distribution (OOD) benchmarks.", "questions": "How would Instructive Learning Framework works on  transformer based model trained on SMILES representation?", "limitations": "The paper mostly focus on GNN as backbone but would be worth discuss more about transformer based model trained on SMILES representation.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Instructive Learning Framework helps the model the generalize better for out-of-distribution molecular property prediction task.", "weaknesses": "The paper mostly focus on GNN as backbone but would be worth discuss more about transformer based model trained on SMILES representation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer **73Xa**,\n\nThank you for your detailed feedback on our InstructMol. We appreciate your insights and suggestions, which are invaluable for refining our work. We are pleased to hear that you found our Instructive Learning Framework effective for improving generalization in OOD molecular property prediction tasks. This was a primary goal of our work, and we're glad it was recognized.\n\nYou noted that our paper primarily focuses on GNNs and suggested that it would be beneficial to discuss the application of our framework with transformer-based models trained on SMILES representations. This is an excellent point. While we concentrated on GNNs due to their strong performance in molecular graph representations, we acknowledge that transformers have shown promising results in modeling sequential data like SMILES.\n\nTo address this, we are currently exploring how our InstructMol can be adapted for transformer-based architectures. To be specific, we leverage an open source Transformer-based algorithm -- SMILES Transformer (ST) [A] as the backbone and evaluate the impact of our instructive learning framework on 10 datasets from MoleculeNet, which is also adopted in its original paper but *with a different splitting strategy*. Note that we adopt a 250K unlabeled dataset for SSL. Preliminary results (see Table below) suggest that the framework's principles are generalizable and can indeed enhance the performance of transformer models on SMILES data. We plan to include a discussion of these findings in the final version of the paper, detailing how the framework's principles can be adapted and optimized for different model architectures.\n\n| Dataset | ESOL $\\downarrow$ | FrSlv $\\downarrow$ | Lipo $\\downarrow$ | MUV $\\uparrow$ | HIV $\\uparrow$ | BACE $\\uparrow$ | BBBP $\\uparrow$ | Tox $21 \\uparrow$ | Sider $\\uparrow$ | ClinTox $\\uparrow$ |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| ST | 1.144 | 2.246 | 1.169 | 0.009 | 0.683 | 0.719 | 0. 900 | 0.706 | 0.559 | 0.963 |\n| ST + InstructMol | **1.013** | **2.089** | **1.058**|  **0.025**|  **0.704**|  **0.733**|  **0.908**|  **0.742**|  **0.570**|  **0.971**|  \n---------------------------------------------\n[A] SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discovery. 2019", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ezVOC6swLy", "reviewer": "Reviewer_dbD2", "summary": "The authors develop a method, called InstructMol, for adding pseudo-labels to any training task by including an \"instructor\" that is trained to discriminate real labels from pseudo-labels, and whose uncertainty is used to modulate the training loss for the primary predictors. The authors show that adding this instructor model improves overall performance over similar methods across a number of property prediction tasks. The authors also show that pretraining delivers state of the art results on the MoleculeNet benchmarks.", "questions": "- I can't tell from the paper if the benefit that comes from InstructMol is due simply to training on more data or for more iterations, simply due to having more labeled data. In which of the experiments is this possibility controlled for? Is it Table 1 where other pseudo-label methods are compared to? If such a control exists, will you please make this explicit in the paper? If not, this seems a critical experiment to run.\n- Why not use TDC benchmarks instead of MoleculeNet? While the datasets are obviously similar, TDC has done some additional work to clean them, provide reasonable splits, etc.", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- novel general pseudo-labeling method \n- novel loss for extracting signal from all pseudo-labels, even when the estimated uncertainty is high\n- top results amongst comparative models, especially the results in Table 1", "weaknesses": "- Using GIN, GAT and GCN for the results in Table 1, but then GEM for those in Table 3, makes it feel like the results are cherry-picked, especially since the GEM+InstructMol results in Table 3 are comparable to and within the error of some of the other methods.\n- It is unclear how the results of Figure 4 were obtained. The reader assumes these are comparable to the GEM+InstructMol of Table 3, but this should be better explained.\n- The distance of the 9 molecules examined in the real world drug discovery section to the training set should be examined, even if the fact that these were patentable suggests they are dissimilar from known molecules. Also, there are likely many more examples like this that could have been provided more prospective evaluation, and only showing one makes the reader again wonder if the example is cherry-picked.\n- It would have been useful to see similar plots to those in Appendix C for other uncertainty estimation methods as a way of showing that InstructMol learns better separation between real and pseudo labels.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer **dbD2**,\n\nThank you for your detailed review and thoughtful comments on our paper. We appreciate your recognition of the strengths, particularly the novelty of our pseudo-labeling method, the innovative loss function for utilizing all pseudo-labels, and the strong performance shown in Table 1. We are pleased that you found our pseudo-labeling approach and loss function novel and effective. Your acknowledgment of the competitive results we achieved, especially in Table 1, is encouraging. Below, we address the concerns and questions you raised:\n\n(1) Your question about the potential benefits from training on more data or iterations is important, and it has also been raised by Reviewer UbNR. First, in response to your query regarding Table 1, we indeed compared InstructMol with other pseudo-labeling methods under controlled conditions to isolate the effects of our approach from simply having more labeled data. For example, UPS [A] introduces an uncertainty-aware pseudo-label selection framework, which enhances pseudo-labeling accuracy by significantly reducing noise during the training process.\n\nSecond, by ruling out the benefit of merely having more labeled data, we attribute the success of InstructMol to two key factors. (1) The target model assigns more accurate and reliable pseudo-labels with the assistance of the instructor. (2) The instructor plays a crucial role in evaluating the reliability of these pseudo-labels and influences their contribution to the loss calculation, as detailed in Equation 2.\n\nInstructMol implements a cooperative-yet-competitive learning scheme, which synergistically enhances both the main molecular prediction task and the companion label confidence prediction task. This approach fosters collaboration between the two tasks by exchanging crucial information: the main task contributes prediction loss data, while the companion task provides label confidence measures. This interaction is a significant factor in the observed performance improvements of our method.\n\n**[A]** In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning\n\n\n\nWe will explicitly clarify this in the paper, detailing the control measures we used to ensure a fair comparison. If this aspect has not been fully addressed, we acknowledge the need for additional experiments and will work to include such controls or explicitly state any limitations.\n\n\n(2) We selected MoleculeNet for its extensive range of datasets and well-established benchmarks, making it a widely accepted standard for evaluating molecular property prediction models. Numerous prior studies, including GROVER [A], 3D-Informax [B], GraphMVP [C], MolCLR [D], Uni-Mol [E], GEM [F], and GPT-GNN [G], have utilized MoleculeNet, which facilitates direct comparisons of our results with these methods. Furthermore, we used the same scaffold splitting strategy as GEM and Uni-Mol, which is recognized as a challenging and biologically relevant approach, ensuring a robust evaluation of our model's performance.\n\nBeyond the standard MoleculeNet benchmarks, we also evaluated InstructMol using the **Graph Out-of-Distribution (GOOD) benchmark**, which systematically assesses the generalization capabilities of graph-based models. Additionally, we explored the predictive strength of InstructMol on bioactivity using the CHEMBL214_Ki dataset from the **ACE benchmark** [H], providing further insights into its practical applicability in real-world drug discovery scenarios.\n\nIn summary, our evaluation strategy aims to be both comprehensive and thorough, covering a wide range of datasets and challenging scenarios. However, we acknowledge the value of the Therapeutic Data Commons (TDC) benchmarks, particularly their enhanced data cleaning and standardized splits. We appreciate your suggestion and will consider incorporating TDC benchmarks in future research to further validate and extend our findings. Thank you for your constructive feedback. \n\n**[A]** Self-supervised graph transformer on large-scale molecular data. NeurIPS 2020.\n\n**[B]** 3d infomax improves gnns for molecular property prediction. ICML 2022.\n\n**[C]** Pre-training molecular graph representation with 3d geometry. ICLR 2022. \n\n**[D]** Molecular contrastive learning of representations via graph neural networks. Nature 2022. \n\n**[E]** Uni-mol: A universal 3d molecular representation learning framework. NeurIPS 2022. \n\n**[F]** Geometry-enhanced molecular representation learning for property prediction. Nature 2022.\n\n**[G]** Strategies for pre-training graph neural networks. ICLR 2020.\n\n**[H]** Exposing the limitations of molecular machine learning with activity cliffs. JCIM 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aBMXhTywgg", "reviewer": "Reviewer_nboj", "summary": "This paper targets the problem of label-scarcity in the domain of molecular property prediction. It can be seen as an improved version of proxy labeling. It utilizes a separate model that measures pseudo-labels’ reliability and helps the target model leverage large-scale unlabeled data. This method applies to both classification and regression tasks. The authors run numerous experiments on predicting molecular properties, OOD generalization, and combination with pre-training models.", "questions": "* In Line 201-202, how are the average increase in AUC-ROC and the average decrease in MSE defined and calculated? Why are there only three numbers for the six classification tasks?\n* I assume the architecture of the InstructMol model is not necessarily the same as the target molecular model. The article does not clearly explain how the InstructMol model is constructed.", "limitations": "The authors briefly touched upon the necessity of developing a self-supervised learning algorithm better aligned with InstructMol than existing methodologies. Could the authors briefly elaborate on the extra compute incurred by the instructor model?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The target problem is both timely and important, with a well-justified motivation. The proposed method alleviates some issues by utilizing an instructor model to predict confidence scores.\n\nOverall, the method is clearly presented, and adequate experiments are conducted and documented.", "weaknesses": "The improvement in model performance is relatively weak, with large standard deviations, and the method is compute-demanding due to the separate model and the iterative procedure. Additionally, some technical details and experimental results are unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer **nboj**,\n\nThank you for your thoughtful review and detailed feedback on our paper. We appreciate your recognition of the significance of addressing label scarcity in molecular property prediction and the contributions of our proposed method. We are glad that you found our work timely, important, and well-motivated. Your positive remarks about the clarity of the presentation and the comprehensiveness of our experiments are encouraging. Below, we respond to the key points you raised:\n\n(1) We acknowledge that while our method shows improvements, the performance gains are sometimes accompanied by large standard deviations. **This variability can arise from the inherent complexity of molecular data and the challenges associated with out-of-distribution (OOD) generalization.** \n\n(2) We apologize for any confusion caused by numerical issues and thanks for your detailed question. Here, we report the average increase in ROC-AUC of six classification tasks for **three backbone architectures**, *i.e.*, GIN, GCN, and GAT, instead of the specific improvements for these six tasks. Therefore, there are only three numbers in Lin 201-202. We will revise this section to ensure clarity and provide a complete breakdown of the results for all tasks.  \n\n(3) You are correct that **the architecture of the InstructMol model is not necessarily the same as the target molecular model**. InstructMol consists of two components: the main model for molecular property prediction and a separate instructor model that predicts the reliability of pseudo-labels. *The choice of architecture for each component can vary depending on the specific task and dataset.* In our experimental implementation, for simplicity, we directly copy and adopt the same architecture of the instructor as the target model. We will expand the discussion in the Appendix to explain the construction of the InstructMol model as you wish. \n\n(4) Regarding the development of a self-supervised learning algorithm better aligned with InstructMol, we recognize that this is an area ripe for further research. As for the additional computational cost incurred by the instructor model, it includes not only the training of an additional model but also the iterative process of pseudo-label assignment. Therefore, the entire computational expense would be $(1 + 1/k)$ times ($k$ is the update frequency) more than the conventional SSL methods, which rely on a single target model for molecular property prediction. However, *this computational burden can be reduced and the training speed would be significantly accelerated if we select a lightweight instructor architecture (e.g., shallow layers and a smaller hidden dimension with a much smaller model size).* We will include a more detailed discussion of this computational cost, along with potential optimizations or alternatives that could mitigate it in the Limitation part.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZjJVm4zxrk", "reviewer": "Reviewer_pmns", "summary": "The paper introduces InstructMol, an innovative learning framework designed to address the challenge of data sparsity in chemical and biological sciences by leveraging large-scale unlabeled data through reliable pseudo-labeling. Unlike traditional methods that rely on transferring knowledge between domains, InstructMol operates within a single domain, eliminating potential discrepancies between pretraining and fine-tuning stages. The authors demonstrate the algorithm's high accuracy using various real-world molecular datasets and out-of-distribution (OOD) benchmarks, demonstrating its effectiveness in enhancing machine learning applications in biochemical research.", "questions": "- Due to limited discussion on computational complexity, it difficult to assess the practicality and scalability of the approach for very large datasets or in resource-constrained environments.\n- It was mentioned that determining 'k' in InstructMol is important; please show the experimental results and discussion related to this.\n- It would be beneficial to have an experiment to determine whether using a poor model for model $f$ can still result in performance compensation due to the confidence score, or if it leads to a decline in performance.\n- Why RMSEs in Figure 3 become worse as the training data increases?\n- What model and data were used for training on \"Real-word Drug Discovery\"?", "limitations": "The authors addressed limitations in Appendix.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Well-written and easy to follow\n- InstructMol effectively addresses the data scarcity issue in biochemical data by leveraging large-scale unlabeled data without the need for domain transfer.\n- Extensive experiments are conducted to demonstrate the efficacy of InstructMol.", "weaknesses": "- The paper does not provide a detailed analysis of the computational complexity or resource requirements of the InstructMol algorithm. \n- While the paper showcases the superior performance of InstructMol, it does not sufficiently address potential overfitting issues that may arise due to the iterative use of pseudo-labels.\n- Although the paper compares InstructMol with several baseline methods, it does not include a thorough comparison with some of the latest advancements in semi-supervised learning and domain adaptation techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer **pmns**,\n\nThank you for your detailed review and insightful feedback on our paper, \"InstructMol.\" We appreciate your recognition of the strengths, particularly in addressing data scarcity in biochemical research and the clarity of our presentation. We are glad that you found the paper well-written and that our approach effectively addresses data sparsity by leveraging large-scale unlabeled data. Your positive remarks on the comprehensiveness of our experiments are encouraging.\n\n(1) We recognize the importance of addressing the practical scalability of InstructMol. The computational cost of the algorithm primarily involves the instructor model and the iterative pseudo-labeling process. Since pseudo-labeling is applied every $k$ epoch, the theoretical complexity of InstructMol is approximately $(1 + 1/k)$ times that of a standard semi-supervised learning (SSL) approach, which trains only the target model. However, in practical implementation, we have observed that InstructMol converges significantly faster than existing SSL methods. For example, in the classification tasks reported in Table 1, the UPS [A] method typically requires around 100 epochs for convergence, whereas InstructMol completes training in approximately 10-20 epochs. This accelerated convergence offers a clearer perspective on the practical implications of using InstructMol on large-scale datasets or in environments with limited computational resources.\n\n**[A]** In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning. ICLR 2021.\n\n(2) The parameter $k$ plays a crucial role in the InstructMol framework. We agree with your point that it is necessary to present the experimental results that explore the impact of different values of 'k' on the model's performance. The table below shows the influence of different $k$ strategies, where **na** indicates a loss explosion without convergence and the number in the bracket corresponds to the number of epochs for convergence. It can be found that a too frequent update would make the training procedure volatile, resulting in training failure, while a large $k$ would significantly increase the training complexity. In contrast, our adaptive decay strategy (ADS) achieves a competitive performance while maintaining a fast training speed. \n\n| k | BBBP  | BACE | ClinTox |\n|---|---|---| ---|\n| 1  | na (--) | 77.9 (9)  | na | \n| 10  | 70.7 (37) | 83.1 (25)  | 86.5 (42)| \n| ADS | 70.5 (13) | 83.3 (11) | 86.2 (18) | \n\n(3) The concern about using a suboptimal model and its impact on performance is valid. This is also one of the major reasons that we conduct experiments in Table 1. To be specific, all backbone architectures, containing GIN [A]/GCN [B]/GAT [C], are simple and \"old-school\" graph-based neural architectures, which were invented many years ago. Their performance is also far worse than state-of-the-art baselines such as Graphormer [D] and GPS [E]. However, our experimental results demonstrate that even when the base model is not optimal,  the confidence scores generated by the instructor model can also positively affect the final performance. This analysis helps in understanding the robustness of InstructMol to variations in model quality. \n\n**[A]** How Powerful are Graph Neural Networks? ICLR 2018. \n\n**[B]** Graph Attention Networks. NeurIPS 2018. \n\n**[C]** Semi-Supervised Classification with Graph Convolutional Networks. ICLR 2017.\n\n**[D]** Do Transformers Really Perform Bad for Graph Representation? NeurIPS 2021.\n\n**[E]** Recipe for a general, powerful, scalable graph transformer. NeurIPS 2022.\n\n\n(4) The observation that RMSEs worsen as training data increases may suggest issues such as noise in the additional data or potential overfitting. Importantly, adding more training data does not automatically enhance a model's generalization to the test set. If the distribution of the additional training samples significantly differs from that of the test domain, this can lead to an out-of-distribution (OOD) transfer problem, resulting in poorer performance on the test set. However, Figure 3 demonstrates that our InstructMol framework can mitigate the distributional gap introduced by new data, thereby enhancing the model's robustness.\n\n\n(5) Thanks for your question about the real-world drug discovery application. We employ the CHEMBL214_Ki dataset from the ACE benchmark [A] and GEM [B] as the base architecture, which shows extraordinary results in molecular property prediction. \n\n**[A]** Exposing the limitations of molecular machine learning with activity cliffs. JCIM 2022. \n\n**[B]** Geometry-enhanced molecular representation learning for property prediction. Nature 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sT4AviRUcf", "reviewer": "Reviewer_UbNR", "summary": "This paper proposes a framework InstructMol for utilizing unlabeled data to help molecular property prediction on out-of-distribution (OOD) domains. The framework combines (1) a molecular model *f* that predicts (pseudo)labels with (2) a binary classifier *g* as instructor that evaluates the probability of labels being pseudo and reweighs *f*’s loss. In the experiments, it is compared to several self-supervised learning (SSL) baselines in predictive accuracy and OOD generalization. The experiments also investigate the effects of unlabeled data size and the instructor model’s behavior as an uncertainty estimator.", "questions": "- In Line 33, domain knowledge being biased is raised as a problem. (1) Could some “bias” be helpful, e.g., by providing correct inductive bias to the model? (2) I suggest citing relevant literature on the scenarios where bias becomes a problem.\n- Line 78, “usually follow significantly different distributions” seems exaggerating. Is there justification?\n- Line 97, could uncertainty quantification methods, e.g., Bayesian NN, SNGP, evidential DL, address the problem for regression tasks?\n- Section titles of 5.3 and 5.4 (Visualization) are confusing.\n- The meaning of “confidence score” is unclear: in the main text (Lines 156, 301), 1 indicates pseudo-label, while Fig. 5 seems the opposite.", "limitations": "Discussed in Appendix.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Utilizing large, unlabeled, and potentially distribution-shifted data is relevant in chemical and materials sciences, and this paper presents an effective method for that.\n- This paper is well-written, the figures are informative, and the experiments are comprehensive.", "weaknesses": "- Sec. 2’s review on related works is comprehensive but could be better organized. \n  - The separation of pretraining and SSL is confusing: techniques like contrastive learning are often viewed as SSL, and some SSL algorithms work as pretraining.\n  - The mention of methods “jointly learning multiple tasks” (Line 171) seems more appropriate in related works than method.\n- The paper presents strong empirical results but not enough interpretations. Compared to SSL, what extra information does InstructMol utilize, or why does it extract information more effectively, that leads to better accuracy/generalizability? Discussing these could provide more insights to future model development.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer **UbNR**,\n\nThank you for your comprehensive and insightful review of our paper. We appreciate your recognition of the strengths and contributions of our work, as well as your constructive feedback on areas for improvement. We are pleased that you found our framework effective in utilizing large, unlabeled, and potentially distribution-shifted data, which is indeed crucial in the fields of chemical and materials sciences. Your positive feedback on the clarity of our writing, the informativeness of the figures, and the comprehensiveness of our experiments is greatly encouraging.\n\n(1) We acknowledge that the review in Section 2 could be better organized. We will restructure this section to more clearly differentiate between different categories of related works, including a more distinct separation of pretraining and SSL. We will also ensure that our discussions on techniques like **contrastive learning** are appropriately classified and explained within the broader context of SSL. Besides, we will move the discussion on **jointly learning multiple tasks** in Line 171 to the related work section according to your suggestion. \n\n(2) We understand the need for a deeper discussion on why InstructMol performs better compared to existing SSL methods, particularly in terms of the extra information it utilizes or extracts. From our humble point of view, InstructMol introduces a **cooperative-yet-competitive learning** scheme to jointly boost the performance of both the main molecular prediction task and the companion label confidence prediction task. Specifically, InstructMol forges collaboration between both tasks by providing extra information to each other, *i.e.*, the main task provides prediction loss while the companion task provides label confidence measures. This is considered the major factor for the performance improvements. \n\n(3) Regarding the point on bias, we agree that some correct inductive biases can indeed provide beneficial inductive biases to the model. For instance, [A, B] construct chemical element knowledge graphs to summarize microscopic associations between elements and facilitate contrastive learning efficiency. Nevertheless, bias can lead to issues sometimes. [C] mentions that the regular distance threshold (*e.g.*,4-5 Angstrom ) to determine the chemical bond between atoms and build the graph connectivity can result in suboptimal performance. We will elaborate on this aspect, providing citations to relevant literature and clarifying the context in which we consider bias problematic.\n\n**[A]** Knowledge graph-enhanced molecular contrastive learning with functional prompt. Nature Machine Intelligence 2023. \n\n**[B]** Molecular contrastive learning with chemical element knowledge graph. AAAI 2022. \n\n**[C]** Discovering and Explaining the Representation Bottleneck of Graph Neural Networks from Multi-order Interactions. IEEE TKDE 2024. \n\n(4) The sensitivity of SSL approaches to distribution shifts between the labeled and unlabeled data has long been a crucial topic. Several prior studies [A,B,C] have demonstrated that SSL models suffer when the distribution of the unlabeled data differs significantly from the labeled data. [D] notice that the performance of SSL methods can degrade under distribution shifts and propose a method (FixMatch) to mitigate this issue by using strong data augmentation techniques. However, we value your comment and will tone down the language used (*usually* to *sometimes*) and provide a more nuanced discussion, supported by examples or references, to justify the statement about different distributions in molecular data.\n\n**[A]** Realistic Evaluation of Deep Semi-Supervised Learning Algorithms. NeurIPS 2018.\n\n**[B]** Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty. NeurIPS 2019.\n\n**[C]** Unsupervised Representation Learning by Predicting Image Rotations. ICLR 2018.\n\n**[D]** FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence. NeurIPS, 2020.\n\n\n(5) Your suggestion to explore uncertainty quantification methods like Bayesian Neural Networks (BNNs), SNGP, and evidential DL is absolutely valuable. For instance, BNNs [A] provide a probabilistic approach to neural networks by estimating the distribution over the network's weights rather than relying on a single set of weights. This probabilistic framework allows BNNs to quantify uncertainty in predictions, which is crucial for estimating the confidence of a regression model. Despite the great potential of BNNs in quantifying the uncertainty, they usually show inferior performance compared to DL methods. We look forward to future efforts in exploring BNNs and other promising mechanisms to take advantage of the abundant unlabeled molecular database. \n\n**[A]** Uncertainty quantification of molecular property prediction with Bayesian neural networks. arXiv 2019. \n\n(6) we have revised the titles of Sections 5.3 and 5.4 (**Discussion, Ablation, and Other Applications**) to more accurately reflect their content and avoid confusion. Additionally, we state that 0 and 1 indicate pseudo-labels and ground truths, respectively, and acknowledge a typo in Line 301. The correct version would be *...to discriminate true labels (confidence score $\\rightarrow$ 1.0) and fake ones (confidence score $\\rightarrow$ 0.0)*. As for Line 156, we have double-checked our analysis and corrected it to \"...forces the target model $f$ to disregard this label, which is actually reliable\". Thanks for your advice on terminology and section titles!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents an instructive learning framework that uses large-scale unlabeled data to enhance molecular property prediction. Unlike traditional methods requiring knowledge transfer between domains, InstructMol operates within a single domain, avoiding discrepancies between pretraining and fine-tuning stages. The framework incorporates a pseudo-labeling mechanism supported by an instructor model that estimates the reliability of these labels. The authors demonstrate InstructMol on molecular datasets and OOD benchmarks. 

The reviewers provided a balanced evaluation of the paper, highlighting several strengths and weaknesses:

**Positive Points:**
* The InstructMol framework is praised for its approach to leveraging unlabeled data for molecular property prediction, particularly its ability to handle OOD scenarios effectively.
* The method addresses a critical problem in biochemical research, namely data scarcity, and demonstrates its applicability through extensive benchmarking, achieving competitive results.

**Negative Points:**
* Several reviewers noted the need for a more detailed discussion on the computational complexity and resource requirements of the InstructMol framework, particularly its scalability to large datasets.
* Concerns were raised about potential overfitting issues due to the iterative use of pseudo-labels and method's robustness to noise introduced by additional training data.
* There was some critique regarding the clarity and organization of specific sections, such as the related work, and the need for more detailed analysis of certain experimental results and their implications.

After considering the reviewers' feedback and the authors' rebuttals, I recommend accepting the paper at NeurIPS. While the reviewers identified areas for improvement, the authors' rebuttals were thorough and addressed many of the concerns. The weaknesses, such as computational complexity and potential overfitting, are acknowledged but do not detract significantly from the overall impact and novelty of the work.
Decision: Accept (poster)