Instructor-inspired Machine Learning for Robust Molecular Property Prediction

Authors: fang  wu
shuting  jin
siyuan  li
stan z li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-24 04:51:45

Abstract: Machine learning catalyzes a revolution in chemical and biological science. However, its efficacy is heavily dependent on the availability of labeled data, and annotating biochemical data is extremely laborious. To surmount this data sparsity challenge, we present an instructive learning algorithm named InstructMol to measure pseudo-labels' reliability and help the target model leverage large-scale unlabeled data. InstructMol does not require transferring knowledge between multiple domains, which avoids the potential gap between the pretraining and fine-tuning stages. We demonstrated the high accuracy of InstructMol on several real-world molecular datasets and out-of-distribution (OOD) benchmarks.

Main paper: 
 introduction: 
 An enduring obstacle in applied chemical and biological sciences is identifying and making chemical compounds or materials that have optimal properties for a given purpose [1, 2] . However, the vast majority of progress in these areas is still achieved primarily through time-consuming and costly trial-and-error experiments. Machine learning (ML) has undergone an unparalleled technological advancement, opening up a myriad of applications across various domains. Its potential is particularly notable in expediting the discovery and development of novel materials, pharmaceuticals, and chemical processes [3] [4] [5] . However, ML models' efficacy heavily relies on the availability of labeled data and the consistency of prediction targets. Meanwhile, the cost of generating new data labels through wet experiments is prohibitively high. Consequently, the size of labeled data in this field is several orders of magnitude lower than the one that can inspire breakthroughs in other ML fields [6] . This data scarcity severely hampers ML in addressing scientific challenges within this realm, impeding its ability to generalize to new molecules [7] .Biochemists have noticed this problem and propose several strategies to overcome the low data limitation (view Fig. 1 ). Firstly, inspired by the remarkable success of self-supervised learning from NLP [8] and CV [9, 10] , researchers apply the pretrain and finetune paradigm [11] [12] [13] [14] to molecule modeling. They boost the performance of various molecular models by pretraining them on massive unlabeled data. However, this benefit can be negligible when a large gap exists between the sample distributions of pretraining and downstream tasks [15] . An alternative option is to use active learning [16] , which iteratively selects new data points to annotate according to the current model's predictions. However, auxiliary labor is still required to enrich the original database. Thirdly, domain knowledge is incorporated to enhance molecular representations, such as providing more high-quality hand-crafted features [17] , constructing motif-based hierarchical molecular graphs [18] , and leveraging knowledge graphs [19] . However, domain knowledge can be biased and is difficult to integrate into different training techniques universally. In this study, we develop InstructMol, a flexible semi-supervised learning (SSL) approach to excavate the abundant unlabeled biochemical data for robust molecular property prediction. It differs from prior studies in two aspects: (1) it utilizes an additional instructor model to predict the confidence of the predicted label, measure its reliability, and generate pseudo-label information for unannotated data; (2) with the help of pseudo-label information guiding the model, the target molecular model can reliably utilize unlabeled data without the need to transfer knowledge between multiple domains, which perfectly avoids the potential gap between pre-training and fine-tuning stages. We demonstrate that InstructMol outpasses existing SSL approaches and achieves state-of-the-art performance in MoleculeNet and several OOD benchmark datasets. Besides, via InstructMol, we accurately predict the properties of all 9 newly discovered drug molecules in the latest patent (ZA202303678A). Extensive experiments showcase the effectiveness of our model in surmounting the challenge of data scarcity, propelling advancements in the chemistry and biology domains. within the neural network (NNs) such as dropout) or data augmentation transformations should not modify model predictions given the same input and impose an auxiliary loss. This line of research includes π-model [29] , temporal ensembling [29] , and mean teachers [30] The proxy-label methods regard proxy labels on unlabeled data as targets and consist of two groups: self-training [31] , where the model itself produces the proxy labels, and multi-view learning [32] , where proxy labels are produced by models trained on different views of the data. The generative models rely on VAE [33] and GAN [34] to capture the joint distribution more accurately. 
preliminaries and background: 
 Task formulation. Suppose we have access to some labeled molecular data D = {(x i , y i )} N i=1 . x i can be in any kind of formats such as 1D sequences, 2D graphs, and 3D structures. y i can be any discrete (e.g., toxicity, and drug reaction) or continuous properties (e.g., water solubility, and free energy). Here we consider continuous property for simple illustration, but our approach can be easily extended to binary-class or multi-label circumstances. The target molecular model f : X → Y can be any category of architectures, including Transformers, GNNs, and geometric NNs. There are also some unseen data D test to evaluate the performance of the learned model. Traditionally, D is divided into the train and validation sets asD train = x train i , y train i N1 i=1 and D val = x val i , y val i N2 i=1. Besides, we can obtain some unlabeled data points, denoted asD ⋆ = {x ⋆ i } M i=1, where the number of unlabeled data M is usually orders of magnitude larger than that of labeled data N .Challenge of proxy-labeling. Labeled data D and unlabeled data D ⋆ sometimes follow significantly different data distributions, i.e., P(x i , y i ) ̸ = P(x ⋆ i , y ⋆ i ). However, many SSL algorithms [29, 35, 36] assume no distributional shift between D and D ⋆ . They are likely only to reinforce the consistent information in the labeled data D to unlabeled examples D ⋆ instead of mining auxiliary information from D ⋆ , without exception for proxy labeling [37] . More importantly, despite the versatility and modality-agnostic of proxy labeling, it achieves relatively poor performance compared to recent SSL approaches [38] . This arises because some pseudo-labels ŷ⋆ i can be severely incorrect during training due to the poor generalization ability of classic DL models [39, 40] . If we directly utilize pseudolabels from a previously learned model for subsequent training, the conformance-biased information in the proceeding epochs could increase confidence in erroneous predictions and eventually lead to a vicious circle of error accumulation [41, 42] . The situation can be even worse when labeled data D contain noises because of unavoidable experimental errors. Accordingly, it becomes essential to grasp the quality and dependability of pseudo-annotations and intelligently select a subset of them to reduce the hidden noise.Motivation of InstructMol. Though confidence is crucial for pseudo-label selection and this selection reduces pseudo-label error rates, NNs' poor calibration renders this solution insufficient. Explicitly, incorrect predictions in poorly calibrated NNs can also have high confidence (i.e., ŷ⋆ i → 0 or ŷ⋆ i → 1) [38] . More importantly, prior studies such as UPS [38] resort to the target model's output ŷi as the confidence indicator and produce hard labels byy ⋆ i = 1 [ŷ ⋆ i ≥ γ 1 ] or y ⋆ i = 1 [ŷ ⋆ i ≤ γ 2 ],where γ 1 and γ 2 are pre-defined thresholds. Nonetheless, this selection mechanism becomes inapplicable if Y is a continuous label space, as networks no longer output class probabilities. For regression tasks, ŷ⋆ i discloses no confidential information, and numerous biochemical problems are regression-based, including molecular property prediction [43] [44] [45] , 3D structure prediction [46] , and binding affinity prediction [47] . This brings a challenge for probabilistic output-based proxy-labeling algorithms [38] . Thus, instead of depending on ŷ⋆ i to judge proxy labels' reliability, we accompany the target molecular model f with an additional instructor model g, which plays the role of a critic and predicts label observability, i.e., whether the label is true or fake. g disentangles the confidence prediction and the property prediction, reducing the noise introduced by the pseudo-labeling process. 
method: 
 The Overall Instructive Learning Framework. We separate the integral workflow of InstructMol into two phases. In the first step, we retain pseudo-labels{ŷ ⋆ i } M i=1 = f {x ⋆ i } M i=1. There are several approaches to creating proxy labels, such as label propagation via neighborhood graphs [48] . Here, we require the molecular model f to directly annotate samples in the unlabeled dataset D ⋆ . Then, in the following step, we construct a new dataset with both labeled and pseudo-labeled samples asD ′ = D ∪ {(x ⋆ i , ŷ⋆ i )} M i=1and proceed training both the target molecular model f and the instructor model g based on this new set. These two procedures are iteratively repeated until f reaches the optimal performance on the validation set D val .To be specific, the instructor model g : (X × Y × H f ) → P forecasts the confidence measure p i (0 ≤ p ≤ 1) of whether the given label y ′ i belongs to the ground-truth label set {y i } N i=1 or the pseudo-label set {y ⋆ i } M i=1 . It digests three items: the data sample with its label (x ′ i , y ′ i ) ∈ D ′ and an additional loss termH f (f (x ′ i ), y ′ i ), where H f is traditionally selected as a root-mean-squared-error (RMSE) loss or a mean-absolute-error (MAE) loss for regression tasks and cross-entropy (CE) loss for classification problems. Here we regard H f (.) as the ingredient of g's input to provide more information about the main molecular property prediction task. At last, the instructor model g is supervised via a binary CE loss (BCE) as:EQUATIONwhere c i ∈ [0, 1] is an integer and represents the observability mask. It indicates whether y i is pseudo-labeled (c i = 0) or not (c i = 1). However, since the number of unlabeled data M is much larger than the number of labeled data N , it is proper to shift the loss function from BCE to a focal loss [49] (FL) for unbalanced classes as:L g D ′ , {ŷ ′ i } N +M i=1 = (x ′ i ,y ′ i )∈D ′ FL(p ′ i , c i ) = (x ′ i ,y ′ i )∈D ′ -(1 -p ′ i ) γ log(p ′ i ), where γ ≥ 0 is a tunable focusing parameter.Meanwhile, the target molecular model f receives the discriminative information {p i } N +M i=1 from the instructor model g and uses it to reweight the importance of different samples in backpropagating its gradient. In other words, the instructor model g guides the target model f to deliver different attention to different labels so that correct labels are attached more importance while erroneous labels are ignored. This can be realized by specially designing the loss of the target model f as:EQUATIONwhere 0 ≤ α ≤ 1 is a hyper-parameter to balance the dominance of labeled and unlabeled data sets. L f (.) transforms the original main task into a cost-sensitive learning problem [50] by imposing With no gradient: 6:ŷ⋆ i ← f (x ⋆ i ), ∀x ⋆ i ∈ D ⋆ ▷ Iteratively assign pseudo-labels every k epochs 7:end if 8:D ′ ← D ∪ {(x ⋆ i , ŷ⋆ i )} M i=1▷ Construct the hybrid database 9:ŷ′ i ← f (x ′ i ) ∀x ′ i ∈ D ′ 10: p i ← g(x ′ i , y ′ i , H f (.)), ∀(x ′ i , y ′ i ) ∈ D ′ ▷ Predict the confidence scores 11: L g D ′ , {ŷ ′ i } N +M i=1← Equation 112:L f D ′ , {p i } N +M i=1← Equation 213:Update the parameters of f and g based on L g (.) and L f (.) 14: end for a group of soft-labeling weights based on the predicted confidence of data labels. That is, for pseudo-labeled samples (i.e., x ′ j ∈ D ⋆ ), the soft-labeling weight becomes 2p j -1. This loss format in Equation 2 induces different behaviors on the loss H f (.) of labeled and pseudolabeled instances, where the judgment {p i } N +M i=1 produced by the instructor model g is leveraged to differentiate their informativeness. Notably, if the instructor model g regards a pseudo-label ŷ⋆ j to be unreliable (i.e., 0.5 > p j > 0), the loss L f (.) chooses to enlarge the gap with the pseudo-label ŷ⋆ j . Meanwhile, once the instructor g fully trusts the pseudo-label ŷ⋆ j , InstructMol forces the target molecular model f to give more effort to inferring this proxy-labeled sample. Generally, the more likely a pseudo-label ŷ⋆ j is considered reliable by the instructor model g (i.e., p j → 1), the stronger it drives the target molecular model f to make further improvement on inferring this label ŷ⋆ j . Otherwise, InstructMol will push f to overturn its previous belief if p j → 0. Noticeably, we can also impose a soft-labeling weight for samples with true labels (i.e., x ′ i ∈ D). For instance, a weight factor of α ≥ 1 can navigate f to concentrate more on samples that are more trusted by g. But we practically discover no significant refinement with this design on labeled data and leave it for future exploration. The whole pseudo-code of InstructMol is depicted in Algorithm 1.Loss Selection for InstructMol. We compare some relevant methods from the literature under the proxy-labeling SSL algorithms in Appendix Tab. 6, containing the vanilla proxy-labeling (PL), curriculum learning for PL (CPL) [51] , UPS, and self-interested coalitional learning (SCL) [52] . It is structured in three main columns that describe the selection for unlabeled samples, loss function, and fitness for regression problems. Except for SCL and InstructMol, all approaches adopt a subset of unannotated instances for training rather than utilizing the entire unlabeled datasets. SCL can take advantage of all data points from D ⋆ , but its main limitation is its improper or even severely wrong loss function design. As the confidence score is close to 1, SCL assigns a strong negative multiplier factor as 1 -α 1-pj → -∞ and forces the target model f to disregard this label, which is actually reliable. On the contrary, when the instructor model g doubts the reliability of ŷ⋆ j , the ratio 1 -α 1-pj becomes 1 -α, driving the target model f to move towards it.Guidelines for InstructMol. Before executing InstructMol, it is natural to first obtain a well-trained molecular target model f 0 through regular supervised learning on D and then initialize an instructor g 0 by discriminating pseudo-labels that are generated by f 0 . This is empirically proven to achieve higher training stability and robustness. Moreover, pseudo-labels are assigned every k epoch and a proper setting of k is critical to the success of InstructMol. If pseudo-labels are updated too frequently, the training procedure will be volatile at the very beginning. While a too-large k would significantly increase the training complexity. Here we adopt an adaptive decay strategy: with an initial value k 0 , the update frequency decreases by a factor of 0.5 until it reaches the minimum threshold k min = 3.Analysis of InstructMol. After the curation of D ′ , there are two distinct learning tasks during the second stage of InstructMol. Specifically, the target model follows the regular routine to predict the molecular properties. Meanwhile, the instructor model strives to differentiate whether the label is real. Notably, some prior works embody a similar idea of jointly learning multiple tasks. For instance, the multi-objective optimization (MOO) methods [53] exploit the shared information and the underlying commonalities between two tasks and solve the problem by minimizing an augmented loss. In addition, GAN [54] makes a generator and a discriminator constantly compete against each other. Nevertheless, MOO cannot handle possible contradictions among different tasks in certain settings, where jointly minimizing the augmented loss may impede both tasks from attaining the global optimal [55] . While GAN has been praised for generating high-quality data, it is notoriously difficult to train and requires a large amount of training data. More essentially, making predictions in an adversarial manner for unlabeled data deviates from our primary goal of pseudo-labeling but is a mature technology for domain adaptation [56] . 
experiments: 
 We carry out a wide scope of experiments in all contexts. Section 5.1 shows the benefits of InstructMol in predicting molecular properties compared with various SSL learning algorithms. Section 5.2 verifies the superiority of InstructMol in lowering the predictive error over existing OOD generalization algorithms. Section 5.3 investigates the possibility of marring InstructMol with cutting-edge pretraining methods and demonstrates its potency by setting a state-of-the-art performance on MoleculeNet. Section 5.4 analyzes pseudo-labels' accuracy, the instructor's behavior, and ablation studies.Data and Setups. We first investigate the effectiveness of InstructMol on three sorts of backbone including GCN [65] , GAT [66] , and GIN [67] , and report their performance on the standard Molecu-leNet [43] . Datasets are divided using scaffold splitting into training, validation, and test sets with a ratio of 8:1:1. We report the mean and standard deviation of the results for three random seeds. More experimental details are put in Appendix A.2.Baselines. Several SSL baselines are chosen including consistency regularization, proxy-label methods, and generative models. π-model [29] applies different stochastic transformations (i.e., dropout) to the networks instead of the input graphs. Semi-GAN [34] introduces a discriminator to classify whether the input is labeled or not. Uncertainty-aware pseudo-label selection (UPS) [38] leverages the prediction uncertainty to guide the pseudo-label selection procedure but is merely applicable to classification problems.Tab. 1 shows that InstructMol significantly improves various ML architectures and outperforms all SSL baselines. For GIN, GAT, and GCN, it leads to an average increase in ROC-AUC of 8.6%, 7.1%, and 6.6%, respectively, for six classification tasks and an average decrease in RMSE of 9.3%, 8.0%, and 7.7% separately for three regression tasks. These statistics demonstrate our approach effectively boosts existing ML models in low-data circumstances for molecular scaffold property prediction, as most datasets in MoleculeNet have only thousands of labeled samples. Besides that, more up-to-date ML models like GIN enjoy stronger benefits of our InstructMol than primitive ones like GCN. It is also worth mentioning that InstructMol overcomes UPS's drawbacks and can be utilized for regression tasks. All evidence clarifies that InstructMol is a more advanced proxy labeling algorithm than the existing SSL mechanisms with stronger virtual screening capacity and broader applications.Data and Setsups. Measuring OOD generalization is particularly relevant in molecular property prediction, where distributional shifts can be large and difficult to handle for ML models. Different molecular datasets obtained by distinct pharmaceutical companies and research groups often contain compounds from vastly different areas of chemical space that exhibit high structural heterogeneity. Towards this goal, we leverage the Graph Out-of-Distributio (GOOD) benchmark [77] , where GOOD-HIV is a small-scale dataset for HIV replication inhibition and GOOD-PCBA includes 128 bioassays and forms 128 binary classification tasks. They are divided into a training set, an in-domain (ID) validation set, an ID test set, and OOD test sets by covariate and concept shift splits.Baselines. The empirical risk minimization (ERM) and several OOD algorithms are considered. IRM [57] searches for data representations that perform well across all environments by penalizing feature distributions with different optimal linear classifiers for each environment. VREx [58] targets both covariate robustness and the invariant prediction. GroupDRO [59] tackles the problem that the distribution minority lacks sufficient training. DANN [60] adversarially trains the regular classifier and a domain classifier to make features indistinguishable. Deep Coral [61] encourages features in different domains to be similar by minimizing the deviation of covariant matrices from different domains. Q-SAVI [62] encodes explicit prior knowledge of the data-generating process into a prior distribution over functions. DIR [64] and Mixup [63] are two graph-specific OOD methods.Results. Tab. 2 documents the statistics, where most OOD algorithms have comparable performance with ERM. The risk interpolation methods like GroupDRO and the risk extrapolation mechanisms like VREx perform favorably against others on multiple shift splits. In contrast, InstructMol significantly exceeds ERM and other OOD baselines in all circumstances. Specifically, InstructMol brings improvements of 2.40%, 2.05%, 7.68%, and 4.69% for GOOD-HIV and GOOD-PCBA's different ID splits, and 3.62%, 6.72%, 9.83%, and 7.51% for GOOD-HIV and GOOD-PCBA's different OOD splits. It can be seen that gains for OOD are greater than benefits for ID, indicating the promise of pseudo-labeling to tackle OOD generalization.Setups and Background. Pretraining and SSL are not mutually exclusive but can collaborate for more robust scientific investigations [78] . So we design a two-step workflow: (1) In the pretraining stages, unlabeled data is first used in a task-agnostic way, and we attain more general molecular representations. Then, those general representations are adapted for a specific task for fine-tuning. (2) In the instructive learning stage, unlabeled data is used again in a task-specific way via InstructMol. We combine GEM [24] and InstructMol and examine their joint effectiveness on MoleculeNet.Baselines. Multiple baselines are selected for a thorough comparison. D-MPNN [68] , MGCN [70] and AttentiveFP [69] are supervised GNN methods. N-gram [71] , PretrainGNN [72] , InfoGraph [73] , GPT-GNN [22] , GROVER [74] , 3D-Infomax [75] , GraphMVP [26] , MolCLR [25] , and Uni-Mol [76] are pretraining methods. We adopt the same scaffold splitting strategy as GEM and Uni-Mol with three repeated runs.Results. The overall performance of InstructMol based on GEM and other baseline methods is summarized in Tab. 3. InstructMol achieves new SOTA results on all MoleculeNet tasks and brings an average improvement of 4.35%. Notably, its benefits are much stronger for regression tasks with a mean decrease of 9.98% in RMSE. Another set of results on MoleculeNet with a different splitting method is in Appendix Tab. 5, where InstructMol also outperforms all baselines. This highlights the necessity and importance of leveraging unlabeled examples to refine and transfer task-specific knowledge after pretraining through instructive learning. It also implies that InstructMol is not incompatible with existing pre-training ML models but can effectively supplement them with an additional instructor model. Empirical evidence of pseudo-labels' accuracy. Pseudo-labels' accuracy is crucial, as it reflects InstructMol's ability to generalize to unseen molecules. However, no ground-truth annotations exist for unlabeled data across the mentioned tasks. To address this issue, we adopt partition-coefficient values (LogP) as the target. A key predictor of drug-likeness featured in the famous "Lipinski's rule of five," LogP can quickly screen large libraries of potential therapeutics using computational tools like XLogP. We utilize a public Kaggle dataset of 14.6k molecules with associated LogP values for training and evaluate the predictions on the much larger ZINC15, whose LogP is computed using RDKit. Furthermore, measurements often number in the tens rather than thousands in the low-data regime of drug discovery. To assess InstructMol's limits, we significantly reduce the training size and investigate its efficacy with only 0.1%, 0.5%, 1%, 5%, and 10% of the entire training samples, resulting in scaffold-split training sets of 14, 73, 146, 730, and 1,326 molecules, respectively. Instructor model's Behavior. The instructor estimates the target model's uncertainty. Therefore, thoroughly evaluating its judgment and contribution is beneficial. Here, we analyze its output value distribution on labeled and unlabeled data in different training stages of an activity cliff estimation task (CHEMBL214_Ki). The plots in Fig. 5 demonstrate the transition of an instructor model. Notably, since we pretrain the instructor before SSL, it performs well in distinguishing fake labels but remains confused with real ones. From the first iterations to 10K iterations, the instructor gradually gains a stronger capacity to discriminate true labels (confidence score → 1.0) and fake ones (confidence score → 0.0). Taking a step further, we draw the distribution of the target model's output in Fig. 6 and show a significant overlap between predictions of labeled and unlabeled data. This undoubtedly excludes the hypothesis that the instructor discriminates labeled and unlabeled data simply based on distinct distributions of their predictions. Even though predictions of labeled and unlabeled data are highly similar, the instructor still succeeds in comprehending their uncertainty and guides the target model to leverage pseudo-labels more cautiously. The instructor model's interpretability is a byproduct of our InstructMol and can be useful for many real-world biochemical problems. Here we give the relevant appendix figures cited in the manuscript. 
conlcusion: 
 This paper presents InstructMol, a novel instructive learning framework, to alleviate the difficulty of experimentally obtaining the ground truth properties of molecular data and to overcome the limitation of the small number of labeled biochemical data points. InstructMol sufficiently cutting-edge pretraining methods for molecular representation, and addresses essential real-world problems.Limitations. Despite the great progress of InstructMol in achieving enhanced molecular learning capacity, there are still minor limitations that require future exploration. For instance, our combination of InstructMol with self-supervised mechanisms is based on existing methodologies such as GEM, Uni-Mol, and GROVE. It is interesting to develop a more suitable self-supervised learning algorithm that can be better aligned with InstructMol. Vanilla PL It should be noted that these 9 new small molecules are highly similar in structures, but IntructMol successfully and confidently differentiates their bioactivity based on their geometries. In addition to that, most experienced medicinal chemists also believe that the two sets of data within 10 times are manipulable and can be recognized as accurate prediction results. The error range of InstructMol's results for the prediction of small molecule properties of new compounds further proves the potential of our model in predicting essential properties of activity cliffs molecular and can effectively get aware of subtle changes in the presence of bioactivity cliffs. 1 [ŷ ⋆ i ≥ γ 1 ] + 1 [ŷ ⋆ i ≤ γ 2 ] H f f x ⋆ j , ŷ⋆ j No CPL 1 [ŷ ⋆ i ≥ γ 1 (e)] + 1 [ŷ ⋆ i ≤ γ 2 (e)] H f f x ⋆ j , ŷ⋆ j No UPS 1 [u(ŷ ⋆ i ) ≤ µ 1 ] 1 [ŷ ⋆ i ≥ γ 1 ] + 1 [u(ŷ ⋆ i ) ≤ µ 2 ] 1 [ŷ ⋆ i ≤ γ 2 ] H f f x ⋆ j , 
 
 Appendix:  


Decision: The paper presents an instructive learning framework that uses large-scale unlabeled data to enhance molecular property prediction. Unlike traditional methods requiring knowledge transfer between domains, InstructMol operates within a single domain, avoiding discrepancies between pretraining and fine-tuning stages. The framework incorporates a pseudo-labeling mechanism supported by an instructor model that estimates the reliability of these labels. The authors demonstrate InstructMol on molecular datasets and OOD benchmarks. 

The reviewers provided a balanced evaluation of the paper, highlighting several strengths and weaknesses:

**Positive Points:**
* The InstructMol framework is praised for its approach to leveraging unlabeled data for molecular property prediction, particularly its ability to handle OOD scenarios effectively.
* The method addresses a critical problem in biochemical research, namely data scarcity, and demonstrates its applicability through extensive benchmarking, achieving competitive results.

**Negative Points:**
* Several reviewers noted the need for a more detailed discussion on the computational complexity and resource requirements of the InstructMol framework, particularly its scalability to large datasets.
* Concerns were raised about potential overfitting issues due to the iterative use of pseudo-labels and method's robustness to noise introduced by additional training data.
* There was some critique regarding the clarity and organization of specific sections, such as the related work, and the need for more detailed analysis of certain experimental results and their implications.

After considering the reviewers' feedback and the authors' rebuttals, I recommend accepting the paper at NeurIPS. While the reviewers identified areas for improvement, the authors' rebuttals were thorough and addressed many of the concerns. The weaknesses, such as computational complexity and potential overfitting, are acknowledged but do not detract significantly from the overall impact and novelty of the work.

Reviews: review: paper_summary: paper_summary: The authors present "InstructMol" which does not require transferring knowledge between multiple domains, which avoids the potential gap between the pretraining and fine-tuning stages. and demonstrate it on real-world molecular datasets and out-of-distribution (OOD) benchmarks.
strength_weakness: strength_weakness: The paper mostly focus on GNN as backbone but would be worth discuss more about transformer based model trained on SMILES representation.
questions: questions: How would Instructive Learning Framework works on  transformer based model trained on SMILES representation?
limitations: limitations: The paper mostly focus on GNN as backbone but would be worth discuss more about transformer based model trained on SMILES representation.
paper_summary: paper_summary: The authors develop a method, called InstructMol, for adding pseudo-labels to any training task by including an "instructor" that is trained to discriminate real labels from pseudo-labels, and whose uncertainty is used to modulate the training loss for the primary predictors. The authors show that adding this instructor model improves overall performance over similar methods across a number of property prediction tasks. The authors also show that pretraining delivers state of the art results on the MoleculeNet benchmarks.
strength_weakness: strength_weakness: - Using GIN, GAT and GCN for the results in Table 1, but then GEM for those in Table 3, makes it feel like the results are cherry-picked, especially since the GEM+InstructMol results in Table 3 are comparable to and within the error of some of the other methods.
- It is unclear how the results of Figure 4 were obtained. The reader assumes these are comparable to the GEM+InstructMol of Table 3, but this should be better explained.
- The distance of the 9 molecules examined in the real world drug discovery section to the training set should be examined, even if the fact that these were patentable suggests they are dissimilar from known molecules. Also, there are likely many more examples like this that could have been provided more prospective evaluation, and only showing one makes the reader again wonder if the example is cherry-picked.
- It would have been useful to see similar plots to those in Appendix C for other uncertainty estimation methods as a way of showing that InstructMol learns better separation between real and pseudo labels.
questions: questions: - I can't tell from the paper if the benefit that comes from InstructMol is due simply to training on more data or for more iterations, simply due to having more labeled data. In which of the experiments is this possibility controlled for? Is it Table 1 where other pseudo-label methods are compared to? If such a control exists, will you please make this explicit in the paper? If not, this seems a critical experiment to run.
- Why not use TDC benchmarks instead of MoleculeNet? While the datasets are obviously similar, TDC has done some additional work to clean them, provide reasonable splits, etc.
limitations: limitations: NA
paper_summary: paper_summary: This paper targets the problem of label-scarcity in the domain of molecular property prediction. It can be seen as an improved version of proxy labeling. It utilizes a separate model that measures pseudo-labels’ reliability and helps the target model leverage large-scale unlabeled data. This method applies to both classification and regression tasks. The authors run numerous experiments on predicting molecular properties, OOD generalization, and combination with pre-training models.
strength_weakness: strength_weakness: The improvement in model performance is relatively weak, with large standard deviations, and the method is compute-demanding due to the separate model and the iterative procedure. Additionally, some technical details and experimental results are unclear.
questions: questions: * In Line 201-202, how are the average increase in AUC-ROC and the average decrease in MSE defined and calculated? Why are there only three numbers for the six classification tasks?
* I assume the architecture of the InstructMol model is not necessarily the same as the target molecular model. The article does not clearly explain how the InstructMol model is constructed.
limitations: limitations: The authors briefly touched upon the necessity of developing a self-supervised learning algorithm better aligned with InstructMol than existing methodologies. Could the authors briefly elaborate on the extra compute incurred by the instructor model?
paper_summary: paper_summary: The paper introduces InstructMol, an innovative learning framework designed to address the challenge of data sparsity in chemical and biological sciences by leveraging large-scale unlabeled data through reliable pseudo-labeling. Unlike traditional methods that rely on transferring knowledge between domains, InstructMol operates within a single domain, eliminating potential discrepancies between pretraining and fine-tuning stages. The authors demonstrate the algorithm's high accuracy using various real-world molecular datasets and out-of-distribution (OOD) benchmarks, demonstrating its effectiveness in enhancing machine learning applications in biochemical research.
strength_weakness: strength_weakness: - The paper does not provide a detailed analysis of the computational complexity or resource requirements of the InstructMol algorithm. 
- While the paper showcases the superior performance of InstructMol, it does not sufficiently address potential overfitting issues that may arise due to the iterative use of pseudo-labels.
- Although the paper compares InstructMol with several baseline methods, it does not include a thorough comparison with some of the latest advancements in semi-supervised learning and domain adaptation techniques.
questions: questions: - Due to limited discussion on computational complexity, it difficult to assess the practicality and scalability of the approach for very large datasets or in resource-constrained environments.
- It was mentioned that determining 'k' in InstructMol is important; please show the experimental results and discussion related to this.
- It would be beneficial to have an experiment to determine whether using a poor model for model $f$ can still result in performance compensation due to the confidence score, or if it leads to a decline in performance.
- Why RMSEs in Figure 3 become worse as the training data increases?
- What model and data were used for training on "Real-word Drug Discovery"?
limitations: limitations: The authors addressed limitations in Appendix.
paper_summary: paper_summary: This paper proposes a framework InstructMol for utilizing unlabeled data to help molecular property prediction on out-of-distribution (OOD) domains. The framework combines (1) a molecular model *f* that predicts (pseudo)labels with (2) a binary classifier *g* as instructor that evaluates the probability of labels being pseudo and reweighs *f*’s loss. In the experiments, it is compared to several self-supervised learning (SSL) baselines in predictive accuracy and OOD generalization. The experiments also investigate the effects of unlabeled data size and the instructor model’s behavior as an uncertainty estimator.
strength_weakness: strength_weakness: - Sec. 2’s review on related works is comprehensive but could be better organized. 
  - The separation of pretraining and SSL is confusing: techniques like contrastive learning are often viewed as SSL, and some SSL algorithms work as pretraining.
  - The mention of methods “jointly learning multiple tasks” (Line 171) seems more appropriate in related works than method.
- The paper presents strong empirical results but not enough interpretations. Compared to SSL, what extra information does InstructMol utilize, or why does it extract information more effectively, that leads to better accuracy/generalizability? Discussing these could provide more insights to future model development.
questions: questions: - In Line 33, domain knowledge being biased is raised as a problem. (1) Could some “bias” be helpful, e.g., by providing correct inductive bias to the model? (2) I suggest citing relevant literature on the scenarios where bias becomes a problem.
- Line 78, “usually follow significantly different distributions” seems exaggerating. Is there justification?
- Line 97, could uncertainty quantification methods, e.g., Bayesian NN, SNGP, evidential DL, address the problem for regression tasks?
- Section titles of 5.3 and 5.4 (Visualization) are confusing.
- The meaning of “confidence score” is unclear: in the main text (Lines 156, 301), 1 indicates pseudo-label, while Fig. 5 seems the opposite.
limitations: limitations: Discussed in Appendix.

Comments: title: Thanks for Feedback
Thanks for Feedback
comment: Dear Reviewer 73Xa,

Thank you for your positive feedback and your continued recommendation for acceptance. We are delighted to hear that our additional experiments have addressed your concerns. Thank you once again for your valuable time and insights!

Best regards,

Authors
Dear Reviewer UbNR,

We are delighted that you have found our rebuttal feedback valuable and maintained your positive score. We greatly appreciate your recognition of our work and would respectfully inquire if there might be any additional opportunities for us to improve our manuscript. Once again, thanks for your constructive feedback, and we would eagerly welcome any further guidance at your convenience!

Best regards,

Authors
Thanks the author for checking the results on SMILES Transformer! I am pleased to see that it helps not only on GNN. I am keeping my score as is.
I would like to thank the authors for their response. Though the calculation of the average increase in ROC-AUC is still not clear to me (e.g., compared with which baseline model),  hopefully they will clarify further in their revisions. I will keep my score as it is.
Thank you for the rebuttal and attempting to address my questions.

Re: question 1 - are you saying that UPS is trained on the same amount of data, or for the same number of iterations, as InstructMol, and so this comparison is the control for that?

Why were none of the weaknesses I pointed out addressed?

I am currently keeping my score as is.
I appreciate the authors' time and effort. While they have addressed most of my concerns, some aspects remain unclear.

- In the experiments related to the proposed \( k \) in the rebuttal, there are still questions about the effectiveness of the adaptive decay strategy.
- I agree that adding training data can introduce more noise. However, I still have concerns that this may indicate a potential weakness in the model's robustness to noise.
I appreciate the authors' comprehensive response, which addresses my concerns well. I am keeping my scores for this paper high at this stage.