Adaptive Pseudo-labeling for Quantum Calculations
Authors: Kexin Huang
Vishnu Sresht
Brajesh Rai
Mykola Bordyuh
Venue: ICLR 2022 Submitted (2022)

Abstract: Machine learning models have recently shown promise in predicting molecular quantum chemical properties. However, the path to real-life adoption requires (1) learning under low-resource constraint and (2) out-of-distribution generalization to unseen, structurally diverse molecules. We observe that these two challenges can be alleviated via abundant labels, which are often not the case in quantum chemistry. We hypothesize that pseudo-labeling on vast array of unlabeled molecules can serve as gold-label proxies to greatly expand the training labeled dataset. The challenge in pseudo-labeling is to prevent the bad pseudo-labels from biasing the model. We develop a simple and effective strategy Pseudo that can assign pseudo-labels, detect bad pseud-labels through evidential uncertainty, and then prevent them from biasing the model using adaptive weighting. Empirically, Pseudo improves quantum calculations accuracy across full data, low data and out-of-distribution settings. 

Under review as a conference paper at ICLR 2022
ADAPTIVE PSEUDO-LABELING FOR QUANTUM CAL- CULATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Machine learning models have recently shown promise in predicting molecular quantum chemical properties. However, the path to real-life adoption requires (1) learning under low-resource constraints and (2) cut-of-distributie
generalization unseen, structurally diverse molecules. We observe that these challenges can be alleviated via abundant labels, often not the case in quantum chemistry. We hypothesize that pseudo-labeling on a vast array of unlabeled molecules can serve as gold-label proxies to expand the training labeled dataset significantly. The challenge in pseudo-labeling is to prevent the bad pseudo-labels from biasing the model. Motivated by entropy minimization framework, we develop a simple and effective strategy PSEUDo that can assign pseudo-labels, detect bad pseudo-labels through evidential uncertainty, and prevent them from biasing the model using adaptive weighting. Empirically, PSEUDo improves quantum calculations accuracy across full data, low data, and uut-of-distributio settings.
INTRODUCTION
Accurate quantum mechanical (QM) calculations of drug-like molecules at CCSDT (coupled cluster single-double-tripee or MP2 (second order Moller-Plesset (Watts et al.]
1992 level of theory, which is essential to characterize biomolecular interactions, continue to be prohibitively expensive, despite recent advances in hardware capabilities. Machine learning (ML) models have astonishing performance in approximating these calculations at fraction of the computational cost (von Lilienfeld & Burke 2020). Such speedups could accelerate the discovery of new therapeutics.
In the absence of large-scale benchmark data sets reporting CCSDT or MP2 level calculations, most publications on this topic have relied on QM9, a standard benchmark of Density Functional Theory (DFT)--evel energy and properties, for training and evaluating QM/ML models. This data contains molecular geometries and energies that were generated using DFT, which provides a faster (compared to CCSDT, for example), but less accurate description of molecular energetics and properties. Previous publications have utilized computed geometries and properties from in in-distribution molecules in QM9 to test and validate model performance, showing excellent performance of QM/MI. models as demonstrated by low mean absolute error (MAE) of predicted energies of other properties relative to the gold otaddataddard DFT calculations (i.e., gold label). However, it is unclear how the reported architectures (e.g., SchNet (Schütt et al. 2017 would perform in the regime of low but accurate data (CCSDT or MP2).
Two challenges remain in the way of the realistic adoption of ML-aided QM caleulations. Firstly, training molecules only cover part of the distribution, and real-world adoption requires out-of- distribution generalization. Secondly, computing QM properties for datasets of the size of QM9 is costly. Widespread applicability of ML for QM (c.g., such as to CCSDT/MP2) thus requires models to generalize well given low-data constraints. Both challenges listed above can be attributed the difficulty and cost of generating experimental data in chemistry and materials science. If we have large and diverse set of labeled molecules, the ML model can generalize to larger chemical spaces and achieve better predictive performance.
Present work. Our key observation is that state-of-the-an ML models can obtain a low-precision estimation of the properties ML predicted label for any unlabelled molecule can be used as a low- owwpprccision estimation of the true label. We hypothesize that by considering these low-precision pseudo-labels as the training
given small set of labeled molecules. This suggests that the

---

Under review as a conference paper at ICLR 2022
targets in the training process, can largely increase the training dataset size and diversity and alleviate the fundamental label scarcity issue, along with the associated challenges in low-data and dustoo-distiibuttin generalization.
In our study, we develop a simple, effective, and model-agnostic pscudo-labeling strategy called PSEUDo. Notably, we solicit extensive array of unlabeled molecules from the PC9 dataset (Glavatskikh et al. (2019) where an ML model assigns pseudo-labels for them. This generates large set of "labeled" training data points, even when only a relatively small number of reference QM properties are available for training. As the unlabeled dataset is diverse and unseen in QM9, it also helps generalization to unseen data. One crucial issue in pseudo-labeling is the introduced bias from low-quality pseudo-labels. To resolve this, we rely on key observation that data point with less evidence/higher model uncertainty is more likely be of low-quality pseudo-label (Section (6). Thus, we use -eder-egeee evidential uncertainty to quantify each unlabeled data and use it to adaptively lower the weight of bad pseudo-labels in the training loss to reduce the bias effect.
In summary, we have made the following contributions: (1) Previous QM focus on in-distribution and label abundant setting while we point out that the realistic ML-aided QM challenges lie in low-data and out-of-distriuttio settings; (2) Pivoting away from the status quo in improving the physics-basee representation, we propose to look at data-centric approaches on learning from the scheduling strategy designed specifically for QM since many classic pseudo-labeling tricks do not work well in QM (e.g.. data augmentation, model noise, student-training re-inirialization); (4) To prevent bad pseudo-label in biasing the training, we devise an adaptive weighting scheme where the weights are generated using evidential uncertainty such that bad pseudo-labels are automatically filtered out; (5) We derive a theoretical connection between the evidential loss and entropy minimization framework; (6) Empirically, we show PSEUDo can improve QM accuracy for any atomistic model across full-data, low-data, and out-of-distribute distribution settings.
vast array of unlabeled molecules; (3) We propose an episodic pseudo-labeling
2 RELATED WORKS
ML-aided quantum calculations. Recently, many ML models have been proposed to improve quantum calculations. They mainly focus on improving the physics-based representation in the full QM9 dataset setting (Schüttt et al.] [017 Unke & Meuwly 2019 Anderson et al., 2019 Lu et al.,
2019; Klicpera et al. 2020 Liu et al. 202 Qiao et al. 2021). In contrast, our work proposes to shift the focus on improving training strategy instead of the model architecture. In addition, PSEUDo is model-agnostic, and it can improve any atomistic model. Additionally, we focus realistic quantum calculations such as learning in the low-data regime and -iitt---iittitttionn inference.
Pseudo-labeling. Pseudd-labelling/elf--rzzzz
generates pseudo-labels for unlabeled data. Numer- ous works exist for how to assign pseudo-labeke labels, notably through trained ML model prediction (Lee) al. 2013), label propagation Shi al. 2018 Iscen et al. 2019), and history cache (Likhomanenko
et al. 2021 Higuchi et al., 2021). PSEUDo is different as focuses on detecting and preventing bad pseudo-label from affecting the model. Also, PSEUDo adopts novel episodic pseudo-label strategy with re-initialized learning rate. (Xie et al.] (020) re-initialize the network as a student when a new pseudo-label set is generated along with noise per epoch. In contrast, PSEUDø has no student, and no noise as both are shown to be ineffective for QM in Section 6 In addition, small perturbational noise in 3D molecular geometry could easily lead to a drastic energy difference. Thus, the noise strategy does not work for QM tasks. More related is a concurrent work (Rizve et al.) (02I) that develops an uncertainty-aware pseudo-laheling strategy, but they introduce additional hyperparameters to remove pseudo-labels at some uncertainties. In contrast, PSEUDØ uses an effective adaptive weighting scheme, along with an episodic pseudo-labeling training schedule. Additionally, PSEUDo is the first method that studies pseudo-label in quantum calculations, which present unique challenges and motivations.
Uncertainty. Model uncertainty is a well-studied subject (Kendall & Gal 2017 Lakshminarayanan
et al. 2017 Blundell et al. 2015). Amiii et al. 2020) use evidential uncertainty to add prior over the gaussian parameters to search for higher-order patterns for regression tasks. PSEUD leverages evidential uncertainty as the uncertainty measure. Note that PSEUDo is uncertainty measure-agnostic. We can easily switch to alternative uncertainty measures. Recently, (Soleimany et al.) 2021) adapt evidential uncertainty and has shown it can successfully help guide property prediction. In contrast,
2

---

Under review as a conference paper at ICLR 2022
Gonerate Pabudo-labal with Uncertainty
Train Gold-Label and Psoudo-Label Adaplive
K epoche
(k==1)
fk
(a.,8)
Y, Ri
Prediction
1 = Bl/r| =
Ju opisodes
Uncertainty # Varlju 1T
Gold-labeled
Unlabeled
Pseudo-labeled
Uncertainty
Figure 1: PSEUDo illustration. In every episode k; PSEUDo assigns pseudo-labels along with their evidential uncertainty using trained neural network (Kk-1) from previous episode. The uncertainty is used as weight to adaptively adjust the loss in this episode's s neural network f(k)'s training to reduce the effect of bad pseudo-labels in inner-loop training with N epochs.
we leverage evidential uncertainty as proxy for pseudo-label quality to tackle low-data and out-of- distribution challenges in realistic quantum calculations setup.
3 PROBLEM FORMULATION
Let X (xI,.. (xI...., (x) } denote N molecules, where each molecule is uniquely defined by 3D
coordinates ((a), c1)) for N; atoms with atom types /y in the corresponding molecule. We then denote y (V1,-.- yN a set of quantum mechanical properties for each molecule. The labeled dataset thus consists of set of pairs of 3D coordinates and scalar labels D = {X,V},
In addition the labeled data, we solicit large quantity of unlabeled data to generate pseudo-labels. J. where M is the size of the unlabeled dataset.
We denote an unlabeled dataset U {x1,..., XMr Given an atomistic model f()), we can generate pseudo-labels y = (ú....., for x; E 24.
UM ), where ÿ fx;)
The problem is to find machine learning-basse based atomistic model f x y that can establish an accurate map from 3D coordinates to the quantum mechanical properties molecules, with the help of pscudo-labeled dataset U.
PSEUDo: ADAPTIVE PSEUDO-LABELING FOR QUANTUM CALCULATIONS
PSEUDo (Figure 1 is an approach for quantum chemical property prediction. Building theoretical motivation from Section [5] PSEUDø solicits pseudo-label on a vast array of an unlabeled dataset to increase the diversity of the training space via an episodic labeling strategy. Then, it adaptively weights the pscudo-labels using evidential uncertainty to allow positive transfer. The overview is in Algorithm
Episodic Pseudo-labeling, We devise pseudo-labeling strategy that can ensure learning from the pseudo-labels the fullest extent for QM. We have made distinct modifications compared to existing works. First is the pseudo-label scheduling. In the standard pseudo-labeling (Lee et al. 2013), pseudo-labels are updated in every update and the model continuously trained. In contrast, we devise an episodic training strategy, where each episode consists of K epochs, and pscudo-labela are regenerated in every episode, while the model is continuously trained. This is important because we observe that fast updates on pseudo-labeling prevents the model to fully learn from all useful information in pseudo-labels, In contrast, our episodic approach gives the model more time to absorb useful information from given set of pseudo-labels. Second modification is the model update. In self-training (Rizve et al. 2021 Xie et al. 2020), a set of pseudo-labe labels are regenerated after K epochs and the model is reinitialized Instead, train the same model throughout episodes. This new strategy allows the model to be exposed to larger number of labels or training data points given the same time frame. For each episode, also reinitialize the learning rate with small step-wise
3

---

Under review as a conference paper at ICLR 2022
decay strategy to allow the model chance to jump out of the local optimum from the previous set of pseudo-labels.
Formally, PSEUDo mainly consists of three stages: in the first stage, regular training is conducted on labeled data D, and the output model is the initialized model f()) In the second stage, the updated model at episode & then conducts inference on the entire unlabeled data 5 = ((k) (24) to generate the pscudo-label set. The per-episode pscudo-label set is then combined with the gold-labeled data to form the training data for the next episode. In the third stage, the model is further trained using the combined dataset to get a new model ((++11 after N epochs. The second and third stages are then reiterated till the loss stops decreasing.
Evidential uncertainty quantification. Pseudo-labels are noisy. Many are incorrect and can po- lentially lead to negative transfer. Thus, it is instrumental in detecting the quality of pseudo-labels. However, there is auxiliary information in the dataset about the pseudo-labels. Thus, we need to quantify it through some proxies that can be assigned without auxiliary information. Our key obser- vation is that low-quality pseudo-labels have high model uncertainty, and high-quality pseudo-labels have low model uncertainty. Another advantage of model uncertainty is that it can be estimated solely from x, if we make it model uncertainty-awar
Building on the theoretical motivation about the connection between evidential uncertainty and entropy minimization Section we use evidential uncertainty as to the proxy for label quality. The evidential modeling of molecular property allows us to derive analytical solution of uncertainty, which could be directly used to weight the pseudo-labels, Formally, we can model the label probabilistically as it is drawn from (y1, , y1) ~ NN,, where the mean μ are variance o² are unknown. To estimate them, we pose a prior
N(7, o20 ~ (a,
where the parameters 0 == (u, o) is an instantiation of the posterior p(u, 0317, or, oz B). The choice of prior allows the factorization p(u, o²) = p(1) p(o²) [Jordan] 2009). The posterior then becomes PermallyGaamma/f a, B) where the maximum likelihood estimation of & can be analytically found as
8 a
E Ely4 = 7, Elo²)
Here, E[o²)] plays the role of the aleatoric (data) uncertainty. The uncertainty of the model prediction can also be calculated, i.e. epistemic uncertainty:
V [ar[u]] = E|o2)/0-
(3)
v(a - 1)"
As the MLE is deterministic, the model can output the four prior parameters {7, U. a, directly where the prediction and uncertainty can be derived from them analytically. The prior optimized
by evidential loss gevi (Amini et al. 2020
3(1 + v) 20
Cevi log y:: 7, ua
+ Alys - (2u + +a)
(4)
where the first term is to maximize the log-likelihood of the posterior predictive, which is derived as the Student's t-distribution. The second term is a regularizer that encourages lower total evidence, which correlates positively to higher epistemic uncertainty, when the gap between ground truth and prediction is high (i.e. when model prediction is with high error). Similarly, encourages lower uncertainty when the model prediction is errorneous. This encourages the model to generate an accurate estimate of uncertainty or the degree of errors for the pseudo-labeled data points. The regularization is controlled by hyperparameter
Adaptive weighting. The evidential uncertainty detects the low-quality pseudo-labels. The next step is to remove the noisy effect from the model training. Naive methods often use removal based on threshold (Rizve et al.) 2021). However, it has two disadvantages: (I) it introduces new hyperparameter the threshold; (2) it removes portion of unlabeled noisy data, which can contain useful information. Instead, propose adaptive weighting mechanism that adapts the evidential loss given the inverse epistemic uncertainty. Intuitively, a higher uncertainty data point requires a lower effect in the loss function because it is more likely that the sample has a low pseudo-label quality,
4

---

Under review as a conference paper at ICLR 2022
Algorithm 1: PSEUDo Algorithm.
Input: Labeled data D = (1x1,81), (x), unlabled data = {x1, .. [Nv]
U + (h W - f
// Initialize with empty unlabeled data
for k E (1, 1I K} do
// Outer-loop with K episodes
T D DUN for (x,, yi) ET do
// Join updated pseudo-labels
// Inner-loop with epochs
0,
(7i, Ui, oi, BB) = (xi)
// Evidental parameters
y; Elu] = 7/i
// Posterior prediction
L = L(y:, 02 Wi)
1/ Adaptive evidential loss via Eq.
((h-1) Update( f(k-1),
// Inner-locp update
end
((k)
((--1)
11 Update teacher model for pseudo-labels
for
x, E 21 do
0, (fis Cis ai Bi.BBB) = (x))
y: 9i
// Infer a new set of pseudo-labele // Update pseudo-labels // Update adaptive weights
U, + (i)
Wi + Varlu]; Barlu// (ā: - 1)/3,
end
end
and we want to reduce its effect on the model. Conversely, if pscudo-label has low uncertainty, the label quality high enough to be used as a proxy of a gold-label. Thus, we should assign higher score in the loss for it. The uncertainty is from the teacher model in the previous episode and is fixed throughout the current episode. Thus, the adaptive weight for each pseudo data point becomes W = Varlu]= The final loss then becomes
L
1
Σ rvvi RED
W: *EN Eieu W;
C.vi
(5)
D
where the first term corresponds to the labeled dataset D does not have weights unlike the second term corresponding to the unlabeled data. This adaptive loss solves two disadvantages since it has zero hyperparameters, and it removes the effect of bad pseudo-labels while retaining all training examples including the noisy ones to maximize the diversity of the training space.
5 PSEUDo MOTIVATION: CONNECTION TO ENTROPY MINIMIZATION
We derive motivation about why evidential uncertainty and the weighting mechanism could be beneficial to pseudo-labeling based on the entropy minimization framework for semi-supervised learning from Grandvalet & Bengio 2004 2006): Lee et al. (2013). Notably, our use of Bayesian modeling enables us to analytically derive a conditional entropy for pseudo-labeled data. We find that evidential loss strongly relates to conditional entropy, and minimizing evidential loss directly minimizes entropy. Secondly, we find the conditional entropy could be decomposed into the inverse epistemic uncertainty and the lig-likel,hood, which motivates weighting mechanism.
In the standard regression setting, one seeks to maximize the likelihood of the model () on the labeled data set D. To utilize the unlabeled data set, we need to extract some useful information on how the model behaves on the unlabeled dataset and inject this information to improve the model. To measure the utility, entropy (Y [24) is introduced (Grandvalet & Bengio 2006) as a proxy to measure the amount of information in unlabeled data:
H( (2)
Σ X.EU
Ey (oppp((lx]]--a log (o x;)]-
(6)
Throughout the text, we are referring to entropy as Shannon entropy. Intuitively, high entropy is indicative of the poor model confidence the unlabeled data point, while the low entropy can
5

---

Under review as a conference paper at ICLR 2022
be interpreted as high confidence of the model predictions. High entropy, associated with random predictions, while low entropy is associated with non-random behavior. Hence force, we hypothesize that small entropy may be indication of signal that our model can benefit from. Small entropy, as seen below, corresponds to high model confidence. and vice versa. Large entropy corresponds to high model uncertainty. Entropy minimization framework casts the regression the following optimization problem:
arg max [log Po ()) X) cH( (2),
where c is the proportionality constant. Intuitively, here, the objective tends maximize the log- likelihood on the labeled dataset while minimizing the entropy on the unlabeled data set at the same time to transfer knowledge from unlabeled data.
In previous works (Schütt et al., 2017 Liu et al. 202I), molecule properties are not modeled probabilistically such that entropy calculation infeasible. In contrast, PSEUDO uses Bayesian modeling approaches that allow us to analytically calculate the entropy. For every molecule x; the machine learning model outputs four parameters fx,)) = (as, 3i, Yr 14). Based on these parameters, the likelihood of label y given the input molecule is given by the Student's t-distribution in the context of evidential regression
py x,) St (y: 7e, 2a. 20:)
evaluated at location parameter Ys. Student's t-distribution scale parameter oa, degrees of freedom. The entropy of the Student's t-distribution given terms of evidential parameters is readily available (Appendix A):
[.(1+v)
M:0, and 20;
2a; +
20 + Vp
v(az)) + log V2a, B(as, [/ llog st.s
1 2
H(v x:)
2
2
where V is digamma function and B(.,-) is beta function. If we take model (epistemic uncertainty) (Eq. 33 as our evidential uncertainty, we can show that minimizing entropy directly relates to minimizing epistemic uncertainty. We plot the relation between the entropy and epistemic uncertainty in the Figure
As the next steps, we aim uncover the dependence of the entropy the model uncertainty of our pseudo-labeling approach. This can be done if we make two simplifications in entropy evaluation. Firstly, to introduce iterations as in pseudo-labeling, we replace entropy with the cross- entropy between two probability distributions: the predictions are generated from the probability distribution Pa(t-1) (y/ x at iteration step 1 and lig-liielihood are evaluated with respect to probability distribution Poçt) x)) at iterative step t:
H()
Σ x.EUI
Ey-pecc-n(slie [- log Po(e) xx)]
(10)
Upon convergence, t + +00, the probability distributions at every iterative step Po(t- Po(t-1)(91x;) (g|x;) R Poçt) (y(xx) and are approximately the same and one can view introduced cross-entropy with respect to time step as entropy. At the earlier stages of training cross-entropy acts as a regularizer encouraging network parameters f(t) to match O(t - 1).
As second approximation, uncover model uncertainty in mathematical formulas, we approximate the probability distribution at time step t - 1. We resort to empirical estimate of the entropy, as done in (Grandvalet & Bengio 2004). We select labels y at the highest mode of probability distribution (e(t-1)(3) x;), which corresponds to y 7 (-1 We obtain the following approximate for the entropy:
H() (2) 2 Hemp ( (2/14)==
Σ gf- log Po(o) - xx))
(11)
X.EU
where the log probabilities are weighted by empirical probabilities weights E evaluated at iterative step t - 1 when plugged into Eq. X (also see Appendix. Eq. [8 exact formula Student's (-distribution)
1
e - -St(y = 76, 22 20x)
(12)
/ 20, 03 ((},a))

---

Under review as a conference paper at ICLR 2022
(4)
22
2.0
16
1.0
50
12
10
0.0
10
20
30
4.0
50
Epistemic uncertainty Var[u]]
Epistemic uncertainty Var(p]
11
20
Empincal weights
0.0
0.55
0.36
84
10
Epistemic uncertainty Var[u]
Epistemic uncortainty Var[u]
Figure 2: (a) Dependence of the entropy (Eq. 9 epistemic uncertainty and virtual observation
parameter a for fixed alcatoric uncertainty E 3 1 1. As the the epistemic uncertainty increases, the entropy is also increases for all values of parameter a. For example, figure (b) demonstrate the trend for a fixed a 2. Figure (c) demonstrates the dependence of empirical weights (Eq. [4] on epistemic uncertainty. The empirical weights tend to decrease the epistemic uncertainty increases.
Figure (d) demonstrates this trend for a fixed a = 2.
To establish relationship between empirical weights 8 and aleatoric B[o / epistemic Varlus] uncertainties we rewrite
st.;
(Var[ue) + E(o²))
(13)
as
8/- (Var[u,) + B V2B( (2B(),a)voo aaaaaaaa
(14)
Empirical coefficients depend on aleatoric, epistemic uncertainties and a; parameter, which can be interpreted as virtual observations in support of the variance estimation [Jordan 2009). In the
limiting case >>>>>> 1 one can approximate beta function via Stirling formula B a;) & v }
and empirical weights become
e R (Var[u.) + -
(15)
We can express the empirical coefficients depend both on aleatoric and epistemic uncertainties in symmetric fashion.
We selected adaptive pseudo-labeling coefficients W; in our pseudo-labeling approach Eq. 5 to be inverse epistemic/model uncertainties. We can see, that those coefficients directly relate to empirical coefficients derived from entropy minimization approach Eq. [4] empirical coefficients also depend on model uncertainty in the inverse fashion. As the model uncertainty increases, the empirical coefficients & tend to decrease minimize the entropy.
7

---

Under review as a conference paper at ICLR 2022
Table 1: Dataset statistics.
Setting
Training Set
Validation Set
Testing Set
Unlabeled Set
OOD Set
Full-data
1 110,000 (QM9)
10,000 (QM9)
10,831 (QM9)
99,234 (PC9)
Low-data-1$9
1,100 (QM9)
10,000 (QM9)
10,831 (QM9)
108,900 (QM9)
Low-data- 10%
11,000 (QM9)
10,000 (QM9)
10,831 (QM9)
99,000 (QM9)
Out-of-distribation
110,000 (QM9)
10,000 (QM9)
10,831 (QM9)
99,234 (PC9)
99,234 (PC9)
Table 2: PSEUDo improves on full data setting. Reported metric is MAE. The lower the better.
Property
Unit
SchNet
PhysNet
Cormorant
MGCN
DimeNot++
SphereNet
PSEUDo-S
PSEUDo-D
meV
41
32.9
36
42.1
24.6 19.5
23.6
32.9 24.7
20.4
ELUMO
meV
34
24.7
36
57.4
18.9
18.2
EXPERIMENTS
6.1 DATASET AND EXPERIMENTAL SETUPS
We evaluate PSEUDo using QM9 dataset Wu et al. follow the previous works (Liu et al. 2021 Klicpera et al. 2020) where a 11,0000/10,000/10,533 dadninn//vaiddationnestt
2018 under two settings. (A) Full-data: We
set is obtained. For the unlabeled data, solicit PC9, dataset of 99,234 molecules that consists of the same elements as QM9, curated by (Glavatskikh et al. (019). (B) Low-data: we set 6% of QM9 full training set the training set (i.e. 110,000) and remove the label of (1-k%) QM9 full training set and make it as the unlabeled set. We evaluate in two k values, and 10, which means only 1,100/11 1.100/11,000 1,000 data points is trained respectively. Dataset statisties summary is in T Note that PC9 has a wider chemical diversity than QM9, demonstrated by wider distribution of distances of chemical bonds and more functional groups (Glavatskikh et al.) 2019
PSEUDo is model-agnostic. We evaluate it with two model backbones SchNet (Schütt et al. 2017) (PSEUDo-S) and DimeNet++ (Klicpera et al.) 2020) (PSEUDo-D)), We do not experiment with the SOTA atomistic model SphereNet (Liu et al., 2021) because it is highly computationally expensive. targets ØHOMO- OLUMO, because the PC9 dataset only has these two
Our result is conducted targets. We use mean absolute eтror as the evaluation metric.
For baselines, compare with 6 state-of-the-an baselines, including SchNet (Schütt al. 2017) PhysNet (Unke & Meuwly, 2019). Cormorant (Anderson et al., 2. MGCN (Lu et al. 2019). DimeNet++ (Klicpera al., 2020), and SphereNet (Liu et al. 2021) We report the best results taken from the original authors' paper while using the same fraction of data split in the full data setting. For PSEUDo, we conduct two hyperparameter tunings on оhомо with SchNet backbone on the validation MAE with full data/low-data setting, respectively. The optimal hyperparameter is then used for both targets. Note that the atomistic model itself has the same hyperparameter as the original authors. Code will be released upon anonymous review period.
6.2 RESULTS
Overview of results. We report performances of PSEUDa in full data (Table low-data (Table out-of-distribution (Table 4 settings and find PSEUDo achieves the best performance across all settings, suggesting the robustness of the lseudo-labeling strategy. Systematic ablation study (Table also show the importance of each module in PSEUDo.
PSEUDo improves on fully supervised QM calculations. We report PSEUDo against 6 state-of-the- art models in Table 2 PSEUDo-D surpasses all baselines in both targets hhоMо, OLUMO- Notably, PSeuDo-D improves the SOTA by 3.2 meV, a significant margin. Particularly, comparing PSEUDO-S consistently improve
with SchNet and PSEUDø-D D with DimeNet++,
find PSEUDø the fully supervised setting by large margin (8.1 meV for SchNet and 4.2 meV for DimeNet++), highlighting the utility of PSEUDo and the high quality of PC9 unlabeled data. It also shows that
8

---

Under review as a conference paper at ICLR 2022
Table 3: PSEUDO improves on low-data regime. Reported metric is MAE. The lower the better.
Low- Low-Data Setting
1% QM9 (1,100)
10% QM9 (11,000)
Property
Unit
SchNet - Pseuie
DimeNet++ PSBUDO
SchNet
PSEUDO
DimeNett+ - PSEUDD
CHOMO
meV
265.4
110.8, 276.2
248.9 18.77 230.2
119.0
30.2,
88.8
81.1 -13.7, 67.4
CLUMO
meV
290.6
-67.8,
232.8
229.3 -6.2 224.1
93.3 78.3
115.0
60.8 -19, 59.2
Table 4: Out-of-distribution best validation MAE.
Property
Unit
SchNet
DimeNet++
PSEUDo-D
HOMO meV
243.4
230.4 184.2
214.4
LUMO
meV
225.0
175.8
this direction of improving learning strategy instead of improving physics-based representation has potentials.
PSEUDO significantly improves on low-data QM calculations. In Table 3 we investigate how PSEUDo can improve on the low-data regime with only 1%, 10% training data point. That is to do QM using only 1,100 and 11,000 known data. This is to simulate the more expensive QM levels such as CCSD(TYMP2. We observe PSEUDo can consistently and significantly improve the prediction in σHоMO, ØLUMO across both low-data settings and both model backbones, suggesting PSEUDo can help prediction in realistic low-data quantum calculations. Notably, in øuUMO with 1% of QM9 data, PSEUDo improves upon SchNet by 57.8 me a considerable margin. We also observe that the gain margin is much more significant when the number of training data is smaller. This showcases the utility of PSEUDo in extremely low-resource settings, such as QM.
PSEUDO improves aut-of-distribution QM calculations. Another realistic challenge to infer accurately on unseen data distribution away from QM9. We conduct inference on the PC9 dataset where has calculated hoоM., ØLUMO- We also find PSEUDo can again significantly improve OOD accuracy over DimeNet++, SOTA method, with over 16.0 meV improvement ØhOMO and 8 8.4 meV improvement on OlUMO, highlighting the robustness of PSEUDO.
Evidential uncertainty highly correlates to label quality. PSEUDo utilizes uncertainty as a proxy of label quality because they are highly correlated for unseen molecules. In this experiment, we want to validate this hypothesis. We train on the complete QM9 training set with evidential uncertainty and then infer on the QM9 testing set. We find that the non- parammertttic Spearman correlation between MAE and epistemic uncertainty is 0.42 with a p-value < le-16. Additionally, we evaluate on PC9 cott-o--ditributtin set, and the Spearman correlation is 0.35 with p-value < le- suggesting our uncertainty is robust measure of label quality.
0.418
Incressing
Increasing Log Epleeeeei Unceriainty
Uncertainty highly correlates to label quality.
Figure 3:
Ablations. In Table 5 we conduct a systematic ablation study using
SchNet the backbone architecture on the fully supervised QM9 setting. We show that each component in PSEUDo is indispensable for PSEUDo. In Table 2 we have reported original authors best performance following standard practices Kliepera al. (2020): Liu al. (2021). To further clearly demonstrate the utility of pscudo-labeling, in -pseudo- label, we keep all hyperparameters the same but remove the pseudo-labeling part. We show our margin. Next, in the -uncertainty ablation, we use a vanilla per-epoch pseudo-labeling strategy with no uncertainty, corresponding to a vanilla pseudo-labeling strategy. We demonstrate it does not work for QM as it even deteriorates compared to no pseudo-labeling (-pseudo-label), calling for specialized strategy design such as ours. Then, to compared with self telfniraining strategy, -student ablation retrains model in every episode as in (Xie et al. 2020) and the decreased performance shows that it is not ideal in QM calculations. Lastly, the -unifomm ablation uses the same weight for all pseudo-labels with uncertainty reweighting. The decreased performance shows the importance of detection and
strategy improve by large

---

Under review as a conference paper at ICLR 2022
Table 5: Ablation using SchNet as backbone the fully supervised setting.
Property
Unit
PSEUDo
-pseudo-label
-uncertainty
-student
-uniform
meV
32.9
38.9
47.7
41.4
37.2
ELUMO
meV
24.7
27.2
32.1
31.4
28.8
adaptive removal of bad pseudo-labels, achieved by our evidential characterization of the molecular property.
7 CONCLUSION
We introduce PSEUDo, simple, effective, model-agnostic pseudo-labeling strategy that can improve quantum calculations accuracy in abundant data, low data and -if---iiittiittion settings. PSEUDo learns from vast unlabeled data by assigning uncertainty-awart pseudo-labels. These pseudo-labels are adaptively selected to be absorbed into the model via an episodic schedule. Unlike eariler methods in QM that focuses on physics-based representation, show the potential of data-centric approach.
REFERENCES
Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. NeurIPS, 2020.
Brandon Anderson, Truong-Son Hy, and Risi Kondor. networks. NeurIPS, 2019.
Cormorant: Covariant molecular neural
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In ICML, pp. 1613-1622. PMLR, 2015.
Marta Glavatskikh, Jules Leguy, Gilles Hunault, Thomas Cauchy, and Benoit Da Mota. Dataset's chemical diversity limits the generalizability of machine learning predictions. Journal of Chemin- formatics, 11(1):1-15, 2019.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. NeurlPS, 2004.
Yves Grandvalet and Yoshua Bengio. Entropy regularization. Semi-Supervised Learning, pages 151-168, 2006.
Yosuke Higuchi, Niko Moritz, Jonathan Le Roux, and Takaaki Hori. Momentum pseudo-labeling for semi-supervised speech recognition. INTERSPEECH. 2021.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi-supervised learning. In CVPR, pp. 5070-5079, 2019.
Michael Jordan. The exponential family: Conjugate priors. 2009.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? NewrIPS, 2017.
Johannes Klicpera, Janek GroB, and Stephan Günnemann. Directional message passing for molecular graphs. ICLR, 2020.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. NeurIPS, 2017.
Dong-Hi Hyyn Lee al. Pseudo-label: The simple and efficient cemi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning. ICML, volume 3, pp. 896, 2013.
10

---

Under review as a conference paper at ICLR 2022
Tatiana Likhomanenko, Qiantong Xu, Jacob Kahn, Gabriel Synnaeve, and Ronan Collobert. slimipl: Hanguagg-model-freee model-free iterative pseudo-labeling. INTERSPEECH, 2021.
Language-
Yi Liu, Limei Wang, Meng Liu, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d graph networks, arXiv:2102.05011 2021.
Chengqiang Lu, Qi Liu, Chao Wang, Zhenya Huang, Peize Lin, and Lixin He. Molecular property prediction: A multilevel quantum interactions modeling perspective. In AAAI, volume 33, pp. 1052-1060, 2019.
Zhuoran Qiao, Anders S Christensen, Frederick R Manby, Matthew Welborn, Anima Anandkumar, and Thomas F Miller III. Unite: Unitary n-body tensor equivariant network with applications to quantum chemistry. arXiv:2105. 14655, 2021.
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pscudo- labeling: An uncertainly-aware pseudo-label selection framework for semi-supervised learning. ICLR, 2021.
Kristof T Schütt, Pieter-Jan Kindermans, Huziel E Sauceda, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Rohert Müller. Schnet: continuous-fllte convolutional neural network for modeling quantum interactions. NeurlPS, 2017.
Weiwei Shi, Yihong Gong, Chris Ding, Zhiheng MaXiaoyu Tao, and Nanning Zheng. Transductive semi-supervised deep learning using min-max features. In ECCV, pp. 299-315, 2018.
Ava Soleimany, Alexander Amini, Samuel Goldman, Daniela Rus, Sangeeta N Bhatia, and Con- nor W Coley. Evidential deep learning for guided molecular property prediction and discovery. ACS Central Science, 2021.
Oliver T Unke and Markus Meuwly. Physnet: neural network for predicting energies, forces, dipole moments, and partial charges. Journal of Chemical Theory and Computation, 156))33678-3633, 2019.
O. Anatole Lilienfeld and Kieron Burke. Retrospective decade of machine learn- ing for chemical discovery. Nature Communications, 11(1):4895, dec 2020. ISSN 1041-
doi: 04010388441467-000-88855555
URL https:/ nature
com Ccmp:rrrcccee//1116600---166666- wttt:/55556tttttt//0. www.nature com com/articles/ 941467-020-185555
John Watts, Jürgen Gauss, and Rodney Bartlett. Open-shell analytical energy gradients for triple excitation many-body, coupled-cluster methods: Mbpt (4), cesd+ (cosd), ecsd (t), and qcisd (t).
Chemical physics letters, 200((-2)::-2 1992.
Zhenqin WU, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: benchmark for molecular machine learning. Chemical Science, 92):513--33, 2018.
Qizhe Xie, Minh- Than-Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In CVPR, pp. 10687-10698, 2020.

---

Under review as a conference paper at ICLR 2022
A
ENTROPY OF STUDENT'S T-DISTRIBUTION
While the entropy of the Student's I-distribution is well known, we derive it for completeness. Student's probability distribution defined in terms of location 7, scale factor o and Vet degrees of freedom is
r( Mett1
1 (y 7)2
11
ply 7, 2 Ust) = St y, 2a Ust)=
(16)
VU 2 ((
Vst
0
where i gamma function. Student's t-distribution can be written in terms of beta function B [(][[() if we take advantage of the fact that ((22) = V T(++y)
(y - 7)²
p(u: 0 Ust) = olyy; Vvs o² B =1)
1 1+
(17)
Ust
In the main text of the manuscript, we used empirical estimate of Student's t-distribution, which corresponds to evaluation the highest mode y = 7. Empirical estimation of probability becomes:
p emp (y 7;; 2 ust) xz St = 7; ost Mx)
(18)
Vva of B 2 21)
If we introduce variable X=2, probability density
Student's t-distribution converts into the standard form with
Oet
{2
v(t; St(tvvar)
1
(19)
Vie B
Ust
A.1 PROPOSITION
Entropy of the generalized and standard Student's t-distributions are related via the formula
Proposition:
Hly; 02 Mse) = Ht use) + log o.:
(20)
Proof: The transformation t g(v) 2-2 is bijective and invertible with the inverse transformation
d.t
y g (t) = ast + 7. The Jacobin of the transformation g is dy g(v) 1. According to the change of variables probability density formula
a.t"
Py (y: o. v.a) = Pr (g(v):
g(v) J. dy
(21)
The equation for the entropy transformation (equation 20 follows directly from the definition of the entropy.
To find the generalized entropy, we just need to calculate the entropy of the standard Student's s i-distribution
H((;va))= = D(t: log
(22)
Ust
log
Vusi B
v(t; dt +
(23)
2'
1/t 1
v(t: dt =log B
BC Vot
log(1 +
+
(24)
2
-x0
Uat
Vst + 1
400
t²
log(1
+
p(t; use) dt.
(25)
2
Ust
12

---

Under review as a conference paper at ICLR 2022
To find the second integral, we make substitution
and obtain
Pat
Mst + 1
1² v(t:
2att dx
log(1
dt
log + x) (1 x)
2
2
Vst
2 BB
Mes
2
V
(26)
Vst
(1+x)
Me
dr
Vet
Ovst Jo
Met x
(1 x)
dx
B(} 2
Ovst.
Vx
B
(27)
Vst +1 0
Vst
Vst
B
= (Vse + 1)
log B
(28)
B(2 "re) Ovat
2 2
Ovat
2
2
log [ Met log r( Mst + 1)
Mst +1
( V Vst -
V (2
-Uuse + 1) Øtst
(29)
2
where digamma function is defined as (x) standard Student's 's t-distribution becomes
(x) T(x)
Putting all the terms together, the entropy of
H((: Me) = Vst +
V
Vst + 1
v
+ log
VVss B 2 2
(30)
2
2
2
The final formula for the entropy of the labels y is given by
Hly 22 Uoe) = 1st +1
Ust + v (1)) log
(v _
+ log 0 (31)
2
2
B PSEUDO-LABELING, ENTROPY MINIMIZATION AND ALEATORIC UNCERTAINTY
In the main section the text, we have considered the case where observed targets (. .,4.) ~ Nuu, a²) are drawn from the Normal distribution with unknown mean and variance μ and o² and we have imposed prior on them. The problem is significantly simplified if we treat μ and o2 in a deterministic way, such that model outputs two parameters f and o². The model here is able to estimate aleatoric (data) uncertainty o² but unable to model epistemic (model) uncertainty. By minimizing negative log-likelihood, the loss is significantly simpler than in Eq.
C. lo ps of) o
(1i p,)
(32)
20
Empirical estimate of the entropy the unlabeled data set becomes
H()lu)=
Σ Ey-po(vlx;))
xx) l-1g pe (y xx)]
Σ
S:m [log (y))))
(33)
X.EU
X.EU
One can notice that the weights are inversely related to with log probability weights E. 2
aleatoric uncertainties & ~ (ot)
13


Reviews:
[{"review_id": "e_APFRHGRJD", "reviewer": "Reviewer_ianT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**strengths**\n* Label scarcity is an important problem when using ML/DL to train models for molecular property prediction, especially when the goal is to get to similar accuracies such as more expensive computational methods such as coupled cluster methods. \n*The empirical results seem encouraging.\n\n**weaknesses**\n* The method is not explained well. The paper can be significantly improved in this aspect. For example:\n\t-  The paper contains pseudocode to explain the algorithm, but there are issues with notation, both in this algorithm and in the main text. \n\t\t* algorithm 1: isn't there a line missing where $f^{(k)}$ is set equal to $f^{(k-1)}$ ? \n\t\t* algorithm 1: In the inner loop over the dataset $\\mathcal T$, the index i is used as a subscript to the update on the neural network function. However, this index is never updated and comes implicitly from the subscript of the datapoint.  I don't understand why the neural network function f needs a subscript here.\n\t\t* algorithm 1: In the inner loop over the dataset $\\mathcal T$,  the $f^{(k-1)}$ model is used to generate predictions, uncertainties and it is updated. Shouldn't the uncertainties come from the teacher model that was used to generate the pseudo labels? \n\t\t* algorithm1: the distinction between the model that produces the pseudolabel and the model that optimizes a loss based on its own predictions and the pseudolabels is not clear. \n\t\n\t- The paragraph on \"Evidential uncertainty quantification\" on page 4 contains many confusing statements and notations. \n\t\t* just below eq 1 is the statement \"dependence of the variance of the distribution of labels and variance for the parameter $\\mu$ allows us to factorize the posterior for$\\mu$ and $\\sigma$ independently.\" But in eq 1 the normal distribution for $\\mu$ depends on $\\sigma$, so it is not factorized independently , but rather p(mu, sigma) = p(mu|sigma)p(sigma). \n\t\t* eq 2 shows the mean of  $\\sigma^2$ as the mean of the inverse gamma distribution. However, in eq 1 $\\sigma$ is distributed according to the inverse gamma distribution, not $\\sigma^2$. \n\t\t* Where in algorithm 1  is the loss in eq 4 used? \n\t- Notation is inconsistent or unclear:\n\t\t* At end of section 3, page 3: f is used to denote the model that generates pseudo labels, as well as for the model that is trained using ground truth labels and pseudolabels. In other words it is used for both the teacher and student model. This gets very confusing for the reader. \n         * The unlabeled dataset is defined as $\\mathcal X_U$ at the end of section 3, but in section 4 it is denoted with $\\mathcal U$. Datapoints in the unlabeled dataset are sometimes denoted with $x_i$ and sometimes with $\\mathcal U_i$ and sometimes with $\\mathcal X_i$. \n\n\n* Section 5 attempts to derive a motivation for the weighting of datapoints depending on the confidence of the model's pseudolabels. However, I found this section very unclear.\n\t* I don't see how the middle plot in figure 2 shows that minimizing the evidential loss can minimize the entropy. A lower total evidence in that plot leads to a higher entropy. \n\t* why are $\\beta=1$ and $\\nu=1$ representative parameters in figure 2.? What is the motivation for this particular choice? How does the plot change if you change these values?\n\t* what does the y axis label on the right plot of figure 2,$p(y=y)$, mean? \n\t*  At the top of page 6 the authors state \"intuitively, we show that minimizing evidential loss can implicitly minimize entropy on the unlabeled data such that it encourages positive information transfer.\" What is the intuition behind minimizing the entropy of the unlabeled data predictions. That would mean you want your model to become confident on unlabeled data? I don't see why that is desirable. \n\t* The discussion and derivation around eq 10, 11 and 12 is very confusing. For eq 10, I don't see what expectation maximization has to do with using a different distribution to sample from compared to the distribution that appears inside the expectation. I don't see how eq 11 is a valid empirical estimate of the expectation in eq 10.  eq. 11 is missing a minus sign. Where does the student distribution come from in eq 12?\n   * top of page 6: states that the second term in eq 4 is the total evidence. The second term contains 2v+a. But in the x-asis of plot figure 2, we see 2v + a. which one is correct?\n\n* It would be useful to comment on the difference between molecules described in QM9 and PC9, so the reader can get a better sense of how hard it is to generalize across these two datasets and how \"out of distribution\" molecules of PC9 are compared to those in the QM9 dataset.\n* Comments on experiments:\n    - what is the explanation for why using pseudolabels deteriorates performance on the $\\epsilon_{homo}$ prediction for SchNet in table 3 for 1% labeled data?\n    -  Although I appreciate the fact that the authors did an ablation study, several things are unclear. What is the difference between -pseudolabels and just a SchNet model?  The -uncertainty and -student baselines perform worse than the SchNet baseline (MAE 41 in table 2) on the $\\epsilon_{homo}$ prediction. How come?\n\n**Minor comments/questions**:\n* CCSD and MP2 acronyms are not defined in introduction.\n* In the introduction the acronym DFT is used before it is defined.\n* SchNet is not accompanied by a reference at the end of paragraph 2 in the introduction.\n* please explain what gold labels means early in the manuscript.", "summary_of_the_paper": "This paper proposes to use pseudo-labeling/self-training for the prediction of molecular properties which are traditionally computed with quantum mechanical chemistry methods such as density functional theory or coupled cluster methods. While several recent works have trained supervised models on the QM9 dataset with labels produced by density function theory, labeled data for more accurate methods such as coupled cluster methods are significantly more expensive to obtain. This paper proposes to approach this label scarcity issue with pseudo-labeling/self-training on unlabeled data. The quality of the pseudo-labels is taken into account as follows: datapoints for which the model produces a pseudolabel with large uncertainty are downweighted compared to datapoints for which the model has high confidence on the pseudolabel. \n\nThe proposed training strategy is used to train models on different versions of QM9 and PC9. The first case considers training on all the labeled data in the training set of QM9, and the data of PC9 is used as unlabeled data. The second case considers only the QM9 dataset, where only 1% of the labels are used (and the rest is used as unlabeled data), or 10% of the labels are used.  \n\nEmpirically the authors observe that using additional unlabeled data with pseudolabels improves the performance of the baseline models, even when only a small fraction of ground truth labels are available. \nThe authors check their hypothesis on the test set of QM9 that the model's uncertainty is well calibrated, in the sense that MAE and the epistemic uncertainty are positively correlated. \n", "summary_of_the_review": "On the positive side, the topic is interesting and timely and I expect this could be of interest to the research community. Furthermore, the empirical results are encouraging. However, as explained above, the paper could be greatly improved in terms of explaining the method. Furthermore, some sections (such as section 5) contain problematic derivations and inconsistent notation. The ablation study is not explained well. At this point I think the weaknesses outweigh the positives, so I recommend rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PapZcY6qtlf", "reviewer": "Reviewer_iJ3j", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strong points**\n\n- At first time, authors adapt and study pseudo-labeling training for quantum calculations problem which has its own challenges and motivations\n- Authors propose to use evidential uncertainty to estimate uncertainty of the pseudo-labels and demonstrate that proposed adaptive weighting mechanism could be viewed as parts of the entropy minimization framework (or entropy minimization implicitly uses adaptive weighting)\n- Demonstrating that algorithm works for different model backbones (SchNet and DimeNet++)\n- In speech and image recognition it was demonstrated that the key components of iterative pseudo-labeling success are model noise (e.g. dropout) and data augmentation. This paper demonstrates that even with no noise (neither in data nor in model) pseudo-labeling can give significant performance boost.\n\n\n**Weak points**\n- Experiment for out of distribution testing is not enough. Ideal test would be evaluating trained models on some other dataset different from labeled and unlabeled data. One suggestion here (if no any other data is available) do the following: for low data 1% and 10% test also on PC9 (as it was not used in training at all) for all models; add one more experiment with 55k QM9 labeled + 55k rest QM9 as unlabeled + test on out of domain PC9. \n- Would be nice to have also upper bound on out of distribution testing: finetune DimeNet++ baseline on PC9 labeled data with evaluating on QM9 and on some (not used) portion of PC9 - this is upper bound on what we can get with QM9 labeled and PC9 unlabeled.\n- Fair “-uncertainty” ablation should follow exactly the same training except the weighting data, so that regenerating pseudo-labels should happen after N epochs. (Current “-uncertainty” ablation is proving another point that we should have some delay and enough training before regenerating pseudo-labels. However,  N is hyperparameter which I guess depends on the problem and data amount).\n- Absense of ablation for low-resource setting as it could give entirely different dynamics of training compared to high-resource (at least this was observed in speech recognition).\n\n**Related work**\n- In speech recognition recently there were developed pseudo-labeled algorithms which continuously trains one model with regenerating pseudo-labels after several epochs https://arxiv.org/abs/2005.09267 and moreover using dynamic cache with pseudo-labels generated by some history model states https://arxiv.org/abs/2010.11524. These works are relevant in the way how authors performs iterative training with continuous model training. Moreover, the latter work (https://arxiv.org/abs/2010.11524) focuses on the low-resource setting of labeled data and solves problem of stable model training in low-resource. Another relevant work, also in speech recognition, is https://arxiv.org/abs/2106.08922: it uses exponential moving averaging of the model to predict pseudo-labels. \n- Other relevant works https://arxiv.org/pdf/1908.02983.pdf, https://openreview.net/forum?id=SJgdnAVKDH\n- It is worth to mention works related to modeling labels probabilistically / Bayesian modeling\n\n**Comments**\n- Algorithm 1: it would be clear to remove index $i$ for $f$, put $f^{k} = f^{(k-1)}$ in the beginning of the inner loop, simply put $Update(f^{k}, L)$. In any case Algorithm 1 needs improvements as right now indices are not clear.\n- Noise strategy is not only on data side - it could be enough to have noise on model side via dropout, stochastic depths, etc. It is worth to mention that models for QM doesn’t not use these widespread techniques, so that main challenge is to introduce some noise into the training process. The latter is done is some extent with introduced uncertainty used to weight samples. Is there any possibility to introduce some noise in the model itself? One of the way is to try to use dynamic cache as in https://arxiv.org/abs/2010.11524 or EMA as in  https://arxiv.org/abs/2106.08922.\n- In Figure 1 better to use \"$K$ epochs\" as $N$ is reserved for $N$ molecules\n- \"In the standard pseudo-labeling (Lee et al., 2013; Rizve et al., 2021), a set of labels are regenerated in every epoch.\" In (Lee et al., 2013) it is done every update, not epoch and the same model is continuously trained while in (Rizve et al., 2021) it is done after full model training, so some $K$ epochs of training, then new pseudo-labels are generated, model is reinitialized and trained from scratch on these new pseudo-labels. \n- Did authors check this statement in their experiments? \"Our key observation is that low-quality pseudo-labels have high model uncertainty, and high-quality pseudo-labels have low model uncertainty.\"\n- \"it removes a large set of unlabeled data, which reduces the diversity of the training space, harming the generalization.\" I do not agree with this statement: In (Rizve et al., 2021) the whole training continues until there will be no uncertain sample anymore, so in general at first pseudo-labeling rounds small portion of unlabeled data is used while at later rounds more data are used as model becomes stronger and more certain, thus diversity and generalization can be still hold. Maybe it is worth to smooth authors' formulation in the way that i) we can learn from noisy data starting right from the begging to extract necessary representations even if model is uncertain.\n- How do authors weight supervised and unsupervised data? Or it is done via Eq. (5) which means use the proportion as it is in the original datasets? Do authors randomly sample batch from joint set of supervised and unsupervised data?\n- Appendix: the Eq. (22) the right hand side should have additional 2 multiplier as variable change $x=t/\\nu_{st}$ is done incorrectly (first you need to write left side as $\\int_{-\\inf}^{+\\inf} ... = 2*\\int_0^{+\\inf} ...$ and then do variable change. This multiplier 2 should be propagated further till formula (28). Next, (26) and (27) are repetitions, so authors can skip (27). Everything else in Appendix is correct.\n- Is there any connection between QM and assumption that $y_i\\sim N(\\mu, \\sigma^2)$?\n- To recheck: in Table 2 Pseudo$\\sigma$-S has the exact same results as PhysNet. Is it correct?\n- Any idea why for 1% train data for HOMO pseudo-labeling doesn’t work with SchNet? Does model diverge as soon as you start involving pseudo-labeled data?\n- Could authors add a small paragraph in experiments on what is the difference between QM9 and PC9 data?\n- Would be great to see the plot of average error and average uncertainty for different episodes, to demonstrate that both error and uncertainty are decreasing over the episodes.\n- Why in Table 5 “-pseudo-label” is not the same as baseline SchNet in Table 2? If authors did optimization of hyperparameters for pseud$\\sigma$ and we see that found hyperparameters even better for baselines - I think it is necessary to put retrained baselines also in Tables to have fair comparison of pseudo-labeling on top of baseline. Ideal case is that I have some baseline and following the paper simply add pseudo-labeling on top and I don’t need to redo optimization of the model hyperparameters again, and only deal with pseudo-labeling hyperparamenters).\n- Does “-uncertainty” ablation diverge actually? Could authors show the training/validation curves? (as quality is worth that even baseline) \n- For “-student” do authors train every episode till full convergence and then regenerate pseudo-labels and retrain from scratch?\n- Please specify that ablations are done on full QM9 (this is obvious from MAE numbers, but would be good to have in the main text).\n- When are first pseudo-labels generated? As model fully converged on supervised data?\n- \"This is important because we observe that per-epoch pseudo-labeling is far from converging, preventing the model from learning from all useful information in pseudo-labels in one episode.” Worse behaviour with per-epoch pseudo-labeling could be due to absence of any data/model augmentation which makes model to be more close to the model state with which pseudo-labels were generated after one epoch. This prevents from training. For future work, I would suggest authors to have a look on the ways to introduce some shift in the model (EMA or dynamic cache, etc.) which can help for convergence and bootstrapping from the model itself. \n- Could authors provide more details on “For each episode, we also reinitialize the learning rate with a small step-wise decay strategy to allow the model a chance to jump out of the local optimum from the previous set of pseudo-labels.” Is it used in ablations too? Could authors report results without this trick? Is it used for both low-resource and high-resource experiments?\n\n**Typos**\n- Abstract typo: \"pseud-labels\" -> \"pseudo-labels\"\n- Would be good for reader not familiar to quantum mechanics to specify abbreviations, e.g. CCSD(T) and MP2, also link to SchNet.\n- Typo \"in QM9 to test and validation model performance\" -> \"in QM9 to test and validate model performance\"\n- typo \"at episode k then conduct inference\" -> \"at episode k then conducts inference\"\n- \"$f (k + 1)$ after N epochs\" -> \"$f^{(k + 1)}$ after N epochs”\n- \"Dependence of the variance for the distribution of labels and variance for the parameter” - not clear English.\nEq (11) the right hand side: should it be $y_i^{t-1}$ -> $\\gamma_i^{t-1}$\nEq. (9) $\\sigma_i$ -> $\\sigma_{st,i}$\n", "summary_of_the_paper": "Self-training, or pseudo-labeling, is very simple and popular approach in different domains like speech and image recognition. \nFor the first time, pseudo-labeling is designed for quantum mechanics (QM) calculations problem. This problem is challenging for pseudo-labeling as it cannot use data augmentation (\"small perturbational noise in 3D molecular geometry could easily lead to a drastic energy difference\") or model noise (like dropout or layer drop due to design of models to work with 3D molecular geometry) which were found to be crucial for pseudo-labeling in other domains. Moreover, the main real case scenario for QM is small amount of supervised data available (low-resource) which makes pseudo-labeling is even harder. In this paper authors propose learning a single model with time-to-time regenerating pseudo-labels and incorporating these pseudo-labeled data according to the model uncertainty. The uncertainty is modeled in the following way: labels are assumed to be from normal distribution and model is designed to predict parameters of this normal distribution, thus model estimates expectation of the label and its dispersion. This uncertainty detects and prevents bad pseudo-label from affecting the model training by adaptively weighting the data according to inverse of uncertainty value. Pseudo-labeling results show that direction of improving learning strategy instead of improving physics-based representation has potentials. With experiments it is demonstrated that proposed pseudo-labeling algorithm, pseud$\\sigma$, improves results over supervised baseline, works across different backbone models and applicable for low-resource regime. Moreover it demonstrates improved results for out of distribution data.", "summary_of_the_review": "Overall authors uses classic semi-supervised algorithm, pseudo-labeling, but they propose necessary modifications like specific learning schedule, similar idea from other domains on bootstrapping a single model, and, the most important, regularizing training with uncertainty estimation for pseudo-labels and proper accounting it during optimization. All these modifications allowed pseudo-labeling to succeed to improve results for quantum mechanics (QM) calculations across different models, labeled data size and out of distribution data. All these modifications make a novel version of pseudo-labeling which is applied to the QM domain at first time and specifically designed for QM. Having all this in mind I recommend to accept the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Gqy5g0WGSX0", "reviewer": "Reviewer_5bAg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n(1) The problem on QM is well-motivated and the paper is well written.\n\n(2) The method part is well explained, even for readers like me who are not working on the self-training problems.\n\n(3) From the technical point of view, this PseudSigma algorithm is interesting and promising. I expect it can be generalized to more general problems, in addition to the QM tasks.\n\n(4) Almost consistent performance improvements can support PseudSigma.\n\n-----\n\nWeaknesses:\n\nI mainly have two concerns (first two), and the remaining ones are just minor points.\n\n(1) There is a gap in why using PseudSigma on QM tasks.\n1. Why this assumption? First, the authors claim that PseudSigma is simple, effective, and model-agnostic (termed SEM for short). However, in Sec 4, the prior comes out abruptly. Similar for Sec 5 Eq 8, the Student-t distribution comes out abruptly. Can authors provide more intuitions of this assumption? And how is this formulation connected to SEM? One point I can imply from this paper is maybe due to the analytical solution? If so, authors should point this out explicitly. Otherwise, authors can provide different view points?\n2. Why QM tasks? Both Sec 4 and Sec 5 are not related to this, i.e., the tasks here can be quite arbitrary. Thus, to better verify the effectiveness of PseudSigma, it is more reasonable to move to the standard self-training research line, with more standard self-training baselines. (This is the second point below.)\n\n(2) Lack of self-training baselines.\nAuthors give a quite comprehensive introduction on self-training in Sec 2, and some are quite interesting, yet none of them is included in Sec 6. For example, `the perturbational noise in 3D conformation can lead to drastic energy differences`. This is a big difference between other applications and QM tasks, and can better reflect the importance of PseudSigma. The authors can consider adding it into baselines.\n\n\n(3) In Sec 5, the authors claim that `molecule properties are not modeled probabilistically ...`, which I cannot fully agree with. So for regression QM tasks, MSE/MAE is indeed modeling the y under Gaussian/Laplacian distribution.\n\n(4) In Sec 2, Pseudo-labeling paragraph, there are two redundant citations (Xie et al.) in one sentence. The authors can remove the second.\n\n(5) In Sec 3, the input X also contains the atom types.\n", "summary_of_the_paper": "This paper starts from two real challenges in QM problems: OOD and low-data issues. Then it proposes a self-training methods called PseudSigma. PseudSigma is a simple, effective, and model-agnostic algorithm with robust empirical improvements.", "summary_of_the_review": "Both the strengths and weaknesses of this paper are obvious.\n\n- From the technical point of view, the PseudSigma is promising.\n- From the story telling point of view, it has some gaps, especially about how PseudSigma is connected to QM tasks.\n\nI would give a borderline rate for now, and see what are authors response during reubttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "5MtjvMsFMtW", "reviewer": "Reviewer_zDAo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Empirical results show improvements over competing methods\n- Connection between evidential uncertainty and entropy minimization is interesting\n\nWeaknesses:\n- Contribution of the work is not clearly stated\n- The technical novelty of the method is very limited: 1) Instead of regenerating pseudo-labels every epoch, they are regenerated less often (after an \"episode\": a set number of epochs). 2) Weighing the pseudo-label contribution to training loss by its uncertainty\n\n\nMinor comments/edits:\n- literature review (e.g. 2nd paragraph) needs references\n- \"concurrent work (Rizve et al., 2021)\" A method from ICLR'21 is hardly \"concurrent work\".", "summary_of_the_paper": "The paper is focused on improving quantum-mechanical properties of molecules in a setting similar to semi-supervised learning, where X is known for samples without Y. In an iterative fashion, the current model is used to predict pseudo-labels for those samples, then pseudo-labels together with labeled samples are used to improve the model. The paper uses existing approaches to quantify the uncertainty in individual pseudo-labels and then uses the uncertainty to weight their contribution to the loss.", "summary_of_the_review": "The paper provides empirical evidence of improvement over competing methods, but the method is technically not very innovative.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were split on this paper: the positive review appreciated (a) how adaptive weighing can be viewed as part of energy minimization, (b) the flexibility of the model to work with different model backbones, (c) the demonstration that even in no-noise settings the method generates noticeable improvements. However, all reviews saw important shortcomings in the (a) few out-of-distribution results, (b) limited ablation studies, (c) clarity of the writing, particularly in notation, (d) explanations of experimental results (e.g., why using pseudolabels sometimes deteriorates performance), (e) assumptions behind the proposed method, (f) lack of self-training baselines, (g) limited technical novelty. Ultimately, the number and severity of the shortcomings outweigh the positive parts of the paper. If the authors take the reviewer’s recommendations into account the paper will be a much stronger submission.
Decision: Reject