Peer review of: Identifying domains of applicability of machine learning models for materials science (2019)

Reviewers' comments: 
 
Reviewer #1 (Remarks to the Author): 
 
The manuscript describes a methodology to improve machine learning predictions of materials 
properties. The idea is to identify regions in feature space where the predictions provided by the 
machine are not reliable, and discard those predictions. The authors test this approach on three 
entries of a recent Kaggle competition and demonstrate that indeed prediction errors go down 
considerably. 
 
In my opinion, the results are not unexpected: if one has a prediction of the uncertainty of a result on 
a large data set, and exclude those where the uncertainties are large, the variance must go down, 
provided the uncertainty estimation was reasonable. I also think there are plenty of other established 
ways to estimate uncertainty in machine learning predictions, for example query by committee or 
other variance estimators. One would expect these to work equally well in this context. 
 
I am also not convinced by the usefulness of this approach (whether it is based on subgroup discovery 
or by uncertainty estimates) other than identifying shortcomings of the representations and/or the 
machine learning approach - which should really be fixed rather than dodged. Often the reasoning for 
a high-throughput machine learning approach like the one simulated in the Kaggle competition is to 
find a needle in a haystack. If we discard a good portion of the haystack, we have just lowered the 
chances of finding the needle. Even worse, one could expect the out-of-ordinary elements to be the 
interesting ones from the scientific point of view. 
 
I find it is a valid argument that this approach can be used to identify weaknesses in descriptors and 
improve them - but I think this may better belong to a more specialised journal. I am not sure yet 
another paper chasing lower mean absolute energy errors is of broad enough interest for the audience 
of Nature Communications. 
 
 
Reviewer #2 (Remarks to the Author): 
 
The paper presents an application of subgroup discovery to several machine learned models which 
predict crystal formation energies. For those datasets, the authors shows significant reductions in MAE 
when the predictions are constrained to sizable fractions of the original data. 
This work should be of interest to the field because it demonstrates the application of a technique that 
I have not seen anyone use in predictive ML for materials and could be relevant to almost any such 
application. 
 
Broadly, I have no substantial concerns with the paper. I have a few suggestions for presentation, but 
these are all minor/easy to fix (see below). 
 
The statistical analysis and machine learning process is appropriate and relativel y standard. Because 
of the large number of details in inherent in a study like this, I hope the authors will be releasing code 
alongside the publication. 
 
The only unusual technical point is the mismatch in loss functions. 
* KRR uses an L2 norm 
* The analysis focuses on MAE (an L1 norm) 
* The SGD uses a relative error loss function 
Any of these is fine, but it’s unusual to see this. Perhaps the authors could explain why they made this 
choice? 

This comes up specifically for “the optimal prediction at each composition of the test set (the median 
energy)”. This is true that median is L1 optimal, but it’s not L2 optimal (which is what the training 
algorithm is optimizing) 
 
Minor points 
* "also to sample novel points from them" -- This statement is not true. Just because you can describe 
a sub-domain does not mean it is easy to sample (unless you are sampling from an enumerated set). 
Consider if the DA has said “PBE band gap energy > x” — finding such a material may be hard. 
* In the section on Jaccard similarity, there is a reference to Figure S1 but Figure S1 does not show 
what is referenced here (I think you left a figure out of supplemental) 
* Table S2 contains the key results from the paper and shouldn’t be relegated to supplemental! I think 
they could be relatively easily incorporated into Figure 3 if you are trying to avoid another main text 
figure/table. 
* in the deeper dive in the SOAP DA model, you should probably reference Figures S1 and S2 for the 
other models. 
* Captions for Figures S1 and S2 should describe what the histograms are. I think I know, but don’t 
make me guess. 
 
Patrick Riley 
Google Accelerated Science 
 
 
Reviewer #3 (Remarks to the Author): 
 
The manuscript at hand presents a novel approach to discover the applicability domain of trained 
machine learning (ML) models on the materials data sets. This approach is based on the previously 
presented method of sub-group discovery, which is equivalent to unsupervised ML techniques. The 
approach is tested on a dummy synthetic data and a small materials data set of formation energies of 
transparent conducting oxides. The final goal is to identify the domain of applicability (DA) of three 
kernel-based ML models. As it’s clearly stated, the challenge of DA in any ML research is timely and of 
interest. Thus, I believe the approach is relevant and arguably new. The text is also well written and 
it’s easy to understand. It conveys a generally acceptable ML workflow for training and validation. 
However, the discussions in the paper are mainly focused on the application of the DA approach as a 
fair metric to compare models and feature representations. I believe a general approach to DA is way 
more important than its application in comparing methods on its basis. There are a few recent 
research papers that try to address this issue from different aspects: 
- Kailkhura et. al., Reliable and Explainable Machine Learning Methods for Accelerated Material 
Discovery, Arxiv 2019 
- Haghighatlari et. al., Thinking Globally, Acting Locally: On the Issue of Training Set Imbalance and 
the Case for Local Machine Learning Models in Chemistry, ChemRxiv 2019 
 
Since it’s difficult to provide benchmark data for unsupervised techniques, these references can be 
used to evaluate the generalizability of the proposed approach. Considering the novelty of the work, I 
recommend the following major concerns and comments be addressed before any potential publication: 
 
1) It seems that each model is trained only once (on one of the training folds) and the SGD-DA 
approach will be applied to the errors of the model and splits of the test set. It is not clear that the 
prediction of the test set by ML model is required to validate the DA or not? If yes please also 
comment on the computational cost for each of the three feature representation methods. 
 2) The follow-up comment is on the optimization algorithm in the SGD method. I understand the 
objective function is explained here, but it seems that the core of the work (optimization algorithm) 
has been left out based upon the previously developed, SGD, technique. However, authors spend a 
generous amount of space to explain feature representations that are developed by other researchers. 
 
3) One of the main concerns is regarding the limited number of features that are used as DA selectors. 
Although authors justify the neutrality and interpretability of the features, there is a place of concern 
for the correlation between these features and actual features that have been used for the training. 
This can be discussed from three different angles: 
a) The choice of selectors may be biased to particular domain knowledge and perform in favor of one 
of the descriptors. This may be a valid point from a chemist point of view, but I believe it’s not really 
fair for the sake of comparison. One can choose selectors that are more correlated with their choice of 
feature representation. 
b) Although the actual features may not be chemically interpretable, they are the same features that 
one needs to calculate before using the model to predict energies. Thus, it sounds Ok to identify DA 
based on the same features. 
c) The number of descriptors in each of the methods are not same. MBTR provides at least two times 
more descriptors than the other two (1400 vs 600). This is way more amount of information extracted 
to be interpreted in a small subgroup and against a limited number of selectors. 
 
4) The other main concern is about the interpretation of the results. As mentioned in the text, the 
coverage of the MBTR is almost half of the SOAP model/representation (%44 vs %76). Thus, the 
MBTR model is applicable in a subspace of the data, which is half of the size of coverage by SOAP. 
Thus the DA of the MBTR is too restricted. I think a higher coverage of the training data is more 
desirable for an ML model with acceptable performance. I suspect if we shrink the subspace of the 
SOAP model elaborately (e.g., from %76 to %50), the number of outliers and thus the subgroup error 
will also shrink. 
 
In addition, there are a number of minor issues that should be addressed before the paper is accepted: 
 
1) on page 6, line 2: impact needs to be italic? 
2) on page 9: Figure2, in the caption: the standard deviation based on the mathematical expression is 
`epsilon` not `s` 
3) Please comment on the accessibility of the codes? or any GitHub repository for SGD/DA method?