Peer review of: Machine learning accurate exchange and correlation functionals of the electronic density (2019)

Reviewers' comments: 
 
Reviewer #1 (Remarks to the Author): 
 
Dick and Fernandez-Serra discuss the development of density functionals based on supervised machine 
learning by using CCSD(T) energies as reference. Their aim is to provide a highly-accurate density (and 
energy) at the speed of a calculation performed with a GGA functional. In this manuscript, they present an 
interesting method that uses atomic orbitals as feature space of the input electron density. A neural network 
is then trained that can predict a new, more accurate density. However, the method has already been 
introduced in a previous publication, while the two manuscripts share many similarities, including test 
examples and figures. In addition, the authors mention in every section (including the title) the word 
“accuracy”, but their results do not fully reflect that statement. For those reasons, I cannot recommend this 
article for publication in Nature Communications. 
 
Technical comments: 
 
- The title is misleading. Reference data from CCSD(T) obtained with a triple-zeta quality basis set should 
not be called as “accurate”. There might be more accurate than GGA functionals, but in order to obtain the 
expected accuracy of post-HF methods and especially coupled-cluster methods, a complete basis set is 
needed (basis set extrapolation, explicitly correlated methods). 
- I do not understand why the authors are trying to map a GGA density generated with planewaves and 
pseudopotentials to data from cluster CCSD(T) calculations. Since all calculations have been performed for 
molecular species, why they do not use a cluster DFT code? 
- The computational details for the CCSD(T) calculations are pretty messy. No quantum chemical code is 
mentioned for the generation of reference data for the sGCML and MB-pol. In addition, using different 
reference data from different codes and approximations (eg. density fitting), makes the analysis of the 
results imbalanced. 
- Efforts to satisfy invariance by data augmentation is inefficient. 
- In the section “data-efficiency”, I don’t understand where the efficiency is if you need a dataset of at least 
500 samples to reach 1 kcal/mol (malonaldehyde). 
- Abstract and introduction mention that density functionals have been developed, so I was expecting one 
unified functional as the outcome of the ML methodology. On the contrary, the manuscript describes the 
development of case-specific functionals. 
- The example of cats and dogs should be removed. 
- Why the authors decide to freeze the previous layers when they add a new layer? The NN that have been 
used are relatively small. 
- I wouldn’t call the improvement of electron densities as “remarkable” (page 9). Figure 9 shows that the 
model fails to provide reasonable densities for specific cases (eg ethanol, water). 
- Overall, a model that claims that can represent the density based on electronic structure descriptors and 
“learn” it from CCSD(T) data should be transferable between atoms with the same size and the same 
number and type of atomic orbitals (same angular momenta). The authors generated higher expectations in 
their introduction and fail to present that novelty in their results sections. 
 
 
 
Reviewer #2 (Remarks to the Author): 
 
Review comments for authors 
 
This paper represents a novel approach for creating density functionals using machine learning techniques, 
and so should be published, possibly in Nature Communications. The paper shows promising results in terms 
of transferability and accuracy for several very different systems. But the presentation needs considerable 
revision before a final decision on importance can be made. 
 
• Lots of essential information on the ML part of this paper are missing. The training procedure is totally 
unclear. For example, what is the loss function being minimized? Is it some combination of density and 
energy? Or “cross validation” has determined rho to lower generalization errors – what procedure exactly? 
The authors must carefully describe all these details, in supplemental info if not in the main text. Even the 
values of r_0 in Eq(1) do not seem to have been reported for each of the functionals used. 
• Almost no information is given about computational cost of the method (not cost of data) and how it scales 
with number of training data. 
• The reference to github for supplemental data is insufficient, there should be a DOI for the codes, etc. (see 
nature guidelines), guaranteeing no changes occur. 
• Figure 2 should have been plotted with the usual log scale on the y-axis, without which it is not very 
useful. 
• Figure 3 is hard to comprehend. According to the caption, the target corrections range over about +- 
0.06eV, which is about the same size as the required accuracy (43 meV). So inserting NO correction at all 
would almost pass this test? This implies the PBE results alone are almost at chemical accuracy for these 
molecules? What am I missing? (first row of tab 1 seems to say PBE errors are about 35 meV…) 
• Fig 4 seems to have been plotted with very few r/r0 values as one can see straightline segments. There 
should be more plots, plus supp info listing the numbers and plotting of errors relative to s66x8. 
• A recent JCP by Paesani and co-workers discusses the decomposition of errors of different functionals. The 
errors should also be compared with the errors of better functionals than PBE (see Chem. Sci., 2019,10, 
8211-8218) 
• In Fig 5, the authors even state that their results should not be compared directly to experiment. So they 
should be compared to MB-pol (without quantum dynamics) to show NeuralXC having much smaller errors 
than PBE. Again, comparison with a better functional for water should also be included. 
• In Fig 6, why is NeuralXC compared to PBE, and not to CCSD(T)? It is almost impossible to judge, with the 
present pictures, if NeuralXC is doing a significantly better job for the density. 
• In several places, the need for methods to work in the thermodynamic limit is mentioned, but none of the 
examples seem to show this. Either this should be demonstrated or removed. 
 
 
 
Reviewer #3 (Remarks to the Author): 
 
The paper presents a further step forward in the development of machine-learnt exchange-correlation 
functionals. The authors build on previously successful ideas such as descriptors based on a basis 
decomposition of the density and Behler-type neural networks to produce NN-based XC functionals which in 
certain cases achieve beyond chemical accuracy. Self-consistent calculations are performed with the fitted 
functionals, which is a strong point of the paper. ML functionals are now actively researched, successes in 
this field and eventual deployment of ML functionals in end-user codes would be a big step forward for 
computational chemistry and computational materials science. For this reason, the paper belongs in the 
literature, and I would recommend eventual acceptance after a revision that needs to address the following: 
 
- The sentence "In this manuscript, we propose a pathway to construct fully machine-learned functionals 
that depend explicitly on the electronic density." could be misleading, as what is done is actually delta-
learning and the functional does not depend only on the density but also on atomic positions. This should be 
highlighted early on as this provides a ML algorithm major help which can be decisive e.g. it is atomic 
position information that allowed previous NN works, albeit on KEF, to get smooth potential curves (CPL 734 
(2019) 136732). 
 
- Details of NNs should be given explicitly. How do the results, and especially transferability, depend on the 
details of atomic NN architectures? (nos. of layers and neurons and their type). It was previously reported 
that NN architecture can critically influence transferability (PCCP 21 (2019) 378), so this is potentially an 
important issue and should be looked into. 
 
- Was SIESTA DZP basis default or tuned? Default basis functions in SIESTA can be very bad, I wonder if 
that could color the results.