Peer review of: Machine learning accurate exchange and correlation functionals of the electronic density (2019)


RR2
Reviewers' comments:
Reviewer #1 (Remarks to the Author):
  The authors have responded to the comments raised by all reviewers. Most of their responses are
 satisfactory. However, there are still a few points that are not clear or they need additional attention by the
 authors:
 - I disagree with the argument that “different atomic species have different (basis) sizes”. For a given basis
 (eg. cc-pVTZ), the number of basis functions is constant for atoms that are in the same row of the periodic
 table. For example, cc-pVTZ has 30 basis functions for atoms Li-Ne
 (https://en.wikipedia.org/wiki/Basis_set_(chemistry)#Correlation-consistent_basis_sets). Based on that,
 and since NeuralXC is using atom-based functions, I was expecting transferability between molecules that
 have different atoms of the same row.
 - I am still concerned about the applicability of NeuralXC. The authors now show in Figure 4 and Table II of
 the SI that PBE and NeuralXC have the same errors for the S66x8 dataset. What NeuralXC learns at the
 end?
 Reviewer #2 (Remarks to the Author):
 The authors have responded to all the points raised by the referees, and the manuscript is much improved.
 However, some of the improvements in presentation have lessened the possible significance of the work.
 And further improvements are still required (see below). But once these are done, and if all questions have
 satisfactory answers, it should be acceptable for Nat Comm
 1. The results on the S66 test set are entirely unconvincing and should be eliminated or mentioned only as
 “no significant improvement over PBE was found.” If the authors compared them with any state of the art
 functional, it would be clear that their numbers for NeuralXC are irrelevant.
 2. The results for densities are also less than convincing, since they consist of colored patches. The density
 differences should be quantified, to show that the NeuralXC densities are significantly closer to the CCSD(T)
 densities than either are to PBE. But the real problem is that in two cases, ethanol and water, they are not.
 In the absence of explanation of why not in those cases, this cannot be used to claim efficacy of the method,
 and all such claims should be removed (or bolstered). It is particularly disturbing when the primary result of
 significance is the improved energetics for water clusters and in condensed phase.
 3. In the hydrocarbon tests, it would be useful to consider error per bond instead of total error, in case it is
 just a question of system size. Also, it should be easy to include one or two double/triple bonds in the
 training set, and then see if that immediately reduces the linear component of the error in 3(b), justifying
 the authors’ hypothesis
 4. Much of the abstract, introduction, and conclusions is spent on very general statements, but nowhere do
 the authors give a precise description of what useful calculations their methodology should now make
 possible. Extrapolating the accuracies shown here, could I now, e.g., calculate the potential of mean force
 for methane or ethanol in water, at PBE cost but with CCSD(T)-like accuracies, using NeuralXC? With so
 many qualifiers on so many of their statements, or lack of precision, it Is not possible for a reader to tell.
 Moreover, given that MB-pol works for water, and the costs are surely greater than running MB-pol, pointing
 out applications beyond water is crucial.
 5. The authors insist on showing only the comparison with PBE, experiment, and their method in the main
 text for the water structure factors, but the convincing comparison is with the MB-POL results, and showing
 that NeuralXC does much better than all the other functionals shown. This should be in the main text. Some
 simple integrated measure of the errors should be reported, to see if there are any measurable differences
 with MB-POL. This again begs the question of what NeuralXC provides that MB-POL does not?