Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper studies approximation of the potential energy of molecules by a message passing architecture. The work builds upon [Gilmer et al., 2017] and the contributions are two-fold:
1) The creation of new datasets to learn and test such architectures on and the augmentation of an existing dataset in order to account for energies at non-equilibrium states.
2) A proposed modification to the MPNN architecture proposed in [Gilmer et al., 2017], in order to account for physical properties in the message-passing procedure.
The performances of the architectures are studied with numerous numerical experiments.
The paper is overall well-written and clear.

The new dataset utility is sound and well-motivated. Unfortunately I can not further motivate upon this, as I am not familiar with this area.

From the point of view of the proposed architecture, the work is quite incremental. The bond type information, previously included as feature, is now transferred to an architectural modification. On the other hand, many different modifications (although no substantially different from each other) are proposed and tested (although no results about the different modifications are reported - it could be nice to have them in an appendix). This motivates the ‘weakly accepted’.

The authors also considered the idea of adding additional learning modules (and a related loss) to help the model learn more 'physics interpretable' hidden states. While it does not seem to give notable gains here, it is an interesting idea and I believe deserves further experimentations in the future.

The experiments are numerous and various, and they offer a very good overview on the goodness of the model (and its limitations). They first compare with the baseline on the augmented dataset, and they show notable gains on the MPNN baseline. The ability of the network to reproduce the energy curve at different interatomic distances is then studied on the different dataset and in different settings, showing gains over the baseline. The authors also report some negative results and experimental interpretation of the model hidden states, which are also an important contribution in my opinion. 

Further comments:

1. No details are given about the training of the models. I think a small paragraph (or larger and reported in the appendix) should be added.

2. Even if it builds upon previous work, the (VI)MPNN model may be further explained. For example, what type of functions are M_t and R? The explanation on the considered modifications of MPNN may be clearer (maybe with the introduction of a more mathematical notation). 

3. How long is the message diffusion (T)? What is the effect of larger / smaller T’s?

4. What’s the point of equations (5) (6) (7)? They are exactly the same and they do not add any information. It would be more useful to explain what type of function R is in my opinion.

5. Table 1, Auxiliary estimates: Are these the results obtained by the model a.ii trained to jointly learn the energy and the properties i) ii) iii) ? In what sense they improve on the baseline? This part was not clear to me.

6. Section 5.2: ‘[…] we combine our proposed physics integration strategies, namely bond type
specialised
node updates (case a.ii) of Section 4.2) and auxiliary estimations of physical properties,
into the VIMPNN model […]’. I do not understand this sentence. Isn’t in fact the VIMPNN architecture the same as MPNN with the modification a.ii? In what sense do you integrate a.ii in it?

7. If I understood correctly, the main final objective is to be able to characterize the minima of the energy. In this case, could you asses the performances of the two methods (MPNN) and (VIMPNN) by measuring some kind of distance from the approximated minima from the actual one?

Typos:

Section 5.3 first line: ‘We investigate the VIMPNN’s the ability …’ -> ‘We investigate the VIMPNN’s ability’

Section 5.2 Fourth sentence: ‘The first seeks to demonstrates a the model’s …’ -> ‘The first seeks to demonstrates the model’s…’
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: The paper tackles the problem of estimating the electronic structure's ground state energy of a given atomic system by means of supervised machine learning, as a fast alternative to conventional explicit methods (DFT).
This is done by improving on a previous method, MPNN, Message Passing Neural Networks, and in particular by including information on bond type as input, so that the NN can learn the appropriate weight for messages going through bonds of that type.  In addition, training on several target labels (multi-regression) is attempted, with the idea that more physical outputs may help building better hidden representations (on this, there are mixed results).  Separately, training sets are enriched with non-equilibrium structures, so as to confront the NN with more diverse data.

The method is tested on 3 kind of training sets. The first is a simple but time-consuming augmentation of QM9, with inter-atomic distances varied, so as to increase the training set's size (and in particular, including non equilibrium configurations). The second consists in a periodic and thus infinite simple crystal structure (with, again, variations in inter-atomic distances, enriching the dataset). The third is a pseudo-cristalline structure with atoms randomly placed on a regular grid, forming a somewhat random structure, also named crystal (this is not a very good name).

The paper is overall rather well written, sometimes being a bit cumbersome (long sentences), but mostly it is stating clearly what is done or discovered.  The work is situated within the existing literature (that I am not familiar with at all).  The idea of using physics to guide architecture choices is gaining a lot of attention recently and seems to be well-suited to this particular problem, and well applied.  Several ways to use the bond type information have been attempted in this work, and several of them are reported and compared (a couple of them are discarded).  The results convincingly show that using bond type information indeed increases performance, both for small systems and for regular crystals.  The impact of performing multi-regression is less important, but still positive.  For large ''random crystals'', the method does not perform very well, and this represents a challenge for future work. Such a confession on the method's limitations is welcomed.

Given the idea (using physical information as bond type) is clearly and honestly presented, produces significant improvement compared with previous works, and has perspective for multiple future developments, I recommend acceptation of this paper.


There are however a number of points that could be improved.

1. There is a physical mistake that is not crucial but should be corrected, when talking about the ground state, and in particular before this sentence: ''accurate ground-state energy estimation of out-of-equilibrium molecule''. Ground state means minimal, T=0K energy level, so by definition it is at equilibrium. Thus, the sentence seems quite contradictory to a physicist.
What DFT and VIMPNN actually compute is the electronic structure's ground state's energy (at fixed positions of the atom kernels). I think this distinction should be mentioned just once, and then you could proceed with saying ground state energy. 
Because of this, I would recommend to edit the title so as to suppress ''out of equilibrium'' from it.  Otherwise readers may think the method deals with non-equilibrium electronic structures (non ground states), which clearly it does not at all, or they may think that it is especially good at estimating energies for out of equilibrium systems, which is not its primary goal.

2. I do not understand very well the training procedure.  Also there are some tests that seem to be interesting and that are not performed (as far as I understood).
Does each training set contains the 90%-150% data augmenations, for each non-augmented training configuration ?
Why is training performed separately for each kind of data set ? Wouldn't the ultimate goal be to transfer learning from a type to another, e.g. from QM9-style to crystal style, etc ?  (as far as I understood, this was not done)
Isn't it interesting to see how much training on an augmented data set (let's say QM9) improves performance on the non-augmented data (the ''true data'' in a sense) ? Although the augmented data is obtained by DFT, and comparing models trained on different data sets is unfair, I think it may be interesting to see if VIMPNN benefits more than MPNN from this strategy (so compare the performance gains of both algorithms obtained by augmenting a data set). This kind of comparison may also be done for the ''augmentation'' of a training set by the concatenation of it with another one (although in that case it may be detrimental to the test accuracy?).
If you actually did some of this, then I misunderstood and I am sorry, but then this also means you should clarify.

3. Section 3.3:
I would not call this a crystal, but more something like ''random finite structure''. This should be done everywhere in the paper.

4. Section 4.1 is a bit too short for the inexperienced reader. I suggest to be a bit more explicit on what is learned

5. Section 4.2: It is nice to say you tried other ways, keep that.
However, try to be more explicit on what is shared and what isn't, in the architecture you finally pick. Is lambda(v,w) a common value for all bonds of the same type, like C-C ? Maybe you could provide an example or some more detailed notation to make your choice fully explicit (after all this is the core of the paper).

6. Section 4.3: could you quickly comment on why you don't use more of the 13 physical observables available in QM9 ?

7. Table 1: do the three last lines correspond to ''no BT information + a single auxiliary estimate'' ?  It seems to be the case, but then you say you will continue with the auxiliaries, in addition to the BT information.  Why don't you display the result of using BT information AND the 3 auxiliary estimates ?  If you did, then I misunderstood, but it would also mean you did not explain well enough a counter-intuitive result (which would be that adding auxiliary information actually hurts the performance of the VIMPNN).

8. Please include a couple of explicit numbers of your training/test/validation sets sizes.

9. section 5.2 could be made more concise. In particular, there is no need in repeating what can be seen directly in the figures (stating numbers). It is useful to comment on the meaning of the results however (as you currently do).

10. section 5.4  is a good idea, promising, however it does not really conclude into a very strong statement, and takes essentially 1 full page, which could be used to better clarify the architecture and/or the training procedure (or to reduce towards the ideal page length).




review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper presents a number of new / extended datasets for the evaluation of ML-based prediction of energies of unstable systems, as well as a network (VIMPNN) that includes a new and better way of including bond-type information. It is also proposed to use auxiliary losses (predicting other chemical properties).

Although I am not an expert in chemistry, the new datasets seem fairly well thought out and their utility is well motivated. The proposed change to the MPNN network architecture is rather simple and hardly physics inspired, but the empirical improvement seems substantial, so this too is a nice contribution. So I have decided to give the “weak accept” rating.

In 4.2 it is explained how different ways of incorporating bond information were evaluated, and it is stated that “best results were obtained in the case a.ii). However, no results are presented to support this claim, leaving the reader to wonder how rigorous this exploration was. I would suggest systematically evaluating the different options and including the results in an appendix. 

In table 1 results are shown for the MPNN baseline, baseline with specialised node updates a.ii, and with auxiliary estimates. However, combinations of these are not evaluated. Nevertheless, if I understood correctly, the VIMPNN method tested later includes all of the separate improvements. It would be good to include experimental results to motivate this.

As acknowledged in the paper, the idea of using bond-type information was already in Gilmer et al. Also, I think the different ways of including bond-type explored in this paper are not really informed by physics. The choice for method a.ii is made based on empirical results. This is not a problem in itself, but I would suggest that the authors change the wording to not over-promise on the physics-inspiredness. E.g. the abstract says “VIMPNN integrates prior knowledge such as the existence of different interatomic bonds”, suggesting that there is more prior knowledge being exploited than just bonds.


Comments:

“It produces comparable accuracy to that of DFT while also improving computation time by 5 orders of magnitude”. 
I assume this speedup is relative to DFT. It would be good to be explicit about that, and also discuss the speed relative to the MPNN baseline (I suppose MPNN and VIMPNN are similar).

“The change of atomic distances are performed isomorphically” - I would say “isometrically”. 

review_title: Official Blind Review #1

Meta-review: The paper considers the problem of estimating the electronic structure's ground state energy of a given atomic system by means of supervised machine learning, as a fast alternative to conventional explicit methods (DFT). For this purpose, it modifies the neural message-passing architecture to account for further physical properties, and it extends the empirical validation to also include unstable molecules. 

Reviewers acknowledged the valuable experimental setup of this work and the significance of the results in the application domain, but were generally skeptical about the novelty of the machine learning model under study. Ultimately, and given that the main focus of this conference is on Machine Learning methodology, this AC believes this work could be more suitable in a more specialized venue in computational/quantum chemistry. 
Decision: Reject