Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an efficient hierarchical molecular grammar learning algorithm for molecular property prediction. The similar property of same production rule motivates the study of data-efficient grammar-induced geometry. Specifically, the authors describes the process of building hierarchical molecular grammar by pre-defined meta grammar and learnable molecular grammar. Results on both small and large polymer datasets demonstrates the effectiveness of the method.
strength_and_weaknesses: Strength:
1. The molecular grammar hyper-graph construction is innovative.
2. The illustration of grammar-induced geometry is clear and interesting. 

Weakness:
1. This paper seems to focus more on grammar tree construction but the detail of how to integrate the prediction model (GRAND) in the paper and induced grammar is not very clear. For example, what if input polymer is not on the induced grammar tree, how do you guarantee every input can be represented on graph.
2. The motivation of using GRAND is not clear. I am assuming it's either because it's performing better or fit the grammar tree. Because the diffusion PDE seems has nothing to do with the proposed framework.
3. The experimental results are not very comprehensive. Admittedly, it shows great performance on two datasets in the paper, however, datasets like ZINC are more frequently used in the literature. Is the proposed method only used for polymers?
clarity,_quality,_novelty_and_reproducibility: The induced grammar part is clear but how to integrate the algorithm in an end-to-end manner is not very clear to me. I hope more clarification can be made and justified to support the claims in the paper.
summary_of_the_review: This paper discusses an efficient training framework for molecular property prediction via grammar-induced geometry. However, I found the motivation and design of the paper is not well supported in the main body of the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a data-efficient property predictor by utilizing a learnable hierarchical molecular grammar that can generate molecules from grammar production rules. The grammar induces an explicit geometry describing the space of molecular graphs, such that a graph neural diffusion on the geometry can be used to effectively predict property values of molecules on small training datasets. On both small and large datasets, the evaluation shows that this approach outperforms a wide spectrum of baselines, including supervised and pre-trained graph neural networks.
strength_and_weaknesses: Strength:

- The proposed method represents grammar production sequences and captures the structure-level similarity between molecules. With a few molecules, this method can utilize the structural relationship between molecules to predict molecular property. It is especially effective for small dataset.

- This paper improves previous molecular hypergraph grammar method [1] by integrating junction tree representation of molecules [2], and proposes a hierarchical molecular grammar to address the computational challenge in construction of grammar-induced geometry. 

Weaknesses:

- The proposed method is limited to small dataset scenarios. While there are many pretraining /transfer learning methods work well on both small and large datasets. 

- Evaluation is weak. In table 1, the proposed method is only validated on two datasets, compared with limited baseline methods. DILI is a representative small dataset and the authors should at least include experiments on DILI datasets to show the effectiveness of the proposed method, which is claimed to be effective on small datasets. Please refer to [3] for experiments on DILI datasets. Pre-trained GIN is neither the SOTA methods. Please the authors include more high-performance baseline methods. Experiments on large datasets in Table 2 only included two datasets, where there are tens of datasets for evaluation.

- Novelty is limited. The proposed method is an incremental improvement based on previous methods [1, 2].
 

[1] Guo, Minghao, et al. "Data-efficient graph grammar learning for molecular generation."ICLR (2022.\
[2] Jin, Wengong, Regina Barzilay, and Tommi Jaakkola. "Junction tree variational autoencoder for molecular graph generation." ICML 2018.\
[3] Ma, Hehuan, et al. "Deep graph learning with property augmentation for predicting drug-induced liver injury." Chemical Research in Toxicology 34.2 (2020): 495-506.
clarity,_quality,_novelty_and_reproducibility: Clarity: needs improvement. Need to check referred publications to understand the proposed method and novel contribution of this submission.
Quality: poor. Experimental evaluation is too weak.
Novelty: poor. Incremental improvement of previous methods.
Reproducibility: no code is provided.
summary_of_the_review: This paper proposes a framework for highly data-efficient property prediction based on a learnable molecular grammar. The intuition behind is that the production rule sequences for molecule generation provide rich information regarding the similarity of molecular structures. The proposed method combines previous existing methods [1, 2] and the novelty is incremental. The major concern is the weak evaluation of the proposed method. I would suggest the authors to include more experimental evaluation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose the use of a learnable hierarchical molecular grammar that can be used for property prediction tasks in the context of drug/materials discovery. This is done by extending the work of Guo et al. (2022) on molecular grammars and adapting it for the supervised learning setting via the usage of a neural diffusion model.
strength_and_weaknesses: Strengths:
* Natural extension of the molecular grammar work proposed by Guo et al. (2022) for the supervised learning case
* The methodology is well motivated and described
* Good level of detail provided in the appendix


Weaknesses:
* Rushed experiment section. Specifically the authors limit themselves to a couple of relatively small datasets when they could test on more widely used benchmarks in molecular ML to demonstrate the superiority of their approach (see summary of the review).
* Computational complexity not clear.
* No code provided to reproduce results / test the existing approach
clarity,_quality,_novelty_and_reproducibility: The work and the proposed method is sufficiently described in the manuscript, and clear effort was spent in making the work understandable.

In terms of novelty, I believe this is the first work that adapts grammar-induced geometries for the task of molecular prediction, albeit these approaches had already been common in approaches such as JT-VAE for molecular generation. 

In terms of reproducibility, while there is enough level of detail in both the main manuscript and appendix, the authors do not provide any code for either the proposed approach or to reproduce the results in the study.
summary_of_the_review: Overall, this is a good paper with a solid theoretical background that deserves credit. I mostly take issue with the (rather limited) result section, which I believe could use further work in terms of tested datasets. 

* The authors could consider testing their method on other well-known benchmarking molecular datasets, as available in the MoleculeNet sets to further improve their claims - or elaborate why they chose to test only on the CROW and Permeability datasets.
* The performance of the GEO-DEG approach seems not to be as strong compared to simpler approaches such as D-MPNN when trained on larger datasets (Table 2, Appendix) - to the degree that in the Lipophilicity dataset results are within 1 std. Could the authors elaborate on why they believe this happens?
* A point that seems to be somewhat absent in the study is the computational complexity required to train and evaluate the proposed learned grammar approach and graph diffusion model on new data. Could the authors provide some extra analyses so that the reader gets an idea on how expensive this methodology is compared to other baseline approaches (e.g. D-MPNN)
* In the process for geometry construction, the authors claim that they find it sufficient to use a maximum depth of 10. Could they elaborate  on how that decision was reached?
* The authors motivate hierarchical molecular graphs by claiming that the construction of grammar-induced geometries is costly. In fact, they mention that they find it infeasable to construct grammars when one considers more than ten production rules. Have the authors tried and compare their results with grammars constructed with only a few production rules?
* For PN and the Pre-trained GIN models, how was the pretraining done? Were these trained on additional polymer data or were they pretrained on other molecular species? Additionally on Figure 4b, it is not entirely clear how the Pre-trained GIN baseline was computed. Did the authors use the remaining 80% of the CROW dataset and finetuned the model there? How was this finetuning performed?


Other points:
* What features were used for the training of the random forest baselines?
* Figure 4b could ideally provide additional error bars
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the problem of molecular property prediction, particularly focusing on the limited availability of labelled data in many domains. It notes the limitations of existing methods targeted towards this scenario, and presents a molecular hypergraph grammar to describe molecules, and this grammar is used to induce a geometry over molecular space to enable effective molecular similarity measurement. To combat the computational challenge of enumerating a full molecular grammar-induced geometry the authors propose a hierarchical grammar consisting of a junction tree meta-geometry and a lower assignment of tree nodes to true molecular graphs. The property prediction then uses a graph neural diffusion model operating on all nodes in the grammar-induced geometry. The performance of this method is evaluated on two small datasets of polymers, and two slightly larger small molecule property prediction datasets. 
strength_and_weaknesses: Strengths: 
* This is a nice extension of molecular grammars to include a more computationally tractable method by inclusion of a hierarchical component.  
* The use of graph neural diffusion to make use of the grammar-induced geometry for molecular property prediction is a good idea -- one would indeed hope that a representation using explicit molecular substructures would ultimately be the best for molecular comparison, and it is helpful that the authors explore this.
* The description of the model is clear. 
* The level of detail given in the appendix regarding the molecular rules and geometry construction is helpful. 

Weaknesses:
* The lack of reference to related work -- the section in the appendix only addresses very general molecular property prediction and not molecular grammars, which in the context of a number of other works approaching similar ideas seems lacking. Concretely, such a section would be highly valuable. In addition, while Guo et al. 2022 is referenced, it does not seem that the strong similarities are acknowledged. Is it possible to compare the performance of this model to one using a non-hierarchical grammar (i.e. does this particular hierarchical grammar improve the molecular property prediction)? In addition, reference is made to the superior computational performance; is it possible to quantify this further? Such comparisons would be very useful. Furthermore, there have been previous works taking a hierarchical approach to the decomposition of molecular structure, such as JT-VAE, and hierVAE. While these works are in molecular generation, the underlying principles bear strong similarity. Extension of the related work section (and possible inclusion in the main body of the paper) would be helpful. 
* The authors strictly address only regression. While this is certainly a more challenging task in general, it does a) make direct comparison to other methods more challenging b) limit the range of datasets on which the method can be evaluated. The small molecule datasets in use appear to be the more challenging regression ones in Molecule Net (i.e. excluding quantum mechanics). As the authors note in the appendix, most related work in molecular property prediction is in classification, so it would be helpful to be able to draw some direct comparison to the state-of-the-art methods there.
* While the authors note that molecular grammars should improve the ability for a model to generalise outside of the domain, and give this as a reason for improved performance of their method over pre-trained GIN and PN, as these were trained on lower average molecular weight datasets than Permeability, they do not justify this with their results. In each of the CROW and Permeability cases their method was trained on the dataset at hand, and so does not have to demonstrate cross-domain adaptation. While GIN and PN are fine-tuned on the datasets, they come with strong intrinsic bias to their original pretraining datasets, and were pretrained for classification in addition. As both were pretrained on a set of small drug-molecule protein-binding data, it is actually remarkable they perform so well on a polymer dataset, where the relevant substructures are likely quite different. A concrete improvement would be demonstration of cross-domain transfer of the model.  
* Overall, the empirical evaluation and discussion felt rushed, the paper would benefit from more expansion around the results. Given the explicit reference to four key questions in the evaluation section, it would be helpful to have these more clearly addressed in the main body of the text, and perhaps slightly less time in description of the model. 
* Section 4.4 "Combination with generative models" is not a strong comparison to the other work in molecular generation, it is not clear how this particular method really compares in performance to, for instance, Guo et al 2022. It is also generally unclear whether the method has made improvements, or this is simply demonstrating that this kind of generation can be done, which was already answered by previous works. 
* Figure 4(b) addresses the effect of changing dataset training size on model performance. 
  * This kind of plot has appeared with other datasets such as FS-Mol. Given that this exploration is very useful, it would be helpful if the authors included analysis of performance on other datasets. 
  * A straight line represents the performance of a Pretrained-GIN, which is therefore presumably not finetuned on training set data. This is an unfair comparison. The Pretrained-GIN should be finetuned on the same data that the author's model has access to, and those results reported, in line with the typical few-shot literature eg. Hu et al. 2020, Stanley et al. 2021. 
  * It would be helpful to see the performance of a few more of the benchmarked models as the training set size is changed.
clarity,_quality,_novelty_and_reproducibility: Quality: 
The work is of good quality in terms of the ideas. The descriptions of the model are also clear and thorough. However, the depth of the empirical evaluation and support of the claims is not as strong as it could be. 

Clarity:
Overall, the paper is clear and easy to read. 
In Figure 4(b) it would be helpful to use number of datapoints to label the axes, rather than percentages, in line with the few-shot literature. 

Originality:
The work brings some original ideas, notably using hierarchical molecular grammars in molecular property prediction, and the use of graph neural diffusion for this purpose. The hierarchical grammar is an additional useful contribution above previous work with no hierarchy, although similar ideas have been seen in molecular graph generation eg. JT-VAE. This work clearly follows the lead of other recent works on molecular grammars and induced geometries and has strong similarity in many of the foundational ideas, so in that sense is extending that area of literature. The main novelty is the application of the ideas in molecular property prediction. 
summary_of_the_review: The paper makes a good contribution with a novel method for molecular property prediction that uses a hierarchical grammar to induce a geometry, then using this to make property predictions. The idea of using such a grammar in a property prediction setting is novel, and the additional hierarchical component is somewhat novel but bears some similarities to previous work on molecular substructures such as JT-VAE, hierVAE, and MoLER. In addition, the molecular grammar itself is very similar to the work of Guo et al. 2022. Therefore, much of the additional contribution comes from the application of these ideas in property prediction rather than molecule generation. This is a nice potential use of such a framework, and indeed could result in better cross-domain adaptation, but it was not clear the authors fully explored or supported claims made in the paper around this. In addition, the limitation of property prediction tasks to regression feels somewhat odd -- while regression is indeed more challenging, not addressing the many unsolved classification problems available in datasets such as MoleculeNet and FS-Mol prevents a strong comparison between this work and previous works being made. 

Overall, the ideas in this paper, while clearly strongly inspired by other works, are good. However, the paper in its current form is marginal because it would benefit from more time spent in experimentation and analysis to more strongly support the claims made. While the ideas are good it is not clear that the method is generally performing well, and whether it is more broadly applicable. If the method is truly very effective, and this can be demonstrated to be so on a wider range of commonly benchmarked tasks, then this would be distinctly impactful to the field. However, as it stands such conclusions are difficult to reach on the basis of the empirical exploration presented here. 

====update in response to authors comments====
Following the response and revisions of the authors, in particular the inclusion of additional evaluation, a discussion of the computational complexity, and more clarification of the baselines, this review is being updated to reflect this.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work addresses the problem of molecule property prediction when the supervised data is scarce.  It builds on prior work of grammar learning for molecules, which has been previously used for molecule generation. In this work it is used to construct a so-called grammar induced geometry (graph) that describes the generation process of molecules where the leaf nodes represent the molecules, and molecules sharing similar generation process are considered to be similar to one another. To capture this generative similarity, it applies GRAND to the grammar induced geometry to allow information to be exchanged between molecules and the labeled molecules are used to train the decoder and diffusion function of GRAND (as well as the molecular grammar) to perform property prediction on the testing leaf nodes.  Results on both small and large polymer datasets demonstrates the effectiveness of the method.

Pros:
+ The idea of using the grammar to induce a learnable prior for molecule similarities is novel and interesting
+ It appears to work well for property prediction when labeled data is very scarce 

Con:
- The writing needs work. The paper spent a lot of space (with redundancy) on the grammar induced geometry, on simple concepts such as BFS, but provided no detail in the main body of the paper on their key technical contribution: how to efficiently construct the geometry and perform molecular grammar learning in an end-to-end manner (the whole thing is deferred to appendix, which is also not self-contained and requires a familiarity with prior work) . It is also unclear how the proposed method can be applied to unseen molecules and ensure that they can be properly represented using the learned grammar (the authors' explanation in their response were similarly opaque and unclear).

- In the experiments presented in the paper, based on their explanation, the proposed method is used in a transductive and semi-supervised manner. But the baselines are all inductive, fully supervised methods. It is not clear the performance gain is not simply due to the transductive and semi-supervised setting.  Comparisons to other transductive and semisupervised methods are missing.


justification_for_why_not_higher_score: The writing needs major work and the empirical contribution is really not clear without comparing to other transductive semi-supervised learning methods.
Decision: Reject