Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors look at molecular property prediction from a few shot learning perspective. They propose an approach, closely following Meta-MGNN, to adapt part of a (optionally pre-trained) graph embedding network in MAML-style per task. A number of well-motivated novelties are introduced: (a) adapt part of the GNN encoder, called property-aware transformation (vs just task-specific attention in Meta-MGNN). (b) explicitly constructing the similarity graph between a test sample and the support set (labeled samples) and add a sparsity loss for the adjacency matrix during training. The results, focusing on 1-shot and 10-shot prediction, convincingly show gains in this regime, and extensive ablations and baselines are presented.

main_review: 
Major strengths:



* Strong motivation for property-specific adaptation + adaptive relation graph learning is presented.  This is done via examples (Fig1), comparisons to works without property-specific adaptation, and examples of how learned relation graphs differ greatly across tasks.
* The proposed model (PAR), though complex, seems to appropriately address the problem setup.
* Strong results (only on 1-shot and 10-shot classification) + comparison to multiple baselines, for each of PAR’s major dimensions.
* Comprehensive ablations of contributions and analysis of their effects.

Weaknesses - none are major, all are expanded upon below.



* Somewhat unclear writing.
* High overall complexity of the approach, adding elements to already-complex IterRefLSTM and Meta-MGNN
* Results only on artificial few-shot settings 

Overall this is a strong paper, hence a score of 8: Top 50% of accepted papers, clear accept

## Detailed review


### Motivation and method

The paper has a strong motivation around few-shot learning and the need for property-specific adaptation of general purpose embeddings.

Also the motivation behind ensuring the relation graph (similarity  is correctly constructed for a task is well-taken, and illustrated convincingly with Fig 4,8,9. Ablations show the gain of constructing and sparsifying the similarity graph.

Questions for the authors:



* Can the adjacency matrix be estimated directly from cosine similarity of embeddings, rather than doing the iterative estimation?

Other minor comments: 



* Appendix Figure 8 row order doesn’t seem to match caption? (0-1 should be ground truth versus grayscale predicted?)
* Figure 4 - these adjacency matrices are the same but according to Fig9 and caption they should be different. Also same comment as above.
* Figure 3 - please label the variants with a short summary to avoid having to cross-reference the text.


### Impact / significance / results



* The ablations are strong.  The three main contributions (beyond results) are ablated, and show large degradations when removed:
    * property-aware embedding
    * fine-tuning of all parameters degrades wrt only final layers
* Comparison to existing work is thorough.
    * 10 seeds for confidence intervals.  Proper comparison to several lines of work which were the basis for PAR.
* Overall this paper addresses a difficult task in a better way, with strong motivation and appropriate novel methods.

Comments:



* It would be helpful to see a section on limitations.
* It would be interesting to see results on full tox21 / other moleculenet datasets, without artificially constraining to 1 or 10-shot classification. However we recognize this is following the standard set by IterRefLSTM and Meta-MGNN.
* See suggested ablation above: “Can the adjacency matrix be estimated directly from cosine similarity of embeddings, rather than doing the iterative estimation?”


### Clarity of writing

Overall the paper is thorough, well-motivated and contains necessary detail.

The **clarity of the paper could be improved**:



* Some concepts should be high-level described (relation graph / property-aware embedding) when the terms are introduced.
* Heavy notation can be reduced and gets in the way of understanding. Authors should look for places to provide a high level summary sentence at the start of the paragraph to summarize the paragraph.
* It would be helpful to be more explicit where the aggregation (GNN node embeddings -> single vector) happens, versus where is task-specific adaptation. 

Specific comments about equations:



* Please move description of g, p, r, c superscripts to the front of [Section 3 - Proposed Method]
    * ‘c’ superscript is used for both classifier parameters $\theta^c$ and context embedding $e^c$, which occur in two different stages?
* What is the $e^p_{x_{Tj}}$ initialization value for equation (2)?

Minor comments and typos

* Several grammatical errors (eg: “while fine-tune theta”, “taking a few gradient descents”)
* Figure 4: (d) 12th task instead of 11th task?
* Parameterize MLP in eq (3) by $\theta^r$ ?
* Also see comments about figures above
limitations_and_societal_impact: It would be helpful to see a section on limitations.
Negative societal impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a novel framework to solve a few-shot molecular property prediction problem w.r.t different properties. It takes GNN as the encoder to obtain property-aware molecular embeddings and projects it to a substructure-aware space according to the target property. Moreover, a relation graph between molecules is learned adaptively to refine the embeddings. Training strategies like gradient-based meta-learning is applied to boost property-aware training. Extensive experiments are conducted over a few-shot property prediction task as well as transfer learning across datasets
main_review: The authors design a novel graph-based property-aware framework for few-shot molecular property prediction and generate a relation graph among molecules to help further refine the molecular embeddings. However, with a closer look, the proposed model seems to be a combination of multiple existing works, e.g., property-aware attention[16], gradient-based meta-learning strategy [9]. It utilizes numbers of modules, and my concern is that how important each module is. It is hard to say all these modules/strategies work effectively since the performance improvement seems limited. The ablation study compares the performance of some variants but it only takes out one module at a time, so there is a chance that certain combinations can obtain competitive performance too.

In addition, the experiment results show that the performance of PAR does not significantly surpass other baseline models w/o pretrain models, instead the pre-GNN + PAR performs the best. The effectiveness of the proposed PAR seems unimpressive. Moreover, the details of the pretrain model implementation are not given, like what dataset is used and what experimental settings are. Also, I am curious that why the training takes so long (2000 epochs), while the original pre-GNN only trains 100 epochs. The dataset does not seem that large, it’s only a few thousand molecules. Additionally, considering such a large training epoch, it’s better to report the training time for the proposed model.

The paper seems to be finished in rush, many unfinished part, e.g., missing citation on P3 L101; no number filling out on P7 L249; the caption of Fig.5 is not consistent with the notation; the dataset description on P7 L216 is not consistent with table 1. The writing needs to be improved.     

limitations_and_societal_impact: See above
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a property-aware adaptive relation networks for few shot molecular property prediction.  They propose a graph based molecule encoder which is trained from large scale tasks to capture a generic representation of the molecules and a property aware embedding function which adapts the generic molecule embedding to be property-aware. Next, an adaptive relation graph learning module is proposed to capture the relationship between similar molecules with respect to a target property. The adaptive relation graph learning module is based on GNNs. A new training strategy, where only the parameters of property aware embedding function and the classifier module is tuned, is also proposed. The authors argue that it useful to separately capture generic molecule representations and those that are specific to the particular task.
main_review: The authors propose a new method for Few shot learning of molecular properties. The proposed method achieves state of the art performance on four classification tasks (Tox21, Sider, MUV, ToxCast). As the authors argue, few shot learning for predicting molecular properties is very important for drug discovery and related applications. Collecting a large number of labels for a particular molecular property is usually expensive. Hence the proposed problem is of great practical interest. 

The proposed method is novel and the experimentation is thorough, despite the evaluation being done only a small number of tasks (see question below about extending it to other tasks). The ablation study disentangles the contribution of each variant on the  final score. The paper is well written and clear. 

Overall, I am quite positive about this paper. I feel it tackles an important problem in a novel way and shows good performance.
limitations_and_societal_impact: Limitations and Questions for authors:
Many of the molecular properties of interest are regression tasks (for example, quantum mechanical properties , binding affinities and so on). Can the authors explain if it is possible to extend the methodology to regression tasks?

The evaluation is done only on Tox21, Sider, MUV and Toxcast.    Does the method transfer to other tasks from the moleculenet dataset?
review_title: null

Meta-review: The paper proposes  a framework for few shot learning for molecular properties prediction. Reviewers have agreed that the problem of few shot training is important in the molecular  space and that the  approach proposed in the paper is novel and elegant. Authors showed evidence of their method working in the few shot regime. Accept 
Decision: Accept (Spotlight)