Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a spanning tree based generative model (STGG) for molecular graphs. STGG sequentially generate a molecule's spanning tree and fill in the residual edges on the way. The spanning tree construction is similar to the standard SMILES representation, but the model operates on a molecular graph rather than a SMILES string. STGG adopts a tree-based transformer with relative positional encoding for tree generation, and a attention-based predictor for residual edge prediction. The method is evaluated on standard ZINC250K, QM9, and MOSES benchmarks and outperform existing baselines.
main_review: Strength:
 * Comprehensive evaluation. The method is evaluated on standard ZINC250K, QM9, MOSES benchmarks and compared with many previous baselines. The model is able to achieve state-of-the-art results on most of the metrics.

Weakness:
 * As noted in the paper, SMILES strings are also constructed by spanning tree algorithms. As shown in Figure 2, there is very little difference between generating a spanning tree versus generating a SMILES string. The only difference between STGG and SMILES string is the adoption of a tree-based transformer.
 * STGG is a node-by-node graph generation method following the depth-first order (DFS) of a graph. CG-VAE (Liu et al. 2018) is also a node-by-node graph generator, but following the breadth-first order (BFS). It would be great if authors can compare STGG with CG-VAE given their similarity.
summary_of_the_review: This paper has different pros and cons. I vote for weak accept because the empirical results seem convincing. Adding comparison with CG-VAE is important and I hope the authors can address this concern during rebuttal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel spanning tree-based graph generation (STGG) framework. The key idea is to represent the molecular graph as a sequence of decisions according to a **novel spanning tree-based grammar**, and then model these decision sequences using the transformer.

To accommodate such a novel algorithm, several interesting techniques are involved, e.g., representing molecules as bipartite graphs, using tree-based positional encoding for transformer. The authors claim that the proposed method allows generating valid molecular structures and inferring the intermediate graph structure.
main_review: **Strengths**
* The proposed method is quite novel. The key idea is to represent the molecular graph as a sequence of decisions according to a **novel spanning tree-based grammar**. Compared to SMILES-based molecular generative methods, STGG allows inferring the intermediate graph structures and takes the graph structure into consideration. The method is general and can be applied to other graph structures.
* The paper is well written. The experiments of the work are comprehensive and well designed. In particular, for PLOGP molecular optimization, they adopt an offline optimization algorithm, which is interesting to me.
* The code is already provided in the Supplementary Material

**Weaknesses or Questions**
* What is the main advantage of STGG over atom-by-atom graph generative models? The Table2 indicates that STGG tend to generate molecules with low novelty and the main advantage is validity.
The authors mention that atom-by-atom methods determine the validity via sample-reject. But actually, we can also mask out invalid decisions for atom-by-atom methods by keeping a record of valency and bond information.
* **Concerns on training data**: To get the training data, the authors use algorithm2 presented in Appendix A to obtain the sequence of decisions from graph structures. For each graph structure, how many sequences do you get? Do you fix the random seed of this algorithm, or generate the sequence on the fly in each batch with different randomness? My key point is that if the training sequence is generated on the fly for each batch with random DFS, the training data for SMILES-TRANSFORMER should also be augmented (instead of using canonical smiles) for a fair comparison in Table 2.
summary_of_the_review: I think the paper reaches the bar of ICLR for the reasons listed above.
However, several concerns remain to be addressed.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new paradigm of graph generation, a transformer network sequentially generates a sequence of decisions to generate a spanning tree for a bipartite graph. These decisions have 7 forms such as attach atom, attach bond, branch start,  etc. After the spanning tree, they add residual edges. The authors also have a focus on generating valid graphs (which is seen in the results) in which they mask out invalid decisions during the generation process itself. 


main_review: Strengths:
I think this work has a lot of novelty and merit. The idea of using a transformer which can view earlier decisions with attention mechanisms increases the possibility of valid molecular generation. This also forms a conceptual departure from highly sequential or reinforcement learning-based methods. They show their results on 3 datasets MOSES, QM9 as well as ZINC250 K. As can be seen in the results, many previous methods do not run well on QM9 and are often overfit for 1-2 benchmarks. The authors also perform ablation tests which show the improvements with several of their design choices including the use of graph structure, the use of a transformer and the use of graph construction and valence masks. 

Weaknesses:  
Although the ablation studies that are used by the author validate some of the design decisions they don't validate or establish the need for several of their most novel decisions: 1. The two-pronged system of first generating spanning trees and then residual edges, 2. the use of a bipartite graph rather than one that uses edges to represent bonds, 3. the use of spanning trees themselves. In addition I would like to see some results on how long it takes to train this transformer model as opposed to other models, due to what I believe is a highly increased number of parameters given the positional encoding and the attention mechanisms. 




summary_of_the_review: Interesting proposal, needs more validation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a method to construct a molecular graph, which is inspired by a spanning tree. In specific, a molecule is constructed by a sequence of actions, each of which adds an atom, adds a bond, starts/ends a branch, adds residual edges for a circular structure, and terminates. The generative process is controlled by a transformer-based neural network, which is specialized to tree construction procedure.
The empirical studies show that the proposed method can learn the distribution of molecules comparably or better than the existing methods, and can be used to molecular optimization.
main_review: ## Strengths
### + Reasonable approach to construct a molecule based on spanning-tree representation
While there are a number of approaches to generate a graph based on spanning-tree representation (i.e., SMILES), most of them treat a SMILES representation as a text and do not exploit the spanning tree. The method proposed in this paper explicitly uses the spanning-tree representation to construct a molecule, and it seems a very reasonable approach to construct a molecule, if inspired from SMILES.

### + Clarification on the known flaws in the penalized log P score
I appreciate the authors clarify the limitation of this benchmark task and propose an alternative evaluation scheme. As stated in Section 5.3, this benchmark does not serve as a proper benchmark task, but not many researchers do not discuss it. This statement is very important to the community.


## Weaknesses
### - Relationship to the existing work
I appreciate the authors summarize the relationship to the existing work in Section 4. I understand that the proposed generation procedure is different from the existing ones, but I don't understand why the difference is important. For example, the authors state that "Our STGG framework differs from this line of research since it proposes a new type of graph-based operations for generating the molecular graph", but do not clarify how it is different, and why the difference is important. Such a comparison is important for readers to understand the essence of the proposed method.

### - No theoretical guarantee to generate valid molecules
I am not convinced with the mechanism to comply the valence rule, and I wonder if there is a theoretical guarantee that this mechanism can comply the rule or there is a counter-example where this mechanism cannot guarantee it.
When constructing a ring, it is desirable that the tail atom has one remaining valence, and the ring closes by adding a residual edge. Is it possible that the tail atom has no remaining valence and the ring cannot be closed? For example, C*=C-C≡C seems not to be rejected by the mechanism, but we cannot close the ring.
% This may be a question, rather than weakness. If there is any misunderstanding, please correct it.

### - Relationship to the classical VAE+BO approaches
As discussed in Section 5.3, one of the major issues in the plogP optimization task is that unrealistic molecules can optimize the score. I consider there had been an implicit agreement that the optimized molecules should resemble the training data, which leads to the classical molecular optimization method combining VAE trained on the real-world molecules and Bayesian optimization. As far as I am aware of, the method by Kajino [Kajino, 19] achieves the best scores among the methods using this approach. While the proposed method can control the trade-off between the score and realisticness, it seems the proposed method is not better than VAE+BO approaches in this setting.

[Kajino, 19] Molecular Hypergraph Grammar with Its Application to Molecular Optimization, ICML-19.

---

Given the discussion below, all of my concerns have been addressed.
summary_of_the_review: I am tending to reject this paper because of the following reason.
While the proposed method is very reasonable if inspired by SMILES, its novelty within the literature of graph-based models is less clear, and thus, I am not very enthusiastic about the proposed method. I would increase the score if I were convinced of the contribution of this work to the literature of graph-based models more clearly. So, I would like the authors to clarify which aspect of the proposed graph generation method is important and why. In addition, I would like to ask the authors to discuss why they don't take the VAE+BO approach to molecular optimization, which can implicitly constrain the output to be similar to the training molecules.

---

Nov 21. Updated the score to 5, given the first response by the authors. But still not positive to accept this paper because of the inconsistency in the experiment on molecular optimization.

--

Nov 24. Since all of my concerns have been addressed in the current version, I would recommend to accept this paper.
review_title: null

Meta-review: This paper proposes a spanning tree-based graph generation framework for molecular graph generation, which is an interesting problem. The tree-based approach is efficient and relatively effective in molecular graph generation tasks, and the empirical results are convincing. There were some concerns during the initial reviews, but all of them have been addressed during the discussion phase. Thus, I recommend this work be accepted.
Decision: Accept (Spotlight)