Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new framework, named ReaKE for learning representations for chemicals to better predict chemical reactions
The main contributions of this work are three folds as claimed by the authors: (1) chemical synthesis KG of reactants, products and; (2) contrastive learning strategies in KG triple level and molecular graph level; (3) joint training on reaction and molecular level which are essentially two types of graph. 
Experiments have demonstrated that ReaKE outperforms other SOTA models and the effectiveness of ReaKE modules with ablation studies. 

strength_and_weaknesses: Strength:
1. Well-established problem and formulation of chemical synthesis problem from the perspective of "multi-view" KG representation learning 
2. The proposed framework is well-designed and mostly technically sound. 
3. Extensive experiments from multiple applications and comparison with a large group of the state-of-the-art baseline approaches

Weaknesses: 
1. The writing clarity needs improvement, especially on the term definitions, and specific mathematical functions. 
2. Insufficient justifications for the rationale of the proposed model and training design.
2. Some important baselines are not included in the comparison, discussion, or related work. It is unclear to distinguish previous work on graph learning and SMILE-based models. 

clarity,_quality,_novelty_and_reproducibility: Clarity and quality: 
Generally, the clarity of this paper is unsatisfactory. As a work of interdisciplinary domains across chemistry and molecular science, graph machine learning and representation learning, many terms, and concepts mentioned in this paper may not be understood universally without formal introduction and clarifications. A notation table or preliminary section for the glossary used in this paper is highly suggested, to explain the entire KG schema as shown in Figure 2(a) in Section 2.1 (Section 2.1 is insufficient to understand the full image). To name a few unclarified terms: 
1. In-depth analysis of the disadvantages of existing methods. All three challenges such as energy flow, ambiguous embedding, sparse embedding space (and smoothness) are not well justified in the introduction and how are these problems observed or demonstrated? 
2. What does it mean by "embeddings are equal in embedding space"? 
3. Template information, reaction condition, and how it is used and guides the model learning
4. Full node features in molecular encoder as one-hot input features (how degree information is included as one-hot)
5. Why are the design of triple-level and graph-level contrastive learning / negative sampling valid? Why is the proposed method for hard negative examples more informative, compared to the original negative sampling?
6. All the tasks are not explicitly explained about the input, output, and goals and how the inference steps are done.

Novelty: 
This work is considered relatively innovative (or marginally innovative) as a straightforward combination of well-known modules as building blocks. Most of the techniques used in this paper, are molecule representations (GNN, graph readout), self-supervised learning, augmentation, and contrastive learning, and knowledge graph embedding (TransE-like structure). It is beneficial to have a comprehensive set of technical approaches that collectively work with the input of relation-level graphs and molecule-level graphs, as an integrated and improved solution. 

Reproducibility:
The level of reproducibility is medium. ReaKE involves many modules as building blocks (GNN, SSL, mini-batch negative sampling, joint training), however, it seems that many implementation details and hyperparameters are not fully mentioned, including visualization of tSNE configuration. The supplementary materials may cover some model specifications but without a pre-released codebase, it is not confident to reproduce the results. Datasets are publicly available research datasets.

Please check the section "summary of the review" for more detailed questions, comments, and suggestions. 
summary_of_the_review: This paper proposed ReaKE as a comprehensive framework for chemical-KG representation learning both from the reaction knowledge graph and the molecular atom graph. The major issue of this paper is clarity on model details which makes it hard to fully understand the motivation and rationale of the learning strategies and training protocol. 

Some other questions and suggestions are as follows: 
1. There is no related work in this paper which brings difficulties to understand the position of the paper. It is suggested that a separate section discuss the distinction and how the baseline models are selected. Some works are missing regarding SSL in graph neural networks [1], graph-sequence hybrid models [2], and multi-view KG joint training [3]. 
2. Why are traditional KG embedding methods on reaction KG (TransE) not usable for comparison for downstream tasks?
3. No report of scalability and computing resources used for the complex learning framework. 
4. As a minor issue, the term "synthetic“ may cause confusion at first when it comes to "synthetic KG" which by default means manually generated datasets instead of real-world ones. "Chemical Synthesis Knowledge Graph (CSKG)" may be used to avoid misunderstanding. 


References: 
1. Xie, Y., Xu, Z., Zhang, J., Wang, Z., & Ji, S. (2022). Self-supervised learning of graph neural networks: A unified review. IEEE Transactions on Pattern Analysis and Machine Intelligence.
2. Wang, Z., Liu, M., Luo, Y., Xu, Z., Xie, Y., Wang, L., ... & Ji, S. (2022). Advanced graph and sequence neural networks for molecular property prediction and drug discovery. Bioinformatics, 38(9), 2579-2586.
3. Hao, J., Ju, C. J. T., Chen, M., Sun, Y., Zaniolo, C., & Wang, W. (2020, September). Bio-JOIE: Joint representation learning of biological knowledge bases. In Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics (pp. 1-10).

For other issues such as clarity and novelty, please check the corresponding sections.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper mainly proposes a chemical synthetic knowledge graph (ReaKE) to handle with different downstream tasks relevant to molecules. The proposed framework mainly addresses three problems: abnormal energy flow, ambiguous embeddings and sparse embedding space. Experimental part shows that the ReaKE improves the results in various tasks.
strength_and_weaknesses: Pros:

- The paper is easy to follow.

- The three problems the paper given are important for the chemical reaction-aware self-supervised methods and hard to completely solve.

- The paper uses the chemical reaction data differently compared to existing works and get good experiment results on various tasks.

Cons:

- The paper does not have a related work part to orderly introduce the existing works and some related works mentioned in the paper are not well introduced. For example, the four GNN models used in experiment part are not introduced in the paper.

- Although the paper has experiments for various datasets, in some tasks such as chemical reaction prediction, reaction classification and yield prediction, there is only one dataset used. I think more datasets should be used to verify the efficiency of the proposed framework.

Concerns:

- In the example in Section 2.1, I cannot understand why one reaction can be transformed into two triplets, as there should be simultaneously two input molecules to get the output molecules.

- In Triplet-level Contrastive Learning in section 2.2.1, the molecules obtained after the drop step might not exist. Is it meaningful to predict these molecules that do not exist in real world?

- In table 1 in section 3.2, the results except MR are stable and almost all the stds are zero. Why?

- In table 2 in section 3.3, the results of DRFP and RXNFP are bad, what is the reason? Are there baselines that can perform better?

- The baselines for MPP tasks are too old. Are there any newer baselines? For example, some NLP based methods are mentioned in introduction. Can these methods applied in MPP tasks?

clarity,_quality,_novelty_and_reproducibility: Clarity, Quality: The paper is clear.

Novelty: The method consists of several points which appear in existing works (contrastive learning, augmentation and negative sampling). For molecular augmentation, the augmentation strategies are similar (lack fundamental differences) with [1,2,3]. For negative sampling, the so-called "reaction-aware negative sampling" is "relation-aware negative sampling", which is common in knowledge graph embedding papers. I do admit no one apply this to molecular related applications in the past.  

Reproducibility: I think most of the results of the experiment part can be reproduced. But I have some problem of experiment part (see Concerns). 



summary_of_the_review: The paper proposes chemical knowledge graph ReaKE to learn better for different downstream tasks. 
The method consists of several points which appear in existing works (contrastive learning, augmentation and negative sampling), but the combination can be new and has not been applied for molecular applications. I would recognize this paper as successfully adapt existing tricks to molecular related applications.

Overall, I think the paper is marginally below the acceptance threshold currently.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a molecular representation learning method based on pre-training with chemical synthetic knowledge graph. The proposed method tackles and solves the limitations in previous works that use chemical reaction knowledge in self-supervised learning.
strength_and_weaknesses: Pros:
1. The paper is well-organized and easy to follow.
2. The proposed approach is simple and useful to deal with the previous limitations.

Cons:
1. The experiments are not clearly described and exhibited:
    - It is unclear what the "TAG" model is. In Appendix A, GCN, GAT, SAGE, and TAG all lack their references. 
    - From the results shown in Table 1 and Table 2, it is not necessary to include the results of ReaKE-SAGE/GAT/GCN, since ReaKE-TAG consistently performs better than the other three variants. 
    - It is unclear whether the results of the baseline models come from their original papers, or from the authors' own experiments. For example, the results of MolR in Table 1 are directly from its original paper [1]. However, the results of MolR in Table 2 and 3 are not found in [1]. The authors do not mention how they get the results. 
    - For the visualization in Figure 3 and 6, the baselines being shown are only DRFP and RXNFP, which perform much worse than the proposed method as shown in Table 2. It would be more convincing to see the visualization of MolR, since it is the most competitive baseline.
    - For the molecule property prediction task, the results of ECFP4 and MolR shown in Figure 5 of this paper and the results in MolR's original paper (Table 3 in [1]) are very different. The authors should clarify this mismatch. Besides, it would be more clear to just use a table to list the values in Figure 5 for comparison. 
2. The orders of Figure 2b and Figure 2c can be exchanged to match the orders of the corresponding paragraphs.
3. The authors should double-check if the related papers are correctly referenced. For example, MolR has been already accepted by ICLR 2022 rather than just on arxiv.
4. Typo: In section 2.2.2, "knoledge" --> "knowledge"

[1] Wang, Hongwei, Weijiang Li, Xiaomeng Jin, Kyunghyun Cho, Heng Ji, Jiawei Han, and Martin Burke. "Chemical-Reaction-Aware Molecule Representation Learning." In International Conference on Learning Representations. 2021.
clarity,_quality,_novelty_and_reproducibility: The clarity of the experiments in this paper is not enough. The quality and novelty of the paper are reasonable. The authors do not provide code for reproducibility.
summary_of_the_review: The motivation of the work is strong, and the proposed approach is simple and practical to deal with the previous limitations. However, the experiments need more clear descriptions and convincing comparisons.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a self-supervised representation method that is based on chemical reaction data. The paper points out that existing methods have disadvantages such as abnormal energy flow,  ambiguous embeddings, and sparse embedding space. To address these disadvantages of the existing reaction-based representation learning, they propose to use a chemical synthetic knowledge graph that connects molecules using reaction templates and individual graphs that represent the 2D structure of the molecules.  They claim that the intorduction of the reaction knowleadge graph can elivate the energy flow and the ambiguous embedding problems. Besides, they propose to use of a functional group-augmented SSL method for reaction triplet representation learning which they believe will solve the sparse embedding space problem.
strength_and_weaknesses: Strength:
They consider the issues such as abnormal energy flow and ambiguous embeddings which were neglected before and proposed valid ways to address them.  The constructive loss defined on the knowledge graph seems to be intuitive and interesting.


weakness:
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and the proposed ideas seem to be novel.  However, I think details about the experimental setup are not very well documented in the appendix, publishing the code in the future could be helpful for reproducibility. 
summary_of_the_review: The paper overall is written clearly, but I have some minor comments or questions as following

1. How does the contrastive knowledge embedding
at the graph, level make the triplet embeddings more evenly distributed on the embedding space? also, there is a typo here in the paper knoledge graph instead of the knowledge graph.



2. I am still a bit confused about the intuition behind this: "The main purpose of triplet-level learning is to construct a
smooth latent embedding space for reactants and products while graph-level learning aims to make
triplet embeddings distributed on the embedding space more evenly"

3. In the section titled "Triplet-level Contrastive Learning",  the loss seems to be simply on the augmentations of the reactant and augmentations of the products, not sure why it is called triplet-level contrastive learning, this title is a bit confusing. I was wondering without any template or triplet, just doing the same augmentation on each of the molecules and applying similar loss would have the same effect. Also in the appendix, the latent of this model seems to separate the reactant and product very well, how did this happen? also, do we really want to separate the reactant and product? because a product in this reaction could be a reactant in another reaction I think.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work proposes a molecular representation learning method based on pre-training on a chemical synthetic knowledge graph.
The proposed framework mainly addresses three problems: abnormal energy flow, ambiguous embeddings and sparse embedding space. Experimental part shows that the ReaKE improves the results in various tasks.

Strength of the paper: 
1. The paper has novel contribution by considering the abnormal energy flow, and ambiguous embedding. 
2. The contrastive loss using knowledge graph. 

Weakness of the paper:
1. Limited technical advancement comparing to existing work MoIR.
2. The experiments lack adequate description. Much of the setup is unclear and hard to reproduce in current description. 

In summary, the paper is not ready to publish in the conference yet. 
justification_for_why_not_higher_score: Limited novelty comparing MoIR. Experiment part is unclear.
Decision: Reject