Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper propose the Heterogeneous Motif Graph Neural Network (HM-GNN) which is based on a motif level graph representation that takes into account commonly occurring motifs like rings in molecules. They apply this to multi-task settings and obtained good experimental performances. They propose an edge sampling scheme to reduce the computational costs needed for training. 
main_review: The core strength of the paper is the strong experimental results. They obtained results that are much better than existing baselines on graph classifications. In terms of their ideas I find their TD-IDF approach natural, and I find the idea of taking into account molecular motifs very appealing and natural as well. 

There are several concerns that I believe the authors should address for this paper to be of publishable quality: 

1. One of my major concerns is that the authors did not provide a sufficiently detailed/rigorous description of the HM-GNN model/any of the related procedures like the edge sampling. The high level description of the model is based on very two short paragraphs in section 2.3, as well as a graphical illustration in figure 3. Lots of details of the model are left out/undescribed. I expect at the very least some pseudocode to describe some details, i.e. the neural network architecture/training, the initialization etc. The GNN for the atom level embedding is also glossed over. I believe that for publication in a venue like ICLR, the authors should at the very least give a more precise and detailed description of the model that they are proposing. There is only a very short appendix provided and many details (e.g. details on the proportion that they used for edge sampling etc)  are nowhere to be found unless the reader delve into their code. 

2. Another one of my concerns is about the Edge sampling scheme. I have no doubts that it will actually speed things up (the complexity depends on the number of edges, so of course if we only select a subset of the edges the process is sped up). What the authors did not provide is any kind of justification, whether from a mathematical or from a molecular perspective, for why they picked this scheme/how they know their scheme isn't deleting important information? The authors propose to "randomly select some molecular nodes as starting nodes" (how many nodes? why random? Wouldn't some nodes be more important than others, especially in molecular settings?) and conduct a BFS where they sample a fixed portion of edges hop by hop (why fixed proportion? how does one pick the proportion?) I understand this is a heuristic that works well in practice but I expect a little more justification and details.  

Overall, I will say that this is a paper with a lot of potential, and many of the ideas are likely to be useful in graph machine learning and molecular representation learning in general. However, the authors wrote the paper in a way where many of the core procedures are described in short paragraphs via words, and where lots of the important details, design decisions, hyperparameters etc are completely left out and are not sufficiently motivated/justified. Similarly ambiguous language is used in the experimental section. I would highly suggest the authors to include more precise descriptions and details in the paper, and to provide more justifications to their heuristics. 
summary_of_the_review: A paper with a lot of potential, but lots of important details are left out and lots of decisions are stated without sufficient justifications. As of right now, this paper is not in publishable state. After some revision however and some improvement in their exposition, I believe this can be a good paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Aiming at the feature representation learning problem of molecular graphs, this paper proposes a heterogeneous motif graph composed of motif nodes and molecular nodes to realize the information interaction between motifs and molecules. The paper uses graph neural networks to learn and combine atom-level and motif-level graph feature representations, so as to improve the representation of molecules.
main_review: ### Strengths
The paper proposes a method of constructing a heterogeneous graph based on motif, which can effectively utilize motif-based features.

### Weaknesses

The proposed method of the paper is interesting, but the baseline for comparison is too weak. For example, the two datasets of OGB shown in Table 2 compare the methods that are ranked very low on the OGB leaderboard, so it is difficult to prove the power of the method. At the same time, the paper did not give a detailed explanation on why GIN was chosen as the feature representation of the learning graph, and from the results in Table 1, GIN does not have obvious advantages compared with other methods. And when doing atom-level feature embedding learning, the edge feature (Bond Type, etc.) is not used, and the edge feature should have a certain degree of contribution to the prediction of molecular properties.

summary_of_the_review: The motif-based heterogeneous graph construction method proposed in this paper is interesting, but the experimental results are difficult to prove the effectiveness of the method. The network architecture is slightly simpler, the chemical bond information of the molecules is not used in the atom-level feature extraction. Therefore, I am not in favor of recommending this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel molecular graph representation learning method by constructing a heterogeneous motif graph. In this graph, molecules and motifs are considered as nodes, and thus forming a heterogeneous graph. Motifs are extracted manually and important motifs are selected by TF-IDF. Moreover, the paper demonstrates that a multi-task learning framework is beneficial, thanks to sharing motifs across different datasets.
main_review: Pros
- The idea of adopting multi-task learning via heterogenous motif graph to address lack of data problem of deep learning for molecular data is interesting. Moreover, the empirical performance through multi-task learning is promising.
- The paper is well organized.

Cons
1. The paper lacks a clear justification on whether two molecules sharing motifs imply special meaning.
	- In other words, is sharing a motif a clear indication that two molecules share common properties? 
	- In addition, a motif-motif edge is generated if at least one atom is shared. Does sharing an atom imply anything chemically?

2. The exact message passing process needs to be described. For example, the sizes of the feature vectors of a motif node and a molecular node should be different, and how they are put together for message passing is not clear.

3. Why are there blanks in Table 1 regarding "MUTAGENICITY."

4. Regarding the edge sampler, 
	- Is BFS done until all the nodes are covered?
	- How efficient is HM-GNN compared with GIN in terms of memory consumption? 
	- The edge sampler seems to be not much different from what is done in GraphSAGE.

5. The literature survey is not sufficient. 
	- There are many recent work on GNNS for molecular graphs especially along the line of self-supervised learning, such as [1,2,3,4]. HM-GNN should be compared with a few methods among them, if not, they should be mentioned in the related work.
	- Moreover, since this is not the first paper to consider "motifs" for GNNs, related work, such as [5,6,7,8] should have been described in the paper.


[1] MoCL: Contrastive Learning on Molecular Graphs with Multi-level Domain Knowledge

[2] InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization

[3] Contrastive Multi-View Representation Learning on Graphs

[4] Self-Supervised Graph Transformer on Large-Scale Molecular Data

[5] Graph Convolutional Networks with Motif-based Attention

[6] Hierarchical Generation of Molecular Graphs using Structural Motifs

[7] Gnnexplainer: Generating explanations for graph neural networks

[8] Motif-Driven Contrastive Learning of Graph Representations

6. Minor comments
	- For reproducibility, best performing hyperparameters could be mentioned.
	- The start of the sentence should be capitalized.

summary_of_the_review: Overall, I think this paper is at the borderline. Although the paper is among the first to use motif for molecular graph representation learning, the paper lacks sufficient literature survey. Moreover, it would have been better if the paper had had discussions about why motifs are helpful in a data-driven perspective.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes learning molecule representations by using motif-level information. The authors construct a heterogeneous graph which consists of molecules and motifs, then learn representations of them using graph neural networks. The learned features are concatenated with the molecule feature learned from a traditional atom-level graph neural networks and fed into an MLP for property prediction.
main_review: Strengths:
1. The idea of constructing a motif-molecule heterogeneous graph is new;
2. The experimental results show that the proposed method can improve the performance significantly;
3. The paper is basically well-written and easy to follow.

Weaknesses:
1. Motif has been explored extensively in the literature, which makes the contribution of novelty for this paper is marginal;
2. Though the idea of constructing a motif-molecule heterogeneous graph is new, it seem unnecessary to introduce motif nodes in this graph;
3. The technical contribution of this paper includes three parts, but the three parts are not necessarily related to each other, which makes the paper very "loose" and more like a combination of small tricks;
4. The part of edge sampling seems trivial;
5. The authors do not test their method on some datasets that are commonly used for molecule property prediction, which makes it hard to comprehensively compare their method with the literature;
6. Some of the technical details are not clearly stated (see below for detail);
7. The quality of figures can be improved.


Details for weaknesses:
1. This paper studies the problem of using motifs (functional groups) in molecule representation learning. However, it is well known that molecule motif is an essential part of molecule structure, which have been extensively explored in the literature.
2. In the molecule-motif graph constructed by the authors, there are motif-motif links (co-occurrence relation) and motif-molecule links (part-whole relation), but no molecule-molecule links. However, motifs are more like a bridge connecting molecules in this big graph, and what the authors finally use is only molecule representation. If this is the case, it seems more direct and reasonable to build a molecule-molecule graph, whose edge weights are calculated based on the motifs they consists of, then the message can pass directly among molecules in GNN. I think the authors can at least take this as a baseline method? Moreover, it is also interesting to investigate if there is any theoretical relationship between the two graphs in terms of running GNN on them.
3. The technical contribution of this paper consists of three parts: constructing a molecule-motif graph, multi-task learning framework, and edge sampling. However, they are not necessarily related to each other. In other words, even if you do not use the molecule-motif graph, you can still build another molecule-molecule graph and use your multi-task and edge sampling strategy on top of this graph. The three parts of contributions are loosely coupled together, making this paper more like a combination of separate tricks.
4. The part of edge sampling seems trivial to me: of course you can randomly delete some edges from a graph to improve the time efficiency, but you do loss information from the original graph. Therefore, it is always a trade-off, which should be discussed in the paper. Moreover, it is also weird why the authors highlight a BFS method on sampling edges: It is easy to implement a method that directly removes edges from the graph while ensuring that all nodes have at least one edge left.
5. There are a lot of molecule property datasets proposed in the literature, for example, BBBP, BACE, HIV, Tox21, ToxCast, SIDER, QM9, ESOL, FreeSolv, etc, but the authors do not test their methods on these commonly used datasets.
6. Some of the technical details are not clearly stated: (1) In section 2.1 the authors say that "By sorting the vocabulary by TF-IDF, we keep the most essential motifs as our final vocabulary", but TF-IDF measures the importance of a motif to a specific molecule, and it is not an independent feature of motifs. How you get the TF-IDF value of motifs? Did you average the TF-IDF values of a motif to all molecules and treat this as the TF-IDF value of this motif? (2) In section 2.2, the authors construct a heterogeneous graph and define edge weights as in Eq. (1), however, PMI could be positive or negative, and TF-IDF can only be positive. How do you deal with negative PMI values in GNN? Should these two type of edge weights be normalized or calibrated to make sure that they are at the same scale in the unified graph?
7. Though the figures in this paper are clear and easy to read, I suggest that the authors reorganize (do not leave too blanks), resize (make sure that texts and shapes across these figures are basically of the same size), and recolor the graphs (do not use colors that are too bright, try to color your graphs using the same style) to make them more beautiful and professional.
summary_of_the_review: In summary, I think the idea of using motif in molecule representation learning is interesting, but the propose method has a lot of design issues that are not well-justified or well-explained. Moreover, the proposed method is not tested on commonly-used datasets. Overall, I think this paper is below the borderline and I tend to reject this paper.
review_title: null

Meta-review: The paper introduces a graph neural network for molecules which takes into account motif-level relationships. The paper received borderline reviews, with three reviewers voting for reject, and one for accept.  After the rebuttal, the reviewers did not change their scores. Overall, it seems that the paper has some merit, with good experimental results. Nevertheless, it suffers from two issues (i) the positioning with respect to other motif-based approaches is not clear enough, making the novelty hard to assess; (ii) there is a lot of room for improvement in terms of clarity. Therefore, the area chair follows the majority of the reviewers' recommendations and recommends a reject.
Decision: Reject