Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper claims that random or learnable graph augmentation methods may inevitably vary molecular semantics and make them inappropriate applied to molecular structure. The authors propose a novel approach to augment the original graph, i.e., converting it to a line graph. The edge attribute fusion tackles the inconsistent representation issue. Two following contrastive losses enhance representation learning.
strength_and_weaknesses: Strength:
1. The paper is well-written and easy to follow. The storyline is clear, and the figures are intuitive.

2. The authors are willing to share the code link after acceptance. The reproducibility of this paper is satisfactory.

3. The experimental setting is very exhaustive. The authors describe almost all the interesting experimental details.

Weakness:
1. The experimental settings are kind of solid but not enough. The experimental section only includes overall performance comparison and ablation study in two different settings. The hyper-parameter sensitivity experiments are missing. And the authors report that the pretraining experiments of their model require 20 hours to converge. I am curious about the efficiency of their method compared with other methods.

2. Some model designs are too simple to come up with. The idea behind the model is kind of interesting. But some model designs are too simple to come up with. For example, the GNN architecture and contrastive loss are similar to other works.

3. The authors claim that edge attribute fusion is essential to fix the inconsistency problem between the representations from two views. But the experimental results shown in figure 3 demonstrate that the model performs worse on half of all the datasets with this attribute fusion.
clarity,_quality,_novelty_and_reproducibility: The reproducibility is good because of the promise of releasing original code and detailed experimental results.
The figures and writing are clear and easy to follow.
The quality of the paper is roughly satisfactory and exceeds the minimum threshold for top conferences. 
The technical novelty is limited.
summary_of_the_review: The paper would benefit from several revisions before publication, including adding some experiments, improving the model design, and checking the necessity of the components carefully.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper fouse on learning good representation for molecuars. The authors introudce line graph to graph contrastive learning and propose two contrastive loss (i.e., inter contrastive loss and intra contrastive loss) to retain molecular semantics. Experiment shows proposed LGCL ahieve SOTA performance on several datasets.
strength_and_weaknesses: **Strength**
Paper is well writen and easy to follow the idea

**Weakness**
Overall, the idea of this paper is not novel and the method propsed by the authors is incremental. 
1. The idea of introducing the line graph of the molecuars to graph representation learning is trivial. It has been explorer in the various GNN papers. Although this paper might be the first one to apply the line graph to the contastive learning, the effort has been made is minor and not significant. It just apply line graph as a contrastive veiws.

2. There is no evidence to show that the proposed intra-local contrastive loss can allivate the oversmooth problem, however, the author claim for that. Need either thertical analysis or empericial result to show this.

3. Some reuslt is much worse than the baseline, can author give some explaination on that.
clarity,_quality,_novelty_and_reproducibility: Clarity: Good, Easy to follow the paper.
Quality: Poor.
Novelty: not novel.
Reproducibility: Not sure.
summary_of_the_review: The author give a trival solution to the learning the represeatation for molecuars. The novelty of this paper is not enough for ICLR.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a pre-training strategy for graph neural networks based on line graphs. The method is then benchmarks on several molecular property prediction tasks.
strength_and_weaknesses: Strengths:
- The work approaches a relevant problem, how to pre-train molecular encoders
- Clearly written and structured paper

Weaknesses:
- The lacks embedding into related works outside of this small community of graph pre-training
- The claimed improvement is hardly relevant since even the most simplistic molecular descriptors (Morgan, ECFP) produce better representations
- Only a single experiment has been performed
- There are technical errors, such as the lack of error bars and confidence intervals, and unclear model and hyperparameter selection 


clarity,_quality,_novelty_and_reproducibility: ## Clarity, Quality, Novelty and Reproducibility

### Clarity
The paper is clearly written and structured and easy to follow.


### Quality

A) The work is not well embedded into related works. It only looks into very closely related works on graph pre-training but not even a bit ahead to other methods for molecular property prediction or to contrastive learning. The authors should embed their work better into related works, and, e.g., check how simple methods and representations usually perform on Tox21[1] or the MoleculeNet benchmarks. 

B) The experimental part is limited and all methods operate at performances that are far below usual approaches to molecular property prediction. The authors only perform experiments for molecular property prediction on the so-called MoleculeNet benchmark. Since the method should generally work for graph pre-training, the authors should also perform experiments on graph that arise in a different domain (e.g. documents or social networks). Furthermore, the usual encoding or representation of molecular graphs are extended-connectivity fingerprints [2] that just encode subgraphs of different size. Using this simple and efficient representation, one can easily outperform the presented method using linear probing (see my table below). Thus, one can save all the computational costs that is used for pre-training the suggested encoder. 
The authors should include baselines into their table (such as linear probing on frequently used molecular descriptors), reference usual performances on these datasets (e.g. in MoleculeNet), and suggest a pre-training method which at least matches the performance of molecular descriptors. 

|          type              |   method                                 | split        | BACE           | BBBP           | ClinTox        | HIV            | MUV            | SIDER          | Tox21          | ToxCast        |
|----------------|--------------------------------|----------|------------|------------|------------|------------|------------|------------|------------|------------|
| Linear probing | ECFP encoder-4096 (reviewer)                 | random   | 91.52±0.0  | 91.25±0.0  | 71.14±0.0  | 80.36±0.0  | 77.05±0.0  | 64.61±0.0  | 76.87±0.0  | 69.43±0.0  |

C) Lack of error bars, confidence intervals and statistical test for many performance metrics. 
The performance metrics in Figure 3 and Table 2 are provided withouth error bars or confidence intervals, such that differences could just arise by chance. The authors should perform repeated training runs, or perform cross-validation to obtain error bars. 

D) Unclear balancing strategies for the loss function and missing hyperparameter selection procedure.
The authors use a combination of three loss functions to pre-train their GNN. However, such combination is usually hard to balance since the losses can be at completely different scales or move to different scales during learning -- a major problem, e.g., in deep reinforcment learning. Suprisingly, in this work the balancing parameters $\alpha$ and $\beta$ could just be set to 1. Generally, it is unclear how the many hyperparameters that come with this architecture (scale parameters, learning rate schedules, GNN architecture, regularization, ...) have been selected. The authors should describe their hyperparameter selection strategy and their metric how to select good hyperparameters. 


### Novelty
The proposed strategy to pre-train graph neural networks appears novel; differences and similarities to other pre-training strategies, e.g. [3], should be described clearer. 


### Reproducibilty
The code has not been provided, such that this work is hardly reproducible. 



### References
[1] Papers with Code, Tox21 (Drug Discovery), https://paperswithcode.com/sota/drug-discovery-on-tox21
[2] Rogers, D., & Hahn, M. (2010). Extended-connectivity fingerprints. Journal of chemical information and modeling, 50(5), 742-754.
[3] Fang, R., Wen, L., Kang, Z., & Liu, J. (2022). Structure-Preserving Graph Representation Learning. arXiv preprint arXiv:2209.00793.
summary_of_the_review: The proposed pre-training strategy is reasonably novel, altough not well embedded into related work. The results and conclusions are hardly relevant since the predictive quality is even below classical molecular descriptors. There are several technical errors in the experimental part.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: **(Motivation:)** This paper points out the limitations of existing graph contrastive learning, which not only alters the semantic of original graphs during perturbation, but also leverages the domain-specific knowledge that might not be generalizable across different graph datasets. 

**(Method:)** To tackle those two problems, the authors propose to use the line graph of the original graph, wherein the edge of the original graph is transformed to the node of the line graph. In particular, the authors maximize the representation similarity of original and its line graph, while minimizing other negative original-line graph pairs, under the existing contrastive learning objective. Also, the authors propose to minimize the inconsistency between node/edge features of the original graph and edge/node features of the line graph, by interchanging their information. Furthermore, the authors propose two additional contrastive losses, which capture edge-level similarities within and between graphs. 

**(Experiment:)** The authors validate the proposed LGCL on the graph classification tasks, showing the propose model outperforms relevant baselines.
strength_and_weaknesses: ### Strengths
* The idea of using line graph transformation for graph contrastive learning, where the positive pair consists of the original and its line graph, is interesting and novel. 
* The problem of information inconsistency between original and line graphs is well justified, which is nicely solved with the proposed edge attribute fusion method. 
* The proposed method empirically outperforms other competitive baselines.
* This paper is well-structured, and easy to follow.

### Weaknesses
* There is a recent work [1] that points out the limitation of contrastive learning: altering the semantics of graphs during perturbation, and this problem-level idea is the same as the authors suggest. Thus, this relevant work [1] should be discussed, and might be compared. 
* The tackled over-smoothing issue incurred by the line graph is unclear. Why the line graph additionally introduces the over-smoothing issue? And how this over-smoothing issue is tackled by the proposed edge-level contrastive losses in Section 4.3 and Section 4.4?
* The experimental setup of Figure 3 is unclear. Does "No Pre-Train w/LG" denote the fine-tuning model that uses representations of both original and line graphs? Then, for the proposed method, is the READOUT function in equation (3) changed to include the line graph representation?

---

[1] Graph Self-supervised Learning with Accurate Discrepancy Learning, NeurIPS 2022.
clarity,_quality,_novelty_and_reproducibility: ### Clarity
* The tackled problems (i.e., existing contrastive learning methods alter the semantic structures of graphs, and also sometimes require the domain-specific knowledge) and proposed methods, based on the line graph-based contrastive learning with edge attribute fusion and local contrastive losses, are clearly described.
* However, the clarity of the over-smoothing issue and the experimental setup of Figure 3 should be further improved (See Weaknesses for details). 

### Quality
* The technical quality of this work is high. The authors define the under-explored challenges of the existing contrastive learning, and tackle them with the novel line graph-based contrastive learning.
* The experimental results show that the proposed LGCL outperforms baselines, which further supports the technical quality of the proposed idea.

### Novelty
* To my knowledge, the main contrastive learning objective based on the similarity learning between original and line graphs is novel. 
* In addition to the main objective, the proposed additional components, such as edge attribute fusion scheme and local contrastive losses, are novel. 

### Reproducibility
* The authors do not provide the source code that lowers the reproducibility of this paper; however, the authors plan to release the source code after the acceptance. Therefore, the reproducibility will be probably high.
summary_of_the_review: While there are some revision points: discussing more work and improving clarify of some sections, I believe they require minor revisions. Thus, given the high quality and novelty of this work with strong empirical results, I recommend the acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new framework for molecular property prediction. The proposed framework consists of (1) a new edge attribute fusion-based graph neural network architecture and (2) three pretraining objectives. The newly proposed GNN architecture additionally uses line graph-based view of a molecule as an additional source of information. Furthermore, the contrastive learning objectives consist of (1) contrastive learning between graph-based and line graph-based molecular representations, (2) intra-contrastive learning between edge representations, and (3) inter-contrastive learning between edge representations. After combining all the components, the proposed method outperforms the existing pretraining frameworks for molecular property prediction.
strength_and_weaknesses: Strength 1: The idea of using line graph as an additional view for a molecule is novel for pretraining. 

Strength 2: The proposed method shows strong empirical performance.

Weakness 1: Most importantly, the proposed method is combination of many ideas, but the current experiments are insufficient to demonstrate significance of such ideas. 

Weakness 1.1: In the ablation studies, simply combining line graph and contrastive learning achieves average performance of 71.16, which is worse than six of the considered baselines. 

Weakness 1.2: After combining edge attribute fusion, the proposed method generates property prediction using different neural network architecture compared to the baselines. If this is the case, the authors should consider the same architecture for the baselines too.

Weakness 1.3: The proposed method is based on combination of three loss functions, while most of the baselines rely on one loss function. To measure the significance of these loss functions, the experiments should be constructed more carefully. For example, one could fix the budget for tuning the hyperparameters and try to combine the previously proposed pretraining objectives.

Weakness 2: Table 2 does not have any “overall” statistics like ranking and average, so it is hard to interpret the table and make conclusion.  

Weakness 3: I suggest comparing with 3D-infomax [1] and MGSSL [2] as a pretraining baseline. I also suggest comparing with works that simultaneously propose graph neural network (or Transformer) architectures with pretraining objective (GROVER [3], MolR [4]). Comparing with other architecture is important since the work also proposes a new architecture for property prediction. 

Weakness 4: I appreciate how the authors showed the candidate hyperparameters for the finetuning experiments. However, there is no detail on which metric is used to choose the hyperparameters. Furthermore, it is not clear whether if the budget used for tuning the pretraining hyperaparameter is the comparable across the baselines. 

[1] 3D Infomax improves GNNs for Molecular Property Prediction, ICML 2022
[2] Motif-based Graph Self-Supervised Learning for Molecular Property Prediction, ICML 2022
[3] Self-Supervised Graph Transformer on Large-Scale Molecular Data, NeurIPS 2020
[4] Chemical-Reaction-Aware Molecule Representation Learning, ICLR 2022
clarity,_quality,_novelty_and_reproducibility: The paper is clear to read and the proposed idea is novel with respect to the existing baselines. Reproducibility of this work will be high if the authors publish their code given acceptance (along with the protocols for selecting the hyperparameters). 
summary_of_the_review: This paper proposes an interesting idea for pretraining of molecular representations. However, I am concerned by the empirical evaluation which is not thorough enough to validate the significance of the proposed work.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper has been evaluated by five reviewers who scored the paper as 3,5,6,6,8. Reviewers liked the paper but also had a number of concerns.

Firstly, they had concerns about the lack of strong motivation for Eq. 10 and 11, or the use of the line graph. Reviewers acknowledged the line graph is popular in molecular graphs but also noted that it was an arbitrary choice with a limited theoretical support why it would be the best choice for the second view of the original graph. It was not clear if other choices for forming views are really as problematic as authors claimed. One of reviewers had a concern also about inferior results compared with ECFP+linear probing and ECFP+MLP based on results from Honda, S., Shi, S., & Ueda, H. R. (2019). Smiles transformer: Pre-trained molecular fingerprint for low data drug discovery. arXiv preprint arXiv:1911.04738. From minor issues, some reviewers noted they had concerns about the statistical significance of results in Table 1 as well as inconsistency of protocols which should have been indicated for clarity: (random/scaffold) and the validation procedure (train/val/test or k-fold cross-validation). 

The above findings were discussed during the AC-reviewers meeting.

On balance, the paper requires another round of reviews and is not ready for publication.
justification_for_why_not_higher_score: After rebuttal and AC-reviewer meeting, a number of concerns remained as explained above.
Decision: Reject