Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper targets the problem that the edges in a graph may have different levels of difficulty to learn and proposes a curriculum learning-based model to gradually incorporate edges during learning according to their difficulties. The difficulty level is obtained by self-supervised learning. Experiments are conducted on 9 synthetic datasets (with 9 different homophily coefficients) and 7 real-world datasets.
strength_and_weaknesses: 
Strengths:

1. The paper is overall easy to understand. The motivation and the background knowledge of the curriculum learning are clearly introduced.

2. The idea to achieve better learning performance on graphs by investigating the difficulty of the edges is novel.

3. In experiments, both synthetic datasets and real-world datasets are used. The synthetic datasets have controllable properties (i.e. homophily coefficient).


Weakness:

1. The main idea is pretty intuitive. The difficulty level of the edges is only intuitively proposed without concrete examples. Therefore, it is unclear what exactly the difficulty levels mean. Moreover, the noise seems to be one component of the difficulty, is there any other concrete factors determining the difficulty?  

2. i.d.d data is not correctly used. What the authors want to express through i.d.d data should be 'independent data', in which the data have no connections. While i.d.d. data refers to different samples drawn independently from identical distributions.

3. In experiments, it is unclear why the difficulty of the edges can correspond to the formation probability.

4. The datasets are rather small. 

5. The proposed method is not consistently better than the baselines in all datasets. Besides, is there any experiment that can demonstrate that it is indeed the curriculum learning part that improves the performance, instead of other factors like model complexity?

clarity,_quality,_novelty_and_reproducibility: The clarity is OK.

The quality is questionable since some key points are not clear.

Novelty is good.

Reproducibility is unclear, and can only be checked by running the code.

summary_of_the_review: Overall, this paper has a novel idea, but the proposed model is not totally clear. Besides, misusing the concepts like i.d.d. makes the paper not rigorous enough.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a method for iterative construction of graph structure in GNN settings. The key is to make use of the homophily -- similar nodes tend to connect. The optimization procedures encourage more edges to be added along the training process. This results in a better optimized GNNs -- boosting up accuracy performance and improved robustness against structural noise.
strength_and_weaknesses: Strength
======
- The paper studies an interesting problem -- optimizing the parameter and graph structure jointly.
- The proposed solution is sensible and is shown empirically to improve performance and robustness.

Weaknesses
==========
- Posing the forward edge selection as curriculum learning is somewhat confusing. It is better to emphasize on the joint optimization aspects.
- Again, adding terms like self-paced, self-supervised learning are also confusing.
- Although homophily is a well-known properties of real-world graphs, it would be interesting to see if this assumption breaks, e.g., redesigning the kernel for graph reconstruction.
clarity,_quality,_novelty_and_reproducibility: The paper is a nice addition to the growing literature of GNNs, emphasizing on structure learning. Overall it is a readable paper, although added concepts like curriculum learning, self-paced learning and self-supervised learning make the messages unclear.
summary_of_the_review: An useful attention to the literature of graph structure learning in GNNs. I'd like to see when the homophily assumption breaks and how the method will be adjusted.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates curriculum learning for graph neural networks (GNNs), aiming at selecting the most important edges in a graph for GNN training. The authors formalize the task as an optimization problem, which has a node-level predictive loss and a structural-penalty loss. A proximal algorithm is proposed for optimization. Experimental results on multiple datasets are encouraging.
strength_and_weaknesses: Strengths:
1. The problem of learning graph structures for GNN is important.
2. The proposed method uses the node representations learned by GNNs to measure the likelihood of each edge, which is quite intuitive.
3. Some improvements are observed in the experiment.

Weaknesses:
1. The writing of the paper can be further improved.
2. The improvements over existing methods are not significant.

Detailed comments:

1. The writing of the paper can be further improved.
The abstract of the paper says that a curriculum learning method is proposed, where edges are given to GNNs according to their difficulty from easy to hard, and the difficulty is measured by a self-supervised learning paradigm. However, in the model section, these claims are not well explained. The edge difficulty and self-supervised learning paradigm are not mentioned. Although readers can roughly get the idea that the edge difficulty is measured by the inner product between corresponding node embeddings, I feel like the abstract and model sections are written by different authors, making it not easy to follow.

2. Many ideas of the paper are not new
The overall loss function of the proposed method is given by Eq. (3) of Algorithm 1. There are three parts, where the first part is a supervised loss, the second part is a graph reconstruction loss, and the third one is a regularization term. The first two terms are widely studied in the graph machine learning literature for GNN training and structure learning. Given these existing works, I feel like this paper does not bring new ideas. Although the authors introduce their method from a perspective of curriculum learning and some smoothing strategies are proposed in Sec. 4.3, I think the contribution is not so significant as the underlying ideas are not so different from existing methods.

3. The improvements over existing methods are not very significant.
The results on real-world datasets are presented in Tab. 2, where the improvements over existing methods are not so significant, and the results on different datasets all have very high standard deviation. To better convince readers, I think it is helpful to consider some larger datasets for evaluation, where the standard deviation over different runs can be smaller.
clarity,_quality,_novelty_and_reproducibility: See the comments above.
summary_of_the_review: See the comments above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces a curriculum learning strategy for GNNs by explicitly modeling the problem that edges in a graph may have different levels of difficulty to learn. It introduces a curriculum learning-based model to gradually incorporate edges during learning according to their difficulties. The difficulty level is obtained by self-supervised learning. 

**Strengths:**
* Experiments are conducted on 9 synthetic datasets (with 9 different homophily coefficients) and 7 real-world datasets. However, the datasets are rather small.
* The paper is overall easy to understand. The motivation and the background knowledge of the curriculum learning are clearly introduced.

**Weaknesses:**
* Reviewers were concerned that the datasets are rather small. In response to this comment, the authors added additional experiments to evaluate the effectiveness on two larger datasets (ogbn-proteins and ogbn-arxiv). 
* Reviewers also noted that it would be interesting to see how the model behaves when the graph homophily assumption breaks. In the rebuttal phase, the authors successfully clarified the method does not require a homophily assumption. It is a general learning framework, and any non-parametric kernels can be utilized in determining how well an edge is expected by its connecting nodes embeddings. 
* Many ideas in the paper are not new. The overall loss function has three parts, where the first part is a supervised loss, the second part is a graph reconstruction loss, and the third one is a regularization term.  Although the authors introduce their method from a perspective of curriculum learning, Reviewers agree the contribution is limited as underlying ideas are similar to existing methods.
* The improvements over existing methods are not significant. The method does not perform consistently better than baselines on real-world datasets; the results on different datasets all have a high standard deviation. 

The reviewers acknowledge the authors' rebuttal, including new experimental results added during the rebuttal. The rebuttal addressed some of the key concerns on the model performance. However, this paper needs further improvement. I suggest the authors address criticisms raised by the reviewers and refine the manuscript for a more solid publication.

justification_for_why_not_higher_score: See the above-mentioned weaknesses.
Decision: Reject