Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a topology optimization method for message passing-based graph learning models, termed Graph Learning Attention Mechanism (GLAM). The proposed model is differentiable and removes edges in an end-to-end manner. The method can be integrated into GCN and GAT. The experimental results show the proposed GLAM can improve the performance of GCNs.
strength_and_weaknesses: Strengths,

S1: The proposed topology optimization method is differentiable;

S2: This paper studies an interesting problem;

S3: This paper is well-written.

Weaknesses,

W1: There is a lack of evidence to support the analysis of the conflations in Section 3, which weakens the motivation of this paper.

W2: Topology optimization is not new in graph learning, especially edge movement.

W3: The improvement from GLAM seems limited in experiments. And the empirical support for the effectiveness of GLAM is weak.

clarity,_quality,_novelty_and_reproducibility: This paper is well-written. Topology optimization is not new in graph learning, especially edge movement. The paper points out a conflict between node embedding and structure learning, which seems new but not convincing.
summary_of_the_review: I recommend a weak reject because of the weak motivation and validation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work aims to tackle the challenges of noisy edges in graph learning problems. It first points out the finding that the demands of structure learning and node embedding are inherently conflicting with each other, so it is not proper to optimize both jointly.  Then, based on their finding, they propose the Graph Learning Attention Mechanism （GLAM） layer, which is a non-heuristic differentiable network layer that can be inserted into existing GNN models and can focus on the graph structure learning objective to learn a task-informed graph structure and benefit the graph learning. 
strength_and_weaknesses: Strength:
1. This work is clearly motivated, and the studied topic is meaningful.
2. The writing of this work is easy-to-follow.

Weakness:
1. Model Evaluation
1) The proposed GLAM is only examined with 3 baselines, which I think is not enough. 
- The authors mentioned that "GSAT was not evaluated on our datasets", I am wondering why GSAT is excluded.
- I think it would be better if the authors can also compare with more challenging baselines, especially some SOTA models on structure learning.
2) no ablation study is provided to examine the GLAM method comprehensively, for some examples, I am curious about the following aspects:
- is it necessary to apply a GLAM layer before every GNN layer? 
- the authors claim that the GLAM can be incorporated with various GNN backbones, however, only an example with GAT is provided. I am curious to see how GLAM performs on other GNN models, such as GCN.

2. Reproducibility
- Though the authors provide the section to clarify the reproducibility and the experimental set-up details claimed to be provided in the code, the code link is missing. So essentially no useful information is provided in the paper, which makes the reproducibility fair.


Some questions:
1. In equations (5) and (6), what is the major benefit of adding the noise term u? is it for the Gumble softmax trick for differentiability?
2. if we apply GAT after GLAM, it seems to me that we are still jointly considering the structure learning and node embedding tasks simultaneously as the model still uses the weight on each edge learned by GAT for message passing. Does it contradict the claim that these two objectives should be considered separately?
3. It looks to me that, the major drawback of GAT is the softmax which expresses the neighbors' importance relatively to other nodes. Then, does it mean, as long as we come up with a way to get rid of this softmax, the limitation can be solved? 
4. In section 5, I don't quite understand, why it would be problematic to add penalties on retained edges?
5. In section 6.2, the authors claims one of the benefits of GLAM is sparsification, I am wondering, is this sparsification mainly for efficiency purpose? If so, then how much speed/memory improvement can this sparsification bring to GAT? If not, then why is sparsification important here?
clarity,_quality,_novelty_and_reproducibility: Clarity: Good

Quality: Fair
- I think the evaluation of the proposed method is not good enough, which makes the quality of this work fair. Please refer to the weakness for more details. 

Novelty: Fine (between Fair and Good)
- I appreciate the observation that the structure learning and node embedding, as claimed in the paper, this has never been done before. I feel this looks like the major novelty of this work.
- The proposed method looks very similar to GAT, but replacing the LeakyReLU in eq.(1) and softmax in eq.(2) with the plus noise and sigmoid in eq.(5), I don't think this part has enough novelty.

Reproducibility: Fair
- Please refer to the weakness for more details.

summary_of_the_review: This is a well-motivated work focusing on how to separate the structure learning and node embedding task to best solve the graph learning problems. Though the writing is clear and the explored topic is meaningful, I have major concerns about the method evaluation. First, it is not compared with comprehensive suitable baselines, and the benefit of the proposed method is not distinctive; second, the authors only provide one simple experimental result (in table 2), and no ablation studies are provided to examine the GLAM method comprehensively; another minor point is, neither the code link nor details of the experimental set-up are provided.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper addresses issues of GNN sensitivity to graph structures. Specifically how noisy edges degrade performance which cannot be mitigated without complete removal. This paper proposed a novel layer Graph Learning Attention Mechanism (GLAM) which isolates structure learning from node embeddings. This technique does not require using costly exogenous structural regularizers or edge-selection heuristics to learn optimal graph structures. Experiments on benchmark datasets including Cora, PubMed, Citeseer, Amazon Photo, and Amazon Computers show comparable classification performance with significantly higher sparsity in structure.
strength_and_weaknesses: The experiments indicate the GLAM layer when applied achieves comparable prediction accuracies to state-of-the-art methods on benchmark datasets but with a significantly sparser graph structure. 

Since the experiments do not show great improvement in prediction accuracies on benchmark datasets, the justification for using this method would rely on applications that can take full advantage of the edge removal capabilities in this paper.

Future experiments can uncover the reasoning behind the drastic differences in the number of removed edges between the first and second GLAM layers. 

During experiments, each dataset has a drastically different % of edges removed during training. Further research may be able to explain this occurrence.

Determining cutoffs for low homophily thresholds for usable datasets can open the door for applications to more diverse datasets and applications.

Does the GLAM layer offer any runtime benefits during training?

clarity,_quality,_novelty_and_reproducibility: This paper is well-written and organized with clearly defined objectives and motivations.

This work builds on existing methods including Graph Attention Networks (GAT). In breaking down the GLAM method, the similarity and novelty of this method from GAT are clearly noted. 

The authors take note of the constraints on the datasets that would benefit from applying the GLAM layer for training.

Source code and setup scripts are provided but redacted for review.

summary_of_the_review: The work in this paper is good with promising results in an interesting direction beyond plainly increasing prediction accuracy on benchmark datasets. Additional work may be needed to expand upon this research for publication.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work presents a graph structure refinement layer named GLAM based on a neighborhood attention schema slightly different from the Softmax attention, which disentangles the optimization of node embeddings and graph structures. The proposed GLAM layer learns more sparse subgraphs than prior arts by giving the known adjacency matrix.
strength_and_weaknesses: Pros:
- The insight of the addressing the conflicts between structure learning and node embedding optimization is interesting
- Simplicity of the methodology
- The paper is well written and easy to follow despite some confusing points

Cons:
- Lack of a more principled motivations and justifications behind the proposed design
- Limited evaluation setup
- Limited technical novelty

clarity,_quality,_novelty_and_reproducibility: Overall, this paper presents an attention mechanism slightly different from the Softmax attention as the graph structure learner extracts sparse subgraphs to be effective topological information to facilitate node embedding learning. I like the idea of disentangling the learning of graph structure and embeddings, but the analysis is less insightful and lacks more principled discussion in Sec. 3. Also, I do not see significant breakthroughs and novelty for the composition of the GLAM layer. For the claimed advantages, there are no substantial improvements or clear justifications in experiments.
summary_of_the_review: Following my comment above, there are several questions:

- What are the exact conflicts between the learning of graph structures and node embeddings? Specifically, in Sec. 3, I have these questions: (1) What are the roles of the permutation invariant aggregator in hindering the use of $\alpha_{ij}$ as the bases of structure learning? (2) I partially agree with your analysis about the Softmax attention, but how to get your conclusion of “For these reasons, …, we should not do away with the Softmax activation to normalize attention coefficients…”? (3) I cannot see your principled analysis to answer why it is not desired to learn edge weights together with node embeddings, e.g., how do you estimate the noisy evolution of local neighbourhoods, and why the improvements over GAT are less significant if your conjectures hold?

- I notice the authors mentioned graph sparsity several times, especially in Sec. 6, but there is no experiment to discuss the consequent benefits, e.g., efficiency and scalability. Have you evaluated the proposal on some larger real-world graphs, e.g., OGB datasets?
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the graph structure learning problem where noisy edges are to be removed to faciliate node embedding learning. It proposes that structure learning and node embedding should be separated as they conflict with each other, which is generally appreciated by the reviewers and me as a good point. However, the current paper suffers from limited evaluation, insignificant empirical improvement, lack of deeper theoretical insights, etc. All reviewers suggest rejecting the paper.
justification_for_why_not_higher_score: It is a unanimous reject by all reviewers, and the author responses did not add new experiments or address the concerns.
Decision: Reject