Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper addresses an important problem of OOD detection and generalization for GNNs. An Energy-based discriminatory framework is explored that is shown to give a some significant advantages like model agnosticity, theoretical gaurantees and practically efficacy. This application and study of energy-based OOD detection methodology for inter-dependent data like graphs has not been done before and is novel. 

They have started off with a simple instance-based energy function used to detect OOD samples, based on negative-log-likelihood loss. The authors show how OOD detection can be enhanced by propagating the energies calculated across neighbors for K-steps in a manner typical to message propagation in GNNs. Further, they have added a regularizing loss to this formulation that induces an energy boundary between in-distribution and OOD instances. 

The experiments compare their variants with image-based and graph-based OOD competitors, with standard choice of metrics for detecting OOD and for calibrating accuracy. They show significant improvements in the OOD detection metrics without trading off the supervised learning of the models by too much. Their results improve upon previous work on OOD detection by a significant margin.

strength_and_weaknesses: Strengths:
- The results seem significant over other state-of-the-art in OOD detection and generalization on graphs.
- Broad choice of GNN encoder backbones, competitor architectures, datasets and experimentation strategies. Multiple ablation studies included.
- Comparison of training times and inference times for all the models used included
- Each variant of their proposed models are compared with each other as well as competitors and a good summary/explanation of each of the experimental data is given.
- Have posited some insights on when regularization may be significant and when they may not help in OOD detection well citing examples from experiments.

Weaknesses:
- The gaps between methods that have access to OOD data to tune on and those that don't is significant. It would be helpful to better delineate these settings in the tables and results in the paper since the conditions in both cases is quite different.
- Provide further justification and rationale for the choice of methodology for creating synthetic OOD data, especially the label leave-out way in datasets like amazon-photos, cora and coauthor-cs.
- The explanation of how their approach is used to regularize learning in the supervised setting is not entirely clear.

​Typos
- Appendix B.1 in Co-author CS:
    - 6805 classes -> 6805 features

clarity,_quality,_novelty_and_reproducibility: The paper is relatively clear and provides a number of significant advances over other approaches for OOD detection on graphs.
summary_of_the_review: This paper provides a new approach for OOD detection and generalization on graphs that incorporates the graph's topology in an energy-based message passing scheme. The approach is well described and the rationale makes sense. The authors provide strong evidence that their approach improves generalization on a number of OOD tasks established on different graph benchmark datasets in both an unsupervised and semi-supervised setting.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed an intrinsic out-of-distribution (OOD) discriminator for semi-supervised learning on graphs based on an energy function called GNN-Safe, to enhance the reliability of the model against OOD data. The authors consider the interdependence between input data and other nodes in the graph rather than treating them independently. Experiments are compared with other models on five real-world datasets.
strength_and_weaknesses: **Strengths:**
1. The experiments in this paper are detailed in comparison with different methods under different datasets and GNN encoders.
2. This paper is well-written and easy to understand.
3. The authors provide proof for all theoretical results.

**Weaknesses:**
1. In the Introduction, it is said that “The scarcity of labeled training data in graph domain requires the model to exploit useful information from the unlabeled portion of observed data.” This statement is confusing because it is not the actual reason why semi-supervised learning is widely used in graph-based learning and performs well [1]. However, this paper claims to use semi-supervised learning to solve the problem of limited labeled data in graph datasets.
2. There is no detailed description of the evaluation metrics. In addition, GPN compares OOD-Acc for different methods, which is also an important metric to evaluate the performance of OOD data [2].
3. In figure 1, compared with chart (a), the distribution center of the green curve (in-distribution) is confusing in chart (b).
4. In Section 4.1, it is said that “use the subgraph DE as in-distribution data and other five subgraphs as OOD data for Twitch.” However, the authors don’t explain why different subgraphs follow different distributions.
 
[1] Graph-based Semi-supervised Learning: A review. Neurocomputing, 2020.  
[2] Graph Posterior Network: Bayesian Predictive Uncertainty for Node Classification. NeurIPS, 2021.  

clarity,_quality,_novelty_and_reproducibility: The motivation of the problem is clearly described in the introduction, and the proposed approach is easy to understand and well supported by several experiments and theoretical proof. The paper focuses on a novel energy-based belief propagation method for boosting OOD detection performance, which is meaningful. The experiment settings are detailed. However, there is no code in the supplementary materials.

summary_of_the_review: This paper is easy to read and proposes a novel approach based on the energy function to detect OOD data. Overall, this work is meaningful in OOD detection on graphs and can be further studied and optimized.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed an energy-based out-of-distribution detection for graph neural network. The contributions include introducing energy-based method to graph out-of-distribution detection, providing some theoretical analysis, presenting energy propagation, and conducting a lot of experiments.
strength_and_weaknesses: Strength:
1. The paper is well-written and the motivation is clear.
2. The numerical results showed that the proposed two methods especially GNN-Safe-r are much better than the baselines.

Weakness:
1. The main idea is almost the same as that of Energy FT (Liu et al. 2020). It seems that the major difference is the proposed methods are for graph neural network rather than MLP and CNN.

2. In the experiments, what is the main difference in terms of the implementation between Energy FT and the proposed GNN-Safe-r?

3. One can observe that in many cases, GNN-Safe was outperformed by Energy FT, which verified the importance of OOD exposure in the training stage. Thus, one may train a binary classifier to classify the in-distribution samples and out-distribution samples into two different classes, which should have a good performance in the testing stage.

4. The OOD exposure violates the principle of OOD detection because it becomes a supervised learning problem with OOD labels.
clarity,_quality,_novelty_and_reproducibility: The clarity and quality are good. However, considering the previous work of Liu et al. 2020, the novelty is not significant enough.
summary_of_the_review: It seems that the main improvement over existing work such as Liu et al 2020 is that the proposed method focuses on GNN. Thus the contribution is not that significant. On the other hand, the numerical results indicate that OOD exposure plays a very important role in the success of the proposed method. It is not clear whether a direct binary classification between in-distribution and out-distribution can yield good OOD performance or not.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a new mechanism based on the energy-models for the task of out of distribution detection on graphs. They show that the graph neural networks trained with supervised loss objective can be intrinsically effective to detect OOD data on which the model should avoid prediction. The energy based method proposed in this work can be directly extracted through simple transformation from the predicted logits of a GNN classifier. Furthermore, the proposed method is agnostic to the choice of the underlying architecture used making it highly generalizable. This is further enhanced using a simple propagation scheme, following the popular label propagation framework, that can provably improve the detection performance by increasing the margin between the in-distribution and out of distribution. Lastly, they propose a direct extension of this by utilizing the energy based scheme in an auxiliary loss regularizer that can be trained along the supervised NLL objective. Experiments performed over a diverse collection of datasets and against SOTA methods clearly demonstrate the proposed method is highly effective.

strength_and_weaknesses: 1. The experiments clearly demonstrate that the proposed approach is highly effective. It is very interesting that the GNN-safe model alone can provide substantially high detection performance.
2. The experiments in table 8 comparing the runtime of all the methods are further useful in demonstrating the effectiveness of the proposed approach.
4. Eq 11 in the appendix should have a sum instead of difference.
5. The authors mention in the last paragraph of page 4 that - “In specific, the generation of a node is conditioned on its neighbors, and thereby an in-distribution node tends to connect with other nodes that are sampled from in-distribution, and vice versa”. It would be more useful to provide some citations for this. Though they mention in the subsequent paragraph that the labels and features for the neighbors can be different, in such heterophilous graphs, the opposite trend may also hold. 
6. There is strong emphasis that the energy of in-distribution samples tend to be lower than the out of distribution samples, however, the opposite trend emerges in figure 1 (a). Do the authors have some intuition of why this is happening?
7. There is no mention of the selection of the hyperparameter $\tau$, eq 8. Details and study of this hyperparameter will be highly relevant.

clarity,_quality,_novelty_and_reproducibility: The writeup of the paper is very clear and the paper is easy to follow. The hyperparameter details along with the empirical evidence presented suggest that the results are likely to be reproducible. Though there are concerns regarding the novelty of the work as the idea of energy based models has been extensively utilized in the same literature of out of distribution detection and with similar formulations.

summary_of_the_review: Based on the questions, comments and concerns raised in the weakness section as well as the issue concerning novelty of the work, I lean towards weak rejection of the work.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: **Summary** This paper studies out-of-distribution detection in the context of graph neural networks (GNNs). This direction of OOD detection with graph-structured data is important, and can be of interest to both graph learning and OOD detection communities. Hence, I am very much in support of this paper's motivation and positioning. 

The core idea of this paper is a nice extension of Liu et al. 2020 [1] from CNNs to GNNs, supported by extensive evaluations and promising results. The methodological novelty was questioned by reviewer 2RDq. During the rebuttal, the authors provided further clarification on the novelty. Compared to [1], the proposed method is designed to handle inter-dependent rather than i.i.d. data, and further, the paper introduces energy belief propagation (which seems new in the context of GNNs). The AC recommends the authors revise the novelty statement to highlight these two points more clearly.

----------------------
**Suggestions for revision [important]** It seems to me the paper draws heavily on the intellectual idea in [1] (both the energy score and energy-based regularization loss). This is fine but they probably should give better credit to the source of inspiration. Because of the lack of a clear attribution of intellectual merit, several places can be flagged as potential plagiarism. I provide a few concrete examples below:

* Page 5, Equation (9), the energy-regularized loss function is mathematically identical to the square hinge loss in [1], though the paper did not cite the paper where the loss originates. Notion wise, the slight change was from $\max(0, x)$ to $\text{ReLU}(x)$. The loss function should not be framed as a new contribution in this paper, since Section 3.3 in spirit follows largely Section 3.3 in [1].

* Page 12, the proof technique of Proposition 1 is identical to Page 4 in [1], which showed the negative log-likelihood loss pushes down energy for in-distribution data. This should be cited properly.

* Page 4, Equation (5), _"...More importantly, without changing the parameterization of the graph neural network,
we can express the free energy function $E(\mathbf{x})$ in terms of the denominator of the softmax activation"_. Note that the sentence is a word-by-word copy of the same sentence on Page 3 of [1], except for replacing "neural network" with graph neural network. 

With that being said, I still think the merit of the paper outweighs the ethical concerns in writing (which are fixable). I thus invite the authors to carefully revise the final version, incorporating the points raised above.

------------------------------
**Recommendation**
Three reviewers recommend acceptance while one reviewer rated the paper marginally below the acceptance threshold. Overall, the AC believes the paper provides valuable contributions to the community. I would like to recommend acceptance, conditioned on the suggested changes incorporated in its final version.

[1] Liu et al. Energy-based Out-of-distribution Detection, NeurIPS 2020. 



justification_for_why_not_higher_score: NA
justification_for_why_not_lower_score: NA
Decision: Accept: poster