Energy-based Out-of-Distribution Detection for Graph Neural Networks
Authors: Qitian Wu
Yiting Chen
Chenxiao Yang
Junchi Yan
Venue: ICLR 2023 poster (2023)
Keywords: graph neural networks
out-of-distribution detection
semi-supervised node classification
energy model

Abstract: Representation learning on semi-structured data, e.g., graphs, has become a central problem in deep learning community as relational structures are pervasive in real situations and induce data inter-dependence that hinders trivial adaptation of existing approaches in other domains where the inputs are assumed to be i.i.d. sampled. However, current models in this regime mostly focus on improving testing performance of in-distribution data and largely ignores the potential risk w.r.t. out-of-distribution (OOD) testing samples that may cause negative outcome if the model is overconfident in prediction on them. In this paper, we identify a provably effective OOD discriminator based on an energy function directly extracted from a graph neural network trained with standard supervised classification loss. This paves a way for a simple and efficient OOD detection model for GNN-based semi-supervised learning on graphs, which we call GNN-Safe. It also has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for in-distribution and OOD samples, which, more critically, can be further strengthened by a non-learning-based structured propagation scheme. Extensive experiments over five real-world datasets validate the practical efficacy of the proposed model for detecting various OOD instances that are inter-connected in a graph with up to 17.0% improvement on average AUROC over competitive peer models and without sacrificing in-distribution testing accuracy.

Published as a conference paper at ICLR 2023
ENERGY-BASED OUT-OFIDISTRIBUTIIN DETECTION FOR GRAPH NEURAL NETWORKS
Qitian Wu, Yiting Chen, Chenxiao Yang, Junchi Yan"
Department of CSE & MoR Lab of Artificial Intelligence, Shanghai Jiao Tong University (echo740, sjtucyt, chr261 95, yanjunchi e jtu .dtu.eeddccm
ABSTRACT
Learning on graphs, where instance nodes are inter-connected, has become one of the central problems for deep learning, as relational structures are pervasive and induce data inter-dependence which hinders trivial adaptation of existing approaches that assume inputs to be i.idd sampled. However, current models mostly focus on improving testing performance of in-distribution data and largely ignore the potential risk w.r.t. cause negative outcome the prediction is overconfident on them. In this paper, we investigate the under-explored problem, OOD detection on graph-structured data, and identify a provably effective OOD discriminator based on an energy function directly extracted from graph neural networks trained with standard classification loss. This paves a way for simple, powerful and efficient OOD detection model for GNN-based learning on graphs, which we call GNNSAFE. It also has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for in-distribution and OOD samples, which, more critically, can be further strengthened by learning-frr foreingg-free energy belief propagation scheme. For comprehensive evaluation, we introduce new benchmark settings that evaluate the model for detecting OOD data from both synthetic and real distribution shifts (cross- domain graph shifts and temporal graph shifts). The results show that GNNSAFF achieves up to 17.0% AUROC improvement serve as simple yet strong baselines such an under-developed area. The codes are available https:: / g thub. com/ qi caaawuu///// Graphooo GNNSO fe.
(OOD) testing samples that may
state-of-the-an the-artsss and it could
INTRODUCTION
Real-world applications often require machine learning systems interact with open world, violating the common assumption that testing and training distributions are identical. This urges the community devote increasing efforts on how enhance models' generalization (Muandet et al., 2013) and reliability (Liang et al., 2018) w.r.t. diito--disttibution (OOD) data. However, most of current approaches are built on the hypothesis that data samples are independently generated (e.g.. image recognition where instances have no interaction). Such premise hinders these models from readily adapting to graph-structured data where node instances have inter-dependence (Zhao et al., 2020; Ma et al., 2021; Wu et al., 2022a).
Out-of-Distribution Generalization. To fill the research gap, a growing number recent studies on graph-related tasks move beyond the single target w.r.t. in-distribution testing performance and turn more attentions to how the model can generalize to perform well on OOD data. One of the seminal works (Wu al., 2022a) formulates the graph-based OOD generalization problem and leverages (causal) invariance principle for devising a new for graph data. Different from grid-structured and independently generated images, distribution shifts concerning graph-structured data can be more complicated and hard to address, which often requires graph-specific technical originality. For instances, Yang et al. (2022c) proposes identify invariant substructures, i.e., a subset of nodes with causal effects to labels, in input graphs to learn
learning approach
"Corresponding author: Junchi Yan who is also affiliated with Shanghai Al Lab. The work was in part supported by National Key Research and Development Program of China (2020AAA0107600), National Natural Science Foundation of China (62222607), STCSM (22511105 1151110

---

Published as a conference paper at ICLR 2023
stable predictive relations across environments, while Yang al. (2022b) resorts to an analogy of thermodynamics diffusion on graphs to build a principled knowledge distillation model for geometric knowledge transfer and generalization.
Detection. One critical challenge that stands in the way for trustworthy AI systems is how equip the deep learning models with enough reliability of realizing what they don't It know, i.e., detecting OOD data on which the models are expected to have low confidence (Amodei al., 2016; Liang et al., 2018). This is fairly important when it comes safety-critical applications such as medical diagnosis (Kukar, 2003), autonomous driving (Dai & Van Gool, 2018), etc. Such a problem is called out-of cuttof-distributionn detection the literature, which aims at discriminating OOD data from the in an-distribution one. While there are a surge of recent works exploring various effective methods for OOD detection (Hendrycks & Gimpel, 2016; Bevandić et al., 2018; Vries & Taylor, 2018; Hein et al., 2019; Hsu et al., 2020; Sun et al., 2021; Bitterwolf et al., 2022), these models again tacitly assume inputs to be independently sampled and are hard to be applied for graph data.
In this paper, we investigate into out-of-distribution detection for learning on graphs, where the model needs to particularly handle data inter-dependence induced by the graph. Our methodology is built on graph neural networks (GNNs) as an encoder for node prvesentaaion/predictii the structural information. As an important step to enhance the reliability of GNN models against OOD testing instances, we identify an intrinsic OOD discriminator from GNN classifier trained with standard learning objective. The key insight of our work is that standard GNN classifiers possess inherent good capability for detecting OOD) samples (i.e., what they don't 't know) from unknown testing observations, with details of the model as described below.
that accommodates
Simplicity and Generality: The out-of-distribution discriminator in our model is based on an energy function that is directly extracted through simple transformation from the predicted logits of a GNN classifier trained with standard supervised classification loss on nn-iistribution data. Therefore, our model can be efficiently deployed in practice, i.c., it does not require training a graph generative model for density estimation or any extra OOD discriminator. Also, the model keeps a general form, i.e., the anergy-based detector is agnostic to GNNs' architectures and can in principle enhance the reliability for arbitrary off-the-sheeff GNNs (or more broadly, graph Transformers) against OOD data.
Theoretical Soundness: Despite simplicity, our model can be provably effective for yielding distinguishable scores for in-distribution and OOD inputs, which can be further reinforced by an energy- -bargy-based belief propagation scheme, a learning-free approach for boosting the detection consen- sus over graph topology. We also discuss how to properly incorporate an auxiliary regularization term when training data contains additional OOD observation outlier exposure, with double guarantees for preserving in-distribution learning and enhancing disti--distiiuution reliability.
Practical Efficacy: We apply model to extensive node classification datasets of different properties and consider various OOD types. When training with standard cross-entropy loss on pure in-distribution data and testing OOD detection inference time, model consistently outperforms SOTA approaches with an improvement of up to 12.9% on average AUROC; when training with auxiliary OOD exposure data regularization and testing on new unseen OOD data, our model outperforms the strong competitors with up to 17.0% improvement on average AUROC.
2 BACKGROUND
Predictive tasks on graphs. We consider a set of instances with indices i E (1, .N I whose generation process involves inter-dependence among each other, represented by an observed graph G = (V, E) where V = {i] < i < N denotes the node set containing all the instances and E E= {ey} } denotes the edge set. The observed edges induce an adjacency matrix A [a.y]N N where 0ij- = 1 if there exists an edge connecting nodes i and j and O otherwise. Moreover, each instance has an input feature vector denoted by x; E RD and a label y; E {1, .1....., , C} where D is the input dimension and C denotes the class number. The N instances are partially labeled and we define I, (resp. I,,) the labeled (resp. unlabeled) node set, i.e.. I = I, U I,,. The goal of standard juppriesue learning graphs is to train node-level classifier f with Y f(X, A), where X [x)iez and y = [ that predicts the labels for in-distribution instances in I,,-
Out-of-distribution detection. Besides decent predictive performance on in in-distribution testing nodes (sampled from the same distribution as training data), we expect the learned classifier is capable
2

---

Published as a conference paper at ICLR 2023
for detecting cut-of-distributioon (OOD) instances that has distinct data tatt-gennraaing distribution with the in- in-distribution ones. More specifically, the goal of OOD detection is to find a proper decision function G' (that is associated with the classifier f) such that for any given input x:
(o.
x is an in-distribution instance, x is an out-of-distribution instance,
G(x,9xif) ({=))=((0 0,
where 9x (1x; ((x; (exx)) Ax) denotes the ego-graph centered at node NN is the set of nodes in the ego-graph (within a certain number of hops) of x on the graph and Ax is its associated adjacency matrix. Our problem formulation here considers neighbored nodes that have inter-dependence with the centered node throughout the Sata-generating process for determining whether it is OOD. In such a case, the OOD decision for each input is dependent on other nodes in the graph, which fairly different from existing works in vision that assume i.i.d. inputs and independent decision for each.
Energy-based model. Recent works (Xie et al., 2016; Grathwohl et al., 2020) establish an equivalence between neural classifier and an energy-
(EBM) (Ranzato et al., 2007) when the inputs are i.i.d.
generated. Assume a classifier as h((x) RDD - TpC which maps an input instance to a C-dimensional
logits, and softmax function over the logits gives predictive categorical distribution: p(y x)
E Ee Millje where h(x) queries the e-th entry of the output. The energy-based model builds
function E(x, v) I2D + R that maps each input instance with an arbitrarily given class label to a -B(xx.a)
scalar value called energy, and induces Boltzmann distribution: p(y|x) Σy < B00
-(x..))
where the denominator Ey F(x.y)) is often called the partition function which marginalizes over y. The equivalance between an EBM and discriminative neural classifier is established by setting the energy as the predicted logit value E(x, (x, y) -x) And. the energy function E(x) for any given input can be
E(x) log Σe Fi(x.1))
y'
The energy score E(x) can be an effective indicator for OOD detection due its nice property: the yielded energy scores for in-distribution data overall incline to be lower than those of OOD data (Liu et al., 2020). However, to the best of our knowledge, current works on energy-based modeling as well as its application for OOD detection mostly focused on i..d. inputs and the power of the energy model for modeling inter-dependent data has remained under-explored.
PROPOSED METHOD
We describe our new model for out-of-distribution detection in seepe-ssuper node classification where instances are inter-dependent. We will first present a basic version of our model with energy- based modeling (Section 3.1) and a novel energy-based belief propagation scheme (Section 3.2). After that, we extend the model with auxiliary OOD training data (if available) (Section 3.3).
3.1 ENERGY-BASED OOD DETECTION WITH DATA DEPENDENCE
To capture dependence among data points, the most commonly adopted model class follows the classic idea of Graph Neural Networks (GNNs) (Scarselli et al., 2008) which iteratively aggregate neighbored nodes' features for updating centered nodes' representations. Such an operation is often called message passing that plays central role in modern GNNs for encoding topological patterns and accommodating global information from other instances. In specific, assume z() as the representation of instance node at the (-th layer and the Graph Convolutional Network (GCN) (Kipf' & Welling, 2017) updates node representation through a layer-wise normalized feature propagation:
z = a (D-1/2 1/2 1/2 -/-2
( = [2 (1-1)) BEZ,
-0= X,
z =
where À is an adjacency matrix with self-loop on the basis of A, D its associated diagonal degree matrix, ( is the weight matrix the (-th layer and a is non-linear activation.
With L layers of graph convolution, the GCN model outputs C-dimensional vector z." as logits
for each node, denoted as ho (x,, Gx:) z (L) where 0 denotes the trainable parameters of the GNN
3

---

Published as a conference paper at ICLR 2023
model h. Induced by the GNN's updating rule, here the predicted logits for instance denoted as a function of the L-order ego-graph centered at instance x, denoted as Gx. The logits are used to derive a categorical distribution with the softmax function for classification:
(x.
ply x, 9x)
E
By connecting to the EBM model that defines the relation between energy function and proba- bility density, we obtain the energy form induced by the GNN model, i.e., E 9x. ys ho) (x, Gx)ls) Notice that the energy presented above is directly obtained from the predicted logits of the GNN classifier (in particular, without changing the parameterization of GNNs). And, the free energy function Exx, ho) that marginalizes y can be expressed in terms of the denominator in Eqn.
E(x, 9xi he)
log Σ hhe (x.
The non-probabilistic energy score given by Eqn. 5 accommodates the information of the instance itself and other instances that have potential dependence based on the observed structures.
For node classification, the GNN model is often trained by minimizing the negative log-likelihood of labeled training data (i.e., supervised classification loss)
Suu = B(x.Grx)) (- ply x. 9x)) = Σ
(2o Gxs )(vil + log Σ e ho (x
VET.
(6)
where Din denotes the distribution where the labeled portion of training data is sampled. We next show that, for any GNN model trained with Eqn. 6, the energy score can provably be well-posed indicator of whether the input data is from in-distribution or out-of-distribution.
Proposition 1. The gradient descent on Ssup will overall decrease the energy E(x, 9xi ho) for any in-distribution instance (x, Gx y) ~ Din-
The proposition shows that the induced energy function Exx, Gxi h2) from the supervisedly trained GNN guarantees an overall trend that the energy scores for in-distribution data will be pushed down. This suggests that it is reasonable use the energy scores for detecting OOD inputs at inference time.
Remark. The results above are agnostic to GNN architectures, i.e., the energy-based OOD discrimi- nator can be applied for arbitrary off tfffth GNNs. Furthermore, beyond defining message passing on fixed input graphs, recently proposed graph Transformers Wu et al. (2022b; 2023) are proven to be powerful encoders for node-level prediction. The energy-based model is also applicable for these Transformer- models and we leave such an extension as future works.
3.2 CONSENSUS BOOSTING WITH ENERGY-BASED BELIEF PROPAGATION
As the labeled data is often scarce in the sun-ssueeeeieed learning setting, the energy scores that are directly obtained from the GNN model trained with supervised loss on labeled data may not suffice to generalize well. The crux for enhancing the generalization ability lies in how to leverage the unlabeled data in training set that could help to better recognize the geometric structures behind data (Belkin et al., 2006; Chong et al., 2020). Inspired by the label propagation (Zhu et al., 2003), a classic non-parametric semi-supervised learning algorithm, we propose energy- belief propagation that iteratively propagates estimated energy scores among inter-connected nodes over the observed structures. We define E(O) [xx, 9x,; ho) ho))eer as vector of initial energy scores for nodes in the graph and consider the propagation updating rule
(7) where 0 < < is parameter controlling the concentration on the energy of the node itself and other linked nodes. After K-step propagation, we can use the final result for estimation, i.e., E(x;, Ix.s ha) E(K) and the decision criterion for OOD detection can be
E (k) a (k-1) + (1 --E)-----EE(-
[E liEz,
G(x, 9x: = { 1, if Gx: S T,
(8)
0, if Exx h9) > T,
4

---

Published as a conference paper at ICLR 2023
where 7 is the threshold and the return of 0 indicates the declaration for OOD. The rationale behind the design of energy belief propagation is to accommodate the physical mechanism in data generation with instance-wise interactions. Since the input graph could reflect the geometric structures among data samples with certain proximity on the manifold, it is natural that the generation of a node is conditioned on its neighbors, and thereby connected nodes tend to be sampled from similar distributions'. Given that the energy score of Eqn. 5 is an effective indicator for out-of-distribution the propagation in the energy space essentially imitates such physical mechanism of data generation, which can presumably reinforce the confidence on detection. Proposition 2 formalizes the overall trend of enforcing consensus for instance-level OOD decisions over graph topology.
Proposition 2. For any given instance X;, if the averaged energy scores of its one-hop neighbored nodes, i.c., Σ, a., E
Σ, "; is less (resp. more) than the energy of its own, i.e., E E then the updated energy given by Eqn. yields FT!)) E !) (resp. Fr(h)) F
Proposition 2 implies that the propagation scheme would push the energy towards the majority of neighbored nodes, which can help to amplify the energy gap between in-distribution samples and the OOD. This result is intriguing, suggesting that the energy model can be boosted by the simple propoagation scheme at inference time without any extra cost for training. We will verify its effectiveness through empirical comparison in experiments.
3.3 ENERGY-REGULARIEN LEARNING WITH BOUNDING GUARANTEES
The model presented so far does not assume any OOD data exposed to training. As an extension of our model, we proceed to consider another setting in prior art of OOD detection, e.g., (Hendrycks et al., 2019b; Liu et al., 2020; Bitterwolf et al., 2022), using extra OOD training data (often called OOD exposure). such a case, we can add hard constraints on the energy gap through a regularization loss Frey that bounds the energy for in-distribution data, i.e., labeled instances in I,, and OOD data, ie., another set of auxiliary training instances Io from a distinct distribution. And, we can define the final training objective as Loup + X&regg where À is trading weight. There is much design flexibility for the regularization term Freg. and here we instantiate it as bounding constraints for absolute energy (Liu et al., 2020): freg
(ReL.U 9x, ha) - -lin) 2
Σ JEI.
(ReL.U (Iout - E (xx, ha)
(9)
VEI.
The above loss function will push the energy values within [tin foue] to be lower (resp. higher) for in-distribution (resp. cit-of-distribution samples, and this is what we expect.
The lingering concern, however, is that the introduction of an extra loss term may interfere with the original supervised learning objective Ceup- In other words, the goal of energy-bounded regularization might be inconsistent with that of supervised learning, and the former may change the global optimum to somewhere the GNN model gives sub-optimal fitting on the labeled instances. Fortunately, the next proposition serves as a justification for the energy regularization in our model.
For any energy regularization freg, satisfying that the GNN model hg- minimizing Creg yields the energy scores upper (resp. lower) bounded by fin (resp. tout) for in-distribution (resp. OOD) samples, where tim < tout are two margin parameters, then the corresponding softmax categorical distribution ply|x, Gxi ha-) also minimizes Lsup-
Proposition 3.
The implication of this proposition is important, suggesting that our used Greg whose optimal energy scores are guaranteed for detecting OOD inputs and stay consistent with the supervised training of the GNN model. We thereby obtain a new training objective which strictly guarantees a desired discrimination between in in-distribution and OOD instances in the training set, and in the meanwhile, does not impair model fitting on in-distribution data.
Admittedly, one disad vantage of the energy- regularrgg learning is that it requires additional OOD data exposed during training. As we will show in the experiments, nonetheless, the energy regularization turns out to be not necessity in quite a few cases in practice.
'hee connected nodes can have different labels or features, even if their data distributions are similar. Here the data distribution is a joint distribution regarding node feature, labels and graph structures.
5

---

Published as a conference paper at ICLR 2023
4 EXPERIMENTS
The performance against out-of-distibbttion data on testing set can be measured by its capability of discriminating OOD testing samples from the in-distribution ones. In the meanwhile, do not expect the degradation of the model's performance on in-distribution testing instances. An ideal model should desirably handle OOD detection without sacrificing the in-distribution accuracy.
4.1 SETUP
Implementation details We call our model trained with pure supervised loss as GNNS and the version trained with additional energy regularization as GNNSAFE++. We basically set the propagation layer number K as 2 and weight as 0.5. For fair comparison, the GCN model with layer depth and hidden size 64 is used as the backbone encoder for all the model.
Datasets and Splits Our experiments are based on five real-world datasets that are widely adopted as -ritthhEExpicoo
node classification benchmarks: Cora, Amwzonn-aaa onoohoo, foauthor-CS, Following recent work OOD generalization over graphs (Wu al., 2022a), we generally consider two ways for creating OOD data in experiments, which can reflect the real- world scenarios involving OOD instances inter-connected in graphs. 1) Multi- aulti-graph scenario: the OOD instances come from a different graph (or subgraph) that is dis- cis-conneet with any node in the training set. 2) Single-graph scenario: the OOD testing instances within the same graph as the training instances, yet the OOD instances are unseen during training. Specifically,
and
For Twitch, data. The subgraph ENGB is used as OOD exposure for training of GNNSAFE++. These subgraphs have different sizes, edge densities and degree distributions, and can be treated as samples from different distributions Wu et al. (2022a).
the subgraph DE in-distribution data and other five subgraphs as OOD
For Arxiv v that is a single graph, we partition the nodes into two-folds: the papers published before 2015 are used as in-distriuution data, and the papers after 2017 are used as OOD data. The papers published at 2015 and 2016 are used as OOD exposure during training.
For Cora, Amazon and Coauthor that have no clear domain information, we synthetically create OOD data via three ways. Structure manipulation: use the original graph as in-distribution data and adopt stochastic block model to randomly generate a graph for OOD data. Feature interpolation: use random interpolation to create node features for OOD data and the original graph as in-distribution data. Label leave-oul; use nodes with partial classes as in-distribution and leave out others for OOD,
For in-distribution data in each dataset, only partial nodes' labels are exposed for supervised training and another subset of nodes is used for validation. Other nodes are used as in in-distribution testing data. We provide detailed dataset and splitting information in Appendix B.I.
Metric We follow common practice using AUROC, AUPR and FPR95 evaluation metrics for OOD detection. The in-distribution performance measured by the Accuracy on testing nodes. In Appendix B.3 we provide more information about these evaluation metries.
Competitors We compare with two classes of models regarding OOD detection. The first family of baseline models focus on OOD detection in vision where inputs (e.g., images) assumed to be i.idd. sampled: MSP (Hendrycks & Gimpel, 2016), ODIN (Liang et al., 2018), Mahalanobis (Lee et al., 2018). OE (Hendrycks et al., 2019b) (using OOD exposure for training), Energy and Energy Fine- Tune (Liu et al., 2020) which also uses OOD exposure for training. We replace their convolutional neural network backbone with the GCN encoder used by our model for processing graph-structune data. The second family of baseline models are particularly designed for handling OOD data in graph machine learning and we compare with two SOTA approaches: the Graph based Kernel Dirichlet GCN method GKDE (Zhao et al., 2020) and Graph Posterior Network GPN (Stadler et al., 2021). We provide more implementation details for these model in Appendix B.2.
4.2 COMPARATIVE RESULTS
How does GNNSAFE++ perform compared with competitive methods? In Table 1 and 2 we report the experimental results of GNNSAFE++ in comparison with the competitive models. We found that GNNS consistently outperform all competitors by large margin throughout all

---

Published as a conference paper at ICLR 2023
Table I:
detection results measured by AUROC (1) / AUPR (1) FPR95 (L)
on Twitch, where nodes in different sub-graphs are OOD data, and Arxiv dataset, where papers published after 2017 are OOD data. The in in-distribution testing aceuracy is reported for calibration. Detailed results each OOD dataset (i.e., sub-graph or year) are presented in Appendix C. GPN
reports out-of-memory issue on Arxiv with 24GB GPU.
Model
OOD Expo
Twitch
Arxiv
AUROC
AUPR
FPR
ID ACC
AUROC
AUPR
FPR
ACC
33.59
49.14
97.45
68.72
63.91
75.85
90.59
53.78
ODIN
No
58.16
72.12
93.96
70.79
55.07
68.85
100.0
51.39
Mahalanobis
No
55.68
66.42
90.13
70.51
56,92
69.63
94.24
$1.59
Energy
51.24
60.81
91.61
70.40
64.20
75.78
90.80
$3.36
GKDE
No
46.48
62.11
95.62
67.44
58.32
72.62
93.84
50.76
GPN
No
51.73
66.36
95.51
68.09
GNNSAFE
66.82
70.97
76.24
70.40
71.06
80.44
87.01
53.39
55.72
70.18
95.07
70.73
69.80
80.15
85.16
$2.39
Energy
Yes
84.50
88.04
61.29
70.52
71.56
80.47
80.59
53.26
GNNSAFE++
Yes
95.36
97.12
33.57
70.18
74.77
83.21
77.43
$3.50
CNVV weo enengy pmoagesiin
GNNSsle
in-dstribution
in-distr bution out of disoribtion
in inddstribtionn
Frequency
out detribution
Treng aergeryy
of dittdiitrbbbit ibution
-14
-35
-03
-0.
-0.5
-26
-23
Energy
Energy
(a) The energy distributions on Twi tch tch where nodes in different sub-graphs are OOD instances
Gressafe we eningy grepagaiion
GNNSale
UINssafee
in-diatr butioe
indistribation
indiiitibution
meqiencs
out distsiiiittnnnionnn
out out-of-iitrrbutinn
out out-sf-ditributionn
Energy Scone
Energy scone
Energy scone
(b) The energy distributions Arxiv where nodes appearing in the future OOD instances
Figure 1: Comparison of the energy-based OOD detection performance yielded by vanilla energy model (described in Section 3.1), using energy-basee propagation (proposed in Section 3.2) and additionally using energy-regularize
training (proposed in Section 3.3). These ablation results demonstrate that the proposed energy propagation and regularized training can both be helpful for enlarging the energy margin between in-distribution and OOD inputs.
cases, increasing average AUROC by 12.8% (resp. 17.0%) and reducing average FPR95 by 44.8% (resp. 21.0%) on Twitch (resp. C Structure) and maintain comparable in-distribution accuracy. This suggests the superiority of our energy- mesey model for OOD detection graphs. Furthermore, without using the energy regularization, GNNSAFE still achieves very competitive OOD detection results, increasing average AUROC by 12.9% Sora-St In partieular, even outperforms OE on all datasets and exceeds Energy FT in most cases on Cora, Ama z on and Coauthox though these two competitors use extra OOD exposure for training. This shows key merit of our proposed model: it can enhance the reliability of GNNs against OOD data without costing extra computation resources. To quantitatively show the efficiency advantage, training and inference times in Table 8 in Appendix C. We can see that GNNSAFE is much faster than the SOTA model GPN during both training and inference and as efficient as other competitors. These results altogether demonstrate the promising efficacy of GNNSAFE++ for detecting OOD anomalies in real graph-related applications where the effectiveness and efficiency are both key considerations.
compare the
How does the energy propagation improve inference-time OOD detection? We study the effectiveness of the proposed energy-based belief propagation by comparing the performance of Energy and GNNSAFE (the Energy model in setting can be seen as ablating the propagation module from GNNSAFE). Results Table and consistently show that the propagation scheme indeed brings up significant performance improvement. Furthermore, in Fig. I we compare the energy scores produced by GNNSAFE (the middle column) and Energy (the left column). The figures show that the propagation scheme indeed help separate the scores for in-distribution and OOD inputs.
7

---

Published as a conference paper at ICLR 2023
Table 2: Out-of-distriibttion detection performance measured by AUROC (1) on datasets Cora,
Amazon, Coauthor with three OOD types (Structure manipulation, Feature interpolation, Label leave-out). Other results for AUPR, FPR95 and in-distribution accuracy are deferred to Appendix C.
Model
OOD Expo
Cora
Ama
Coauthor
S
F
L
MSP
No
70.90
85.39
91.36
98.27
97.31
93.97
95.30
97.05
94.88
ODIN
No
49.92
49,88
49.80
93.24
81.15
65.97
52.14
51.54
51.44
Mahalanobis
No
46.68
49.93
67.62
71.69
76.50
73.25
80.46
93.23
85.36
Energy
No
71.73
86.15
91.40
98.51
97.87
93.81
96.18
97.88
95.87
GKDE
No
68.61
82.79
57.23
76.39
58,96
65.58
65.87
80.69
61.15
GPN
No
77.47
85,88
90.34
97.17
87.91
92.72
34,67
72.56
83.65
GNNSAFE
No
87.52
93.44
92.80
99.58
98.55
97.35
99.60
99.64
97.23
OE
Yes
67.98
81.83
89,47
99.60
98.39
95.39
97.86
99.04
96.04
Energy FT
Yes
75,88
88.15
91.36
98.83
98.55
97.35
98.84
99,43
96,23
GNNSAFE++
Yes
90.62
95.56
92.75
99.82
99.64
97.51
99.99
99.97
97.89
MSP
Erorgy
Energy FT
GhNBafe
GNNS3lle*
Srustre
Foature
Label
AUROC
AUROC
AURGO
40
MLP
OCN
GAT
JKhat
Mietop
GCH
GAT
JOHN
Mtelop
MLP
GCN
GAT
J0HN
Malko
Figure 2: Performance comparison of MSP, Energy, Energy FT, GNNSAFE and GNNSAFE+ w.r.t different encoder backbones on Cora with three OOD types.
Such results also verify our hypothesis in Section 3.2 that the structured propagation can leverage the topological information to boost OOD detection with data inter-dependence.
How does the reguaaieell training help OOD detection? We next investigate into the effectiveness of the introduced mnergy-regularized training. In comparison with GNNSAFE, we found that GNNSAFE++ indeed yields better relative performance across the experiments of Table and 2, which is further verified by the visualization in Fig. 1 where we compare the energy scores of the models trained w/o and w/ the regularization loss (the middle and right columns, respectively) and that GNNSAFE-+ can indeed produce larger gap between in-distribution and OOD data. Furthermore, we notice that the performance gains achieved in Table 2 are not that considerable as Table 1. In particular, the AUROCS given by GNNSAFE and GNNSAFE++ are comparable on Amazon and Coauthor. The possible reason is that the OOD data on the multi multi-graph dataset Twitch and large temporal dataset Arxiv is introduced by the context information and the distribution shift may not be that significant, which makes it harder for discrimination, and the energy regularization can significantly facilitate OOD detection. On the other hand, the energy regularization on auxiliary OOD) training data may not necessary for cases (like other three datasets) where the distribution shifts between OOD and in-distribution data are intense enough.
4.3 ANALYSIS OF
We next analyze the impact of using different encoder backbones and prrpe-pprramttrs, to shed more lights on the practical efficacy and usage of our proposed model.
Will different encoder backbones impact the detection results? Fig. 2 compares the performance of MSP, Energy, Energy FT, GNNSAFE and GNNS w.r.t. using different encoder backbones including MLP, GAT (Velickovic et al., 2018), JKNet (Xu et al., 2018), MixHop (Abu-El- Hau-El-Haije -aijaaa al., 2019) on Cora with three OOD types. We found that different backbones indeed impact the detection performance and in particularly, GNN backbones bring up significantly better AUROC than MI.P due to their stronger expressiveness for encoding topological features. However, we observe that the relative performance of five models mostly consistent with the results in Table 2 where we use GCN as the backbone and two GNNSAFE models outperform other competitors in nearly all the cases. In the right figure, even with MLP backbone, two GNN models perform competitively as the counterparts with GNN backbones, which also validates the effectiveness of our energy propagation scheme that helps to leverage topological information over graph structures.
How do the propagation depth and self-loop strength impact the inference-time perfor- mance? Fig. 3(a) plots the results of GNNSAFE+ with propagation step K increasing from
8

---

Published as a conference paper at ICLR 2023
1 to 64 and self-loop strength a varing between 0.1 and 0.9. As shown in the figure, the AUROC scores first go up and then degrade as the propagation steps increase. The reason is that a moderate number of propagation steps can indeed help to leverage structural dependence for boosting OOD detection yet too large propagation depth could potentially overly smooth the estimated energy and make the detection scores indistinguishable among different nodes. Similar trend is observed for which contributes to better performance with the value set as moderate value, c.g., 0.5 or 0.7. This presumably because too large a would counteract the effect of neighbored information and too small a would overly concentrate on the other nodes.
How do the regularization weight and margins impact the training? We discuss the performance variation of +NNSAFE++ w.r.t the configurations of ti,, tout and À in Fig. 3(b) and (c). We found that the performance GNNSAFE++ largely dependent on which controls the strength of energy regularization and its optimal values also vary among different datasets, though we merely tune it with a coarse searching space (0.01, 0.1, 1.0]. Besides, a proper setting for the margin in and fout yields stably good performance while too large margin would lead to performance degradation.
5 RELATED WORKS
Out-of -ittii-iittiitttion detection for neural networks Detecting OOD samples which the models should have low confidence has been extensively studied by recent literature. From different tech- nical aspects, prior proposes to adopt confidence scores from pre-trained models (Hendrycks & Gimpel, 2016; Hendrycks et al., 2019a), add regularization on auxiliary OOD training data (Bevandić al., 2018; Liu et al., 2020; Mohseni et al., 2020), or train a generative model for detecting OOD examples (Ren et al., 2019). However, these methods mostly assume instances (e.g., images) are i..dd generated, and ignore the widely existing problem scenario that involves inter-dependent data.
Out-of-distribution detection on graphs In contrast, OOD detection on graph data that contains inter-dependent nodes has remained as an under-explored research area. The recent work Li et al. (2022) proposes a flexible generative framework that involves node features, labels and graph structures and derives a posterior distribution for OOD detection. Moreover, Bazhenov et al. (2022) compares different OOD detection schemes from uncertainty estimation perspective. These works focus on graph classification where each instance itself a graph and can be treated as i.i.d. samples as well. In terms of node classification where the inter-dependence of instances becomes non-negligible, Zhao et al. (2020) and Stadler et al. (2021) propose Bayesian GNN models that can detect OOD nodes graphs. Different from these approaches, the OOD discriminator in our model is directly extracted from the predicted logits of standard GNNs trained with the supervised classification loss, which keeps simple and general form that is agnostic to GNN architectures. Also, our model is applicable for thf-th discriminative GNN classifiers and is free from training complicated graph generative models for density estimation or any extra OOD discriminator.
Out-of- distributinn generalization Within the general picture of learning with distribution shifts, another related problem orthogonal to ours is to enhance the model's predictive performance on OOD testing data, i.e., OOD generalization (Muandet et al., 2013). The OOD data on graphs is of various types dependent on specific distribution shifts in different scenarios and datasets, including but not limited to cross-domain graph shifts and temporal graph shifts used for experiments in Wu et al. (2022a), subgroup attribute shifts in node classification (Ma al., 2021; Bi al., 2023), molecule size/scaffold shifts in molecular property prediction (Yang et al., 2022c), temporal context shifts in recommender systems (Yang et al., 2022a), feature space expansion in tabular predictive tasks (Wu et al., 2021), etc. As an exploratory work in OOD detection on graphs, we focus on the cross-domain shifts and temporal shifts in our experiments as two real-world scenarios, and one promising future direction is to extend the evaluation of OOD detection to other distribution shifts on graphs.
6 CONCLUSIONS
We have shown in this paper that graph neural networks trained with supervised loss can intrinsically be an effective OOD discriminator for detecting OOD testing data on which the model should avoid prediction. Our model is simple, general, and well justified by theoretical analysis. We also verify its practical efficacy through extensive experiments on real-world datasets with various OOD data types. We hope work can empower GNNs for real applications where safety considerations matter.

---

Published as a conference paper at ICLR 2023
REFERENCES
Sami Abu-EI-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard. Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. Mixhop: Higher-order graph convolutional ar- chitectures via sparsified neighborhood mixing. In International Conference on Machine Learning. pp. 21-29, 2019.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint N.Iv:1606606563. 2016.
Gleb Bazhenov, Sergei Ivanov, Maxim Panov, Alexey Zaytsev, and Evgeny Burnaev. Towards ood detection in graph classification from uncertainty estimation perspective. arXiv preprint arXiv:2206. 10691, 2022.
Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. Manifold regularization: geometric frame- work for learning from labeled and unlabeled examples. Journal of machine learning research, 7 (11), 2006.
Petra Bevandić, Ivan Krešo, Marin OrSić, and Siniša Segvić. Discriminative out-of-distribution detection for semantic segmentation. arXiv preprint arXi:: 2018.
Wendong Bi, Bingbing Xu, Xiaoqian Sun, Li Xu, Huawei Shen, and Xueqi Cheng. Predicting the silent majority on graphs: Knowledge transferable graph neural network. In The Web Conference, 2023.
Julian Bitterwolf, Alexander Meinke, Maximilian Augustin, and Matthias Hein. Breaking down out-of -istiii-iiiitttion detection: Many methods based on ood training data estimate combination of the same core quantities. In International Conference on Machine Leurning, pp. 2041-2074. PMLR, 2022.
Yanwen Chong, Yun Ding, Qing Yan, and Shaoming Pan. Graph-based semi-supervised learning: A review. Neurocomputing, 408:216-2368 230, 2020.
Dengxin Dai and Luc Van Gool. Dark model adaptation: Semantic image segmentation from daytime to nighttime. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pp. 38 1819-38.4. IEEE, 2018.
Terrance Veerie and Graham W Taylor. Learning confidence for ousto--distributiin detection in neural networks. arXiv preprint 1rXiv:/802. 04865, 2018.
Will Grathwohl, Kuan-Chich Wang.
Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In International Conference on Learning Representations 2020.
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high- confidence predictions far away from the training data and how to mitigate the problem. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 41-50, 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint 16X.0:16 2016.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. A benchmark for anomaly segmentation. 2019a.
Dan Hendryeks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019b.
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out-of- distribution image without learning from out-o--distiibuttin data. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10951-10960. 2020.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong. Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118 18-22333,, 2020.
10

---

Published as a conference paper at ICLR 2023
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
Matjaž Kukar. Transductive reliability estimation for medical diagnosis. Artificial Intelligence in Medicine, 29(1-))811-06, 2003.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distriutiion samples and adversarial attacks. In Advances in neural information processing systems, 2018.
Zenan Li, Qitian Wu, Fan Nie, and Junchi Yan. Graphde: A generative framework for debiased learning and out-of- distributiion detection on graphs. In Advances Neural Information Processing Systems, 2022.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribttion image detection in neural networks. In International Conference Learning Representations, 2018.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of of-distribution detection. Advances in Neural Information Processing Systems, 33:2 164 21475, 2020.
Jiaqi Ma, Junwei Deng, and Qiaozhu Mei. Subgroup generalization and fairness of graph neural networks. Advances in Neural Information Processing Systems, 34: 1048-1061, 2021.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based recommendations styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pp. 43-52, 2015.
Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable distriiibbt detection. In AAAI Conference on Artificial Intelligence, pp. 5216-5223, 2020.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning (ICML), pp. 10-18, 2013.
Marc' Aurelio Ranzato, Y-Lan Boureau, Sumit Chopra, and Yann LeCun. A unified mergy-bbsse framework for unsupervised learning. In Artificial Intelligence and Statistics, pp. 371-379. PMLR, 2007.
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snock, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. Advances in neural information processing systems, 32, 2019.
Benedek Rozemberezki and Rik Sarkar. Twitch gamers: a dataset for evaluating proximity preserving and structural role-based node embeddings, 2021.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks, 20(1):61-80, 2008.
Prithviraj Sen, Galilco Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rac Rad. Collective classification in network data. Al magazine, 29(3):93 2008
Arnab Sinha. Zhihong Shen, Yang Song, Hao Ma, Darrin Eide. Bo-June Hsu, and Kuansan Wang. overview of microsoft academic service (mas) and applications. In Proceedings of the 24th international conference on world wide web, pp. 243-246, 2015.
Maximilian Stadler, Bertrand Charpentier, Simon Geisler, Daniel Zügner,, and Stephan Günnemann. Graph posterior network: Bayesian predictive uncertainty for node classification. Advances in Neural Information Processing Systems, 34: 14:18033-18048 2021.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. Advances in Neural Information Processing Systems, 34:144-157, 2021.
11

---

Published as a conference paper at ICLR 2023
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations (ICLR), 2018.
Qitian Wu, Chenxiao Yang, and Junchi Yan. Towards open-world feature extrapolation: An inductive graph learning approach. In Advances in Neural Information Processing Systems, 2021.
Qitian Wu, Hengrui Zhang, Junchi Yan, and David Wipf. Handling distribution shifts graphs: An
invariance perspective. International Conference on Learning Representations, 2022a.
Qitian Wu, Wentao Zhao, Zenan Li, David Wipf, and Junchi Yan. Nodeformer: A scalable graph struc- ture learning transformer for node classification. In Advances in Neural Information Processing
Systems, 2022b.
Qitian Wu, Chenxiao Yang, Wentao Zhao, Yixuan He, David Wipf, and Junchi Yan. DIFFormer: Scalable (graph) transformers induced by energy constrained diffusion. International Conference on Learning Representations, 2023.
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Ying Nian Wu. A theory of generative convnet. In International Conference Machine Learning, pp- 2635-66322222 2644, 2016.
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In ICML, 2018.
Chenxiao Yang, Qitian Wu, Qingsong Wen, Zhiqiang Zhou, Liang Sun, and Junchi Yan. Towards out- of-distribution sequential event prediction: causal treatment. Advances in Neural Information Processing Systems, 2022a.
Chenxiao Yang, Qitian Wu, and Junchi Yan. Geometric knowledge distillation: Topology compression for graph neural networks. In Advances in Neural Information Processing Systems, 2022b.
Nianzu Yang, Kaipeng Zeng, Qitian Wu, Xiaosong Jia, and Junchi Yan. Learning substructure invariance for disttiiibutionn molecular representations. In Advances in Neural Information Processing Systems, 2022c.
Xujiang Zhao, Feng Chen, Shu Hu, and Jin-Hee Cho. Uncertainty aware semi-supervised learning on graph data. Advances in Neural Information Processing Systems, 13:12827-12836 2020.
Xiaojin Zhu, Zoubin Ghahramani, and John D. Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In International Conference on Machine Learning (ICML), pp. 912-919, 2003.
A PROOFS FOR TECHNICAL RESULTS
A.1 PROOF FOR PROPOSITION
Proof. The proof is kind of straight-forward as followed similar reasoning line of the section 1 in Liu et al. (2020). The gradient of Cull w.r.t is given by
TCnll 00
@hg(Xi, Gz.)
The Gx. yel
No(xx18x1110
Σ
00
00
=d==1 No(xx0aa)ee
VEI+r
@F Gx,, Ui)
0E Gx,, c)
Σ
00
00
2d==1 é (241
TEIer
0E 0E(x,, (x,,
Gx, Mii he)
Exx, Gx, c; he)
= - -1-ppy- yi xi, 9x.))
Σ ply = xi, Px.)
20
00
(10)
We can see from the above equation that the training procedure overall minimizing the first-order gradi- ent of Lall will decrease the energy score E(x;, Gx., ys he) and increase the energy E(x;, Gx. c; he). In a general sense, the energy value produced by the trained model tends to become lower for any in-distribution instance (x, Gx, v) D.n-
12

---

Published as a conference paper at ICLR 2023
A.2 PROOF FOR PROPOSITION 2
If Σ, Ey F
Proof.
E 1-1
we can re-write the update in Eqn. 7 as the scalar-form for each
2zay
instance:
+ - -a) Eg
-a) Oij ais (k-1)
E
F. = aF. (k-1)
Dj Oij
(1I)
> aFt (k 1) a)) Fi! (2(
(1-1))
E E(
The opposite result can be similarly proved for the case
Σ, ais
A.3 PROOF FOR PROPOSITION
Proof. We define Gxi hg. as the energy yielded by the model hgs minimizing the regularization loss Freg. And, higt denotes the model yielding the optimal predictive softmax distribution that minimizes the supervised classification loss Leups i...
eh (x. 9x)
argmin (()x DI log Go)].
(12)
Ecneeoo ehat (x, Gx)(c]
Notice that Fx(x, 9xi ha) log Ee ( Therefore we have
Fxi ho-) E(x, ho-) E(x, 9xi hat) log Σ not (x,0x))
log
e Elxxx
hot (x.9x)
(13)
= - log
c=1
The above relationship suggests that the energy E(x, 9x; hg- is secretely equivalent to the energy
induced by the predicted logits hag (x, 9x) - F((x, Gxi hag) + E(x, Gxi ). We can thereby denote the predictive softmax distribution of E(x, Gxi he-) as
eh (x.0x)) Bll
p(u|x,9>)
De ch (x.00)))) Blx
(14)
Eg_ e (x.
The above predictive distribution exactly minimizes Csup as defined by Eqn. 12. We thus have proven that the optimal energy that minimizes Lreg intrinsically induce the predictive softmax distribution that minimizes Lrup- On the other hand, given our assumption that the optimal energy function can well distinguish the scores for in-distribution and OOD inputs, thus obtain proper choices for the regularization loss Creg whose optimal energy satisfies the two requirements mentioned upfront in Section 3.3.
B EXPERIMENTAL DETAILS
We supplement experiment details for reproducibility. Our implementation based on Ubuntu 16.04.6. Cuda 10.2, Pytorch 1.9.0 and Pytorch Geometrie 2.0.3. Most of the experiments running with a NVIDIA 2080Ti with LIGB memory, except that for cases where the model requires larger GPU memory NVIDIA 3090 with 24GB memory for experiments.
13

---

Published as a conference paper at ICLR 2023
B.1 DATASET INFORMATION
The datasets used in our experiment are all public available as common benchmarks for evaluating graph learning models. For oginnArxivv, we use the preprocessed dataset and data loader provided by the OGB package". For other datasets, we use the data loader provided by the Pytorch Geometric package".
Cora (Sen et al., 2008) is a citation network where each node denotes a published paper and each edge indicates the citation relationship. The goal is predict each paper's topic as the label class. This dataset contains 2,708 nodes, 5,429 edges, 1,433 features and 7 classes. We use three different ways for synthetically creating OOD data as mentioned Section 4.1. For araiiing/vllddationnessing splits in in-distribution data, use follow the semi-supervised learning setting by Kipf & Welling (2017) and use its provided splits.
Amazon - Photo (McAuley al., 2015) is an item co- co-purchasing network on Amazon where each node denotes a product and each edge indicates that two linked products are frequently purchased together. The node label denotes the category of the product. This dataset contains 7,650 nodes, 238,166 162 edges, 745 features, and 8 classes. Similar to Cora, we use three ways for synthetically creating OOD data. And, for in-distribution data, we follow the common practice (Kipf & Welling, 2017) and use random splits with 1:1:8 for raining/aaldaationn/ssingg
Coauthor-Cs (Sinha al., 2015) is coauthor network of computer science. Nodes denote authors and they are connected as edges if the two authors co-authored a paper. The goal is to predict the respective field of study for authors based on their papers' keywords as node features. This dataset contains 18,333 nodes, 163,788 edges, 6,805 features, and 15 classes. We also use the aforementioned three ways to construct OOD data and for in-distribution data, we follow the common practice using random splits with 1:1:8 for iaaniingvvaliaation/eesee
Twi tritcch-xppp i t (Rozemberezki & Sarkar, 2021) is multi-graph dataset, where each subgraph corresponds to social network in one region. Nodes in this dataset represent game players on Twitch and edges indicate two users follow each other. Node features are embeddings of games played by the Twitch users, and the label class indicates whether user streams mature content. The node numbers of the subgraphs are ranged from 1,912 to 9,498, with edge numbers from 31, 31,299 to 153,133 and shared feature dimension 2,545. As mentioned in Section 4.1 data where we use random splits with 1:1:8 for saining/validation/essiin used OOD exposure and ES, FR, RU are used for OOD testing data.
subgraph DE as in-distribution Besides, subgraph ENGB
ogbn-Arx v (Hu et al., 2020) is a relatively large graph dataset that records the citation information from 1960 2020. Each node denotes a paper with subject area as the label to predict. Edges indicate citation relationship and each node has 128-dimensional feature vector obtained by the word embeddings of its title and abstract. As described in Section 4. we use the time information for partitioning in-distribution and OOD data. For the in-distribution portion, we also use 1:1:8 random splits for waliii/g//liiaattt/eettnng Notice that here our partition for the in-distribution and OOD data disables us to use the original splitting by Hu et al. (2020) which used time information for splitting training and testing nodes. In our setting, the random splitting also guarantees that the training, validation and testing nodes (of in-distribution data) come from an identical distribution that is distinct from that of OOD data.
B.2 IMPLEMENTATION DETAILS
Architectures As mentioned in Section 4.1, our basic encoder backbone is set as a GCN, and also consider other models such as MLP, GAT, JKNet and MixHop for further discussions. We summary the architectural information of all the encoder backbones used in our experiments as follows.
GCN: 2 GCNConv layers with hidden size 64, ReLU activation, self- and batch normal- ization.
* MLP: 2 fully-connected layers with hidden size 64 and ReLU activation.
14

---

Published as a conference paper at ICLR 2023
GAT: GATConv layers with hidden size 64, ELU activation, head number [2, 1] and batch normalization.
JKNet: 2 GCNConv layers with hidden size 64, ReLU activation, self-loop and batch normalization. The jumping knowledge module is using max pooling.
MixHop: 2 MixHop layers with hop number 2, hidden size 64, ReLU activation, and batch normalization.
Hyper-paranne parpeepprraeters As described in Section 4. we consider the following hypermeters default setting: the propagation step K = 2 and the propagation self-loop weight a = 0.5. For GNNSAFE, we only consider grid-search for the learning rate within f0.1, 0.01, 0.001 } using validation set. For GNNSAFE++, we use the validation set for additionally search for margin
tin (-9, -7, -5}, tout {-1, -2 3, -4} and the regularization weight {0.01, 0.1, 1.0).
The model GNNSAFE is trained with the supervised classification loss on labeled training data, and GNNSAFE++ trained with the supervised loss plus the regularization loss on the OOD training data. We use the supervised classification loss (i.e., gross-entropy) on validation set as the indicator of epoch selection for reporting the testing result. More specifically, in each run, we train the model with 200 epochs as a fixed budget and report the testing performance produced by the epoch where the model yields the lowest classification loss on validation data.
Training Details
Competitors For competitor models, we their implementations provided by the original papers and do some adaptation for our setting. As mentioned in Section 4.1, we use the same encoder backbone GCN (with the same hidden size 64 and layer number 2) for baselines MSP, ODIN, Mahalanobis, Energy and Energy FT. For GKDE and GPN, we use their public implementation, refer to their syper-parameter settings reported in the paper and do some fine-tuning for different datasets.
B.3 EVALUATION METRICS
We provide more details concerning the evaluation metrics we used in our experiment. We follow the common practice of OOD detection literature and use AUROC, AUPR and FPR95 for evaluating the performance for detecting OOD data in the testing set. These metrics are independent of the threshold т. Specifically, AUROC refers to the Area Under the Receiver Operating Characteristic (ROC) curve. The ROC curve presents trade-off between the true positive rate and the false positive rate under different thresholds in 0 ~ 1. Intuitively. the AUROC can be understood as the probability that for any pair of positive example (i.e., in-distribution sample) and negative example -iittr-disttiittioon sample), the positive one has a larger estimated score than the negative, However, AUROC is not always an ideal metric, especially when the positive class and negative class are imbalanced. Under such a case, the AUPR adjusts for different positive/negative base rates. AUPR refers to the Area Under the PR curve which presents the trade-off between the precision and recall under different thresholds in O ~ 1. Furthermore, FPR95 is also widely used metric which refers to the False Positive Rate at 95% true positive rate (TPR). FPR95 can be interpreted as the probability that a citoo--distribution sample is misclassified as in-distribution when the TPR is as high 95%.
C ADDITIONAL EXPERIMENT RESULTS
We supplement more experimental results in this section. In specific, we report detailed OOD detection performance on each OOD dataset (different subgraphs for Twitch and different years for Arxiv) in Table 3 and 4 complementary for Table 1 in the maintext. Besides, in Table 5, 6 and 7 we report the AUROC/AUPR/FPRS5 and in in-distribution testing accuracy on Cora, Amazon and Coauthor as complementary for Table 2 in the maintext. In Table 8 compare the training time per epoch and inference time of our model and the competitors. Moreover, we compare the training time per epoch and inference time of GNNSAFE+ and the competitors in Table 8. Also, we present more hyper-paramett analysis results in Fig. 3 including a, K, tins tout and X.
15

---

Published as a conference paper at ICLR 2023
Table Out-of-distribution detection performance measured by ACROCK()AUPPK())PPPSS)) on OOD sub sub-graphs ES, FR and RU of Twitch dataset.
Model
OOD Expo
Twitch-Ba AUPR
witch--a FR AUPR
Twisch-RU
AUROC
FPR95
AUROC
FPR95
AUROC
AUPR
FPR95
MSP
No
37,72
53.08
98.09
21.82
38.27
99.25
41.23
56.06
95.01
ODIN
83.83
80.43
33.28
59.82
64.63
92.57
58.67
72.58
93.98
Mahalanobis
No
45.66
58.82
95.48
40.40
46.69
95.54
55.68
66.42
90.13
Energy
No
38.80
54.26
95.70
57.21
61.48
91.57
57.72
66.68
87.57
GKDE
No
48.70
61.05
95.37
49.19
52.94
95.04
46.48
62.11
GPN
No
53.00
64.24
95.05
51.25
55.37
50.89
65.14
99.93
OE
Yes
55.97
69.49
94.94
45.66
$4.03
95.48
$5.72
70.18
95.07
Energy
80.73
87.56
76.76
79.66
81,20
93.12
30.72
GNE
No
49.07
$7.62
63.49
66.25
90.80
87.90
89.05
43.95
+N
Yes
94.54
97.17
44,06
93.45
95.44
51.06
98.10
98.74
5.59
Table 4: out-of-distribution detection performance measured by A on OOD datasets of papers published 2018, 2019 and 2020, respectively, on Arxiv.
Arxiv-2011
20-8
2019
Arxiv- 20110
2020
Model
OOD Expo
AUROC
AUPR
FPR95
AUROC
AUPR
FPR95
AUROC
AUPR
FPR95
70.63
63.07
67.00
ODIN
No
53.49
63.06
100.0
53.95
56.07
100.0
55.78
87.41
100.0
Mahalanobis
No
$7.08
65.09
93.69
56.76
$7.85
94.01
56.92
85.95
95.01
Energy
61.75
70.41
65.78
67.70
GKDE
No
56,29
66.78
94.31
57.87
62.34
93.97
60.79
88.74
93.31
GPN
No
67.72
75.74
72.15
72.35
Energy FT
Yes
69,58
76.31
82.10
70.58
72.03
81.30
74.53
93.08
78.36
GNNSAFE
No
66.47
74.99
89.44
68.36
71.57
88.02
78.35
94.76
83.57
GNNSAFE+*
70.4
78.62
81.47
72.16
75.43
79.33
81.75
95.57
71.50
Table duitt--diit detection performance measured by AUROCC))AUPRR(/PPRRSS) on Cora with three OOD types (structure manipulation, feature interpolation, label leave-out).
Cora-Ste Dora-Strucsuur AUPR
Coza-FeatLze
Cora-Label
AUROC
MSP
70.90
45.73
$7.30
75.50
85.39
73.70
61.88
75.30
91.36
78.03
34,99
88.92
ODIN
49.92
27.01
100.0
74,90
49.88
26.96
100.0
75.00
49.30
24.27
100.0
88.92
Mahalanobis
Energy
71.73
16.08
76.00
86.15
74.12
65.81
76.10
91.10
78.14
11.08
$8.92
GKDE
68.61
44.26
84
73.70
82.79
66.52
68.24
74, 80
$7.23
27.50
$3.95
89.87
GPN
77.47
53.25
76.22
76.50
85.88
73.79
56.17
77.00
90.34
77.40
37.42
91.46
Enery
75.88
49.18
67.73
75.50
88.15
75.99
47.53
75.30
91.36
78:28 49
37.83
90.51
GNNSAFE
87.52
77 /6
73.15
75.80
93.44
88.19
38.92
76,
92.80
82. 21
30.83
88 92
GNNSAFT4
Table 6: Out-of-distribttion detection performance measured by AUROCCHAAPPK(//PPSSSS on Amazon with three OOD types (structure manipulation, feature interpolation, label leave-out).
Model
AUROC
Ama zon- Structure
AUPR
FPR95
ID ACC
AUROC
Amazen-Faen Cecen---eauuee
AUPR
EPR95
ID ACC
AUROC
Amazor- Laaee.
AUPR
FPR95
ID ACC
MSP
98.27
98.54
6.13
92.84
97.31
95.16
8.72
92.89
93.97
01.32
26.65
95.36
93.24
92.84
81.15
78.47
100.0
92.71
57.
Mahalonohis
71.69
79.01
99.91
92.79
76.50
71.14
76.12
92.86
71.25
66.89
74
95.76
Energy
98.51
95.72
-197
92.36
97.87
95.61
6.00
92.96
93.81
91.13
28.48
95.72
GKDE
65.20
97.17
90.05
901
OE
99.60
99.61
0.51
92.61
98.39
96.21
4.34
92,30
95.39
92.55
17.72
95.72
Energy FT
98.83
99.14
1.31
92.79
98.68
96.82
2.84
92.52
96.61
94.99 44.92
13.78
94.33
92.81
GNNS
09.82
99.89
0.00
92.22
99.64
99.68
0.13
92.39
97.51
97.07
6.18
95.84
16

---

Published as a conference paper at ICLR 2023
Table 7: Out-of-distribution detection performance measured by (UROCC))AUPPR on Coauthor with three OOD types (structure manipulation, feature interpolation, label leave-out).
Model
Coauthor-St AUPR
Coautho hoottrr-ttuuuu
Coaut hor-Label
AUROC
FPR95
11 ACC
AUROC
AUPR
FPR95
I0 ACC
AUROC
AUPP
FPR995
ID ACC
ODIN
52.14
48.83
99.92
92.34
51.54
45.30
100.0
92.39
51. .4
74.79
100.0
95.15
Mahalanobis
80.46
76.65
70.75
92.33
93.23
90.85
28.10
92.34
85.36
93.61
45.41
95.19
Energy
96.18
95.25
18.02
92.75
97.88
97.69
9.75
92.75
95.87
98.34
13.69
95.20
GPN
34.67
40.21
99.57
89.45
81.77
80.56
74,46
$7.05
93.24
97.55
34.78
91.63
OE
97.86
96.81
9.25
92.60
99.04
98.80
4,44
92.64
96.01
98.50
18.17
95.10
Energy
98.84
3,97
92.61
2.25
92.50
96.23
98.51
17:07
95.20
GNNSAFE
99.60
99.69
0.26
92.73
99.64
99.66
0.51
92.73
97. 23
96 98.98
12.05
95.21
GNNSAFB+4
99.99
99.99
0.02
92.92
99.97
99.95
0.09
92.87
97.89
99.24
9.43
95.24
Table 8: Comparison of training time per epoch and inference time all the models on five datasets.
Cora
Ama zon
Coauthor TR (s)
Twi tc
Arx v
Model
TR (s)
(s)
TR (s)
(s)
(s)
(s)
IN (s)
TR (s)
(s)
MSP
0.008
0.013
0.011
0.023
0.060
0.166
0.008
0.040
0.043
0.173
ODIN
0.007
0.025
0.009
0.053
0.055
0.311
0.008
0.081
0.018
0.010
Mahalanobis
0,008
0.251
0.013
0.306
0.073
1.081
0.009
0.006
0.024
0,034
Energy
0.013
0.016
0.016
0.022
0.114
0.160
0.011
0.048
0.063
0.192
GKDE
0,007
0.015
0,008
0.025
0.069
0.236
0.006
0.043
0.030
0.171
GPN
10.02
40.30
6.480
25,86
28.58
114.44
2.850
13.88
OF
0.014
0.015
0.018
0.024
0.118
0.165
0.012
0.044
0.077
0.252
Energy FT
0.013
0.018
0.018
0.024
0.131
0.171
0.016
0.052
0.069
0.218
GNNSAFE
0.015
0.015
0.015
0.023
0.115
0.164
0.011
0.049
0.062
0,219
GNNSAFE++
0.015
0.019
0.015
0.027
0.141
0.189
0.016
0.052
0.075
0.239
93.0
82.5
25
2-01
2010.5
8:0.1
0--0.
7.m77
12 28
Depth X
(a)
(b)
(c)
Figure 3: (a) Hyper-parameter analysis for propagation steps K and self-loop strength a. (b)
Impact of margin hyper-parameter In and fout- (c) Impact of weight hyper-parameter for energy regularization GNNSAFE++. We Cora with structure manipulation for experiments.
17


Reviews:
[{"review_id": "hJm4d6cIcA", "reviewer": "Reviewer_QtVR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses an important problem of OOD detection and generalization for GNNs. An Energy-based discriminatory framework is explored that is shown to give a some significant advantages like model agnosticity, theoretical gaurantees and practically efficacy. This application and study of energy-based OOD detection methodology for inter-dependent data like graphs has not been done before and is novel. \n\nThey have started off with a simple instance-based energy function used to detect OOD samples, based on negative-log-likelihood loss. The authors show how OOD detection can be enhanced by propagating the energies calculated across neighbors for K-steps in a manner typical to message propagation in GNNs. Further, they have added a regularizing loss to this formulation that induces an energy boundary between in-distribution and OOD instances. \n\nThe experiments compare their variants with image-based and graph-based OOD competitors, with standard choice of metrics for detecting OOD and for calibrating accuracy. They show significant improvements in the OOD detection metrics without trading off the supervised learning of the models by too much. Their results improve upon previous work on OOD detection by a significant margin.\n", "summary_of_the_review": "This paper provides a new approach for OOD detection and generalization on graphs that incorporates the graph's topology in an energy-based message passing scheme. The approach is well described and the rationale makes sense. The authors provide strong evidence that their approach improves generalization on a number of OOD tasks established on different graph benchmark datasets in both an unsupervised and semi-supervised setting.", "strength_and_weaknesses": "Strengths:\n- The results seem significant over other state-of-the-art in OOD detection and generalization on graphs.\n- Broad choice of GNN encoder backbones, competitor architectures, datasets and experimentation strategies. Multiple ablation studies included.\n- Comparison of training times and inference times for all the models used included\n- Each variant of their proposed models are compared with each other as well as competitors and a good summary/explanation of each of the experimental data is given.\n- Have posited some insights on when regularization may be significant and when they may not help in OOD detection well citing examples from experiments.\n\nWeaknesses:\n- The gaps between methods that have access to OOD data to tune on and those that don't is significant. It would be helpful to better delineate these settings in the tables and results in the paper since the conditions in both cases is quite different.\n- Provide further justification and rationale for the choice of methodology for creating synthetic OOD data, especially the label leave-out way in datasets like amazon-photos, cora and coauthor-cs.\n- The explanation of how their approach is used to regularize learning in the supervised setting is not entirely clear.\n\n​Typos\n- Appendix B.1 in Co-author CS:\n    - 6805 classes -> 6805 features\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RN2OYkv_roL", "reviewer": "Reviewer_PiYE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new mechanism based on the energy-models for the task of out of distribution detection on graphs. They show that the graph neural networks trained with supervised loss objective can be intrinsically effective to detect OOD data on which the model should avoid prediction. The energy based method proposed in this work can be directly extracted through simple transformation from the predicted logits of a GNN classifier. Furthermore, the proposed method is agnostic to the choice of the underlying architecture used making it highly generalizable. This is further enhanced using a simple propagation scheme, following the popular label propagation framework, that can provably improve the detection performance by increasing the margin between the in-distribution and out of distribution. Lastly, they propose a direct extension of this by utilizing the energy based scheme in an auxiliary loss regularizer that can be trained along the supervised NLL objective. Experiments performed over a diverse collection of datasets and against SOTA methods clearly demonstrate the proposed method is highly effective.\n", "summary_of_the_review": "Based on the questions, comments and concerns raised in the weakness section as well as the issue concerning novelty of the work, I lean towards weak rejection of the work.", "strength_and_weaknesses": "1. The experiments clearly demonstrate that the proposed approach is highly effective. It is very interesting that the GNN-safe model alone can provide substantially high detection performance.\n2. The experiments in table 8 comparing the runtime of all the methods are further useful in demonstrating the effectiveness of the proposed approach.\n4. Eq 11 in the appendix should have a sum instead of difference.\n5. The authors mention in the last paragraph of page 4 that - “In specific, the generation of a node is conditioned on its neighbors, and thereby an in-distribution node tends to connect with other nodes that are sampled from in-distribution, and vice versa”. It would be more useful to provide some citations for this. Though they mention in the subsequent paragraph that the labels and features for the neighbors can be different, in such heterophilous graphs, the opposite trend may also hold. \n6. There is strong emphasis that the energy of in-distribution samples tend to be lower than the out of distribution samples, however, the opposite trend emerges in figure 1 (a). Do the authors have some intuition of why this is happening?\n7. There is no mention of the selection of the hyperparameter $\\tau$, eq 8. Details and study of this hyperparameter will be highly relevant.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PX5BTgqvVV", "reviewer": "Reviewer_BViK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed an intrinsic out-of-distribution (OOD) discriminator for semi-supervised learning on graphs based on an energy function called GNN-Safe, to enhance the reliability of the model against OOD data. The authors consider the interdependence between input data and other nodes in the graph rather than treating them independently. Experiments are compared with other models on five real-world datasets.", "summary_of_the_review": "This paper is easy to read and proposes a novel approach based on the energy function to detect OOD data. Overall, this work is meaningful in OOD detection on graphs and can be further studied and optimized.", "strength_and_weaknesses": "**Strengths:**\n1. The experiments in this paper are detailed in comparison with different methods under different datasets and GNN encoders.\n2. This paper is well-written and easy to understand.\n3. The authors provide proof for all theoretical results.\n\n**Weaknesses:**\n1. In the Introduction, it is said that “The scarcity of labeled training data in graph domain requires the model to exploit useful information from the unlabeled portion of observed data.” This statement is confusing because it is not the actual reason why semi-supervised learning is widely used in graph-based learning and performs well [1]. However, this paper claims to use semi-supervised learning to solve the problem of limited labeled data in graph datasets.\n2. There is no detailed description of the evaluation metrics. In addition, GPN compares OOD-Acc for different methods, which is also an important metric to evaluate the performance of OOD data [2].\n3. In figure 1, compared with chart (a), the distribution center of the green curve (in-distribution) is confusing in chart (b).\n4. In Section 4.1, it is said that “use the subgraph DE as in-distribution data and other five subgraphs as OOD data for Twitch.” However, the authors don’t explain why different subgraphs follow different distributions.\n \n[1] Graph-based Semi-supervised Learning: A review. Neurocomputing, 2020.  \n[2] Graph Posterior Network: Bayesian Predictive Uncertainty for Node Classification. NeurIPS, 2021.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Dm2v-e2C38", "reviewer": "Reviewer_2RDq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed an energy-based out-of-distribution detection for graph neural network. The contributions include introducing energy-based method to graph out-of-distribution detection, providing some theoretical analysis, presenting energy propagation, and conducting a lot of experiments.", "summary_of_the_review": "It seems that the main improvement over existing work such as Liu et al 2020 is that the proposed method focuses on GNN. Thus the contribution is not that significant. On the other hand, the numerical results indicate that OOD exposure plays a very important role in the success of the proposed method. It is not clear whether a direct binary classification between in-distribution and out-distribution can yield good OOD performance or not. ", "strength_and_weaknesses": "Strength:\n1. The paper is well-written and the motivation is clear.\n2. The numerical results showed that the proposed two methods especially GNN-Safe-r are much better than the baselines.\n\nWeakness:\n1. The main idea is almost the same as that of Energy FT (Liu et al. 2020). It seems that the major difference is the proposed methods are for graph neural network rather than MLP and CNN.\n\n2. In the experiments, what is the main difference in terms of the implementation between Energy FT and the proposed GNN-Safe-r?\n\n3. One can observe that in many cases, GNN-Safe was outperformed by Energy FT, which verified the importance of OOD exposure in the training stage. Thus, one may train a binary classifier to classify the in-distribution samples and out-distribution samples into two different classes, which should have a good performance in the testing stage.\n\n4. The OOD exposure violates the principle of OOD detection because it becomes a supervised learning problem with OOD labels.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster