EGonc : Energy-based Open-Set Node Classification with substitute Unknowns
Authors: Qin Zhang
Zelin Shi
Shirui Pan
Junyang Chen
Huisi Wu
Xiaojun Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Open-set classification
Energy-based models
Graph learning
Primary area: graph_neural_networks
TL;DR: An energy-based open graph learning method for open-set node classification.

Abstract: Open-set Classification (OSC) is a critical requirement for safely deploying machine learning models in the open world, which aims to classify samples from known classes and reject samples from out-of-distribution (OOD). 
Existing methods exploit the feature space of trained network and attempt at estimating the uncertainty in the predictions.
However, softmax-based neural networks are found to be overly confident in their predictions even on data they have never seen before and
the immense diversity of the OOD examples also makes such methods fragile.
To this end, we follow the idea of estimating the underlying density of the training data to decide whether a given input is close to the in-distribution (IND) data and adopt Energy-based models (EBMs) as density estimators. 
A novel energy-based generative open-set node classification method, \textit{EGonc}, is proposed to achieve open-set graph learning. 
Specifically, we generate substitute unknowns to mimic the distribution of real open-set samples firstly, based on the information of graph structures. 
Then, an additional energy logit representing the virtual OOD class is learned from the residual of the feature against the principal space, and matched with the original logits by a constant scaling. This virtual logit serves as the indicator of OOD-ness. 
EGonc has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for IND and OOD samples. 
Comprehensive experimental evaluations of EGonc also demonstrate its superiority.

EGonc Energy-based Open-Set Node Classification with substitute Unknowns
Qin Zhang'
Zelin Shi!
Huisi Wu' Xiaojun Chen' Shenzhen University Griffith University
Shirui Pan Junyang Chen'
ginzhangdszu. edu. en, shizel1n2021@maill szu edu. cn, s. pan@griffith edu. en, (junyangchen, hswu, x xceeen) @szu. edu. cn
Abstract
Open-set Classification (OSC) is a critical requirement for safely deploying ma- chine learning models in the open world, which aims classify samples from known classes and reject samples from out-of-distriutiion (OOD). Existing meth- ods exploit the feature space of trained network and attempt estimating the uncertainty in the predictions. However, softmax-basse neural networks are found be overly confident in their predictions even on data they have never seen be- fore and the immense diversity of the OOD examples also makes such methods fragile. To this end, follow the idea of estimating the underlying density the training data to decide whether given input is close to the in-distribution (IND) data and adopt Energy-based models (EBMs) as density estimators. A novel based generative open-set node classification method, EGonc, is proposed achieve open-set graph learning. Specifically, we generate substitute unknowns to mimic the distribution of real open-set samples firstly, based on the information graph structures. Then, an additional energy logit representing the virtual OOD class is learned from the residual of the feature against the principal space, and matched with the original logits by a constant scaling. This virtual logit serves as the indicator of OOD-ness. EGonc has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for IND and OOD samples. Comprehensive experimental evaluations of EGone also demonstrate its superiority.
Introduction
Learning on graphs, where instance nodes inter-connected, has become one of the central problems for deep learning, as relational structures are pervasive and induce data inter-dependence which hinders trivial adaptation of existing approaches that assume inputs be i.i.d. sampled (56). However, current models mostly focus on improving testing performance of in-distribution (IND) data and largely ignore the potential risk w.r.t. out-of-distribution (OOD) testing samples that may cause negative outcome if the prediction is overconfident on them [31]
Real-world applications often require machine learning systems interact with open world, violating the common assumption that testing and training distributions are identical. This urges the community to devote increasing efforts how to enhance models' generalization [9] and reliability [7 w.r.f. ofutt---iistiiuution data, by classifying samples from known classes and rejecting samples from dittof-distribution.
Existing methods addressing this task normally are achieved by assuming that a model is more confident in its predictions for closed-set samples than for open-set samples [55 g Based on this
"Correpooding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

assumption, threshold is applied to a model's 's confidence score to separate unknown samples from known ones. Despite its apparent intuitiveness [20), this method often fails because deep neural networks tend to be over-confident in their predictions, assigning high confidence scores even to
unknown inputs 19
42 71 Furthermore, determining an optimal threshold for distinguishing
known from unknown classes is a challenging and time-consuming task [38] 12 22] Others exploit
the feature space of trained network and attempt at estimating the density of in-distribution (IND)
features to address OOD detection [29], such as Gaussian Mixture Models [31 44 nearest neighbors
distribution [49] and energy logits 34 8] However, the immense diversity of the OOD examples makes such methods fragile [52]. For example, GMMs' density explicitly decreases when moving away from training data, making them effective for far- far-OOD detection, while energy logits benefits from the classifier training obtain strong results on near-OOD samples
[29] 522.
To overcome these limitations, researchers try to make advantages of unrelated samples in the open- set environment [21]. Nevertheless, the methods using real open-set samples to help the training of models often require substantial amount of data 23 48 5 and insouciantly selected data does not really help (66). Well Well-designed outlier samples are difficult to obtain and require experts for identification and labeling 66 25).
In this paper, we follow the idea of estimating the underlying density of the training data to decide whether given input is close to the IND data and adopt Energy- -ased models (EBMs) as density estimators. To further improve the OOD detection performance, we try to generate substitute unknown samples carefully under the guidance of graph structure, to mimic the distribution of real open-set classes. Specifically, novel energy-based generative open-set node classification method, EGone, proposed achieve open-set graph learning. Based on the generated substitute unknowns, an additional energy logit representing the virtual OOD class is learned from the residual of the feature against the principal space, and matched with the original logits by a constant scaling. This virtual logit serves as the indicator of OOD-ness. EGonc has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for IND and OOD samples. Comprehensive experimental evaluations of EGonc also demonstrate its superiority. To sum up, the contributions of this paper are summarized as follows:
A novel method, EGone, for open-set set node classification is proposed by redefining the open-world graph learning paradigm based on the energy model and elaborate unknown-substitue generation; EGone has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for IND) and OOD samples. The adopted energy regularization loss has consistent effects with the cross-entropy loss as well as with the tailored complement entropy loss on the known classes, and that they are not mutually exclusive;
No open-set data (samples of unknown classes or any side information of unknown classes) is required during training and validation. EGonc has an explicit classifier for unknowns and does not require threshold tuning:
EGone is agnostic to specific GNN architecture and demonstrates robust generalization capabilities. Experiments conducted on benchmark graph datasets illustrate the commendable performance achieved by EGonc.
2
Background
2.1 Open-set Node Classification
This paper focuses on the node classification problem for graph. Consider graph denoted as G = (V, X). where V= (u;li 1, N is a set of N nodes in the graph. E = fer ji.j = N.....N.i. j} is a set of edges between pairs of nodes i, and Vy. X E R Nxd denotes the feature matrix of nodes, and d is the dimension of node features. 2, X indicates the feature vector associated with each node n2. The topological structure of G is represented as an adjacency matrix A E NxN where Ar, 1 if the nodes i and dy are connected, iee,, Bes, E E, otherwise A., 0. Y E R NxC is the label matrix of G, where C' is the already- known node classes. If node i E V is associated with a label c, Vi = 1, otherwise, Vi
For typical closed-set node classification problem, GNN encoder Se, takes node features X and adjacency matrix as input, aggregates the neighborhood information and outputs representations. Then, classifier fo. is used to classify the nodes into C' already-known classes. The GNN encoder

---

and the classifier are optimized to minimize the expected risk 63 in Eq. I with the assumption that test data Dr and train data Dr share the same feature space and label space, i.e.,
arg min ((x.)))DD, [y + s(0g. Ozj A)) FEH
where H is the hypothesis space, I(() is the indicator function which outputs if the expression holds and 0 otherwise. Generally, it can be optimized with cross-entropy to discriminate between known classes.
In open-set node classification problem, given graph G = (V, -xx,) denotes the train nodes. Di (Xx: Y() ) is the test nodes, where X = U. (1. .........C..... C,C++,..... S.J. is the set of nodes that belong to seen classes that already appeared in Dr and U is the set of nodes that do not belong to any seen class (i.e., unknown class nodes). Open-set node classification aims to learn a (C + ))-clss classifier fe. that f(0g. Oo: X A) {Xee A} + y.y {l,..., C, unknown). with the minimization of the expected risk [63
I = arg min E SEH (a.D)~D, T(y # [(0g. 0 A))
(2)
where is the adjacency matrix for X The predicted class unknown y contains group of novel categories, which may contain more than one class. Thus, the overall risk aims to classify known classes while also detecting the samples from unseen categories as class unknown.
It is worth mentioning that open-set node classification problem is different with out-of-distributie (OOD) detection problem. The objective of OOD detection is to identify new sample whether from unknown classes or not, which is a binary classification challenge. Thereby, OOD detection techniques are not directly applicable open-set node classification scenarios. This emphasizes the crucial importance of conducting research on open-set node classification, highlighting the distinct requirements and challenges within this domain.
From close-set to open-set classifier, an intuitive way is thresholding 20 Taking the max output probability as confidence score, i.e., conf maas.mt......fftt.. A), it assumes the model is more confident in closed-set instances than open-set ones. Then we can extend a closed-set classifier by
arg max Se(wi, A), con >T
9:
c=I....
unknown,
otherwise.
where is a threshold. However, due to the overconfidence phenomena of deep neural networks, the output confidence of known and unknown is both high []]. As a result, tuning a threshold that well separates known from unknown is hard and time-consuming.
Differently, the approach presented in this paper redefines the paradigm of open-set graph learning from the perspective of energy models. well-designed generated unknown substitutes are introduced to aid in training of the energy model performance and elaborate loss function to improve classification performance. There are advantages of adding class instead of optimizing threshold as in previous methods 46 54 I], since does not require real open-set samples in validation and also eliminates the difficulty of tuning the threshold.
2.2 baseg Models
Energy-based models 30 use the energy-function Eg which defines a density over the data æ as Po- expl Es (z)) where are learnable parameters and Z(0) = ezp(-Eg) dr is the normalizing constant of the EBM. Ze ensures that the induced density function Equation integrates to I. However, -igh-dimensional integral. On the positive side, Eg can be any function Eg RD + IR placing no restrictions on the transformations compared to Normalizing Flows 68
Z(0)
Ze is often hard to compute or even approximate since
We can directly use the energy for OOD detection. That is, since a data point having high probability equivalent to having low energy as po(x) [x)x)) This means are not required estimate the normalizing constant (() in practice by considering a decision threshold 7 and binary classifier
G for OOD data
[0, if Eo(x) < T
G(x, 0, 0)
1, if Ex(x) >T
3

---

where 1 corresponds to IND samples and 0 to OOD samples.
We can additionally employ EBMs C-category classification problem discriminative EBMs
13
Suppose fo RDD + R is a classifier assigning logits for C classes for data point x E RR D The logits can be interpreted unnormalized probabilities of the joint distribution pa pa(z, yielding the marginal distribution over x as po(x) = Eg po(x, y). For training, the factorization can be used, le., po(x, (Ir 2) = log po(x) + log pa (yl ar), and a standard Cross Entropy loss employed to optimize pe (y (z).
=xp(f(2] [v)) 2
2
usually
To sum up, the connection between Energy- Model and discriminative neural classifier is established by defining the energy function as the negation of the predicted logit value: Exx, y) (x) Moreover, the energy function E(x) can computed for any given input
E(x) 1 log Σe -(x.'')
(5)
Nevertheless, to the best of our knowledge, existing research on energy-based modeling as well as its application is mostly concentrated OOD detection, where the primary emphasis is on i.i.d. inputs while the power of modeling inter-dependent data has remained further exploration. Besides, there have been relatively few attempts to apply energy-base models in open-world scenarios, and this is an endeavor we will also pursue.
3 Methodology
We propose a unified open-set node classification framework based energy scores and generated substitute unknowns. The energy scores mitigate a critical problem of softmax confidence with arbitrarily high values for OOD examples [18] and the differences of energies between in-distribution and out-of-distribution allow effective differentiation.
Specifically, GNN encoder fo, takes node features X and adjacency matrix A input, aggregates the neighborhood information and outputs representation he for each node in its k-th layer, = 1,..., K., and K. is the total number of layers. Thus, in the k-th layer, for node u; the GNN encoder aggregates neighbor information from the k-1 layer into a neighborhood representation:
h = f (01; h
k-1 nny .j N,)
where he E Rd is the hidden representation at the k-th layer, N₁ is the neighborhood of node and ho X.
Meanwhile, we mimic novel patterns by generating substitute unknown nodes through manifold mixup [7] Targeted node pairs are selected and mixed up at the middle layer of the GNN, and two kinds of substitutes are created: inter-class unknown substitutes that separate known classes from each other and external unknown substitutes that separate known from unknown. Generated substitutes and original known class nodes are input into the remaining layers together to learn discriminative representations.
Then, a classifier fea is learned to classify the nodes into C' + classes according to their energy scores. The energy scores of node i is obtained through energy propagation upon the graph topology, i.e.
E = (t N.)
where E.- = =- log Ey (h Ho'1 is the initial energy score of U;.
To conclude, the proposed EGone method consists of three main modules: 1) substitute node generation module create substitute unknown class nodes; 2) energy propagation module to obtain energy score of each node; and 3) an open-set classifier learning module to guarantee the classification of known classes and the rejection of the unknown class. We introduce them in details.
3.1 Substitute Unknown Generation
As mentioned previously, insouciantly selected unrelated data does not really help (66 the open-set classification. Thus, try to generate substitute unknown nodes strategically positioned at class

---

boundaries. These nodes play crucial role in the learning process by providing supervision and distinguishing between known and unknown nodes. Specifically, we generate two types of substitutes, inter-class unknown substitutes and external unknown substitutes, through manifold mixup [51] 17
For well-trained classifier, nodes with common categories tend to have similar characteristics, while those from different classes often have distinct ones. Ideally, well-defined boundaries separate the different classes. Considering that nodes close to these boundaries may have less representative features for their own classes, we chose to generate substitute unknowns using nodes close to these boundaries regions. As a result, samples belonging to different classes, but located close proximity, become optimal candidates for substitute generation. i.e., pairs of nodes in distinct classes connected by an edge. To be specifical, given two connected nodes from different categories, denoted as {(2,, Ve), (T by)}, where # Vj and aij = 1, their embeddings in the k-thh layer are h and hy. derived by inputting the graph into the network for The inter-class unknown substitute (Tz Mi) is obtained as follows:
ã = ah (01; A) (1 a) hy (01:2,
(8)
- C+
Where a E [0, is randomly sampled from Beta distribution, typically centred around 0.5. Vi C + denotes the categories index of ã;, indicating it belongs to the unknown class. Two edges between (z,, Æ2)) and (Iy, R,)) are also introduced into the graph. We refer to the set of generated inter-class samples as Xim- The purpose of Xin to simulate the distribution of unknown classes nodes existing between known classes. The mixed hidden representation ã; is then passed to fol to obtain the final representation T.ç- S (*:))
pk+1- +-IK
In addition the inter-class unknown substitutes, we also generate external unknown substitutes to reflect the unknown distributions at the periphery. We use peripheral nodes of each known category along with their respective class center generate these external unknown substitutes. The first kind of peripheral nodes are leaf nodes that belong to known classes, denoted as I; E Xur, s.t. E, Aij- = 1. The second kind are nodes with low classification confidence, namely those nodes with the top T least confident scores within each known classes. In this way, it allows to identify both structured- based and semantic-
peripheral nodes. We denote the set of peripheral nodes as Xper- Then, we obtain the embedding of class centers in the k-th layer based on the ground-truth annotations, i.e., h/f (01; æi, A), c = I,..., C, where X represents the nodes labeled with the class And we can use the manifold mixup on peripheral nodes and their respective inverted class centers [(rr, (I().): Ni E Xper: yi E (I..... C}} to derive the external unknown substitutes in the k-thh layer, i.e.
((e)) = [x _x,ex=
x. (8 2., A) + +n(-
8. = C +
where 3 > 0 and Y > 0 represent two hyperparameters used to adjust the distance and relative position between the generated samples and the existing known class samples. Furthermore, edges connecting (æ,, x; E Xper are also included in the graph. We denote the set of generated external unknown substitutes as Xex and define unknown substitutes as X Xmm Xext-
3.2 Energy Propagation
To further capture the diverse distribution of the unknown classes. we introduced an energy-based model. Through formula F((x,) y) = - h-(x)(x))x)) [u). we can establish bridge between the energy function and an open-set classifier.
Proposition 1. The energy score (Eq. (5) can be an powerful indicator for OOD detection due to its valuable trait: the yielded energy scores for in-distribution data typically exhibit a lower tendency than those of OOD data.
As an energy model graphs, in order to fully leverage the topological structure information of the graph, we apply energy propagation to the topological structure of the graph.
E*() = () + (1 - D) (k(k-1)
(10)
Here, A D 112 is the symmetrically normalized adjacency matrix of A, and D represents the degree matrix, E(*) = (k)
Iz E Du U X With 0 < Ç < as parameter, determine the
5

---

energy weights for individual nodes with respect to themselves and other connected nodes. The motivation behind topologiacl energy propagation incorporate the underlying physical processes involved in data generation by exploiting local interactions between instances. Since similar nodes are often adjacent in the graph, the model can learn relationships between these nodes through energy propagation. This helps to better capture local structures and similarities within the graph.
Proposition 2. For node 24, if its energy score E(*=1) is greater (resp. less) than the average of the energy scores of its one-hop neighbors at the current layer, i.e., 2. A. A, 1) E; A, then the updated )(
energy score of its own yiclds E!!A
( (resp, E
Due structural homogeneity (60 28 known classes samples and unknown substitutes often connect within their respective distributions, indicating a correlation. Therefore, by using this energy propagation formula, it could ensure the average energy scores of the known classes samples decrease while the average energy scores of the unknown substitutes increase.
3.3 Open-set Classifier Learning
To address the diverse compositions of known and unknown categories, it is imperative extract invariant information from both known and unknown classes. Consequently, the backbone GNN network fe, shared across the C + classes, facilitating the learning of class distribution and node representations.
To calibrate the closed-set (known classes) classifier to an open-set classifier, additional category introduced in the final classification layer to handle unknown predictions. Suppose the weights of the closed-set classifier are denoted as Welose E R where de represents the dimension of the embeddings in the final layer of the GNN, the open-set classifier formed by combining the dlseed-set classifier and the substitute classifier, i.e., 02 = [Watose: Wsab] E Rdex(C+1) where Wyub E ddxx represents the weights associated with the substitute classifier. For simplicity, the bias term is omitted in this context. Then, the integrated classifier is trained using both the original known classes samples and the generated substitute samples, i.e., D'r = Drr U (Xsub, Yc+1), X Xuu using the cross entropy loss, i.e.,
4 =
Ice( B4) + A1 (ZL.
Σ
E Z2EN
Icn(00 1+
(11)
where 9; = 8(Fe (Se, (I;)) is the output vector of the open-set classifier for the input node a;; through softmax s.)). (yc, v:) = yi log ÿi is the cross entropy loss.
4 loss primarily uses information from the ground-truth class to maximize the likelihood, it tends to overlook the influence of complementary classes (i.e., incorrect classes) ( Therefore, we incorporate the concept of complement entropy 3 to complement the softmax cross entropy. to neutralize the effects of complementary classes.
Specifically, for the generated unknown substitutes, we minimize their inherent complementary loss, i.e., minimize the average of entropy over the C' known classes, considering that the generated substitutes may be adjacent to any of the known classes. Conversely, for nodes belong to known classes, they predominantly adjacent
the generated substitutes, i.e., the C + class, encourage the substitute classifier to assign the second-largest probability to these nodes. In light of these considerations, we introduce tailored complement entropy loss for EGonc as follows:
Σ (DD,
(CHE 1)+
Σ I
(ooE(ii y1)
(12)
where He \y denotes the prediction logits after excluding the probability of its corresponding ground- truth label. IC.E = = - - I-Ii., Bic__ log I- _ VV(,, Ui) E Drr- The first term of l2 aims to minimize the misclassification of known class nodes into other known classes. By employing l2 loss, the world classifier Jo, can accurately classify node belonging to known classes into their respective known class, while leveraging the substitute classifier to establish an appropriate boundary for distinguishing between the known and the unknown.

---

To mitigate the impact of energy attenuation on unknown substitutes during the energy propagation, we further introduce strict constraints via energy regularization 13 to the model. Drawing inspiration from the Elastic Network 73 10], we observe parallel in structure and purpose between regular- ization terms and their associated error terms. light of this observation, we adopt a strategy that incorporates both linear loss and quadratic loss terms to penalize errors effectively. The linear error terms facilitate the generation of sparse solutions, thereby enhancing the model's robustness, while the quadratic error terms adept at accommodating outliers and achieving better fit to the data. By integrating these two kinds of error terms, we aim to enhance the adaptability and generalization capacity of the model.
k1 Σ (zind( ZrEDir
Σ ((oood(Z))) K2( Σ a,EDr
oWiin(Zz)))+
Σ E
aBoood (zzz) ())))
(13)
where (.) is the LeakyReI.U function. k1 k2 are the weights for linear and quadratic enror terms, respectively. li, laut are hyperparameters. Find (a:) = F( A; he) lin, Var, E D'r and (x)) = fout - Fx A; ho). Vzz E Xub represent the energy error terms correspond to the IND data and the OOD data, respectively. Here, E. (I,, A; hg) represents the energy value of sample I, at the final layer. In this way, the energy scores learned with the constraint of la are of benefit to open-set classification, and it also mitigate the excessive attenuation energy scores for unknown substitutes caused by energy propagation.
Proposition 3. For any energy regularization 13 that ensures that the GNN model hg- minimizing 13- where ti < fout are two margin parameters, then the corresponding softmax categorical distribution plylær Gz; hgg)) ) also minimizes 4, while p(C + 1|x, 9z: hg-) similarly minimizes lz. This means l3 is optimized in the same direction as 1 and l2 for ind data.
Finally, the total loss of EGonc is combination of cross entropy, the tailored complement entropy loss, and energy regularization loss, i.e., Lomal 1 + A212 + Agla
(14)
where d2 and A3 are the hyperparameters balance the losses. The algorithm of EGonc and its complexity analysis are provided in Appendix D
4 Experiments
Experiments were carried out to validate the performance of EGone. These experiments include: open-set node classification comparison, ablation study, parameter study, and generalization analysis. Code are available at http:gghiwh...o////o////////oooa
Datasets. Experiments to evaluate the performance for open-set node classification were mainly performed on five benchmark graph datasets [54] 72]. namely Cora Citeseer DBLF" PubMed
and Ogbn arxiv (24 45), which are widely used citation network datasets. Statistics are presented
in Appendix E.2
Metrics. Accuracy and Macro-F1 are used for performance evaluation.
Implementation Details. Generally, EGone adopt GCN 228 as the backbone neural network for experimental evaluation unless otherwise specified. Detailed model parameters and platform
information are given in Appendix E.3
Test settings. Two kinds of open-set classification evaluations were conducted consider short or long distances between known classes and unknown classes, i.e., near open-set classification and far open-set classification. Specifically, in the near open-set classification experiment, for each dataset, the data of several classes were held out as the unknown classes for testing and the remaining classes
"herps//pubmme..nhhiii nlm. www..ii..gmm..mmmmmmmm.... gov/downlodd/ hhtts://githubbc soapsnap- stanford/ogb/

---

Table 1: Near open-set classification five citation network datasets with unknown class (u=1)
in the inductive learning setting. Numbers reported are all percentage (%).
Cora
Citeseer
DBLP
Pubmed
Ogbn_arxiv
arxiv
Average
Methods
Acc
F1
Acc
F1
Acc
Acc
F1
Acc
F1
Acc
F1
GCN_soft soft
70.6
67.6
44.6
38.9
63.8
59.2
28.9
29.9
49.8
17.5
51.5
42.6
GCN_sip
69.2
64.7
45.3
44.5
63.5
58.7
28.9
29.8
48.8
9.5
51.1
41.4
GCN LIN_soft_
73.6
73.8
57.3
54.5
65.0
62.4
49.7
48.6
47.3
20.6
58.6
52.0
LEN_sig_*
79.7
80.1
62.1
54.6
69.2
68.2
45.1
46.0
46.0
8.3
60.4
51.4
Openmax
74.6
75.1
56.2
54.5
67.2
67.2
49.1
48.7
45.5
16.3
58.5
52.4
DOC
77.8
78.1
66.0
56.7
69.9
69.2
45.6
46.2
46.7
20.7
61.2
52.2
PROSER
83.2
83.7
73.7
63.6
71.7
72.6
71.0
58.4
53.0
31.1
70.5
61.9
OpenWGL
78.1
78.9
64.1
60.8
71.4
72.2
65.3
63.4
45,4
20.7
64.9
60.2
GNNSAFE
79.6
81.0
69.8
60.3
72.5
74.1
70.1
66.8
51.2
24.2
68.6
61.3
Pry
84.3
84.8
75.5
71.0
77.3
79.0
73.7
70.2
62.7
33.0
74.7
67.6
EGonc
84.5
84.9
75.8
71.5
79.1
80.8
80.2
75.5
63.0
33.0
76.5
69.1
were considered as the known classes. 70% of the known class nodes were sampled for training, 10% for validation and 20% for testing. In the far open-set classification experiment, nodes from other datasets were used as unknown class samples for testing, other than the nodes from the dataset used in training.
Besides, comparison is also provided for different settings in terms of the availability of side information on unknown classes during training or validation, known as the inductive learning setting and the transductive learning setting. In experiments with inductive learning setting, there is not any information about the real unknown class (such as the features or side information of unknown classes) used during training or evaluation, while under the transductive learning setting, the whole graph (including sampled known class nodes and unlabeled unknown class nodes) are input during model training or validation.
Baselines. We compare EGonc with ten baselines, which can be classified to four categories.
1) Closed-set classification methods: GCN soft and GCN _Sg. They are GCNs 28 with a softmax layer multiple 1-vs-rest of sigmoids layer output layer.
2) Open-set classification methods with thresholds: CCN__sof__T, GCN _ig__, GCN_sig_r DOC [46] and Open WGI. [54]. 'he threshold is chosen from 0..1,0.2.... ... 0.9) as the descrip-
Openmax 1.
tion the original paper, to perform open-set recognition.
3) Generative Open-set Classification methods: PROSER 70 and g Pry 67
4) Energy-based methods: GNNSAFE [56].
A detailed introduction of the baselines can be found in the Appendix E.1
4.1 Open-set node classification comparison
Since real-world scenarios are complex, where seen and unseen differs in diverse tasks, we evaluate our model terms of open-set classification from two aspects: near open-set classification, and far open-set classification under inductive and transductive learning setting, respectively, introduced in test settings.
4.1.1 Near open-set classification
Table presents the accuracy and macro-F1 scores of the methods applied to the near open-set classificatin task in the inductive learning setting, where the last class of each dataset is designated as the unknown class (i.e,, = 1), and the remaining classes are used for model training. It can be observed that EGone outperforms all the baselines on five datasets. This shows that EGone can better differentiate between known class and an unknown class, though they are similar to each other. Specifically, compared to the second-best method G2 Pay, BGone achieves 2.41 and 2.22% improvements average terms of accuracy and Fl the five datasets,

---

Table 2: Accuracy and macro-F1 scores of EGonc and its variants with respect to different losses and generation strategies.
Components
Cora
Citeseer
DBLP
Pubmed
O L_arxiv
Average
2
3
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
Fl
84.2
84.7
75.2
69.0
76.5
77.7
70.1
47.3
61.9
34.1
73.6
62.6
V
84.3
84.8
75.5
71.0
77.3
79.0
73.7
70.2
62.7
33.0
74.7
67.6
v
84.5
84.9
75.8
71.5
79.1
80.8
80.2
75.5
63.0
33.0
76.5
69.1
far
X
rand
X
X
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
82.7
83.2
73.5
69.6
69.5
71.3
70.4
67.2
60.1
30.0
71.2
64.3
83.7
84.0
75.5
66.9
72.3
72.7
71.8
68.5
62.3
29.3
73.1
64.3
81.3 82.2
74.6
63.7
71.2
71.5
66.9 70.0
61.9
32.3
71.8
63.3
84.2 84.7
75.3
70.8
75.3
76.9
73.4
68.7
62.3
31.4
74.1
66.5
84.1 84.6
75.4
70.9
75.5
74.8
71.4
66.9
61.5
29.5
73.6
65.3
84.0
84.4
75.7
71.2
72.0
71.7
73.0
69.1
61.9
32.0
73.3
65.7
84.5
84.9
75.8
71.5
79.1
80.8
80.2
75.5
63.0
33.0
76.5
69.1
We illustrate detailed classification accuracy in terms of known classes and unknown classes in Appendix E.4] And the results shows that in order to gain the ability of unknown class detection, there slight decrease in the performance of known class classification, i.e. from 79.5% 76.3% on average, comparing EGonc to closed-set classification method GCN soft. However, the unknown class detection accuracy is improved from 0% 76. 1% on average, which remarkable. Besides, we also evaluate the performance of our model in terms of multiple unknown classes. The results are demonstrated in Appendix En5 and can be observed that EGone consistently outperforms baselines. We further evaluate our model in terms of near open-set classification under transductive learning setting. The results are shown Appendix E.6 EGone consistently performs best.
4.1.2 Far open-set classification
For far open-set classification, follow- ing the protocol defined in (38), the models are trained and validated by training and validation instances of the original dataset (IND data). While for testing, instances from another dataset are augmented to the origi- nal as open-set classes (OOD data). for example, Co_ Ci means that Core as IND data and Citeseer as OOD data.
Table 3: Accuracy and macro-F1 for far open-set classifi-
cation on benchmark datasets. Numbers reported are all percentage (%)
Co Ci
Ci DB
DB Pub
Average
Methods
Acc
F1
Acc FI
Acc
FI
Acc
FI
GCN_ soft
43.0 58.9
38.4
42.5
41.9
53.7
41.1 51.7
GCN sig
41.6 57.5
36.3
42.1
41.6
45.2
39.8
48.3
GCN soft
81.2 77.6
86.2
71.1
85.0
75.6
84.1
74.8
GCN sig T
69.4 51.8
68.7
48.0
79.8
69.1
72.6 56.3
Openmax
56.2 55.1
69.6
60.3
69.6 58.7
65.1 58.0
The results for shown in Ta- ble 3 It is found that EGone can handle far out-of-distribution classes from diverse inputs and achieve no- ticeable performance improvement compared to other open-set clas- sification methods. Surprisingly, the simple thresholding approach of SCN__soft_
DOC
69.4 57.8
75.5
62.3
78.0 70.7
74.3 63.6
PROSER
78.5 79.1
81.5
66.4
78.6 69.0
79.5
71.5
OpEnWGL
80.6
76.7
44.6
11.9
84.6 70.7
69.9
53.1
GNNSAFE
79.3
79.9
80.9
65.9
80.0 65.0
80.1
70.3
Pay
81.3
80.5
87.5
74.4
86.5 72.3
85.1
75.7
EGone
81.7 81.0
88.1
75.2
87.2 72.8
85.7 76.3
GCN soft_ 5 achieves comparable performance with EGone. This inspires us to combine the genera- tive methods with discriminative methods for far open-set classification in future work.
4.2 Ablation Study
We compare variants of EGone with respect to loss function and substitute-unknoot generation strategy to demonstrate its effect. As shown in Table we firstly verify the effect of each loss we adopted, i.e., (C + cllass cross-entropy loss 4, tailored complement entropy loss l2, and energy regularization loss lg. The results show that these three losses are indispensable to open-set node classification. Then verify the effect of the substitute unknown samples generated. Specifically,

---

we compare the performance of EGonc with respect to different kinds of (substitute) unknown samples: Xar which are real far OOD samples, Xnnd which are substitute unknowns generated via miaup with random parent nodes; Xim which are generated inter-class substitute unknowns, and Nex which are generated external substitute unknowns. From the results, we can see that EGone generally achieves higher accuracy with the assistance of auxiliary unknown class samples. However, well-designed unknown substitutes are most beneficial for open-set node classification.
4.3 Generalization Analysis
The proposed model has no specific requirement on the GNN architecture for classification, The unknown-class substitute generation strategy and energy propagation take into account the topological properties of graph data. Therefore, they efficiently achieve the generation of representative unknown class samples and perform in-depth exploration of unkown-class features across different backbones. This design contributes to the model's performance in open-set classification tasks under different backbones. Table ilustrates the performance of the proposed EGonc with different GNN architec- tures, including GCN, GAT and GraphSage. The results confirm the effectiveness and generalization ability of EGonc for open-set node classification.
Table 4: Accuracy and macro-Fl scores of open-se setn-set classification methods with different backbone neural network. Numbers reported are all percentage (%).
Cora
Citeseer
Dblp
PubMed
Ogbn
arxiv
Methods
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
GCN soft_T
73.6
73.8
57.3
54.5
65.0
62.4
49.7
48.6
47.3
20.6
GCN DOC
77.8
78.1
66.0
56.7
69.9
69.2
45.6
46.2
46.7
20.7
GCN Openmax
74.6
75.1
56.2
54.5
67.2
67.2
49.1
48.7
45.5
16.3
GCN G2y
84.3
84.8
75.5
71.0
77.3
79.0
73.7
70.2
62.7
33.0
GCN EGonc
84.5
84.9
75.8
71.5
79.1
80.8
80.2
75.5
63.0
33.0
GAT_soft__ _
71.6
69.2
58.9
51.1
65.4
66.6
43.2
43.7
49.1
16.7
GAT _DC
71.1
72.6
62.4
59.5
64.2
61.8
42.1
42.9
48.3
16.2
GAT SAT_Openman
66.3
63.4
48.6
48.9
62.5
56.9
48.6
47.0
32.2
8.4
GAT_GaPP
80.4
81.0
75.2
70.9
72.9
73.7
71.7
47.0
53.7
22.6
GAT_EGon
80.8
81.3
75.3
71.0
73.1
74.0
74.3
63.6
56.1
24.5
Grappsagee_oftt_>
72.7
72.9
63.5
51.2
64.3
64.0
46.6
46.9
51.5
16.0
Graphsage_DOO DOC
76.0
75.4
63.6
59.9
68.9
72.2
44.6
45.7
49.5
14.7
Graphsage_Opente Openmax
71.1
70.6
47.9
48.7
62.3
56.9
44.4
45.1
43.2
8.0
Graphsage g² Pry
87.2
87.3
78.6
76.9
74.4
74.7
72.8
64.9
62.8
36.5
Graphsage_ EGonc
87.3
87.3
79.5
77.4
78.0
79.6
73.0
65.0
63.4
38.4
Limitation and Conclusion
This paper proposed novel energy-based generative open-set node classification method, EGonc, by estimating the underlying density of the training data to decide whether a given input is close to the IND data. To obtain better OOD detection capability, we generate substitute unknowns mimic the distribution of real open-set samples, and use energy logit to represent the indicator of OOD-ness. Under constraint of cross entropy loss, complement entropy loss, and energy regularization loss, EGonc achieves superior effectiveness for unknown class detection and known class classification, which is validated by experiments. EGone has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for IND and OOD samples. EGone also has good generalization since it has specific requirement on the GNN architecture.
10

---

Acknowledgments
This research was supported by National Natural Science Foundation of China (62206179, 92270122). Guangdong Provincial Natural Science Foundation (202211515010129, 2023A1515012584))
References
[1] Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In IEEE conference on computer vision and pattern recognition, pages 1563-1572, 2016.
[2] Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. Advances neural information processing systems, 13, 2000.
[3] Hao- Yun Chen, Pei-Hsin Wang. Chun-Hao Liu, Shih-Chieh Chang, Jia- Yu Pan, Yu-Ting Chen, Wei Wei, and Da- Da-Cheng Juan. Complement objective training. In International Conference on Learning Representations, 2018
[4] Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International Conference on Machine Learning, pages 1725-1735. 2020.
[5] Yue Chen, Yalong Bai, Wei Zhang, and Tao Mei. Destruction and construction learning for fine-grained image recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5157-5166, 2019.
[6] Yifeng Ding, Zhanyu Ma, Shaoguo Wen, Jiyang Xie, Dongliang Chang, Zhongwei Si, Ming Wu, and Haibin Ling. Ap-enn: Weakly supervised attention pyramid convolutional neural network for fine-graie graingr visual classification. IEEE Transactions on Image Processing, 00:2826-2836 2021.
[7] Yilun Du, Joshua Meier, Jerry Ma, Rob Fergus, and Alexander Rives. Energy-based models for dromic-resolution protein conformations. In 8th International Conference on Learning Representations, ICLR 2020, 2020.
[8] Sven Elflein. Out-of-distribution detection with energy-based models. arXiv:2302. 12002, 2023.
arXiv preprint
[9] Shaohua Fan, Xiao Wang, Chuan Shi, Peng Cui, and Bai Wang. Generalizing graph neural networks on -fitof-distribution graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[10] Jerome Friedman, Trevor Hastie, and Rob Tibshirani. Regularization paths for generalized linear models via coordinate descent. Journal of statistical software, 33(1):1,201 2010.
[11] Gao, Song, B Poole, YN Wu, and DP Kingma. Learning energy-based models by diffusion recovery likelihood. In International Conference on Learning Representations (ICLR 202/). 2021.
[12] Zong Yuan Ge, Sergey Demyanov, Zetao Chen, and Rahil Garnavi. Generative openmax for multi-class open set elassification. In British Machine Vision Conference. BMVA Press, 2017.
[13] Will Grathwohl, Chin- Wang, Benrik Jacobsen, David Duvenaud. Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXiv: arXiv: 992...26....... 2019.
[14] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. International conference on machine learning, pages 1321-1330, 2017.
[15] Tuomas Haarojj, Haoran Tang, Pieter Abbecl, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pages 1352-1361, 2017.
[16) Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Neural information processing systems, 30, 2017.

---

[17] Xiaotian Han, Zhimeng Jiang, Ninghao Liu, and Xia Hu. G-mixup: Graph data augmentation for graph classification. In International Conference on Machine Learning, pages 9230-8248.
PMLR, 2022.
[18] Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 41-50, 2019.
[19] Matthis Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-sonfidence predictions far away from the training data and how to mitigate the problem. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 41-50, 2019.
[20] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distributiin examples in neural networks. International Conference on Learning Representations, 2017.
[21] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint 1rXiv:1812.04660 2018.
[22] Marcel Hoffmann, Lukas Galke, and Ansgar Scherp. Open-world lifelong graph learning. In 2023 International Joint Conference on Neural Networks (IJCNN), pages 1-9. IEEE, 2023.
[23] Tao Hu, Honggang Qi, Qingming Huang, and Yan Lu. See better before looking closer: Weakly supervised data augmentation network for fine- ane-grained visual classification. arXiv preprint arXiv:1901.0988 2019.
[24] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118 11-22118-2213, 2020.
[25] Ruyi Ji, Longyin Wen, Libo Zhang. Dawei Du, Yanjun Wu, Chen Zhao, Xianglong Liu, and Feiyue Huang. Attention convolutional binary neural tree for fine- ane-ggained visual categorization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pages 10468-10477, 2020.
[26] Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and S Yu Philip. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Transactions on Neural Networks and Learning Systems, 33(2):494-514, 2021.
[27] Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint 1rXiv:1611.07300 2016.
[28] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference Learning Representations, 2017.
[29] Mare Lafon, Clément Rambour, and Nicolas Thome. Heat: Hybrid energy based model in the feature space for out-of-distribution detection. In NeurlPS ML Safety Workshop. 2022.
[30] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on energy- based learning. Predicting structured data, 1(0), 2006.
[31] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018
[32] Zhao Li, Xin Shen, Yuhang Jiao, Xuming Pan, Pengcheng Zou, Xianling Meng, Chengwei Yao, and Jiajun Bu. Hierarchical bipartite graph neural networks: Towards large-scale e-commerce applications. In IEEE International Conference on Data Engineering, pages 1677-1688, 2020.
[33] Qi Liu, Miltiadis Allamanis, Mare Brockschmidt, and Alexander Gaunt. Constrained graph variational autoencoders for molecule design. Neural information processing systems, 31, 2018.
[34] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based -ustr--dittiniution detection. Advances in neural information processing systems, 14:1444-221755 2020.
12

---

[35] Xin Liu, Mingyu Yan. Lei Deng, Guoqi Li, Xiaochun Ye, Dongrui Fan, Shirui Pan, and Yuan Xie. Survey on graph neural network acceleration: An algorithmic perspective. arXiv preprint arXiv:2202004822, 2022.
[36] Yixin Liu, Ming Jin, Shirui Pan, Chuan Zhou, Yu Zheng, Feng Xia, and Philip Yu. Graph self-supervised learning: A survey, IEEE Transactions on Knowledge and Data Engineering, 2022.
[37] Yadan Luo, Zijian Wang, Zi Huang, and Mahsa Baktashmotlagh. Progressive graph learning for open-set domain adaptation. In International Conference on Machine Learning, pages 6468 6478, 2020.
[38] Pramuditha Perera, Vlad I Morariu, Rajiv Jain, Varun Manjunatha, Curtis Wigington, Vicente Ordonez, and Vishal M Patel. recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11814-11823 2020.
feature representations for open-set
[39] Phillip E Pope. Soheil Kolouri, Mohammad Rostami, Charles E Martin, and Heiko Hoffmann. Explainability methods for graph convolutional neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10772-10781,
[40] Marc' Aurelio Ranzato, Y-Lan Boureau, Sumit Chopra, and Yann LeCun. A unified energy- based framework for unsupervised learning. Artificial Intelligence and Statistics, pages 371-379. PMLR, 2007.
[41] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks, 20(1):61-80, 2008.
[42] Walter J Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recognition. IEEE transactions on pattern analysis and machine intelligence, 35(7): 1757-1772, 2012.
[43] Walter J Scheirer, Lalit P Jain, and Terrance E Boult. Probability models for open set recognition. IEEE transactions on pattern analysis and machine intelligence, 36(1 (66611):231722244 2014.
[44] Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for superviee outlier detection. arXiv preprint arXiv: 103. 12051, 2021.
[45] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi- Rad. Collective classification in network data. magazine, 29(3):93-93, 2008
[46] Lei Shu, Hu Xu, and Bing Liu. Doc: Deep open classification of text documents. In Proceedings of the 2017 Conference on Empirical Methods in Natural Lunguage Processing, pages 2911- 1- 2916, 2017.
[47] Sireesha, Srinivasa Rao, and M Vijay Kumar. Graph theory based transformation existing distribution network into clusters of multiple micro-grids for reliability enhancement. Materials Today: Proceedings, 80:2921-2923 2023.
[48] Ming Sun, Yuchen Yuan, Feng Zhou, and Errui Ding. Multi-attention multi-class constraint for fine-graine image recognition. In Proceedings of the european conference computer vision (ECCV), pages 805-821, 2018.
[49] Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning, pages 20827-20840. PMLR, 2022.
[50] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
[51] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pages 6438-6447 2019.
13

---

[52] Haoqi Wang. Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: virtual-logit matching. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 4921-4930, 2022.
with
[53] Max Welling and Thomas N Kipf. Semi-supervised classification with graph convolutional networks. In J. International Conference on Learning Representations, 2017.
[54] Man Wu, Shirui Pan, and Xingquan Zhu. Openwgl: Open-world graph learning. In IEEE international conference on data mining, pages 681-690, 2020.
[55] Man Wu, Shirui Pan, and Xingquan Zhu. Openwgl: open-world graph learning for unseen class node classification. Knowledge and Information Systems, pages 1 1-26, 2021.
[56) Qitian Wu, Yiting Chen, Chenxiao Yang, and Junchi Yan. Energy-baset detection for graph neural networks. arXiv preprint arXiv:2302. 02914, 2023.
[57] Z Wu, S Pan, G Long, J Jiang, and Zhang. Graph wavenet for deep -enttalleemprall graph modeling. In The International Joint Conference on Artificial Intelligence, 2019.
[58] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32(1 2:4 2020.
[59] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:/810.000 1810.0082, 2018.
[60] Keyulu Xu. Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International conference machine learning, pages 5453-5462. PMLR, 2018.
[61] Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gnnexplainer: Generating explanations for graph neural networks. Neural information processing systems, 32, 2019.
[62] Jiaxuan You, Jonathan M Selman, Rex Ying, and Jure Leskovec. Identity-aware graph neural networks. In AAAI Conference on Artificial Intelligence, volume 35, pages 10737-10745, 2021.
[63] Yang Yu, Yaig Qu, Nan Li, and Zimin Guo. Open-category classification by adversarial sample generation. In International Joint Conference on Artificial Intelligence, pages 3357- 3363, 2017.
[64] Hanqing Zeng, Muhan Zhang, Yinglong Xia, Ajitesh Srivastava, Andrey Malevich, Rajgopal Kannan, Viktor Prasanna, Long Jin, and Ren Chen. Decoupling the depth and scope of graph neural networks. Neural Information Processing Systems, 14:19665-1967, 2021.
[65] He Zhang, Bang Wu, Xingliang Yuan, Shirui Pan, Hanghang Tong, and Jian Pei. Trustworthy graph neural networks: Aspects, methods and trends. arXiv preprint arXiv:2205.0742 2022.
[66] Jingyang Zhang, Nathan Inkawhich, Randolph Linderman, Yiran Chen, and Hai Li. Mix- ture outlier exposure: Towards out-of-distribution detection in fine-grained environments. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5531-5540, 2023.
[67] Qin Zhang, Zelin Shi, Xiaolin Zhang. Xiaojun Chen, Philippe Fournier- Vournier-Viger, and Shirui Pan. G2pxy: generative open-set node classification on graphs with proxy unknowns. In Proceedings of the Thirry-Second International Joint Conference on Artificial Intelligence, pages 2023.
[68] Junbo Zhao, Michael Mathicu, and Yann LeCun. Easrgy-based generative adversarial networks. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
[69] Xin Zheng, Yixin Liu, Shirui Pan, Miao Zhang, Di Jin, and Philip $ Yu. Graph neural networks for graphs with heterophily: A survey, arXiv preprint arXiv:2202.07088 2022.
14

---

[70] Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Learning placeholders for open-set recognition. In the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2021.
[71] Shuang Zhou, Xiao Huang, Ninghao Liu, Qiaoyu Tan, and Fu-Lai Chung. Unseen anomaly detection on networks via heuti-hyperssphree learning. In Proceedings of the 2022 SIAM International Conference Data Mining (SDM), pages 262-270. SIAM, 2022.
[72] Qi Zhu, Chao Zhang, Chanyoung Park, Carl Yang, and Jiawei Han. Shift-robust node classifica- tion via graph adversarial clustering. arXiv preprini arXiv:2203. 25222003..02,, 2022.
[73] Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B: Statistical Methodology, 67(2)3301-320, 2005.
15

---

A Appendix supplemental material
B Related works
B.1
Graph Neural Networks
Graph neural networks deep learning based models that apply neural networks to graph learning and representation [41]. They are used in wide variety of tasks [53 50] and have a strong theoretical foundation [39 61.. GNNs were shown yield state-of-the-ar performance on many graph-related tasks 36 35 and have applications in various fields such as e-commerce 32]. traffic analysis [57].
chemistry and for knowledge bases [26
The original GNN model brought the idea of combining the design of modern neural networks
with graph learning [58], e.g.. Graph Convolutional Networks (GCNs) [28), Graph Attention
Networks (GAT') (50, Graph Isomorphism Networks (GINs) [59], and GraphSage 16]
Then
various improvements have been made to the basic GNNs for different purposes 33
35
69
65
For instance, to alleviate the over-smoothing problem, decoupling mechanisms (44] and identity
mapping 4 were introduced respectively. To increase the expressive power of most GNNs by the I-Weisfeiler-Lehman (I-WL) graph isomorphism test, identity-aware Graph Neural Networks [62] were proposed.
B.2
Open Set Node Classification
require classifiers capable of categorizing known classes and detecting objects of unknown types during testing. Two main paradigms have been investigated: generative models and discriminative models. Generative models improve prediction performance by emulating the distribution of unknown class nodes, while discriminative models use a threshold to distinguish between known and unknown classes, thereby improving prediction accuracy.
Open set recognition and classification
Prior methods, such as OpenMax [[ and DOC [46], mainly focus on addressing the overconfidence issue of deep neural networks for unknown classes, in image and text classification problems, respectively. Specifically, on graph structured data, OpenWGL [54] employs an uncertainty loss in the form of a graph reconstruction loss (27] unlabeled data. PGL Brl extends previous unsupervised domain adaption framework for the open-set scenario with graph neural networks. However, it focuses the unsupervised domain adaption and conditional shift problem. With the development of deep learning, several novel approaches have been proposed. SRNC [72) further presents a domain adaption framework for the two most common kinds of data shifts in graph structured data, i.e., open-set and closed-set data shift, by combining a graph adversarial clustering technique shift-robust node classification. MHGL [7T] extends traditional out-of-distribution (OOD) detection tasks to open-set scenarios, employing Pattern Distribution Estimator (PDE) and Hyerssee Graph Learning algorithm to model fine- mmne-graiedd patterns and identify previously unseen anomalous patterns. GOOD (22) proposes weakly supervised relevance feedback approach, named Open-WRF, mitigate sensitivity to thresholds. Pry [67] use the unknown proxies generated via mixup to efficiently anticipate the distribution of novel classes and achieve excellent prediction performance under the constraint of both cross entropy loss and complement entropy loss.
B.3 Energy-based Models
Energy-based modeling as well as its application is mostly concentrated on out-of--istiibution detection. Grathowohl al. [13 found that a standard discriminative classifier of p(y]zz) can be interpreted as an energy-based model (EBM) [40] for the joint distribution p(z, y). Junbo Zhjao, Michael Mathien al. introduces new model of Generative Adversarial Networks, called Energy- based Generative Adversarial Network (EBGAN) (68), which features the discriminator being viewed as an energy function, introducing a new theoretical perspective and method for Generative Adversarial Networks. Tuomas Haarnoja, Haoran Tang et al. apply EBMs to the field of reinforcement learning, and introduces a novel reinforcement learning method soft Q-learning [5] using deep energy-based polices for continuous states and actions, focusing on maximum entropy policies and improved task exploration. Yilun Du, Joshua Meier innovatively applies an energy-based model to score protein conformations at the atomic level, introducing machine learning methods into the
16

---

field of protein design. Ruiqi Gao. Yang Song et al. introduces method called Diffusion Recovery Likelihood [T] effectively learn and sample from Energy-Based Models (EBMs) by learning from the diffused versions of the data, overcoming the challenges of training and sampling EBMs on high-dimensional datasets. Marela Elias Ramzi et al. introduced the HEAT [29). which leverages the versatility of the EBM framework to provide strong OOD detection method. Qitian Wu, Yiting Chen et al. applied EBM to the task of OOD detection in graph domains and proposed GNNSAFE [56]. GNNSAFE is method for detecting anomilies in graph data, which can improve the robustness and generalization of GNN. Impressively, the use unknown classes samples information to improve EBM performance is discussed in GNNSAFE.
C Proofs for Proposition
C.1 Proof for Proposition 1
The gradient of | is given by
04,
(0; A)
C+1 Oh: A)
e Al(00iAAA
Σ
00
00
00
C+1 6 1+ (02
ED
OE (0t,, A, U1)
DE (I,, A,
Σ
00
CC41 e
1 (1:00)))))
IED
OE (x,, A, Ui ha)
OE (x,, A, c, ho)
= (1 (y = y/ X1, A))
Σ ply = c xi, A)
00
00
(15)
The gradient of l2 is given by
0l2
ah: 10, A)(c+1)
C+1
(0;
A)
Σ
+
Σ
00
00
00
Let e (0(000))))o)))
iED
j
0E (z,, A, 11)
C+1
DE (z,, A.j)
00
00
Σet1 e (0:2)1A))1))
iED
jf
DE (x,, A, C 1;h0)
OE (x,, A.j; ho)
(1 - (y C X5,
p' Xi, A)
00
00
(16)
where p' (y= = x,, A)
is an approximate probability distribution established
Di.u, 16
0-1
on the complementary classes
In the Eq. 15 and Eq. 16), We computed the gradient with respect to the parameters. The information presented the preceding equation indicates that the training process for Drs, which aims to minimize the first-order gradient of [ + A2lz, leads to reduction in the energy score. In broader context, the energy output generated by the trained model tends to decrease for any instance originating from the distribution Drr- Besides, for the unknown substitutes X we constructed, we will adjust their energy scores by the la to keep the energy model in an optimal state for fulfilling its function.
17

---

C.2 Proof for Proposition 2
We can demonstrate the proposition when considering M (k-1) rewrite the Eq. (10) from the perspective of a single vector:
Aus
Eg Ais < =,:t-1)
We need
E = ab!h-1) + +11 - Σ Aij
(N-1)
Asu
E
(17)
< <a 1
+ (1 Fo)
E, as (aa))(u11)
similar prove can be applied to establish the contrary outcome in the case
E. (k(-1))
2, Avs
F = OF. (((h-1)) + (1 - a) E, A, (k-1)
E, Ass
(18)
a (aE(N11) + (1 a)E(k-1)
E(k-1)
C.3 Proof for Proposition 3
At first, we define Ex:; hgg) the energy scores yielded by the model fig- minimizing the energy regularization loss 13. Then, hep and hag denotes the model yielding the optimal predictive softmax distribution that minimizing the entropy loss and the model yiclding the optimal predictive softmax distribution that minimizing the tailored complement entropy loss l2. i.e.
e Not (x,, 9z)(v
Blan p(v1=,9)1
(19)
EC+1 e Not xtac x7c]
e Not (x, (2)
orggitihcc--iusssshhccseseeeaee- pCC 1/
(20)
hat Eeety e l7d
At the same time, considering that Flxx,, G2; -log ICt1 co((x.))) Therefore start from this equation and we have
C+1
E(x, Gn, ho-) = E(x, Gni E(x, ((x,)92;ho)) -lagg)loggo log
Σ
h (o.00)))
c=1
-Ela,Grshor) + Ela.Gasthtt))
N (m.Sa)
-log(e
(21)
C+1
h (t
-log
The above equation implies an equivalence relationship between E(z, Gz, ha-) and E(x, Goi he-) E(x, Gui hg+)) - log EC+1 (m.e2) Meanwhile, continuing to consider that E(x, (2: ha: -log CC+1 (x.8x), we can further demonstrate that the predictive softmax probability distribu- tion obtained from E(x, Ga, h
18

---

Mot Mot Bla
plyla, 9x)
Eae e Not (aSSil - Ela B
Not (1n9)))))))
(22)
C+1 Mot (otlassldd)
Σ C+1 (0 c=
The above predictive softmax probability distribution exactly minimizes I defined by Eq (9). We thus have proven that the optimal energy that minimizes 13 intrinsically induce the predictive softmax distribution that minimizes l for Dur-
Similarly, when we consider the relationship between the l2 and l3, we can start again from E(x, Noi ho) = llg CC+1 cho(z.Ge)) and get
(xx, Gus ha-) -log Σ
oz (m. E(n. ((z.)sihh
(23)
The above equation implies an equivalence relationship between E(x, Gz. ha-) and E(x, Gzs ha-) E(r, 92; haz) - log EC+1 é Aof (w.02)) We thus can further demonstrate that the predictive softmax probability distribution with respect to class C obtained from Exx, Gu, he-) is
Not (c. Ela.S.iho E(z,
pCC + 11), = C+1 M (n-la
Bla.Ga c=l - Bl
Mot (2,90)0411
E(x. ) E(
C+1 e en
Fla,Baihor R4
N (nneal Egj e 0+1
(24)
0+1 Noz (n.
Moj (a.Saa)
Eosy e
e
Mo Ma (m.
M (0.G))0441
= (1 0++ Mog (1.
Eejn hog (2.(0))))1))
Considering that the energy scores are not computed for a specific class of labels, but rather are computed considering all classes, and the loss la tends to reduce the energy scores for x Dr-- Meanwhile, combine this with the fact that the loss la tends to drive higher probability of output for class C, it reasonable assume that the probability of this probability distribution is not overly large for class Then, The former term can be regarded as coefficient modifying the latter term. We thus have proven that the optimal energy that minimizes ly intrinsically induce the predictive softmax distribution that minimizes l2 certain extent for D
D Algorithm
The algorithm of EGone illustrated in Algorithm The given graph data are inputted into a GNN model (Line 4), followed by the generation of inter-class and external unknown substitutes. Then generated substitutes and the original training data are input into the rest of neural network and learn an open-set classifier (Lines 6-10). As new graphs are fed in for training (Line 6). the energy scores for the known classes samples and unknown class samples are calculated (Line 7). Under the constraint of the total loss, the model is trained and optimized.
19

---

Algorithm EGGon: open-set node classification
Require: G E, X): graph with links and features;
Du |G. train set with labeled nodes;
X U: test set where S are the known classes appeared in training and are the unknown
classes;
Ensure: f(X;, V). E fI...., C, unknown).
1: Obtain the inter-class node pairs { V1 ) (Zi, Uj) } Dir 8.1. Yi F y,&aay =
2: Obtain the peripheral nodes that are leaf nodes and low confident nodes.
3: while not convergence do
4: For the first m = 1,... layer:
hTTh Pd (01; hT ATT iJ E NG), Va, Drr
5: At the k-th layer:
Create unknown substitutes Sut using Eq. (8 & 9 Augment the substitutes to known class samples:
Dr, Dir (X sub C--1)
6: For the = k + 1,
kj layers:
h fm (01; h
mm .j E N.), Va, E De
7: For the k1 K1...., K. layers:
= (01; K m- hg
N.), Va, E De
E. = E ET j.j j NG), Va, E Dr
8:
For open-set classifier layer:
Obtain cross entropy loss as Eq. 11
Obtain complement entropy loss Eq. 12
Obtain energy regularization loss as Eq. 13)
9:
Back-propagate loss gradient using Eq. (14) and update weights
10:
if early stopping condition is satisfied then Terminate
12:
end if
end while
D.1 Complexity Analysis
Detailed training process of EGonc is illustrated in Algorithm 1. In terms of complexity, compared with normal closed-set node classification, our extra computation comes from three parts: 1) unknown substitute generation, which has linear complexity with the quantity of edges and the quantity of generated substitutes, i.e., O(EE) + [x 2) energy scores calculation, which has linear complexity with the quantity of the known class samples after augmentation, i.e., ODD,,) 3) substitute classifier training, the complexity of this part depends on the adopted backbone GNN and the quantity of generated substitutes. Assume O(.fol) be the time GNN for and classifier fo, takes compute single node embedding and make prediction, the complexity of substitute classifier training is O[Xxaabl ffo)) Overall, the extra complexity linear with the number of edges in the original graph and the number of generated substitutes. Experimental results on parameter analysis of generated substitutes quantity (see Fig. [ show that relatively small number of substitutes can achieve quite good performance. Thus, the extra complexity is reasonable for open-set node classification.
E Supplementary experiments
E.1 Baseline Details
GCN_sof soft, GCN_sig GCN 128 is adopted for graph learning. We constructed four baselines using GCN, where GCN_soff soft and GCN_sig use softmax and sigmoid layers, respectively, as the final layer for training and classification. Here adopt GCN with sigmod layer baselines since softmax is observed to overconfident to unknown class samples.
GCN _ssos_off, SCN_sig_rs Additionally, GCN_soft, _soft_r and GCN_sig perform classification based on these two model using fixed thresholds chosen in {0.1, 0.2,...,,..33
Openmax: Openmax 1 an open-set recognition model based on "activation vectors". iee. penultimate layer of the network.
20

---

By modeling the distance from "mean activation vector" using the extreme value distribution, softmax scores are calibrated for each class and updated Openmax scores, which are then used for open-set classification. GCN is further combined with Openmax for open-set node classification on graph data.
DOC: DOC [46] is an open-world classification method for text classification. It uses multiple 1-vs-rest of sigmoids the final output layer and defines an automatic threshold setting mechanism. GCN is combined with DOC to allow comparison with the proposed model. GCN is used to obtain the node representations and DOC is used to do open-set classification and rejection.
PROSER: PROSER (70) a novel and highly effective open-set recognition method used in the
field of image processing. We paired it with GCN for graph-based comparison.
OpenWGl:: OpenWGl. 5 employs uncertainty loss in the form of graph reconstruction loss
on unlabeled data and using an adaptive threshold to detect the unknown class samples.
GNNSAFE: GNNSAFE [56] proposes effective OOD discriminator based on energy function derived from graph neural networks (GNNs) trained with standard classification loss. We apply from the OOD detection domain to the open-set classification domain.
G² Pry Pry 667 uses the generated proxies generated via mixup, with the help of cross entropy loss and complement entropy loss, shows excellent performance in open-set classification
E.2 Datasets Stastics
Table 5: Statistics of the experimental datasets.
Dataset
Nodes
Edges
Features
Labels
Cora
2708
5429
1433
7
Citeseer
3312
4732
3703
DBLP
17716
105734
1639
4
PubMed
19717
44325
500
3
Ogbn_arxiv
169343
1166243
128
40
E.3
Implementation Details
The GCN is configured with two hidden GCN layers in the dimension of 512 and 128, followed by an additional multilayer perceptron layer of size 64. EGonc implemented with PyTorch and the networks are optimized using stochastic gradient descent with a learning rate of le -3 The balance parameters X1, X2 and À are chosen by grid search in the interval from 10-2 102 with step size of 10'.
The baseline methods evaluated according the instructions reported in the original papers with the same parameter configuration unless otherwise specified, and the best results are selected. For each experiment, the baselines and the proposed method were applied using the same training, validation, and testing datasets. The hyperparameters were tuned to get the best performance on the validation set. All the experiments were conducted on a workstation equipped with an Intel(R) Xeon(R) Gold 6226R CPU and an Nvidia A100.
E.4 Detailed performance on IND and 0OD classes for near open-set classification
We show the detailed classification accuracy in terms of known elasses (IND) classes) and unknown classes (OOD classes) respectively, in Table The experiment was conducted under the same setting of Table [ It shows that in order to gain the ability of unknown class detection, there is a slight decrease in the performance of known class classification, i.c. from 79.5% to 76.3% on average, comparing EGonc to closed-set classification method GCN soft. However, the unknown class detection accuracy is improved from 0% to 76. 76.1% 1% on average, which is remarkable. Compared to other open-set classification methods, such as the globally second best method, g Pry, the average performance of anknown-clane node detection is improved from 70.7% 7% to 76. (6.1% % while the performance of the known-class classification experienced a slight decrease from 78.6% to 76.3% on average. When compared with the globally third best method, PROSER, the average performance of unknown-class node detection is improved from 63.6% to 76. % while the performance of the known-class classification still gain certain increase from 73.0% 76.3% on average.
21

---

Table 6: Detailed classification accuracy in terms of known classes in--istributton, ind) and unknown classes (out-of-distribution ood) for near open-set classification on five datasets with one unknown
class (u 1) under inductive learning. Numbers reported all percentage (%).
Cora
Citeseer
DBLP
Pubmed
arxiv
Average
Methods
ind
ood
ind
ood
ind
ood
ind
0od
ind
ood
ind
ood
overall
GCN_son
89.7
00.0
71.8
00.0
92.9
00.0
93.2
00.0
50.1
00.0
79.5
00.0
51.5
GCN sig
87.8
00.0
73.0
00.0
92.5
00.0
93.0
00.0
49.1
00.0
79.1
00.0
51.1
GCN =
87.2
23.3
61.4
50.5
85.6
19.2
70.4
40.3
47.4
35.3
70.4
33.7
58.6
GCN_sig s
85.6
57.9
67.1
53.8
79.9
45.8
78.1
30.2
46.2
10.0
71.4
39.5
60.4
Openmax
86.6
30.0
60.4
49.4
85.4
27.6
73.4
38.1
45.7
19.3
70.3
32.9
58.5
DOC
84.0
54.9
67.7
63.2
79.8
48.1
78.8
30.7
46.8
32.3
71.4
45.8
61.2
PROSER
83.2
82.7
72.5
76.0
78.2
58.0
78.0
70.2
53.1
31.2
73.0
63.6
70.5
O
83.4 58.6
59.9 66.4
60.0 76.6
87.9
55.2
45.3
58.7
71.9
58.5
64.9
GNNSAFE
86.6 53.3
74.0
62.4
85.6
43.8
92.2
60.1
51.3
33.1
77.9
50.5
68.6
G2 Pxy
83.8 86.5
73.6
78.7
85.1
60.3
87.6
67.5
62.8
60.3
78.6
70.7
74.7
EGone
84.3 84.9
74.5
78.1
79.4
80.7
80.3
80.1
63.0
69.1
76.3
76.1
76.5
Table 7: Near open-set classification accuracy on three datasets with multiple unknown classes under the inductive learning setting. Numbers reported are all percentage (%).
Cora
Citeseer
Ogbn
arxiv
Methods
(u=2)
(u=3)
(u=2)
(u=3)
(u=5)
(u=10)
(u=15)
(u=20)
Acc
Fl
Acc
F1
Acc
Fl
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
FI
GCN_soff
48.6 50.3
34.2
42.6
28.0
31.6 14.8
18.1 48.7 15.0
47.7
17.6
42.4
17.4
29.5
13.7
GCN sig
49.5 50.7
34.8 41.7
27.3
32.1
14.8
17.7 49.4 12.2
45.4
9.4
42.3
9.7
26.6
9.3
GCN soft
69.9 68.3 64.3
61.7
51.3
61.1 39.2 44.3
15,0
17.8
49.0
17.3
56.2
14.1
GCN sig
71.6 68.9 65.2 53.2 63.7 48.9
57.6 44.9 46.4
48.2
9.9
47.4
7.0
61.0
9.0
Openmax
61.2 52.7
56.2 49.7 45.7
43.0 32.9 42.6 14.3
47.6
17.3
48.3
17.6
54.4
10.0
DOC
71.8 69.9
62.6 56.9 71.8 53.4 68.8 47.4 45.3 15.2
46.6
16.7
46.8
7.0
60.2
12.0
PROSER
80.6 80.9
75.8 72.4 73.7 63.4 68.0 53.7 60.4
27.1
53.2
22.6
54.6
30.1
53.3
19.2
OpenWGL
73.1 71.6
69.3 66.9
73.5 41.0 62.5 47.7 46.8 20.1
53.4
21.9
56.7
22.6
65.2
9.1
GNNSAFE
78.3 80.5
80.8 66.9
79.4 60.6 76.3 57.7 55.8 25.7
51.9
24.7
53.9
18.9 59.4
19.7
G2 Pry
82.2 83.4
80.2 68.9
77.3 65.3
79.4 55.5
68.9
46.1
55.1
26.5
63.3
30.4
65.3
23.9
EGone
83.3 83.9
81.3 70.9
77.6 65.6
80.0 55.7
69.5 46.7
55.2 26.8
63.8
30.3
65.5
27.3
E.5
Near open-set classification on multiple unknown classes
Further, we explore the performance in terms of multiple unknown classes. Table 7 we provide the results of inductive learning near open-set classification with multiple unknown classes, i.e., 2,3 for the Cora and Citeseer datasets and u=5,10,10 10,1, for ogbn arxiv dataset, respectively. It can be observed that EGone consistently performs best.
ogbn_arxiv
E.6 Near open-set classification under transductive learning setting
Table presents the accuracy and Fl scores of the methods applied to the near open-set classification task with one unknown class (the last class), under the transductive learning setting, i.e., the features of unknown class nodes are used to facilitate the model training or validation. It is observed that EGonc still consistently outperforms the baselines. Specifically, compared with the second-bes method i.e., G Pry, EGone achieves 3. 3.11% and 2.89% improvements average in terms of accuracy and F1, and achieves 9.94% and 13.6% improvements over the third-best method (PROSER) on average in terms of accuracy and F1, respectively. Furthermore, by comparing Table and Table it is observed that the involving of real unknown class information during training or validation benefits the model, which is reasonable.
E.7 Parameter learning quantity of substitutes.
The substitute unknown node generation module of EGone is crucial and the quantity of generated substitutes need to be predefined. We assess the performance of EGone with different numbers of
22

---

Table 8: Near open-set classification five citation network datasets with unknown class (u=1)
under in the transductive learning setting. Numbers reported are all percentage (%).
Cora
Citeseer
DBLP
Pubmed
Ogbn_arxin
Average
Methods
Acc
F1
Acc
F1
Acc
Acc
F1
Aсc
F1
Acc
F1
GCN_soff soft
70.8
68.2
44.7
38.9
62.9
57.0
29.2
29.7
50.2
18.4
51.6
42.4
GCN_sig
68.8
64.5
44.6
40.1
63.4
59.2
29.0
29.5
46.8
8.4
50.5
40.3
GCN CCN_soft_r
78.1
78.9
67.3
57.0
67.3
67.7
68.9
27.2
49.6
19.0
66.2
50.0
GCN GCN_sig__
78.3
78.5
65,4
55.3
71.4
71.5
69.0
27.2
45.9
7.7
66.0
48.0
Openmax
77.2
76.9
57.5
56.7
69.0
70.6
55.0
52.1
49.2
18.9
61.6
55.0
DOC
77.3
77.9
65.1
55.3
71.7
72.0
68.4
34.2
49.9
19.4
66.5
51.8
PROSER
84.7
83.6
74.3
66.6
75.3
71.6
72.8
60.8
55.0
30.7
72.4
62.7
OpenWGL
83.3
83.5
70.0
65.4
74.3
74.2
71.2
68.0
46.0
20.0
69.0
62.2
GNNSAFE
80.7
81.9
73.1
62.2
74.2
75.8
73.5
69.9
52.8
24.1
70.8
62.8
Pry
90.7
89.7
76.3
71.8
77.5
79.5
78.0
73.4
63.7
31.4
77.2
69.2
EGonc
91.2
90.4
77.2
72.9
79.4
80.7
86.5
80.5
63.8
31.6
79.6
71.2
0.9
0.8
0.7
Performance
Cora-Acc
0.6
Cora-F1
Citeseer-Acc
0.5
Citeseer-Fl
Dblp-Acc
Dblp-F1
0.4
Pubmed-Acc
Pubmed-F1 F1
0.3
Quantity of generated substitutes
Figure 1: Performance of EGonc under different quantity of substitutes.
generated substitutes, i.e., {0.1, 0.5, 1,2,5) 5} times the average number of nodes in each known class, the influence of the quantity of substitutes. Results are shown in Fig. I It is observed that both accuracy and macro-F1 are stable while the quantity of generated substitutes varies a lot. Thus, relatively small number of substitutes can be used for better efficiency.
E.8 Parameter learning weights of losses.
ablation study, we verify that the cross-entropy loss (Eq. (TT). tailored complement entropy loss (Eq. (12). and energy regularization loss (Eq. (13) all indispensable. Here further assess the performance of EGonc with various weights these three losses, i.e., A1 A2, and λ3. with the range of (0.01, 0.1, 1, 10. 100). The results shown in Fig. 2 We can that it is better to choose small values for the weights generally. And the wights for tailored complement entropy loss (A2) are relatively large compared to the weights of other losses (A1, Aa). For example, the best value for X2 for Cora, Citeseer, Dblp and 10 for Pubmed while the best values for x and Ag are either 0.01 0.1. It illustrates the importance of complement entropy loss in open-set node classification task.
23

---

Cora-Acc
Corai11
Gleseer-Act
ChesserF1
0.6
0.5
0.4
Dbip-Acc
0.2
Dbip FI
0.1
Pubmed-FF
0.01
0.0
for CIE pmax
(b)
Weight CrE praxy
Cora
Corsif1
0.7
Cleseer-Act
ChtsserFF1
0.6
reformance
0.4
0.3
Dblp-Acc
Dbip FI
0.1
Pubmed-FF
0.01
0.0
Weight tallered loss
(d)
Weight tallored CoE
Cora-Ace
0.8
Cara.51
0.7
0.7
Citeseen-cc
Citeseer-F3
0.6
0.5
0.4
04
0.3
0.2
Dblp-F1
0.1
Pubmed-Acc
Pubmed--1
0.0
0.02
(c)
Weight regulericat
Weight for regularication loss
Figure 2: Performance of EGonc with different value of weights for losses we adopted.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [TODOJYes, the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: (TODO)Yes, the paper discusses the limitations of the work performed by the authors the "Limitation and Conclusion" section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer: [Yes]
Justification:
[TODO]We provide complete and correct proofs in the appendix section
"Proofs for Propositions
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: [ThDO]The paper provides pseudocode, model details, and experimental details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
26

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: [TODO]We will provide access the data and code, along with sufficient instructions to reproduce the main experimental results when the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://mips.cc.
aublic/guudees/Cdeetuuii.. casssionoooyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wttps/cccppclcccuua.eeaeeecccuea
wtteee///oooSSeissss.ee.ee. www.ssssiicggcooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: (TODD]Yes, please refer to H.3 in the appendix and test settings in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: [TODO]No, the paper does not include error bars, confidence intervals, or statistical significance tests for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: [TODO]No, the article only provides the required computational resources, but does not provide detailed information such as memory, execution time, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:
[Yes]
Justification: YToDO)Yes, it does comply.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [TODO]The paper is foundational research, does not involve societal impact, and does not discuss societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
28

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [TODO]The article poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: [TODO]The article specifies the sources of the referenced assets much as possible.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
29

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: [TODO]We will strive convey these details using structured templates as much as possible, but due to time constraints, they may not be comprehensive.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [TODO]The article does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [ThDO]The article does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
30

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "sdSfvXb4rn", "reviewer": "Reviewer_qzuy", "summary": "In this paper, the authors proposes a new generative open set node classification method (EGonc) to address the challenge of Open Set Classification (OSC) for safely deploying machine learning models in an open world. Traditional softmax-based methods are overly confident on unseen data, making them vulnerable to out-of-distribution (OOD) samples. EGonc uses graph structure to generate substitute unknowns simulating real open set samples and employs Energy-Based Models (EBM) for density estimation. The method learns additional energy logits from feature residuals to indicate OOD-ness, ensuring a distinguishable margin between in-distribution (IND) and OOD detection scores. Experimental results demonstrate the superiority of EGonc.", "questions": "1. Why were these datasets chosen, and what advantages do they offer in validating the proposed method?\n2. Why was the 2017 GCN by Kipf and Welling chosen as the backbone neural network for experimental evaluation, instead of using other models?", "limitations": "See the weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Novel Approach: The paper introduces a new method (EGonc) for open set classification that leverages energy-based models, which is a significant departure from traditional softmax-based methods.\n2. Theoretical Guarantees: EGonc comes with strong theoretical properties that ensure a distinguishable margin between in-distribution and out-of-distribution samples.\n3. Robustness to OOD Variability: By simulating real open set samples and learning virtual OOD classes, the method enhances robustness against the diversity of OOD examples.", "weaknesses": "1. The motivation primarily focuses on the limitations of softmax-based neural networks without delving deeply into the broader implications and potential impact of improved open set classification across various domains and real-world applications.\n2. The advantages of the proposed method could be strengthened by providing more concrete examples of practical scenarios where current OSC methods fail and how EGonc specifically addresses these failures, making a stronger case for its real-world applicability and importance.\n3. While the method shows promise, its applicability and performance in domains outside of those tested in the experiments need further validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Review 1:**\n\n**Weakness 1:** The motivation primarily focuses on the limitations of softmax-based neural networks without delving deeply into the broader implications and potential impact of improved open set classification across various domains and real-world application.\n\n**Answer W1:** Thanks for your comment. A majority of neural network based classification models use softmax to achieve classification and that is why we mainly focus on softmax-based neural networks. As shown in experiments, in various fields and practical applications, our method can improve the adaptability and generalization of the models in terms of open-set classification scenarios, compared to softmax-based neural networks for closed-world classification, thereby broadening their application scope. In addition, in **Table 1** of the main text, we also include sigmoid-based neural networks (GCN\\_sig and GCN\\_sig\\_$\\tau$) as baselines.\nWe will add these analysis into the revision. \n\n**Weakness 2:** The advantages of the proposed method could be strengthened by providing more concrete examples of practical scenarios where current OSC methods fail and how EGonc specifically addresses these failures, making a stronger case for its real-world applicability and importance.\n\n**Answer W2:** Thank you for your suggestion. We will specifically present the bad cases of EGonc, $\\mathcal{G}^2pxy$ and GNNSAFE across five datasets in the revision, and we will visualise their open set node classification results in the Appendix, make the differences between the three models more intuitively.\n\n**Weakness 3 \\& Question 1:** While the method shows promise, its applicability and performance in domains outside of those tested in the experiments need further validation. Why were those datasets chosen, and what advantages do they offer in validating the proposed method?  \n\n**Answer W3 \\& Q1:** Thanks for your comment and your question. In the field of open-set node classification on graphs, these datasets are the most commonly used. For example, [1] conducted experiments using Cora, Citeseer, Dblp, and Pubmed; our experiments mainly followed their setup and additionally included the obgn\\_arxiv dataset to evaluate the model's performance on large-scale data. Specifically, these datasets have the following advantages: First, these datasets (Cora, Citeseer, Dblp, Pubmed, and Ogbn\\_arxiv) are widely used in the academic community, are publicly available and accessible, and have a high degree of representativeness and ubiquity, which allows for a fair and equitable measurement of model performance and enhances the transparency and credibility of research. Second, the datasets come from a variety of sources, including computer science (Citeseer, Dblp), medicine (Pubmed), and arxiv preprints (Ogbn\\_arxiv), which allows us to validate the applicability and robustness of the proposed methodology in different domains. Third, they have various dataset sizes, including small datasets (Cora, Citeseer), medium datasets (Dblp, Pubmed) and large datasets (Ogbn\\_arxiv), which allow us to evaluate the performance of the models with different dataset sizes. \n\n[1]: Man Wu, Shirui Pan, and Xingquan Zhu. Openwgl: open-world graph learning for unseen class node classification. Knownledge and Information Systems, pages 1-26 2021.\n\n**Question 2:** Why was the 2017 GCN by Kipf and Welling chosen as the backbone neural network for experimental evaluation, instead of using other models?\n\n**Answer 2:** Thanks for your question. Actually, the proposed method, EGonc, is agnostic to specific GNN architectures and demonstrates robust generalization capabilities. We conduct experiments on different GNN architectures, including GCN, GAT and GraphSage. The results are shown in **Table 8** in **Appendix E.10**, which have confirm the effectiveness and generalization ability of EGonc for open-set node classification.\nConsidering the space limitation, we only illustrate the experimental results in terms of GCN-based EGonc in the body of the article, since GCN is one of the most widely used and one of the most representative GNNs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i84XErZ4aH", "reviewer": "Reviewer_1Tms", "summary": "This paper focuses on energy-based open-set node classification, and adopted a generative method to obtain the explicit specific score of a node belonging to the ‘unknown‘ class. To generate good substitute unknowns, it adopted energy-based models to estimate the density of classes and guarantee the nice theoretical properties of the proposed method with an overall distinguishable margin between the detection scores from IND and OOD samples. Extensive experiments are conducted and show superior performance.", "questions": "See above.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1)\tThe paper is well-structured and overall easy to follow.\n\n(2)\tThe proposed method can obtain an explicit specific score of belonging to OOD data for each input, which provide more information and show the confidence of the model on its decision. \n\n(3)\tThe proposed method has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for IND and OOD samples, the adopted energy regularization loss has consistent effects with the cross-entropy loss as well as with the tailored complement entropy loss on the known classes, and that they are not mutually exclusive.\n\n(4)\tThe proposed method is agnostic to specific GNN architecture, which demonstrate its generalization for some content.", "weaknesses": "(1)\tIn the main part of the paper, some important experiment settings are missing, which makes the readers feel confused about the motivations of the experiments.\n\n(2)\tWhat are the differences between near open-set classification and far open-set classification? Why do they matter? If a method can achieve good far open-set classification, it should also be good at near open-set classification?\n\n(3)\tFor section 3, the model design is quite complex, the illustration should be more straightward and make each term and each symbol has enough explanation.\n\n(4)\tSome sentences should be improved to make them clearer, such as line 271-273: “Inspired by the Elastic Network Zou and Hastie [2005], Friedman et al. [2010], and considering the similarity in form and function between regularization terms and corresponding error terms…”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Review 2:**\n\n**Weakness 1:** In the main part of the paper, some important experiment settings are missing, which makes the readers feel confused about the motivations of the experiments.\n\n**Answer W1:** Thanks for your comment. A more detailed  experimental setting can be found in **subsection E.4** in **Appendix**. Due to space limitations, we have simplified and shortened this part in the main context. We will move the important details from Appendix in the revision. \n\n**Weakness 2:** What are the differences between near open-set classification and far open-set classification? why do they matter? If a method can achieve good far open-set classification, it should also be good at near open-set classification?\n\n**Answer W2:** Thanks for your question. Near open-set classification is to identify the fine-grained OOD samples while also classify the known class samples. Far open-set classification is to identify the coarse-grained OOD samples and also classify the known class samples. In near open-set classification, near ood data may be similar to some of the known classes, thus it requires the model to discern between these easily confusing  categories, and learn very clear and specific boundaries for known classes. In comparison, far ood data normally significantly differ  from known classes. It matters since recognizing categories that are significantly different from the training data is essential to ensuring model robustness. In some content,  we think near open-set classification is more challenging since it requires learning very precise class boundaries to distinguish between fine-grained differences in categories.\n\n**Weakness 3:** For section 3, the model design is quite complex, the illustration should be more straightward and make each term and each symbol has enough explanation.\n\n**Answer W3:** Thanks for your suggestion. We will improve the illustration of **Section 3** and make each term and symbol explained more clearly.\n\n**Weakness 4:** Some sentences should be improved to make them clearer, such as line 271-273:\"Inspired by the Elastic Network Zou and Hastie[3005], Friedman et al.[2010], and considering the similarity in form and function between regularization terms and corresponding error terms...\"\n\n**Answer W4:** Thanks for your suggestion. We will revise these sentences and improve the whole paper in terms of writing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kxwvDYskCa", "reviewer": "Reviewer_ena4", "summary": "This paper proposed a new energy-based generative open-set node classification method to achieve open-set graph learning. It uses energy-based models to estimate the underlying density of the seen classes and to evaluate the probability of a given input belonging to the IND classes or OOD classes. Besides, it mimics the distribution of real open-set classes by generating substitute unknown samples under the guidance of graph structure. Besides, the proposed method has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for IND and OOD samples.", "questions": "See Weaknesses part.", "limitations": "See Weaknesses part.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1)This paper studies a significant and interesting problem, and the method can be used in a wide range of real-world applications. \n2)The paper is overall well motivated. The proposed model is reasonable and sound. Theoretical analysis is performed. \n3)Comprehensive experiments, including near open-set classification, far open-set classification, ablation studies and parameter learning, are conducted. The results demonstrate good performance.", "weaknesses": "1) What is difference between open-set node classification and out-of-distribution detection problem? The authors should illustrate the differences and whether the proposed method can solve these two problems simultaneously.\n2) In section 3.1, it claims that the insouciantly selected unrelated data does not help the open-set classification, thus substitute unknown nodes near the class boundaries are generated. Why randomly selected data does not help? Aren’t they real-world open-set data which can help the open-set learning?\n3) It is a good idea to imitating data from open-set distribution by generating pseudo out-of-distribution samples, however the diversity of the generated data is essential. The method used in the paper is manifold mixup, does this method can ensure the diversity of the generated data?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Review 3:**\n\n**Weakness 1:** What is difference between open-set node classification and out-of-distribution detection problem? The authors should illustrate the differences and whether the proposed method can solve these two problems simultaneously.\n\n**Answer W1:** Thanks for your suggestion. We will take this into account and emphasise it in the next version. \n OOD detection is to identify the unknown samples that the model did not learn during the training, which is a binary-class classification problem, i.e., a test sample is an OOD sample or not. Open-set classification requires to identify unknown class samples and classify the known class samples, i.e. it is a multiple-class classification problem, which is more difficult than OOD detection problem. Thus, OOD detection methods cannot be directly applied to the open set classification scenarios. We will add these content in the revision. \n\n**Weakness 2:** In Section 3.1, it claims that the insouciantly selected unrelated data does not help the open-set classification, thus substitute unknown nodes near the class boundaries are generated. Why randomly selected data does not help? Aren't they real-world open-set data which can help the open-set learning?\n\n**Answer W2:**  Thanks for your question. Randomly selected, unrelated data typically does not contain feature information relevant to known classes, providing no boundary confidence to help the model establish classification boundaries. This effectively introduces noise into the open-set classification task, interfering with the model's learning process. In contrast, the unknown class nodes constructed near the class boundaries can provide rich boundary information for establishing classification boundaries, thereby helping the model to understand and recognise potential unknown class categories, improving the model's robustness and classification performance. Although these randomly selected unrelated data may belong to open-set data in the real world, they do not always contain information we need to support open-set learning. In particular, open-set learning relies more on samples that can provide boundary information to help the model determine decision boundaries and narrow the decision space.\n\n**Weakness 3:** It is a good idea to imitating data from open-set distribution by generating pseudo out-of-distribution samples, however the diversity of the generated data is essential. The method used in paper is manifold mixup, does this method can ensure the diversity of the generated data?\n\n**Answer W3:** Thanks for your question. Diversity of generated data is critical when creating pseudo-unknown class points. The construction of clear and unambiguous decision boundaries depends on the high quality and diversity of the generated data. Manifold Mixup is a method that performs data mixing in manifold space, aiming to generate new pseudo-samples by linearly mixing samples. While Manifold Mixup can increase data diversity, it also depends on the mixing strategy used and the characteristics of the data itself. In future work, we will consider the possibility of combining Manifold Mixup with other methods, such as data augmentation and adversarial sample generation, to further ensure the diversity of the generated data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QslCxN5YGC", "reviewer": "Reviewer_exuD", "summary": "This paper introduces a new method EGonc for open-set node classification in graphs. EGonc performs open-set classification by incorporating energy-based models into the graph learning diagram. The training of the model only requires in-distribution data, while out-of-distribution training data are generated synthetically. In the experiments, the authors compare EGonc against a wide range of baseline methods on several benchmarks and provide ablation studies.", "questions": "1. What's the intuition behind the unknown substitute generation formula (equations 8 and 9)? What is the intuition of including $l_2$ and $l_3$ into the learning objective? Can you use examples to explain them?\n2. In line 127, the second kind of substitute is generated from nodes with low classification confidence. Would that also be affected by the over-confident problem?\n3. In equation 14, how the $\\lambda$'s are selected?\n4. I suggest moving line 155-160 (the introduction of GNN) to the background section.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "1. The paper focuses on open-set node classification in graph learning, which is an important problem in real-world applications.\n2. Incorporating energy-based models into graph learning is a new method that has not been explored.\n3. The proposed method shows good performance, beating a wide range of baselines as is shown in the experiments.", "weaknesses": "1. An important claim made in the paper is not supported. The paper claims that the proposed method \"has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for IND and OOD samples\", but there is no theoretical analysis supporting this claim. \n\n2. The experiments are not sufficient. First, using one class as the out-of-distribution (OOD) class is very different from the real-world scenario where OOD examples can be quite diverse. Second, the paper does not compare against several important OOD detection baselines[1][2]. The proposed method has a negative impact on the in-distribution accuracy, while the methods in [1][2] will not. Third, the error bars are not reported. Fourth, an ablation study for energy propagation is missing, which seems to be an important component of the method.\n\n3. The presentation needs major improvement. First, there are a lot of format issues (e.g., missing reference in line 127, texts in the equations are not corrected formatted). Second, the intuitions behind $l_2$ and $l_3$ are not well-explained. It is not clear what are the roles of the two losses. It would be more helpful to use examples to show the intent of the two losses. Same for equation 8 and 9. Third, Proposition 1 uses vague descriptions, which is not valid for a mathematical statement. \n\n[1] Liu, Weitang, et al. \"Energy-based out-of-distribution detection.\" Advances in neural information processing systems 33 (2020): 21464-21475.\n\n[2] Sun, Yiyou, et al. \"Out-of-distribution detection with deep nearest neighbors.\" International Conference on Machine Learning. PMLR, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1:** An important claim made …\n\n**A-W1:** We support the claim through three propositions. The defined propositions can be found in Section 3, and the proofs can be found in Appendix C. In proposition 1, we proved loss $l_1$ and $l_2$ would result in lower energy scores for the model's outputs on training data, i.e. IND data, by taking the partial derivatives of the model parameters $\\theta$ with respect to these losses. In Proposition 2, we prove that the overall energy scores of known class samples decrease during the energy propagation, while the overall energy scores of substitute nodes increase, implicitly enlarging the decision boundary between IND and OOD samples. In proposition 3, we prove that loss $l_3$, i.e. the energy regularization loss, reduces the energy scores for IND data and increase the energy scores of OOD data, having the same effect as $l_1$ and $l_2$，and leading to a decrease in the energy scores of IND data. With these three propositions, the proposed method can guarantee an overall distinguishable margin between the detection scores for IND and OOD samples. Thank you. We will add more analysis and explanation in the revision, to improve the clarity.\n\n**W2:** The experiments…\n\n**AW2-1:**  We have conducted comprehensive experiments, including near open-set classification with one unknown class (Table 1 and 7) or multiple unknown classes (Table 6), and far open-set classification with multiple unknown classes (Table 3). We also discussed the experiments of different settings in terms of inductive learning (Table 1 and 6) and transductive learning (Table 7). In addition, we also conducted ablation study (Table 2), parameter analysis (Figure 1\\&2), and backbone architectures (Table 8).\n\n **AW2-2:**  For the papers given here ([1][2]), the main task of these articles is OOD detection, which is a different task with open-set node classification. The goal of OOD detection is to identify the unknown class samples, which is a binary-class classification problem, i.e., a test sample is an OOD sample or not. However, open-set classification requires to identify unknown class samples and classify the known class samples. It is a multiple-class classification problem, which is more difficult then OOD detection. Thus, OOD detection methods cannot be directly applied to open set classification. And this is the main reason we did not include OOD detection methods as baselines. In addition, the proposed method does not reduce IND accuracy much. As shown in Table 5, the average performance across five datasets shows that, compared to the closed-world method GCN, the IND classification accuracy of our method only decreases 3.2\\% ( from 79.5\\% to 76.6\\%), while the accuracy in OOD has been improved from 0\\% to 76.1\\%. \n\n**AW2-3**: Thanks. Many classic articles [3][4] in this task do not include error bars, and we followed their format. But we agree that error bars can provide more detailed information, and we will add them in the revision.  \n    \n[3] M. Wu, et al. Openwgl: open-world graph learning for unseen class node classification. Knownledge and Information Systems, pages 1-26 2021.\n[4] L. Shu, et al. Doc:Deep open classification of text document. Empirical Methods in Natural Language Processing, pages 2911-2916, 2017.\n   \n**AW2-4** : Thanks. We have conducted this ablation study, as shown in Table 2, i.e. the first two versions without loss $l_3$. Comparing the results of the $l_1$+$l_2$ and $l_1$+$l_2$+$l_3$, we can see that the energy propagation module contributes to the improvement of the model performance, which demonstrates its significance of as an important component.\n\n**W3:** The presentation…\n\n**AW3-1:**  We will address the format issues, and carefully proofread the whole article to improve the overall presentation.\n\n**AW3-2:** The intuition of $l_2$ is related to data generation process. It ensures that the known class samples are sufficiently close to their own class center in the output space.  \n\nThe intuition of $l_3$ is to control the energy values of known class samples being low, while keeping the energy values of substitute nodes being high, thereby allowing the energy model to remain in an optimal state where it can function effectively. Thus, the role of the loss function $l_3$ is to adjust the energy scores and keep the energy model in an optimal state.\n\nThe intuition of Eq. 8 is to generate substitute node at the boundary between two different known classes. For example, we can draw a line between a node from class A and a node from class B, then take a node near the center of this line as constructed inter-class unknown substitute. \n\nThe intuition of Eq. 9 is to generate substitute nodes located at the periphery of known classes. The goal is to enhance the model’s ability to recognize unknown classes located at the periphery. We achieve this by selecting class centers of known classes and peripheral nodes, with semantic and structural criteria, then mapping them to generate substitute nodes located at the peripheral of known classes. \n    \n**AW3-3:** We will revise it and make it more concise and consistent with mathematical statement. Thanks,\n\n**Q1:** What's the intuition behind…\n\n**A1:** Please refer to **AW3-2** .\n\n**Q2:** In line 127, the second kind of substitute…\n\n**A2:** Thanks. Overconfidence has very slight effect on the generation of our second kind of substitute nodes, since we only need a part of unconfident nodes to help the generation, and we do not require all the unconfident nodes. Thus, there is redundant space to allow the existence of nodes which should be unconfident but are misclassified with overconfidence. \n\n**Q3:** In equation 14…\n\n**A3:** We used a grid search to determine the values of these two hyperparameters. The results are given in E.9 in the Appendix.\n\n**Q4:** line 155-160…\n\n**A4:** We will revise it. Thanks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new energy-based open-set node classification method for open-set learning tasks. Initially, the paper received borderline scores. During the rebuttal phase, the authors successfully addressed most concerns, leading to a consensus to accept the paper.

Please follow the reviewers' suggestions to integrate the rebuttal content into the original paper.
Decision: Accept (poster)