Title: EGonc : Energy-based Open-Set Node Classification with substitute Unknowns
Venue: NeurIPS 2024
Accepted: yes
Keywords: Open-set classification

Energy-based models

Graph learning
Abstract: Open-set Classification (OSC) is a critical requirement for safely deploying machine learning models in the open world, which aims to classify samples from known classes and reject samples from out-of-distribution (OOD). 
Existing methods exploit the feature space of trained network and attempt at estimating the uncertainty in the predictions.
However, softmax-based neural networks are found to be overly confident in their predictions even on data they have never seen before and
the immense diversity of the OOD examples also makes such methods fragile.
To this end, we follow the idea of estimating the underlying density of the training data to decide whether a given input is close to the in-distribution (IND) data and adopt Energy-based models (EBMs) as density estimators. 
A novel energy-based generative open-set node classification method, \textit{EGonc}, is proposed to achieve open-set graph learning. 
Specifically, we generate substitute unknowns to mimic the distribution of real open-set samples firstly, based on the information of graph structures. 
Then, an additional energy logit representing the virtual OOD class is learned from the residual of the feature against the principal space, and matched with the original logits by a constant scaling. This virtual logit serves as the indicator of OOD-ness. 
EGonc has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for IND and OOD samples. 
Comprehensive experimental evaluations of EGonc also demonstrate its superiority.

EGonc : Energy-based Open-Set Node Classification with substitute Unknowns

Abstract

Open-set Classification (OSC) is a critical requirement for safely deploying machine learning models in the open world, which aims to classify samples from known classes and reject samples from out-of-distribution (OOD). Existing methods exploit the feature space of trained network and attempt at estimating the uncertainty in the predictions. However, softmax-based neural networks are found to be overly confident in their predictions even on data they have never seen before and the immense diversity of the OOD examples also makes such methods fragile. To this end, we follow the idea of estimating the underlying density of the training data to decide whether a given input is close to the in-distribution (IND) data and adopt Energy-based models (EBMs) as density estimators. A novel energy-based generative open-set node classification method, EGonc , is proposed to achieve open-set graph learning. Specifically, we generate substitute unknowns to mimic the distribution of real open-set samples firstly, based on the information of graph structures. Then, an additional energy logit representing the virtual OOD class is learned from the residual of the feature against the principal space, and matched with the original logits by a constant scaling. This virtual logit serves as the indicator of OOD-ness. EGonc has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for IND and OOD samples. Comprehensive experimental evaluations of EGonc also demonstrate its superiority.

1 Introduction

Learning on graphs, where instance nodes are inter-connected, has become one of the central problems for deep learning, as relational structures are pervasive and induce data inter-dependence which hinders trivial adaptation of existing approaches that assume inputs to be i.i.d. sampled [56]. However, current models mostly focus on improving testing performance of in-distribution (IND) data and largely ignore the potential risk w.r.t. out-of-distribution (OOD) testing samples that may cause negative outcome if the prediction is overconfident on them [31].

Real-world applications often require machine learning systems to interact with an open world, violating the common assumption that testing and training distributions are identical. This urges the community to devote increasing efforts on how to enhance models' generalization [9] and reliability [47] w.r.t. out-of-distribution data, by classifying samples from known classes and rejecting samples from out-of-distribution.

Existing methods in addressing this task normally are achieved by assuming that a model is more confident in its predictions for closed-set samples than for open-set samples [55, 8]. Based on this

assumption, a threshold is applied to a model's confidence score to separate unknown samples from known ones. Despite its apparent intuitiveness [20], this method often fails because deep neural networks tend to be over-confident in their predictions, assigning high confidence scores even to unknown inputs [19, 14, 1, 42, 71]. Furthermore, determining an optimal threshold for distinguishing known from unknown classes is a challenging and time-consuming task [38, 12, 22]. Others exploit the feature space of trained network and attempt at estimating the density of in-distribution (IND) features to address OOD detection [29], such as Gaussian Mixture Models [31, 44], nearest neighbors distribution [49] and energy logits [34, 8]. However, the immense diversity of the OOD examples makes such methods fragile [52]. For example, GMMs' density explicitly decreases when moving away from training data, making them effective for far-OOD detection, while energy logits benefits from the classifier training to obtain strong results on near-OOD samples [29, 52].

To overcome these limitations, researchers try to make advantages of unrelated samples in the openset environment [21]. Nevertheless, the methods using real open-set samples to help the training of models often require a substantial amount of data [23, 48, 5], and insouciantly selected data does not really help [66]. Well-designed outlier samples are difficult to obtain and require experts for identification and labeling [6, 25].

In this paper, we follow the idea of estimating the underlying density of the training data to decide whether a given input is close to the IND data and adopt Energy-based models (EBMs) as density estimators. To further improve the OOD detection performance, we try to generate substitute unknown samples carefully under the guidance of graph structure, to mimic the distribution of real open-set classes. Specifically, a novel energy-based generative open-set node classification method, EGonc , is proposed to achieve open-set graph learning. Based on the generated substitute unknowns, an additional energy logit representing the virtual OOD class is learned from the residual of the feature against the principal space, and matched with the original logits by a constant scaling. This virtual logit serves as the indicator of OOD-ness. EGonc has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for IND and OOD samples. Comprehensive experimental evaluations of EGonc also demonstrate its superiority. To sum up, the contributions of this paper are summarized as follows:

A novel method, EGonc, for open-set node classification is proposed by redefining the open-world graph learning paradigm based on the energy model and elaborate unknown-substitute generation;

EGonc has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for IND and OOD samples. The adopted energy regularization loss has consistent effects with the cross-entropy loss as well as with the tailored complement entropy loss on the known classes, and that they are not mutually exclusive;

No open-set data (samples of unknown classes or any side information of unknown classes) is required during training and validation. EGonc has an explicit classifier for unknowns and does not require threshold tuning;

EGonc is agnostic to specific GNN architecture and demonstrates robust generalization capabilities.

Experiments conducted on benchmark graph datasets illustrate the commendable performance achieved by EGonc.

2 Background

2.1 Open-set Node Classification

̸

This paper focuses on the node classification problem for a graph. Consider a graph denoted as G = ( V, E, X ) , where V = { v i | i = 1 , . . . , N } is a set of N nodes in the graph. E = { e i,j | i, j = 1 , . . . , N. i = j } is a set of edges between pairs of nodes v i and v j . X ∈ R N × d denotes the feature matrix of nodes, and d is the dimension of node features. x i ∈ X indicates the feature vector associated with each node v i . The topological structure of G is represented as an adjacency matrix A ∈ R N × N , where A i,j = 1 if the nodes v i and v j are connected, i.e. , ∃ e i,j ∈ E , otherwise A i,j = 0 . Y ∈ R N × C is the label matrix of G , where C is the already-known node classes. If node v i ∈ V is associated with a label c , y i,c = 1 , otherwise, y i,c = 0 .

For a typical closed-set node classification problem, a GNN encoder f θ g takes node features X and adjacency matrix A as input, aggregates the neighborhood information and outputs representations. Then, a classifier f θ c is used to classify the nodes into C already-known classes. The GNN encoder

and the classifier are optimized to minimize the expected risk [63] in Eq. (1), with the assumption that test data D te and train data D tr share the same feature space and label space, i.e. ,

̸

where H is the hypothesis space, I ( · ) is the indicator function which outputs 1 if the expression holds and 0 otherwise. Generally, it can be optimized with cross-entropy to discriminate between known classes.

In open-set node classification problem , given a graph G = ( V, E, X ) , D tr = ( X,Y ) denotes the train nodes. D te = ( X te , Y te ) is the test nodes, where X te = S ∪ U , Y te = { 1 , . . . , C, C +1 , . . . } . S is the set of nodes that belong to seen classes that already appeared in D tr and U is the set of nodes that do not belong to any seen class ( i.e. , unknown class nodes). Open-set node classification aims to learn a ( C +1) -class classifier f θ c that f ( θ g , θ c ; X te , A ) : { X te , A } → Y , Y = { 1 , . . . , C, unknown } , with the minimization of the expected risk [63]:

̸

where A is the adjacency matrix for X te. The predicted class unknown ∈ Y contains a group of novel categories, which may contain more than one class. Thus, the overall risk aims to classify known classes while also detecting the samples from unseen categories as class unknown .

It is worth mentioning that open-set node classification problem is different with out-of-distribution (OOD) detection problem . The objective of OOD detection is to identify a new sample whether from unknown classes or not, which is a binary classification challenge. Thereby, OOD detection techniques are not directly applicable to open-set node classification scenarios. This emphasizes the crucial importance of conducting research on open-set node classification, highlighting the distinct requirements and challenges within this domain.

From close-set to open-set classifier , an intuitive way is thresholding [20]. Taking the max output probability as confidence score, i.e., conf = max c =1 ,...,C f c ( x, A ) , it assumes the model is more confident in closed-set instances than open-set ones. Then we can extend a closed-set classifier by

where τ is a threshold. However, due to the overconfidence phenomena of deep neural networks, the output confidence of known and unknown is both high [1]. As a result, tuning a threshold that well separates known from unknown is hard and time-consuming.

Differently, the approach presented in this paper redefines the paradigm of open-set graph learning from the perspective of energy models. well-designed generated unknown substitutes are introduced to aid in training of the energy model performance and elaborate loss function to improve classification performance. There are advantages of adding a class instead of optimizing a threshold as in previous methods [46, 54, 1], since it does not require real open-set samples in validation and also eliminates the difficulty of tuning the threshold.

2.2 Energy-based Models

Energy-based models [30] use the energy-function E θ which defines a density over the data x as p θ = exp ( -E θ ( x )) Z ( θ ) , where θ are learnable parameters and Z ( θ ) = ∫ exp ( -E θ ) dx is the normalizing constant of the EBM. Z θ ensures that the induced density function Equation integrates to 1. However, Z θ is often hard to compute or even approximate since it is a high-dimensional integral. On the positive side, E θ can be any function E θ : R D → R placing no restrictions on the transformations compared to Normalizing Flows [68].

We can directly use the energy for OOD detection. That is, since a data point having high probability is equivalent to having low energy as p θ ( x ) ∝ -E θ ( x ) . This means we are not required to estimate the normalizing constant Z ( θ ) in practice by considering a decision threshold τ and binary classifier G for OOD data as

where 1 corresponds to IND samples and 0 to OOD samples.

We can additionally employ EBMs to C -category classification problem as discriminative EBMs [13]. Suppose f θ : R D → R C is a classifier assigning logits for C classes for a data point x ∈ R D . The logits can be interpreted as unnormalized probabilities of the joint distribution p θ ( x, y ) = exp ( f ( x )[ y ]) Z ( θ ) , yielding the marginal distribution over x as p θ ( x ) = ∑ y p θ ( x, y ) . For training, the factorization can be used, i.e. , log p θ ( x, y ) = log p θ ( x ) + log p θ ( y | x ) , and a standard Cross Entropy loss [2] is usually employed to optimize p θ ( y | x ) .

To sum up, the connection between Energy-Based Model and a discriminative neural classifier is established by defining the energy function as the negation of the predicted logit value: E ( x , y ) = -h ( x ) [ y ] . Moreover, the energy function E ( x ) can be computed for any given input

Nevertheless, to the best of our knowledge, existing research on energy-based modeling as well as its application is mostly concentrated on OOD detection, where the primary emphasis is on i.i.d. inputs while the power of modeling inter-dependent data has remained further exploration. Besides, there have been relatively few attempts to apply energy-based models in open-world scenarios, and this is an endeavor we will also pursue.

3 Methodology

We propose a unified open-set node classification framework based on energy scores and generated substitute unknowns. The energy scores mitigate a critical problem of softmax confidence with arbitrarily high values for OOD examples [18] and the differences of energies between in-distribution and out-of-distribution allow effective differentiation.

Specifically, a GNN encoder f θ 1 takes node features X and adjacency matrix A as input, aggregates the neighborhood information and outputs representation h k i for each node v i in its k -th layer, k = 1 , . . . , K , and K is the total number of layers. Thus, in the k -th layer, for node v i , the GNN encoder aggregates neighbor information from the k -1 layer into a neighborhood representation:

where h k i ∈ R d k is the hidden representation at the k -th layer, N i is the neighborhood of node v i and h 0 = X .

Meanwhile, we mimic novel patterns by generating substitute unknown nodes through manifold mixup [51, 17]. Targeted node pairs are selected and mixed up at the middle layer of the GNN, and two kinds of substitutes are created: inter-class unknown substitutes that separate known classes from each other and external unknown substitutes that separate known from unknown. Generated substitutes and original known class nodes are input into the remaining layers together to learn discriminative representations.

Then, a classifier f θ 2 is learned to classify the nodes into C +1 classes according to their energy scores. The energy scores of node v i is obtained through energy propagation upon the graph topology, i.e. ,

where E 0 i = -log ∑ y ′ e f θ 2 ( h K i )[ y ′ ] is the initial energy score of v i .

To conclude, the proposed EGonc method consists of three main modules: 1) a substitute node generation module to create substitute unknown class nodes; 2) a energy propagation module to obtain energy score of each node; and 3) an open-set classifier learning module to guarantee the classification of known classes and the rejection of the unknown class. We introduce them in details.

3.1 Substitute Unknown Generation

As mentioned previously, insouciantly selected unrelated data does not really help [66] the open-set classification. Thus, we try to generate substitute unknown nodes strategically positioned at class

boundaries. These nodes play a crucial role in the learning process by providing supervision and distinguishing between known and unknown nodes. Specifically, we generate two types of substitutes, inter-class unknown substitutes and external unknown substitutes, through manifold mixup [51, 17].

̸

For a well-trained classifier, nodes with common categories tend to have similar characteristics, while those from different classes often have distinct ones. Ideally, well-defined boundaries separate the different classes. Considering that nodes close to these boundaries may have less representative features for their own classes, we chose to generate substitute unknowns using nodes close to these boundaries regions. As a result, samples belonging to different classes, but located in close proximity, become optimal candidates for substitute generation. i.e. , pairs of nodes in distinct classes connected by an edge. To be specifical, given two connected nodes from different categories, denoted as { ( x i , y i ) , ( x j , y j ) } , where y i = y j and a ij = 1 , their embeddings in the k -th layer are h k i and h k j , derived by inputting the graph into the network f 1 ∼ k θ 1 . The inter-class unknown substitute (˜ x i , ˜ y i ) is obtained as follows:

Where α ∈ [0 , 1] is randomly sampled from a Beta distribution, typically centred around 0 . 5 . ˜ y i = C +1 denotes the categories index of ˜ x i , indicating it belongs to the unknown class. Two edges between ( x i , ˜ x i ) and ( x j , ˜ x i ) are also introduced into the graph. We refer to the set of generated inter-class samples as X int. The purpose of X int is to simulate the distribution of unknown classes nodes existing between known classes. The mixed hidden representation ˜ x i is then passed to f k +1 ∼K θ 1 to obtain the final representation ˜ h K i = f K θ 1 (˜ x i ) .

In addition to the inter-class unknown substitutes, we also generate external unknown substitutes to reflect the unknown distributions at the periphery. We use peripheral nodes of each known category along with their respective class center to generate these external unknown substitutes. The first kind of peripheral nodes are leaf nodes that belong to known classes, denoted as x i ∈ X tr , s.t. ∑ j A i,j = 1 . The second kind are nodes with low classification confidence, namely those nodes with the top T least confident scores within each known classes. In this way, it allows us to identify both structuredbased and semantic-based peripheral nodes. We denote the set of peripheral nodes as X per. Then, we obtain the embedding of class centers in the k -th layer based on the ground-truth annotations, i.e. , h k ( c ) = 1 | X c | ∑ x i ∈ X c h k i ( θ 1 ; x i , A ) , c = 1 , . . . , C, where X c represents the nodes labeled with the class c . And we can use the manifold mixup on peripheral nodes and their respective inverted class centers { ( x i , y i ) , ( x ( y i ) , y i ) , x i ∈ X per , y i ∈ { 1 , . . . , C }} to derive the external unknown substitutes in the k -th layer, i.e.,

where β > 0 and γ > 0 represent two hyperparameters used to adjust the distance and relative position between the generated samples and the existing known class samples. Furthermore, edges connecting ( x i , ˜ x i ) , x i ∈ X per are also included in the graph. We denote the set of generated external unknown substitutes as X ext and define unknown substitutes as X sub = X int ∪ X ext .

3.2 Energy Propagation

To further capture the diverse distribution of the unknown classes. we introduced an energy-based model. Through formula E ( x , y ) = -h ( x ) [ y ] , we can establish a bridge between the energy function and an open-set classifier.

Proposition 1. The energy score (Eq. (5)) can be an powerful indicator for OOD detection due to its valuable trait: the yielded energy scores for in-distribution data typically exhibit a lower tendency than those of OOD data.

As an energy model on graphs, in order to fully leverage the topological structure information of the graph, we apply energy propagation to the topological structure of the graph.

Here, ˆ A = D -1 / 2 AD -1 / 2 is the symmetrically normalized adjacency matrix of A , and D represents the degree matrix, E ( k ) = [ E ( k ) i ] , x i ∈ D tr ∪ X sub. With 0 < ζ < 1 as parameter, determine the

energy weights for individual nodes with respect to themselves and other connected nodes. The motivation behind topological energy propagation is to incorporate the underlying physical processes involved in data generation by exploiting local interactions between instances. Since similar nodes are often adjacent in the graph, the model can learn relationships between these nodes through energy propagation. This helps to better capture local structures and similarities within the graph.

Proposition 2. For a node x i , if its energy score E ( k -1) i is greater (resp. less) than the average of the energy scores of its one-hop neighbors at the current layer, i.e. , ∑ j ˆ A ij E ( k -1) j ∑ j ˆ A ij , then the updated energy score of its own yields E ( k ) i < E ( k -1) i , (resp, E ( k ) i > E ( k -1) i ).

Due to structural homogeneity [60, 28], known classes samples and unknown substitutes often connect within their respective distributions, indicating a correlation. Therefore, by using this energy propagation formula, it could ensure the average energy scores of the known classes samples decrease while the average energy scores of the unknown substitutes increase.

3.3 Open-set Classifier Learning

To address the diverse compositions of known and unknown categories, it is imperative to extract invariant information from both known and unknown classes. Consequently, the backbone GNN network f θ 1 is shared across the C +1 classes, facilitating the learning of class distribution and node representations.

To calibrate the closed-set (known classes) classifier to an open-set classifier, an additional category is introduced in the final classification layer to handle unknown predictions. Suppose the weights of the closed-set classifier are denoted as w close ∈ R d K × C where d K represents the dimension of the embeddings in the final layer of the GNN, the open-set classifier is formed by combining the closed-set classifier and the substitute classifier, i.e. , θ 2 = [ w close , w sub ] ∈ R d K × ( C +1) , where w sub ∈ R d K × 1 represents the weights associated with the substitute classifier. For simplicity, the bias term is omitted in this context. Then, the integrated classifier is trained using both the original known classes samples and the generated substitute samples, i.e. , D tr = D tr ∪ ( X sub , Y C+1 ) , X sub = X int ∪ X ext , using the cross entropy loss, i.e. ,

where ˆ y i = s ( f θ 2 ( f θ 1 ( x i ))) is the output vector of the open-set classifier for the input node x i through softmax s ( · ) . l CrE (ˆ y i , y i ) = -y i log ˆ y i is the cross entropy loss.

l 1 loss primarily uses information from the ground-truth class to maximize the likelihood, it tends to overlook the influence of complementary classes ( i.e. , incorrect classes) [3]. Therefore, we incorporate the concept of complement entropy [3] to complement the softmax cross entropy, to neutralize the effects of complementary classes.

Specifically, for the generated unknown substitutes, we minimize their inherent complementary loss, i.e. , minimize the average of entropy over the C known classes, considering that the generated substitutes may be adjacent to any of the known classes. Conversely, for nodes belong to known classes, they are predominantly adjacent to the generated substitutes, i.e. , the C + 1 class, we encourage the substitute classifier to assign the second-largest probability to these nodes. In light of these considerations, we introduce a tailored complement entropy loss for EGonc as follows:

̸

where ˆ y i \ y i denotes the prediction logits after excluding the probability of its corresponding groundtruth label. l CoE = -∑ C +1 c =1 ,c = y i ˆ y i,c 1 -ˆ y i,y i log ˆ y i,c 1 -ˆ y i,y i , ∀ ( x i , y i ) ∈ D tr . The first term of l 2 aims to minimize the misclassification of known class nodes into other known classes. By employing l 2 loss, the open-world classifier f θ 2 can accurately classify node belonging to known classes into their respective known class, while leveraging the substitute classifier to establish an appropriate boundary for distinguishing between the known and the unknown.

To mitigate the impact of energy attenuation on unknown substitutes during the energy propagation, we further introduce a strict constraints via energy regularization l 3 to the model. Drawing inspiration from the Elastic Network [73, 10], we observe a parallel in structure and purpose between regularization terms and their associated error terms. In light of this observation, we adopt a strategy that incorporates both linear loss and quadratic loss terms to penalize errors effectively. The linear error terms facilitate the generation of sparse solutions, thereby enhancing the model's robustness, while the quadratic error terms are adept at accommodating outliers and achieving a better fit to the data. By integrating these two kinds of error terms, we aim to enhance the adaptability and generalization capacity of the model.

where σ ( · ) is the LeakyReLU function. k 1 , k 2 are the weights for linear and quadratic error terms, respectively. t in , t out are hyperparameters. E ind ( x i ) = E ( x i , A ; h θ ) -t in , ∀ x i ∈ D tr and E ood ( x j ) = t out -E ( x j , A ; h θ ) , ∀ x j ∈ X sub represent the energy error terms correspond to the IND data and the OOD data, respectively. Here, E ( x i , A ; h θ ) represents the energy value of sample x i at the final layer. In this way, the energy scores learned with the constraint of l 3 are of benefit to open-set classification, and it also mitigate the excessive attenuation of energy scores for unknown substitutes caused by energy propagation.

Proposition 3. For any energy regularization l 3 that ensures that the GNN model h θ ∗ minimizing l 3 , where t in < t out are two margin parameters, then the corresponding softmax categorical distribution p ( y | x, G x ; h θ ∗ ) also minimizes l 1 , while p ( C +1 | x, G x ; h θ ∗ ) similarly minimizes l 2 . This means l 3 is optimized in the same direction as l 1 and l 2 for ind data.

Finally, the total loss of EGonc is a combination of cross entropy, the tailored complement entropy loss, and energy regularization loss, i.e. ,

where λ 2 > 0 and λ 3 > 0 are the hyperparameters to balance the losses. The algorithm of EGonc and its complexity analysis are provided in Appendix D.

4 Experiments

Experiments were carried out to validate the performance of EGonc. These experiments include: open-set node classification comparison , ablation study , parameter study , and generalization analysis . Code are available at https://github.com/hiromisyo/EGonc .

Datasets. Experiments to evaluate the performance for open-set node classification were mainly performed on five benchmark graph datasets [54, 72], namely Cora 2 , Citeseer 3 , DBLP 4 , PubMed 5 , and Ogbn_arxiv 6 [24, 45], which are widely used citation network datasets. Statistics are presented in Appendix E.2.

Metrics. Accuracy and Macro-F1 are used for performance evaluation.

Implementation Details. Generally, EGonc adopt GCN [28] as the backbone neural network for experimental evaluation unless otherwise specified. Detailed model parameters and platform information are given in Appendix E.3.

Test settings. Two kinds of open-set classification evaluations were conducted to consider short or long distances between known classes and unknown classes, i.e. , near open-set classification and far open-set classification. Specifically, in the near open-set classification experiment, for each dataset, the data of several classes were held out as the unknown classes for testing and the remaining classes

were considered as the known classes. 70% of the known class nodes were sampled for training, 10% for validation and 20% for testing. In the far open-set classification experiment, nodes from other datasets were used as unknown class samples for testing, other than the nodes from the dataset used in training.

Besides, a comparison is also provided for different settings in terms of the availability of side information on unknown classes during training or validation, known as the inductive learning setting and the transductive learning setting. In experiments with inductive learning setting, there is not any information about the real unknown class (such as the features x i or side information of unknown classes) used during training or evaluation, while under the transductive learning setting, the whole graph (including sampled known class nodes and unlabeled unknown class nodes) are input during model training or validation.

Baselines. We compare EGonc with ten baselines, which can be classified to four categories.

1) Closed-set classification methods: GCN_soft and GCN_sig . They are GCNs [28] with a softmax layer or a multiple 1-vs-rest of sigmoids layer as output layer.

2) Open-set classification methods with thresholds: GCN_soft_ τ , GCN_sig_ τ , Openmax [1], DOC [46] and OpenWGL [54]. The threshold is chosen from { 0 . 1 , 0 . 2 , . . . , 0 . 9 } or as the description of the original paper, to perform open-set recognition.

3) Generative Open-set Classification methods: PROSER [70] and G 2 Pxy [67].

4) Energy-based methods: GNNSAFE [56].

A detailed introduction of the baselines can be found in the Appendix E.1.

4.1 Open-set node classification comparison

Since real-world scenarios are complex, where seen and unseen differs in diverse tasks, we evaluate our model in terms of open-set classification from two aspects: near open-set classification, and far open-set classification under inductive and transductive learning setting, respectively, as introduced in test settings .

4.1.1 Near open-set classification

Table 1 presents the accuracy and macro-F1 scores of the methods applied to the near open-set classification task in the inductive learning setting, where the last class of each dataset is designated as the unknown class ( i.e. , u = 1), and the remaining classes are used for model training. It can be observed that EGonc outperforms all the baselines on the five datasets. This shows that EGonc can better differentiate between a known class and an unknown class, though they are similar to each other. Specifically, compared to the second-best method G 2 Pxy , EGonc achieves 2.41% and 2.22% improvements on average in terms of accuracy and F1 on the five datasets,

We illustrate detailed classification accuracy in terms of known classes and unknown classes in Appendix E.4. And the results shows that in order to gain the ability of unknown class detection, there is a slight decrease in the performance of known class classification, i.e. from 79.5% to 76.3% on average, comparing EGonc to closed-set classification method GCN_soft. However, the unknown class detection accuracy is improved from 0% to 76.1% on average, which is remarkable. Besides, we also evaluate the performance of our model in terms of multiple unknown classes. The results are demonstrated in Appendix E.5 and it can be observed that EGonc consistently outperforms the baselines.We further evaluate our model in terms of near open-set classification under transductive learning setting. The results are shown in Appendix E.6. EGonc consistently performs best.

4.1.2 Far open-set classification

For far open-set classification, following the protocol defined in [38], the models are trained and validated by training and validation instances of the original dataset (IND data). While for testing, instances from another dataset are augmented to the original test set as open-set classes (OOD data). for example, Co_Ci means that Core as IND data and Citeseer as OOD data.

The results for are shown in Table 3. It is found that EGonc can handle far out-of-distribution classes from diverse inputs and achieve noticeable performance improvement compared to other open-set classification methods. Surprisingly, the simple thresholding approach of

GCN_soft_ τ achieves comparable performance with EGonc. This inspires us to combine the generative methods with discriminative methods for far open-set classification in future work.

4.2 Ablation Study

We compare variants of EGonc with respect to loss function and substitute-unknown generation strategy to demonstrate its effect. As shown in Table 2, we firstly verify the effect of each loss we adopted, i.e. , ( C +1) -class cross-entropy loss l 1 , tailored complement entropy loss l 2 , and energy regularization loss l 3 . The results show that these three losses are indispensable to open-set node classification. Then we verify the effect of the substitute unknown samples we generated. Specifically,

we compare the performance of EGonc with respect to different kinds of (substitute) unknown samples: X far which are real far OOD samples, X rand which are substitute unknowns generated via mixup with random parent nodes; X int which are generated inter-class substitute unknowns, and X ext which are generated external substitute unknowns. From the results, we can see that EGonc generally achieves higher accuracy with the assistance of auxiliary unknown class samples. However, well-designed unknown substitutes are most beneficial for open-set node classification.

4.3 Generalization Analysis

The proposed model has no specific requirement on the GNN architecture for classification. The unknown-class substitute generation strategy and energy propagation take into account the topological properties of graph data. Therefore, they efficiently achieve the generation of representative unknown class samples and perform in-depth exploration of unkown-class features across different backbones. This design contributes to the model's performance in open-set classification tasks under different backbones. Table 4 ilustrates the performance of the proposed EGonc with different GNN architectures, including GCN, GAT and GraphSage. The results confirm the effectiveness and generalization ability of EGonc for open-set node classification.

5 Limitation and Conclusion

This paper proposed a novel energy-based generative open-set node classification method, EGonc, by estimating the underlying density of the training data to decide whether a given input is close to the IND data. To obtain better OOD detection capability, we generate substitute unknowns to mimic the distribution of real open-set samples, and use energy logit to represent the indicator of OOD-ness. Under constraint of cross entropy loss, complement entropy loss, and energy regularization loss, EGonc achieves superior effectiveness for unknown class detection and known class classification, which is validated by experiments. EGonc has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for IND and OOD samples. EGonc also has good generalization since it has no specific requirement on the GNN architecture.

Acknowledgments

This research was supported by National Natural Science Foundation of China (62206179, 92270122), Guangdong Provincial Natural Science Foundation (2022A1515010129, 2023A1515012584).

References

Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In IEEE conference on computer vision and pattern recognition , pages 1563-1572, 2016.

Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. Advances in neural information processing systems , 13, 2000.

Hao-Yun Chen, Pei-Hsin Wang, Chun-Hao Liu, Shih-Chieh Chang, Jia-Yu Pan, Yu-Ting Chen, Wei Wei, and Da-Cheng Juan. Complement objective training. In International Conference on Learning Representations , 2018.

Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International Conference on Machine Learning , pages 1725-1735, 2020.

Yue Chen, Yalong Bai, Wei Zhang, and Tao Mei. Destruction and construction learning for fine-grained image recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5157-5166, 2019.

Yifeng Ding, Zhanyu Ma, Shaoguo Wen, Jiyang Xie, Dongliang Chang, Zhongwei Si, Ming Wu, and Haibin Ling. Ap-cnn: Weakly supervised attention pyramid convolutional neural network for fine-grained visual classification. IEEE Transactions on Image Processing , 30:2826-2836, 2021.

Yilun Du, Joshua Meier, Jerry Ma, Rob Fergus, and Alexander Rives. Energy-based models for atomic-resolution protein conformations. In 8th International Conference on Learning Representations, ICLR 2020 , 2020.

Sven Elflein. Out-of-distribution detection with energy-based models. arXiv preprint arXiv:2302.12002 , 2023.

Shaohua Fan, Xiao Wang, Chuan Shi, Peng Cui, and Bai Wang. Generalizing graph neural networks on out-of-distribution graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Jerome Friedman, Trevor Hastie, and Rob Tibshirani. Regularization paths for generalized linear models via coordinate descent. Journal of statistical software , 33(1):1, 2010.

R Gao, Y Song, B Poole, YN Wu, and DP Kingma. Learning energy-based models by diffusion recovery likelihood. In International Conference on Learning Representations (ICLR 2021) , 2021.

ZongYuan Ge, Sergey Demyanov, Zetao Chen, and Rahil Garnavi. Generative openmax for multi-class open set classification. In British Machine Vision Conference . BMVA Press, 2017.

Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXiv:1912.03263 , 2019.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning , pages 1321-1330, 2017.

Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 1352-1361, 2017.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Neural information processing systems , 30, 2017.

Xiaotian Han, Zhimeng Jiang, Ninghao Liu, and Xia Hu. G-mixup: Graph data augmentation for graph classification. In International Conference on Machine Learning , pages 8230-8248. PMLR, 2022.

Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 41-50, 2019.

Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 41-50, 2019.

Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. International Conference on Learning Representations , 2017.

Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXiv:1812.04606 , 2018.

Marcel Hoffmann, Lukas Galke, and Ansgar Scherp. Open-world lifelong graph learning. In 2023 International Joint Conference on Neural Networks (IJCNN) , pages 1-9. IEEE, 2023.

Tao Hu, Honggang Qi, Qingming Huang, and Yan Lu. See better before looking closer: Weakly supervised data augmentation network for fine-grained visual classification. arXiv preprint arXiv:1901.09891 , 2019.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems , 33:22118-22133, 2020.

Ruyi Ji, Longyin Wen, Libo Zhang, Dawei Du, Yanjun Wu, Chen Zhao, Xianglong Liu, and Feiyue Huang. Attention convolutional binary neural tree for fine-grained visual categorization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10468-10477, 2020.

Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and S Yu Philip. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Transactions on Neural Networks and Learning Systems , 33(2):494-514, 2021.

Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 , 2016.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations , 2017.

Marc Lafon, Clément Rambour, and Nicolas Thome. Heat: Hybrid energy based model in the feature space for out-of-distribution detection. In NeurIPS ML Safety Workshop , 2022.

Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on energybased learning. Predicting structured data , 1(0), 2006.

Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems , 31, 2018.

Zhao Li, Xin Shen, Yuhang Jiao, Xuming Pan, Pengcheng Zou, Xianling Meng, Chengwei Yao, and Jiajun Bu. Hierarchical bipartite graph neural networks: Towards large-scale e-commerce applications. In IEEE International Conference on Data Engineering , pages 1677-1688, 2020.

Qi Liu, Miltiadis Allamanis, Marc Brockschmidt, and Alexander Gaunt. Constrained graph variational autoencoders for molecule design. Neural information processing systems , 31, 2018.

Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Advances in neural information processing systems , 33:21464-21475, 2020.

Xin Liu, Mingyu Yan, Lei Deng, Guoqi Li, Xiaochun Ye, Dongrui Fan, Shirui Pan, and Yuan Xie. Survey on graph neural network acceleration: An algorithmic perspective. arXiv preprint arXiv:2202.04822 , 2022.

Yixin Liu, Ming Jin, Shirui Pan, Chuan Zhou, Yu Zheng, Feng Xia, and Philip Yu. Graph self-supervised learning: A survey. IEEE Transactions on Knowledge and Data Engineering , 2022.

Yadan Luo, Zijian Wang, Zi Huang, and Mahsa Baktashmotlagh. Progressive graph learning for open-set domain adaptation. In International Conference on Machine Learning , pages 6468-6478, 2020.

Pramuditha Perera, Vlad I Morariu, Rajiv Jain, Varun Manjunatha, Curtis Wigington, Vicente Ordonez, and Vishal M Patel. Generative-discriminative feature representations for open-set recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11814-11823, 2020.

Phillip E Pope, Soheil Kolouri, Mohammad Rostami, Charles E Martin, and Heiko Hoffmann. Explainability methods for graph convolutional neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10772-10781, 2019.

Marc'Aurelio Ranzato, Y-Lan Boureau, Sumit Chopra, and Yann LeCun. A unified energybased framework for unsupervised learning. In Artificial Intelligence and Statistics , pages 371-379. PMLR, 2007.

Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks , 20(1):61-80, 2008.

Walter J Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recognition. IEEE transactions on pattern analysis and machine intelligence , 35(7): 1757-1772, 2012.

Walter J Scheirer, Lalit P Jain, and Terrance E Boult. Probability models for open set recognition. IEEE transactions on pattern analysis and machine intelligence , 36(11):2317-2324, 2014.

Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self-supervised outlier detection. arXiv preprint arXiv:2103.12051 , 2021.

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina EliassiRad. Collective classification in network data. AI magazine , 29(3):93-93, 2008.

Lei Shu, Hu Xu, and Bing Liu. Doc: Deep open classification of text documents. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pages 29112916, 2017.

R Sireesha, C Srinivasa Rao, and M Vijay Kumar. Graph theory based transformation of existing distribution network into clusters of multiple micro-grids for reliability enhancement. Materials Today: Proceedings , 80:2921-2928, 2023.

Ming Sun, Yuchen Yuan, Feng Zhou, and Errui Ding. Multi-attention multi-class constraint for fine-grained image recognition. In Proceedings of the european conference on computer vision (ECCV) , pages 805-821, 2018.

Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning , pages 20827-20840. PMLR, 2022.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations , 2018.

Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David LopezPaz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning , pages 6438-6447, 2019.

Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual-logit matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4921-4930, 2022.

Max Welling and Thomas N Kipf. Semi-supervised classification with graph convolutional networks. In J. International Conference on Learning Representations , 2017.

Man Wu, Shirui Pan, and Xingquan Zhu. Openwgl: Open-world graph learning. In IEEE international conference on data mining , pages 681-690, 2020.

Man Wu, Shirui Pan, and Xingquan Zhu. Openwgl: open-world graph learning for unseen class node classification. Knowledge and Information Systems , pages 1-26, 2021.

Qitian Wu, Yiting Chen, Chenxiao Yang, and Junchi Yan. Energy-based out-of-distribution detection for graph neural networks. arXiv preprint arXiv:2302.02914 , 2023.

Z Wu, S Pan, G Long, J Jiang, and C Zhang. Graph wavenet for deep spatial-temporal graph modeling. In The International Joint Conference on Artificial Intelligence , 2019.

Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems , 32(1):4-24, 2020.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 , 2018.

Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International conference on machine learning , pages 5453-5462. PMLR, 2018.

Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gnnexplainer: Generating explanations for graph neural networks. Neural information processing systems , 32, 2019.

Jiaxuan You, Jonathan M Gomes-Selman, Rex Ying, and Jure Leskovec. Identity-aware graph neural networks. In AAAI Conference on Artificial Intelligence , volume 35, pages 10737-10745, 2021.

Yang Yu, Wei-Yang Qu, Nan Li, and Zimin Guo. Open-category classification by adversarial sample generation. In International Joint Conference on Artificial Intelligence , pages 33573363, 2017.

Hanqing Zeng, Muhan Zhang, Yinglong Xia, Ajitesh Srivastava, Andrey Malevich, Rajgopal Kannan, Viktor Prasanna, Long Jin, and Ren Chen. Decoupling the depth and scope of graph neural networks. Neural Information Processing Systems , 34:19665-19679, 2021.

He Zhang, Bang Wu, Xingliang Yuan, Shirui Pan, Hanghang Tong, and Jian Pei. Trustworthy graph neural networks: Aspects, methods and trends. arXiv preprint arXiv:2205.07424 , 2022.

Jingyang Zhang, Nathan Inkawhich, Randolph Linderman, Yiran Chen, and Hai Li. Mixture outlier exposure: Towards out-of-distribution detection in fine-grained environments. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 5531-5540, 2023.

Qin Zhang, Zelin Shi, Xiaolin Zhang, Xiaojun Chen, Philippe Fournier-Viger, and Shirui Pan. G2pxy: generative open-set node classification on graphs with proxy unknowns. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence , pages 4576-4583, 2023.

Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial networks. In 5th International Conference on Learning Representations, ICLR 2017 , 2017.

Xin Zheng, Yixin Liu, Shirui Pan, Miao Zhang, Di Jin, and Philip S Yu. Graph neural networks for graphs with heterophily: A survey. arXiv preprint arXiv:2202.07082 , 2022.

Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Learning placeholders for open-set recognition. In the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4401-4410, 2021.

Shuang Zhou, Xiao Huang, Ninghao Liu, Qiaoyu Tan, and Fu-Lai Chung. Unseen anomaly detection on networks via multi-hypersphere learning. In Proceedings of the 2022 SIAM International Conference on Data Mining (SDM) , pages 262-270. SIAM, 2022.

Qi Zhu, Chao Zhang, Chanyoung Park, Carl Yang, and Jiawei Han. Shift-robust node classification via graph adversarial clustering. arXiv preprint arXiv:2203.15802 , 2022.

Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B: Statistical Methodology , 67(2):301-320, 2005.

A Appendix / supplemental material

B Related works

B.1 Graph Neural Networks

Graph neural networks are deep learning based models that apply neural networks to graph learning and representation [41]. They are used in a wide variety of tasks [53, 50] and have a strong theoretical foundation [39, 61]. GNNs were shown to yield state-of-the-art performance on many graph-related tasks [36, 35] and have applications in various fields such as e-commerce [32], traffic analysis [57], chemistry [33] and for knowledge bases [26].

The original GNN model brought the idea of combining the design of modern neural networks with graph learning [58], e.g. , Graph Convolutional Networks (GCNs) [28], Graph Attention Networks (GATs) [50], Graph Isomorphism Networks (GINs) [59], and GraphSage [16]. Then various improvements have been made to the basic GNNs for different purposes [36, 35, 69, 65]. For instance, to alleviate the over-smoothing problem, decoupling mechanisms [64] and identity mapping [4] were introduced respectively. To increase the expressive power of most GNNs by the 1-Weisfeiler-Lehman (1-WL) graph isomorphism test, identity-aware Graph Neural Networks [62] were proposed.

B.2 Open Set Node Classification

Open set recognition and classification [1, 43] require classifiers capable of categorizing known classes and detecting objects of unknown types during testing. Two main paradigms have been investigated: generative models and discriminative models. Generative models improve prediction performance by emulating the distribution of unknown class nodes, while discriminative models use a threshold to distinguish between known and unknown classes, thereby improving prediction accuracy.

Prior methods, such as OpenMax [1] and DOC [46], mainly focus on addressing the overconfidence issue of deep neural networks for unknown classes, in image and text classification problems, respectively. Specifically, on graph structured data, OpenWGL [54] employs an uncertainty loss in the form of a graph reconstruction loss [27] on unlabeled data. PGL [37] extends a previous unsupervised domain adaption framework for the open-set scenario with graph neural networks. However, it focuses on the unsupervised domain adaption and conditional shift problem. With the development of deep learning, several novel approaches have been proposed. SRNC [72] further presents a domain adaption framework for the two most common kinds of data shifts in graph structured data, i.e. , open-set and closed-set data shift, by combining a graph adversarial clustering technique to shift-robust node classification. MHGL [71] extends traditional out-of-distribution (OOD) detection tasks to open-set scenarios, employing Pattern Distribution Estimator (PDE) and Multi-Hypersphere Graph Learning algorithm to model fine-grained patterns and identify previously unseen anomalous patterns. GOOD [22] proposes a weakly supervised relevance feedback approach, named Open-WRF, to mitigate sensitivity to thresholds. G 2 Pxy [67] use the unknown proxies generated via mixup to efficiently anticipate the distribution of novel classes and achieve excellent prediction performance under the constraint of both cross entropy loss and complement entropy loss.

B.3 Energy-based Models

Energy-based modeling as well as its application is mostly concentrated on out-of-distribution detection. Grathowohl et al. [13] found that a standard discriminative classifier of p ( y | x ) can be interpreted as an energy-based model (EBM) [40] for the joint distribution p ( x, y ) . Junbo Zhjao, Michael Mathien et al. introduces a new model of Generative Adversarial Networks, called Energybased Generative Adversarial Network (EBGAN) [68], which features the discriminator being viewed as an energy function, introducing a new theoretical perspective and method for Generative Adversarial Networks. Tuomas Haarnoja, Haoran Tang et al. apply EBMs to the field of reinforcement learning, and introduces a novel reinforcement learning method soft Q-learning [15] using deep energy-based polices for continuous states and actions, focusing on maximum entropy policies and improved task exploration. Yilun Du, Joshua Meier et al. innovatively applies an energy-based model [7] to score protein conformations at the atomic level, introducing machine learning methods into the

field of protein design. Ruiqi Gao, Yang Song et al. introduces a method called Diffusion Recovery Likelihood [11] to effectively learn and sample from Energy-Based Models (EBMs) by learning from the diffused versions of the data, overcoming the challenges of training and sampling EBMs on high-dimensional datasets. MareLafun, Elias Ramzi et al. introduced the HEAT [29], which leverages the versatility of the EBM framework to provide a strong OOD detection method. Qitian Wu, Yiting Chen et al. applied EBM to the task of OOD detection in graph domains and proposed GNNSAFE [56]. GNNSAFE is a method for detecting anomilies in graph data, which can improve the robustness and generalization of GNN. Impressively, the use of unknown classes samples information to improve EBM performance is discussed in GNNSAFE.

C Proofs for Proposition

C.1 Proof for Proposition 1

The gradient of l 1 is given by

The gradient of l 2 is given by

̸

̸

where p ′ ( y = j | x i , A ) = e h k i ( θ ; x i ,A ) [ j ] ∑ C +1 c ′ ̸ = y i e h k i ( θ ; x i ,A ) [ c ′ ] is an approximate probability distribution established on the complementary classes

In the Eq. (15) and Eq. (16), We computed the gradient with respect to the parameters. The information presented in the preceding equation indicates that the training process for D tr, which aims to minimize the first-order gradient of l 1 + λ 2 l 2 , leads to a reduction in the energy score. In a broader context, the energy output generated by the trained model tends to decrease for any instance originating from the distribution D tr. Besides, for the unknown substitutes X sub we constructed, we will adjust their energy scores by the l 3 to keep the energy model in an optimal state for fulfilling its function.

̸

̸

̸

̸

̸

C.2 Proof for Proposition 2

We can demonstrate the proposition 2 when considering ∑ j ˆ A ij E ( k -1) j ∑ j ˆ A ij < E ( k -1) i . We need to rewrite the Eq. (10) from the perspective of a single vector:

A similar prove can be applied to establish the contrary outcome in the case ∑ j a ij E ( k -1) j ∑ j ˆ A ij > E ( k -1) i

C.3 Proof for Proposition 3

At first, we define E ( x ; h θ ∗ ) as the energy scores yielded by the model h θ ∗ minimizing the energy regularization loss l 3 . Then, h θ + 1 and h θ + 2 denotes the model yielding the optimal predictive softmax distribution that minimizing the cross entropy loss l 1 and the model yielding the optimal predictive softmax distribution that minimizing the tailored complement entropy loss l 2 . i.e. ,

̸

At the same time, considering that E ( x, G x ; h θ ) = -log ∑ C +1 c =1 e h θ ( x, G x ) . Therefore we start from this equation and we have

The above equation implies an equivalence relationship between E ( x, G x , h θ ∗ ) and E ( x, G x ; h θ ∗ ) -E ( x, G x ; h θ + 1 ) -log ∑ C +1 c =1 e h θ + 1 ( x, G x ) . Meanwhile, continuing to consider that E ( x, G x ; h θ ∗ ) = -log ∑ C +1 c =1 e h θ ∗ ( x, G x ) , we can further demonstrate that the predictive softmax probability distribution obtained from E ( x, G x , h θ ∗ ) is

The above predictive softmax probability distribution exactly minimizes l 1 as defined by Eq (19). We thus have proven that the optimal energy that minimizes l 3 intrinsically induce the predictive softmax distribution that minimizes l 1 for D tr .

Similarly, when we consider the relationship between the l 2 and l 3 , we can start again from E ( x, G x ; h θ ) = -log ∑ C +1 c =1 e h θ ( x, G x ) and get

The above equation implies an equivalence relationship between E ( x, G x , h θ ∗ ) and E ( x, G x ; h θ ∗ ) -E ( x, G x ; h θ + 2 ) -log ∑ C +1 c =1 e h θ + 2 ( x, G x ) . We thus can further demonstrate that the predictive softmax probability distribution with respect to class C + 1 obtained from E ( x, G x , h θ ∗ ) is

̸

̸

Considering that the energy scores are not computed for a specific class of labels, but rather are computed considering all classes, and the loss l 3 tends to reduce the energy scores for x ∈ D tr . Meanwhile, combine this with the fact that the loss l 2 tends to drive a higher probability of output for class C , it is reasonable to assume that the probability of this probability distribution is not overly large for class y . Then, The former term can be regarded as a coefficient modifying the latter term. We thus have proven that the optimal energy that minimizes l 3 intrinsically induce the predictive softmax distribution that minimizes l 2 to a certain extent for D tr .

D Algorithm

The algorithm of EGonc is illustrated in Algorithm 1. The given graph data are inputted into a GNN model (Line 4), followed by the generation of inter-class and external unknown substitutes. Then the generated substitutes and the original training data are input into the rest of neural network and learn an open-set classifier (Lines 6-10). As new graphs are fed in for training (Line 6), the energy scores for the known classes samples and unknown class samples are calculated (Line 7). Under the constraint of the total loss, the model is trained and optimized.

̸

Algorithm 1 EGonc: open-set node classification

Require:

: G = ( V, E, X ) : a graph with links and features;

D tr = { G,Y } : train set with labeled nodes;

X te = S ∪ U : test set where S are the known classes appeared in training and U are the unknown classes;

Ensure: f ( X te →Y ) , Y ∈ { 1 , . . . , C, unknown } .

1: Obtain the inter-class node pairs { ( x i , y i ) , ( x j , y j ) } ∈ D tr s.t. y i = y j & a ij = 1

̸

2: Obtain the peripheral nodes that are leaf nodes and low confident nodes.

3: while not convergence do

4: For the first m = 1 , . . . k layer: h i m = f m ( θ 1 ; h m -1 i , h m -1 j , j ∈ N i ) , ∀ x i ∈ D tr

5: At the k -th layer:

Create unknown substitutes X sub using Eq. (8) & (9)

Augment the substitutes to known class samples:

D

tr

=

D

tr

∪

(

X

sub

, Y

C+1

)

6: For the m = k +1 , . . . , k -1

1 layers: h i m = f m ( θ 1 ; h m -1 i , h m -1 j , j ∈ N i ) , ∀ x i ∈ D tr

7: For the m = k 1 , . . . , K layers: h i m = f m ( θ 1 ; h m -1 i , h m -1 j , j ∈ N i ) , ∀ x i ∈ D tr E i m = f m ( E m -1 i , E m -1 j , j ∈ N i ) , ∀ x i ∈ D tr

8: For open-set classifier layer:

Obtain cross entropy loss as Eq. (11)

Obtain complement entropy loss as Eq. (12)

Obtain energy regularization loss as Eq. (13)

9: Back-propagate loss gradient using Eq. (14) and update weights

10: if early stopping condition is satisfied then

11: Terminate

12: end if

13: end while

D.1 Complexity Analysis

Detailed training process of EGonc is illustrated in Algorithm 1. In terms of complexity, compared with normal closed-set node classification, our extra computation comes from three parts: 1) unknown substitute generation, which has linear complexity with the quantity of edges and the quantity of generated substitutes, i.e. , O ( | E | + | X sub | ) ; 2) energy scores calculation, which has linear complexity with the quantity of the known class samples after augmentation, i.e. , O ( D tr ) 3) substitute classifier training, the complexity of this part depends on the adopted backbone GNN and the quantity of generated substitutes. Assume O ( | f θ | ) be the time GNN f θ 1 and classifier f θ 2 takes to compute a single node embedding and make a prediction, the complexity of substitute classifier training is O ( | X sub | · | f θ | ) . Overall, the extra complexity is linear with the number of edges in the original graph and the number of generated substitutes. Experimental results on parameter analysis of generated substitutes quantity (see Fig. 1) show that a relatively small number of substitutes can achieve quite good performance. Thus, the extra complexity is reasonable for open-set node classification.

E Supplementary experiments

E.1 Baseline Details

GCN_soft, GCN_sig : GCN [28] is adopted for graph learning. We constructed four baselines using GCN, where GCN_soft and GCN_sig use softmax and sigmoid layers, respectively, as the final layer for training and classification. Here we adopt GCN with sigmod layer as baselines since softmax is observed to be overconfident to unknown class samples.

GCN_soft_ τ , GCN_sig_ τ : Additionally, GCN_soft_ τ and GCN_sig_ τ perform classification based on these two model using fixed thresholds chosen in { 0 . 1 , 0 . 2 , . . . , 0 . 9 } .

Openmax: Openmax [1] is an open-set recognition model based on 'activation vectors', i.e. , penultimate layer of the network.

By modeling the distance from 'mean activation vector' using the extreme value distribution, softmax scores are calibrated for each class and updated to Openmax scores, which are then used for open-set classification. GCN is further combined with Openmax for open-set node classification on graph data.

DOC: DOC [46] is an open-world classification method for text classification. It uses multiple 1-vs-rest of sigmoids as the final output layer and defines an automatic threshold setting mechanism. GCN is combined with DOC to allow comparison with the proposed model. GCN is used to obtain the node representations and DOC is used to do open-set classification and rejection.

PROSER: PROSER [70] is a novel and highly effective open-set recognition method used in the field of image processing. We paired it with GCN for graph-based comparison.

OpenWGL: OpenWGL [54] employs an uncertainty loss in the form of graph reconstruction loss on unlabeled data and using an adaptive threshold to detect the unknown class samples.

GNNSAFE: GNNSAFE [56] proposes an effective OOD discriminator based on an energy function derived from graph neural networks (GNNs) trained with standard classification loss. We apply it from the OOD detection domain to the open-set classification domain.

G 2 Pxy : G 2 Pxy [67] uses the generated proxies generated via mixup, with the help of cross entropy loss and complement entropy loss, shows excellent performance in open-set classification

E.2 Datasets Stastics

E.3 Implementation Details

The GCN is configured with two hidden GCN layers in the dimension of 512 and 128, followed by an additional multilayer perceptron layer of size 64. EGonc is implemented with PyTorch and the networks are optimized using stochastic gradient descent with a learning rate of 1 e -3 . The balance parameters λ 1 , λ 2 and λ 3 are chosen by a grid search in the interval from 10 -2 to 10 2 with a step size of 10 1 .

The baseline methods are evaluated according to the instructions reported in the original papers with the same parameter configuration unless otherwise specified, and the best results are selected. For each experiment, the baselines and the proposed method were applied using the same training, validation, and testing datasets. The hyperparameters were tuned to get the best performance on the validation set. All the experiments were conducted on a workstation equipped with an Intel(R) Xeon(R) Gold 6226R CPU and an Nvidia A100.

E.4 Detailed performance on IND and OOD classes for near open-set classification

We show the detailed classification accuracy in terms of known classes (IND classes) and unknown classes (OOD classes) respectively, in Table 6. The experiment was conducted under the same setting of Table 1. It shows that in order to gain the ability of unknown class detection, there is a slight decrease in the performance of known class classification, i.e. from 79.5% to 76.3% on average, comparing EGonc to closed-set classification method GCN_soft. However, the unknown class detection accuracy is improved from 0% to 76.1% on average, which is remarkable. Compared to other open-set classification methods, such as the globally second best method, G 2 Pxy , the average performance of unknown-class node detection is improved from 70.7% to 76.1% while the performance of the known-class classification experienced a slight decrease from 78.6% to 76.3% on average. When compared with the globally third best method, PROSER, the average performance of unknown-class node detection is improved from 63.6% to 76.1% while the performance of the known-class classification still gain a certain increase from 73.0% to 76.3% on average.

E.5 Near open-set classification on multiple unknown classes

Further, we explore the performance in terms of multiple unknown classes. In Table 7, we provide the results of inductive learning on near open-set classification with multiple unknown classes, i.e. , u = 2,3 for the Cora and Citeseer datasets and u=5,10,15,20 for ogbn_arxiv dataset, respectively. It can be observed that EGonc consistently performs best.

E.6 Near open-set classification under transductive learning setting

Table 8 presents the accuracy and F1 scores of the methods applied to the near open-set classification task with one unknown class (the last class), under the transductive learning setting, i.e. , the features of unknown class nodes are used to facilitate the model training or validation. It is observed that EGonc still consistently outperforms the baselines. Specifically, compared with the second-best method i.e. , G 2 Pxy , EGonc achieves 3.11% and 2.89% improvements on average in terms of accuracy and F1, and achieves 9.94% and 13.6% improvements over the third-best method (PROSER) on average in terms of accuracy and F1, respectively. Furthermore, by comparing Table 1 and Table 8, it is observed that the involving of real unknown class information during training or validation benefits the model, which is reasonable.

E.7 Parameter learning on quantity of substitutes.

The substitute unknown node generation module of EGonc is crucial and the quantity of generated substitutes need to be predefined. We assess the performance of EGonc with different numbers of

0.9

0.8

0.7

Performance

Cora-Acc

0.6

Cora-F1

Citeseer-Acc

0.5

Citeseer-F1

1O

Dblp-AcC

Dblp-F1

0.4--

O

Pubmed-Acc

Pubmed-F1

0.3

0.1

0.5

1

2

Quantity of generated substitutes

generated substitutes, i.e. , { 0 . 1 , 0 . 5 , 1 , 2 , 5 } times the average number of nodes in each known class, to see the influence of the quantity of substitutes. Results are shown in Fig. 1. It is observed that both accuracy and macro-F1 are stable while the quantity of generated substitutes varies a lot. Thus, a relatively small number of substitutes can be used for better efficiency.

E.8 Parameter learning on weights of losses.

In ablation study, we verify that the cross-entropy loss (Eq. (11)), tailored complement entropy loss (Eq. (12)), and energy regularization loss (Eq. (13)) are all indispensable. Here we further assess the performance of EGonc with various weights of these three losses, i.e. , λ 1 , λ 2 , and λ 3 , with the range of { 0 . 01 , 0 . 1 , 1 , 10 , 100 } . The results are shown in Fig. 2. We can see that it is better to choose small values for the weights generally. And the wights for tailored complement entropy loss ( λ 2 ) are relatively large compared to the weights of other two losses ( λ 1 , λ 3 ). For example, the best value for λ 2 is 1 for Cora, Citeseer, Dblp and 10 for Pubmed while the best values for λ 1 and λ 3 are either 0.01 or 0.1. It illustrates the importance of complement entropy loss in open-set node classification task.

(a)

(b)

(c)

(d)

(e)

(f)

Cora-Acc

0.8-

0.8-

Cora-F1

0.7 --

0.7 --(

Citeseer-Acc..

Citeseer-F1

0.6-

0.6-

Performance

Performance

0.5

0.5

0.4-

0.4

0.3

0.3-

0.2-

0.2-

10

Dblp-F1

Dblp-AcC

0.1 --

0.1-.-O

Pubmed-Acc

Pubmed-F1

0.0-

0.01

0.1

1

10

100

10 0

0.1

1

10

100

Weight for CrE loss on proxy class

Weight for CrE loss on proxy class

0.8---

Cora-Acc

0.8-

Cora-F1

0.7--

0.7--

Citeseer-Acc

Citeseer-F1

0.6--

0.6-

Performance

0.5

0.4-

0.4-

0.3

0.3

0.2-

0.2-

Dblp-F1

Dblp-AcC

0.1---

0.1--O

Pubmed-Acc

Pubmed-F1

0.00.01

0.0 0.01

0.1

1

10

100

0.1

1

10

100

Cora-Acc

0.8

0.8-

Cora-F1

0.7

0.7 --

Citeseer-Acc

Citeseer-F1

0.6-

0.6-

Performance

0.5

0.4-

0.4-

0.3-

0.3

0.2---

0.2-

101

O

Dblp-AcC

Dblp-F1

0.1 ---

0.1--

Pubmed-Acc

Pubmed-F1

0.0-

0.01

0.1

1

10

100

0.0

0.01

0.1

1

10

100

Weight for energy regularization loss

Weight forenergy regularization loss

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: [TODO] Yes, the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: [TODO] Yes, the paper discusses the limitations of the work performed by the authors in the "Limitation and Conclusion" section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: [TODO] We provide complete and correct proofs in the appendix section "Proofs for Propositions

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: [TODO] The paper provides pseudocode, model details, and experimental details.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: [TODO] We will provide access to the data and code, along with sufficient instructions to reproduce the main experimental results when the paper is accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: [TODO] Yes, please refer to E.3 in the appendix and test settings in the experiment section.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: [TODO] No, the paper does not include error bars, confidence intervals, or statistical significance tests for the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: [TODO] No, the article only provides the required computational resources, but does not provide detailed information such as memory, execution time, etc.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: [TODO] Yes, it does comply.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: [TODO] The paper is foundational research, does not involve societal impact, and does not discuss societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [TODO] The article poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: [TODO] The article specifies the sources of the referenced assets as much as possible.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: [TODO] We will strive to convey these details using structured templates as much as possible, but due to time constraints, they may not be comprehensive.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [TODO] The article does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [TODO] The article does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new method EGonc for open-set node classification in graphs. EGonc performs open-set classification by incorporating energy-based models into the graph learning diagram. The training of the model only requires in-distribution data, while out-of-distribution training data are generated synthetically. In the experiments, the authors compare EGonc against a wide range of baseline methods on several benchmarks and provide ablation studies.\n\nSTRENGTHS:\n1. The paper focuses on open-set node classification in graph learning, which is an important problem in real-world applications.\n2. Incorporating energy-based models into graph learning is a new method that has not been explored.\n3. The proposed method shows good performance, beating a wide range of baselines as is shown in the experiments.\n\nWEAKNESSES:\n1. An important claim made in the paper is not supported. The paper claims that the proposed method \"has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for IND and OOD samples\", but there is no theoretical analysis supporting this claim. \n\n2. The experiments are not sufficient. First, using one class as the out-of-distribution (OOD) class is very different from the real-world scenario where OOD examples can be quite diverse. Second, the paper does not compare against several important OOD detection baselines[1][2]. The proposed method has a negative impact on the in-distribution accuracy, while the methods in [1][2] will not. Third, the error bars are not reported. Fourth, an ablation study for energy propagation is missing, which seems to be an important component of the method.\n\n3. The presentation needs major improvement. First, there are a lot of format issues (e.g., missing reference in line 127, texts in the equations are not corrected formatted). Second, the intuitions behind $l_2$ and $l_3$ are not well-explained. It is not clear what are the roles of the two losses. It would be more helpful to use examples to show the intent of the two losses. Same for equation 8 and 9. Third, Proposition 1 uses vague descriptions, which is not valid for a mathematical statement. \n\n[1] Liu, Weitang, et al. \"Energy-based out-of-distribution detection.\" Advances in neural information processing systems 33 (2020): 21464-21475.\n\n[2] Sun, Yiyou, et al. \"Out-of-distribution detection with deep nearest neighbors.\" International Conference on Machine Learning. PMLR, 2022.\n\nQUESTIONS:\n1. What's the intuition behind the unknown substitute generation formula (equations 8 and 9)? What is the intuition of including $l_2$ and $l_3$ into the learning objective? Can you use examples to explain them?\n2. In line 127, the second kind of substitute is generated from nodes with low classification confidence. Would that also be affected by the over-confident problem?\n3. In equation 14, how the $\\lambda$'s are selected?\n4. I suggest moving line 155-160 (the introduction of GNN) to the background section.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a new energy-based generative open-set node classification method to achieve open-set graph learning. It uses energy-based models to estimate the underlying density of the seen classes and to evaluate the probability of a given input belonging to the IND classes or OOD classes. Besides, it mimics the distribution of real open-set classes by generating substitute unknown samples under the guidance of graph structure. Besides, the proposed method has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for IND and OOD samples.\n\nSTRENGTHS:\n1)This paper studies a significant and interesting problem, and the method can be used in a wide range of real-world applications. \n2)The paper is overall well motivated. The proposed model is reasonable and sound. Theoretical analysis is performed. \n3)Comprehensive experiments, including near open-set classification, far open-set classification, ablation studies and parameter learning, are conducted. The results demonstrate good performance.\n\nWEAKNESSES:\n1) What is difference between open-set node classification and out-of-distribution detection problem? The authors should illustrate the differences and whether the proposed method can solve these two problems simultaneously.\n2) In section 3.1, it claims that the insouciantly selected unrelated data does not help the open-set classification, thus substitute unknown nodes near the class boundaries are generated. Why randomly selected data does not help? Aren’t they real-world open-set data which can help the open-set learning?\n3) It is a good idea to imitating data from open-set distribution by generating pseudo out-of-distribution samples, however the diversity of the generated data is essential. The method used in the paper is manifold mixup, does this method can ensure the diversity of the generated data?\n\nQUESTIONS:\nSee Weaknesses part.\n\nLIMITATIONS:\nSee Weaknesses part.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on energy-based open-set node classification, and adopted a generative method to obtain the explicit specific score of a node belonging to the ‘unknown‘ class. To generate good substitute unknowns, it adopted energy-based models to estimate the density of classes and guarantee the nice theoretical properties of the proposed method with an overall distinguishable margin between the detection scores from IND and OOD samples. Extensive experiments are conducted and show superior performance.\n\nSTRENGTHS:\n(1)\tThe paper is well-structured and overall easy to follow.\n\n(2)\tThe proposed method can obtain an explicit specific score of belonging to OOD data for each input, which provide more information and show the confidence of the model on its decision. \n\n(3)\tThe proposed method has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for IND and OOD samples, the adopted energy regularization loss has consistent effects with the cross-entropy loss as well as with the tailored complement entropy loss on the known classes, and that they are not mutually exclusive.\n\n(4)\tThe proposed method is agnostic to specific GNN architecture, which demonstrate its generalization for some content.\n\nWEAKNESSES:\n(1)\tIn the main part of the paper, some important experiment settings are missing, which makes the readers feel confused about the motivations of the experiments.\n\n(2)\tWhat are the differences between near open-set classification and far open-set classification? Why do they matter? If a method can achieve good far open-set classification, it should also be good at near open-set classification?\n\n(3)\tFor section 3, the model design is quite complex, the illustration should be more straightward and make each term and each symbol has enough explanation.\n\n(4)\tSome sentences should be improved to make them clearer, such as line 271-273: “Inspired by the Elastic Network Zou and Hastie [2005], Friedman et al. [2010], and considering the similarity in form and function between regularization terms and corresponding error terms…”\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors proposes a new generative open set node classification method (EGonc) to address the challenge of Open Set Classification (OSC) for safely deploying machine learning models in an open world. Traditional softmax-based methods are overly confident on unseen data, making them vulnerable to out-of-distribution (OOD) samples. EGonc uses graph structure to generate substitute unknowns simulating real open set samples and employs Energy-Based Models (EBM) for density estimation. The method learns additional energy logits from feature residuals to indicate OOD-ness, ensuring a distinguishable margin between in-distribution (IND) and OOD detection scores. Experimental results demonstrate the superiority of EGonc.\n\nSTRENGTHS:\n1. Novel Approach: The paper introduces a new method (EGonc) for open set classification that leverages energy-based models, which is a significant departure from traditional softmax-based methods.\n2. Theoretical Guarantees: EGonc comes with strong theoretical properties that ensure a distinguishable margin between in-distribution and out-of-distribution samples.\n3. Robustness to OOD Variability: By simulating real open set samples and learning virtual OOD classes, the method enhances robustness against the diversity of OOD examples.\n\nWEAKNESSES:\n1. The motivation primarily focuses on the limitations of softmax-based neural networks without delving deeply into the broader implications and potential impact of improved open set classification across various domains and real-world applications.\n2. The advantages of the proposed method could be strengthened by providing more concrete examples of practical scenarios where current OSC methods fail and how EGonc specifically addresses these failures, making a stronger case for its real-world applicability and importance.\n3. While the method shows promise, its applicability and performance in domains outside of those tested in the experiments need further validation.\n\nQUESTIONS:\n1. Why were these datasets chosen, and what advantages do they offer in validating the proposed method?\n2. Why was the 2017 GCN by Kipf and Welling chosen as the backbone neural network for experimental evaluation, instead of using other models?\n\nLIMITATIONS:\nSee the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}