EGonc : Energy-based Open-Set Node Classification with substitute Unknowns

Authors: qin  zhang
zelin  shi
shirui  pan
junyang  chen
huisi  wu
xiaojun  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 05:13:53

Abstract: Open-set Classification (OSC) is a critical requirement for safely deploying machine learning models in the open world, which aims to classify samples from known classes and reject samples from out-of-distribution (OOD). 
Existing methods exploit the feature space of trained network and attempt at estimating the uncertainty in the predictions.
However, softmax-based neural networks are found to be overly confident in their predictions even on data they have never seen before and
the immense diversity of the OOD examples also makes such methods fragile.
To this end, we follow the idea of estimating the underlying density of the training data to decide whether a given input is close to the in-distribution (IND) data and adopt Energy-based models (EBMs) as density estimators. 
A novel energy-based generative open-set node classification method, \textit{EGonc}, is proposed to achieve open-set graph learning. 
Specifically, we generate substitute unknowns to mimic the distribution of real open-set samples firstly, based on the information of graph structures. 
Then, an additional energy logit representing the virtual OOD class is learned from the residual of the feature against the principal space, and matched with the original logits by a constant scaling. This virtual logit serves as the indicator of OOD-ness. 
EGonc has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for IND and OOD samples. 
Comprehensive experimental evaluations of EGonc also demonstrate its superiority.

Main paper: 
 introduction: 
 Learning on graphs, where instance nodes are inter-connected, has become one of the central problems for deep learning, as relational structures are pervasive and induce data inter-dependence which hinders trivial adaptation of existing approaches that assume inputs to be i.i.d. sampled [56] . However, current models mostly focus on improving testing performance of in-distribution (IND) data and largely ignore the potential risk w.r.t. out-of-distribution (OOD) testing samples that may cause negative outcome if the prediction is overconfident on them [31] .Real-world applications often require machine learning systems to interact with an open world, violating the common assumption that testing and training distributions are identical. This urges the community to devote increasing efforts on how to enhance models' generalization [9] and reliability [47] w.r.t. out-of-distribution data, by classifying samples from known classes and rejecting samples from out-of-distribution.Existing methods in addressing this task normally are achieved by assuming that a model is more confident in its predictions for closed-set samples than for open-set samples [55, 8] . Based on this assumption, a threshold is applied to a model's confidence score to separate unknown samples from known ones. Despite its apparent intuitiveness [20] , this method often fails because deep neural networks tend to be over-confident in their predictions, assigning high confidence scores even to unknown inputs [19, 14, 1, 42, 71] . Furthermore, determining an optimal threshold for distinguishing known from unknown classes is a challenging and time-consuming task [38, 12, 22] . Others exploit the feature space of trained network and attempt at estimating the density of in-distribution (IND) features to address OOD detection [29] , such as Gaussian Mixture Models [31, 44] , nearest neighbors distribution [49] and energy logits [34, 8] . However, the immense diversity of the OOD examples makes such methods fragile [52] . For example, GMMs' density explicitly decreases when moving away from training data, making them effective for far-OOD detection, while energy logits benefits from the classifier training to obtain strong results on near-OOD samples [29, 52] .To overcome these limitations, researchers try to make advantages of unrelated samples in the openset environment [21] . Nevertheless, the methods using real open-set samples to help the training of models often require a substantial amount of data [23, 48, 5] , and insouciantly selected data does not really help [66] . Well-designed outlier samples are difficult to obtain and require experts for identification and labeling [6, 25] .In this paper, we follow the idea of estimating the underlying density of the training data to decide whether a given input is close to the IND data and adopt Energy-based models (EBMs) as density estimators. To further improve the OOD detection performance, we try to generate substitute unknown samples carefully under the guidance of graph structure, to mimic the distribution of real open-set classes. Specifically, a novel energy-based generative open-set node classification method, EGonc, is proposed to achieve open-set graph learning. Based on the generated substitute unknowns, an additional energy logit representing the virtual OOD class is learned from the residual of the feature against the principal space, and matched with the original logits by a constant scaling. This virtual logit serves as the indicator of OOD-ness. EGonc has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for IND and OOD samples. Comprehensive experimental evaluations of EGonc also demonstrate its superiority. To sum up, the contributions of this paper are summarized as follows:• A novel method, EGonc, for open-set node classification is proposed by redefining the open-world graph learning paradigm based on the energy model and elaborate unknown-substitute generation; • EGonc has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for IND and OOD samples. The adopted energy regularization loss has consistent effects with the cross-entropy loss as well as with the tailored complement entropy loss on the known classes, and that they are not mutually exclusive; • No open-set data (samples of unknown classes or any side information of unknown classes) is required during training and validation. EGonc has an explicit classifier for unknowns and does not require threshold tuning; • EGonc is agnostic to specific GNN architecture and demonstrates robust generalization capabilities.• Experiments conducted on benchmark graph datasets illustrate the commendable performance achieved by EGonc.2 Background 
open-set node classification: 
 This paper focuses on the node classification problem for a graph. Consider a graph denoted as G = (V, E, X), where V = {v i |i = 1, . . . , N } is a set of N nodes in the graph. E = {e i,j |i, j = 1, . . . , N. i ̸ = j} is a set of edges between pairs of nodes v i and v j . X ∈ R N ×d denotes the feature matrix of nodes, and d is the dimension of node features. x i ∈ X indicates the feature vector associated with each node v i . The topological structure of G is represented as an adjacency matrix A ∈ R N ×N , where A i,j = 1 if the nodes v i and v j are connected, i.e., ∃e i,j ∈ E, otherwise A i,j = 0. Y ∈ R N ×C is the label matrix of G, where C is the already-known node classes. If node v i ∈ V is associated with a label c, y i,c = 1, otherwise, y i,c = 0.For a typical closed-set node classification problem, a GNN encoder f θg takes node features X and adjacency matrix A as input, aggregates the neighborhood information and outputs representations. Then, a classifier f θc is used to classify the nodes into C already-known classes. The GNN encoder and the classifier are optimized to minimize the expected risk [63] in Eq. ( 1), with the assumption that test data D te and train data D tr share the same feature space and label space, i.e., f * = arg min f ∈H E (x,y)∼Dte I(y ̸ = f (θ g , θ c ; x, A))where H is the hypothesis space, I(•) is the indicator function which outputs 1 if the expression holds and 0 otherwise. Generally, it can be optimized with cross-entropy to discriminate between known classes.In open-set node classification problem, given a graph G = (V, E, X), D tr = (X, Y ) denotes the train nodes. D te = (X te , Y te ) is the test nodes, where X te = S ∪ U , Y te = {1, . . . , C, C + 1, . . .}. S is the set of nodes that belong to seen classes that already appeared in D tr and U is the set of nodes that do not belong to any seen class (i.e., unknown class nodes). Open-set node classification aims to learn a(C + 1)-class classifier f θc that f (θ g , θ c ; X te , A) : {X te , A} → Y, Y = {1, . . . , C, unknown},with the minimization of the expected risk [63] :EQUATION)where A is the adjacency matrix for X te . The predicted class unknown ∈ Y contains a group of novel categories, which may contain more than one class. Thus, the overall risk aims to classify known classes while also detecting the samples from unseen categories as class unknown. (EQUATIONwhere τ is a threshold. However, due to the overconfidence phenomena of deep neural networks, the output confidence of known and unknown is both high [1] . As a result, tuning a threshold that well separates known from unknown is hard and time-consuming.Differently, the approach presented in this paper redefines the paradigm of open-set graph learning from the perspective of energy models. well-designed generated unknown substitutes are introduced to aid in training of the energy model performance and elaborate loss function to improve classification performance. There are advantages of adding a class instead of optimizing a threshold as in previous methods [46, 54, 1] , since it does not require real open-set samples in validation and also eliminates the difficulty of tuning the threshold.Energy-based models [30] use the energy-function E θ which defines a density over the data x asp θ = exp(-E θ (x)) Z(θ), where θ are learnable parameters and Z(θ) = exp(-E θ )dx is the normalizing constant of the EBM. Z θ ensures that the induced density function Equation integrates to 1. However, Z θ is often hard to compute or even approximate since it is a high-dimensional integral. On the positive side, E θ can be any function E θ : R D → R placing no restrictions on the transformations compared to Normalizing Flows [68] .We can directly use the energy for OOD detection. That is, since a data point having high probability is equivalent to having low energy as p θ (x) ∝ -E θ (x). This means we are not required to estimate the normalizing constant Z(θ) in practice by considering a decision threshold τ and binary classifier G for OOD data asEQUATIONwhere 1 corresponds to IND samples and 0 to OOD samples.We can additionally employ EBMs to C-category classification problem as discriminative EBMs [13] . Suppose f θ : R D → R C is a classifier assigning logits for C classes for a data point x ∈ R D . The logits can be interpreted as unnormalized probabilities of the joint distribution p θ (x, y) = exp(f (x)[y])Z(θ), yielding the marginal distribution over x as p θ (x) = y p θ (x, y). For training, the factorization can be used, i.e.,log p θ (x, y) = log p θ (x) + log p θ (y|x), and a standard Cross Entropy loss [2] is usually employed to optimize p θ (y|x).To sum up, the connection between Energy-Based Model and a discriminative neural classifier is established by defining the energy function as the negation of the predicted logit value: E(x, y) = -h(x) [y] . Moreover, the energy function E(x) can be computed for any given inputEQUATIONNevertheless, to the best of our knowledge, existing research on energy-based modeling as well as its application is mostly concentrated on OOD detection, where the primary emphasis is on i.i.d. inputs while the power of modeling inter-dependent data has remained further exploration. Besides, there have been relatively few attempts to apply energy-based models in open-world scenarios, and this is an endeavor we will also pursue. 
methodology: 
 We propose a unified open-set node classification framework based on energy scores and generated substitute unknowns. The energy scores mitigate a critical problem of softmax confidence with arbitrarily high values for OOD examples [18] and the differences of energies between in-distribution and out-of-distribution allow effective differentiation.Specifically, a GNN encoder f θ1 takes node features X and adjacency matrix A as input, aggregates the neighborhood information and outputs representation h k i for each node v i in its k-th layer, k = 1, . . . , K, and K is the total number of layers. Thus, in the k-th layer, for node v i , the GNN encoder aggregates neighbor information from the k-1 layer into a neighborhood representation:EQUATIONwhere h k i ∈ R d k is the hidden representation at the k-th layer, N i is the neighborhood of node v i and h 0 = X.Meanwhile, we mimic novel patterns by generating substitute unknown nodes through manifold mixup [51, 17] . Targeted node pairs are selected and mixed up at the middle layer of the GNN, and two kinds of substitutes are created: inter-class unknown substitutes that separate known classes from each other and external unknown substitutes that separate known from unknown. Generated substitutes and original known class nodes are input into the remaining layers together to learn discriminative representations.Then, a classifier f θ2 is learned to classify the nodes into C + 1 classes according to their energy scores. The energy scores of node v i is obtained through energy propagation upon the graph topology, i.e.,EQUATIONwhereE 0 i = -log y ′ e f θ 2 (h K i )[y ′ ] is the initial energy score of v i .To conclude, the proposed EGonc method consists of three main modules: 1) a substitute node generation module to create substitute unknown class nodes; 2) a energy propagation module to obtain energy score of each node; and 3) an open-set classifier learning module to guarantee the classification of known classes and the rejection of the unknown class. We introduce them in details.As mentioned previously, insouciantly selected unrelated data does not really help [66] the open-set classification. Thus, we try to generate substitute unknown nodes strategically positioned at class boundaries. These nodes play a crucial role in the learning process by providing supervision and distinguishing between known and unknown nodes. Specifically, we generate two types of substitutes, inter-class unknown substitutes and external unknown substitutes, through manifold mixup [51, 17] .For a well-trained classifier, nodes with common categories tend to have similar characteristics, while those from different classes often have distinct ones. Ideally, well-defined boundaries separate the different classes. Considering that nodes close to these boundaries may have less representative features for their own classes, we chose to generate substitute unknowns using nodes close to these boundaries regions. As a result, samples belonging to different classes, but located in close proximity, become optimal candidates for substitute generation. i.e., pairs of nodes in distinct classes connected by an edge. To be specifical, given two connected nodes from different categories, denoted as {(x i , y i ), (x j , y j )}, where y i ̸ = y j and a ij = 1, their embeddings in the k-th layer are h k i and h k j , derived by inputting the graph into the network f 1∼k θ1 . The inter-class unknown substitute (x i , ỹi ) is obtained as follows:EQUATIONWhere α ∈ [0, 1] is randomly sampled from a Beta distribution, typically centred around 0.5. ỹi = C + 1 denotes the categories index of xi , indicating it belongs to the unknown class. Two edges between (x i , xi ) and (x j , xi ) are also introduced into the graph. We refer to the set of generated inter-class samples as X int . The purpose of X int is to simulate the distribution of unknown classes nodes existing between known classes. The mixed hidden representation xi is then passed to f k+1∼K θ1 to obtain the final representation hK i = f K θ1 (x i ). In addition to the inter-class unknown substitutes, we also generate external unknown substitutes to reflect the unknown distributions at the periphery. We use peripheral nodes of each known category along with their respective class center to generate these external unknown substitutes. The first kind of peripheral nodes are leaf nodes that belong to known classes, denoted as x i ∈ X tr , s.t. j A i,j = 1. The second kind are nodes with low classification confidence, namely those nodes with the top T least confident scores within each known classes. In this way, it allows us to identify both structuredbased and semantic-based peripheral nodes. We denote the set of peripheral nodes as X per . Then, we obtain the embedding of class centers in the k-th layer based on the ground-truth annotations, i.e.,h k (c) = 1 |X c | xi∈X c h k i (θ 1 ; x i , A), c = 1, . . . , C, where X c represents the nodes labeled with the class c. And we can use the manifold mixup on peripheral nodes and their respective inverted class centers {(x i , y i ), (x (yi) , y i ), x i ∈ X per , y i ∈ {1, . . . , C}} to derive the external unknown substitutes in the k-th layer, i.e., xi = βh k i (θ 1 ;EQUATIONwhere β > 0 and γ > 0 represent two hyperparameters used to adjust the distance and relative position between the generated samples and the existing known class samples. Furthermore, edges connecting (x i , xi ), x i ∈ X per are also included in the graph. We denote the set of generated external unknown substitutes as X ext and define unknown substitutes as X sub = X int ∪ X ext .To further capture the diverse distribution of the unknown classes. we introduced an energy-based model. Through formula E(x, y) = -h(x) [y] , we can establish a bridge between the energy function and an open-set classifier.Proposition 1. The energy score (Eq. ( 5)) can be an powerful indicator for OOD detection due to its valuable trait: the yielded energy scores for in-distribution data typically exhibit a lower tendency than those of OOD data.As an energy model on graphs, in order to fully leverage the topological structure information of the graph, we apply energy propagation to the topological structure of the graph.EQUATIONHere, Â = D -1/2 AD -1/2 is the symmetrically normalized adjacency matrix of A, and D represents the degree matrix, is greater (resp. less) than the average of the energy scores of its one-hop neighbors at the current layer, i.e., j Âij E (k-1)E (k) = E (k) i , x i ∈ D tr ∪ X sub . With 0 < ζ <j j Âij, then the updated energy score of its own yieldsE (k) i < E (k-1) i , (resp, E (k) i > E (k-1) i).Due to structural homogeneity [60, 28] , known classes samples and unknown substitutes often connect within their respective distributions, indicating a correlation. Therefore, by using this energy propagation formula, it could ensure the average energy scores of the known classes samples decrease while the average energy scores of the unknown substitutes increase.To address the diverse compositions of known and unknown categories, it is imperative to extract invariant information from both known and unknown classes. Consequently, the backbone GNN network f θ1 is shared across the C + 1 classes, facilitating the learning of class distribution and node representations.To ) , where= [w close , w sub ] ∈ R d K ×(C+1w sub ∈ R d K ×1represents the weights associated with the substitute classifier. For simplicity, the bias term is omitted in this context. Then, the integrated classifier is trained using both the original known classes samples and the generated substitute samples, i.e., D tr = D tr ∪ (X sub , Y C+1 ), X sub=Xint∪Xext , using the cross entropy loss, i.e.,EQUATIONwhere ŷi = s(f θ2 (f θ1 (x i ))) is the output vector of the open-set classifier for the input node x i through softmax s(•). l CrE (ŷ i , y i ) = -y i log ŷi is the cross entropy loss.l 1 loss primarily uses information from the ground-truth class to maximize the likelihood, it tends to overlook the influence of complementary classes (i.e., incorrect classes) [3] . Therefore, we incorporate the concept of complement entropy [3] to complement the softmax cross entropy, to neutralize the effects of complementary classes.Specifically, for the generated unknown substitutes, we minimize their inherent complementary loss, i.e., minimize the average of entropy over the C known classes, considering that the generated substitutes may be adjacent to any of the known classes. Conversely, for nodes belong to known classes, they are predominantly adjacent to the generated substitutes, i.e., the C + 1 class, we encourage the substitute classifier to assign the second-largest probability to these nodes. In light of these considerations, we introduce a tailored complement entropy loss for EGonc as follows:EQUATIONwhere ŷi \y i denotes the prediction logits after excluding the probability of its corresponding groundtruth label.l CoE = - C+1 c=1,c̸ =yi ŷi,c 1-ŷi,y i log ŷi,c 1-ŷi,y i , ∀(x i , y i ) ∈ D tr .The first term of l 2 aims to minimize the misclassification of known class nodes into other known classes. By employing l 2 loss, the open-world classifier f θ2 can accurately classify node belonging to known classes into their respective known class, while leveraging the substitute classifier to establish an appropriate boundary for distinguishing between the known and the unknown.To mitigate the impact of energy attenuation on unknown substitutes during the energy propagation, we further introduce a strict constraints via energy regularization l 3 to the model. Drawing inspiration from the Elastic Network [73, 10] , we observe a parallel in structure and purpose between regularization terms and their associated error terms. In light of this observation, we adopt a strategy that incorporates both linear loss and quadratic loss terms to penalize errors effectively. The linear error terms facilitate the generation of sparse solutions, thereby enhancing the model's robustness, while the quadratic error terms are adept at accommodating outliers and achieving a better fit to the data. By integrating these two kinds of error terms, we aim to enhance the adaptability and generalization capacity of the model.l 3 = k 1 ( xi∈Dtr σ(E ind (x i )) + xj ∈Xsub σ(E ood (x j ))) + k 2 ( xi∈Dtr σ(E ind (x i )) 2 + xj ∈Xsub σ(E ood (x j )) 2 ) (13)where σ(•) is the LeakyReLU function. k 1 , k 2 are the weights for linear and quadratic error terms, respectively. t in , t out are hyperparameters.E ind (x i ) = E(x i , A; h θ ) -t in , ∀x i ∈ D tr and E ood (x j ) = t out -E(x j , A; h θ ), ∀x j ∈ X subrepresent the energy error terms correspond to the IND data and the OOD data, respectively. Here, E (x i , A; h θ ) represents the energy value of sample x i at the final layer. In this way, the energy scores learned with the constraint of l 3 are of benefit to open-set classification, and it also mitigate the excessive attenuation of energy scores for unknown substitutes caused by energy propagation. Proposition 3. For any energy regularization l 3 that ensures that the GNN model h θ * minimizing l 3 , where t in < t out are two margin parameters, then the corresponding softmax categorical distribution p(y|x, G x ; h θ * ) also minimizes l 1 , while p(C + 1|x, G x ; h θ * ) similarly minimizes l 2 . This means l 3 is optimized in the same direction as l 1 and l 2 for ind data.Finally, the total loss of EGonc is a combination of cross entropy, the tailored complement entropy loss, and energy regularization loss, i.e.,EQUATION)where λ 2 > 0 and λ 3 > 0 are the hyperparameters to balance the losses. The algorithm of EGonc and its complexity analysis are provided in Appendix D. 
experiments: 
 Experiments were carried out to validate the performance of EGonc. Baselines. We compare EGonc with ten baselines, which can be classified to four categories.• 1) Closed-set classification methods: GCN_soft and GCN_sig. They are GCNs [28] with a softmax layer or a multiple 1-vs-rest of sigmoids layer as output layer.• 2) Open-set classification methods with thresholds: GCN_soft_τ , GCN_sig_τ , Openmax [1] , DOC [46] and OpenWGL [54] . The threshold is chosen from {0.1, 0.2, . . . , 0.9} or as the description of the original paper, to perform open-set recognition.• 3) Generative Open-set Classification methods: PROSER [70] and G 2 P xy [67] .• 4) Energy-based methods: GNNSAFE [56] .A detailed introduction of the baselines can be found in the Appendix E.1.Since real-world scenarios are complex, where seen and unseen differs in diverse tasks, we evaluate our model in terms of open-set classification from two aspects: near open-set classification, and far open-set classification under inductive and transductive learning setting, respectively, as introduced in test settings. Surprisingly, the simple thresholding approach of GCN_soft_τ achieves comparable performance with EGonc. This inspires us to combine the generative methods with discriminative methods for far open-set classification in future work.We compare variants of EGonc with respect to loss function and substitute-unknown generation strategy to demonstrate its effect. As shown in Table 2 , we firstly verify the effect of each loss we adopted, i.e., (C + 1)-class cross-entropy loss l 1 , tailored complement entropy loss l 2 , and energy regularization loss l 3 . The results show that these three losses are indispensable to open-set node classification. Then we verify the effect of the substitute unknown samples we generated. Specifically, we compare the performance of EGonc with respect to different kinds of (substitute) unknown samples: X far which are real far OOD samples, X rand which are substitute unknowns generated via mixup with random parent nodes; X int which are generated inter-class substitute unknowns, and X ext which are generated external substitute unknowns. From the results, we can see that EGonc generally achieves higher accuracy with the assistance of auxiliary unknown class samples. However, well-designed unknown substitutes are most beneficial for open-set node classification.The proposed model has no specific requirement on the GNN architecture for classification. The unknown-class substitute generation strategy and energy propagation take into account the topological properties of graph data. Therefore, they efficiently achieve the generation of representative unknown class samples and perform in-depth exploration of unkown-class features across different backbones. This design contributes to the model's performance in open-set classification tasks under different backbones. Table 4 ilustrates the performance of the proposed EGonc with different GNN architectures, including GCN, GAT and GraphSage. The results confirm the effectiveness and generalization ability of EGonc for open-set node classification. A Appendix / supplemental material B Related worksGraph neural networks are deep learning based models that apply neural networks to graph learning and representation [41] . They are used in a wide variety of tasks [53, 50] and have a strong theoretical foundation [39, 61] . GNNs were shown to yield state-of-the-art performance on many graph-related tasks [36, 35] and have applications in various fields such as e-commerce [32] , traffic analysis [57] , chemistry [33] and for knowledge bases [26] .The original GNN model brought the idea of combining the design of modern neural networks with graph learning [58] , e.g., Graph Convolutional Networks (GCNs) [28] , Graph Attention Networks (GATs) [50] , Graph Isomorphism Networks (GINs) [59] , and GraphSage [16] . Then various improvements have been made to the basic GNNs for different purposes [36, 35, 69, 65] .For instance, to alleviate the over-smoothing problem, decoupling mechanisms [64] and identity mapping [4] were introduced respectively. To increase the expressive power of most GNNs by the 1-Weisfeiler-Lehman (1-WL) graph isomorphism test, identity-aware Graph Neural Networks [62] were proposed.Open set recognition and classification [1, 43] require classifiers capable of categorizing known classes and detecting objects of unknown types during testing. Two main paradigms have been investigated: generative models and discriminative models. Generative models improve prediction performance by emulating the distribution of unknown class nodes, while discriminative models use a threshold to distinguish between known and unknown classes, thereby improving prediction accuracy.Prior methods, such as OpenMax [1] and DOC [46] , mainly focus on addressing the overconfidence issue of deep neural networks for unknown classes, in image and text classification problems, respectively. Specifically, on graph structured data, OpenWGL [54] employs an uncertainty loss in the form of a graph reconstruction loss [27] on unlabeled data. PGL [37] extends a previous unsupervised domain adaption framework for the open-set scenario with graph neural networks. However, it focuses on the unsupervised domain adaption and conditional shift problem. With the development of deep learning, several novel approaches have been proposed. SRNC [72] further presents a domain adaption framework for the two most common kinds of data shifts in graph structured data, i.e., open-set and closed-set data shift, by combining a graph adversarial clustering technique to shift-robust node classification. MHGL [71] extends traditional out-of-distribution (OOD) detection tasks to open-set scenarios, employing Pattern Distribution Estimator (PDE) and Multi-Hypersphere Graph Learning algorithm to model fine-grained patterns and identify previously unseen anomalous patterns. GOOD [22] proposes a weakly supervised relevance feedback approach, named Open-WRF, to mitigate sensitivity to thresholds. G 2 P xy [67] use the unknown proxies generated via mixup to efficiently anticipate the distribution of novel classes and achieve excellent prediction performance under the constraint of both cross entropy loss and complement entropy loss.Energy-based modeling as well as its application is mostly concentrated on out-of-distribution detection. Grathowohl et al. [13] found that a standard discriminative classifier of p(y|x) can be interpreted as an energy-based model (EBM) [40] for the joint distribution p(x, y [56] . GNNSAFE is a method for detecting anomilies in graph data, which can improve the robustness and generalization of GNN. Impressively, the use of unknown classes samples information to improve EBM performance is discussed in GNNSAFE.C.1 Proof for Proposition 1The gradient of l 1 is given by∂l 1 ∂θ = i∈Dtr - ∂h k i (θ; x i , A) [yi] ∂θ + C+1 c=1 ∂h k i (θ; x i , A) [c] ∂θ e h k i (θ;xi,A) [c] C+1 c ′ =1 e h k i (θ;xi,A) [c ′ ] = i∈Dtr ∂E (x i , A, y i ) ∂θ - C+1 c=1 ∂E (x i , A, c) ∂θ e h k i (θ;xi,A) [c] C+1 c ′ =1 e h k i (θ;xi,A) [c ′ ] = (1 -p (y = y i | x i , A)) ∂E (x i , A, y i ; h θ ) ∂θ - c̸ =yi p (y = c | x i , A) ∂E (x i , A, c; h θ ) ∂θ .(15) The gradient of l 2 is given byEQUATION)wherep ′ (y = j | x i , A) = e h k i (θ;x i ,A) [j] C+1 c ′ ̸ =y i e h k i (θ;x i ,A) [c ′ ]is an approximate probability distribution established on the complementary classesIn the Eq. ( 15) and Eq. ( 16), We computed the gradient with respect to the parameters. The information presented in the preceding equation indicates that the training process for D tr , which aims to minimize the first-order gradient of l 1 + λ 2 l 2 , leads to a reduction in the energy score. In a broader context, the energy output generated by the trained model tends to decrease for any instance originating from the distribution D tr . Besides, for the unknown substitutes X sub we constructed, we will adjust their energy scores by the l 3 to keep the energy model in an optimal state for fulfilling its function.We can demonstrate the proposition 2 when considering j Âij E (k-1)j j Âij < E (k-1) i. We need to rewrite the Eq. ( 10) from the perspective of a single vector:EQUATIONA similar prove can be applied to establish the contrary outcome in the case j aij E (k-1)EQUATION)At first, we define E(x; h θ * ) as the energy scores yielded by the model h θ * minimizing the energy regularization loss l 3 . Then, h θ + 1 and h θ + 2 denotes the model yielding the optimal predictive softmax distribution that minimizing the cross entropy loss l 1 and the model yielding the optimal predictive softmax distribution that minimizing the tailored complement entropy loss l 2 . i.e.,EQUATIONEQUATIONAt the same time, considering that E(x, G x ; h θ ) = -log C+1 c=1 e h θ (x,Gx) . Therefore we start from this equation and we haveEQUATION)The above equation implies an equivalence relationship between E(x, G x , h θ * ) and E(x, G x ; h θ * ) - Gx) , we can further demonstrate that the predictive softmax probability distribution obtained fromE(x, G x ; h θ + 1 ) -log C+1 c=1 e h θ + 1 (x,Gx ) . Meanwhile, continuing to consider that E(x, G x ; h θ * ) = -log C+1 c=1 e h θ * (x,EQUATION)The above predictive softmax probability distribution exactly minimizes l 1 as defined by Eq (19) . We thus have proven that the optimal energy that minimizes l 3 intrinsically induce the predictive softmax distribution that minimizes l 1 for D tr .Similarly, when we consider the relationship between the l 2 and l 3 , we can start again from Gx) and getE(x, G x ; h θ ) = -log C+1 c=1 e h θ (x,EQUATION)The above equation implies an equivalence relationship between E(x, G x , h θ * ) and E(x,G x ; h θ * ) - E(x, G x ; h θ + 2 ) -log C+1 c=1 e h θ +2 (x,Gx ) . We thus can further demonstrate that the predictive softmax probability distribution with respect to class C + 1 obtained fromEQUATIONConsidering that the energy scores are not computed for a specific class of labels, but rather are computed considering all classes, and the loss l 3 tends to reduce the energy scores for x ∈ D tr . Meanwhile, combine this with the fact that the loss l 2 tends to drive a higher probability of output for class C, it is reasonable to assume that the probability of this probability distribution is not overly large for class y. Then, The former term can be regarded as a coefficient modifying the latter term. We thus have proven that the optimal energy that minimizes l 3 intrinsically induce the predictive softmax distribution that minimizes l 2 to a certain extent for D tr .The algorithm of EGonc is illustrated in Algorithm 1. The given graph data are inputted into a GNN model (Line 1: Obtain the inter-class node pairs {(x i , y i ), (x j , y j )} ∈ D tr s.t. y i ̸ = y j &a ij = 1 2: Obtain the peripheral nodes that are leaf nodes and low confident nodes. For the first m = 1, . . . k layer:h m i = f m (θ1; h m-1 i , h m-1 j , j ∈ Ni), ∀xi ∈ Dtr 5:At the k-th layer:Create unknown substitutes X sub using Eq. ( 8) & ( 9) Augment the substitutes to known class samples:D tr = D tr ∪ (X sub , Y C+1 ) 6:For the m = k + 1, . . . , k 1 -1 layers:h m i = f m (θ1; h m-1 i , h m-1 j , j ∈ Ni), ∀xi ∈ Dtr 7:For the m = k 1 , . . . , K layers:h m i = f m (θ1; h m-1 i , h m-1 j , j ∈ Ni), ∀xi ∈ Dtr E m i = f m (E m-1 i , E m-1 j , j ∈ Ni), ∀xi ∈ Dtr 8:For open-set classifier layer:Obtain cross entropy loss as Eq. ( 11) Obtain complement entropy loss as Eq. ( 12) Obtain energy regularization loss as Eq. ( 13) 9:Back-propagate loss gradient using Eq. ( 14) and update weights 10:if early stopping condition is satisfied then end if 13: end whileDetailed training process of EGonc is illustrated in Algorithm 1. In terms of complexity, compared with normal closed-set node classification, our extra computation comes from three parts: 1) unknown substitute generation, which has linear complexity with the quantity of edges and the quantity of generated substitutes, i.e., O(|E| + |X sub |); 2) energy scores calculation, which has linear complexity with the quantity of the known class samples after augmentation, i.e., O(D tr ) 3) substitute classifier training, the complexity of this part depends on the adopted backbone GNN and the quantity of generated substitutes. Assume O(|f θ |) be the time GNN f θ1 and classifier f θ2 takes to compute a single node embedding and make a prediction, the complexity of substitute classifier training isO(|X sub | • |f θ |).Overall, the extra complexity is linear with the number of edges in the original graph and the number of generated substitutes. Experimental results on parameter analysis of generated substitutes quantity (see Fig. 1 ) show that a relatively small number of substitutes can achieve quite good performance. Thus, the extra complexity is reasonable for open-set node classification.• GCN_soft, GCN_sig : GCN [28] is adopted for graph learning. We constructed four baselines using GCN, where GCN_soft and GCN_sig use softmax and sigmoid layers, respectively, as the final layer for training and classification. Here we adopt GCN with sigmod layer as baselines since softmax is observed to be overconfident to unknown class samples. • GCN_soft_τ , GCN_sig_τ : Additionally, GCN_soft_τ and GCN_sig_τ perform classification based on these two model using fixed thresholds chosen in {0.1, 0.2, . . . , 0.9}. • Openmax: Openmax [1] is an open-set recognition model based on "activation vectors", i.e., penultimate layer of the network.By modeling the distance from "mean activation vector" using the extreme value distribution, softmax scores are calibrated for each class and updated to Openmax scores, which are then used for open-set classification. GCN is further combined with Openmax for open-set node classification on graph data. • DOC: DOC [46] is an open-world classification method for text classification. It uses multiple 1-vs-rest of sigmoids as the final output layer and defines an automatic threshold setting mechanism. GCN is combined with DOC to allow comparison with the proposed model. GCN is used to obtain the node representations and DOC is used to do open-set classification and rejection. • PROSER: PROSER [70] is a novel and highly effective open-set recognition method used in the field of image processing. We paired it with GCN for graph-based comparison. • OpenWGL: OpenWGL [54] employs an uncertainty loss in the form of graph reconstruction loss on unlabeled data and using an adaptive threshold to detect the unknown class samples. • GNNSAFE : GNNSAFE [56] proposes an effective OOD discriminator based on an energy function derived from graph neural networks (GNNs) trained with standard classification loss. We apply it from the OOD detection domain to the open-set classification domain. The GCN is configured with two hidden GCN layers in the dimension of 512 and 128, followed by an additional multilayer perceptron layer of size 64. EGonc is implemented with PyTorch and the networks are optimized using stochastic gradient descent with a learning rate of 1e -3 . The balance parameters λ 1 , λ 2 and λ 3 are chosen by a grid search in the interval from 10 -2 to 10 2 with a step size of 10 1 .The baseline methods are evaluated according to the instructions reported in the original papers with the same parameter configuration unless otherwise specified, and the best results are selected.For each experiment, the baselines and the proposed method were applied using the same training, validation, and testing datasets. The hyperparameters were tuned to get the best performance on the validation set. All the experiments were conducted on a workstation equipped with an Intel(R) Xeon(R) Gold 6226R CPU and an Nvidia A100.We show the detailed classification accuracy in terms of known classes (IND classes) and unknown classes (OOD classes) respectively, in Table 6 . The experiment was conducted under the same setting of Table 1 . It shows that in order to gain the ability of unknown class detection, there is a slight decrease in the performance of known class classification, i.e. from 79.5% to 76.3% on average, comparing EGonc to closed-set classification method GCN_soft. However, the unknown class detection accuracy is improved from 0% to 76.1% on average, which is remarkable. Compared to other open-set classification methods, such as the globally second best method, G 2 P xy, the average performance of unknown-class node detection is improved from 70.7% to 76.1% while the performance of the known-class classification experienced a slight decrease from 78.6% to 76.3% on average. When compared with the globally third best method, PROSER, the average performance of unknown-class node detection is improved from 63.6% to 76.1% while the performance of the known-class classification still gain a certain increase from 73.0% to 76.3% on average. The substitute unknown node generation module of EGonc is crucial and the quantity of generated substitutes need to be predefined. We assess the performance of EGonc with different numbers of It is observed that both accuracy and macro-F1 are stable while the quantity of generated substitutes varies a lot. Thus, a relatively small number of substitutes can be used for better efficiency.E.8 Parameter learning on weights of losses.In ablation study, we verify that the cross-entropy loss (Eq. ( 11)), tailored complement entropy loss (Eq. ( 12)), and energy regularization loss (Eq. ( 13)) are all indispensable. Here we further assess the performance of EGonc with various weights of these three losses, i.e., λ 1 , λ 2 , and λ 3 , with the range of {0.01, 0.1, 1, 10, 100}. The results are shown in Fig. 2 . We can see that it is better to choose small values for the weights generally. And the wights for tailored complement entropy loss (λ 2 ) are relatively large compared to the weights of other two losses (λ 1 , λ 3 ). For example, the best value for λ 2 is 1 for Cora, Citeseer, Dblp and 10 for Pubmed while the best values for λ Justification: [TODO]We will provide access to the data and code, along with sufficient instructions to reproduce the main experimental results when the paper is accepted.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: [TODO]Yes, please refer to E.3 in the appendix and test settings in the experiment section.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: [TODO]The paper is foundational research, does not involve societal impact, and does not discuss societal impact. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: [TODO]The article poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: [TODO]The article specifies the sources of the referenced assets as much as possible.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: [TODO]We will strive to convey these details using structured templates as much as possible, but due to time constraints, they may not be comprehensive.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: [TODO]The article does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: [TODO]The article does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.https://graphsandnetworks.com/the-cora-dataset/https://networkrepository.com/citeseer.phphttps://dblp.uni-trier.de/xml/https://pubmed.ncbi.nlm.nih.gov/download/https://github.com/snap-stanford/ogb/ 
 
 Appendix:  


Decision: This paper proposes a new energy-based open-set node classification method for open-set learning tasks. Initially, the paper received borderline scores. During the rebuttal phase, the authors successfully addressed most concerns, leading to a consensus to accept the paper.

Please follow the reviewers' suggestions to integrate the rebuttal content into the original paper.

Reviews: review: paper_summary: paper_summary: In this paper, the authors proposes a new generative open set node classification method (EGonc) to address the challenge of Open Set Classification (OSC) for safely deploying machine learning models in an open world. Traditional softmax-based methods are overly confident on unseen data, making them vulnerable to out-of-distribution (OOD) samples. EGonc uses graph structure to generate substitute unknowns simulating real open set samples and employs Energy-Based Models (EBM) for density estimation. The method learns additional energy logits from feature residuals to indicate OOD-ness, ensuring a distinguishable margin between in-distribution (IND) and OOD detection scores. Experimental results demonstrate the superiority of EGonc.
strength_weakness: strength_weakness: 1. The motivation primarily focuses on the limitations of softmax-based neural networks without delving deeply into the broader implications and potential impact of improved open set classification across various domains and real-world applications.
2. The advantages of the proposed method could be strengthened by providing more concrete examples of practical scenarios where current OSC methods fail and how EGonc specifically addresses these failures, making a stronger case for its real-world applicability and importance.
3. While the method shows promise, its applicability and performance in domains outside of those tested in the experiments need further validation.
questions: questions: 1. Why were these datasets chosen, and what advantages do they offer in validating the proposed method?
2. Why was the 2017 GCN by Kipf and Welling chosen as the backbone neural network for experimental evaluation, instead of using other models?
limitations: limitations: See the weaknesses.
paper_summary: paper_summary: This paper focuses on energy-based open-set node classification, and adopted a generative method to obtain the explicit specific score of a node belonging to the ‘unknown‘ class. To generate good substitute unknowns, it adopted energy-based models to estimate the density of classes and guarantee the nice theoretical properties of the proposed method with an overall distinguishable margin between the detection scores from IND and OOD samples. Extensive experiments are conducted and show superior performance.
strength_weakness: strength_weakness: (1)	In the main part of the paper, some important experiment settings are missing, which makes the readers feel confused about the motivations of the experiments.

(2)	What are the differences between near open-set classification and far open-set classification? Why do they matter? If a method can achieve good far open-set classification, it should also be good at near open-set classification?

(3)	For section 3, the model design is quite complex, the illustration should be more straightward and make each term and each symbol has enough explanation.

(4)	Some sentences should be improved to make them clearer, such as line 271-273: “Inspired by the Elastic Network Zou and Hastie [2005], Friedman et al. [2010], and considering the similarity in form and function between regularization terms and corresponding error terms…”
questions: questions: See above.
limitations: limitations: Yes
paper_summary: paper_summary: This paper proposed a new energy-based generative open-set node classification method to achieve open-set graph learning. It uses energy-based models to estimate the underlying density of the seen classes and to evaluate the probability of a given input belonging to the IND classes or OOD classes. Besides, it mimics the distribution of real open-set classes by generating substitute unknown samples under the guidance of graph structure. Besides, the proposed method has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for IND and OOD samples.
strength_weakness: strength_weakness: 1) What is difference between open-set node classification and out-of-distribution detection problem? The authors should illustrate the differences and whether the proposed method can solve these two problems simultaneously.
2) In section 3.1, it claims that the insouciantly selected unrelated data does not help the open-set classification, thus substitute unknown nodes near the class boundaries are generated. Why randomly selected data does not help? Aren’t they real-world open-set data which can help the open-set learning?
3) It is a good idea to imitating data from open-set distribution by generating pseudo out-of-distribution samples, however the diversity of the generated data is essential. The method used in the paper is manifold mixup, does this method can ensure the diversity of the generated data?
questions: questions: See Weaknesses part.
limitations: limitations: See Weaknesses part.
paper_summary: paper_summary: This paper introduces a new method EGonc for open-set node classification in graphs. EGonc performs open-set classification by incorporating energy-based models into the graph learning diagram. The training of the model only requires in-distribution data, while out-of-distribution training data are generated synthetically. In the experiments, the authors compare EGonc against a wide range of baseline methods on several benchmarks and provide ablation studies.
strength_weakness: strength_weakness: 1. An important claim made in the paper is not supported. The paper claims that the proposed method "has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for IND and OOD samples", but there is no theoretical analysis supporting this claim. 

2. The experiments are not sufficient. First, using one class as the out-of-distribution (OOD) class is very different from the real-world scenario where OOD examples can be quite diverse. Second, the paper does not compare against several important OOD detection baselines[1][2]. The proposed method has a negative impact on the in-distribution accuracy, while the methods in [1][2] will not. Third, the error bars are not reported. Fourth, an ablation study for energy propagation is missing, which seems to be an important component of the method.

3. The presentation needs major improvement. First, there are a lot of format issues (e.g., missing reference in line 127, texts in the equations are not corrected formatted). Second, the intuitions behind $l_2$ and $l_3$ are not well-explained. It is not clear what are the roles of the two losses. It would be more helpful to use examples to show the intent of the two losses. Same for equation 8 and 9. Third, Proposition 1 uses vague descriptions, which is not valid for a mathematical statement. 

[1] Liu, Weitang, et al. "Energy-based out-of-distribution detection." Advances in neural information processing systems 33 (2020): 21464-21475.

[2] Sun, Yiyou, et al. "Out-of-distribution detection with deep nearest neighbors." International Conference on Machine Learning. PMLR, 2022.
questions: questions: 1. What's the intuition behind the unknown substitute generation formula (equations 8 and 9)? What is the intuition of including $l_2$ and $l_3$ into the learning objective? Can you use examples to explain them?
2. In line 127, the second kind of substitute is generated from nodes with low classification confidence. Would that also be affected by the over-confident problem?
3. In equation 14, how the $\lambda$'s are selected?
4. I suggest moving line 155-160 (the introduction of GNN) to the background section.
limitations: limitations: Yes

Comments: comment: Thank you so much for your valuable comments and suggestions on our paper. We appreciate your expertise and careful consideration. And if we have addressed your major concerns, we would like to kindly request if you could increase your score of our paper. We will improve the manuscript further as you suggested. Thanks so much.
Thank you so much for your valuable comments and suggestions on our manuscript. We appreciate your expertise and careful consideration. We will improve the manuscript further as you suggested. Thanks again.
Thank you so much for your valuable comments and suggestions on our manuscript. We appreciate your expertise and careful consideration.  And we will improve the presentation further and enhance the clarity of our work. Thanks again.
Thank you for the response, which solves most of my concerns, and I will maintain my score on this paper.
Thank you for your detailed response to the concerns raised. I appreciate the effort you've put into addressing each point.

After reviewing your explanations, I will increase my score, as many of my concerns have been addressed. However, I hope the presentation can be improved further to enhance the clarity of your work.
Thank you for your detailed response. I will keep my positive rating.