Title: Revisiting Score Propagation in Graph Out-of-Distribution Detection
Venue: NeurIPS 2024
Accepted: yes
Keywords: Graph Neural Network

Out-of-distribution Detection
Abstract: The field of graph learning has been substantially advanced by the development of deep learning models, in particular graph neural networks. However, one salient yet largely under-explored challenge is detecting Out-of-Distribution (OOD) nodes on graphs. Prevailing OOD detection techniques developed in other domains like computer vision, do not cater to the interconnected nature of graphs. This work aims to fill this gap by exploring the potential of a simple yet effective method -- OOD score propagation, which propagates OOD scores among neighboring nodes along the graph structure. This post hoc solution can be easily integrated with existing OOD scoring functions, showcasing its excellent flexibility and effectiveness in most scenarios. However, the conditions under which score propagation proves beneficial remain not fully elucidated. Our study meticulously derives these conditions and, inspired by this discovery, introduces an innovative edge augmentation strategy with theoretical guarantee. Empirical evaluations affirm the superiority of our proposed method, outperforming strong OOD detection baselines in various scenarios and settings.

Revisiting Score Propagation in Graph Out-of-Distribution Detection

Abstract

The field of graph learning has been substantially advanced by the development of deep learning models, in particular graph neural networks. However, one salient yet largely under-explored challenge is detecting Out-of-Distribution (OOD) nodes on graphs. Prevailing OOD detection techniques developed in other domains like computer vision, do not cater to the interconnected nature of graphs. This work aims to fill this gap by exploring the potential of a simple yet effective method OOD score propagation, which propagates OOD scores among neighboring nodes along the graph structure. This post hoc solution can be easily integrated with existing OOD scoring functions, showcasing its excellent flexibility and effectiveness in most scenarios. However, the conditions under which score propagation proves beneficial remain not fully elucidated. Our study meticulously derives these conditions and, inspired by this discovery, introduces an innovative edge augmentation strategy with theoretical guarantee. Empirical evaluations affirm the superiority of our proposed method, outperforming strong OOD detection baselines in various scenarios and settings. To ensure reproducibility, we have made our code and relevant data publicly available at https://github.com/longfei-ma/GRASP .

1 Introduction

Graph-like data structures are ubiquitous in many domains, such as social networks [87, 40], molecular chemistry [19, 82], and recommendation systems [84, 45]. As graph neural networks increasingly serve as powerful tools for navigating this complex data landscape, a compelling yet under-explored issue emerges: Out-of-Distribution (OOD) node detection. Imagine a recommender system suggesting irrelevant or even harmful products to users, or a bioinformatics algorithm misusing an unknown protein. This gives rise to the importance of OOD detection in graph data, which determines whether an input is in-distribution (ID) or OOD and enables the model to take precautions.

While existing OOD detection methods have shown promising results in computer vision [66, 27, 18, 15, 95], natural language procession [11, 57] and tabular data analytics [68], their effectiveness diminishes when applied to graph data [77]. These conventional techniques operate under the assumption that data points are independently sampled, which misaligns with the interconnected nature of graphs. To better leverage the structural knowledge from the graph, OOD score propagation [77] has been employed to enhance graph OOD detection performance by directly propagating the computed OOD scores along the graph structure (as shown in Figure 1). Although this strategy has shown promising results on some datasets,

Raw

OOD Score

g

1

+

g

2

+

g

3

+

g

4

4

g

1

g

2

g

3

g

4

g

1

+

g

2

2

Post-

propagation

OOD Score

Propagation

ID

OOD

…

<latexi sh

1_b

64="OLNDnwmSp/

72Q

od

Z

y

C

0

>A

c

V

F

r

fq

E

u

T

g

W

R

J

9

k

3

5

8

P

vI

z

K

H

U

MY

j

G

X

<latexi sh

1_b

64="m

PU

NS

YX/D5Qf

Z

G

k

>A

B

n

c

V

8

EJ3

r

q

9L

0

M

F

7

oWy2

Tp

I

Cj

z

u

+O

d

g

H

w

R

v

K

<latexi sh

1_b

64="q

JO

HG

MKd

F/mY

L5RI2

>A

C

n

c

VD

S

N

3

W

r

k

B

g

pQ

0

j

ZT

f+

X

9z

7

w

y

8

v

E

o

U

P

u

<latexi sh

1_b

64="d5Sy3wV

uY

28MJ

o

k

TIf

0

>A

B

n

c

N

E

Ur q/

9L

K

F

p

Q

+

z

O

D

7

m

G

Z

j

g

W

X

v

C

P

R

H

<latexi sh

1_b

64="BHA

qfS

C

78/o dKFYn

OE

>

c

V

N

9L

0m

M

Q

Wy2

TpZ

I

P

j

kzX

w

G

u

D

v

+

R

5

g

<latexi sh

1_b

64="RD

FwI

Gp3

Of

7C

HKy802U

>A

B

n

c

V

NS

EJ

r

q/

9L

mk

M

Q

oW

T

Z

P

j

zX

Y

u

+

d

g

5

1. Shall we propagate?

2. How to improve it?

1

0

1

1

0

0

0.33

0.67

0.33

0.33

0.67

0.67

1

0

1

0

0

1

0.5

0.4

0.5

0.4

0.4

0.33

Error Rate: 0

Error Rate: 100%

Error Rate: 33%

Error Rate: 16%

Score

Distribution

Score

Distribution

(b) The case when propagation is helpful.

(a) The case when propagation is harmful.

Worse

Better

Score Propagation (with More Inter-edges)

Score Propagation (with More Intra-edges)

Inter-edges

Intra-edges

ID

Node

OOD

Node

Annotation

the reasons behind its effectiveness and the conditions under which it works are not clear. To dive deep into it, our research embarks on addressing two research questions related to OOD score propagation:

Question 1: "Will naive OOD score propagation always help graph OOD Detection?" The short answer is no. This method can be ineffective on graphs where inter-edges ( ID-to-OOD ) are predominant. Using the examplse in Figure 2(a), where only inter-edges exist, prior to conducting score propagation, all ID and OOD nodes can be fully distinguished. However, after performing score propagation along these edges, the ID and OOD nodes are completely misclassified. Conversely, after adding intra-edges and making them donimate, as shown in Figure 2(b), score propagation would be beneficial to distinguish ID and OOD nodes. These two examples intuitively illustrate how the ratio of intra-edges and inter-edges can impact the effectiveness of OOD score propagation. We substantiate this intuition in Section 3. This finding naturally paves the way for subsequent questions.

Question 2: "How to derive a better score propagation strategy for graph OOD detection?" Building on our prior findings, we propose a graph augmentation strategy as presented in Section 4. Specifically, our strategy selects a subset G of the training set and puts additional edges to the nodes within G . Beyond its practical implications, our solution is theoretically supported: When G predominantly connects to ID data over OOD data, our strategy can provably enhance the post-propagation OOD detection outcomes.

We summarize our contributions as below:

Theoretical understanding: We delve deeply into the mechanism of score propagation to understand its potential for graph OOD detection and elucidate the conditions under which it thrives, providing an understanding that extends beyond existing knowledge.

Practical solution: To counter the identified challenge of inter-edges' domination, we propose GR aphA ugmented S core P ropagation (GRASP), an innovative edge augmentation strategy with theoretical guarantee. By strategically adding edges to a chosen subset G of the training set, as detailed in Section 4, our method aims to enhance the intra-edge ratio, thereby boosting OOD detection outcomes post-propagation.

Empirical studies : We demonstrate the superior performance of the proposed method on extensive graph OOD detection benchmarks, different pre-trained methodologies [34, 69, 7, 94], and different OOD scoring functions. Under the same condition, our proposed strategy substantially reduces the FPR95 by 17.87 %and 32.21 %compared to the strongest graph OOD detection baselines on common and large-scale benchmarks respectively. Comprehensive analyses are also provided to validate the effectiveness of the proposed approach and the correctness of the theoretical findings.

2 Preliminaries

Problem setup . We consider a traditional semi-supervised node classification setting with the additional unlabeled nodes from the out-of-distribution class. Let G = {V , E} denote the graph with nodes V and edges E , where the node set V with size N are attributed with data matrix X ∈ R N × d . The structure of graph G is described by the adjacency matrix A ∈ { 0 , 1 } N × N . We let the corresponding row-stochastic matrices as ¯ A = D -1 A , where D is the diagonal matrix with

D ii = ∑ j A ij . The N nodes are partially labeled, so we let V l and V u represent the labeled and unlabeled node sets respectively, i.e, V = V l ∪V u . Given a training set D tr = { ( x i , y i ) } i ∈V l with x i as the i -th row of X and y i ∈ Y ≜ { 1 , · · · , C } , the goal of node classification is to learn a mapping f : V → R C from the nodes to the probability of each class.

This can be achieved by having an OOD detector, in tandem with the node classification model f . OOD detection can be formulated as a binary classification problem. At test time, the goal of OOD detection is to decide whether an unlabeled node i ∈ V u is from ID or OOD. The decision can be made via a level set estimation:

Out-of-distribution detection. When deploying a model in the real world, a reliable classifier should not only accurately classify known in-distribution (ID) nodes, but also identify 'unknown' nodes or OOD nodes. Formally, we can represent the unlabeled node set by V u = V uid ∪ V uood where V uid and V uood represent the in-distribution (ID) node and OOD node respectively. The goal of the graph OODdetection is to derive an algorithm to decide if a node i ∈ V u is from V uood or V uid .

where nodes with higher scores g ( x i ) are classified as ID and vice versa, and λ is the threshold commonly chosen so that a high fraction (e.g., 95%) of ID data is correctly classified.

In this paper, we consider post hoc OOD detection methods to produce g ( x i ) which does not require expensive re-training. As an example, a classical way to compute g ( x i ) is Maximum Softmax Probability (MSP) [23] which is given by the maximum softmax value. We include details of the considered OOD detection methods in Appendix B.

3 Will propagation always help Graph OOD Detection?

In the introduction, we delineate the limitations of OOD score propagation using a concrete example and elucidate the intuition that it may fail when inter-edges dominate. In this section, we formally delineate the conditions under which OOD score propagation works. We start by showing the formal definition of propagation.

Define OOD scoring propagation . Given a raw OOD scoring vector ˆ g ∈ R N with ˆ g i = g ( x i ) , the propagated scoring vector is given by:

where k ∈ N + are hyperparameters.

Is it necessarily the case that g outperforms ˆ g ? The answer is NO . We elucidate with the theoretical insight below.

Theoretical Insight. As discussed in the Introduction from Figure 2, when the number of ID-to-ID and OOD-to-OOD edges surpasses that of ID-to-OOD edges, the propagation mechanism tends to 'aggregate" the scores associated with the ID and OOD nodes respectively, which further amplify the separability between them. Conversely, when the number of ID-to-OOD edges are more than the other types of edges, the scores for both ID and OOD nodes become undistinguishable post-propagation.

The example above offers the insight that the relative performance of g compared to ˆ g is contingent upon the structural dynamics of the network, specifically the distribution of edges. To formally articulate this relationship, we adopt a probabilistic framework for modeling edges. Specifically, we assume that the edge follows a Bernoulli distribution characterized by parameters η intra and η inter for intra-edges ( ID-to-ID and OOD-to-OOD ) and inter-edges ( ID-to-OOD ), respectively:

In the context of probabilistic modeling, the subsequent Theorem 3.1 can be established to formalize the inherent understanding.

Theorem 3.1. (Informal) (a) When η intra ≫ η inter , it is highly likely that the propagation algorithm will yield enhanced performance in OOD detection. (b) When η intra ≈ η inter or even η intra < η inter , the score propagation is likely to be either ineffective or detrimental to the performance.

We also provide the formal version below (Theorem 3.2) which provides a mathematical foundation for understanding how varying the Bernoulli parameters influence the efficacy of the propagation in the context of OOD detection. We provide the detailed proof in Appendix A.

Theorem 3.2. (Formal) For any two test ID/OOD node set S id ⊂ V uid , S ood ⊂ V uood with equal size N s , let the ID-vs-OOD separability M sep defined on an OOD scoring vector ˆ g R N as

If M sep (ˆ g ) > 0 and η intra -η inter > 1 /N s , for some ϵ > 0 and constant c , we have P sep ( A ˆ g ) sep (ˆ g ) ϵ 1 exp ( cϵ 2 ˆ g 2 ) .

Summary. This section has presented a comprehensive theoretical evidence to substantiate the claim that propagation through the adjacency matrix A does not necessarily enhance out-of-distribution (OOD) detection in graphs. Moreover, Theorem 3.2 reveals that the critical factor in enhancing post-propagation performance lies in improving the ratio of intra-edges within the graph structure. These insights serve as a direct motivation for the augmentation strategy in the next section.

4 An Augmented Score Propagation Strategy

The findings from the preceding section give rise to a subsequent thought: " Can we improve the propagation strategy for graph OOD detection performance? " In an ideal scenario, if an oracle were to indicate that a particular subset in the test set belongs exclusively to the ID or OOD, one could augment the graph by adding intra-edges or removing inter-edges. This would consequently improve the ratio of intra-edges η intra , leading to enhanced OOD detection performance post-propagation.

However, such an oracle does not exist in practical settings, and even approximating such a subset proves to be a difficult task. Existing literature has suggested the use of pseudo-labels assigned to nodes [36, 79, 2, 73, 55]. Nonetheless, these studies also caution that this approach is susceptible to 'confirmation bias", whereby errors in estimation are inadvertently amplified.

To circumvent it, this paper proposes the solution for adding edges to a subset of the training set V l , which is assured to be in-distribution data. We start by showing the theoretical underpinnings that adding such a subset can, under specified conditions, contribute to improved OOD detection performance after propagation.

4.1 Theoretical Insight

Our approach involves adding the edges to a subset G of training data and then propagating the out-of-distribution (OOD) scoring vector using the enhanced adjacency matrix. Specifically, when edges are added to G , this action can be mathematically represented as incorporating a perturbation matrix E = e G e ⊤ G into A, as demonstrated in Figure 3. Here, e S ∈ R N denotes an indicator vector for a set S ⊂ V , where the vector takes the value of 1 if the index i ∈ S and value 0 otherwise. A sufficient condition for the efficacy of this augmentation strategy in enhancing post-propagation OOD detection performance is outlined in Theorem 4.1.

Theorem 4.1. (Informal) For a subset G in the training set, augmenting G by adding edges to all its nodes can lead to improved post-propagation OOD detection performance, provided that the following condition is met: G has more edges to ID data than OOD data.

We also provide the formal version below (Theorem 4.2) that incorporates a perturbation analysis. This analysis elucidates how edge augmentation in the training set can positively influence the

<latexi sh

1_b

64="EJ8j+WL

KBY

RH

7

mqUC

S

zc

>A

X

VD

gN

Oy

r

f

9

oP

Iv

Q

k

M

/

2

Z

5p

0F

d

G

T

3

u

n

w

<latexi sh

1_b

64="S2gUnD3

5H7zQ

0Z

P

d

>A

B

c

V

N

8

EJ

r

q/

9L

G

R

K

wm

I

Cj

k

X

W

M

Y

u

+O

f

o

T

y

v

p

F

<latexi sh

1_b

64="QpDMvq uPU75GXz

0

kN

>A

B

c

V

S8

EJ3

W

O

K

R9

j

r

/

2

d

L

Cf

+

g

m

F

T

w

y

H

o

Z

<latexi sh

1_b

64="

9Nr8

TAL3Cj

Hd

WFnYE

Dv

>

B

c

V

S

J

U

q/

0

GPR

K

g

wm7k

I

Q

zX

u

M

pf

o

5

+

O

y

<latexi sh

1_b

64="

QcXF

7Y

Hj

PD8C

o9

L3

kT

>A

B

n

V

NS

EJ

Ur q/

0

G

R

y

p

+g

W

z

5uwO

m

M

v

K

f

d2

Z

I

<latexi sh

1_b

64="TG3

E

Cyn k7XR82SB/jAm

FQ

>

+

c

VDL

N

pr fUZdu

ovg

50W

M

P

w

Y

O

z

9

J

q

K

I

H

<latexi sh

1_b

64="

r

q

JF

vC

QX

o+BHf

9U

A

>

c

VDLSgN

EOy

Y

P

K

I

Twm

G

20puM

Z

d

j

8

n

k

7

W

/

3

R

5

z

<latexi sh

1_b

64="S2gUnD3

5H7zQ

0Z

P

d

>A

B

c

V

N

8

EJ

r

q/

9L

G

R

K

wm

I

Cj

k

X

W

M

Y

u

+O

f

o

T

y

v

p

F

<latexi sh

1_b

64="QpDMvq uPU75GXz

0

kN

>A

B

c

V

S8

EJ3

W

O

K

R9

j

r

/

2

d

L

Cf

+

g

m

F

T

w

y

H

o

Z

<latexi sh

1_b

64="

9Nr8

TAL3Cj

Hd

WFnYE

Dv

>

B

c

V

S

J

U

q/

0

GPR

K

g

wm7k

I

Q

zX

u

M

pf

o

5

+

O

y

<latexi sh

1_b

64="

QcXF

7Y

Hj

PD8C

o9

L3

kT

>A

B

n

V

NS

EJ

Ur q/

0

G

R

y

p

+g

W

z

5uwO

m

M

v

K

f

d2

Z

I

…

…

…

1

1

v

1

v

2

v

3

v

N

A

=

1

1

…

1

1

…

…

1

1

v

1

v

2

v

3

v

N

Add

edges to

G

=

{

v

1

, v

2

}

E

propagation algorithm's ability to enhance OOD detection. For the sake of the main intuition, we provide the analysis on A instead of ¯ A for simplicity. We provide the detailed proof in Appendix A.

Theorem 4.2. (Formal) For any two test ID/OOD node set S id ⊂ V uid , S ood ⊂ V uood with size N s , let the ID-vs-OOD separability M sep defined on a non-negative OOD scoring vector ˆ g ∈ R N as

Let E S ↔ S ′ ⊂ E to denote the edge set of edges between two node sets S and S ′ , where S, S ′ ⊂ V . If we can find a node set G ⊂ V l such that |E G ↔ S id | > |E G ↔ S ood | , we have

The Theorem 4.2 shows a critical principle for enhancing propagation: the optimal strategy entails the addition of edges to the subset G such that there are more edges to ID data than OOD data. For some S id , S ood in the test set, the goal is to find the set

where N g is a hyperparameter to control the size of G ∗ . Inspired by the optimization target, we proceed to present our pragmatic algorithmic approach.

4.2 Graph-Augmented Score Propagation (GRASP)

Our augmentation approach hinges on the selection of a subset, G , from the training set, as exemplified in Equation 2. Two principal challenges arise in implementing this: (1) We cannot directly determine the number of edges linked to ID/OOD data because these reside in the test set and their labels remain unknown. (2) An exhaustive search to find a subset is computationally expensive, as the number of combinatorial possibilities increases in a factorial manner. In this paper, we tackle these challenges by providing the practical approximation method.

Selection of Sid / Sood. Our discussion begins by detailing the methodology to select the subset from the test ID/OOD dataset, symbolized by S id and S ood in Equation 2. A straightforward approach to obtain the most likely ID is by selecting nodes with the largest confidence and the least for OOD in class predictions. Following [23], we employ the max softmax probability (MSP) as a representation of confidence. The selected sets can be defined as:

where λ α denotes the α -th percentile of the MSP scores corresponding to nodes in V u . To offer a clear view, Figure 4 portrays S id and S ood in the marginal regions highlighted in orange. Selecting a subset in the leftmost and rightmost regions reduces the error when identifying the ID/OOD subsets, given that overlapping between ID and OOD predominantly occurs around the central region of the distribution.

Selection of G . Upon establishing S id and S ood , the next step is to determine G using Equation 2. Directly enumerating every possible G is impractical. Instead, we adopt a greedy approach, prioritizing the node with the highest "likelihood" score. To elucidate, for each node i ∈ V l , the score can be

<latexi sh

1_b

64="

98E+q

/7R

p

CSNvrD

WJ

>A

B

n

c

V

3U

L

0mk

M

F

Q

o

w2

Zjf

O

H

G

Y

y

I

d

u

X

5

T

zg

K

P

<latexi sh

1_b

64="+I

J

kvKBFcZ7j

R

8

>

3

V

LS

U

2

W

du

o

g

p

y

r

M

Q

YO

f

X

P

/

n

m

5

HTz

9

G

q

w

where k ∈ N + are hyperparameters.

OOD

ID

λ

1

-

α

λ

α

S

id

S

ood

Frequency

Distribution of MSP Score in

V

u

computed as the ratio of the edge count to S id over S ood :

where we incorporate an addition of 1 in the denominator to circumvent division by zero. Subsequently, G can be expressed as:

where τ β stands for the β -th percentile of h ( i ) scores for nodes in V l . Once G is defined, edge augmentation can be executed as demonstrated in Section 4.1. The OOD score is then propagated with the new adjacency matrix A + = A + e G e ⊤ G in place:

Complexity analysis. While our algorithm introduces the fully connected matrix E , our method can be effciently implemented by matrix-vector multiplication, leading to computational footprint in terms of runtime and memory usage with O ( N +2 k |E| + n ) and O ( N + |E| + n ) respectively after propagation for k times, where n is node count of subset G . We provide the comprehensive complexity analysis in Appendix D.6.

5 Experiments

Datasets. We conduct extensive experiments using 10 real-world datasets that span diverse domains, scales, and structures (homophily or heterophily). A high-level summary of the dataset statistics is provided in Table 1, with a detailed information of the datasets and the comprehensive description of ID/OOD split in Appendix C. Specifically, Cora [61] serves as a widely recognized citation network. Amazon-Photo [52] represents a co-purchasing network on Amazon. Coauthor-CS [62] portrays a coauthor network within the realm of computer science. Moreover, Chameleon and Squirrel [59] are two notable Wikipedia networks, predominantly utilized as heterophilic graph benchmarks. We

additionally incorporate 5 large-scale graphs to evaluate our proposed methods: Reddit2 [88] and ogbn-products [24] are large homophily datasets; ArXiv-year , Snap-patents , and Wiki [44] are recently proposed large-scale heterophily benchmarks.

Remark on homophily/heterophily. In Table 1, datasets are also categorized based on the attribute of homophily, denoting the tendency of nodes with the same class to connect. Conversely, the heterophily graph demonstrates a tendency for nodes of disparate classes to connect. This characteristic not only presents a challenge for node classification but also for graph OOD detection. The underlying

<latexi sh

1_b

64="

oN

9B

EvJ

q02YC

kK

M

>A

X

c

V

LS

F

3U

g

w

RIp

r

m

Z

G

/

TdO

y

zu

H

8W5

n

P

j7

Q

f

<latexi sh

1_b

64="I

Aw/j9

uY

qyvrVBpk0oWOE

>

7X

c

NS

M

J2

+

T

CR

U

m

D

g

8

Z

P

H

5Q

K

3d

f

F

n

z

G

L

<latexi sh

1_b

64="kQKS

v

HLUzP2

dW

B

5I

Y0M

>A

9

c

VD

g

F

TX7

+q

y

u

G

o

J

m

E

/

3

R

8

O

w

n

j

Nr

f

C

p

Z

reason is that the OOD data is from different classes with ID, and heterophily exacerbates the ratio of inter-edge connections between ID and OOD, which is deemed undesirable for graph OOD detection according to Theorem 3.2.

Implementation Details. Our graph OOD detection technique operates in a post hoc fashion utilizing a pre-trained network and so can be used in various pre-trained network seamlessly. We present results evaluated on Graph Convolutional Network (GCN) [34] in the main paper to save space and put detailed results of other architectures in Appendix D.3. All pre-trained models possess a layer depth of 2. With the pre-trained network, we proceed to execute the graph OOD detection. By default, we report the performance of the augmented propagation (GRASP) on the MSP score [23]. The compatibility with other OOD scoring functions is also shown in Table 6. We set the propagation number k as 8, with percentile values α = 5 and β = 50 . The sensitivity analysis of the hyperparameters is included in Appendix D.4.

Metrics. Following the convention in literature [23, 47, 65], we use AUROC and FPR95 as evaluation metrics for OOD detection.

5.1 Main Results

GRASP consistently achieves superior performance. We provide results of 5 common small-scale benchmarks and 5 large-scale datasets in Table 2 and Table 3 respectively, wherein only the averaged results over 5 runs are presented to save space and the detailed results with standard errors of these two scale datasets are shown in Table 12 and Table 13 respectively. From the results we can see that our proposed methodology (GRASP) consistently demonstrates promising performance. The comparative analysis encompasses a broad spectrum of post hoc competitive Out-of-Distribution (OOD) detection techniques in existing literature and training-based methods tailored for graph OOD detection. We categorize the baseline methods into two groups: (a) Traditional OOD detection methods including MSP [23], Energy [47], ODIN [43], Mahalanobis [37], and KNN [67]; (b) Graph OOD detection methods including GKDE [92], GPN [64], OODGAT [63], and GNNSafe [77]. In these tables, we present GRASP results based on the MSP score. Noteworthy findings include: (a) The traditional OOD detection methods exhibit suboptimal performance in the realm of graph OOD detection. For instance, GRASP reduced the average FPR95 by 17.87 %and 32.21 %compared to the strongest traditional OOD detection method GNNSafe and Mahalanobis on common and large-scale benchmarks, respectively. This outcome is anticipated given their lack of specificity in design towards graph data. (b) GRASP outperforms existing baselines by a large margin, surpassing the best baseline GNNSafe by 17.87 %and 40 %concerning average FPR95 on two scale benchmarks respectively. These results further corroborate that the theoretically motivated solution GRASP is also appealing to use in practice.

GRASP is also competitive on large-scale graph datasets. Contrasted with the small-scale benchmarks in Table 2, the large-scale scenario in Table 3 presents more challenges due to a large number of nodes and edges. From Table 3 we can see that all baseline OOD detection methodologies exhibit suboptimal performance on large-scale benchmarks, while our method GRASP robustly performs the best.

GRASP exhibits significant advantages over training-based baselines. In addition to contrasting with post hoc methods, we extend our comparison to a parallel line of graph Out-Of-Distribution

(OOD) detection research, which focuses on refining the training strategy to improve graph OOD detection performance. The compared methods include GKDE [92], GPN [64] and OODGAT [63]. While these approaches necessitate a costly re-training procedure, they perform mediocrely across all small-scale datasets and even run out-of-memory on almost all large-scale benchmarks, rendering them impractical for real-world deployment.

GRASP is performant on challenging heterophily datasets. As indicated in Table 2 and 3, GNNSafe, which performs well on homophily datasets, experiences significant degradation on the difficult heterophily benchmarks due to its naïve propagation mechanism. In contrast, GRASP maintains optimal performance on these hard scenarios.

5.2 A Comprehensive Analysis of GRASP

Ablation study on augmentation policy. Recall that the key part of our method GRASP is the augmentation policy that consists in adding edges to the training nodes with top 50% scores of h ( i ) , which corresponds to the nodes on the right side of Figure 5. We ablate the contributions of h ( i ) by comparision with alternative augmentation approaches that utilize h ( i ) differently in Table 4, specifically, (1) selecting 50% of training nodes with the lowest h ( i ) values (left side of Figure 5), (2) randomly selecting 50% of training nodes, corresponding to randomly picking node indices from the x-axis of Figure 5, (3) directly adding edges to S id and S ood within the test set ( i.e. , TestAug ), and (4) a classic graph augmentation method named GAug [91],

∣

∣

E

{

i

}↔V

uid

∣

∣

∣

∣

E

{

i

}↔V

uood

∣

∣

0

200

Sorted training node indices

i

0

1k

which adds or removes edges based on an edge predictor that disregards h ( i ) completely. We have the following key observations:

( a ) Selection by h ( i ) is effective. For example, our strategy using the top 50% scores of h ( i ) outperforms that uses random 50%, which, in turn, outperforms the low 50% way. This is because a higher score of h ( i ) implies higher edge count towards ID data than to OOD data, which can increase the ratios of intra-edges and improve OOD detection performance after propagation. In contrast, the alternative augmentation method GAug , which does not consider the score h ( i ) at all, performs even worse than the Low 50% policy.

<latexi sh

1_b

I

V

7S

Q

LCf

Xd

HE0

>

c

B

wJ

F

2

Mv

8

To

D

k

g

NU

W

r/

+

9

O

3P

u

j

Y

z

K

p

R

G

y

<latexi sh

1_b

64="59MV

d/o

8yr

S

g

JNvQ

n

Bk

>A

CLH

c

D

F

3U

2

q

0

w

RIp

Y

m

j

f

Z

G

z

W

O

K

7

X

E

+

P

T

u

<latexi sh

1_b

64="gQ0

ZrH

Bkqf9

/

V

JU

>A

c

NS8

E

3

L

m

M

F

2

po

+

z

u

D

G

j

Y

w

y

I

d

W

X

7

v5

C

K

P

T

n

R

O

( b ) Directly adding edges to S id and S ood within the test set is sub-optimal . Specifically, employing this method leads to nearly 20 %lower than that achieved with GRASP, which substantiates the notion that 'confirmation bias" can adversely affect the graph OOD detection.

Overall, the ablation study suggests that our proposed augmentation policy is crucial to OOD detection performance.

GRASP can effectively boost performance of challenging nodes connected by inter-edges. As stated in the introduction 1, the reason OOD score propagation does not always work is the confusion between ID and OOD nodes resulting from propagation along the inter-edges. For example in heterophily datasets, where connected nodes tend to possess different labels, OOD nodes are more likely to appear on the inter-edges. To assess the capability of our proposed augmentation propagation method to address this challenge, we present in Table 5 the accuracy of detecting OOD nodes connected by intra-edges ( intra ) and inter-edges ( inter ) respectively, using the original MSP without any propagation, naive propagation based on MSP ( MSP+prop ), and our proposed augmentation propagation ( MSP+GRASP ) respectively. From the results we can see that naive propagation performs well only on strong homophily datasets, while on strong heterophily datasets (datasets highlighted in bold in the table), its performance is even worse than without propagation, as expected. In contrast, employing our augmentation method still results in substantial performance gain after propagation on these challenging datasets.

GRASP is compatible with a wide range of OOD scoring methods. In Table 6, we demonstrate the compatibility of GRASP with various alternative scoring functions. Each method generates OOD scores to form a scoring vector; GRASP is then applied to facilitate score propagation. The use of GRASP markedly surpasses the performance of its non-augmented counterpart across all datasets.

Remark on other empirical findings. We include other empirical findings in Appendix. Specifically, in Table 14, we prove that GRASP also achieves superior performance on the other GNN architectures in the literature. In Figure 7, we demonstrate a strong positive correlation between ratios of intraedges and the corresponding OOD detection performance, validating the correctness of Theorem 3.2.

We also show in Table 15 that GRASP can increase the ratio of intra-edges after augmentation, consequently boosting OOD performance. This substantiates the correctness of Theorem 4.2. What's more, we conduct a thorough accounting of computation/memory demands compared to baselines on all scale datasets in Table 16 and Table 17 respectively, which underscore the strong practicality of our approach. Lastly, in Appendix D.7, we investigate the other propagation mechanisms in the literature and their impact on graph OOD detection performance emperically.

6 Related Work

Out-of-distribution Detection. The primary focus within this realm has been on the development of scoring functions for OOD detection. These works can be broadly categorized into two main streams: (1) output-based methods [23, 43, 47, 71, 25, 72, 95, 26, 15, 90, 16, 21], and (2) featurebased methods including the Mahalanobis distance [37, 60, 56] and KNN distance [67]. These methodologies are predominantly applied in domains such as computer vision, where samples are inherently independent of each other. However, these techniques are not designed to adeptly handle data structures like graphs, where samples are inter-connected.

Out-of-distribution detection for graph data. Graph anomaly detection has a rich history [12, 74, 89, 75, 48, 14, 50, 32, 46, 51]. In recent years, the OOD detection in graph data introduced fresh challenges, particularly with multi-class classification for in-distribution data, escalating the difficulty in discerning outlier data. Some of the works focus on graph-level OOD detection [42, 49, 4, 76]. For node-level OOD detection, GKDE [92] and GPN [64] apply Bayesian Network models to estimate uncertainties to detect OOD nodes. However, Bayesian-based approaches can encounter impediments such as inaccurate predictions and high computational demands, which limit their broader applicability [83]. GNNSafe [77] emerges as the work employing post hoc energy-based score to perform OOD detection. Given the merits of post hoc methods, our study first provides a comprehensive understanding of the OOD score propagation in Graphs, extending beyond existing knowledge.

Graph Data Augmentation. Graph Data Augmentation is a common technique in graph machine learning [20, 6, 58, 29, 93, 91, 33, 53, 13, 3] to improve the node classification performance. Existing methods operate exclusively on in-distribution (ID) data. Furthermore, their test set data also originates from the in-distribution and shares the same classes as the training set. In contrast, our data augmentation is purposefully crafted for OOD detection, supported by the theoretical explanation.

7 Conclusions

In this research, we delve into an important yet under-explored challenge in the realm of graph data: Out-of-Distribution (OOD) detection. Recognizing the inadequacies of traditional OOD detection techniques in the context of graph data, our exploration centered on the potential of score propagation as a viable and efficient solution. Our findings reveal the specific conditions under which score propagation will be helpful-in situations where the ratio of intra-edges surpasses that of inter-edges. Motivated by this finding, our edge augmentation strategy selectively adds edges to a specific subset G of the training set, which provably improves post-propagation OOD detection outcomes under certain conditions. Extensive empirical evaluations reinforced the merit of our approach.

References

Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In international conference on machine learning , pages 21-29. PMLR, 2019.

Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Pseudolabeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN) , pages 1-8. IEEE, 2020.

Mehdi Azabou, Venkataramana Ganesh, Shantanu Thakoor, Chi-Heng Lin, Lakshmi Sathidevi, Ran Liu, Michal Valko, Petar Veliˇ ckovi´ c, and Eva L Dyer. Half-hop: A graph upsampling approach for slowing down message passing. 2023.

Gleb Bazhenov, Sergei Ivanov, Maxim Panov, Alexey Zaytsev, and Evgeny Burnaev. Towards ood detection in graph classification from uncertainty estimation perspective. arXiv preprint arXiv:2206.10691 , 2022.

Gleb Bazhenov, Denis Kuznedelev, Andrey Malinin, Artem Babenko, and Liudmila Prokhorenkova. Evaluating robustness and uncertainty of graph models under structural distributional shifts. Advances in Neural Information Processing Systems , 36, 2024.

Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 3438-3445, 2020.

Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International conference on machine learning , pages 1725-1735. PMLR, 2020.

Yongqiang Chen, Yatao Bian, Kaiwen Zhou, Binghui Xie, Bo Han, and James Cheng. Does invariant graph learning via environment augmentation learn invariance? Advances in Neural Information Processing Systems , 36, 2024.

Zhengyu Chen, Teng Xiao, Kun Kuang, Zheqi Lv, Min Zhang, Jinluan Yang, Chengqiang Lu, Hongxia Yang, and Fei Wu. Learning to reweight for generalizable graph neural network. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 8320-8328, 2024.

Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. International Conference on Learning Representations , 2021.

Pierre Colombo, Eduardo Dadalto, Guillaume Staerman, Nathan Noiry, and Pablo Piantanida. Beyond mahalanobis distance for textual ood detection. Advances in Neural Information Processing Systems , 35:17744-17759, 2022.

Kaize Ding, Kai Shu, Xuan Shan, Jundong Li, and Huan Liu. Cross-domain graph anomaly detection. IEEE Transactions on Neural Networks and Learning Systems , 33(6):2406-2415, 2021.

Kaize Ding, Zhe Xu, Hanghang Tong, and Huan Liu. Data augmentation for deep graph learning: A survey. ACM SIGKDD Explorations Newsletter , 24(2):61-77, 2022.

Kaize Ding, Qinghai Zhou, Hanghang Tong, and Huan Liu. Few-shot network anomaly detection via cross-network meta-learning. In Proceedings of the Web Conference 2021 , pages 2448-2456, 2021.

Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. In The Eleventh International Conference on Learning Representations , 2022.

Xuefeng Du, Yiyou Sun, and Yixuan Li. When and how does in-distribution label help out-ofdistribution detection? In Forty-first International Conference on Machine Learning , 2024.

Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds , 2019.

Ido Galil, Mohammed Dabbah, and Ran El-Yaniv. A framework for benchmarking class-out-ofdistribution detection and its application to imagenet. In The Eleventh International Conference on Learning Representations , 2022.

Johannes Gasteiger, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. In International Conference on Learning Representations , 2019.

Johannes Gasteiger, Stefan Weißenberger, and Stephan Günnemann. Diffusion improves graph learning. Advances in neural information processing systems , 32, 2019.

Soumya Suvra Ghosal, Yiyou Sun, and Yixuan Li. How to overcome curse-of-dimensionality for out-of-distribution detection? In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 19849-19857, 2024.

Shurui Gui, Meng Liu, Xiner Li, Youzhi Luo, and Shuiwang Ji. Joint learning of label and environment causal independence for graph out-of-distribution generalization. Advances in Neural Information Processing Systems , 36, 2024.

Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations , 2016.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems , 33:22118-22133, 2020.

Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. Advances in Neural Information Processing Systems , 34:677689, 2021.

Wenjian Huang, Hao Wang, Jiahao Xia, Chengyan Wang, and Jianguo Zhang. Density-driven regularization for out-of-distribution detection. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Paul F Jaeger, Carsten Tim Lüth, Lukas Klein, and Till J Bungert. A call to reflect on evaluation practices for failure detection in image classification. In The Eleventh International Conference on Learning Representations , 2022.

Tianrui Jia, Haoyang Li, Cheng Yang, Tao Tao, and Chuan Shi. Graph invariant learning with subgraph co-mixup for out-of-distribution generalization. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 8562-8570, 2024.

Wei Jin, Yao Ma, Xiaorui Liu, Xianfeng Tang, Suhang Wang, and Jiliang Tang. Graph structure learning for robust graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages 66-74, 2020.

Wei Jin, Tong Zhao, Jiayuan Ding, Yozen Liu, Jiliang Tang, and Neil Shah. Empowering graph representation learning with test-time graph transformation. In The Eleventh International Conference on Learning Representations , 2022.

Mingxuan Ju, Tong Zhao, Wenhao Yu, Neil Shah, and Yanfang Ye. Graphpatcher: Mitigating degree bias for graph neural networks via test-time augmentation. Advances in Neural Information Processing Systems , 36, 2024.

Hwan Kim, Byung Suk Lee, Won-Yong Shin, and Sungsu Lim. Graph anomaly detection with graph neural networks: Current status and challenges. IEEE Access , 2022.

Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 , 2016.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations , 2017.

Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. International Conference on Learning Representations , 2019.

Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML , volume 3, page 896. Atlanta, 2013.

Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems , 31, 2018.

Jure Leskovec, Jon Kleinberg, and Christos Faloutsos. Graphs over time: densification laws, shrinking diameters and possible explanations. In Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining , pages 177-187, 2005.

Jure Leskovec and Andrej Krevl. Snap datasets: Stanford large network dataset collection, 2014.

Chang Li and Dan Goldwasser. Encoding social information with graph convolutional networks forpolitical perspective detection in news media. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 2594-2604, 2019.

Xiner Li, Shurui Gui, Youzhi Luo, and Shuiwang Ji. Graph structure and feature extrapolation for out-of-distribution generalization. arXiv preprint arXiv:2306.08076 , 2023.

Zenan Li, Qitian Wu, Fan Nie, and Junchi Yan. Graphde: A generative framework for debiased learning and out-of-distribution detection on graphs. Advances in Neural Information Processing Systems , 35:30277-30290, 2022.

Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In International Conference on Learning Representations , 2018.

Derek Lim, Felix Hohne, Xiuyu Li, Sijia Linda Huang, Vaishnavi Gupta, Omkar Bhalerao, and Ser Nam Lim. Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods. Advances in Neural Information Processing Systems , 34:20887-20902, 2021.

Changshu Liu, Liangjian Wen, Zhao Kang, Guangchun Luo, and Ling Tian. Self-supervised consensus representation learning for attributed graph. In Proceedings of the 29th ACM international conference on multimedia , pages 2654-2662, 2021.

Kay Liu, Yingtong Dou, Yue Zhao, Xueying Ding, Xiyang Hu, Ruitong Zhang, Kaize Ding, Canyu Chen, Hao Peng, Kai Shu, et al. Bond: Benchmarking unsupervised outlier node detection on static attributed graphs. Advances in Neural Information Processing Systems , 35:27021-27035, 2022.

Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Advances in neural information processing systems , 33:21464-21475, 2020.

Yang Liu, Xiang Ao, Zidi Qin, Jianfeng Chi, Jinghua Feng, Hao Yang, and Qing He. Pick and choose: a gnn-based imbalanced learning approach for fraud detection. In Proceedings of the web conference 2021 , pages 3168-3177, 2021.

Yixin Liu, Kaize Ding, Huan Liu, and Shirui Pan. Good-d: On unsupervised graph out-ofdistribution detection. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining , pages 339-347, 2023.

Yixin Liu, Zhao Li, Shirui Pan, Chen Gong, Chuan Zhou, and George Karypis. Anomaly detection on attributed networks via contrastive self-supervised learning. IEEE transactions on neural networks and learning systems , 33(6):2378-2392, 2021.

Xiaoxiao Ma, Ruikun Li, Fanzhen Liu, Kaize Ding, Jian Yang, and Jia Wu. Graph anomaly detection with few labels: A data-centric approach. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2153-2164, 2024.

Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based recommendations on styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval , pages 43-52, 2015.

Hyeonjin Park, Seunghun Lee, Sihyeon Kim, Jinyoung Park, Jisu Jeong, Kyung-Min Kim, Jung-Woo Ha, and Hyunwoo J Kim. Metropolis-hastings data augmentation for graph neural networks. Advances in Neural Information Processing Systems , 34:19010-19020, 2021.

Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , pages 1532-1543, 2014.

Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11557-11568, 2021.

Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan. A simple fix to mahalanobis distance for improving near-ood detection. arXiv preprint arXiv:2106.09022 , 2021.

Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter J Liu. Out-of-distribution detection and selective generation for conditional language models. In The Eleventh International Conference on Learning Representations , 2022.

Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. 2020.

Benedek Rozemberczki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. Journal of Complex Networks , 9(2):cnab014, 2021.

Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self-supervised outlier detection. In Proceedings of the International Conference on Learning Representations , 2021.

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina EliassiRad. Collective classification in network data. AI magazine , 29(3):93-93, 2008.

Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Hsu, and Kuansan Wang. An overview of microsoft academic service (mas) and applications. In Proceedings of the 24th international conference on world wide web , pages 243-246, 2015.

Yu Song and Donglin Wang. Learning on graphs with out-of-distribution nodes. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1635-1645, 2022.

Maximilian Stadler, Bertrand Charpentier, Simon Geisler, Daniel Zügner, and Stephan Günnemann. Graph posterior network: Bayesian predictive uncertainty for node classification. Advances in Neural Information Processing Systems , 34:18033-18048, 2021.

Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. Advances in Neural Information Processing Systems , 34:144-157, 2021.

Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning , pages 20827-20840. PMLR, 2022.

Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning (ICML) . PMLR, 2022.

Dennis Ulmer, Lotta Meijerink, and Giovanni Cinà. Trust issues: Uncertainty estimation does not enable reliable ood detection on medical tabular data. In Machine Learning for Health , pages 341-354. PMLR, 2020.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. International Conference on Learning Representations , 2017.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual-logit matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4921-4930, 2022.

Qizhou Wang, Feng Liu, Yonggang Zhang, Jing Zhang, Chen Gong, Tongliang Liu, and Bo Han. Watermarking for out-of-distribution detection. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Xiao Wang, Hongrui Liu, Chuan Shi, and Cheng Yang. Be confident! towards trustworthy graph neural networks via confidence calibration. Advances in Neural Information Processing Systems , 34:23768-23779, 2021.

Xuhong Wang, Baihong Jin, Ying Du, Ping Cui, Yingshui Tan, and Yupu Yang. One-class graph neural networks for anomaly detection in attributed networks. Neural computing and applications , 33:12073-12085, 2021.

Yanling Wang, Jing Zhang, Shasha Guo, Hongzhi Yin, Cuiping Li, and Hong Chen. Decoupling representation learning and classification for gnn-based anomaly detection. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval , pages 1239-1248, 2021.

Yili Wang, Yixin Liu, Xu Shen, Chenyu Li, Kaize Ding, Rui Miao, Ying Wang, Shirui Pan, and Xin Wang. Unifying unsupervised graph-level anomaly detection and out-of-distribution detection: a benchmark. arXiv preprint arXiv:2406.15523 , 2024.

Qitian Wu, Yiting Chen, Chenxiao Yang, and Junchi Yan. Energy-based out-of-distribution detection for graph neural networks. In The Eleventh International Conference on Learning Representations , 2022.

Qitian Wu, Fan Nie, Chenxiao Yang, Tianyi Bao, and Junchi Yan. Graph out-of-distribution generalization via causal intervention. arXiv preprint arXiv:2402.11494 , 2024.

Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020.

Bingbing Xu, Huawei Shen, Qi Cao, Keting Cen, and Xueqi Cheng. Graph convolutional networks using heat kernel for semi-supervised learning.

Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International conference on machine learning , pages 5453-5462. PMLR, 2018.

Yujun Yan, Jiong Zhu, Marlena Duda, Eric Solarz, Chandra Sripada, and Danai Koutra. Groupinn: Grouping-based interpretable neural network for classification of limited, noisy brain data. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pages 772-782, 2019.

Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2110.11334 , 2021.

Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages 974-983, 2018.

Junchi Yu, Jian Liang, and Ran He. Mind the label shift of augmentation-based graph ood generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11620-11630, 2023.

Haonan Yuan, Qingyun Sun, Xingcheng Fu, Ziwei Zhang, Cheng Ji, Hao Peng, and Jianxin Li. Environment-aware dynamic graph learning for out-of-distribution generalization. Advances in Neural Information Processing Systems , 36, 2024.

Reza Zafarani, Mohammad Ali Abbasi, and Huan Liu. Social media mining: an introduction . Cambridge University Press, 2014.

Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. Graphsaint: Graph sampling based inductive learning method. In International Conference on Learning Representations , 2019.

Ge Zhang, Jia Wu, Jian Yang, Amin Beheshti, Shan Xue, Chuan Zhou, and Quan Z Sheng. Fraudre: Fraud detection dual-resistant to graph inconsistency and imbalance. In 2021 IEEE International Conference on Data Mining (ICDM) , pages 867-876. IEEE, 2021.

Jinsong Zhang, Qiang Fu, Xu Chen, Lun Du, Zelin Li, Gang Wang, xiaoguang Liu, Shi Han, and Dongmei Zhang. Out-of-distribution detection based on in-distribution data patterns memorization with modern hopfield energy. In The Eleventh International Conference on Learning Representations , 2023.

Tong Zhao, Yozen Liu, Leonardo Neves, Oliver Woodford, Meng Jiang, and Neil Shah. Data augmentation for graph neural networks. In Proceedings of the aaai conference on artificial intelligence , volume 35, pages 11015-11023, 2021.

Xujiang Zhao, Feng Chen, Shu Hu, and Jin-Hee Cho. Uncertainty aware semi-supervised learning on graph data. Advances in Neural Information Processing Systems , 33:12827-12836, 2020.

Cheng Zheng, Bo Zong, Wei Cheng, Dongjin Song, Jingchao Ni, Wenchao Yu, Haifeng Chen, and Wei Wang. Robust graph representation learning via neural sparsification. In International Conference on Machine Learning , pages 11458-11468. PMLR, 2020.

Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in neural information processing systems , 33:7793-7804, 2020.

Yao Zhu, YueFeng Chen, Chuanlong Xie, Xiaodan Li, Rong Zhang, Hui Xue', Xiang Tian, bolun zheng, and Yaowu Chen. Boosting out-of-distribution detection with typical features. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

A Proofs

Theorem A.1. (Recap of Theorem 3.2) For any two test ID/OOD node set S id ⊂ V uid , S ood ⊂ V uood with equal size N s , let the ID-vs-OOD separability M sep defined on a OOD scoring vector ˆ g ∈ R N as

If M sep (ˆ g ) > 0 and η intra -η inter > 1 /N s , for some ϵ > 0 and constant c , we have

Proof. Without losing the generality, we set the ˆ g i = 0 , if i ∈ S ood ∪ S id , since we only care about the detection results in the given test node set S ood and S id .

The M sep (ˆ g ) can be re-written as

According to General Hoeffding's inequality (Theorem 2.6.3) in [70], we know that

where c is some constant value.

Since ˆ g i = 0 , if i ∈ V l ,

Combining together, we have

Theorem A.2. (Recap of Theorem 4.2) For any two test ID/OOD node set S id ⊂ V uid , S ood ⊂ V uood with equal size N s , let the ID-vs-OOD separability M sep defined on a non-negative OOD scoring vector ˆ g ∈ R N as

Let E S ↔ S ′ ⊂ E to denote the edge set of edges between two node sets S and S ′ , where S, S ′ ⊂ V . If we can find a node set G ⊂ V l such that |E G ↔ S id | > |E G ↔ S ood | , we have

where E = e G e ⊤ G and δ > 0 .

Proof. The M sep (ˆ g ) can be re-written as

Then we have

We can then directly derive the proof by expanding

where the second and the third equation are derived by the fact that G ⊂ V l and then we have e ⊤ S id e G = 0 and e ⊤ S ood e G = 0 .

B Details of Baselines

For the reader's convenience, we summarize in detail a few common techniques for defining OOD scores that measure the degree of ID-ness on a given input. By convention, a higher (lower) score is indicative of being in-distribution (out-of-distribution).

MSP [23] This method proposes to use the maximum softmax score as the OOD score. For each node i , we use F OODD ( i ) = max c ∈ [ C ] f c ( i ) as the OOD score.

ODIN [43] This method improves OOD detection with temperature scaling and input perturbation. In all experiments, we set the temperature scaling parameter T = 1000 . For graph neural network, we found the input perturbation does not further improve the OOD detection performance and hence we set ϵ = 0 .

Mahalanobis [37] This method uses multivariate Gaussian distributions to model class-conditional distributions of softmax neural classifiers and uses Mahalanobis distance-based scores for OOD detection. The mean µ c of each multivariate Gaussian distribution with class c and a tied covariance Σ are estimated based on training samples. We define the confidence score M ( x ) using the Mahalanobis distance between test sample x and the closest class-conditional Gaussian distribution.

Energy [47] This method proposes using energy score for OOD detection. The energy function maps the logit outputs to a scalar E ( x i ; f ) ∈ R , which is relatively lower for ID data. Note that [47] used the negative energy score for OOD detection, in order to align with the convention that S ( x ) is higher (lower) for ID (OOD) data.

KNN [67] This method uses the k -th nearest neighbor distance between a test graph node and the training set as the OOD score. We use k = 10 for all experiments in this paper.

C Dataset Details

We adopt ten publicly available benchmarks used for graph learning, covering diverse domains, scales, and structures (homophily or heterophily). For five homophily datasets Cora , Amazon-Photo , Coauthor-CS , Reddit2 and ogbn-products , we use the data loader provided by the Pytorch Geometric package 3 . For the remaining five heterophily datasets, we directly use the pickle file or download from the given hyperlinks proposed by [44].

Cora [61] is a 7-class citation network comprising 2,708 nodes, 5,429 edges and 1,433 features. In this network, each node represents a published paper, each edge signifies a citation relationship, and the label class is each paper's topic, which is the goal to predict.

Amazon-Photo [52] is an 8-class item co-purchasing network on Amazon, which contains 7,650 nodes, 238,162 edges and 745 features. In this network, each node denotes a product, each edge indicates that two linked products are frequently purchased together, and the node label denotes the category of the product.

Coauthor-CS [62] is a 15-class coauthor network of computer science, which contains 18,333 nodes, 163,788 edges and 6,805 features. In this network, nodes denote authors and there is an edge between two authors if co-authored a paper. And the label represents the study field for the authors.

Reddit2 [88] is a large homophily post network with 41 classes, where nodes are posts based on user comments and the task is to predict communities of online posts based on user comments.

ogbn-products [24] is a super large homophily undirected and unweighted graph with 47 classes, representing an Amazon product co-purchasing network. Nodes represent products sold in Amazon, and edges between two products indicate that the products are purchased together. The task is to predict the category of a product in a multi-class classification setup, where the 47 top-level categories are used for target labels.

Chameleon and Squirrel [59] are two Wikipedia networks with 5 classes, where nodes represent web pages and edges represent hyperlinks between them. Node features represent several informative nouns in the Wikipedia pages and the task is to predict the average daily traffic of the web page [17].

arXiv-year [24] is the ogbn-arXiv network with different labels and is altered to be heterophily, in which the class labels are set to be the year that the paper is posted, instead of subject area in the original paper. The nodes are arXiv papers, and directed edges connect a paper to other papers that it cites. The node features are averaged word2vec token features of both the title and abstract of the paper. The five classes are chosen by partitioning the posting dates so that class ratios are approximately balanced [44].

snap-patents [39, 38] is a big dataset of utility patents in the US. Each node represents a patent and edges connect patents that cite each other. Node features are derived from patent metadata [44]. Like arXiv-year , this dataset is changed to set the task to predict the time at which a patent was granted, which is also five classes.

wiki [44] is a super big dataset of Wikipedia articles, which are crawled and cleaned from the internet. Nodes represent pages and edges represent links between them. Node features are derived from the average GloVe embeddings [54] of the titles and abstracts and labels indicate total page views over a 60-day period, categorized into five classes based on quintiles.

The complete information and statistics of all these datasets aforementioned are summarized in Table 7.

D Additional Experiments

D.1 Graph ID Classification Details

In this section, we provide a detailed report on the ID ACC results of nine popular GNN pretrained backbones used in our paper, including GCN [34], GAT [69], GCNJK [81], GATJK [81], APPNP [35], MixHop [1], GPR-GNN [10], GCNII [7], and H2GCN [94]. The results of all the nine architectures on five common benchmarks and architecture GCN on five large-scale benchmarks are presented in Table 8 and Table 9 respectively. In addition, we present the ID ACC results of three training-based methods on the ten benchmarks. It is important to note that these training-based methods only yield results on five small-scale datasets and the moderately sized arxiv-year dataset. They fail to produce results on the remaining four large datasets, as shown in Tables 10 and Table 11.

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

D.2 Detailed Main Graph OOD Detection Results

D.3 Graph OOD Detection Results on Various Backbones

GRASP can seamlessly be applied to various GNN backbones. In this section, we report the OOD detection performance of all baselines on various popular architectures in Table 14

D.4 Analysis of Hyper-parameters Sensitivity

We show the sensitivity of hyper-parameters α , β and k in Figure 6. The vertical axis in the figure represents the average AUROC values across common datasets. The performance comparison in the bar plot for each hyper-parameter is reported by fixing other hyper-parameters. We see that within the range of chosen hyperparameter values, our proposed method's performance does not vary significantly, which constantly outperforms baselines by a large margin.











































































k























D.5 Relationship between Ratio of Intra-Edges and OOD Detection performance























As shown in Figure 7, the ratio of intra-edges and OOD detection performance have strong positive correlation. This validates the correctness of Theorem 4.2. The detailed the number in Figure 7 are listed in Table 15. From the table we can see that after using our proposed augmentation, the ratio of intra-edges increases and in turn boosts the OOD detection performance.

D.6 Computational Complexity Analysis

Our method's computational footprint in terms of runtime and memory usage after propagation for k times is:

Time Complexity: O ( N +2 k |E| + n )

Memory Complexity: O ( N + |E| + n )

While our algorithm introduces the fully connected matrix E , which brings about explosive growth of connections after each propagation and results in challenges in performing related computations, we adeptly transform the complex and time-consuming structural computations into simple matrix-vector multiplication operations, making original expensive operation tractable and leading to efficient linear complexity. Below, we provide the support from both mathmatical and empirical points.

Complexity analysis. Our method consists of two major computational modules, specifically, ( a ) calculating connections between training nodes and testing nodes; ( b ) augmentation propagation.

( a ) Calculating connections between training nodes and testing nodes. We utilize two indicator vectors, I id and I ood , both with a length of nodes number N , to represent the number of connections each node has with S id and S ood , respectively. The initial values of I id are set to 1 at the indices corresponding to S id and 0 at the remaining indices (corresponding to S ood ). The initial value of I id is denoted as I (0) id . Similarly, I ood has a initial value of 1 at all indices corresponding to S ood and 0 at the indices corresponding to S id . We also represent the initial value of I ood as I (0) ood . After propagation for k times, the connections between each node with S id are:

In the above equation, AI ( i ) id can be computed in O ( |E id | ) with sparse matrix multiplication for each propagation, where |E id | represents number of edges connected to S id nodes.

( b ) Augmentation propagation. As described by Equation 6 in the main paper, given a raw OOD scoring vector ˆ g ∈ R N , the propagated scoring vector using augmentated adjacency matrix after

The above abalysis and conclusion also applys to computing I ( k ) ood , resulting computational complexity O ( |E ood | ) , where |E ood | is number of edges connected to S ood nodes. This leads to time and memory costs O ( |E id | + |E ood | ) = O ( |E| ) for calculating needed number of connections in each propagation and O ( k |E| ) and O ( |E| ) for time and memory costs after propagation for k times, respectively.

propagation for k times is given by:

In the above equation,

E ˆ g can be computed with time/space complexity O ( n ) by a simple summation operation ( g i in G will be replaced by ∑ i ∈ G g i ) to get rid of the matrix multiplication.

D -1 + is an O ( N ) operation since it is scaling over all elements in the vector.

A ˆ g can be computed in O ( |E| ) with sparse matrix multiplication.

Above all, the time/memory complexity of augmentated propagation is O ( N + |E| + n ) for each propagation and the time and memory costs are O ( N + k |E| + n ) and O ( N + |E| + n ) for k times respectively with memory reutilization.

Integrating the above two considerations, the time complexity of our algorithm after propagation for k times is O ( N +2 k |E| + n ) , while the space complexity is O ( N + |E| + n ) . Notably, the utilization of space reuse results in a linear space complexity.

Empirical results. Then we conduct comprehensive experiments comparing the time and space costs of our algorithm with various baselines (both post-hoc and training-based approaches) in Table 16 and Table 17. From the experimental results, it is evident that our algorithm is highly efficient across all datasets.

Based on the experimental results in Table 16 and Table 17, we have:

In comparison to training-based methods, post-hoc methods exhibit significantly lower runtime and memory consumption.

Considering that the minimum time and space complexity required to run a graph algorithm is O ( N + |E| ) , as outlined in our algorithm complexity analysis, our algorithm incurs limited additional time and space costs, specifically O ( |E| ) and O ( n ) on time and memory costs respectively in each propagation. This can be validated by the small extra overhead incurred by our algorithm compared to MSP or Energy from the table.

Compared with training-based methods, our algorithm demonstrates substantially lower time and space demands on five large-scale datasets. And when compared to post-hoc baselines on these datasets, the overhead of our method is also reasonable. The performance of our method on these large-scale datasets underscores the strong practicality of our approach.

D.7 Explore How Different Propagation Mechanisms Impact the Findings.

In this section, we investigate the effect of different propagation mechanisms on graph OOD detection performance, like higher-order diffusion and the other various classical propagation mechanisms in the literature, including Personalized PageRank (PPR) [35], Heat Kernel Diffusion (GraphHeat) [80], Graph Diffusion Convolution (GDC) [20], Mixing Higher-Order Propagation (MixHop) [1], and Generalized PageRank (GPR) [10]. The emperical observations are as follows:

From the analysis of the hyper-parameter (order of propagation) and its impact on the AUROC in Figure 6, we find that OOD detection performance benefits from propagation orders within a reasonable range. However, excessive propagation (greater than 16) may be detrimental.

The AUROC results of the other various classical propagation mechanisms on graph OOD detection are shown in Table 18 and Table 19. We can see that these propagation mechanisms perform inconsistently among various datasets. In contrast, our propagation mechanism GRASP constantly demonstrates superior performance among all datasets.

E Limitations

While the proposed method of OOD score propagation shows promise in improving OOD detection in graph learning, there are limitations worth noting. The effectiveness of the method heavily relies on the connectivity and structure of the graph. In scenarios where the graph exhibits random connectivity patterns, the propagation of OOD scores with the proposed solution may no longer be effective. Addressing these limitations will be crucial for ensuring the robustness and generalizability of the proposed OOD detection approach in graph learning.

F Broader Impact

This paper on detecting Out-of-Distribution (OOD) nodes in graph learning holds significant potential for various fields and applications. By addressing a salient yet under-explored challenge in graph neural networks, the proposed method of OOD score propagation offers a promising avenue for enhancing the robustness and reliability of graph-based machine learning systems. The implications extend beyond the realm of graph learning, as OOD detection is a critical component in many realworld applications, including anomaly detection, fraud detection, and network security. Improving OODdetection in graph structures can lead to more accurate and reliable decision-making in domains such as social network analysis, recommendation systems, and biological network analysis. Overall, this research has the potential to drive advancements in graph-based machine learning and contribute to the development of more robust and reliable AI systems with broader societal impact.

G Other Related Works

Graph OOD Generalization. Unlike graph OOD detection, which aims to increase the gap between ID and OOD for effective OOD detection, graph OOD generalization aims to learn the invariant aspects behind ID and OOD. By enhancing these invariant points, the gap between ID and OOD data is reduced to improve the robustness of graph models on OOD data. The distinction among different proposed methods lies in identifying different invariant factors. GTRANS [30] leverages contrastive learning to learn the optimal perturbations of features and structures to enhance the GNN model's robustness against OOD. GRAPHPATCHER [31] improves GNN's robustness against low-degree scenarios during testing by corrupting a portion of nodes and reconstructing the original structure. LiSA [85] employs variational subgraph generators to generate label-invariant subgraphs for data augmentation, thereby enhancing the robustness of GNN models. [9] leverages the stability of correlation and employs a graph decorrelation method to learn stable associations, raising the graph OOD generalization capability. [78] boosts generalization by eliminating confounding bias. [41] extrapolates structure and feature spaces to generate OOD graph data, which is then used for data augmentation to improve OOD generalization. [28] strengthens graph OOD generalization by extracting invariant subgraphs and learning invariant patterns behind graphs based on these subgraphs. [8] introduces a set of minimal assumptions for feasible invariant graph learning. [5] investigates OOD generalization under different structural shifts. [22] incorporates label and environment causal independence to discover causal subgraphs for intensifying OOD generalization. [86] improves the OOD generalization of dynamic graphs by learning spatio-temporal invariant patterns from an environment learning perspective.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provide theoretical contribution in Section 3 and Section 4, and experimental results in Section 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] .

Justification: We present the limitation discussion in Appendix E.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the assumptions and complete proof in Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Section 5 and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification:

https://github.com/longfei-ma/GRASP

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Appendix D.6

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Appendix F

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification:

https://github.com/longfei-ma/GRASP

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors attempt to tackle the task of detecting Out-of-Distribution (OOD) nodes in graph data. To this end, the authors propose an augmentation method, namely Graph-Augmented Score Propagation (GRASP). GRASP performs the task of OOD detection in graphs by increasing the ratio of intra-edges (i.e., edges only connecting in-distribution nodes or OOD nodes). Theoretical analysis is also provided to support the effectiveness of the proposed approach. Experimental results show that GRASP can outperform existing OOD detection methods in various datasets.\n\nSTRENGTHS:\n1. The proposed method is introduced in an informative manner.\n2. The paper is easy to follow.\n3. The comparative experiments are comprehensive.\n4. The hyperparameter sensitivity analysis is thorough, providing readers with valuable insights.\n\nWEAKNESSES:\n1. At the beginning of the article, the author might overly emphasize the contribution of this article, to some extent giving the impression to readers that it is the first study on graph OOD detection. However, there have already been some papers in this area.​ For instance, 1) \"Learning on Graphs with Out-of-Distribution Nodes,\" in KDD 2022 (cited); 2) \"Generalizing graph neural networks on out-of-distribution graphs,\" in TPAMI 2023 (not cited); 3) \"Good-d: On unsupervised graph out-of-distribution detection (cited),\" in WSDM 2023. While their core idea differs from this paper, their contributions should not be overlooked. Given these observations, it is highly suggested that the authors conduct broader investigations of existing works, discuss the differences between the proposed methods and other related methods, and adjust the contributions of the paper.\n2. The baselines selected in the experiments may not be sufficiently novel. This makes the experimental results are not convincing enough.\n3. The proposed method needs more motivation from both theoretical and practical aspects. For example, are there any observations or previous studies supporting the assumptions that edges follow a Bernoulli distribution? Are there concrete (or real-world) examples that can demonstrate the importance of the number of intra-edges and inter-edges in OOD detection?\n\nQUESTIONS:\n1. The authors emphasize multiple times the importance of the number of intra-edges and inter-edges in OOD detection, providing theoretical proof. However, could a concrete example be provided to illustrate this concept more intuitively?\n2. Does the assumption that edges follow a Bernoulli distribution receive support in most real-world datasets, practical applications, or previous studies?\n3. In Table 3, the FPR on the dataset reddit2 is significantly lower than that of its comparison models. Could the author provide further explanation for this remarkable improvement?\n4. Is there any underlying pattern in the ratio of intra-edges to inter-edges when achieving the best OOD detection performance across different datasets?\n5. Can authors conduct broader investigations of recent related works, explicitly discuss the differences between the proposed and other related methods, and adjust the contributions of the paper (if possible)?\n\nLIMITATIONS:\nThe authors have discussed the limitations of the proposed method in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This study investigates the effectiveness of score propagation in graph raph Out-of-Distribution (OOD) detection. It explores the conditions under which score propagation can be beneficial and proposes an edge augmentation strategy (GRASP) to improve its performance. The authors provide theoretical guarantees for GRASP and demonstrate its performance compared to existing OOD detection methods on several graph datasets.

STRENGTHS:
1) The study delves into the mechanisms of score propagation and derives conditions for its effectiveness. This theoretical foundation is solid and extends the understanding of graph OOD detection.

2) The proposed GRASP method is a practical and efficient solution for improving OOD detection performance. The results showed that GRASP has achieves the SOTA performance.

3.) The paper is generally in well-written and easy to follow.

WEAKNESSES:
1) it is strongly suggested to release the source code of experiments.

2) Considering the imporvement is margin in Table 2 and 3, it is also suggested that the authors should provide a standard error with statistical significance analysis in the results.

QUESTIONS:
1. How does the performance of GRASP vary with different OOD distributions (e.g., uniform, normal, outliers)?

2. Can the authors provide further insights into how GRASP works and how it interprets the graph structure?

3. How does the performance of GRASP scale with the size of the graph and the number of nodes? Is there any systemic analysis?

LIMITATIONS:
I did not find significant limitation of this study, one point may concern me that the effectiveness of GRASP relies on the connectivity and structure of the graph. In scenarios with random connectivity patterns, the method may not be as effective..
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a methodology called Graph-Augmented Score Propagation to improve OOD detection performance on graphs. The key idea of the paper is an edge augmentation strategy which selectively adds edges to a subset of training nodes, which is combined with score propagation for the OOD node detection task on the graphs. Theoretical analyses is provided which links the OOD score propagation to the intra-edge vs inter-edge ratios between ID and OOD samples. Experimental results are provided on several real world graph datasets with the measurement of OOD detection metrics for demonstrating the effectiveness of their method.\n\nSTRENGTHS:\n1. The paper is generally well written, well motivated and easy to follow. \n\n2. Theoretical analysis of the setting when OOD score propagation will be effective is helpful for future work in this direction, and for developing OOD detection methods for graphs. \n\n3. The key idea of the work for OOD detection is presented as a post hoc strategy, hence the practical applicability of the method is good. \n\n4. Results in Table 2 and 3 are convincing.\n\nWEAKNESSES:\n1. The creation of the subset G is dependent on the selection of S_id and S_ood examples which uses the MSP as a measure of for creation of these sets. However it has been well-discussed that MSP suffers from several practical issues such as overconfidence [Hendrycks and Gimpel,  ICLR ‘17], poor generalization [Lee et. al, NeurIPS ‘18] and calibration issues [Guo et al, ICML ‘17]. Because of GRASP’s dependence on the MSP for creating the set G, the overall method can be non-robust. \n\n\n2. Error bars are missing from all the results, which is important for assessing the consistency of the proposed method, since the data augmentation could be significantly affected by the randomness.\n\nQUESTIONS:\nHow would the authors ensure robustness of their detection method when the confidence could vary? (see weakness pt 1 above) Did the authors do any experiments to evalute the robustness of their method and how would they address it?\n\nLIMITATIONS:\nThe authors mention the limitations of their work in Appendix section E and societal impact in Appendix Section F.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}

comments: SUMMARY:
This work aims to detect out-of-distribution (OOD) nodes on a graph by exploring useful OOD score propagation methods. It introduces a novel edge augmentation strategy, with a theoretical guarantee. The approach's superiority is empirically demonstrated, outperforming OOD detection baselines in various scenarios and settings.

STRENGTHS:
S1. The paper conducts an in-depth analysis and empirically validates the beneficial conditions for  OOD score propagation in a graph.

S2. The paper introduces the GRASP methodology, which addresses the limitations of previous score propagation methods and demonstrates superior performance in node-level OOD detection tasks.

WEAKNESSES:
W1. Does the strategy of utilizing GRASP with additional edges compromise classification accuracy within the distribution? In other words, how can you balance the homogeneity of graph classification with the homogeneity of OOD detection tasks when adding edges?  

W2. While this method is suitable for node-level OOD detection, it appears to be more challenging to adapt it to subgraph-level tasks or graph-level tasks.

QUESTIONS:
Q1. Is the classification accuracy in Appendix D.1 assessed before or after employing GRASP? Is there a comparison between the two?  

Q2. Do similar cases discussed in Figure 2 exist within real-world datasets?  

Q3. What is the accuracy of the edge-adding method presented in the text, specifically how many to intra-edges, and how many to inter-edges? What impact would incorrectly added edges have on the results?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-30