Title: Uncertainty Quantification over Graph with Conformalized Graph Neural Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: Graph Neural Networks

Conformal Prediction

Uncertainty Quantification
Abstract: Graph Neural Networks (GNNs) are powerful machine learning prediction models on graph-structured data. However, GNNs lack rigorous uncertainty estimates, limiting their reliable deployment in settings where the cost of errors is significant. We propose conformalized GNN (CF-GNN), extending conformal prediction (CP) to graph-based models for guaranteed uncertainty estimates. Given an entity in the graph, CF-GNN produces a prediction set/interval that provably contains the true label with pre-defined coverage probability (e.g. 90%). We establish a permutation invariance condition that enables the validity of CP on graph data and provide an exact characterization of the test-time coverage. Moreover, besides valid coverage, it is crucial to reduce the prediction set size/interval length for practical use. We observe a key connection between non-conformity scores and network structures, which motivates us to develop a topology-aware output correction model that learns to update the prediction and produces more efficient prediction sets/intervals. Extensive experiments show that CF-GNN achieves any pre-defined target marginal coverage while significantly reducing the prediction set/interval size by up to 74% over the baselines. It also empirically achieves satisfactory conditional coverage over various raw and network features.

Uncertainty Quantification over Graph with Conformalized Graph Neural Networks

Abstract

Graph Neural Networks (GNNs) are powerful machine learning prediction models on graph-structured data. However, GNNs lack rigorous uncertainty estimates, limiting their reliable deployment in settings where the cost of errors is significant. We propose conformalized GNN (CF-GNN), extending conformal prediction (CP) to graph-based models for guaranteed uncertainty estimates. Given an entity in the graph, CF-GNN produces a prediction set/interval that provably contains the true label with pre-defined coverage probability (e.g. 90%). We establish a permutation invariance condition that enables the validity of CP on graph data and provide an exact characterization of the test-time coverage. Besides valid coverage, it is crucial to reduce the prediction set size/interval length for practical use. We observe a key connection between non-conformity scores and network structures, which motivates us to develop a topology-aware output correction model that learns to update the prediction and produces more efficient prediction sets/intervals. Extensive experiments show that CF-GNN achieves any pre-defined target marginal coverage while significantly reducing the prediction set/interval size by up to 74% over the baselines. It also empirically achieves satisfactory conditional coverage over various raw and network features.

1 Introduction

Graph Neural Networks (GNNs) have shown great potential in learning representations for graphstructured data, which has led to their widespread adoption in weather forecasting [29], drug discovery [31], and recommender systems [46], etc. As GNNs are increasingly deployed in high-stakes settings, it is important to understand the uncertainty in the predictions they produce. One prominent approach to uncertainty quantification is to construct a prediction set/interval that informs a plausible range of values the true outcome may take. A large number of methods have been proposed to achieve this goal [17, 49, 28, 44]. However, these methods often lack theoretical and empirical guarantees regarding their validity, i.e. the probability that the prediction set/interval covers the outcome [2]. This lack of rigor hinders their reliable deployment in situations where errors can be consequential.

Conformal prediction [43] (CP) is a framework for producing statistically guaranteed uncertainty estimates. Given a user-specified miscoverage level α ∈ (0 , 1) , it leverages a set of 'calibration' data to output prediction sets/intervals for new test points that provably include the true outcome with probability at least 1 -α . Put another way, the conformal prediction sets provably only miss the test outcomes at most α fraction of the time. With its simple formulation, clear guarantee and

distribution-free nature, it has been successfully applied to various problems in computer vision [2, 4], causal inference [30, 22, 47], time series forecasting [11, 48], and drug discovery [21].

Despite its success in numerous domains, conformal prediction has remained largely unexplored in the context of graph-structured data. One primary challenge is that it is unclear if the only, yet crucial, assumption for CP-exchangeability between the test and calibration samples-holds for graph data. When applying conformal prediction, exchangeability is usually ensured by independence among the trained model, the calibration data, and test samples (see Appendix B for more discussion). However, in the transductive setting, GNN training employs all nodes within the same graph-including test points-for message passing, creating intricate dependencies among them. Thus, to deploy conformal prediction for graph data, the first challenge is to identify situations where valid conformal prediction is possible given a fitted GNN model that already involves test information.

Efficiency is another crucial aspect of conformal prediction for practical use: a prediction set/interval with an enormous set size/interval length might not be practically desirable even though it achieves valid coverage. Therefore, the second major challenge is to develop a graph-specific approach to reduce the size of the prediction set or the length of the prediction interval (dubbed as inefficiency hereafter for brevity) while retaining the attractive coverage property of conformal prediction.

Present work. We propose conformalized GNN (CF-GNN), 1 extending conformal prediction to GNN for rigorous uncertainty quantification over graphs. We begin by establishing the validity of conformal prediction for graphs. We show that in the transductive setting, regardless of the dependence among calibration, test, and training nodes, standard conformal prediction [43] is valid as long as the score function (whose definition will be made clear in Section 2) is invariant to the ordering of calibration and test samples. This condition is easily satisfied by popular GNN models. Furthermore, we provide an exact characterization of the empirical test-time coverage.

Subsequently, we present a new approach that learns to optimize the inefficiencies of conformal prediction. We conduct an empirical analysis which reveals that inefficiencies are highly correlated along the network edges. Based on this observation, we add a topology-aware correction model that updates the node predictions based on their neighbors. This model is trained by minimizing a differentiable efficiency loss that simulates the CP set sizes/interval lengths. In this way, unlike the raw prediction that is often optimized for prediction accuracy, the corrected GNN prediction is optimized to yield smaller/shorter conformal prediction sets/intervals. Crucially, our approach aligns with the developed theory of graph exchangeability, ensuring valid coverage guarantees while simultaneously enhancing efficiency.

We conduct extensive experiments across 15 diverse datasets for both node classification and regression with 8 uncertainty quantification (UQ) baselines, covering a wide range of application domains. While all previous UQ methods fail to reach pre-defined target coverage, CF-GNN achieves the pre-defined empirical marginal coverage. It also significantly reduces the prediction set sizes/interval lengths by up to 74% compared with a direct application of conformal prediction to GNN. Such improvement in efficiency does not appear to sacrifice adaptivity: we show that CF-GNN achieves strong empirical conditional coverage over various network features.

2 Background and Problem Formulation

Let G = ( V , E , X ) be a graph, where V is a set of nodes, E is a set of edges, and X = { x v } v ∈V is the attributes, where x v ∈ R d is a d -dimensional feature vector for node v ∈ V . The label of node v is y v ∈ Y . For classification, Y is the discrete set of possible label classes. For regression, Y = R .

Transductive setting. We focus on transductive node classification/regression problems with random data split. In this setting, the graph G is fixed. At the beginning, we have access to { ( x v , y v ) } v ∈D as the 'training' data, as well as test data D test with unknown labels { y v } v ∈D test . Here D and D test are disjoint subsets of V . We work on the prevalent random split setting where nodes in D and D test are randomly allocated from the entire graph, and the test sample size is m = |D test | . The training node set D is then randomly split into D train / D valid / D calib of fixed sizes, the training/validation/calibration set, correspondingly. A perhaps nonstandard point here is that we withhold a subset D calib as 'calibration' data in order to apply conformal prediction later on. During the training step, the data

<latexi sh

1_b

64="p9cWonIJjLFOH

u

0QAm

g

>

VD

S

N

3U

2vq yE

w

+

X

P

KYT

f

k

z

R

8

rd7

5

Z

G

C

M

<latexi sh

1_b

64="g nSp

J8

m

H

V

7k

L

v

>A

B+X

c

D

N

F

2

r

fUZdu

E

wT

CW

y

5M

0

9

IR

/

O

G

j

q3

Y

o

Q

K

P

z

<latexi sh

1_b

64="830FnL

ZdI

pm rc

V

GMo

J

>A

CHX

D

SgNBE

z

Y

v

y

Qw

5

W

O

u7

K

P

+

2

R

j

/

q

k

T

9

∇

Calibrate

Train+Val

Test

or

or

No label

ˆ

µ

(

X

i

)

˜

µ

(

X

i

)

Topology-aware Conformal Correction

GNN

ϑ

Standard GNN Training

GNN

θ

Standard Conformal Prediction

Regression

{

,

}

1

-

α

Coverage Guaranteed

Given permutation

invariance (Theorem 1)

Classification

3

1

4

2

5

ˆ

η

3

1

+ˆ

η

-

ˆ

η

or

or

or

or

or

ϑ

Inefficiency

1

2

3

L

{ ( x v , y v ) } v ∈D train ∪D valid , the attribute information in { x v } v ∈D calib ∪D test and the entire graph structure ( V , E ) are available to the GNN to compute training nodes representations, while { y v } v ∈D calib ∪D test are not seen.

Graph Neural Networks (GNNs). GNNs learn compact representations that capture network structure and node features. A GNN generates outputs through a series of propagation layers [12], where propagation at layer l consists of the following three steps: (1) Neural message passing. GNN computes a message m ( l ) uv = MSG ( h ( l -1) u , h ( l -1) v ) for every linked nodes u, v based on their embeddings from the previous layer h ( l -1) u and h ( l -1) v . (2) Neighborhood aggregation. The messages between node u and its neighbors N u are aggregated as ̂ m ( l ) u = AGG ( m ( l ) uv | v ∈ N u ) . (3) Update. Finally, GNN uses a non-linear function to update node embeddings as h ( l ) u = UPD ( ̂ m ( l ) u , h ( l -1) u ) using the aggregated message and the embedding from the previous layer. The obtained final node representation is then fed to a classifier or regressor to obtain a prediction µ ( X ) .

̂ Conformal prediction. In this work, we focus on the computationally efficient split conformal prediction method [43]. 2 Given a predefined miscoverage rate α ∈ [0 , 1] , it proceeds in three steps: (1) non-conformity scores. CP first obtains any heuristic notion of uncertainty called non-conformity score V : X × Y → R . Intuitively, V ( x, y ) measures how y "conforms" to the prediction at x . An example is the predicted probability of a class y in classification or the residual value V ( x, y ) = | y -̂ µ ( x ) | in regression for a predictor ̂ µ : X → Y . (2) Quantile computation. CP then takes the 1 -α quantile of the non-conformity scores computed on the calibration set. Let { ( X i , Y i ) } n i =1 be the calibration data, where n = |D calib | , and compute ̂ η = quantile( { V ( X 1 , Y 1 ) , · · · , V ( X n , Y n ) } , (1 -α )(1 + 1 n )) . (3) Prediction set/interval construction. Given a new test point X n +1 , CP constructs a prediction set/interval C ( X n +1 ) = { y ∈ Y : V ( X n +1 , y ) ≤ ̂ η } . If { Z i } n +1 i =1 := { ( X i , Y i ) } n +1 i =1 are exchangeable, 3 then V n +1 := V ( X n +1 , Y n +1 ) is exchangeable with { V i } n i =1 since ̂ µ is given. Thus, ̂ C ( X n +1 ) contains the true label with predefined coverage rate [43]: P { Y n +1 ∈ C ( X n +1 ) } = P { V n +1 ≥ Quantile ( { V 1 , . . . , V n +1 } , 1 -α ) ≥ 1 -α due to exchangeability of { V i } n +1 i =1 . This framework works for any non-conformity score. CF-GNN is similarly non-conformity score-agnostic. However, for demonstration, we focus on two popular scores, described in detail below.

Adaptive Prediction Set (APS). For the classification task, we use the non-conformity score in APS proposed by [37]. It takes the cumulative sum of ordered class probabilities till the true class. Formally, given any estimator ̂ µ j ( x ) for the conditional probability of Y being class j at X = x , j = 1 , . . . , |Y| , we denote the cumulative probability till the k -th most promising class as V ( x, k ) = ∑ k j =1 µ π ( j ) ( x ) ,

<latexi sh

1_b

64="zR

SC2PoU

XZK0y

TW

j+

A

>

c

VD

N

FJ3

vqB

ru

p

w

E

Y

9Q

m

/

8

G

n

d

g

I

H

O

Mf

5

7

<latexi sh

1_b

64="

W

Ef5A

XBH

7

J

Ik2GdCm

>

+3

c

VDLS

N

F

U

v

u

p

8

n

Zw

M

z

Q

/

o

0R

Y

T

O

g

qr

j

y

9

P

K

<latexi sh

1_b

64="SEf3CZzW

2myA

rvVLKG

+Xk5Q

>

B8

c

D

gN

O

M

H

Y

J

w

j

9

7

U

u

P

n

R

0

q

I

F

d

po

/

T

<latexi sh

1_b

64="

5gL

AN

zOHTX

7Mq v0

/E

>

B83

c

V

S

Ur

9

Cp

I

D

W

YGu

K

y

R

w

F

k

Z

2

P

Q

j

d

f

n

o

+

J

m

<latexi sh

1_b

64="g

o

SnYd

IC

q+

T

/k

5

p3

>A

B9H

c

V

N

8

E

Ur

Fv

y

wX

mO2

D

P

L

u

7M

z

W

R0

G

J

K

f

Q

<latexi sh

1_b

64="

IBLG

Z

k

ED

M5/pO

HUAT

d

>

9

c

V

NS8

3

r

q

F

Y

K

gv2

J

ou

Wz

7

RQ

n+

w

PX

f

0

C

y

mj

where π is a permutation of Y so that ̂ µ π (1) ( x ) ≥ ̂ µ π (2) ( x ) ≥ · · · ≥ ̂ µ π ( |Y| ) ( x ) . Then, the prediction set is constructed as C ( x ) = { π (1) , · · · , π ( k ∗ ) } , where k ∗ = inf { k : ∑ k j =1 µ π ( j ) ( x ) ≥ η } .

̂ ̂ ̂ ̂ In its vanilla form, the non-conformity score (including APS and CQR) in CP does not depend on the calibration and test data. That means, { x v } v ∈D calib ∪D test are not revealed in the training process of V , which is the key to exchangeability. In contrast, GNN training typically leverages the entire graph, and hence the learned model depends on the calibration and test attributes in a complicated way. In the following, for clarity, we denote any non-conformity score built on a GNN-trained model as

̂ ̂ Conformalized Quantile Regression (CQR). For the regression task, we use CQR in [36]. CQR is based on quantile regression (QR). QR obtains heuristic estimates ̂ µ α/ 2 ( x ) and ̂ µ 1 -α/ 2 ( x ) for the α/ 2 -th and 1 -α/ 2 -th conditional quantile functions of Y given X = x . The non-conformity score is V ( x, y ) = max { ̂ µ α/ 2 ( x ) -y, y -̂ µ 1 -α/ 2 ( x ) } , interpreted as the residual of true label projected to the closest quantile. The prediction interval is then C ( x ) = [ µ α/ 2 ( x ) -η, µ 1 -α/ 2 ( x ) + η ] .

Evaluation metrics. The goal is to ensure valid marginal coverage while decreasing the inefficiency as much as possible. Given the test set D test , the empirical marginal coverage is defined as Coverage := 1 |D test | ∑ i ∈D test I ( Y i ∈ C ( X i )) . For the regression task, inefficiency is measured as the interval length while for the classification task, the inefficiency is the size of the prediction set: Ineff := 1 |D test | ∑ i ∈D test | C ( X i ) | . The larger the length/size, the more inefficient. Note that inefficiency of conformal prediction is different from accuracy of the original predictions. Our method does not change the trained prediction but modifies the prediction sets from conformal prediction.

to emphasize its dependence on the entire graph, where z v = ( x v , Y v ) for v ∈ V .

3 Exchangeability and Validity of Conformal Prediction on Graph

To deploy CP for graph-structured data, we first study the exchangeability of node information under the transductive setting. We show that under a general permutation invariant condition (Assumption 1), exchangeability of the non-conformity scores is still valid even though GNN training uses the calibration and test information; this paves the way for applying conformal prediction to GNN models. We develop an exact characterization of the test-time coverage of conformal prediction in such settings. Proofs of these results are in Appendix A.1.

Assumption 1. For any permutation π of D calib ∪ D test, the non-conformity score V obeys where ( V π , E π ) represents a graph where D calib ∪ D test nodes (indices) are permuted according to π .

Assumption 1 imposes a permutation invariance condition for the GNN training, i.e., model output/non-conformity score is invariant to permuting the ordering of the calibration and test nodes (with their edges permuted accordingly) on the graph. To put it differently, different selections of calibration sets do not modify the non-conformity scores for any node in the graph. GNN models (including those evaluated in our experiments) typically obey Assumption 1, because they only use the structures and attributes in the graph without information on the ordering of the nodes [24, 16, 12].

For clarity, we write the calibration data as { ( X i , Y i ) } n i =1 , where X i = X v i , and v i ∈ D calib is the i -th node in the calibration data under some pre-defined ordering. Similarly, the test data are { ( X n + j , Y n + j ) } j m =1 , where X n + j = X v j , and v j ∈ D test is the j -th node in the test data. We write

V i is a random variable that depends on the training process and the split of calibration and test data. The next lemma shows that under Assumption 1, the non-conformity scores are still exchangeable.

Lemma 2. In the transductive setting described in Section 2, conditional on the entire unordered graph ( V , E ) , all the attribute and label information { ( x v , y v ) } v ∈V , and the index sets D train and D ct := D calib ∪ D test, the unordered set of the scores [ V i ] n + m i =1 are fixed. Also, the calibration scores { V i } n i =1 are a simple random sample from { V i } n + m i =1 . That is, for any subset { v 1 , . . . , v n } ⊆ { V i } n + m i =1 of size n , P ( { V i } n i =1 = { v 1 , . . . , v n } ∣ ∣ { V i } n + m i =1 ) = 1 / ( n |D ct | ) .

Based on Lemma 2, we next show that any permutation-invariant non-conformity score leads to valid prediction sets, and provide an exact characterization of the distribution of test-time coverage.

Theorem 3. Given any score V obeying Assumption 1 and any confidence level α ∈ (0 , 1) , we define the split conformal prediction set as C ( x ) = { y : V ( x, y ) ≤ η } , where

̂ Then P ( Y n + j ∈ ̂ C ( X n + j )) ≥ 1 -α , ∀ j = 1 , . . . , m . Moreover, define ̂ Cover = 1 m ∑ m j =1 1 { Y n + j ∈ ̂ C ( X n + j ) } . If the V i 's, i ∈ D calib ∪ D test, have no ties almost surely, then for any t ∈ (0 , 1) ,

where Φ HG ( · ; N,n,k ) denotes the cumulative distribution function of a hyper-geometric distribution with parameters N,n,k (drawing k balls from an urn wherein n out of N balls are white balls).

Figure 2 plots the probability density functions (p.d.f.) of ̂ Cover at a sequence of t ∈ [0 , 1] fixing n = 1000 while varying m . The exact distribution described in Theorem 3 is useful in determining the size of the calibration data in order for the test-time coverage to concentrate sufficiently tightly around 1 -α . More discussion and visualization of ̂ Cover under different values of ( n, m ) are in Appendix A.2. Note that similar exchangeability and validity results are obtained in several concurrent works [15, 33], yet without exact characterization of the test-time coverage.

4 CF-GNN: Conformalized Graph Neural Networks

0

10

20

30

40

50

0.85

0.90

0.95

1.00

coverage

p.d.f.

m

100

250

500

750

1000

2000

We now propose a new method called CF-GNN to reduce inefficiency while maintaining valid coverage. The key idea of CF-GNN is to boost any given non-conformity score with graphical information. The method illustration is in Figure 1 and pseudo-code is in Appendix C.

Efficiency-aware boosting. Standard CP takes any pre-trained predictor to construct the prediction set/interval (see Section 2). A key observation is that the training stage is not aware of the post-hoc stage of conformal prediction set/interval construction. Thus, it is not optimized for efficiency. Our high-level idea is to include an additional correction step that boosts the non-conformity score, which happens after the model training and before conformal prediction. To ensure flexibility and practicality, our framework is designed as a generic wrapper that works for any pre-trained GNN model, without changing the base model training process.

Motivation: Inefficiency correlation. Our approach to boosting the scores is based on exploiting the correlation among connected nodes. Since the connected nodes usually represent entities that interact in the real world, there can be strong correlation between them. To be more specific, it is well established in network science that prediction residuals are correlated along edges [20]. Such a result implies a similar phenomenon for inefficiency: taking CQR for regression as an example, the prediction interval largely depends on the residual of the true outcome from the predicted quantiles. Hence, the prediction interval lengths are also highly correlated for connected nodes. We empirically verify this intuition in Figure 3, where we plot the difference in the prediction interval lengths for connected/unconnected node pairs in the Anaheim dataset using vanilla CQR for GCN. 4 In Figure 3, we observe that inefficiency has a topological root: connected nodes usually have similar residual scales, suggesting the existence of rich neighborhood information for the residuals. This motivates us to utilize such information to correct the scores and achieve better efficiency.

Topology-aware correction model. Based on the motivation above, we update the model predictions using the neighbor predictions. However, the relationship in neighborhood predictions could be complex (i.e. beyond homophily), making heuristic aggregation such as averaging/summing/etc. overly simplistic and not generalizable to all types of graphs. Ideally, we want to design a general and powerful mechanism that flexibly aggregates graph information. This requirement could be perfectly

⊗}√(⊕]{⌋˜̂

/

/

/

/

/

/

/

/

/

/

〈˜{√]√glyph[arrowbt]

⊕]{⌋˜̂

…

…

ˆ

η

V

cor

-

calib

V

cor

-

test

DiffQuantile(

{

V

(

X

i

, Y

i

)

}

,

ˆ

α

)

V

(

X

1

, Y

1

)

V

(

X

2

, Y

2

)

V

(

X

M

, Y

M

)

…

GNN

ϑ

˜

µ

(

X

1

)

˜

µ

(

X

2

)

…

˜

µ

(

X

N

)

Prediction Interval Length Proxy

ϑ

L

=

1

N

∑

i

∈V

ct

∑

k

σ

((

V

(

X

i

, k

)

-

ˆ

η

)

/τ

)

L

=

1

N

∑

i

∈V

ct

(˜

µ

1

-

α/

2

(

X

i

) + ˆ

η

)

-

(˜

µ

α/

2

(

L

=

1

N

∑

i

∈V

ct

(˜

µ

1

-

α/

2

(

X

i

) + ˆ

η

)

-

(˜

µ

α/

2

(

X

i

)

-

ˆ

η

)

∇

L

∮{˜˜˜]̂]˜{̂glyph[arrowbt](⊔∐√(˜√glyph[arrowvertexdbl]˜˜{((∑∐]√(}˜(⊗}̂˜√

fulfilled by GNN message passing as it represents a class of learnable aggregation functions over graphs. Therefore, we use a separate GNN learner parameterized by ϑ for the same network G but with modified input node features; specifically, we use the base GNN prediction ( X 0 = ̂ µ ( X ) ) as input, and output ˜ µ ( X ) = GNN ϑ ( ̂ µ ( X ) , G ) . We will then use ˜ µ ( X ) as the input for constructing conformal prediction sets/intervals. Note that this second GNN model is a post-hoc process and only requires base GNN predictions, instead of access to the base GNN model.

Training with conformal-aware inefficiency loss. Given the hypothesis class, it remains to devise a concrete recipe for training the correction model GNN ϑ parameters. Recall that as with many other prediction models, a GNN model is typically trained to optimize prediction loss (i.e.cross-entropy loss or mean squared error) but not geared towards efficiency for the post-hoc conformal prediction step. We design GNN ϑ to be efficiency-aware by proposing a differentiable inefficiency loss that GNN ϑ optimizes over; this allows integration of GNN message passing to exploit the neighborhood information and also ensures a good ˜ µ ( · ) that leads to efficient prediction sets in downstream steps.

We first withhold a small fraction γ of the calibration dataset and use it for the correction procedure. The remaining data is used as the 'usual' calibration data for building the conformal prediction set. We then further split the withheld data into a correction calibration set V cor -cal and correction testing set V cor -test , to simulate the downstream conformal inference step. Given ˜ µ ( X ) = GNN ϑ ( ̂ µ ( X ) , G ) and a target miscoverage rate α , the framework follows three steps:

̂ Since the non-conformity score is usually differentiable, it only requires differentiable quantile calculation where there are well-established methods available [6, 5].

(2) Differentiable inefficiency proxy: we then construct a differentiable proxy c of the inefficiency on V cor -test by simulating the downstream conformal prediction procedures. We propose general formulas to construct c that are applicable for any classification and regression tasks respectively:

a. Inefficiency loss instantiation for Classification : The desirable proxy is to simulate the prediction set size using D cor -test as the 'test' data and D cor -calib as the 'calibration' data. For class k and node i in D cor -test , the non-conformity score is V ( X i , k ) for class k , where V ( · ) , for instance, is the APS score in Section 2. Then, we define the inefficiency proxy as

b. Inefficiency loss instantiation for Regression : The desirable proxy is to simulate the prediction interval length. For node i in V cor -test , the conformal prediction interval is [˜ µ α/ 2 ( X i ) -̂ η, ˜ µ 1 -α/ 2 ( X i ) + ̂ η ] . Thus, the per-sample prediction interval length could be directly calculated as where σ ( x ) = 1 1+ e -x is the sigmoid function and τ is a temperature hyper-parameter [40]. It can be interpreted as a soft assignment of class k to the prediction set. When τ → 0 , it becomes a hard assignment. The per-sample inefficiency proxy is then readily constructed as c i = 1 |Y| ∑ k ∈Y c i,k .

√∖√∐⌈√˜(/˜∖

<latexi sh

1_b

64="UH+m

7gjckEJ

K

wv uG

q

0

>A

C

X

Z

T

M

FI

d

B

W

R

S

z

L

V

p

r

y8

o

D

Y

Q

n/5

O

3

f

P

9

N

2

<latexi sh

1_b

64="

gy/MTS

5C+9

vdA

DI

Num0J

>

H

c

Z

L

F

Yn

V

r

W

G

R

K

p

2

Q

k

E

B

j

qz8

O

3

U

X

o

w

7

<latexi sh

1_b

64="zR

SC2PoU

XZK0y

TW

j+

A

>

c

VD

N

FJ3

vqB

ru

p

w

E

Y

9Q

m

/

8

G

n

d

g

I

H

O

Mf

5

7

<latexi sh

1_b

64="N

Lf

+CVwudJ

3

/k

FD

HQ

>A

B9

c

EK2

W

OX

I

G

m

M

v

Y

p

T

n

7

j

o

P8

0

q

Z

r

z

R

U

y

5

<latexi sh

1_b

64="

L2

JwdW

p7n

A

0

Ur

MB

c

I

>

9H

VD

S

N

F

f

Z

u

g

k

G

O

y

8

m

T

o

/

v

Y

z

R

j

X

5

q

+

C

K

Q

3

P

E

<latexi sh

1_b

64="rCI

W8mGJ

0

F

OS

p

kA

>

B9H

c

VDL

N

2

f

Zdu

g

Q

y

n

T

o

/

U

v

Y

z

R

j

X

5

q

+

7

w

K

M

3

P

E

<latexi sh

1_b

64="

Z

wnC

gWf

z

v

BG

M

uI+HE

>A

3

c

VDLS

N

FJ

U

2

d

kp

8

m

0

7

Y

O

PyK

T

X

59

q

r

/

j

R

Q

o

<latexi sh

1_b

64="HqUjdfRQ

Lc

N5

p

Z

SOM

>A

B+3

VD

FJ

2vW

u

k

8

E

n

w

C

m

z

I

/

o

0

Y

T

X

7

g

r

y

9

G

P

K

<latexi sh

1_b

64="

H

z+

Q

u7XW

BD

/c

>A

3

V

LS

N

FJ

U

2v

d

p

8

E

n

Zw

C

Mm

I

o

0R

Y

T

5g qr

j

y

9

G

K

<latexi sh

1_b

64="M/3q2

U9vp

NOw yW

5VG8K

Y

>A

CFn

c

DLSg

BEJz

r

H

Po

I

m

f

d

Z

R

Q

u

+

j

0

X

7

k

T

<latexi sh

1_b

64="5/

+SN28

CcQM

L

dnr

w

>A

jX

VF

v9

KqP

g

7

p

m

u

o

W

D

Z

y

zJH

GE

B

k

0

T

3

R

O

U

I

f

Y

<latexi sh

1_b

64="NFwun

LIQ

0

CmU

+

v9

GPY

>A

H3

c

VD

SgM

r

q

E

B

KT

R

y

zdj

Z

7

J

8

f

k

p

X

5

oO

/

W

2

<latexi sh

1_b

64="75GRuQ3n

UB

w

PE

>A

CfH

c

Fd

M

2

X

9

rog8

WmKJ

0

S

Nj

I

/

p

D

O

k

y

L

+

Y

<latexi sh

1_b

64="5/

+SN28

CcQM

L

dnr

w

>A

jX

VF

v9

KqP

g

7

p

m

u

o

W

D

Z

y

zJH

GE

B

k

0

T

3

R

O

U

I

f

Y

sh

64="

AN

7Mq

/E

B83

V

Ur

Cp

D

YGu

y

w

k

2

Q

<latexi

1_b

5gL

zOHTX

v0

>

c

S

9

I

W

K

R

F

Z

P

j

d

f

n

o

+

J

m

X

i

)

-

ˆ

η

)

Since GNN ϑ maps intervals to intervals and do not pose a constraint on the prediction, it may incur a trivial optimized solution where ˜ µ 1 -α/ 2 ( X ) < ˜ µ α/ 2 ( X ) . Thus, we pose an additional consistency regularization term: (˜ µ 1 -α/ 2 ( X ) -̂ µ 1 -α/ 2 ( X )) 2 +(˜ µ α/ 2 ( X ) -̂ µ α/ 2 ( X )) 2 . This regularizes the updated intervals to not deviate significantly to reach the trivial solution.

Conditional coverage. A natural question is whether optimizing the efficiency of conformal prediction may hurt its conditional validity. 5 In Section 5, we empirically demonstrate satisfactory conditional coverage across various graph features, which even improves upon the direct application of APS and CQR to graph data. We conjecture that it is because we correct for the correlation among nodes. However, theoretical understanding is left for future investigation.

(3) Inefficiency loss: finally, the inefficiency loss is an average of inefficiency proxies L ineff = 1 |V cor -test | ∑ i c i . The GNN ϑ is optimized using backpropagation in an end-to-end fashion.

Graph exchangeability. The post-hoc correction model is GNN -based, thus, it is permutationinvariant. Thus, it satisfies the exchangeability condition laid out in our theory in Section 3. Empirically, we demonstrate in Section 5 that CF-GNN achieves target empirical marginal coverage.

Computational cost. We remark that CF-GNN scales similarly as base GNN training since the correction step follows a standard GNN training procedure but with a modified input attribute and loss function. Notably, as the input to the correction model usually has a smaller attribute size (the number of classes for classification and 2 for regression), it has smaller parameter size than standard GNN training. In addition, it is also compatible with standard GNN mini-batching techniques.

General loss functions. Finally, we note that the choice of our loss function can be quite general. For instance, one may directly optimize for conditional validity by choosing a proper loss function.

5 Experiments

We conduct experiments to demonstrate the advantages of CF-GNN over other UQ methods in achieving empirical marginal coverage for graph data, as well as the efficiency improvement with CF-GNN. We also evaluate conditional coverage of CF-GNN and conduct systematic ablation and parameter analysis to show the robustness of CF-GNN.

Evaluation setup. For node classification, we follow a standard semi-supervised learning evaluation procedure [24], where we randomly split data into folds with 20%/10%/70% nodes as D train / D valid / D calib ∪ D test . For the node regression task, we follow a previous evaluation procedure from [20] and randomly split the data into folds with 50%/10%/40% nodes as D train / D valid / D calib ∪D test . We conduct 100 random splits of calibration/testing sets to estimate the empirical coverage. Using the test-time coverage distribution in Figure 2 to ensure that coverage is concentrated tightly around 1α , we modify the calibration set size to min { 1000 , |D calib ∪ D test | / 2 } , and use the rest as the test sample. For a fair comparison, we first train 10 runs of the base GNN model and then fix the predictions (i.e. the input to UQ baselines and CF-GNN). In this way, we ensure that the gain is not from randomness in base model training. The hyperparameter search strategy and configurations for CF-GNN and baselines can be found in Appendix D.1.

Models & baselines to evaluate coverage. For classification, we first use general statistical calibration approaches including temperate scaling [13], vector scaling [13], ensemble temperate scaling [49]. We also use SOTA GNN-specific calibration learners including CaGCN [44] and GATS [17]. The prediction set is the set of classes from highest to lowest scores until accumulative scores exceed 1α . For regression, we construct prediction intervals using quantile regression (QR) [25], Monte Carlo dropouts (MC dropout) [9], and Bayesian loss to model both aleatoric and epistemic uncertainty [23]. More information about baselines can be found in Appendix D.2.

Models & baselines to evaluate efficiency. As smaller coverage always leads to higher efficiency, for a fair comparison, we can only compare methods on efficiency that achieve the same coverage. Thus, we do not evaluate UQ baselines here since they do not produce exact coverage and are thus not comparable. While any CP-based methods produce exact coverage, to the best of our knowledge,

Task

UQ Model

DBLP

CiteSeer

PubMed

Computers

Photo

CS

Physics

Covered?

Node

classif.

Temp. Scale.

0.946

±

.003

7

0.920

±

.009

7

0.952

±

.004

"

0.899

±

.002

7

0.929

±

.002

7

0.962

±

.002

"

0.957

±

.001

"

0.969

±

.000

"

7

Vector Scale.

0.944

±

.004

7

0.921

±

.009

7

0.951

±

.004

"

0.899

±

.003

7

0.932

±

.002

7

0.963

±

.002

"

0.958

±

.001

"

0.969

±

.000

"

7

Ensemble TS

0.947

±

.003

7

0.920

±

.008

7

0.953

±

.003

"

0.899

±

.002

7

0.930

±

.002

7

0.964

±

.002

"

0.958

±

.001

"

0.969

±

.000

"

7

CaGCN

0.939

±

.005

7

0.922

±

.004

7

0.949

±

.005

7

0.898

±

.003

7

0.926

±

.003

7

0.956

±

.002

"

0.954

±

.003

"

0.968

±

.001

"

7

GATS

0.939

±

.005

7

0.921

±

.004

7

0.951

±

.005

"

0.898

±

.002

7

0.925

±

.002

7

0.957

±

.002

"

0.957

±

.001

"

0.968

±

.000

"

7

CF-GNN

0.952

±

.001

"

0.952

±

.001

"

0.953

±

.001

"

0.953

±

.001

"

0.952

±

.001

"

0.953

±

.001

"

0.952

±

.001

"

0.952

±

.001

"

"

Task

UQ Model

Anaheim

Chicago

Education

Election

Income

Unemploy.

Twitch

Covered?

Node

regress.

QR

0.943

±

.031

7

0.950

±

.007

7

0.959

±

.001

"

0.956

±

.004

"

0.960

±

.005

"

0.954

±

.004

"

0.900

±

.015

7

7

MCdropout

0.553

±

.022

7

0.427

±

.015

7

0.423

±

.013

7

0.417

±

.008

7

0.532

±

.022

7

0.489

±

.016

7

0.448

±

.017

7

7

BayesianNN 0.967

±

.001

"

0.955

±

.003

"

0.957

±

.002

"

0.958

±

.009

"

0.970

±

.004

"

0.960

±

.001

"

0.923

±

.006

7

7

CF-GNN

0.957

±

.003

"

0.954

±

.002

"

0.951

±

.001

"

0.950

±

.001

"

0.951

±

.001

"

0.951

±

.001

"

0.954

±

.001

"

"

there are no existing graph-based conformal prediction methods for transductive settings. Thus, we can only compare with the direct application of conformal prediction (CP) to base GNN. In the main text, we only show results for GCN [24] as the base model; results of three additional popular GNN models (GraphSAGE [16], GAT [42], and SGC [45]) are deferred to Appendix D.4.

Datasets. We evaluate CF-GNN on 8 node classification datasets and 7 node regression datasets with diverse network types such as geographical network, transportation network, social network, citation network, and financial network. Dataset statistics are in Appendix D.3.

5.1 Results

CF-GNN achieves empirical marginal coverage while existing UQ methods do not. We report marginal coverage of various UQ methods with target coverage at 95% (Table 1). There are three key takeaways. Firstly, none of these UQ methods achieves the target coverage for all datasets while CF-GNN does, highlighting the lack of statistical rigor in those methods and the necessity for a guaranteed UQ method. Secondly, it validates our theory from Section 3 that CF-GNN achieves designated coverage in transductive GNN predictions. Lastly, CF-GNN achieves empirical coverage that is close to the target coverage while baseline UQ methods are not. This controllable feature of CF-GNN is practically useful for practitioners that aim for a specified coverage in settings such as planning and selection.

CF-GNN significantly reduces inefficiency. We report empirical inefficiency for 8 classification and 7 regression datasets (Table 2). We observe that we achieve consistent improvement across datasets with up to 74% reduction in the prediction set size/interval length. We additionally conduct the same experiments for 3 other GNN models including GAT, GraphSAGE, and SGC in Appendix D.4 and we observe that performance gain is generalizable to diverse architecture choices. Furthermore, CF-GNN yields more efficient prediction sets than existing UQ methods even if we manually adjust the nominal level of them to achieve 95% empirical coverage (it is however impossible to do so in practice, here we do this for evaluation). For instance, the best calibration method GATS yields an average prediction size of 1.82 on Cora when the nominal level is tuned to achieve 95% empirical coverage. In contrast, CF-GNN has an average size of 1.76, smaller than GATS. In Appendix D.5, we also observe that CF-GNN also reduces inefficiency for advanced conformal predictor RAPS for classification task. In addition, we find that CF-GNN yields little changes to the prediction accuracy of the original GNN model (Appendix D.7).

CF-GNN empirically maintains conditional coverage. While CF-GNN achieves marginal coverage, it is highly desirable to have a method that achieves reasonable conditional coverage, which was the motivation of APS and CQR. We follow [37] to evaluate conditional coverage via the Worst-Slice (WS) coverage, which takes the worst coverage across slices in the feature space (i.e. node input features). We observe that CF-GNN achieves a WS coverage close to 1 -α , indicating satisfactory conditional coverage (Cond. Cov. (Input Feat.) row in Table 3). Besides the raw features, for each node, we also construct several network features (which are label agnostic) including clustering

coefficients, betweenness, PageRank, closeness, load, and harmonic centrality, and then calculate the WS coverage over the network feature space. We observe close to 95% WS coverage for various network features, suggesting CF-GNN also achieves robust conditional coverage over network properties. We also see that the direct application of CP (i.e. without graph correction) has much smaller WS coverage for classification, suggesting that adjusting for neigborhood information in CF-GNN implicitly improves conditional coverage.

Ablation. We conduct ablations in Table 4 to test two main components in CF-GNN, topology-aware correction model, and inefficiency loss. We first remove the inefficiency loss and replace it with standard prediction loss. The performance drops as expected, showing the power of directly modeling inefficiency loss in the correction step. Secondly, we replace the GNN correction model with an MLP correction model. The performance drops significantly, showing the importance of the design choice of correction model and justifying our motivation on inefficiency correlation over networks.

Parameter analysis. We conduct additional parameter analysis to test the robustness of CF-GNN. We first adjust the target coverage rate and calculate the inefficiency (Figure 5(1)). CF-GNN consistently beats the vanilla CP across all target coverages. Moreover, we adjust the fraction γ of the holdout calibration data in building the inefficiency loss, and observe that CF-GNN achieves consistent improvement in inefficiency (Figure 5(2)). We also observe a small fraction (10%) leads to excellent performance, showing that our model only requires a small amount of data for the inefficiency loss and leaves the majority of the calibration data for downstream conformal prediction.

6 Related Works

We discuss here related works that are closest to the ideas in CF-GNN and provide extended discussion on other related works in Appendix E.

(1) Uncertainty quantification (UQ) for GNN: Many UQ methods have been proposed to construct model-agnostic uncertain estimates for both classification [13, 49, 14, 27, 1] and regression [25, 41,

1

2

38, 9, 28, 26, 35, 23, 19]. Recently, specialized calibration methods for GNNs that leverage network principles such as homophily have been developed [44, 17]. However, these UQ methods can fail to provide a statistically rigorous and empirically valid coverage guarantee (see Table 1). In contrast, CF-GNN achieves valid marginal coverage in both theory and practice.

(2) Conformal prediction for GNN: The application of CP to graph-structured data remains largely unexplored. [7] claims that nodes in the graph are not exchangeable in the inductive setting and employs the framework of [3] to construct prediction sets using neighborhood nodes as the calibration data for mitigating the miscoverage due to non-exchangeability. In contrast, we study the transductive setting where certain exchangeability property holds; thus, the method from [3] are not comparable to ours. Concurrent with our work, [15] studies the exchangeability under transductive setting and proposes a diffusion-based method for improving efficiency, which can be viewed as an instantiation of our approach where the GNN correction learns an identity mapping; [33] studies exchangeability in network regression for of non-conformity scores based on various network structures, with similar observations as our Theorem 3. Other recent efforts in conformal prediction for graphs include [32, 34] which focus on distinct problem settings.

(3) Efficiency of conformal prediction: How to achieve desirable properties beyond validity is an active topic in the CP community; we focus on the efficiency aspect here. One line of work designs 'good' non-conformity scores in theory such as APS [37] and CQR [36]. More recent works take another approach, by modifying the training process of the prediction model. CF-GNN falls into the latter case, although our idea applies to any non-conformity score. ConfTr [40] also modifies training for improved efficiency. Our approach differs from theirs in significant ways. First, we develop a theory on CP validity on the graph data and leverage topological principles that are specialized to graph to improve efficiency while ConfTr focuses on i.i.d. vision image data. Also, ConfTr happens during base model training using the training set, while CF-GNN conducts post-hoc correction using withheld calibration data without assuming access to base model training, making ConfTr not comparable to us. Finally, we also propose a novel loss for efficiency in regression tasks.

7 Conclusion

In this work, we extend conformal prediction to GNNs by laying out the theoretical conditions for finite-sample validity and proposing a flexible graph-based CP framework to improve efficiency. Potential directions for future work include generalizing the inefficiency loss to other desirable CP properties such as robustness and conditional coverage; extensions to inductive settings or transductive but non-random split settings; extensions to other graph tasks such as link prediction, community detection, and so on.

References

Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U Rajendra Acharya, et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion , 76:243-297, 2021.

Anastasios Angelopoulos, Stephen Bates, Jitendra Malik, and Michael I Jordan. Uncertainty sets for image classifiers using conformal prediction. ICLR , 2021.

Rina Foygel Barber, Emmanuel J Candès, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. arXiv:2202.13415 , 2022.

Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael Jordan. Distribution-free, risk-controlling prediction sets. Journal of the ACM (JACM) , 68(6):1-34, 2021.

Mathieu Blondel, Olivier Teboul, Quentin Berthet, and Josip Djolonga. Fast differentiable sorting and ranking. In ICML , pages 950-959. PMLR, 2020.

Victor Chernozhukov, Iván Fernández-Val, and Alfred Galichon. Quantile and probability curves without crossing. Econometrica , 78(3):1093-1125, 2010.

Jase Clarkson. Distribution free prediction sets for node classification. LoG , 2022.

Rina Foygel Barber, Emmanuel J Candès, Aaditya Ramdas, and Ryan J Tibshirani. The limits of distribution-free conditional predictive inference. Information and Inference: A Journal of the IMA , 10(2):455-482, 2021.

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML , pages 1050-1059. PMLR, 2016.

Jiayi Gao, Jiaxing Li, Ke Zhang, and Youyong Kong. Topology uncertainty modeling for imbalanced node classification on graphs. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1-5. IEEE, 2023.

Isaac Gibbs and Emmanuel Candès. Adaptive conformal inference under distribution shift. NeurIPS , 34, 2021.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In ICML , pages 1263-1272. JMLR. org, 2017.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In ICML , pages 1321-1330. PMLR, 2017.

Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchisescu, and Richard Hartley. Calibration of neural networks using splines. ICLR , 2021.

Soroush H. Zargarbashi, Simone Antonelli, and Aleksandar Bojchevski. Conformal prediction sets for graph neural networks. In ICML , 2023.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. NeurIPS , 30, 2017.

Hans Hao-Hsun Hsu, Yuesong Shen, Christian Tomani, and Daniel Cremers. What makes graph neural networks miscalibrated? NeurIPS , 2022.

Vladislav Ishimtsev, Alexander Bernstein, Evgeny Burnaev, and Ivan Nazarov. Conformal k -nn anomaly detector for univariate data streams. In Conformal and Probabilistic Prediction and Applications , pages 213-227. PMLR, 2017.

Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson. What are bayesian neural network posteriors really like? In ICML , pages 4629-4640. PMLR, 2021.

Junteng Jia and Austion R Benson. Residual correlation in graph neural network regression. In KDD , pages 588-598, 2020.

Ying Jin and Emmanuel J Candès. Selection by prediction with conformal p-values. arXiv:2210.01408 , 2022.

Ying Jin, Zhimei Ren, and Emmanuel J Candès. Sensitivity analysis of individual treatment effects: A robust conformal inference approach. Proceedings of the National Academy of Sciences , 120(6):e2214889120, 2023.

Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? NeurIPS , 30, 2017.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. ICLR , 2017.

Roger Koenker and Kevin F Hallock. Quantile regression. Journal of Economic Perspectives , 15(4):143-156, 2001.

Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In ICML , pages 2796-2804. PMLR, 2018.

Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. NeurIPS , 32, 2019.

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. NeurIPS , 30, 2017.

Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Alexander Pritzel, Suman Ravuri, Timo Ewalds, Ferran Alet, Zach Eaton-Rosen, et al. Graphcast: Learning skillful medium-range global weather forecasting. arXiv:2212.12794 , 2022.

Lihua Lei and Emmanuel J Candès. Conformal inference of counterfactuals and individual treatment effects. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 2021.

Michelle M Li, Kexin Huang, and Marinka Zitnik. Graph representation learning in biomedicine and healthcare. Nature Biomedical Engineering , pages 1-17, 2022.

Robert Lunde. On the validity of conformal prediction for network data under non-uniform sampling. arXiv preprint arXiv:2306.07252 , 2023.

Robert Lunde, Elizaveta Levina, and Ji Zhu. Conformal prediction for network-assisted regression. arXiv:2302.10095 , 2023.

Ariane Marandon. Conformal link prediction to control the error rate. arXiv preprint arXiv:2306.14693 , 2023.

Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. NeurIPS , 32, 2019.

Yaniv Romano, Evan Patterson, and Emmanuel Candès. Conformalized quantile regression. NeurIPS , 32, 2019.

Yaniv Romano, Matteo Sesia, and Emmanuel Candès. Classification with valid and adaptive coverage. NeurIPS , 33:3581-3591, 2020.

Simon J Sheather and James Stephen Marron. Kernel quantile estimators. Journal of the American Statistical Association , 85(410):410-416, 1990.

Maximilian Stadler, Bertrand Charpentier, Simon Geisler, Daniel Zügner, and Stephan Günnemann. Graph posterior network: Bayesian predictive uncertainty for node classification. Advances in Neural Information Processing Systems , 34:18033-18048, 2021.

David Stutz, Ali Taylan Cemgil, Arnaud Doucet, et al. Learning optimal conformal classifiers. ICLR , 2022.

Ichiro Takeuchi, Quoc Le, Timothy Sears, Alexander Smola, et al. Nonparametric quantile estimation. 2006.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. ICLR , 2018.

Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world . Springer Science & Business Media, 2005.

Xiao Wang, Hongrui Liu, Chuan Shi, and Cheng Yang. Be confident! towards trustworthy graph neural networks via confidence calibration. NeurIPS , 34:23768-23779, 2021.

Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In ICML , pages 6861-6871. PMLR, 2019.

Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. Graph neural networks in recommender systems: a survey. ACM Computing Surveys , 55(5):1-37, 2022.

Mingzhang Yin, Claudia Shi, Yixin Wang, and David M Blei. Conformal sensitivity analysis for individual treatment effects. Journal of the American Statistical Association , pages 1-14, 2022.

Margaux Zaffran, Olivier Féron, Yannig Goude, Julie Josse, and Aymeric Dieuleveut. Adaptive conformal predictions for time series. In ICML , pages 25834-25866. PMLR, 2022.

Jize Zhang, Bhavya Kailkhura, and T Yong-Jin Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In ICML , pages 11117-11128. PMLR, 2020.

Xujiang Zhao, Feng Chen, Shu Hu, and Jin-Hee Cho. Uncertainty aware semi-supervised learning on graph data. Advances in Neural Information Processing Systems , 33:12827-12836, 2020.

A Deferred details for Section 3

A.1 Technical proofs for theoretical results

Proof of Lemma 2. Hereafter, we condition on the entire unordered graph, all the attribute and label information, and the index sets D train and D ct. We define the scores evaluated at the original node indices as

By Assumption 1, for any permutation π of D ct, we always have

That is, given D ct, the evaluated score at any v ∈ D ct remains invariant no matter which subset of D ct is designated as D calib. This implies that the scores are fixed after conditioning:

where we use [] to emphasize unordered sets. Thus, the calibration scores { V i } n i =1 is a subset of size n of [ v v ] v ∈D ct . Note that under random splitting in the transductive setting, any permutation π of D ct occurs with the same probability, which gives the conditional probability in Lemma 2.

Proof of Theorem 3. Throughout this proof, we condition on the entire unordered graph, all the attribute and label information, and the index sets D train and D ct. By Lemma 2, after the conditioning, the unordered set of { V i } n + m i =1 is fixed as [ v v ] v ∈D ct , and { V i } n i =1 is a simple random sample from [ v v ] v ∈D ct . As a result, any test sample V ( X n + j , Y n + j ) , j = 1 , . . . , m is exchangeable with { V i } n i =1 . By standard theory for conformal prediction [43], this ensures valid marginal coverage , i.e., P ( Y n + j ∈ C ( X n + j )) ≥ 1 -α , where the expectation is over all the randomness.

̂ We now proceed to analyze the distribution of ̂ Cover . For notational convenience, we write N = m + n , and view D ct as the 'population'. In this way, { V i } n i =1 is a simple random sample from [ v v ] v ∈D ct . For every η ∈ R , we define the 'population' cumulative distribution function (c.d.f.)

which is a deterministic function. We also define the calibration c.d.f. as

which is random, and its randomness comes from which subset of D ct is D calib. By definition,

Since the scores have no ties, we know

The test-time coverage can be written as

Now we characterize the distribution of ̂ η . For any η ∈ R , by the definition of ̂ η , P ( ̂ η ≤ η ) = P ( n ̂ F n ( η ) ≥ ( n +1)(1 -q ) ) = P ( n ̂ F n ( η ) ≥ glyph[ceilingleft] ( n +1)(1 -q ) glyph[ceilingright] ) .

Note that n ̂ F n ( η ) = ∑ v ∈D calib 1 { v v ≤ η } is the count of data in D calib such that the score is below η . By the simple random sample (i.e., sampling without replacement), n ̂ F n ( η ) follows a hyper-geometric distribution with parameter N,n,NF ( η ) . That is,

Denoting the c.d.f. of hypergeometric distribution as Φ HG ( k ; N,n,NF ( η )) , we have

Then, for any t ∈ [0 , 1] ,

Since F ( · ) is monotonely increasing,

where F -1 ( s ) = inf { η : F ( η ) ≥ s } for any s ∈ [0 , 1] . Plugging in the previous results on the distribution of η , we have

where the second equality uses the fact that F ( η ) ∈ { 0 , 1 /N,.. . , ( N -1) /N, 1 } , hence F ( F -1 ( s )) = glyph[ceilingleft] Ns glyph[ceilingright] /N for any s ∈ [0 , 1] . By tower property, such an equation also holds for the unconditional distribution, marginalized over all the randomness. This completes the proof of Theorem 3.

A.2 Additional visualization of test-time coverage

In this part, we provide more visualization of the distributions of test time coverage ̂ Cover under various sample size configurations. We note that such results also apply to standard application of split conformal prediction when the non-conformity score function V is independent of calibration and test samples, so that Assumption 1 is satisfied.

Figures 6 and 7 plot the p.d.f. of ̂ Cover for α = 0 . 05 and α = 0 . 1 , respectively, when fixing n and varying the test sample size m . The y -axis is obtained by computing P ( t k -1 < ̂ Cover ≤ t k ) / ( t k -t k -1 ) at x = ( t k -1 + t k ) / 2 for a sequence of evenly spaced { t k } ∈ [0 , 1] . All figures in this paper for p.d.f.s are obtained in the same way. We see that ̂ Cover concentrates more tightly around the target value 1 -α as m and n increases.

Figures 8 and 9 plot the p.d.f. of ̂ Cover for α = 0 . 05 and α = 0 . 1 , respectively, where we fix N = m + n but vary the calibration sample size n . This mimics the situation where the total number of nodes on the graph is fixed, while we may have flexibility in collecting data as the calibration set. We observe a tradeoff between the calibration accuracy determined by n and the test-sample concentration determined by n . The distribution of ̂ Cover is more concentrated around 1 -α when m and n are relatively balanced.

0

10

20

30

0.85

0.90

0.95

1.00

coverage

p.d.f.

m

50

100

250

500

750

1000

0

10

20

30

40

50

0.85

0.90

0.95

1.00

coverage

p.d.f.

m

100

250

500

750

1000

2000

0

20

40

60

0.85

0.90

0.95

1.00

coverage

p.d.f.

m

100

250

500

750

1000

2000

0

5

10

15

20

25

0.75

0.80

0.85

0.90

0.95

1.00

coverage

p.d.f.

m

50

100

250

500

750

1000

0

10

20

30

0.75

0.80

0.85

0.90

0.95

1.00

coverage

p.d.f.

m

100

250

500

750

1000

2000

0

10

20

30

40

0.75

0.80

0.85

0.90

0.95

1.00

coverage

p.d.f.

m

100

250

500

750

1000

2000

0

5

10

15

20

0.85

0.90

0.95

1.00

coverage

p.d.f.

n

50

200

300

400

450

0

10

20

30

0.85

0.90

0.95

1.00

coverage

p.d.f.

n

100

250

500

750

900

0

10

20

30

40

0.85

0.90

0.95

1.00

coverage

p.d.f.

n

500

1000

1250

1500

1750

0

5

10

15

0.75

0.80

0.85

0.90

0.95

1.00

coverage

p.d.f.

n

50

200

300

400

450

0

5

10

15

20

0.75

0.80

0.85

0.90

0.95

1.00

coverage

p.d.f.

n

100

250

500

750

900

0

10

20

30

0.75

0.80

0.85

0.90

0.95

1.00

coverage

p.d.f.

n

500

1000

1250

1500

1750

B Discussion on full conformal prediction, split conformal prediction

In this part, we discuss the relation of our application of conformal prediction to full conformal prediction and split conformal prediction, two prominent conformal prediction methods proposed by Vovk and his coauthors in [43]. Split conformal prediction is mostly widely used due to its computational efficiency, where exchangeability is usually ensured by independence (which is not obvious for graph data) as we discussed briefly in the introduction.

Full conformal prediction (FCP) is arguably the most versatile form of conformal prediction. Given calibration data Z i = ( X i , Y i ) ∈ X × Y , i = 1 , . . . , n , and given a test point X n +1 ∈ X whose outcome Y n +1 ∈ Y is unknown, at every hypothesized value y ∈ Y , FCP uses any algorithm S to train the following scores

where S is symmetric in the arguments Z 1 , . . . , Z i -1 , Z i +1 , . . . , Z n , ( X n +1 , y ) , as well as

Here, for 1 ≤ i ≤ n , S y i intuitively measures how well the observation ( X i , Y i ) conforms to the observations Z 1 , . . . , Z i -1 , Z i +1 , . . . , Z n , ( X n +1 , y ) with the hypothesized value of y . For instance, when using a linear prediction model, it can be chosen as the prediction residual

S y i = | Y i -X glyph[latticetop] i ̂ θ y | , where ̂ θ y is the ordinary least squares coefficient by a linear regression of Y 1 , . . . , Y i -1 , Y i +1 , . . . , Y n , y over X 1 , . . . , X i -1 , X i +1 , . . . , X n , X n +1 . More generally, one may train a prediction model ̂ µ y : X → Y using Z 1 , . . . , Z i -1 , Z i +1 , . . . , Z n , ( X n +1 , y ) , and set S y i = | Y i -̂ µ y ( X i ) | . For a confidence level α ∈ (0 , 1) , the FCP prediction set is then

Since the original form of FCP involves training n + 1 models at each hypothesized value y , its computation can be very intense. It is thus impractical to directly apply FCP to GNN models (i.e., imagining S as the GNN training process on the entire graph with a hypothesized outcome y ).

Split conformal prediction (SCP) is a computationally-efficient special case of FCP that is most widely used for i.i.d. data. The idea is to set aside an independent fold of data to output a single trained model. To be specific, we assume access to a given non-conformity score V : X × Y → R , i.i.d. calibration data Z i = ( X i , Y i ) n i =1 , and an independent test sample ( X n +1 , Y n +1 ) from the same distribution with Y n +1 unobserved. Here by a 'given' score, we mean that it is obtained without knowing the calibration and test sample; usually, it is trained on an independent set of data { ( X j , Y j ) } j ∈D train before seeing the calibration and test sample. Then define V i = V ( X i , Y i ) for i = 1 , . . . , n . The SCP prediction set is

The above set is usually convenient to compute, because we only need one single model to obtain V . The validity of SCP usually relies on the independence of V to calibration and test data as we mentioned in the introduction. However, the application of SCP to GNN model is also not straightforward: as we discussed in the main text, the model training step already uses the calibration and test samples, and the nodes are correlated.

Indeed, our method can be seen as a middle ground between FCP and SCP: it only requires one single prediction model as SCP does, but allows to use calibration and test data in the training step as FCP does. In our method introduced in the main text, there exists a fixed function V : Y × Y → R (provided by APS and CQR) such that

S y i = V ( ̂ µ ( X i ) , Y i ) , S y n +1 = V ( ̂ µ ( X n +1 , y ) , where ̂ µ is the final output from the second GNN model whose training process does not utilize the outcomes Y 1 , . . . , Y n and y , but uses the features X 1 , . . . , X n and X n +1 .

Algorithm 1: Pseudo-code for CF-GNN algorithm.

C Algorithm overview

We describe the pseudo-code of CF-GNN in Algorithm 1.

D Deferred details for experiments

D.1 Hyperparameters

Table 5 reports our set of hyperparameter ranges. We conduct 100 iterations of Bayesian Optimization for CF-GNN with the validation set inefficiency proxy as the optimization metric. To avoid overfitting, each iteration only uses the first GNN run. The optimized hyperparameters are then used for all 10 GNN runs and we then reported the average and standard deviation across runs. Each experiment is done with a single NVIDIA 2080 Ti RTX 11GB GPU.

D.2 Baseline Details

We report the details about baselines below and the hyperparameter range in Table 6.

Temperature Scaling [13] divides the logits with a learnable scalar. It is optimized over NLL loss in the validation set.

Vector Scaling [13] has a scalar to scale the logits for each class dimension and adds an additional classwide bias. It is optimized over NLL loss in the validation set.

Ensemble Temperature Scaling [49] learns an ensemble of uncalibrated, temperature-scaled calibrated calibrators.

CaGCN [44] uses an additional GCN model that learns a temperature scalar for each node based on its neighborhood information.

GATS [17] identifies five factors that affect GNN calibration and designs a model that accounts for these five factors by using per-node temperature scaling and attentive aggregation from the local neighborhood.

QR [25] uses a pinball loss to produce quantile scores. It is CQR without the conformal prediction adjustment.

MCdropout [9] turns on dropout during evaluation and produces K predictions. We then take the 95% quantile of the predicted distribution. We also experimented with taking a 95% confidence interval but 95% quantile has better coverage, thus we adopt the quantile approach.

BayesianNN [23] model the label with normal distribution and the model produces two heads, where one corresponds to the mean and the second log variance. We then calculate the standard deviation as the square root of the exponent of log variance. Then we take the [mean-1.96*standard deviation, mean+1.96*standard deviation] for the 95% interval.

D.3 Dataset

For node classification, we use the common node classification datasets in Pytorch Geometric package. For node regression, we use datasets in [20]. We report the dataset statistics at Table 7.

D.4 Marginal coverage and inefficiency across GNN architectures

Weadditionally conduct marginal coverage and inefficiency comparisons of CF-GNN over the vanilla CP across 4 different GNN architectures: GCN, GAT, GraphSAGE, and SGC. The result for marginal

DBLP

CiteSeer

PubMed

Computers

Photo

CS

Physics

Anaheim

Chicago

Education

Election

Income

Unemploy

Twitch

0.0

0.5

1.0

Empirical Coverage

CF-GNN Base

GCN

GraphSAGE

GAT

SGC

coverage is in Figure 10. The result for inefficiency is in Table 8. We observe consistent improvement in inefficiency reduction across these architectures, suggesting CF-GNN is a GNN-agnostic efficiency improvement approach.

D.5 CF-GNN with Regularized Adaptive Prediction Sets

To further showcase that CF-GNN is a versatile framework that adapts to any advancement in non-conformity scores, we experiment on RAPS [2], which regularizes APS to produce a smaller prediction set size. We report the performance using the GCN backbond in Table 9. We observe that CF-GNN still obtains impressive inefficiency reduction compared to the vanilla application of RAPS to GNN.

D.6 Conditional coverage on full set of network features

We report the full set of network features and calculate the worse-slice coverage in Table 10 for target coverage of 0.9 and Table 11 for a target coverage of 0.95. We observe that CF-GNN achieves satisfactory conditional coverage across a wide range of diverse network features.

D.7 Prediction accuracy versus uncertainty calibration

As we discussed in the main text, the original GNN trained towards optimal prediction accuracy does not necessarily yield the most efficient prediction model; this is corrected by the second GNN in CF-GNN which improves the efficiency of conformal prediction sets/intervals. With our approach, one can use the output of the original GNN for point prediction while that of the second GNN for efficient uncertainty quantification, without necessarily overwriting the first accurate prediction model. However, a natural question here still remains, which is that whether applying the second

GNN drastically changes the prediction accuracy. This question is more relevant to the classification problem since for regression our method only adjusts the confidence band. For classification, we consider top-1 class prediction as the 'point prediction'. We present its accuracy "Before" and "After" the correction in Table 12, which shows that this correction typically does not result in a visible change in accuracy. In addition, in a new experiment on Cora, we find that 100% of the top-1 class from the base GNN are in CF-GNN's prediction sets. The potential to develop steps that explicitly consider point prediction accuracy is an exciting avenue for future research.

E Extended Related Works

Uncertainty quantification for graph neural networks. Uncertainty quantification (UQ) is a wellstudied subject in general machine learning and also recently in GNNs. For multi-class classification, the raw prediction scores are often under/over-confident and thus various calibration methods are proposed for valid uncertainty estimation such as temperate scaling [13], vector scaling [13], ensemble temperate scaling [49], and so on [14, 27, 39, 1]. Recently, specialized calibration methods that leverage network principles such as homophily have been developed: examples include CaGCN [44] and GATS [17]. In regression, various methods have been proposed to construct prediction intervals, such as quantile regression [25, 41, 38], bootstrapping with subsampling, model ensembles, and dropout initialization [9, 28, 26, 35], and bayesian approaches with strong modeling assumptions on parameter and data distributions [23, 19]. However, these UQ methods can fail to provide statistically

rigorous and empirically valid coverage guarantee (see Table 1). In contrast, CF-GNN achieves valid marginal coverage in both theory and practice. Uncertainty quantification has also been leveraged to deal with out-of-distribution detection and imbalanced data in graph neural networks [50, 10]. While it is not the focus here, we remark that conformal prediction can also be extended to tackle such issues [18], and it would be interesting to explore such applications for graph data.

Conformal prediction for graph neural networks. As we discussed, the application of conformal prediction to graph-structured data remains largely unexplored. At the time of submission, the only work we awared of is [7], who claims that nodes in the graph are not exchangeable in the inductive setting and employs the framework of [3] to construct conformal prediction sets using neighborhood nodes as the calibration data. In contrast, we study the transductive setting where certain exchangeablility property holds and allows for flexibility in the training step. We also study the efficiency aspect that is absent in [7]. In addition, there have been concurrent works [15, 33] that observe similar exchangeability and validity of conformal prediction in either transductive setting or other network models. In particular, [15] proposes a diffusion-based method that aggregates nonconformity scores of neighbor nodes to improve efficiency, while our approach learns the aggregation of neighbor scores, which is more general than their approach. [32] studies the exchangeability for node regression under certain network models instead of our transductive setting with GNNs, and without considering the efficiency aspect. With a growing recent interest in conformal prediction for graphs, there are even more recent works that focus on validity [32] and link prediction [34].

Efficiency of conformal prediction. While conformal prediction enjoys distribution-free coverage for any non-conformity score based on any prediction model, its efficiency (i.e., size of prediction sets or length of prediction intervals) varies with specific choice of the scores and models. How to achieve desirable properties such as efficiency is a topic under intense research in conformal prediction. To this end, one major thread designs good non-conformity scores such as APS [37] and CQR [36]. More recent works take another approach, by modifying the training process of the prediction model to

further improve efficiency. This work falls into the latter case. Our idea applies to any non-conformity scores, as demonstrated with APS and CQR, two prominent examples of the former case. Related to our work, ConfTr [40] also simulates conformal prediction so as to train a prediction model that eventually leads to more efficient conformal prediction sets. However, our approach differs from theirs in significant ways. First, ConfTr modifies model training, while CF-GNN conducts post-hoc correction without changing the original prediction. Second, ConfTr uses the training set to simultaneously optimize model prediction and efficiency of conformal prediction, while we withhold a fraction of calibration data to optimize the efficiency. Third, our approach specifically leverages the rich topological information in graph-structured data to achieve more improvement in efficiency. Finally, we also propose a novel loss for efficiency in regression tasks.

Peer reviews:
comments: SUMMARY:
Conformal Prediction (CP) outputs a prediction set that contains the true label with a certain likelihood given assumptions on exchangeability. It is a well-known and popular uncertainty quantification (UQ) technique.  The authors propose a technique that unites GNNs and CP called conformalized GNN (CF-GNN). The technique is crucially permutation equivariant and topology aware.

STRENGTHS:
The paper tackles an interesting problem of unifying CP and GNNs. UQ is an important area of research and CP offers strong sound guarantees to tackle such a problem. The paper is well-written and I appreciated that the authors keep important considerations such as permutation equivariance, efficiency and topology in mind when developing their technique. The evaluation is extensive and shows good results.

WEAKNESSES:
The technique focuses on a transductive setting only. While this is still a valid and interesting setting, the fact that the technique focuses on this setting only is a weakness.

QUESTIONS:
Is there a reason for the focus on the transductive setting? Are there any clear challenges in inductive node-level or inductive graph-level tasks for instance. I believe inductive graph-level tasks to be an extremely interesting approach as doing UQ on molecular predictions may be a very important application. Another important application could also be link prediction.

LIMITATIONS:
The authors address the transductive focus as a limitation which is an important consideration. I don't foresee potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-30

comments: SUMMARY:
The authors propose Conformalized Graph Neural Networks (CF-GNNs), which extends conformal prediction to graphs for uncertainty quantification. The framework allows a GNN to produce confidence intervals for its predictions, based on an uncertainty estimation on a withhold calibration set. Under permutation invariance condition, they provide theoretical guarantee for the coverage of test-time confidence intervals. Furthermore, a learnable correction model is introduced to empirically produce more efficient prediction intervals. Experiments show that CF-GNNs acheive a better coverage rate over baselines and meanwhile have a more efficient (smaller) confidence interval than a naive conformal prediction.

STRENGTHS:
- [originality] the authors extend conformal prediction to node-level task and give theoretical coverage guarantee. As far as I know, this adaption and theorical results are new.
- [significance] this work makes solid theoretical and empirical contributions to uncertainty quantifications of graphs. The proposed method has a greater coverage rate and a smaller prediction intervals size.

WEAKNESSES:
See Questions.

QUESTIONS:
- At Line 276 the authors mention "As smaller coverage always leads to higher efficiency, for a fair comparison, we can only compare methods on efficiency that achieve the same coverage. Thus, we do not evaluate UQ baselines here since they do not produce exact coverage and are thus not comparable". Do you think there is a way to make some of the other UQ methods empirically reach $(1-\alpha)$ coverage so that we can get some senses of their size of prediction sets for comparison?

LIMITATIONS:
No limitation discussed. No potential negative social impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new approach, known as conformalized Graph Neural Networks (CF-GNN), designed to bring reliable uncertainty estimates to graph-structured data prediction models. The study's primary contribution is the innovative adaptation of conformal prediction (CP) to Graph Neural Networks (GNNs). The proposed CF-GNN model is capable of generating prediction sets or intervals that encapsulate the true label, with a level of coverage probability (90%) that can be predefined.\n\nSTRENGTHS:\nThe idea of extending conformal prediction to GNNs is quite interesting and timely. The authors establish a permutation invariance condition that justifies the application of CP on graph data and gives a precise outline of test-time coverage. The empirical evidence from numerous experiments validates the effectiveness of CF-GNN, demonstrating that it can meet any specified target marginal coverage while drastically reducing prediction set or interval size by up to 74% when compared to baseline models. The overall presentation is easy-to-follow, and the technical contribution is non-trivial.\n\nWEAKNESSES:\nI don't see major flaws in this manuscript, but the introduction to how to construct correlation datasets can be more clear. In addition, the manuscript seems to miss a portion of works [1, 2, 3] on quantifying the uncertainty in GNN predictions. If the proposal of conformal prediction is to quantifying the uncertainty of GNN prediction, then other approaches on uncertainty quantification of node classification with GNNs should not be neglected.\n\n[1] Stadler, Maximilian, et al. \"Graph posterior network: Bayesian predictive uncertainty for node classification.\" Advances in Neural Information Processing Systems 34 (2021): 18033-18048.  \n[2] Zhao, Xujiang, et al. \"Uncertainty aware semi-supervised learning on graph data.\" Advances in Neural Information Processing Systems 33 (2020): 12827-12836.  \n[3] Gao, Jiayi, et al. \"Topology Uncertainty Modeling For Imbalanced Node Classification on Graphs.\" ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.\n\nQUESTIONS:\nPlease refer to the Weakness section.\n\nLIMITATIONS:\nI am not seeing limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a conformal prediction method tailored for graph-structured data. The proposed correction method is topology-aware and based on an empirical observation that inefficiencies correlate highly with network edges. The method updates node predictions based on its neighbors, and it is trainable alongside the GNN model. They also show how regular conformal prediction methods work under\n\nSTRENGTHS:\n[1] The well-motivated problem, clean writing, and detailed related works.\n\n[2] The first conformal prediction method for graph-structured data with exchangeability and validity assumptions is GNN agnostic and intuitive.\n\n[3] Rigorous proof and method to show exchangeability and validity of conformal prediction on graph-structured data for the first time in the literature.\n\n[4] Capable of achieving conditional coverage, which is a stronger version of marginal coverage.\n\n[5] In-depth experiments and ablation studies show the efficacy and efficiency of the proposed method.\n\nWEAKNESSES:\n[1] Inductive settings for GNN problems are more realistic compared to transductive settings. It is also not motivated why authors start with a transductive setting. \n\n[2] There could be ablation studies/experiments over the conformity score functions, such as testing with RAPS[1].\n\n[3] Even though the original coverage definition is used to measure inefficiency, for graph-structured data, there is an inherent non-IIDness in the data. Therefore each sample having equal weights in the coverage calculation is not appropriate. Weighting based on the degree of a node could be a great idea. Would love to discuss this part during the rebuttals.\n\n[4] Why is there no experimental comparison with DAPS[2] (ICML23), which is also a conformal prediction method for node prediction? \n\n[5] The paper's assumption of exchangeability is strict. What happens if exchangeability does not hold?\n\n\n[1] Angelopoulos, Anastasios, et al. \"Uncertainty sets for image classifiers using conformal prediction.\" arXiv preprint arXiv:2009.14193 (2020).\n\n[2[ Zargarbashi, Soroush H., Simone Antonelli, and Aleksandar Bojchevski. \"Conformal Prediction Sets for Graph Neural Networks.\" (2023).\n\nQUESTIONS:\n(1) Why do authors not tackle with inductive node classification problem?\n\n(2) What happens if permutation invariance does not hold?\n\n(3) What is the reason for not using RAPS[1]?\n\n(5) In lines 133-134, while employing conformal prediction. is it even possible to change the trained prediction?\n\n(6) My intuitive understanding is that if inefficiency is correlated with the network edges, why degree does not affect the coverage? I feel that coverage for graph data should be redefined for graph-structured data. For example, each sample could be weighted based on the degree of the node.\n\n(7) In what circumstances does exchangeability not hold?\n\n\n[1] Angelopoulos, Anastasios, et al. \"Uncertainty sets for image classifiers using conformal prediction.\" arXiv preprint arXiv:2009.14193 (2020).\n\nLIMITATIONS:\nThe authors have addressed limitations perfectly. As they also mentioned, this method is valid for transductive settings, but it will not be optimal for inductive settings. Authors also plan to extend it for inductive settings and link prediction tasks. Also, this method is heavily based on the exchangeability assumptions. It is unclear how to modify the algorithm because exchangeability assumptions do not hold.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of providing faithful and \"efficient\" uncertainty estimates for GNNs. Here faithful means the unknown groundtruth is contained in the prediction set with a probability higher than a threshold; efficient means the prediction set should be as small as possible. Specifically, the proposed method is based on the conformal prediction method, which uses a separate calibration set to determine the threshold used to decide whether a class (for classification) is included in the prediction set. \n\nThe authors first show that conformal prediction can be applied to GNNs as long as the samples in the calibration and test sets are exchangeable. This further leads to the observation that the test time coverage of the predictions fluctuates a lot when the number of test samples is small. The authors term this problem as the \"inefficiency\" during conformal prediction.\n\nTo improve both faithfulness and efficiency of the uncertainty estimates, the authors propose to train a new GNN that uses the original GNN's predictions as input and adjusted predictions as output. Instead of employing the original predictive loss, the paper proposes a loss that is a differentiable proxy of the efficiency metric. This model is trained with a pseudo calibration and test set split from the original validation set.\n\nExperiments show better uncertainty estimates compared to well-known approaches such as MC dropout, although the baselines are not designed specifically for GNNs.\n\nSTRENGTHS:\n- The paper formally justifies that conformal prediction can be used under the exchangeability assumption.\n\n- The proposed method is simple yet effective: directly optimizing for better uncertainty quantification on a potentially much smaller model.\n\nWEAKNESSES:\n- It seems that the calibration GNN needs to be trained for every $\\alpha$ separately if we want to have multiple calibration thresholds. It would be nice to show whether it is possible to train a model that adapts to multiple $\\alpha$s.\n\n- It would be useful to show the tradeoff between the predictive performance and the uncertainty quantification performance. For example, the change in top-1 accuracy of the calibrated and original prediction.\n\n- Could the calibration GNN suffer from overfitting? In an extreme case, if $\\mathcal{V}_{\\mathrm{cor \\_cal}}$  and $\\mathcal{V}_{\\mathrm{cor \\_test}}$ are both small, it might be possible that the calibration GNN minimizes its loss by overfitting the labels and make all Vs trivially 0? Maybe this is one reason for only giving the calibration GNN the output of the base GNN?\n\nQUESTIONS:\n- To evaluate $\\hat{\\eta}$, all samples from the calibration set are needed. If the graph is too large to be processed in a single batch, is it possible to train the calibration GNN in a mini-batch fashion? How would this affect the performance?\n\n- It would be nice to show the training cost of the proposed method. I would expect training the calibration GNN could be much faster, but it is better to have more quantitative results.\n\nLIMITATIONS:\nI do not see potential negative societal impact of the work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}