SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this work, we extend conformal prediction to GNNs by laying out the theoretical conditions for finite-sample validity and proposing a flexible graph-based CP framework to improve efficiency. Potential directions for future work include generalizing the inefficiency loss to other desirable CP properties such as robustness and conditional coverage; extensions to inductive settings or transductive but non-random split settings; extensions to other graph tasks such as link prediction, community detection, and so on.
K.H. and J.L. gratefully acknowledge the support of DARPA under Nos. HR00112190039 (TAMI), N660011924033 (MCS); ARO under Nos. W911NF-16-1-0342 (MURI), W911NF-16-1-0171 (DURIP); NSF under Nos. OAC-1835598 (CINES), OAC-1934578 (HDR), CCF-1918940 (Expeditions), NIH under No. 3U54HG010426-04S1 (HuBMAP), Stanford Data Science Initiative, Wu Tsai Neurosciences Institute, Amazon, Docomo, GSK, Hitachi, Intel, JPMorgan Chase, Juniper Networks, KDDI, NEC, and Toshiba. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding entities.
[46] Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. Graph neural networks in recommender systems: a survey. ACM Computing Surveys, 55(5):1-37, 2022.
[47] Mingzhang Yin, Claudia Shi, Yixin Wang, and David M Blei. Conformal sensitivity analysis for individual treatment effects. Journal of the American Statistical Association, pages 1-14, 2022.
[48] Margaux Zaffran, Olivier Feron, Yannig Goude, Julie Josse, and Aymeric Dieuleveut. Adaptive conformal predictions for time series. In ICML, pages 25834-25866. PMLR, 2022.
[49] Jize Zhang, Bhavya Kailkhura, and T Yong-Jin Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In ICML, pages 11117-11128. PMLR, 2020.
[50] Xujiang Zhao, Feng Chen, Shu Hu, and Jin-Hee Cho. Uncertainty aware semi-supervised learning on graph data. Advances in Neural Information Processing Systems, 33:12827-12836, 2020.
A Deferred details for Section 3
A.1 Technical proofs for theoretical results
Proof of Lemma 2. Hereafter, we condition on the entire unordered graph, all the attribute and label information, and the index sets  $\mathcal{D}_{\mathrm{train}}$  and  $\mathcal{D}_{\mathrm{ct}}$ . We define the scores evaluated at the original node indices as
$$
v _ {v} = V (\mathbf {x} _ {v}, y _ {v}; \{z _ {i} \} _ {i \in \mathcal {D} _ {\text {t r a i n}} \cup \mathcal {D} _ {\text {v a l i d}}}, \{\mathbf {x} _ {v} \} _ {v \in \mathcal {D} _ {\text {c a l i b}} \cup \mathcal {D} _ {\text {t e s t}}}, \mathcal {V}, \mathcal {E}), v \in \mathcal {D} _ {\text {c a l i b}} \cup \mathcal {D} _ {\text {t e s t}} \subseteq \mathcal {V}.
$$
By Assumption 1, for any permutation  $\pi$  of  $\mathcal{D}_{\mathrm{ct}}$ , we always have
$$
v _ {v} = V \left(\mathbf {x} _ {v}, y _ {v}; \{z _ {i} \} _ {i \in \mathcal {D} _ {\text {t r a i n}} \cup \mathcal {D} _ {\text {v a l i d}}}, \left\{\mathbf {x} _ {\pi (v)} \right\} _ {v \in \mathcal {D} _ {\text {c a l i b}} \cup \mathcal {D} _ {\text {t e s t}}}, \mathcal {V} _ {\pi}, \mathcal {E} _ {\pi}\right).
$$
That is, given  $\mathcal{D}_{\mathrm{ct}}$ , the evaluated score at any  $v \in \mathcal{D}_{\mathrm{ct}}$  remains invariant no matter which subset of  $\mathcal{D}_{\mathrm{ct}}$  is designated as  $\mathcal{D}_{\mathrm{calib}}$ . This implies that the scores are fixed after conditioning:
$$
[ V _ {1}, \dots , V _ {n + m} ] = [ v _ {v} ] _ {v \in \mathcal {D} _ {\mathrm {c t}}},
$$
where we use  $\left[\right]$  to emphasize unordered sets. Thus, the calibration scores  $\{V_i\}_{i=1}^n$  is a subset of size  $n$  of  $[v_v]_{v \in \mathcal{D}_{\mathrm{ct}}}$ . Note that under random splitting in the transductive setting, any permutation  $\pi$  of  $\mathcal{D}_{\mathrm{ct}}$  occurs with the same probability, which gives the conditional probability in Lemma 2.
Proof of Theorem 3. Throughout this proof, we condition on the entire unordered graph, all the attribute and label information, and the index sets  $\mathcal{D}_{\mathrm{train}}$  and  $\mathcal{D}_{\mathrm{ct}}$ . By Lemma 2, after the conditioning, the unordered set of  $\{V_i\}_{i=1}^{n+m}$  is fixed as  $[v_v]_{v \in \mathcal{D}_{\mathrm{ct}}}$ , and  $\{V_i\}_{i=1}^n$  is a simple random sample from  $[v_v]_{v \in \mathcal{D}_{\mathrm{ct}}}$ . As a result, any test sample  $V(X_{n+j}, Y_{n+j})$ ,  $j = 1, \ldots, m$  is exchangeable with  $\{V_i\}_{i=1}^n$ . By standard theory for conformal prediction [43], this ensures valid marginal coverage, i.e.,  $\mathbb{P}(Y_{n+j} \in \widehat{C}(X_{n+j})) \geq 1 - \alpha$ , where the expectation is over all the randomness.
We now proceed to analyze the distribution of  $\widehat{\text{Cover}}$ . For notational convenience, we write  $N = m + n$ , and view  $\mathcal{D}_{\mathrm{ct}}$  as the 'population'. In this way,  $\{V_i\}_{i=1}^n$  is a simple random sample from  $[v_v]_{v \in \mathcal{D}_{\mathrm{ct}}}$ . For every  $\eta \in \mathbb{R}$ , we define the 'population' cumulative distribution function (c.d.f.)
$$
F (\eta) = \frac {1}{N} \sum_ {v \in \mathcal {D} _ {\mathrm {c t}}} \mathbb {1} \left\{v _ {v} \leq \eta \right\},
$$
which is a deterministic function. We also define the calibration c.d.f. as
$$
\widehat {F} _ {n} (\eta) = \frac {1}{n} \sum_ {v \in \mathcal {D} _ {\mathrm {c a l i b}}} \mathbb {1} \{v _ {v} \leq \eta \} = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {1} \{V _ {i} \leq \eta \},
$$
which is random, and its randomness comes from which subset of  $\mathcal{D}_{\mathrm{ct}}$  is  $\mathcal{D}_{\mathrm{calib}}$ . By definition,
$$
\widehat {\eta} = \inf  \{\eta \colon \widehat {F} _ {n} (\eta) \geq (1 - q) (1 + 1 / n) \}.
$$
Since the scores have no ties, we know
$$
\hat {F} _ {n} (\hat {\eta}) = \lceil (1 - q) (n + 1) \rceil / n.
$$
The test-time coverage can be written as
$$
\begin{array}{l} \widehat {\text {C o v e r}} = \frac {1}{m} \sum_ {j = 1} ^ {m} \mathbb {1} \left\{V _ {n + j} \leq \widehat {\eta} \right\} \\ = \frac {1}{N - n} \left(\sum_ {v \in \mathcal {D} _ {\mathrm {c t}}} \mathbb {1} \left\{v _ {v} \leq \hat {\eta}\right) - \sum_ {v \in \mathcal {D} _ {\mathrm {c a l i b}}} \mathbb {1} \left\{v _ {v} \leq \hat {\eta}\right)\right) \\ = \frac {N}{N - n} F (\widehat {\eta}) - \frac {n}{N - n} \widehat {F} _ {n} (\widehat {\eta}) = \frac {N}{N - n} F (\widehat {\eta}) - \frac {\lceil (1 - q) (n + 1) \rceil}{N - n}. \\ \end{array}
$$
Now we characterize the distribution of  $\widehat{\eta}$ . For any  $\eta \in \mathbb{R}$ , by the definition of  $\widehat{\eta}$
$$
\mathbb {P} (\widehat {\eta} \leq \eta) = \mathbb {P} \big (n \widehat {F} _ {n} (\eta) \geq (n + 1) (1 - q) \big) = \mathbb {P} \big (n \widehat {F} _ {n} (\eta) \geq \lceil (n + 1) (1 - q) \rceil \big).
$$
Note that  $n\widehat{F}_n(\eta) = \sum_{v\in \mathcal{D}_{\mathrm{calib}}}\mathbb{1}\{v_v\leq \eta \}$  is the count of data in  $\mathcal{D}_{\mathrm{calib}}$  such that the score is below  $\eta$ . By the simple random sample (i.e., sampling without replacement),  $n\widehat{F}_n(\eta)$  follows a hyper-geometric distribution with parameter  $N,n,NF(\eta)$ . That is,
$$
\mathbb {P} (n \widehat {F} _ {n} (\eta) = k) = \frac {\binom {N F (\eta)} {k} \binom {N - N F (\eta)} {n - k}}{\binom {N} {n}}, \quad 0 \leq k \leq N F (\eta).
$$
Denoting the c.d.f. of hypergeometric distribution as  $\Phi_{\mathrm{HG}}(k;N,n,NF(\eta))$ , we have
$$
\mathbb {P} (\widehat {\eta} \leq \eta) = 1 - \Phi_ {\mathrm {H G}} \big (\lceil (n + 1) (1 - q) \rceil - 1; N, n, N F (\eta) \big).
$$
Then, for any  $t\in [0,1]$
$$
\begin{array}{l} \mathbb {P} \big (\widehat {\text {C o v e r}} \leq t \big) = \mathbb {P} \left(\frac {N}{N - n} F (\widehat {\eta}) - \frac {\lceil (1 - q) (n + 1) \rceil}{N - n} \leq t\right) \\ = \mathbb {P} \left(F (\widehat {\eta}) \leq \frac {\lceil (1 - q) (n + 1) \rceil + (N - n) t}{N}\right). \\ \end{array}
$$
Since  $F(\cdot)$  is monotonely increasing,
$$
\mathbb {P} \big (\widehat {\operatorname {C o v e r}} \leq t \big) = \mathbb {P} \bigg (\widehat {\eta} \leq F ^ {- 1} \bigg (\frac {\lceil (1 - q) (n + 1) \rceil + (N - n) t}{N} \bigg) \bigg),
$$
where  $F^{-1}(s) = \inf \{\eta \colon F(\eta) \geq s\}$  for any  $s \in [0,1]$ . Plugging in the previous results on the distribution of  $\widehat{\eta}$ , we have
$$
\begin{array}{l} \mathbb {P} \left(\widehat {\text {C o v e r}} \leq t\right) = 1 - \Phi_ {\mathrm {H G}} \left(\lceil (n + 1) (1 - q) \rceil - 1; N, n, N F \left(F ^ {- 1} \left(\frac {\lceil (1 - q) (n + 1) \rceil + (N - n) t}{N}\right)\right)\right) \\ = 1 - \Phi_ {\mathrm {H G}} \left(\lceil (n + 1) (1 - q) \rceil - 1; N, n, N \frac {\lceil (1 - q) (n + 1) \rceil + (N - n) t \rceil}{N}\right) \\ = 1 - \Phi_ {\mathrm {H G}} \left(\lceil (n + 1) (1 - q) \rceil - 1; N, n, \lceil \lceil (1 - q) (n + 1) \rceil + (N - n) t \rceil\right) \\ = 1 - \Phi_ {\mathrm {H G}} \left(\lceil (n + 1) (1 - q) \rceil - 1; N, n, \lceil (1 - q) (n + 1) \rceil + \lceil (N - n) t \rceil\right) \\ \end{array}
$$
where the second equality uses the fact that  $F(\eta) \in \{0,1 / N,\dots ,(N - 1) / N,1\}$ , hence  $F(F^{-1}(s)) = \lceil N s\rceil /N$  for any  $s \in [0,1]$ . By tower property, such an equation also holds for the unconditional distribution, marginalized over all the randomness. This completes the proof of Theorem 3.
A.2 Additional visualization of test-time coverage
In this part, we provide more visualization of the distributions of test time coverage  $\widehat{\text{Cover}}$  under various sample size configurations. We note that such results also apply to standard application of split conformal prediction when the non-conformity score function  $V$  is independent of calibration and test samples, so that Assumption 1 is satisfied.
Figures 6 and 7 plot the p.d.f. of  $\widehat{\text{Cover}}$  for  $\alpha = 0.05$  and  $\alpha = 0.1$ , respectively, when fixing  $n$  and varying the test sample size  $m$ . The  $y$ -axis is obtained by computing  $\mathbb{P}(t_{k-1} < \widehat{\text{Cover}} \leq t_k) / (t_k - t_{k-1})$  at  $x = (t_{k-1} + t_k) / 2$  for a sequence of evenly spaced  $\{t_k\} \in [0,1]$ . All figures in this paper for p.d.f.s are obtained in the same way. We see that  $\widehat{\text{Cover}}$  concentrates more tightly around the target value  $1 - \alpha$  as  $m$  and  $n$  increases.
Figures 8 and 9 plot the p.d.f. of  $\widehat{\text{Cover}}$  for  $\alpha = 0.05$  and  $\alpha = 0.1$ , respectively, where we fix  $N = m + n$  but vary the calibration sample size  $n$ . This mimics the situation where the total number of nodes on the graph is fixed, while we may have flexibility in collecting data as the calibration set. We observe a tradeoff between the calibration accuracy determined by  $n$  and the test-sample concentration determined by  $n$ . The distribution of  $\widehat{\text{Cover}}$  is more concentrated around  $1 - \alpha$  when  $m$  and  $n$  are relatively balanced.
Figure 6: P.d.f. of test-time coverage  $\widehat{\text{Cover}}$  for  $n = 500$  (left), 1000 (middle), 2000 (right) and  $\alpha = 0.05$  with curves representing different values of  $m$ , the test sample size.
Figure 7: P.d.f. of test-time coverage  $\widehat{\text{Cover}}$  for  $n = 500$  (left), 1000 (middle), 2000 (right) and  $\alpha = 0.1$  with curves representing different values of  $m$ , the test sample size.
Figure 8: P.d.f. of test-time coverage  $\widehat{\text{Cover}}$  for  $N = m + n = 500$  (left), 1000 (middle), 2000 (right) and  $\alpha = 0.05$  with curves representing different values of  $n$ , the calibration sample size.
Figure 9: P.d.f. of test-time coverage  $\widehat{\text{Cover}}$  for  $N = m + n = 500$  (left), 1000 (middle), 2000 (right) and  $\alpha = 0.1$  with curves representing different values of  $n$ , the calibration sample size.
B Discussion on full conformal prediction, split conformal prediction
In this part, we discuss the relation of our application of conformal prediction to full conformal prediction and split conformal prediction, two prominent conformal prediction methods proposed by Vovk and his coauthors in [43]. Split conformal prediction is mostly widely used due to its computational efficiency, where exchangeability is usually ensured by independence (which is not obvious for graph data) as we discussed briefly in the introduction.
Full conformal prediction (FCP) is arguably the most versatile form of conformal prediction. Given calibration data  $Z_{i} = (X_{i},Y_{i})\in \mathcal{X}\times \mathcal{Y}$ ,  $i = 1,\dots ,n$ , and given a test point  $X_{n + 1}\in \mathcal{X}$  whose outcome  $Y_{n + 1}\in \mathcal{V}$  is unknown, at every hypothesized value  $y\in \mathcal{V}$ , FCP uses any algorithm  $S$  to train the following scores
$$
S _ {i} ^ {y} = S \left(X _ {i}, Y _ {i}; Z _ {1}, \dots , Z _ {i - 1}, Z _ {i + 1}, \dots , Z _ {n}, \left(X _ {n + 1}, y\right)\right), \quad i = 1, \dots , n,
$$
where  $S$  is symmetric in the arguments  $Z_{1},\ldots ,Z_{i - 1},Z_{i + 1},\ldots ,Z_{n}$ $(X_{n + 1},y)$ , as well as
$$
S _ {n + 1} ^ {y} = S \left(X _ {n + 1}, y; Z _ {1}, \dots , Z _ {n}\right).
$$
Here, for  $1 \leq i \leq n$ ,  $S_{i}^{y}$  intuitively measures how well the observation  $(X_{i}, Y_{i})$  conforms to the observations  $Z_{1}, \ldots, Z_{i-1}, Z_{i+1}, \ldots, Z_{n}, (X_{n+1}, y)$  with the hypothesized value of  $y$ . For instance, when using a linear prediction model, it can be chosen as the prediction residual
$$
S _ {i} ^ {y} = | Y _ {i} - X _ {i} ^ {\top} \widehat {\theta} ^ {y} |,
$$
where  $\widehat{\theta}^y$  is the ordinary least squares coefficient by a linear regression of  $Y_{1},\ldots ,Y_{i - 1},Y_{i + 1},\ldots ,Y_{n},y$  over  $X_{1},\ldots ,X_{i - 1},X_{i + 1},\ldots ,X_{n},X_{n + 1}$ . More generally, one may train a prediction model  $\widehat{\mu}^{y}\colon \mathcal{X}\to \mathcal{Y}$  using  $Z_{1},\ldots ,Z_{i - 1},Z_{i + 1},\ldots ,Z_{n},(X_{n + 1},y)$ , and set  $S_i^y = |Y_i - \widehat{\mu}^y (X_i)|$ . For a confidence level  $\alpha \in (0,1)$ , the FCP prediction set is then
$$
\widehat {C} (X _ {n + 1}) := \left\{y \colon \frac {1 + \mathbb {1} \left\{S _ {i} ^ {y} > S _ {n + 1} ^ {y} \right\}}{n + 1} \leq \alpha \right\}.
$$
Since the original form of FCP involves training  $n + 1$  models at each hypothesized value  $y$ , its computation can be very intense. It is thus impractical to directly apply FCP to GNN models (i.e., imagining  $S$  as the GNN training process on the entire graph with a hypothesized outcome  $y$ ).
Split conformal prediction (SCP) is a computationally-efficient special case of FCP that is most widely used for i.i.d. data. The idea is to set aside an independent fold of data to output a single trained model. To be specific, we assume access to a given non-conformity score  $V \colon \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$ , i.i.d. calibration data  $Z_{i} = (X_{i}, Y_{i})_{i=1}^{n}$ , and an independent test sample  $(X_{n+1}, Y_{n+1})$  from the same distribution with  $Y_{n+1}$  unobserved. Here by a "given" score, we mean that it is obtained without knowing the calibration and test sample; usually, it is trained on an independent set of data  $\{(X_{j}, Y_{j})\}_{j \in \mathcal{D}_{\text{train}}}$  before seeing the calibration and test sample. Then define  $V_{i} = V(X_{i}, Y_{i})$  for  $i = 1, \ldots, n$ . The SCP prediction set is
$$
\widehat {C} (X _ {n + 1}) = \left\{y \colon \frac {1 + \mathbb {1} \left\{V _ {i} > V (X _ {n + 1} , y) \right\}}{n + 1} \leq \alpha \right\}.
$$
The above set is usually convenient to compute, because we only need one single model to obtain  $V$ . The validity of SCP usually relies on the independence of  $V$  to calibration and test data as we mentioned in the introduction. However, the application of SCP to GNN model is also not straightforward: as we discussed in the main text, the model training step already uses the calibration and test samples, and the nodes are correlated.
Indeed, our method can be seen as a middle ground between FCP and SCP: it only requires one single prediction model as SCP does, but allows to use calibration and test data in the training step as FCP does. In our method introduced in the main text, there exists a fixed function  $V\colon \mathcal{Y}\times \mathcal{Y}\to \mathbb{R}$  (provided by APS and CQR) such that
$$
S _ {i} ^ {y} = V (\widehat {\mu} (X _ {i}), Y _ {i}), \quad S _ {n + 1} ^ {y} = V (\widehat {\mu} (X _ {n + 1}, y),
$$
where  $\widehat{\mu}$  is the final output from the second GNN model whose training process does not utilize the outcomes  $Y_{1},\ldots ,Y_{n}$  and  $y$ , but uses the features  $X_{1},\ldots ,X_{n}$  and  $X_{n + 1}$ .
Algorithm 1: Pseudo-code for CF-GNN algorithm.
while not done do
Input: Graph  $G = (\mathcal{V},\mathcal{E},\mathbf{X})$ ; a trained base GNN model  $\mathrm{GNN}_{\theta}$ ; non-conformity score function  $V(X,Y)$ ; pre-specified mis-coverage rate  $\alpha$ , Randomly initialized  $\vartheta$  for the conformal correction model  $\mathrm{GNN}_{\vartheta}$ .
for  $i$  in  $\{1,\dots,|\mathcal{V}_{\mathrm{cor - calib}}\cup \mathcal{V}_{\mathrm{cor - test}}|\}$  do  $\begin{array}{rlr}{\widehat{\mu}(X_i)=\mathrm{GNN}_\theta(\mathbf{X}_i,G)}&{}&{\mathrm{~//~Base~GNN~outpuet~scores}}\\ {\widehat{\mu}(X_i)=\mathrm{GNN}_\vartheta(\widehat{\mu}(X_i),G)}&{}&{\mathrm{~//~Correction~model~output~scores}}\end{array}$    
end   
 $n,m = |\mathcal{V}_{\mathrm{cor - calib}}|,|\mathcal{V}_{\mathrm{cor - test}}|$  // Size of correction calib/test set   
 $\widehat{\alpha} = \frac{1}{n + 1} *\alpha$  // Finite-sample correction   
 $\widehat{\eta} = \mathrm{DiffQuantile}(\{V(X_i,Y_i)|i\in \mathcal{V}_{\mathrm{cor - calib}}\}) / /$  Compute non-conformity scores   
if Classification then   
 $\mathcal{L}_{\mathrm{Ineff}} = \frac{1}{m}\sum_{i\in \mathcal{V}_{\mathrm{cor - test}}}\frac{1}{|\mathcal{V}|}\sum_{k\in \mathcal{Y}}\sigma (\frac{V(X_i,k) - \widehat{\eta}}{\tau})$    
// Inefficiency proxy for classification tasks   
end   
if Regression then   
 $\mathcal{L}_{\mathrm{Ineff}} = \frac{1}{m}\sum_{i\in \mathcal{V}_{\mathrm{cor - test}}}(\tilde{\mu}_{1 - \alpha /2}(X)_i + \widehat{\eta}) - (\tilde{\mu}_{\alpha /2}(X)_i - \widehat{\eta})$    
// Inefficiency proxy for regression tasks   
 $\mathcal{L}_{\mathrm{Ineff}} + = \gamma \frac{1}{m}\sum_{i\in \mathcal{V}_{\mathrm{cor - test}}}(\tilde{\mu}_{1 - \alpha /2}(X)_i - \widehat{\mu}_{1 - \alpha /2}(X)_i)^2 +(\tilde{\mu}_{\alpha /2}(X)_i - \widehat{\mu}_{\alpha /2}(X)_i)^2$    
// Consistency regularization term   
end   
 $\vartheta \gets \vartheta -\nabla_{\vartheta}\mathcal{L}_{\mathrm{Ineff}}$  // Optimizing  $\vartheta$  to reduce inefficiency
C Algorithm overview
We describe the pseudo-code of CF-GNN in Algorithm 1.
D Deferred details for experiments
Table 5 reports our set of hyperparameter ranges. We conduct 100 iterations of Bayesian Optimization for CF-GNN with the validation set inefficiency proxy as the optimization metric. To avoid overfitting, each iteration only uses the first GNN run. The optimized hyperparameters are then used for all 10 GNN runs and we then reported the average and standard deviation across runs. Each experiment is done with a single NVIDIA 2080 Ti RTX 11GB GPU.
Table 5: Hyperparameter range for CF-GNN.
Task	Param.	Range
Classification	GNNθHidden dimension	[16,32,64,128,256]
Learning rate	[1e-1, 1e-2, 1e-3, 1e-4]
GNNθNumber of GNN Layers	[1,2,3,4]
GNNθBase Model	[GCN, GAT, GraphSAGE, SGC]
τ	[10, 1, 1e-1, 1e-2, 1e-3]
Regression	GNNθHidden dimension	[16,32,64,128,256]
Learning rate	[1e-1, 1e-2, 1e-3, 1e-4]
GNNθNumber of GNN Layers	[1, 2, 3, 4]
GNNθBase Model	[GCN, GAT, GraphSAGE, SGC]
Reg. loss coeff. γ	[1, 1e-1]
We report the details about baselines below and the hyperparameter range in Table 6.
1. Temperature Scaling [13] divides the logits with a learnable scalar. It is optimized over NLL loss in the validation set.
2. Vector Scaling [13] has a scalar to scale the logits for each class dimension and adds an additional classwide bias. It is optimized over NLL loss in the validation set.
3. Ensemble Temperature Scaling [49] learns an ensemble of uncalibrated, temperature-scaled calibrated calibrators.
4. CaGCN [44] uses an additional GCN model that learns a temperature scalar for each node based on its neighborhood information.
5. GATS [17] identifies five factors that affect GNN calibration and designs a model that accounts for these five factors by using per-node temperature scaling and attentive aggregation from the local neighborhood.
6. QR [25] uses a pinball loss to produce quantile scores. It is CQR without the conformal prediction adjustment.
7. MC dropout [9] turns on dropout during evaluation and produces  $K$  predictions. We then take the 95% quantile of the predicted distribution. We also experimented with taking a 95% confidence interval but 95% quantile has better coverage, thus we adopt the quantile approach.
8. BayesianNN [23] model the label with normal distribution and the model produces two heads, where one corresponds to the mean and the second log variance. We then calculate the standard deviation as the square root of the exponent of log variance. Then we take the [mean-1.96*standard deviation, mean+1.96*standard deviation] for the 95% interval.
Table 6: Hyperparameter range for baselines.
Baseline	Param.	Range
Temperature Scaling	No hyperparameter	Not Applicable
Vector Scaling	No hyperparameter	Not Applicable
Ensemble Temp Scaling	No hyperparameter	Not Applicable
CaGCN	Dropout	[0.3, 0.5, 0.7]
Hidden dimension	[16, 32, 64, 128, 256]
Number of GNN Layers	[1,2,3,4]
Weight Decay	[0, 1e-3, 1e-2, 1e-1]
GATS	Dropout	[0.3, 0.5, 0.7]
Hidden dimension	[16, 32, 64, 128, 256]
Number of GNN Layers	[1,2,3,4]
Weight Decay	[0, 1e-3, 1e-2, 1e-1]
MC Dropout	Number of Predictions	[100, 500, 1,000]
BayesianNN	No hyperparameter	Not Applicable
D.3 Dataset
For node classification, we use the common node classification datasets in Pytorch Geometric package. For node regression, we use datasets in [20]. We report the dataset statistics at Table 7.
D.4 Marginal coverage and inefficiency across GNN architectures
We additionally conduct marginal coverage and inefficiency comparisons of CF-GNN over the vanilla CP across 4 different GNN architectures: GCN, GAT, GraphSAGE, and SGC. The result for marginal
Table 7: Dataset statistics.
Domain	Dataset	Task	# Nodes	# Edges	# Features	# Labels
Citation	Cora	Classification	2,995	16,346	2,879	7
DBLP	Classification	17,716	105,734	1,639	4
CiteSeer	Classification	4,230	10,674	602	6
PubMed	Classification	19,717	88,648	500	3
Co-purchase	Computers	Classification	13,752	491,722	767	10
Photos	Classification	7,650	238,162	745	8
Co-author	CS	Classification	18,333	163,788	6,805	15
Physics	Classification	34,493	495,924	8,415	5
Transportation	Anaheim	Regression	914	3,881	4	-
Chicago	Regression	2,176	15,104	4	-
Geography	Education	Regression	3,234	12,717	6	-
Election	Regression	3,234	12,717	6	-
Income	Regression	3,234	12,717	6	-
Unemployment	Regression	3,234	12,717	6	-
Social	Twitch	Regression	1,912	31,299	3,170	-
Figure 10: Empirical coverage across 15 datasets with 10 independent runs of GNN, using CF-GNN.
coverage is in Figure 10. The result for inefficiency is in Table 8. We observe consistent improvement in inefficiency reduction across these architectures, suggesting CF-GNN is a GNN-agnostic efficiency improvement approach.
D.5 CF-GNN with Regularized Adaptive Prediction Sets
To further showcase that CF-GNN is a versatile framework that adapts to any advancement in non-conformity scores, we experiment on RAPS [2], which regularizes APS to produce a smaller prediction set size. We report the performance using the GCN backbond in Table 9. We observe that CF-GNN still obtains impressive inefficiency reduction compared to the vanilla application of RAPS to GNN.
D.6 Conditional coverage on full set of network features
We report the full set of network features and calculate the worse-slice coverage in Table 10 for target coverage of 0.9 and Table 11 for a target coverage of 0.95. We observe that CF-GNN achieves satisfactory conditional coverage across a wide range of diverse network features.
D.7 Prediction accuracy versus uncertainty calibration