SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As we discussed in the main text, the original GNN trained towards optimal prediction accuracy does not necessarily yield the most efficient prediction model; this is corrected by the second GNN in CF-GNN which improves the efficiency of conformal prediction sets/intervals. With our approach, one can use the output of the original GNN for point prediction while that of the second GNN for efficient uncertainty quantification, without necessarily overwriting the first accurate prediction model. However, a natural question here still remains, which is that whether applying the second
Table 8: Empirical inefficiency measured by the size/length of the prediction set/interval for node classification/regression. The result uses APS for classification and CQR for regression. We report the average and standard deviation calculated from 10 GNN runs with each run of 100 conformal splits.
Task	GNN Model	GCN	GraphSAGE	GAT	SGC
Dataset	CP→CF-GNN	CP→CF-GNN	CP→CF-GNN	CP→CF-GNN
Node classifier.	Cora	3.80±.28→-58.44%→1.58±.22	6.73±.19→-76.50%→1.58±.15	4.14±.16→-62.53%→1.55±.10	3.88±.19→-62.23%→1.47±.10
DBLP	2.43±.03→-49.20%→1.23±.01	3.91±.01→-68.27%→1.24±.01	2.02±.06→-37.51%→1.26±.01	2.44±.04→-48.93%→1.24±.02
CiteSeer	3.86±.11→-74.50%→0.99±.01	5.88±.02→-83.07%→1.00±.01	3.18±.25→-68.56%→1.00±.01	3.79±.14→-73.43%→1.01±.02
PubMed	1.60±.02→-19.44%→1.29±.04	1.93±.28→-36.95%→1.22±.03	1.37±.02→-10.54%→1.23±.02	1.60±.02→-24.27%→1.21±.02
Computers	3.56±.13→-50.22%→1.77±.11	6.00±.10→-45.74%→3.26±.48	2.33±.11→-8.18%→2.14±.12	3.44±.14→-45.69%→1.87±.15
Photo	3.79±.13→-57.03%→1.63±.17	4.52±.47→-37.22%→2.84±.67	2.24±.21→-19.38%→1.81±.16	3.81±.14→-62.86%→1.41±.05
CS	7.79±.29→-55.83%→3.44±.33	14.68±.02→-88.63%→1.67±.14	6.87±.48→-73.08%→1.85±.10	7.76±.25→-73.92%→2.02±.22
Physics	3.11±.07→-65.36%→1.08±.10	4.91±.01→-72.97%→1.33±.08	2.00±.19→-45.23%→1.09±.06	3.10±.08→-57.22%→1.32±.12
Average Improvement	-53.75%	-63.75%	-40.63%	-56.07%
Node regress.	Anaheim	2.89±.39→-25.00%→2.17±.11	2.37±.05→-23.12%→1.82±.07	3.12±.38→-31.27%→2.14±.11	2.94±.24→-24.90%→2.21±.16
Chicago	2.05±.07→-0.48%→2.04±.17	2.08±.05→-7.90%→1.92±.09	1.95±.04→-68.15%→0.62±.93	2.02±.03→-1.37%→1.99±.07
Education	2.56±.02→-5.07%→2.43±.05	2.20±.04→+8.44%→2.38±.08	2.48±.05→-2.76%→2.41±.04	2.55±.02→-2.80%→2.48±.04
Election	0.90±.01→+0.21%→0.90±.02	0.87±.01→-0.80%→0.86±.02	0.89±.00→-1.23%→0.88±.02	0.90±.00→-0.42%→0.90±.02
Income	2.51±.12→-4.58%→2.40±.05	2.08±.04→+32.23%→2.75±.23	2.35±.03→-0.23%→2.34±.07	2.42±.01→+3.04%→2.49±.04
Unemploy.	2.72±.03→-10.83%→2.43±.04	2.75±.06→-12.90%→2.39±.05	2.80±.08→-14.56%→2.40±.04	2.72±.02→-11.05%→2.42±.04
Twitch	2.43±.10→-1.36%→2.39±.07	2.48±.09→-3.06%→2.40±.07	2.50±.14→-5.53%→2.36±.07	2.42±.08→-1.43%→2.38±.06
Average Improvement	-6.73%	-1.02%	-17.68%	-5.56%
Table 9: Comparison with other non-conformity scores that reduce inefficiency.
Size	CP → CF-GNN
Cora	1.67±.11-15.35%→1.42±.05
DBLP	1.39±.02-5.00%→1.32±.01
CiteSeer	1.30±.07-19.85%→1.04±.04
PubMed	1.23±.01+2.40%→1.26±.02
Computers	1.58±.02-4.59%→1.51±.05
Photo	1.34±.01-10.47%→1.20±.01
CS	1.29±.04-6.13%→1.21±.02
GNN drastically changes the prediction accuracy. This question is more relevant to the classification problem since for regression our method only adjusts the confidence band. For classification, we consider top-1 class prediction as the "point prediction". We present its accuracy "Before" and "After" the correction in Table 12, which shows that this correction typically does not result in a visible change in accuracy. In addition, in a new experiment on Cora, we find that  $100\%$  of the top-1 class from the base GNN are in CF-GNN's prediction sets. The potential to develop steps that explicitly consider point prediction accuracy is an exciting avenue for future research.
E Extended Related Works
Uncertainty quantification for graph neural networks. Uncertainty quantification (UQ) is a well-studied subject in general machine learning and also recently in GNNs. For multi-class classification, the raw prediction scores are often under/over-confident and thus various calibration methods are proposed for valid uncertainty estimation such as temperate scaling [13], vector scaling [13], ensemble temperate scaling [49], and so on [14, 27, 39, 1]. Recently, specialized calibration methods that leverage network principles such as homophily have been developed: examples include CaGCN [44] and GATS [17]. In regression, various methods have been proposed to construct prediction intervals, such as quantile regression [25, 41, 38], bootstrapping with subsampling, model ensembles, and dropout initialization [9, 28, 26, 35], and bayesian approaches with strong modeling assumptions on parameter and data distributions [23, 19]. However, these UQ methods can fail to provide statistically
Table 10: CF-GNN achieves conditional coverage. We use Cora/Twitch as an example classification/regression dataset.
Target: 0.9	Classification	Regression
Model	CP	CF-GNN	CP	CF-GNN
Marginal Cov.	0.90±.02	0.90±.01	0.91±.02	0.91±.03
Cond. Cov. (Input Feat.)	0.89±.04	0.90±.03	0.90±.07	0.86±.08
Cond. Cov. (Cluster)	0.82±.07	0.89±.03	0.90±.06	0.88±.07
Cond. Cov. (Between)	0.82±.06	0.89±.03	0.86±.08	0.88±.07
Cond. Cov. (PageRank)	0.71±.08	0.87±.05	0.87±.09	0.89±.07
Cond. Cov. (Load)	0.83±.05	0.90±.03	0.86±.08	0.88±.07
Cond. Cov. (Harmonic)	0.89±.04	0.87±.05	0.88±.08	0.91±.06
Cond. Cov. (Degree)	0.79±.05	0.89±.04	0.86±.08	0.89±.06
Table 11: CF-GNN achieves conditional coverage. We use Cora/Twitch as an example classification/regression dataset.
Target: 0.95	Classification	Regression
Model	CP	CF-GNN	CP	CF-GNN
Marginal Cov.	0.95±.01	0.95±.01	0.96±.02	0.96±.02
Cond. Cov. (Input Feat.)	0.94±.02	0.94±.03	0.95±.04	0.94±.05
Cond. Cov. (Cluster)	0.89±.06	0.93±.04	0.96±.03	0.96±.03
Cond. Cov. (Between)	0.81±.06	0.95±.03	0.94±.05	0.94±.05
Cond. Cov. (PageRank)	0.78±.06	0.94±.03	0.94±.05	0.94±.05
Cond. Cov. (Load)	0.81±.06	0.94±.03	0.94±.05	0.95±.05
Cond. Cov. (Harmonic)	0.88±.04	0.95±.03	0.96±.04	0.95±.04
Cond. Cov. (Degree)	0.83±.05	0.88±.06	0.94±.04	0.94±.04
rigorous and empirically valid coverage guarantee (see Table 1). In contrast, CF-GNN achieves valid marginal coverage in both theory and practice. Uncertainty quantification has also been leveraged to deal with out-of-distribution detection and imbalanced data in graph neural networks [50, 10]. While it is not the focus here, we remark that conformal prediction can also be extended to tackle such issues [18], and it would be interesting to explore such applications for graph data.
Conformal prediction for graph neural networks. As we discussed, the application of conformal prediction to graph-structured data remains largely unexplored. At the time of submission, the only work we aware of is [7], who claims that nodes in the graph are not exchangeable in the inductive setting and employs the framework of [3] to construct conformal prediction sets using neighborhood nodes as the calibration data. In contrast, we study the transductive setting where certain exchangeability property holds and allows for flexibility in the training step. We also study the efficiency aspect that is absent in [7]. In addition, there have been concurrent works [15, 33] that observe similar exchangeability and validity of conformal prediction in either transductive setting or other network models. In particular, [15] proposes a diffusion-based method that aggregates nonconformity scores of neighbor nodes to improve efficiency, while our approach learns the aggregation of neighbor scores, which is more general than their approach. [32] studies the exchangeability for node regression under certain network models instead of our transductive setting with GNNs, and without considering the efficiency aspect. With a growing recent interest in conformal prediction for graphs, there are even more recent works that focus on validity [32] and link prediction [34].
Efficiency of conformal prediction. While conformal prediction enjoys distribution-free coverage for any non-conformity score based on any prediction model, its efficiency (i.e., size of prediction sets or length of prediction intervals) varies with specific choice of the scores and models. How to achieve desirable properties such as efficiency is a topic under intense research in conformal prediction. To this end, one major thread designs good non-conformity scores such as APS [37] and CQR [36]. More recent works take another approach, by modifying the training process of the prediction model to
Table 12: CF-GNN does not change the top-1 class prediction accuracy for classification tasks.
Dataset	Before	After
Cora	0.844±0.004	0.843±0.016
DBLP	0.835±0.001	0.832±0.002
CiteSeer	0.913±0.002	0.911±0.002
further improve efficiency. This work falls into the latter case. Our idea applies to any non-conformity scores, as demonstrated with APS and CQR, two prominent examples of the former case. Related to our work, ConfTr [40] also simulates conformal prediction so as to train a prediction model that eventually leads to more efficient conformal prediction sets. However, our approach differs from theirs in significant ways. First, ConfTr modifies model training, while CF-GNN conducts post-hoc correction without changing the original prediction. Second, ConfTr uses the training set to simultaneously optimize model prediction and efficiency of conformal prediction, while we withhold a fraction of calibration data to optimize the efficiency. Third, our approach specifically leverages the rich topological information in graph-structured data to achieve more improvement in efficiency. Finally, we also propose a novel loss for efficiency in regression tasks.