SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Motivation: Inefficiency correlation. Our approach to boosting the scores is based on exploiting the correlation among connected nodes. Since the connected nodes usually represent entities that interact in the real world, there can be strong correlation between them. To be more specific, it is well established in network science that prediction residuals are correlated along edges [20]. Such a result implies a similar phenomenon for inefficiency: taking CQR for regression as an example, the prediction interval largely depends on the residual of the true outcome from the predicted quantiles. Hence, the prediction interval lengths are also highly correlated for connected nodes. We empirically verify this intuition in Figure 3, where we plot the difference in the prediction interval lengths for connected/unconnected node pairs in the Anaheim dataset using vanilla CQR for GCN. In Figure 3, we observe that inefficiency has a topological root: connected nodes usually have similar residual scales, suggesting the existence of rich neighborhood information for the residuals. This motivates us to utilize such information to correct the scores and achieve better efficiency.
Topology-aware correction model. Based on the motivation above, we update the model predictions using the neighbor predictions. However, the relationship in neighborhood predictions could be complex (i.e. beyond homophily), making heuristic aggregation such as averaging/summing/etc. overly simplistic and not generalizable to all types of graphs. Ideally, we want to design a general and powerful mechanism that flexibly aggregates graph information. This requirement could be perfectly
4That is, we take the predicted quantiles from GCN and directly builds prediction intervals using CQR.
Figure 3: Inefficiency is correlated in the network. Connected nodes have significantly smaller gaps in prediction interval length compared to unconnected nodes.
Figure 4: We simulate the downstream conformal step and optimize for inefficiency directly. We first produce differentiable quantile  $\hat{\eta}$  using  $V(X_{i},Y_{i})$  from  $\nu_{\mathrm{cor - cal}}$ . We then construct a prediction set size/interval length proxy on  $\nu_{\mathrm{cor - test}}$  and directly minimize inefficiency loss by updating  $\mathrm{GNN}_{\vartheta}$ .
fulfilled by GNN message passing as it represents a class of learnable aggregation functions over graphs. Therefore, we use a separate GNN learner parameterized by  $\vartheta$  for the same network  $G$  but with modified input node features; specifically, we use the base GNN prediction  $(\mathbf{X}_0 = \widehat{\mu}(X))$  as input, and output  $\tilde{\mu}(X) = \mathrm{GNN}_{\vartheta}(\widehat{\mu}(X), G)$ . We will then use  $\tilde{\mu}(X)$  as the input for constructing conformal prediction sets/intervals. Note that this second GNN model is a post-hoc process and only requires base GNN predictions, instead of access to the base GNN model.
Training with conformal-aware inefficiency loss. Given the hypothesis class, it remains to devise a concrete recipe for training the correction model  $\mathrm{GNN}_{\vartheta}$  parameters. Recall that as with many other prediction models, a GNN model is typically trained to optimize prediction loss (i.e. cross-entropy loss or mean squared error) but not geared towards efficiency for the post-hoc conformal prediction step. We design  $\mathrm{GNN}_{\vartheta}$  to be efficiency-aware by proposing a differentiable inefficiency loss that  $\mathrm{GNN}_{\vartheta}$  optimizes over; this allows integration of GNN message passing to exploit the neighborhood information and also ensures a good  $\tilde{\mu}(\cdot)$  that leads to efficient prediction sets in downstream steps.
We first withhold a small fraction  $\gamma$  of the calibration dataset and use it for the correction procedure. The remaining data is used as the "usual" calibration data for building the conformal prediction set. We then further split the withheld data into a correction calibration set  $\mathcal{V}_{\mathrm{cor - cal}}$  and correction testing set  $\mathcal{V}_{\mathrm{cor - test}}$ , to simulate the downstream conformal inference step. Given  $\tilde{\mu}(X) = \mathrm{GNN}_{\vartheta}(\hat{\mu}(X), G)$  and a target miscoverage rate  $\alpha$ , the framework follows three steps:
(1) Differentiable quantile: we compute a smooth quantile  $\widehat{\eta}$  based on the  $\nu_{\mathrm{cor - cal}}$  by
$$
\widehat {\eta} = \operatorname {D i f f Q u a n t i l e} \left(\left\{V \left(X _ {i}, Y _ {i}\right) \mid i \in \mathcal {V} _ {\text {c o r - c a l}} \right\}, (1 - \alpha) \left(1 + 1 / \left| \mathcal {V} _ {\text {c o r - c a l}} \right|\right). \right.
$$
Since the non-conformity score is usually differentiable, it only requires differentiable quantile calculation where there are well-established methods available [6, 5].
(2) Differentiable inefficiency proxy: we then construct a differentiable proxy  $\mathbf{c}$  of the inefficiency on  $\overline{\mathcal{V}_{\mathrm{cor - test}}}$  by simulating the downstream conformal prediction procedures. We propose general formulas to construct  $\mathbf{c}$  that are applicable for any classification and regression tasks respectively:
a. Inefficiency loss instantiation for Classification: The desirable proxy is to simulate the prediction set size using  $\mathcal{D}_{\mathrm{cor - test}}$  as the "test" data and  $\mathcal{D}_{\mathrm{cor - calib}}$  as the "calibration" data. For class  $k$  and node  $i$  in  $\mathcal{D}_{\mathrm{cor - test}}$ , the non-conformity score is  $V(X_{i}, k)$  for class  $k$ , where  $V(\cdot)$ , for instance, is the APS score in Section 2. Then, we define the inefficiency proxy as
$$
\mathbf {c} _ {i, k} = \sigma (\frac {V (X _ {i} , k) - \widehat {\eta}}{\tau}),
$$
where  $\sigma(x) = \frac{1}{1 + e^{-x}}$  is the sigmoid function and  $\tau$  is a temperature hyper-parameter [40]. It can be interpreted as a soft assignment of class  $k$  to the prediction set. When  $\tau \to 0$ , it becomes a hard assignment. The per-sample inefficiency proxy is then readily constructed as  $\mathbf{c}_i = \frac{1}{|\mathcal{Y}|} \sum_{k \in \mathcal{Y}} \mathbf{c}_{i,k}$ .
b. Inefficiency loss instantiation for Regression: The desirable proxy is to simulate the prediction interval length. For node  $i$  in  $\mathcal{V}_{\mathrm{cor - test}}$ , the conformal prediction interval is  $[\tilde{\mu}_{\alpha /2}(X_i) - \hat{\eta},\tilde{\mu}_{1 - \alpha /2}(X_i) + \hat{\eta} ]$ . Thus, the per-sample prediction interval length could be directly calculated as
$$
\mathbf {c} _ {i} = (\tilde {\mu} _ {1 - \alpha / 2} (X _ {i}) + \widehat {\eta}) - (\tilde {\mu} _ {\alpha / 2} (X _ {i}) - \widehat {\eta}).
$$
Since  $\mathrm{GNN}_{\vartheta}$  maps intervals to intervals and do not pose a constraint on the prediction, it may incur a trivial optimized solution where  $\tilde{\mu}_{1 - \alpha /2}(X) < \tilde{\mu}_{\alpha /2}(X)$ . Thus, we pose an additional consistency regularization term:  $(\tilde{\mu}_{1 - \alpha /2}(X) - \widehat{\mu}_{1 - \alpha /2}(X))^2 + (\tilde{\mu}_{\alpha /2}(X) - \widehat{\mu}_{\alpha /2}(X))^2$ . This regularizes the updated intervals to not deviate significantly to reach the trivial solution.
(3) Inefficiency loss: finally, the inefficiency loss is an average of inefficiency proxies  $L_{\mathrm{ineff}} = \frac{1}{|\mathcal{V}_{\mathrm{cor - test}}|}\sum_i\mathbf{c}_i$ . The GNN  $\vartheta$  is optimized using backpropagation in an end-to-end fashion.
Conditional coverage. A natural question is whether optimizing the efficiency of conformal prediction may hurt its conditional validity.<sup>5</sup> In Section 5, we empirically demonstrate satisfactory conditional coverage across various graph features, which even improves upon the direct application of APS and CQR to graph data. We conjecture that it is because we correct for the correlation among nodes. However, theoretical understanding is left for future investigation.
Graph exchangeability. The post-hoc correction model is GNN-based, thus, it is permutation-invariant. Thus, it satisfies the exchangeability condition laid out in our theory in Section 3. Empirically, we demonstrate in Section 5 that CF-GNN achieves target empirical marginal coverage.
Computational cost. We remark that CF-GNN scales similarly as base GNN training since the correction step follows a standard GNN training procedure but with a modified input attribute and loss function. Notably, as the input to the correction model usually has a smaller attribute size (the number of classes for classification and 2 for regression), it has smaller parameter size than standard GNN training. In addition, it is also compatible with standard GNN mini-batching techniques.
General loss functions. Finally, we note that the choice of our loss function can be quite general. For instance, one may directly optimize for conditional validity by choosing a proper loss function.
We conduct experiments to demonstrate the advantages of CF-GNN over other UQ methods in achieving empirical marginal coverage for graph data, as well as the efficiency improvement with CF-GNN. We also evaluate conditional coverage of CF-GNN and conduct systematic ablation and parameter analysis to show the robustness of CF-GNN.
Evaluation setup. For node classification, we follow a standard semi-supervised learning evaluation procedure [24], where we randomly split data into folds with  $20\% /10\% /70\%$  nodes as  $\mathcal{D}_{train} / \mathcal{D}_{valid} / \mathcal{D}_{calib}\cup \mathcal{D}_{test}$ . For the node regression task, we follow a previous evaluation procedure from [20] and randomly split the data into folds with  $50\% /10\% /40\%$  nodes as  $\mathcal{D}_{train} / \mathcal{D}_{valid} / \mathcal{D}_{calib}\cup \mathcal{D}_{test}$ . We conduct 100 random splits of calibration/testing sets to estimate the empirical coverage. Using the test-time coverage distribution in Figure 2 to ensure that coverage is concentrated tightly around  $1 - \alpha$ , we modify the calibration set size to  $\min \{1000,|\mathcal{D}_{calib}\cup \mathcal{D}_{test}| / 2\}$  and use the rest as the test sample. For a fair comparison, we first train 10 runs of the base GNN model and then fix the predictions (i.e. the input to UQ baselines and CF-GNN). In this way, we ensure that the gain is not from randomness in base model training. The hyperparameter search strategy and configurations for CF-GNN and baselines can be found in Appendix D.1.
Models & baselines to evaluate coverage. For classification, we first use general statistical calibration approaches including temperate scaling [13], vector scaling [13], ensemble temperate scaling [49]. We also use SOTA GNN-specific calibration learners including CaGCN [44] and GATS [17]. The prediction set is the set of classes from highest to lowest scores until accumulative scores exceed  $1 - \alpha$ . For regression, we construct prediction intervals using quantile regression (QR) [25], Monte Carlo dropouts (MC dropout) [9], and Bayesian loss to model both aleatoric and epistemic uncertainty [23]. More information about baselines can be found in Appendix D.2.
Models & baselines to evaluate efficiency. As smaller coverage always leads to higher efficiency, for a fair comparison, we can only compare methods on efficiency that achieve the same coverage. Thus, we do not evaluate UQ baselines here since they do not produce exact coverage and are thus not comparable. While any CP-based methods produce exact coverage, to the best of our knowledge,
${}^{5}$  Conditional coverage asks for  $\mathbb{P}\left( {{Y}_{n + j} \in  \widehat{C}\left( {{X}_{n + j}}\right)  \mid  {X}_{n + j} = x}\right)  \approx  1 - \alpha$  for all  $x \in  \mathcal{X}$  . Although exact conditional validity is statistically impossible [8],approximate conditional validity is a practically important property that APS and CQR are designed for. See Section 5 for common ways to assess conditional coverage.
Table 1: Empirical marginal coverage of node classification(upper table) and node regression tasks(lower table). The result takes the average and standard deviation across 10 GNN runs with 100 calib/test splits.  $\checkmark$  means that the UQ method reaches the target coverage (i.e. coverage  $\geq 0.95$ ) while  $\times$  means that it fails to reach it. The last column "Covered" becomes  $\checkmark$  if a UQ method reaches target coverage for all datasets and  $\times$  vice versa.
Task	UQ Model	Cora	DBLP	CiteSeer	PubMed	Computers	Photo	CS	Physics	Covered?
Node
classify.	Temp. Scale.	0.946±.003 X	0.920±.009 X	0.952±.004√	0.899±.002 X	0.929±.002 X	0.962±.002√	0.957±.001√	0.969±.000√	X
Vector Scale.	0.944±.004 X	0.921±.009 X	0.951±.004√	0.899±.003 X	0.932±.002 X	0.963±.002√	0.958±.001√	0.969±.000√	X
Ensemble TS	0.947±.003 X	0.920±.008 X	0.953±.003√	0.899±.002 X	0.930±.002 X	0.964±.002√	0.958±.001√	0.969±.000√	X
CaGCN	0.939±.005 X	0.922±.004 X	0.949±.005 X	0.898±.003 X	0.926±.003 X	0.956±.002√	0.954±.003√	0.968±.001√	X
GATS	0.939±.005 X	0.921±.004 X	0.951±.005√	0.898±.002 X	0.925±.002 X	0.957±.002√	0.957±.001√	0.968±.000√	X
CF-GNN	0.952±.001√	0.952±.001√	0.953±.001√	0.953±.001√	0.952±.001√	0.953±.001√	0.952±.001√	0.952±.001√	✓
Task	UQ Model	Anaheim	Chicago	Education	Election	Income	Unemploy.	Twitch	Covered?
Node regress.	QR	0.943±.031X	0.950±.007X	0.959±.001√	0.956±.004√	0.960±.005√	0.954±.004√	0.900±.015X	X
MC dropout	0.553±.022X	0.427±.015X	0.423±.013X	0.417±.008X	0.532±.022X	0.489±.016X	0.448±.017X	X
BayesianNN	0.967±.001√	0.955±.003√	0.957±.002√	0.958±.009√	0.970±.004√	0.960±.001√	0.923±.006X	X
CF-GNN	0.957±.003√	0.954±.002√	0.951±.001√	0.950±.001√	0.951±.001√	0.951±.001√	0.954±.001√	✓
there are no existing graph-based conformal prediction methods for transductive settings. Thus, we can only compare with the direct application of conformal prediction (CP) to base GNN. In the main text, we only show results for GCN [24] as the base model; results of three additional popular GNN models (GraphSAGE [16], GAT [42], and SGC [45]) are deferred to Appendix D.4.
Datasets. We evaluate CF-GNN on 8 node classification datasets and 7 node regression datasets with diverse network types such as geographical network, transportation network, social network, citation network, and financial network. Dataset statistics are in Appendix D.3.
CF-GNN achieves empirical marginal coverage while existing UQ methods do not. We report marginal coverage of various UQ methods with target coverage at  $95\%$  (Table 1). There are three key takeaways. Firstly, none of these UQ methods achieves the target coverage for all datasets while CF-GNN does, highlighting the lack of statistical rigor in those methods and the necessity for a guaranteed UQ method. Secondly, it validates our theory from Section 3 that CF-GNN achieves designated coverage in transductive GNN predictions. Lastly, CF-GNN achieves empirical coverage that is close to the target coverage while baseline UQ methods are not. This controllable feature of CF-GNN is practically useful for practitioners that aim for a specified coverage in settings such as planning and selection.
CF-GNN significantly reduces inefficiency. We report empirical inefficiency for 8 classification and 7 regression datasets (Table 2). We observe that we achieve consistent improvement across datasets with up to  $74\%$  reduction in the prediction set size/interval length. We additionally conduct the same experiments for 3 other GNN models including GAT, GraphSAGE, and SGC in Appendix D.4 and we observe that performance gain is generalizable to diverse architecture choices. Furthermore, CF-GNN yields more efficient prediction sets than existing UQ methods even if we manually adjust the nominal level of them to achieve  $95\%$  empirical coverage (it is however impossible to do so in practice, here we do this for evaluation). For instance, the best calibration method GATS yields an average prediction size of 1.82 on Cora when the nominal level is tuned to achieve  $95\%$  empirical coverage. In contrast, CF-GNN has an average size of 1.76, smaller than GATS. In Appendix D.5, we also observe that CF-GNN also reduces inefficiency for advanced conformal predictor RAPS for classification task. In addition, we find that CF-GNN yields little changes to the prediction accuracy of the original GNN model (Appendix D.7).
CF-GNN empirically maintains conditional coverage. While CF-GNN achieves marginal coverage, it is highly desirable to have a method that achieves reasonable conditional coverage, which was the motivation of APS and CQR. We follow [37] to evaluate conditional coverage via the Worst-Slice (WS) coverage, which takes the worst coverage across slices in the feature space (i.e. node input features). We observe that CF-GNN achieves a WS coverage close to  $1 - \alpha$ , indicating satisfactory conditional coverage (Cond. Cov. (Input Feat.) row in Table 3). Besides the raw features, for each node, we also construct several network features (which are label agnostic) including clustering
Table 2: Empirical inefficiency measured by the size/length of the prediction set/interval for node classification (left table)/regression(right table). A smaller number has better efficiency. We show the relative improvement  $(\%)$  of CF-GNN over CP on top of the  $\rightarrow$ . The result uses APS for classification and CQR for regression with GCN as the base model. Additional results on other GNN models are at Appendix D.4. We report the average and standard deviation of prediction sizes/lengths calculated from 10 GNN runs, each with 100 calibration/test splits.
Task	Dataset	CP → CF-GNN
Node
classify.	Cora	3.80±.28→53.61%1.76±.27
DBLP	2.43±.03→-49.13%1.23±.01
CiteSeer	3.86±.11→-74.27%0.99±.02
PubMed	1.60±.02→-19.05%1.29±.03
Computers	3.56±.13→-49.05%1.81±.12
Photo	3.79±.13→-56.28%1.66±.21
CS	7.79±.29→-62.16%2.95±.49
Physics	3.11±.07→-62.81%1.16±.13
Average Improvement	-53.75%
Task	Dataset	CP → CF-GNN
Node regress.	Anaheim	2.89±.39→-25.00%2.17±.11
Chicago	2.05±.07→-0.48%2.04±.17
Education	2.56±.02→-5.07%2.43±.05
Election	0.90±.01→+0.21%2.90±.02
Income	2.51±.12→-4.58%2.40±.05
Unemploy	2.72±.03→-10.83%2.43±.04
Twitch	2.43±.10→-1.36%2.39±.07
Average Improvement	-6.73%
Table 3: CF-GNN achieves conditional coverage, measured by Worse-Slice Coverage [37]. We use Cora/Twitch as an example classification/regression dataset. Results on other network features and results on target coverage of 0.9 can be found in Appendix D.6.
Target: 0.95	Classification	Regression
Model	CP	CF-GNN	CP	CF-GNN
Marginal Cov.	0.95±.01	0.95±.01	0.96±.02	0.96±.02
Cond. Cov. (Input Feat.)	0.94±.02	0.94±.03	0.95±.04	0.94±.05
Cond. Cov. (Cluster)	0.89±.06	0.93±.04	0.96±.03	0.96±.03
Cond. Cov. (Between)	0.81±.06	0.95±.03	0.94±.05	0.94±.05
Cond. Cov. (PageRank)	0.78±.06	0.94±.03	0.94±.05	0.94±.05
Table 4: Ablation. For Size/length, we use Cora/Anaheim dataset with GCN backbone. Each experiment is with 10 independent base model runs with 100 conformal split runs.
Topology -aware	Ineff. Loss	Size	Length
✓	✓	1.76±.27	2.17±.11
✓	X	2.42±.35	2.23±.10
X	✓	2.35±.47	2.32±.18
X	X	3.80±.28	2.89±.39
coefficients, betweenness, PageRank, closeness, load, and harmonic centrality, and then calculate the WS coverage over the network feature space. We observe close to  $95\%$  WS coverage for various network features, suggesting CF-GNN also achieves robust conditional coverage over network properties. We also see that the direct application of CP (i.e. without graph correction) has much smaller WS coverage for classification, suggesting that adjusting for neighborhood information in CF-GNN implicitly improves conditional coverage.
Ablation. We conduct ablations in Table 4 to test two main components in CF-GNN, topology-aware correction model, and inefficiency loss. We first remove the inefficiency loss and replace it with standard prediction loss. The performance drops as expected, showing the power of directly modeling inefficiency loss in the correction step. Secondly, we replace the GNN correction model with an MLP correction model. The performance drops significantly, showing the importance of the design choice of correction model and justifying our motivation on inefficiency correlation over networks.
Parameter analysis. We conduct additional parameter analysis to test the robustness of CF-GNN. We first adjust the target coverage rate and calculate the inefficiency (Figure 5(1)). CF-GNN consistently beats the vanilla CP across all target coverages. Moreover, we adjust the fraction  $\gamma$  of the holdout calibration data in building the inefficiency loss, and observe that CF-GNN achieves consistent improvement in inefficiency (Figure 5(2)). We also observe a small fraction  $(10\%)$  leads to excellent performance, showing that our model only requires a small amount of data for the inefficiency loss and leaves the majority of the calibration data for downstream conformal prediction.
We discuss here related works that are closest to the ideas in CF-GNN and provide extended discussion on other related works in Appendix E.
(1) Uncertainty quantification (UQ) for GNN: Many UQ methods have been proposed to construct model-agnostic uncertain estimates for both classification [13, 49, 14, 27, 1] and regression [25, 41,
Figure 5: (1) Parameter analysis on inefficiency given different target coverage rate  $1 - \alpha$ . (2) Parameter analysis on inefficiency given calibration set holdout fraction. Analyses use Cora/Anaheim for classification/regression.
38, 9, 28, 26, 35, 23, 19]. Recently, specialized calibration methods for GNNs that leverage network principles such as homophily have been developed [44, 17]. However, these UQ methods can fail to provide a statistically rigorous and empirically valid coverage guarantee (see Table 1). In contrast, CF-GNN achieves valid marginal coverage in both theory and practice.
(2) Conformal prediction for GNN: The application of CP to graph-structured data remains largely unexplored. [7] claims that nodes in the graph are not exchangeable in the inductive setting and employs the framework of [3] to construct prediction sets using neighborhood nodes as the calibration data for mitigating the miscoverage due to non-exchangeability. In contrast, we study the transductive setting where certain exchangeability property holds; thus, the method from [3] are not comparable to ours. Concurrent with our work, [15] studies the exchangeability under transductive setting and proposes a diffusion-based method for improving efficiency, which can be viewed as an instantiation of our approach where the GNN correction learns an identity mapping; [33] studies exchangeability in network regression for of non-conformity scores based on various network structures, with similar observations as our Theorem 3. Other recent efforts in conformal prediction for graphs include [32, 34] which focus on distinct problem settings.
(3) Efficiency of conformal prediction: How to achieve desirable properties beyond validity is an active topic in the CP community; we focus on the efficiency aspect here. One line of work designs "good" non-conformity scores in theory such as APS [37] and CQR [36]. More recent works take another approach, by modifying the training process of the prediction model. CF-GNN falls into the latter case, although our idea applies to any non-conformity score. ConfTr [40] also modifies training for improved efficiency. Our approach differs from theirs in significant ways. First, we develop a theory on CP validity on the graph data and leverage topological principles that are specialized to graph to improve efficiency while ConfTr focuses on i.i.d. vision image data. Also, ConfTr happens during base model training using the training set, while CF-GNN conducts post-hoc correction using withheld calibration data without assuming access to base model training, making ConfTr not comparable to us. Finally, we also propose a novel loss for efficiency in regression tasks.