SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The general framework for shift-detection can be found in the following figure, Figure 3.
Figure 3: The procedure of detecting a dataset shift using dimensionality reduction and then a two-sample statistical test. The dimensionality reduction is applied to both the detection-training (source) and test (target) data, prior to being analyzed using statistical hypothesis testing. This figure is taken from [37].
B.1 Proof for Theorem 4.2
$$
\mathcal {B} _ {\theta_ {i}} \triangleq b _ {i} ^ {*} (m, m \cdot \hat {c} _ {i} (\theta_ {i}, S _ {m}), \frac {\delta}{k}),
$$
$$
\mathcal {C} _ {\theta_ {i}} \triangleq c (\theta_ {i}, \mathcal {P}).
$$
Consider the i-th iteration of SGR over a detection-training set  $S_{m}$ , and recall that,  $\theta_{i} = \kappa_{f}(x_{z}), x_{z} \in S_{m}$  (see Algorithm 1). Therefore,  $\theta_{i}$  is a random variable (between zero and one), since it is a function of a random variable  $(x \in S_{m})$ . Let  $\mathbf{Pr}_{S_m}\{\theta_i = \theta'\}$  be the probability that  $\theta_{i} = \theta'$ .
$$
\begin{array}{l} \Pr_ {S _ {m}} \left\{\mathcal {C} _ {\theta_ {i}} <   \mathcal {B} _ {\theta_ {i}} \right\} \\ = \int_ {0} ^ {1} d \theta^ {\prime} \mathbf {P r} _ {S _ {m}} \left\{\mathcal {C} _ {\theta_ {i}} <   \mathcal {B} _ {\theta_ {i}} \mid \theta_ {i} = \theta^ {\prime} \right\} \cdot \mathbf {P r} _ {S _ {m}} \left\{\theta_ {i} = \theta^ {\prime} \right\} \\ = \int_ {0} ^ {1} d \theta^ {\prime} \mathbf {P r} _ {S _ {m}} \left\{\mathcal {C} _ {\theta^ {\prime}} <   \mathcal {B} _ {\theta^ {\prime}} \right\} \cdot \mathbf {P r} _ {S _ {m}} \left\{\theta_ {i} = \theta^ {\prime} \right\}. \\ \end{array}
$$
Given that at the  $i$ -th iteration  $\theta_{i} = \theta^{\prime}$ , and considering that  $\mathcal{B}_{\theta^{\prime}}$  is derived using Lemma 4.1 (refer to Algorithm 1), we obtain,
$$
\mathbf {P r} _ {S _ {m}} \left\{\mathcal {C} _ {\theta^ {\prime}} <   \mathcal {B} _ {\theta^ {\prime}} \right\} <   \frac {\delta}{k}.
$$
$$
\begin{array}{l} \Pr_ {S _ {m}} \left\{\mathcal {C} _ {\theta_ {i}} <   \mathcal {B} _ {\theta_ {i}} \right\} \\ = \int_ {0} ^ {1} d \theta^ {\prime} \mathbf {P r} _ {S _ {m}} \left\{\mathcal {C} _ {\theta^ {\prime}} <   \mathcal {B} _ {\theta^ {\prime}} \right\} \cdot \mathbf {P r} _ {S _ {m}} \left\{\theta_ {i} = \theta^ {\prime} \right\} \\ <   \int_ {0} ^ {1} d \theta^ {\prime} \frac {\delta}{k} \cdot \mathbf {P r} _ {S _ {m}} \left\{\theta_ {i} = \theta^ {\prime} \right\} \\ = \frac {\delta}{k} \cdot \left(\int_ {0} ^ {1} d \theta^ {\prime} \mathbf {P r} _ {S _ {m}} \left\{\theta_ {i} = \theta^ {\prime} \right\}\right) \\ = \frac {\delta}{k}. \tag {5} \\ \end{array}
$$
The following application of the union bound completes the proof,
$$
\mathbf {P r} _ {S _ {m}} \left\{\exists i: \mathcal {C} _ {\theta_ {i}} <   \mathcal {B} _ {\theta_ {i}} \right\} \leq \sum_ {i = 1} ^ {k} \mathbf {P r} _ {S _ {m}} \left\{\mathcal {C} _ {\theta_ {i}} <   \mathcal {B} _ {\theta_ {i}} \right\} <   \sum_ {i = 1} ^ {k} \frac {\delta}{k} = \delta .
$$
C Exploring Model Sensitivity: Evaluating Accuracy on Shifted Datasets
In this section, we present Table 3, which displays the accuracy (when applicable) as well as the degradation from the original accuracy over the ImageNet dataset, of the considered models on each of the simulated shifts mentioned in Section 6.1.1.
Shift Dataset	ResNet50	MovileNetV3	ViT-Degradation
Acc.	ImageNet Degradation	Acc.	ImageNet Degradation	Acc.	ImageNet Degradation
FGSM ε = 7 · 10-5	76.68%	-3.7%	62.09%	-3.15%	72.51%	-2.95%
FGSM ε = 1 · 10-4	75.19%	-5.19%	60.72%	-4.52%	71.49%	-3.97%
FGSM ε = 3 · 10-4	66.15%	-14.23%	52.09%	-13.15%	65.06%	-10.4%
FGSM ε = 5 · 10-4	59.23%	-21.15%	44.45%	-20.79%	58.9%	-16.56%
PGD ε = 1 · 10-4	74.64%	-5.74%	60.63%	-4.61%	71.35%	-4.11%
GAUSSIAN σ = 0.1	79.02%	-1.36%	62.82%	-2.42%	71.79%	-3.67%
GAUSSIAN σ = 0.3	74.63%	-5.75%	55.06%	-10.18%	50.86%	-24.6%
GAUSSIAN σ = 0.5	68.56%	-11.82%	42.55%	-22.69%	22.25%	-53.21%
GAUSSIAN σ = 1	46.1%	-34.28%	13.82%	-51.42%	0.56%	-74.9%
ZOOM 50%	65.55%	-14.83%	36.96%	-28.28%	46.04%	-29.42%
ZOOM 70%	74.31%	-6.07%	53.53%	-11.71%	62.69%	-12.77%
ZOOM 90%	78.6%	-1.78%	61.28%	-3.96%	72.08%	-3.38%
ROTATION θ = 5°	76.7%	-3.68%	62.42%	-2.82%	71.27%	-4.19%
ROTATION θ = 10°	72.4%	-7.98%	58.22%	-7.02%	67.29%	-8.17%
ROTATION θ = 20°	68.29%	-12.09%	49.96%	-15.28%	62.38%	-13.08%
ROTATION θ = 25°	70.08%	-10.3%	50.95%	-14.29%	60.97%	-14.49%
Table 3: Shifted dataset accuracy and comparison with ImageNet. We display the accuracy for each shifted dataset and model combination, along with the accuracy degradation when compared to the original ImageNet dataset.
D Extended Empirical Results
In this section, we present a detailed analysis of our empirical findings on the ResNet50 architecture. We report the results for each window size,  $|W_{k}| \in \{10, 20, 50, 100, 200, 500, 1000\}$ , and for several shift cases discussed in Section 6.1.1. In particular, we show the detection performance of all the discussed methods, for the following shifts: FGSM (Table 4), ImageNet-O (Table 5), ImageNet-A (Table 6), and the Zoom out shift,  $90\%$  (Table 7).
Method	Window size
AUROC ↑ / AUPR-In ↑ / AUPR-Out ↑ / DetectionError ↓ / FPR@95TPR ↓
10	20	50	100	200	500	1000
KS	Softmax Embeddings	32/45/40/47/92	47/55/46/44/91	64/72/59/34/69	72/72/75/38/77	80/87/69/18/36	100/100/100/2/4	100/100/100/0/0
54/58/55/43/86	32/39/48/49/100	54/64/49/39/80	41/44/48/50/99	37/48/45/47/92	60/70/59/30/60	71/77/61/33/68
MMD	Softmax Embeddings	36/48/42/45/90	44/56/44/42/82	51/53/51/48/93	41/44/54/49/97	50/52/50/48/94	48/52/51/45/93	55/55/55/47/94
61/56/60/48/95	57/57/59/45/93	72/73/67/36/73	63/70/56/38/71	63/69/55/37/75	67/70/61/39/74	70/79/59/28/54
Single-instance	SR Entropy	34/45/40/47/93	69/68/72/42/82	43/52/50/45/90	54/54/61/47/93	62/64/58/42/86	66/73/59/35/72	72/69/73/43/86
42/49/44/47/94	65/60/65/47/92	49/55/49/45/89	59/53/63/49/98	60/66/58/39/77	59/59/56/45/90	64/61/63/46/90
CBD (Ours)	71/64/75/45/92	77/82/75/25/51*	88/90/85/20/39	84/86/84/25/49	99*/99*/99*/3*/5*	98/98/98/5/10	100/100/100/2/2
Table 4: Comparison of different evaluation metrics over ResNet50 with the discussed baselines methods, over the FGSM shift with  $\epsilon = 0.0001$ . The best performing method is highlighted in bold; we add the superscript * to the bolded result when it is statistically significant.
Method	Window size
10	AUROC ↑ / AUPR-In ↑ / AUPR-Out ↑ / DetectionError ↓ / FPR@95TPR ↓
20	50	100	200	500	1000
KS	Softmax Embeddings	62/59/60/46/94	70/61/75/48/94	98/98/98/6/11	99/99/99/5/10	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0
85/89/76/18/35	97/98/97/6*12*	99/99/99/5/9	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0
MMD	Softmax Embeddings	43/53/47/44/87	74/75/73/39/72	94/92/96/33/51	97/97/98/31/27	97/97/98/32/26	100/100/100/0/0	100/100/100/0/0
96/97/97*10/20	95/94/97/33/39	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	94/95/97/31/46	100/100/100/0/0
Single-instance	SR Entropy	62/63/60/43/87	31/41/38/48/98	47/56/44/41/86	56/57/63/45/85	51/51/53/48/97	37/41/42/50/100	42/44/47/49/100
64/66/68/40/81	39/42/53/50/99	41/52/45/44/90	58/66/52/37/75	54/51/55/49/99	52/55/53/48/90	84/85/84/26/52
CBD (Ours)	61/61/61/47/92	84/89/77/18/37	99/99/99/5/8	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0
Table 5: Comparison of different evaluation metrics over ResNet50 with the discussed baselines methods, over the ImageNet-O shift. The best performing method is highlighted in bold; we add the superscript * to the bolded result when it is statistically significant.
Method	Window size
AUROC ↑ / AUPR-In ↑ / AUPR-Out ↑ / DetectionError ↓ / FPR@95TPR ↓
10	20	50	100	200	500	1000
KS	Softmax Embeddings	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	101/100/100/0/0
98/98/98/7/14	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	102/100/100/0/0
MMD	Softmax Embeddings	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	100/100/100/0	100/100/100/0/0	100/100/100/0/0
100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	103/100/100/0/0	100/100/100/0/0
Single-instance	SR Entropy	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	106/100/100/0/0	100/100/100/0/0
100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	125/100/100/0/0	100/100/100/0/0	102/100/100/0/0	100/100/100/0/0
CBD (Ours)	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	100/100/100/0/0	125/100/100/0/0	100/100/100/0/0	106/100/100/0/0	100/100/100/0/0
Table 6: Comparison of different evaluation metrics over ResNet50 with the discussed baselines methods, over the ImageNet-A shift. The best performing method is highlighted in bold; we add the superscript * to the bolded result when it is statistically significant.
Method	Window size
AUROC ↑ / AUPR-In ↑ / AUPR-Out ↑ / DetectionError ↓ / FPR@95TPR ↓
10	20	50	100	200	500	1000
KS	Softmax Embeddings	42/49/48/47/93	53/52/54/47/97	61/69/57/37/74	71/72/69/39/77	91/92/91*15/30	100*100*100*2/3*	100/100/100/0/0
46/51/52/46/92	25/45/38/44/87	56/58/54/44/86	46/46/53/49/98	35/44/42/49/97	27/38/38/50/99	41/46/46/47/97
MMD	Softmax Embeddings	48/51/50/46/93	46/44/52/50/100	51/57/52/45/88	50/56/52/46/88	52/57/49/46/88	51/57/47/44/90	53/53/54/49/96
57/56/61/48/91	54/63/49/41/83	68/67/73/40/82	37/53/41/43/83	28/41/38/49/97	31/38/41/50/100	18/35/34/49/100
Single-instance	SR Entropy	28/38/38/50/100	45/45/48/51/99	44/53/43/43/87	52/59/53/42/84	65/68/59/41/85	74/73/78/38/78	84/84/83/31/62
29/38/39/50/100	50/47/56/51/100	51/53/58/46/91	61/63/66/41/83	71/74/61/36/72	76/73/81/39/77	87/86/87/29/58
CBD (Ours)	70/61/77/47/95	69/73/68/35/70	81/86/75/22/43	72/75/71/34/67	76/82/71/23/46	94/94/94/15/31	100/100/100/0/0
Table 7: Comparison of different evaluation metrics over ResNet50 with the discussed baselines methods, over the Zoom out (90%) shift. The best performing method is highlighted in bold; we add the superscript * to the bolded result when it is statistically significant.
In this section, we conduct multiple experiments to analyze the various components of our framework; all those experiments are conducted using a ResNet50. We explore several hyper-parameter choices, including  $C_{\mathrm{target}}$ ,  $\delta$ , and  $\kappa_f$ . More specifically, we consider  $C_{\mathrm{target}} \in \{1, 10, 100\}$ , and  $\delta \in \{0.1, 0.01, 0.001, 0.0001\}$ , and two different CFs  $\kappa_f$ , namely SR and Entropy-based.
To evaluate the performance of our detectors under varying hyper-parameters, we have selected a single metric that we believe to be the most important, namely, AUROC [13]. Additionally, since performance may vary depending on window size, we display the average AUROC across all window sizes that we have considered in our experiments. These window sizes include:  $\{10, 20, 50, 100, 200, 500, 1000\}$ . In Figure 4, we summarize our findings by displaying the average AUROC value as a function of the chosen hyper-parameters. These results are presented as heatmaps.
(a)  $\kappa_{f} = (1 - \text{Entropy})$ .
(b)  $\kappa_{f} = \mathrm{SR}$
Figure 4: AUROC performance of our detector under different choices of hyper-parameters.
Figure 4a, displays our AUROC detector's performance when we use Entropy-based as our CF. We observe that the optimal choice of hyper-parameters is  $\delta = 0.01$  and  $C_{\mathrm{target}} = 10$ , resulting in the highest performance. However, increasing the value of  $C_{\mathrm{target}}$  leads to a more consistent and robust detector, as changes in the value of  $\delta$  do not significantly affect the detector's performance. Additionally, we note that using  $C_{\mathrm{target}} = 1$  yields relatively poor performance, indicating that a single coverage choice is insufficient to capture the characteristics of the distribution represented by the sample  $S_{m}$ . Similar results are obtained when using SR as the CF, as shown in Figure 4b. These results suggest that selecting a high value of  $C_{\mathrm{target}}$  and a low value of  $\delta$  is the most effective approach for ensuring a robust detector. Finally, the heatmaps demonstrate that Entropy-based CF outperforms (by a low margin) SR, in terms of detection performance.
F Upper Coverage Bound Development and Application
In this section, we discuss the complementary problem to that outlined in Section 4.1, which is solved by Algorithm 3; we also show its application for detecting distribution shifts, Algorithm 4. Finally, we introduce a unified algorithm, Algorithm 5, for detecting distribution shifts, while accounting for both under-coverage and over-coverage, details follow.
Problem statement. For a classifier  $f$ , a detection-training sample  $S_{m} \sim \mathcal{P}^{m}$ , a confidence parameter  $\delta > 0$  and a desired coverage  $c^{*} > 0$ , our goal is to use  $S_{m}$  to find a  $\theta$  value (which implies a selection function  $g_{\theta}$ ) such that the coverage satisfies,
$$
\Pr_ {S _ {m}} \left\{c (\theta , \mathcal {P}) > c ^ {*} \right\} <   \delta . \tag {6}
$$
This means that over coverage should occur with probability of at most  $\delta$ .
The pseudo code of the algorithm that finds the optimal coverage upper bound (with confidence  $\delta$ ) - SGC-UP, appears in Algorithm 3.
Algorithm 3: Selection with guaranteed coverage - Upper bound (SGC-UP)
1 Input: train set:  $S_{m}$ , confidence-rate function:  $\kappa_{f}$ , confidence parameter  $\delta$ , target coverage:  $c^{*}$ .
2 Sort  $S_{m}$  according to  $\kappa_{f}(x_{i})$ ,  $x_{i} \in S_{m}$  (and now assume w.l.o.g. that indices reflect this ordering).
3  $z_{\mathrm{min}} = 1, z_{\mathrm{max}} = m$
4 for  $i = 1$  to  $k = \lceil \log_2m\rceil$  do
5  $z = \lceil (z_{\mathrm{min}} + z_{\mathrm{max}}) / 2\rceil$    
6  $\theta_{i} = \kappa_{f}(x_{z})$    
7 Calculate  $\hat{c}_i(\theta_i,S_m)$    
8 Solve for  $b_{i}^{*}(m,m\cdot \hat{c}_{i}(\theta_{i},S_{m}),\frac{\delta}{k})$  {see Lemma F.1}   
9 if  $b_{i}^{*}(m,m\cdot \hat{c}_{i}(\theta_{i},S_{m}),\frac{\delta}{k})\geq c^{*}$  then   
10  $|z_{\mathrm{min}} = z$    
11 else   
12  $|z_{\mathrm{max}} = z$    
13 end if
14 end for
15 Output: bound:  $b_{k}^{*}(m,m\cdot \hat{c}_{k}(\theta_{k},S_{m}),\frac{\delta}{k})$ , threshold:  $\theta_{k}$ .
To deduce Equation (6) we make use of Lemma F.1, which is nearly identical to Lemma 4.1.
Lemma F.1. Let  $\mathcal{P}$  be any distribution and consider a  $CF$  threshold  $\theta$  with a coverage of  $c(\theta, \mathcal{P})$ . Let  $0 < \delta < 1$  be given and let  $\hat{c}(\theta, S_m)$  be the empirical coverage w.r.t. the set  $S_m$ , sampled i.i.d. from  $\mathcal{P}$ . Let  $b^*(m, m \cdot \hat{c}(\theta, S_m), \delta)$  be the solution of the following equation:
$$
\arg \min  _ {b} \left(\sum_ {j = 0} ^ {m \cdot \hat {c} (\theta , S _ {m})} \binom {m} {j} b ^ {j} (1 - b) ^ {m - j} \leq \delta\right). \tag {7}
$$
$$
\boldsymbol {P r} _ {S _ {m}} \left\{c (\theta , \mathcal {P}) > b ^ {*} (m, \hat {c} (\theta , S _ {m}), \delta) \right\} <   \delta . \tag {8}
$$
The following is a uniform convergence theorem for SGC-UP (Algorithm 3) procedure stating that all the calculated bounds are valid simultaneously with a probability of at least  $1 - \delta$ .
Theorem F.2. Assume  $S_{m}$  is sampled i.i.d. from  $\mathcal{P}$ , and consider an application of Algorithm 3. For  $k = \lceil \log_2 m \rceil$ , let  $b_i^* (m, m \cdot \hat{c}_i(\theta_i, S_m), \frac{\delta}{k})$  and  $\theta_i$  be the values obtained in the  $i^{th}$  iteration of Algorithm 3. Then,
$$
\left. \right. \boldsymbol {P r} _ {S _ {m}} \left\{\exists i: c \left(\theta_ {i}, \mathcal {P}\right) > b _ {i} ^ {*} (m, m \cdot \hat {c} _ {i} \left(\theta_ {i}, S _ {m}\right), \frac {\delta}{k}) \right\} <   \delta .
$$
The proof for Theorem F.2 can be easily deduced from the proof of Theorem 4.2, given in Appendix B.1.
F.1 Application of Upper Bound Algorithm for Detecting Distribution Shifts
In this section, we demonstrate the application of Algorithm 3 for detecting shifts in distribution. Specifically, Algorithm 3 is useful in identifying distribution shifts towards 'over-confidence,' where the model exhibits excessive confidence in its predictions on input data. It's important to note that this scenario was not empirically observed in our experiments, leading to the exclusion of this section from the main body of the paper.
Our 'over-confidence' detection algorithm is analogous to Algorithm 2. We start by applying SGC-UP to  $C_{\text{target}}$  target coverages uniformly spread between the interval [0.1, 1], excluding the coverage of 1. Each application  $j$  of SGC-UP on the same sample  $S_m \sim \mathcal{P}_m$  with a target coverage of  $c_j^{*;\mathrm{up}}$  produces a pair:  $(b_j^{*,\mathrm{up}}, \theta_j^{\mathrm{up}})$ , which represents a threshold and a bound, respectively.
Define,  $\delta^{\mathrm{up}}(\hat{c} |b^{*}) = \mathbb{1}[\hat{c}\geq b^{*}]$ , thus, given a window of k samples from an alternative distribution,  $W_{k}\sim Q^{k}$  we define the sum of upper bound violations,  $V^{\mathrm{up}}$ , as follows,
$$
\begin{array}{l} V ^ {\mathrm {u p}} = \frac {1}{C _ {\text {t a r g e t}}} \sum_ {j = 1} ^ {C _ {\text {t a r g e t}}} \left(\hat {c} \left(\theta_ {j} ^ {\mathrm {u p}}, W _ {k}\right) - b _ {j} ^ {*; \mathrm {u p}}\right) \cdot \delta^ {\mathrm {u p}} \left(\hat {c} \left(\theta_ {j} ^ {\mathrm {u p}}, W _ {k}\right) \mid b _ {j} ^ {*; \mathrm {u p}}\right) \tag {9} \\ = \frac {1}{k \cdot C _ {\text {t a r g e t}}} \sum_ {j = 1} ^ {C _ {\text {t a r g e t}}} \sum_ {i = 1} ^ {k} \left(g _ {\theta_ {j} ^ {\mathrm {u p}}} \left(x _ {i}\right) - b _ {j} ^ {*; \mathrm {u p}}\right) \cdot \delta^ {\mathrm {u p}} \left(\hat {c} _ {j} ^ {\mathrm {u p}} | b _ {j} ^ {*; \mathrm {u p}}\right), \\ \end{array}
$$
where we obtain the last equality by using  $\hat{c}_j^{\mathrm{up}}\triangleq \hat{c} (\theta_j^{\mathrm{up}},W_k) = \frac{1}{k}\sum_{i = 1}^{k}g_{\theta_j^{\mathrm{up}}}(x_i)$ .
Similarly to  $V$  in Equation (4), the term  $V^{\mathrm{up}}$  denotes the magnitude of the violation, where  $V^{\mathrm{up}} = 0$  indicates an absence of over-confidence in the predictions.
A pseudocode of our algorithm for detecting over-confidence in predictions appears in Algorithm 4.
Algorithm 4: Coverage-Based Detection - over confidence
1 // Fit  
2 Input Training:  $S_{m}, \delta, \kappa_{f}, C_{\text{target}}$   
3 Generate  $C_{\text{target}}$  uniformly spread coverages  $c^{*}$   
4 for  $j = 1$  to  $C_{\text{target}}$  do  
5 |  $b_{j}^{*\mathrm{up}}, \theta_{j}^{\mathrm{up}} = \mathrm{SGC - UP}(S_{m}, \delta, c_{j}^{*}, \kappa_{f})$   
6 end for  
7 Output Training:  $\{(b_{j}^{*\mathrm{up}}, \theta_{j}^{\mathrm{up}})\}_{j=1}^{C_{\text{target}}}$   
8 // Detect  
9 Input Detection:  $\{(b_{j}^{*\mathrm{up}}, \theta_{j}^{\mathrm{up}})\}_{j=1}^{C_{\text{target}}}, \kappa_{f}, \alpha, k$   
10 while True do  
11 Receive window  $W_{k} = \{x_{1}, x_{2}, \ldots, x_{k}\}$   
12 Calculate  $V^{\mathrm{up}}$  (see Equation (9))  
13 Obtain p-value from t-test,  $H_{0}: V^{\mathrm{up}} = 0, H_{1}: V^{\mathrm{up}} > 0$   
14 if  $p_{value} < \alpha$  then  
15 Shiftdetected ← True  
16 Output Detection: Shiftdetected,  $p_{value}$   
17 end if  
18 end while
F.2 Unified Algorithm for Distribution Shift Detection
In this section, we introduce a unified algorithm designed to detect shifts in distribution, accommodating both scenarios of over-confidence and under-confidence in predictions.
Define sum of violations (SOV),
$$
\mathrm {S O V} \triangleq V + V ^ {\mathrm {u p}} = \frac {1}{C _ {\text {t a r g e t}}} \sum_ {j = 1} ^ {C _ {\text {t a r g e t}}} \left(\hat {c} _ {j} ^ {\mathrm {u p}} - b _ {j} ^ {*; \mathrm {u p}}\right) \cdot \delta^ {\mathrm {u p}} \left(\hat {c} _ {j} ^ {\mathrm {u p}} \mid b _ {j} ^ {*; \mathrm {u p}}\right) + \left(b _ {j} ^ {*} - \hat {c} _ {j}\right) \cdot \delta \left(\hat {c} _ {j} \mid b _ {j} ^ {*}\right). \tag {10}
$$
A pseudocode of our unified algorithm for detecting both over and under confidence in predictions appears in Algorithm 5.
Algorithm 5: Coverage-Based Detection - Unified
1 // Fit  
2 Input Training:  $S_{m}, \delta, \kappa_{f}, C_{\text{target}}$   
3 Generate  $C_{\text{target}}$  uniformly spread coverages  $c^{*}$   
4 for  $j = 1$  to  $C_{\text{target}}$  do  
5  $b_{j}^{*}, \theta_{j} = \text{SGC}(S_{m}, \delta, c_{j}^{*}, \kappa_{f})$   
6  $b_{j}^{*}; \text{up}, \theta_{j}^{\text{up}} = \text{SGC}-\text{UP}(S_{m}, \delta, c_{j}^{*}, \kappa_{f})$   
7 end for  
8 Output Training:  $\{(b_{j}^{*}, \theta_{j})\}_{j=1}^{C_{\text{target}}}, \{(b_{j}^{*}; \text{up}, \theta_{j}^{\text{up}})\}_{j=1}^{C_{\text{target}}}$   
9 // Detect  
10 Input Detection:  $\{(b_{j}^{*}, \theta_{j})\}_{j=1}^{C_{\text{target}}}, \{(b_{j}^{*}; \text{up}, \theta_{j}^{\text{up}})\}_{j=1}^{C_{\text{target}}}, \kappa_{f}, \alpha, k$   
11 while True do  
12 Receive window  $W_{k} = \{x_{1}, x_{2}, \ldots, x_{k}\}$   
13 Calculate SOV {see Equation (10)}  
14 Obtain p-value from t-test,  $H_{0}: \text{SOV} = 0, H_{1}: \text{SOV} > 0$   
15 if  $p_{value} < \alpha$  then  
16 Shift Detected ← True  
17 Output Detection: Shift Detected,  $p_{value}$   
18 end if  
19 end while