Table 1: Recalibration Results for Balanced Classification. For each dataset and atypicality quantile, the best results are marked in bold. We provide the standard errors next to the means over 10 random seeds.
Atypicality	Uncalibrated	TS	AdaTS	Atypicality-Aware
ECE	RMSE	ECE	RMSE	ECE	RMSE	ECE	RMSE
GMM	ResNet152	ImageNet	0.2	0.029 ± 0.001	0.077 ± 0.001	0.014 ± 0.001	0.066 ± 0.002	0.014 ± 0.001	0.064 ± 0.002	0.016 ± 0.001
0.4	0.039 ± 0.001	0.086 ± 0.001	0.020 ± 0.001	0.074 ± 0.001	0.020 ± 0.002	0.075 ± 0.002	0.019 ± 0.001
0.6	0.050 ± 0.001	0.097 ± 0.001	0.026 ± 0.001	0.079 ± 0.002	0.025 ± 0.002	0.081 ± 0.002	0.026 ± 0.000
0.8	0.062 ± 0.001	0.106 ± 0.001	0.024 ± 0.001	0.076 ± 0.001	0.023 ± 0.002	0.078 ± 0.002	0.024 ± 0.001
1.0	0.084 ± 0.001	0.123 ± 0.001	0.037 ± 0.001	0.088 ± 0.001	0.033 ± 0.004	0.085 ± 0.003	0.026 ± 0.001
ResNet18	ImageNet	0.2	0.017 ± 0.001	0.074 ± 0.001	0.031 ± 0.001	0.084 ± 0.001	0.028 ± 0.004	0.081 ± 0.004	0.016 ± 0.001
0.4	0.016 ± 0.001	0.074 ± 0.001	0.020 ± 0.001	0.079 ± 0.001	0.023 ± 0.003	0.079 ± 0.003	0.015 ± 0.001
0.6	0.025 ± 0.001	0.080 ± 0.001	0.023 ± 0.001	0.077 ± 0.001	0.024 ± 0.003	0.080 ± 0.002	0.019 ± 0.001
0.8	0.040 ± 0.001	0.092 ± 0.001	0.024 ± 0.001	0.078 ± 0.001	0.029 ± 0.002	0.082 ± 0.003	0.018 ± 0.001
1.0	0.054 ± 0.001	0.100 ± 0.001	0.033 ± 0.001	0.085 ± 0.001	0.032 ± 0.003	0.083 ± 0.003	0.017 ± 0.002
ResNet50	ImageNet	0.2	0.021 ± 0.001	0.069 ± 0.002	0.018 ± 0.001	0.071 ± 0.001	0.017 ± 0.001	0.068 ± 0.002	0.018 ± 0.001
0.4	0.029 ± 0.001	0.079 ± 0.001	0.023 ± 0.001	0.077 ± 0.001	0.020 ± 0.002	0.074 ± 0.001	0.022 ± 0.001
0.6	0.041 ± 0.001	0.091 ± 0.001	0.023 ± 0.001	0.078 ± 0.001	0.026 ± 0.002	0.079 ± 0.002	0.026 ± 0.001
0.8	0.042 ± 0.001	0.092 ± 0.001	0.024 ± 0.001	0.078 ± 0.001	0.024 ± 0.001	0.080 ± 0.002	0.020 ± 0.001
1.0	0.078 ± 0.000	0.118 ± 0.000	0.049 ± 0.001	0.095 ± 0.001	0.047 ± 0.004	0.096 ± 0.003	0.031 ± 0.001
RoBERTa	MNLI	0.2	0.005 ± 0.001	0.062 ± 0.004	0.013 ± 0.001	0.077 ± 0.002	0.006 ± 0.002	0.063 ± 0.005	0.006 ± 0.001
0.4	0.016 ± 0.001	0.091 ± 0.004	0.013 ± 0.001	0.083 ± 0.002	0.010 ± 0.002	0.075 ± 0.005	0.008 ± 0.001
0.6	0.034 ± 0.002	0.116 ± 0.002	0.015 ± 0.001	0.092 ± 0.004	0.016 ± 0.001	0.093 ± 0.005	0.017 ± 0.002
0.8	0.061 ± 0.002	0.153 ± 0.003	0.031 ± 0.002	0.113 ± 0.004	0.024 ± 0.002	0.104 ± 0.004	0.028 ± 0.002
1.0	0.065 ± 0.002	0.156 ± 0.002	0.028 ± 0.002	0.112 ± 0.003	0.021 ± 0.002	0.106 ± 0.004	0.023 ± 0.003
WideResNet28	CIFAR10	0.2	0.003 ± 0.000	0.041 ± 0.001	0.009 ± 0.000	0.057 ± 0.000	0.003 ± 0.000	0.051 ± 0.002	0.001 ± 0.000
0.4	0.005 ± 0.001	0.051 ± 0.002	0.007 ± 0.001	0.055 ± 0.001	0.002 ± 0.001	0.047 ± 0.002	0.002 ± 0.000
0.6	0.004 ± 0.001	0.050 ± 0.003	0.008 ± 0.001	0.058 ± 0.001	0.004 ± 0.001	0.049 ± 0.004	0.002 ± 0.000
0.8	0.009 ± 0.001	0.066 ± 0.003	0.004 ± 0.001	0.056 ± 0.002	0.003 ± 0.001	0.058 ± 0.003	0.002 ± 0.001
1.0	0.142 ± 0.002	0.240 ± 0.002	0.073 ± 0.002	0.168 ± 0.003	0.046 ± 0.002	0.129 ± 0.004	0.041 ± 0.002
WideResNet28	CIFAR100	0.2	0.007 ± 0.001	0.059 ± 0.003	0.018 ± 0.001	0.084 ± 0.001	0.021 ± 0.003	0.089 ± 0.005	0.021 ± 0.001
0.4	0.029 ± 0.001	0.110 ± 0.002	0.005 ± 0.001	0.070 ± 0.003	0.008 ± 0.002	0.073 ± 0.003	0.005 ± 0.001
1.0	0.161 ± 0.002	0.201 ± 0.002	0.052 ± 0.001	0.136 ± 0.002	0.044 ± 0.004	0.124 ± 0.007	0.049 ± 0.001
0.8	0.257 ± 0.004	0.297 ± 0.002	0.106 ± 0.003	0.191 ± 0.003	0.101 ± 0.003	0.187 ± 0.002	0.105 ± 0.003
1.0	0.371 ± 0.004	0.351 ± 0.002	0.105 ± 0.004	0.200 ± 0.003	0.101 ± 0.004	0.199 ± 0.003	0.109 ± 0.005
KNN	ResNet152	ImageNet	0.2	0.035 ± 0.001	0.085 ± 0.001	0.014 ± 0.001	0.062 ± 0.002	0.016 ± 0.001
0.4	0.039 ± 0.001	0.085 ± 0.001	0.015 ± 0.001	0.067 ± 0.001	0.018 ± 0.001	0.069 ± 0.002	0.017 ± 0.001
0.6	0.035 ± 0.001	0.083 ± 0.001	0.021 ± 0.001	0.076 ± 0.002	0.024 ± 0.003	0.079 ± 0.004	0.019 ± 0.001
0.8	0.057 ± 0.002	0.102 ± 0.001	0.031 ± 0.001	0.084 ± 0.001	0.033 ± 0.003	0.085 ± 0.003	0.032 ± 0.001
1.0	0.099 ± 0.001	0.129 ± 0.001	0.036 ± 0.001	0.090 ± 0.001	0.042 ± 0.006	0.093 ± 0.004	0.037 ± 0.001
ResNet18	ImageNet	0.2	0.025 ± 0.001	0.077 ± 0.002	0.020 ± 0.001	0.074 ± 0.001	0.037 ± 0.011	0.115 ± 0.042	0.019 ± 0.001
0.4	0.017 ± 0.001	0.072 ± 0.001	0.021 ± 0.001	0.076 ± 0.001	0.045 ± 0.255	0.120 ± 0.046	0.019 ± 0.001
0.6	0.021 ± 0.001	0.074 ± 0.001	0.027 ± 0.001	0.083 ± 0.001	0.052 ± 0.299	0.136 ± 0.558	-
1.0	-	-	-	-	-	-	-
1.5	-	-	-	-	-	-	-
1.5	-	-	-	-	-	-	-
1.5	-	-	-	-	-	-	-
1.5	-	-	-	-	-	-	-
1.5	-	-	-	(1) - (2) - (3) - (4) - (5) - (6) - (7) - (8) - (9) - (11) - (12) - (13) - (14) - (15) - (16) - (17) - (18) - (19) - (2) - (3) - (4) - (5) - (6) - (7) - (8) - (9) - (11) - (12) - (13) - (14) - (15) - (16) - (17) - (18) - (19) - (2) - (3) - (4) - (5) - (6) -(7) - (8) - (9) - (11) - (12) - (13) - (14) - (15) - (16) - (17) - (18) - (19) - (2) - (3) - (4) - (5) - (6) - (7) - (8) - (9) - (11) - (12) - (2) - (3) - (4) - (5) - (6) - (7) - (8) - (9) - (11) - (12) - (2) - (3) - (4) - (5) - (6) - (7) - (8) - (9) - (11) - (12) - (2) - (3) - (4) - (6) - (7) - (8) - (9) - (11) - (12) - (2) - (3) - (4) - (5) - (6) - (7) - (8) - (9) - (11) - (12) - (2) - (3) - (4) - (5) - (6) - (7) - (8) - (11) - (12) - (2) - (3) - (4) - (5) - (6) - (7) - (8) - (9) - (11) - (12) - (2) - (3) - (4) - (5) - (6) - (7) - (8) - (9) - (11) - (12) - (2)
-	-	-	-	-	-
1.5	-	-	-	-	-	-	-
1.5	-	-	-	-	-	-	-
1.5	-	-	-	-	-	-	-
1.5	-	-	-	-	-
1.5	-	-	-	-	-
1.5	-	-	-	-	-
1.5	-	-	-	-	-
1.5	-	-	-	-	-
1.5	-	-	-
1.5	-	-	-
1.5	-	-	-
1.5	-	-	-
1.5	-	-	-
1.5	-	-	-