Table 3: Average and standard deviation of the distance-based visual alignment and reliability scores across five seeds on the open test set. Bold indicates the best performance in each category, and underline is the second best. Deep Ensemble does not have a standard deviation since it uses the output of all five models.
Visual Alignment (↓)	Reliability score (↑)
Must-Act	Must-Abstain	Uncertain	Average	RS0	RS450	RS900
Category 1	Category 2	Category 3	Category 4	Category 5	Category 6	Category 7	Category 8
ViT [11]
SP	0.261±0.051	0.556±0.029	0.367±0.038	0.793±0.057	0.808±0.057	0.787±0.056	0.792±0.059	0.671±0.032	0.629±0.021	313	-245837	-491987
ASP	0.208±0.036	0.514±0.033	0.325±0.022	1.000±0.000	1.000±0.000	1.000±0.000	1.000±0.000	0.767±0.010	0.727±0.007	253	-285047	-570347
MD [35]	0.390±0.030	0.658±0.025	0.485±0.023	0.725±0.021	0.721±0.023	0.726±0.023	0.664±0.025	0.623±0.012	0.624±0.005	270	-275580	-551430
KNN [67]	0.382±0.047	0.634±0.029	0.484±0.033	0.679±0.058	0.696±0.050	0.679±0.049	0.674±0.067	0.612±0.034	0.605±0.020	282	-264768	-529818
TAPUDD [13]	0.375±0.070	0.628±0.073	0.468±0.074	0.809±0.079	0.809±0.084	0.835±0.065	0.768±0.089	0.678±0.024	0.671±0.017	253	-285047	-570347
OpenMax [3]	0.238±0.027	0.536±0.033	0.344±0.022	0.804±0.050	0.816±0.037	0.804±0.059	0.766±0.055	0.696±0.025	0.626±0.020	335	-229165	-458665
MC-Dropout [16]	0.210±0.036	0.516±0.032	0.326±0.022	0.968±0.009	0.970±0.010	0.968±0.009	0.968±0.010	0.749±0.014	0.709±0.005	253	-285047	-570347
Deep Ensemble [32]	0.305	0.571	0.400	0.712	0.732	0.705	0.713	0.628	0.596	376	-205274	-410924
Swin Transformer [38]
SP	0.106±0.004	0.362±0.014	0.221±0.017	0.793±0.016	0.828±0.043	0.800±0.022	0.829±0.028	0.625±0.031	0.571±0.015	363	-225537	-451437
ASP	0.085±0.007	0.329±0.008	0.182±0.020	0.998±0.000	0.998±0.000	0.998±0.000	0.998±0.000	0.736±0.009	0.666±0.003	294	-268356	-537006
MD [35]	0.296±0.018	0.512±0.012	0.364±0.012	0.700±0.012	0.743±0.014	0.723±0.017	0.685±0.021	0.575±0.007	0.575±0.006	326	-248974	-498274
KNN [67]	0.370±0.017	0.580±0.008	0.456±0.018	0.549±0.025	0.590±0.013	0.545±0.022	0.554±0.035	0.543±0.007	0.523±0.012	526	-115124	-230774
TAPUDD [13]	0.201±0.053	0.427±0.048	0.278±0.046	0.876±0.058	0.889±0.048	0.898±0.049	0.844±0.073	0.663±0.022	0.635±0.013	294	-268356	-537006
OpenMax [3]	0.099±0.008	0.358±0.013	0.225±0.029	0.831±0.037	0.810±0.023	0.817±0.032	0.724±0.084	0.656±0.030	0.565±0.011	399	-208401	-417201
MC-Dropout [16]	0.092±0.007	0.338±0.008	0.191±0.020	0.947±0.001	0.957±0.006	0.951±0.002	0.953±0.003	0.705±0.011	0.642±0.003	294	-268356	-537006
Deep Ensemble [32]	0.132	0.377	0.253	0.725	0.766	0.734	0.768	0.584	0.542	434	-187666	-375766
DenseNet [27]
SP	0.094±0.017	0.258±0.023	0.183±0.019	0.813±0.017	0.852±0.015	0.819±0.012	0.864±0.036	0.614±0.008	0.562±0.007	392	-211558	-423508
ASP	0.079±0.013	0.224±0.023	0.159±0.018	0.997±0.000	0.997±0.000	0.997±0.000	0.997±0.000	0.747±0.008	0.650±0.004	312	-260238	-520788
MD [35]	0.170±0.016	0.323±0.022	0.250±0.025	0.873±0.014	0.866±0.019	0.854±0.009	0.825±0.032	0.620±0.022	0.598±0.006	339	-247611	-495561
KNN [67]	0.272±0.021	0.448±0.021	0.360±0.017	0.612±0.019	0.640±0.022	0.615±0.019	0.664±0.014	0.565±0.009	0.522±0.002	482	-157468	-315418
TAPUDD [13]	0.310±0.039	0.393±0.025	0.364±0.044	0.862±0.021	0.831±0.023	0.837±0.018	0.810±0.028	0.645±0.017	0.631±0.004	320	-249880	-500080
OpenMax [3]	0.093±0.015	0.288±0.023	0.199±0.027	0.764±0.049	0.817±0.054	0.734±0.058	0.823±0.058	0.590±0.016	0.539±0.025	461	-165589	-331639
MC-Dropout [16]	0.087±0.014	0.263±0.024	0.204±0.017	0.953±0.003	0.953±0.002	0.954±0.005	0.964±0.004	0.718±0.009	0.637±0.003	312	-260238	-520788
Deep Ensemble [32]	0.109	0.276	0.209	0.767	0.814	0.775	0.825	0.581	0.545	396	-209754	-419904
ConvNeXt [39]
SP	0.211±0.020	0.461±0.032	0.354±0.024	0.661±0.039	0.772±0.023	0.671±0.026	0.767±0.028	0.583±0.016	0.560±0.008	427	-180923	-362273
ASP	0.162±0.015	0.398±0.026	0.299±0.021	0.998±0.000	0.999±0.000	0.999±0.000	0.998±0.000	0.729±0.003	0.698±0.007	283	-268817	-537917
MD [35]	0.439±0.019	0.583±0.023	0.504±0.017	0.699±0.021	0.663±0.023	0.692±0.031	0.642±0.069	0.600±0.009	0.603±0.013	350	-213400	-427150
KNN [67]	0.376±0.007	0.574±0.014	0.484±0.009	0.613±0.027	0.654±0.013	0.627±0.024	0.621±0.019	0.565±0.004	0.564±0.010	451	-169649	-339749
TAPUDD [13]	0.448±0.018	0.578±0.013	0.518±0.019	0.796±0.016	0.732±0.016	0.799±0.014	0.752±0.035	0.656±0.007	0.660±0.010	278	-263422	-527122
OpenMax [3]	0.183±0.010	0.408±0.026	0.318±0.027	0.944±0.007	0.914±0.012	0.960±0.005	0.888±0.052	0.708±0.004	0.665±0.010	286	-261614	-523514
MC-Dropout [16]	0.166±0.016	0.403±0.026	0.303±0.021	0.941±0.003	0.958±0.002	0.942±0.001	0.955±0.003	0.699±0.002	0.671±0.006	283	-268817	-537917
Deep Ensemble [32]	0.228	0.480	0.368	0.621	0.743	0.633	0.738	0.563	0.547	455	-166045	-332545
MLP-Mixer [68]
SP	0.240±0.048	0.556±0.055	0.333±0.032	0.842±0.051	0.829±0.059	0.828±0.034	0.850±0.037	0.674±0.024	0.644±0.021	347	-231403	-463153
ASP	0.212±0.041	0.524±0.054	0.303±0.038	0.999±0.000	0.999±0.000	0.999±0.000	0.999±0.000	0.749±0.017	0.723±0.013	260	-285490	-571240
MD [35]	0.431±0.044	0.682±0.042	0.499±0.017	0.649±0.021	0.651±0.022	0.640±0.012	0.621±0.034	0.609±0.013	0.598±0.009	374	-209326	-419026
KNN [67]	0.414±0.034	0.680±0.038	0.491±0.017	0.621±0.013	0.641±0.013	0.610±0.014	0.634±0.013	0.584±0.015	0.584±0.015	402	-194448	-389298
TAPUDD [13]	0.586±0.013	0.742±0.020	0.631±0.021	0.624±0.023	0.600±0.021	0.620±0.022	0.603±0.034	0.619±0.014	0.628±0.012	297	-231003	-462303
OpenMax [3]	0.290±0.045	0.630±0.060	0.372±0.065	0.662±0.127	0.681±0.140	0.648±0.174	0.630±0.122	0.631±0.050	0.568±0.057	358	-218792	-437942
MC-Dropout [16]	0.213±0.041	0.525±0.054	0.303±0.037	0.977±0.008	0.974±0.009	0.975±0.005	0.977±0.006	0.737±0.017	0.710±0.013	260	-285490	-571240
Deep Ensemble [32]	0.297	0.597	0.370	0.735	0.714	0.719	0.743	0.614	0.599	376	-207524	-415424