SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note the score  $s$  in Algorithm 1 is
$$
s [ i ] = \sum_ {j \in \mathcal {T}} w (i, j) - 2 \sum_ {j \in \mathcal {N}} w (i, j) = \sum_ {j \in \mathcal {T} \backslash \mathcal {N}} w (i, j) - \sum_ {j \in \mathcal {N}} w (i, j).
$$
Thus the change in the objective value by moving data  $i$  to another partition is  $s[i] - \lambda$  for  $i\in \mathcal{T}\backslash \mathcal{N}$  and  $-s[i] + \lambda$  for  $i\in \mathcal{N}$ , which can be represented as  $v[i](s[i] - \lambda)$ . Therefore, moving a sample with a positive value of  $v[i](s[i] - \lambda)$  to another partition guarantees an increase in the objective function value. Because a cut value in a graph is bounded, the algorithm converges to the local optimum by the monotone convergence theorem.
A.2 Empirical convergence analysis
We conduct an empirical study on the convergence of Algorithm 1. Specifically, we randomly sample 100,000 data from ImageNet and construct a relation graph. We compare the set-level Algorithm 1 and the original Kerningham-Lin algorithm in Figure 11. The figure indicates that both algorithms converge to the local optimum, while our set-level algorithm converges faster. Additionally, we observe that the set-level algorithm achieves a lower objective value, verifying its effectiveness in large-scale settings.
Figure 11: Empirical convergence analysis of max-cut algorithms. Set denotes Algorithm 1 and Single denotes the Kerningham-Lin algorithm which updates a single node at each optimization step.
A.3 Computation time comparison
In this section, we measure the time spent on detection algorithms. We use 1 RTX3090-Ti GPU and conduct experiments on the full ImageNet training set. Table 4 compares computation time for Algorithm 1 and feature calculation. Note that all existing methods based on neural networks, including ours, require the calculation of features. The table shows that Algorithm 1 (excluding
Table 4: Time spent (s) for label error detection on ImageNet 1.2M dataset. Feature indicates the total computing time for calculating feature embeddings of all data points, and Gradient means the total computing time for calculating network gradient on each data point. Algorithm 1 indicates the time spent by our algorithm, excluding feature calculation.
Model	Feature	Algorithm 1	Gradient
MAE-Base	2300	400	6000
MAE-Large	6900	420	21000
Table 5: Time spent per sample (ms) for OOD detection on ImageNet with various models. Unary refers to unary scoring methods utilizing logit or probability score for each data point.
Model	Unary	Relation
MAE-Large	12.2	12.3
ResNet-50	8.1	8.2
feature calculation) requires significantly less computation time than forward computation. We also observe that our algorithm efficiently scales up to larger neural networks which have a larger number of feature embedding dimensions. It is also worth noting that computing gradient takes a much longer time and also requires a large memory budget, demonstrating the efficiency of our algorithm in large-scale label error detection.
In Table 5, we measure the time spent for OOD detection on the full ImageNet training set. The computation of our similarity kernel is embarrassingly parallelizable on GPUs. As shown in the table, the overhead time for computing our outlier scores is negligible compared to the time spent for the neural networks' forward computation on a single data point. We can further reduce the time and memory requirements by measuring the outlier score on a subset of the training set (Figure 7).
A.4 Interpretation of relation function
We establish an understanding of our relation function in Equation (1) by drawing a connection to the influence function [39]. For simplicity, we consider the influence function with a single checkpoint, where the influence between data  $x_{i}$  and  $x_{j}$  is given by  $\nabla_w\ell (x_i)^\top \nabla_w\ell (x_j)$ . Here,  $\ell$  denotes the loss function, and  $w$  denotes the weight of the checkpoint. We consider the influence function at the feed-forward layer, where  $\ell (x_{i}) = h(\mathbf{f}_{i}^{\prime}) = h(w^{\top}\mathbf{f}_{i})$ , following the convention [39]. By the chain rule, we can decompose the weight gradient as  $\nabla_w\ell (x_i) = \nabla_{\mathbf{f}'}h(\mathbf{f}_i')\mathbf{f}_i^\top$ , and represent the influence as  $\nabla_{\mathbf{f}'}h(\mathbf{f}_i')^\top \nabla_{\mathbf{f}'}h(\mathbf{f}_j')\cdot \mathbf{f}_i^\top \mathbf{f}_j$ . In contrast, our relation function has a form of  $1(y_{i} = y_{j})\cdot |s(\mathbf{f}_{i},\mathbf{f}_{j})\cdot c(\mathbf{p}_{i},\mathbf{p}_{j})|^{t}$ .
The main distinction between our relation function and the influence function is the existence of the feature gradient term  $\nabla_{\mathbf{f}^{\prime}}h(\mathbf{f}_{i}^{\prime})$ . As observed in Barshan et al. [2], outliers have a large feature-gradient norm, leading to difficulties in detecting label errors. Specifically, let us consider the weight  $w$  at the classifier layer, where the function  $h$  is the softmax cross-entropy loss function. As Pruthi et al. [39], we can express the feature gradient inner-product as
$$
\nabla_ {\mathbf {f} ^ {\prime}} h \left(\mathbf {f} _ {i} ^ {\prime}\right) ^ {\top} \nabla_ {\mathbf {f} ^ {\prime}} h \left(\mathbf {f} _ {j} ^ {\prime}\right) = \left(\mathbf {y} _ {i} - \mathbf {p} _ {i}\right) ^ {\top} \left(\mathbf {y} _ {j} - \mathbf {p} _ {j}\right),
$$
where  $\mathbf{y}_i$  denote the one-hot label. The equation above shows that the correctly classified data with  $\mathbf{y}_i \approx \mathbf{p}_i$  yields near zero inner-product values, whereas outliers with high entropy predictions exhibit large inner-product values. Consequently, existing influence-based label error detection methods, which detect label errors by identifying data with high influence values, have degraded performance in the presence of outliers [2].
Our relation function differs from influence functions in that it separates label and prediction information using a label comparison term  $1(y_{i} = y_{j})$  and a compatibility term  $c(\mathbf{p}_i,\mathbf{p}_j)$ , respectively. Outlier data typically have a high entropy of model predictions, resulting in lower compatibility values with other data [14]. On the other hand, normal data with label errors exhibit high compatibility values with other normal data. Our detection algorithms exploit these differences and achieve improved detection performance compared to the influence functions.
B Additional discussions
Why does relation graph work? We discuss the conceptual differences between our relation graph-based approach and previous baselines. Firstly, our method is data-centric, whereas the previous
Table 6: Hyperparameter settings. We use identical hyperparameters for all experiments of each task regardless of model types.
Task	t (Equation (3))	λ (Algorithm 1)
Label error	4	0.05
Outlier	6	-
OOD	1	-
Table 7: Label error detection performance over a range of  $\lambda$  values (ImageNet 8% noise, MAE-Large).
Metric \ λ	0.0	0.01	0.05	0.1
AP	0.522	0.524	0.527	0.527
TNR95	0.692	0.693	0.695	0.693
approaches rely on a unary score by models. Our approach identifies problematic data by comparing them to other data, which leads to more reliable identification of problematic data than unary scoring methods that are vulnerable to overfitting [38]. Secondly, our approach aggregates global relational information, whereas previous methods rely on local information such as  $k$ -nearest distance [50]. Considering all edge connections, our method obtains more representative information about the data distribution. Through temperature parameter  $t$  and efficient graph algorithms, we effectively process the entire relations and achieve the improved identification of problematic data.
Limitations and future works There are several promising future directions for our work. Firstly, the current experiments are limited to the classification task, and it would be valuable to apply our approach to a wider range of tasks, such as segmentation or generative models. These tasks may introduce new and interesting categories of problematic data arising from different label spaces and data structures. Secondly, integrating our method with human annotation and model training processes will also be valuable. This could involve using our approach to identify inconsistencies in label assignments or to conduct a fine-grained evaluation of models.
Broader impact Regarding social impact, we anticipate that our method will largely reduce the cost of the data cleaning and annotation process, while also providing developers with valuable insights into their data and training process through our data relation map. We do not anticipate negative social effects, as our method does not deal with socially sensitive issues.
C Experiment settings
Models For label error detection, we train models on datasets with label noise. In the case of ImageNet, we fine-tune the pre-trained MAE models following the official training codes², which train MAE-Large for 50 epochs and MAE-Base for 100 epochs. The MAE-Large model has a feature dimension of 1024, where the MAE-Base model has a dimension of 768. It is worth noting that the masked auto-encoding pre-training process of MAE does not utilize label information. We use 8 RTX3090-Ti GPUs for the training and 1 RTX3090-Ti GPU for executing our algorithm. In the case of ESC-50, we fine-tune the AST model with a 768 feature dimension for 25 epochs following the official training codes³. In the case of language domain tasks, we fine-tune RoBERTa-Base with a 768 feature dimension following the official training codes⁴, where we train models for 5 epochs. For other models, we use the trained models provided by the Timm library⁵. For all experiments, we use the inputs of the classification layers as feature embeddings. In the case of RoBERTa, this corresponds to the encoder output of the [CLS] token.
Hyperparameter As shown in Figure 8, we observe that a large value of temperature  $t$  benefits label error detection, while a moderate temperature value around 1 shows the best performance on OOD detection. We summarized the hyperparameter used in our experiments in Table 6. Note we use  $\lambda = 0.05$  in Algorithm 1 after scaling the label noisiness score to have a maximum absolute value of 1. We observe that the conservative estimation of noisy set  $\mathcal{N}$  by using small positive  $\lambda$  values leads to better results than  $\lambda = 0$ , as shown in Table 7. The results in the table confirm that our method
$^{2}$ https://github.com/facebookresearch/mae
<sup>3</sup>https://github.com/YuanGongND/ast
4 https://github.com/facebookresearch/fairseq/tree/main/examples/roberta
5 https://github.com/rwrightman/pytorch-image-models
Table 8: Validation top-1 accuracy of MAE-Large trained on ImageNet with noisy labels.
Label Noise Ratio	0.	0.04	0.08	0.12	0.15
Top-1 Accuracy	85.89	84.96	84.15	82.88	81.50
Table 9: Label error detection AP by the temporal model ensemble with 4 MAE-Large checkpoints (ImageNet,  $8\%$  label noise). In parenthesis, we denote the performance gain compared to the detection by a single converged model.
Entropy	Least-conf.	CWE	Loss	TracIn	Margin	Relation
0.246 (0.007)	0.282 (0.001)	0.397 (0.031)	0.465 (0.041)	0.449 (0.034)	0.544 (0.06)	0.562 (0.036)
performs effectively over a wide range of  $\lambda$  values, outperforming the best scores of 0.484 AP and 0.521 TNR95 from the six baselines tested.
Other tricks We find that small noisy kernel values accumulate errors as we consider large numbers of data. To resolve this issue, we clamp small similarity kernel values that fall below an absolute value of 0.03 as zero in Equation (3).
C.2 Synthetic label noise
In Section 4.1, we conduct controlled experiments by generating synthetic label noise. Specifically, we flip labels of a certain percentage of training data with the top-2 prediction of trained models on correctly classified data. For the label flip, we use MAE-Huge for ImageNet and RoBERTa-Large for language datasets. Note that we did not use these models for detecting label errors to prevent possible correlations. In the case of speech domain, we use the identical AST architecture. We note that the original ImageNet training set may contain label issues which can lead to misleading experimental results [34]. Therefore, we remove about  $4\%$  of data that are likely to have label issues by following Northcutt et al. [33] with MAE-Huge, resulting in a total of 1,242,890 data samples. We conduct label error detection experiments on this pre-cleaned training set. In Table 8, we provide the top-1 validation accuracy of MAE-Large trained on training sets with noisy labels, demonstrating the importance of label noise identification and cleaning.
C.3 Outlier detection setting
As outlier detection is not well-benchmarked in modern computer vision [59], we design an experimental setup with the ImageNet dataset and two outlier datasets: SUN [58] and iNaturalist [53], each with 10k outlier data. To ensure an appropriate outlier data ratio ( $\sim 8\%$ ), we use ImageNet-100 [21], a subset of ImageNet consisting of 120k data from 100 classes. We adopt the outlier detection setting of Wang et al. [56], where the training set has the outlier data with random labels. To consider multiple types of outlier data, we construct two noisy training sets: ImageNet-100 with SUN and ImageNet-100 with iNaturalist. We train a ViT-Base model [8] from scratch for 300 epochs on these noisy training datasets and measure outlier detection performance using the trained model.
D.1 Temporal model ensemble
Following Pruthi et al. [39], we measure the label error detection performance by using the temporal model ensemble. Specifically, we average the label noisiness scores from 4 checkpoints that are uniformly sampled throughout training. Table 9 shows that this technique improves the performance of all methods, with our approach still exhibiting the best performance. These results confirm the effectiveness of temporal ensembles when more computation and storage are available.
Figure 12: Normalized histogram of label noisiness scores by simple edge aggregation (Sum) and max-cut. Arrows indicate differences between the density functions.
D.2 Analysis on edge aggregation
In this section, we analyze the difference between label noisiness scores calculated using max-cut and simple edge sum. Specifically, in Figure 12, we draw histograms of label noisiness scores for two groups of data: one with clean labels and the other with label errors. The figure shows that the score calculated through max-cut shifts the density in a positive direction for data with clean labels and in a negative direction for data with label errors, compared to the simple edge sum. This indicates that the label noisiness score calculated with max-cut better separates label errors from the clean data. In this case, TNR95 increases from 0.628 to 0.647 by applying the max-cut.
D.3 Label error detection
In this section, we provide the exact performance values for Figure 5, including AUROC results. We report label error detection performances under various noise levels in Table 10, and provide performances according to the number of data in Table 11. The tables confirm that our relation graph approach achieves the best label error detection performances in all three metrics, regardless of the noise ratio and the number of data. In Tables 12 to 14, we provide performance values of all baselines considered in Table 1.
D.4 OOD detection
In Tables 15 and 16, we provide OOD detection results on individual datasets mentioned in Section 4.2: Places, SUN, iNaturalist, and Textures. We report the OOD detection performance of MAE-Large in Table 15 and the performance of ResNet-50 in Table 16. The tables demonstrate that our approach achieves the best OOD detection performance on three out of four datasets considered, while achieving the best overall performance. Furthermore, our method shows the best performance on all three metrics with both models, which highlights its effectiveness in detecting OOD data.
E Additional qualitative results
E.1 Relation map
In Figure 13, we present additional relation maps on ImageNet. We draw the relation maps with MAE-Large and ResNet-50. Note, MAE-Large utilizes masked auto-encoding pre-training whereas ResNet-50 is trained from scratch. From the figure, we observe that the models exhibit similar distributions of positive and negative relations for each data point. However, the ResNet model tends to have an overall larger variance of relation, indicating that the pre-training process reduces the relation variance and is helpful in forming relationships between data.
The visualization tool helps us to comprehend the distribution of complementary and conflicting relations associated with a data point, aiding dataset analysis and debugging. For example, in Figure 13, the relation map of the Ram sample exhibits a combination of positive and negative relations. Specifically, it exhibits positive relations with other Ram class samples while showing negative relations with samples from the Big Horn class, which are visually challenging to distinguish.
This observation suggests the need for multi-labeling or refinement of the label space definition. In this way, we can intuitively identify problems in the dataset by using the visualization tool.
E.2 Qualitative results
We present problematic data samples detected by our algorithm. Specifically, Figure 14 shows ImageNet samples with label errors and their most conflicting data samples having negative relation values. Figure 15 shows the outlier data detected by our algorithm on the ImageNet validation set, indicating the existence of inappropriate data for evaluation. We also present results on SST2 [55], a binary text sentiment classification dataset, in Tables 17 and 18.
Table 10: Label error detection performance on a range of label error ratios (MAE-Large, ImageNet).
Method\Ratio	AUROC	AP	TNR95
4%	8%	12%	15%	4%	8%	12%	15%	4%	8%	12%	15%
Entropy	0.790	0.796	0.801	0.803	0.153	0.246	0.334	0.389	0.115	0.137	0.162	0.181
Least-conf.	0.829	0.836	0.843	0.847	0.176	0.282	0.380	0.443	0.223	0.270	0.327	0.368
CWE	0.845	0.845	0.845	0.844	0.301	0.397	0.471	0.506	0.135	0.160	0.188	0.212
Loss	0.864	0.864	0.865	0.865	0.328	0.424	0.497	0.530	0.229	0.276	0.334	0.374
TracIn	0.889	0.888	0.887	0.885	0.303	0.415	0.498	0.536	0.503	0.521	0.556	0.582
Margin	0.876	0.875	0.876	0.876	0.403	0.484	0.544	0.568	0.346	0.392	0.457	0.505
Relation (Ours)	0.917	0.914	0.910	0.904	0.437	0.526	0.590	0.611	0.695	0.695	0.687	0.683
Table 11: Label error detection performance according to the number of data (MAE-Large, ImageNet,  $8\%$  label noise).
Method\#data	AUROC	AP	TNR95
12k	25k	100k	400k	1.2M	12k	25k	100k	400k	1.2M	12k	25k	100k	400k	1.2M
Entropy	0.806	0.801	0.797	0.796	0.796	0.250	0.248	0.248	0.246	0.246	0.161	0.161	0.144	0.137	0.137
Least-conf.	0.844	0.841	0.836	0.837	0.836	0.289	0.287	0.284	0.283	0.282	0.255	0.269	0.275	0.277	0.27
CWE	0.852	0.850	0.845	0.845	0.845	0.390	0.394	0.392	0.394	0.397	0.169	0.184	0.177	0.163	0.16
Loss	0.869	0.868	0.863	0.864	0.864	0.415	0.421	0.418	0.421	0.424	0.260	0.298	0.282	0.284	0.276
TracIn	0.891	0.890	0.886	0.887	0.888	0.408	0.415	0.409	0.412	0.415	0.554	0.541	0.523	0.524	0.521
Margin	0.878	0.878	0.874	0.875	0.875	0.481	0.486	0.477	0.481	0.484	0.376	0.424	0.412	0.398	0.392
Relation (Ours)	0.910	0.912	0.912	0.914	0.914	0.502	0.515	0.518	0.523	0.526	0.679	0.692	0.692	0.696	0.695
Table 12: Label error detection performance on various scales of MAE (ImageNet,  $8\%$  noise).
Scale	Metric	Entropy	Least-conf.	CWE	Loss	TracIn	Margin	Relation
Base	AP	0.231	0.280	0.373	0.412	0.393	0.477	0.514
TNR95	0.120	0.217	0.132	0.223	0.488	0.342	0.672
Large	AP	0.246	0.282	0.397	0.424	0.415	0.484	0.526
TNR95	0.137	0.270	0.160	0.276	0.521	0.392	0.695
Table 13: Label error detection performance on speech (ESC-50) and language (SST2, MNLI) datasets with  $10\%$  label noise. For ESC-50, we use the AST model [10], and for SST2 and MNLI, we use the RoBERTa-Base model [30].
Dataset	Metric	Entropy	Least-conf.	CWE	Loss	TracIn	Margin	Relation
ESC-50	AP	0.715	0.737	0.720	0.737	0.739	0.737	0.779
TNR95	0.784	0.783	0.790	0.783	0.789	0.793	0.847
SST2	AP	0.227	0.227	0.861	0.861	0.854	0.861	0.881
TNR95	0.175	0.175	0.850	0.850	0.837	0.850	0.870
MNLI	AP	0.199	0.197	0.753	0.764	0.724	0.754	0.758
TNR95	0.103	0.104	0.494	0.497	0.510	0.514	0.660
Table 14: Label error detection AP on ImageNet validation set. All the model scales are Large.
Model	Entropy	Least-conf.	CWE	Loss	TracIn	Margin	Relation
MAE	0.558	0.609	0.688	0.703	0.695	0.708	0.733
BEIT	0.614	0.644	0.707	0.719	0.718	0.718	0.737
ConvNeXt	0.587	0.625	0.696	0.710	0.700	0.713	0.735
ConvNeXt-22k	0.617	0.642	0.707	0.722	0.719	0.724	0.744
Table 15: OOD detection performance on ImageNet with MAE-Large.
Method\OOD dataset	AUROC	AP	TNR95
ALL	Places	SUN	iNat.	Textures	ALL	Places	SUN	iNat.	Textures	ALL	Places	SUN	iNat.	Textures
MSP	0.857	0.835	0.833	0.907	0.853	0.818	0.543	0.567	0.699	0.514	0.428	0.386	0.315	0.651	0.376
Max Logit	0.824	0.787	0.790	0.881	0.850	0.808	0.531	0.557	0.700	0.570	0.138	0.106	0.088	0.320	0.216
Mahalanobis	0.875	0.819	0.841	0.948	0.904	0.815	0.471	0.489	0.722	0.532	0.521	0.407	0.474	0.847	0.599
Energy	0.776	0.728	0.738	0.829	0.837	0.757	0.446	0.467	0.592	0.553	0.074	0.060	0.050	0.127	0.154
ReAct	0.896	0.862	0.872	0.944	0.909	0.857	0.586	0.611	0.746	0.634	0.550	0.449	0.463	0.793	0.547
KL-Matching	0.877	0.848	0.857	0.928	0.874	0.818	0.500	0.526	0.708	0.455	0.548	0.473	0.490	0.738	0.511
Iterative sampling	0.444	0.377	0.409	0.457	0.600	0.382	0.127	0.134	0.145	0.173	0.073	0.051	0.061	0.125	0.131
Local outlier factor	0.556	0.492	0.521	0.594	0.665	0.431	0.151	0.160	0.183	0.173	0.156	0.113	0.136	0.283	0.240
KNN	0.901	0.861	0.884	0.946	0.922	0.854	0.550	0.593	0.736	0.648	0.590	0.487	0.560	0.808	0.626
Relation (Ours)	0.911	0.883	0.894	0.951	0.921	0.874	0.618	0.653	0.782	0.642	0.636	0.547	0.587	0.810	0.641
Table 16: OOD detection performance on ImageNet with ResNet-50.
Method\OOD dataset	AUROC	AP	TNR95
ALL	Places	SUN	iNat.	Textures	ALL	Places	SUN	iNat.	Textures	ALL	Places	SUN	iNat.	Textures
MSP	0.847	0.829	0.836	0.896	0.814	0.782	0.469	0.501	0.641	0.369	0.496	0.471	0.461	0.634	0.352
Max Logit	0.844	0.827	0.833	0.894	0.807	0.767	0.445	0.468	0.605	0.331	0.482	0.463	0.448	0.622	0.340
Mahalanobis	0.693	0.644	0.628	0.735	0.823	0.567	0.228	0.209	0.270	0.375	0.265	0.194	0.229	0.436	0.371
Energy	0.836	0.821	0.825	0.883	0.798	0.721	0.398	0.407	0.503	0.269	0.481	0.463	0.448	0.622	0.340
ReAct	0.624	0.610	0.623	0.626	0.650	0.446	0.185	0.189	0.188	0.128	0.361	0.302	0.370	0.429	0.371
KL-Matching	0.838	0.811	0.822	0.894	0.817	0.784	0.439	0.463	0.693	0.404	0.307	0.268	0.224	0.477	0.291
Iterative sampling	0.624	0.530	0.578	0.659	0.809	0.516	0.167	0.190	0.229	0.380	0.150	0.112	0.134	0.223	0.269
Local outlier factor	0.648	0.556	0.602	0.686	0.824	0.533	0.175	0.199	0.244	0.397	0.186	0.143	0.167	0.272	0.315
KNN	0.822	0.743	0.783	0.884	0.922	0.764	0.335	0.384	0.546	0.774	0.380	0.295	0.372	0.587	0.496
Relation (Ours)	0.870	0.830	0.853	0.922	0.879	0.818	0.493	0.543	0.708	0.513	0.515	0.429	0.498	0.691	0.465
Clumber
Norwich terrier
Water ouzel
MAE
Figure 13: Data relation maps on ImageNet with MAE-Large and ResNet-50. We denote the assigned label above each image. The color represents the relation value at the last checkpoint. The x-axis is the standard deviation and the y-axis is the mean value of the relation values throughout training.
Bald eagle (-0.761)
Bald eagle (-0.760)
Bald eagle (-0.759)
Lesser panda (-0.761)
Lesser panda (-0.758)
Lesser panda (-0.757)
Monarch (-0.763)
Monarch (-0.754)
Monarch (-0.753)
Hartebeest (-0.763)
Hartebeest (-0.761)
Steam locomotive (-0.751)
Steam locomotive (-0.749)
Redshank
Ruddy turnstone (-0.761)
Ruddy turnstone (-0.758)
Figure 14: The first column shows the data samples detected by our label error detection algorithm using MAE-Large on ImageNet. We present the samples with the most conflicting relation next to the detected samples. We denote the assigned label and the corresponding relation value above the image.
Ruddy turnstone (-0.757)
African grey
Breastplate
Alligator lizard
Gasmask
Barber chair
Figure 15: Data samples with the highest outlier score by our method on the ImageNet validation set. We denote the assigned label above the image.