SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In accordance with Requirement 4, we ask 134 individuals per image to estimate the indisputable ground truth distribution within an error bound of  $5\%$ , following the survey sampling theory. Proofs are provided in Appendix F. Additionally, we calculate the Fleiss' Kappa [15] to assess two types of consistency among the MTurk workers' answers: intra-annotator and inter-annotator consistency. Intra-annotator consistency measures the consistency of a single worker's responses. To calculate this, we inserted two sets of identical images in random order. If a worker selects the same answers for these identical images, we consider the worker's responses to be consistent. Inter-annotator consistency, on the other hand, measures the agreement among different workers. Our results show an intra-annotator consistency value of  $\kappa = 0.91$ , indicating almost perfect agreement, and an inter-annotator consistency value of  $\kappa = 0.80$ , demonstrating substantial agreement. Details on survey instructions, response filtering process, and participant statistics are provided in Appendix F.
3.4 Dataset
We prepare three datasets: the train set, the open test set, and the closed test set. The train set is a subset of ImageNet-21K [58], consisting only of Category 1 samples. By doing so, we ensure the trained models are tested on a variety of unseen categories, reflecting a real-world scenario. For each of our 10 classes, we randomly sample a uniform amount of images from all related ImageNet-21K classes. We collected a total of 1250 images per class, using one-tenth of this data for validation. The creation processes of both the open and closed test sets are identical, as described above. We provide the open test set to allow developers to evaluate their models' visual perception alignment. Developers
<sup>5</sup>We leveraged open-sourced code available at https://github.com/hendrycks/robustness
Some might wonder why the machines should settle for aligning with human visual perception, rather than aiming to correctly classify even the most corrupted images (i.e. aim for superhuman visual perception). We provide arguments for the necessity of the former in Appendix D.2.2.
Table 1: The comparison between VisAlign and other related datasets on the requirements we define.  $\triangle$  indicates that only a subset of our scenarios are covered.
Dataset	Req. 1	Req. 2	Req. 3	Req. 4
ImageNet-C [22]	X	X	△	✓
ImageNet-A [25]	X	X	X	✓
OpenOOD [74]	X	X	△	✓
Background Challenge [73]	X	X	X	✓
MNIST [34]	X	✓	X	✓
CIFAR10 [30]	X	✓	X	✓
CIFAR10H [52]	X	✓	△	✓
PLEX [69]	X	X	✓	✓
Park et al. [49]	✓	X	△	X
DCIC [65]	X	X	△	✓
VisAlign	✓	✓	✓	✓
wishing to evaluate their models on the closed test set can submit their models to us. Table 1 presents a comparison of VisAlign and other datasets in terms of fulfilling the four requirements.
4 Metrics
We introduce a distance-based metric to measure AI-human visual alignment. Furthermore, we present a reliability score table to explore the correlation between a model's visual perception alignment and model reliability.
4.1 Distance-Based Visual Perception Similarity Metric
We propose a distance-based metric to measure the distance between two multinomial distributions: the human visual distribution and the model output distribution over 11 classes (10 mammals + abstention). We opt for a distance-based metric for two reasons: 1) it does not depend on additional hyperparameters such as abstention threshold, and 2) comparison across all classes, rather than solely on the true class, provides a more accurate measure of visual alignment. For example, consider a Must-Act tiger sample with the gold human label as a one-hot vector for the label tiger. Suppose one model outputs a probability of 0.7 for tiger and 0.3 for abstention, and another model yields a probability of 0.7 for tiger and 0.1 for zebra, elephant, and giraffe respectively. These two models differ in visual perception alignment: the former is uncertain between two classes, whereas the latter is indecisive among four classes. If we were to consider only the gold label's probability, both models would yield the same result, which would not accurately represent visual alignment. Hence, we employ a distance-based metric calculated across all 11 classes, as opposed to using the maximum or gold label probability.
Specifically, we employ the Hellinger distance [47] to measure the difference between the two probability distributions as summarized in Eq. 1. Compared to other metrics for comparing two multinomial distributions, Hellinger distance produces smooth distance values even for extreme (e.g., one-hot) distributions (unlike KL Divergence [10]) and considers all classes while calculating the distance (unlike Total Variation distance). For instance, given a human visual distribution of [1., 0., 0.] and two model output distributions [0.3, 0., 0.7] and [0.3, 0.4, 0.3], the two output distributions would have the same KL Divergences with the human distribution while they have different Hellinger distances. Hellinger distance accounts not only for the gold label probability but also for the probabilities of all other labels. Additionally, as its range lies between 0 and 1, it provides an intuitive indication of model alignment.
$$
h (P, Q) = \frac {1}{\sqrt {2}} \sum_ {i} \| \sqrt {p} _ {i} - \sqrt {q} _ {i} \| _ {2} \tag {1}
$$
4.2 Reliability Score with Abstention
We also assess the model's reliability based on its final action. This process involves two steps. First, a model abstains if the abstention probability surpasses an abstention threshold,  $\gamma$ ; otherwise, it makes a prediction. Next, if a model decides to act, its prediction is one of the 10 mammal classes
with the highest prediction probability. Table 2 details the reliability scores for each case. We devise separate metrics for Must-Act and Must-Abstain instances. For Uncertain samples, they are treated as Must-Act if the probability of the original label exceeds a threshold  $\lambda$ ; otherwise, they are treated as Must-Abstain. We set an initial  $\lambda$  value at 0.5, but this can be adjusted according to the specific objective. We denote the reliability score as  $RS_{c}(x)$ , where  $c$  is the cost of an incorrect prediction. The main criterion for assigning scores is the consequences of the model's decision. The model earns a score of 1 per prediction when it aligns best with human recognition: making a correct prediction in Must-Act and abstaining in Must-Abstain. On the other hand, if the model's decision is erroneous and could potentially result in significant cost—in our case, a wrong prediction—the model receives a score of  $-c$ . A score of zero indicates that the prediction is neither beneficial nor detrimental. Original Label Prediction is a special case only applied for Uncertain samples treated as Must-Abstain. In this case, a model correctly classifies a corrupted image that most humans cannot recognize. Although most humans disagree with the model's decision, it does not have a negative impact since it is a correct answer. The total score,  $RS_{c}$ , is the summation over all test samples,  $\sum_{i} RS_{c}(x_{i})$ .
The proper value of cost  $c$  depends on the industry and the use case.  $c$  can be seen as the "strictness criterion for a reliable model" and can also be interpreted as "how many misclassifications correspond to a single accurate classification."  $c$  can be set as an integer ranging from 0 to the total size of the test set. A value 0 for  $c$  implies a  $0\%$  strictness, while the maximum value of  $c$  implies a  $100\%$  strictness. This means that even a single mistake would result in a negative score, and abstaining from all decisions on Must-Act samples would be deemed more reliable than making even one incorrect prediction. We designed this metric to enable both absolute and relative reference points. As an absolute reference point, if the final score is at or above 0 (non-negative reliability score), it demonstrates
Table 2: Reliability score table. The optimal outcomes earn a score of 1. Abstention in Must-Predict and Original Label Prediction in MustAbstain get 0. The worst case receives  $-c$ , where  $c$  is the cost value. *Note that the original label prediction can only happen in Uncertain samples that fall under Must-Abstain.
Sample Type	Model Action	RSc(x)
Must-Act	Correct Prediction	+1
Incorrect Prediction	-c
Abstention	0
Must-Abstain	Original Label Prediction*	0
Other Prediction	-c
Abstention	+1
that the model satisfies the user-defined minimum reliability. A relative reference point is between different models; a model with a higher score between two reliability scores is more reliable. In this paper, we set the value of  $c$  as 0, 450, or 900.
We perform experiments with Transformer-based [71], CNN-based [33], and MLP-based models. We use ViT [11] and Swin Transformer [38] for Transformer-based models, and DenseNet [27] and ConvNeXt [39] for CNN-based models. For the MLP-based model, we use MLP-Mixer [68]. All models are trained on our train set and tested on the open test set.
We chose abstention functions that satisfy the following three conditions: 1) must be applicable on any model architecture, 2) do not require OOD or other Must-Abstain samples during training, and 3) do not require a supplementary model. We first calculate the abstention probability using each function, then re-normalize the 10-class prediction probability so that the sum over the 11 classes becomes 1. Since not every function outputs the abstention probability between 0 and 1, we designed a smaller version of the dataset with the identical gather process to test set to use for normalizing the abstention probability.
- Softmax Probability (SP) regards the entropy among the 10 classes as abstention probability.
- Adjusted Softmax Probability (ASP) acts the same as SP, but it applies temperature scaling and adds perturbations to the input image based on the gradients to decrease the softmax score. This method is inspired by ODIN [26].
- Mahalanobis detector (MD) [35] determines abstention probability based on the minimum Mahalanobis distance [40] calculated from each class distribution's mean and variance.
- KNN [67] uses the shortest  $k$ -Nearest Neighbor (KNN) distance between the feature of the test sample and the in-class features as an abstention probability.
- TAPUDD [13] extracts features from train set and split into  $m$  clusters using Gaussian Mixture Model (GMM). It determines the abstention probability based on the shortest Mahalanobis distance calculated from all clusters.
- OpenMax [3] represents each class as a mean activation vector (MAV) in the penultimate layer of the network. Next, the test sample distance from the corresponding class MAV is used to calculate the abstention probability.
- MC-Dropout [16] and Deep Ensemble [32] approximate model uncertainty using multiple predictions given by different dropouts and ensemble of networks, respectively. The average of the entropies over the 10 classes of each prediction determines the abstention probability.
5.2 Visual Alignment and Reliability Score
Table 3 presents both the distance-based visual alignment and the reliability scores on the open test set for all model and abstention function combinations. One key observation is that the performance differences between models are not significantly distinct, suggesting that visual alignment is more influenced by abstention functions than by the model architectures. For Must-Act categories, distance-based functions (MD, KNN, and TAPUDD) exhibit better visual alignment. Conversely, for Must-Abstain samples, probability-based methods (SP and ASP) align better with human perception. This implies that distance-based abstentions are generally more inclined to act, while probability-based abstentions are more likely to abstain. In Uncertain category, all abstention functions demonstrate
similar visual alignment performances, predominantly ranging from 0.5 and 0.6. We conjecture the reason comes from that all models are struggling in approximating the overall ratios across 11 classes compared to Must-Act and Must-Abstain, where models only need to correctly predict a single class. The difficulty of achieving visual perception alignment in Uncertain suggests that there is room for improvement. KNN [67] has the best visual alignment across all categories on average. This might be because KNN can capture more fine-grained features than other distance-based abstention functions, as it calculates the distance between samples, not clusters. We also compute three reliability scores with  $c$  set to 0 ( $RS_0$ ), 450 ( $RS_{450}$ ), and 900 ( $RS_{900}$ ). The resulting ratios of each action type are shown in Appendix G.1. Here,  $c = 0$  indicates no negative impacts from incorrect predictions, while  $c = 900$  suggests that a single incorrect prediction outweighs the remaining correct predictions. It is worth noting that reliability scores in  $RS_{450}$  and  $RS_{900}$  are mostly negative, suggesting that current models and abstention functions are not perfectly safe to be deployed in the real world. Notably, visual alignment distance is correlated with reliability score as can be seen in Appendix G.2.
Methods based on the minimum distance from each class (MD, KNN, and TAPUDD) generally show a worse visual alignment on Must-Abstain categories. We conjecture that the reason comes from using the shortest distance to in-class clusters. If an embedding contains one clear in-class feature, the distance to the corresponding class would be short, leading the model to make a prediction. On the other hand, methods based on entropy or uncertainty show weak alignment on Must-Act categories. With these methods, the model has to be not only confident that its predicted class is correct but also that the remaining classes are incorrect. Considering the confidence in all classes makes it more challenging for visual alignment in Must-Act categories. An abstention function which takes advantage of both distance-based and probability-based methods is needed to perform well on visual alignment. The distance should be sample-wise to capture the nuanced characteristics of the samples. Overall, our experiments show that no methods perform well across all categories. There is much room for improvement in visual alignment, a field in which our dataset will become an essential tool for benchmarking new methods.
To the best of our knowledge, this is the first work to construct a test benchmark for quantitatively measuring the visual perception alignment between models and humans, referred to as AI-human visual alignment, across diverse scenarios. Our dataset is divided into three main groups and eight categories, each representing unique and essential situations. Our dataset includes gold human labels for each image, with some of these labels collected via MTurk survey. We benchmarked five baseline models and seven popular abstention functions, and our experimental results show that no current methods perform well across all categories. This suggests there is room for improvement in visual alignment. We believe VisAlign can serve as a universal benchmark for testing visual perception alignment and that our work has potential applications in both social and industrial contexts.
Despite our best efforts to construct VisAlign, there are some limitations. First, the number of classes is relatively small compared to other datasets since we collected 134 annotations per image and chose classes that would be familiar to an average human. Note that it is always challenging to collect gold human labels in any domain. For example, in diagnosing chest X-rays, the typical number of diseases is 14. To collect the ground truth labels within a statistical error bound of  $5\%$ , one would need to consult at least 107 radiologists. Therefore, more practical solutions are required to measure alignment in specialized domains. Another limitation comes from the nature of uncertainty. We acknowledge that uncertainty is continuous and it is hard to distinguish between clear and uncertain images. Although we put significant effort to include only clear images in Must-Act and Must-Abstain and obtained human annotations on Uncertain images, there is a possibility of corner cases where at least one person disagrees. Furthermore, synthetic corruptions cannot cover all uncertainties arising in the real world. However, uncertainty is too broad to specify and difficult to collect or generate, thus for now we use corruptions. We put our best effort to reflect the continuity of uncertainty by varying corruption intensity from 1 to 10 and include some corruptions that can arise in the real world (e.g., pixelation). We detailed further discussions on uncertainty in Appendix I. Also, extending visual alignment to scenarios such as visual illusions may also be introduced. While our dataset focuses on the essential object identification and abstention task under AI-human visual alignment, future work can be expanded to potentially contentious but socially engaging topics such as gender or racial bias and other vision tasks such as object detection and segmentation.
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant (No.2019-0-00075, No.2022-0-00984) and National Research Foundation of Korea (NRF) grant (NRF-2020H1D3A2A03100945, NRF-2021H1D3A2A03038607), funded by the Korea government (MSIT). Marzyeh Ghassemi receives support from the Herman L. F. von Helmholtz Career Development Professorship and CIFAR Azrieli Global Scholar award.
The following section is answers to questions listed in datasheets for datasets.
- For what purpose was the dataset created?
VisAlign is created to serve as a benchmark for measuring visual perception alignment between AI models and humans.
The authors of this paper.
This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant (No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST)) and National Research Foundation of Korea (NRF) grant (NRF-2020H1D3A2A03100945), funded by the Korea government (MSIT).
- What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)?
VisAlign contains eight different types of images and their corresponding gold human labels.
There are a total of 12500 images in the train set, distributed equally among the 10 classes. The open test set and the closed test each contain 900 images: 100 images each in Categories 1 to 7 and 200 images in Category 8.
- Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set?
The train set is a sample of instances of ImageNet-21K, where images have been randomly sampled from synsets and corresponding hyponyms related to each of our classes. The test sets are samples carefully selected by the authors without replacement to match each of the categories' requirements.
- What data does each instance consist of?
Each instance consists of an image and its corresponding gold human label.
- Is there a label or target associated with each instance?
Yes, the label represents the gold label (e.g., human visual perception).
- Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text. N/A.
- Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)?  
N/A.
- Are there recommended data splits (e.g., training, development/validation, testing)?
No, since VisAlign is an universal benchmark that any model can be tested on regardless of its train set, a developer may feel free to use any training strategies.
- Are there any errors, sources of noise, or redundancies in the dataset? N/A.
- Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)?
The dataset relies on open source databases: ImageNet [60], ImageNet21K [58], ImageNet-C [22], DomainNet [50], and ImageNet-R [24].
- Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals' non-public communications)?  
N/A.
- Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? N/A.
- Does the dataset relate to people? Yes.
- Does the dataset identify any subpopulations (e.g., by age, gender)? N/A.
- Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? N/A.
- Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals race or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? N/A.
- How was the data associated with each instance acquired?
We leveraged open source datasets. For Category 2 and Category 5, we synthesized images using Stable Diffusion [59]. For Category 3, we manually applied FGSM [20] on samples in Category 1. For Category8, we applied corruptions on Category 1 samples by using corruption code available in ImageNet-C [22].
- What mechanisms or procedures were used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)?
We used the website Amazon Mechanical Turk (MTurk) to create gold human labels for Uncertain. After the poll, we used Excel, Google Sheets, and Python to process and label the collected data.
- If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)?
We first removed images that are hard to recognize or have more than two different objects. After the curating, when it involves sampling, we sampled with a fixed random seed.
- Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?
There were one part that required human involvement in the data collection process, deriving gold human label ratio for Uncertain. We provided $ 0.05 for classifying 25 images. We did not put any restrictions on participants.
Over what timeframe was the data collected?
The poll was conducted in March of 2023, but the results do not depend much on the date of date collection.
- Were any ethical review processes conducted (e.g., by an institutional review board)? N/A.
- Does the dataset relate to people? Yes.
- Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)?
We obtained via Amazon Mechanical Turk MTurk website.
- Were the individuals in question notified about the data collection? Yes.
- Did the individuals in question consent to the collection and use of their data? Yes.
- If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? N/A.
- Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted?
  - The dataset does not have individual-specific information.
- Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? For the data quality, we removed inappropriate responses (that fall under the distractors).
- Was the "raw" data saved in addition to the preprocessed/cleaned/ labeled data (e.g., to support unanticipated future uses)?  
N/A.
- Is the software that was used to preprocess/clean label the data available?
Preprocessing, cleaning, and labeling are done via Excel, Google Sheets, and Python.