SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiyoung Lee $^{1}$ , Seungho Kim $^{1}$ , Seunghyun Won $^{2}$ , Joonseok Lee $^{3}$ , Marzyeh Ghassemi $^{4,5,6}$ , James Thorne $^{1}$ , Jaeseok Choi $^{7}$ , O-Kil Kwon $^{7}$ , Edward Choi $^{1}$
$^{1}$ KAIST,  $^{2}$ Seoul National University Bundang Hospital,  $^{3}$ Seoul National University,  $^{4}$ MIT,  $^{5}$ University of Toronto,  $^{6}$ Vector Institute,  $^{7}$ Kangwon National University Hospital  $^{1}$  {jiyounglee0523, shokim, thorne, edwardchoi}  $@$  kaist.ac.kr  $^{2}$  shwon0213@gmail.com,  $^{3}$ joonseok2010@gmail.com,  $^{4}$ mghassem@mit.edu  $^{7}$  {gobiobotia, okkwon}  $@$  kangwon.ac.kr
AI alignment refers to models acting towards human-intended goals, preferences, or ethical principles. In this paper, we focus on the models' visual perception alignment with humans, further referred to as AI-human visual alignment. Specifically, we propose a new dataset for measuring AI-human visual alignment in terms of image classification. In order to evaluate AI-human visual alignment, a dataset should encompass samples with various scenarios and have gold human perception labels. Our dataset consists of three groups of samples, namely Must-Act (i.e., Must-Classify), Must-Abstain, and Uncertain, and further divided into eight categories. All samples have a gold human perception label; even Uncertain (e.g., severely blurry) sample labels were obtained via crowd-sourcing. The validity of our dataset is verified by sampling theory, statistical theories related to survey design, and experts in the related fields. Using our dataset, we analyze the visual alignment and reliability of five popular visual perception models and eight abstention methods. Our code and data is available at https://github.com/jiyounglee-0523/VisAlign.
AI alignment [62] seeks to align models to act towards human-intended goals [48, 78], preferences [66, 61], or ethical principles [28]. Misaligned models may show unexpected and unsafe behaviors which can bring about negative outcomes, including loss of human lives [54, 78]. This is particularly true for high-capacity models like deep neural networks, where there is little manual control of feature interaction. In such cases, analyzing the alignment between models and humans can be a proxy measure for safe behavior [45]. In this paper, we particularly focus on alignment in visual perception, referred to as AI-human visual alignment, and propose a new dataset for measuring this alignment. Note that recent work in AI-human alignment tends to focus on societal topics with ethical implications, such as racial or gender bias [70, 12, 42]. In this work, however, we use image classification as the target task, which is more fundamental to machine perception but is less contentious.
Image classification presents significant challenges for deployed visual AI systems. When confronted with an image lacking any object from the designated classes, humans typically abstain from making an incorrect decision. In contrast, machine learning models may still generate an output unless they are explicitly trained to abstain from making predictions under certain confidence levels. Similarly, when an image provides imperfect information (e.g., due to blurred vision or a dark environment), human decisions tend to waver between a correct prediction and abstention. Conversely, machines often
VisAlign: Al-Human Visual Alignment Dataset
Figure 1: The overview of VisAlign. The example images are given with reference to the class Zebra. Category 1. A photo-realistic image of a zebra. Category 2. A zebra crossing a road. Category 3. A slight noise is added to the Category 1 image. Category 4. A picture of a truck. Category 5. A head and two limbs of an elephant with the remaining body of a zebra. Category 6. A donkey. Category 7. A zebra illustrated on a piece of clothing. Category 8. Two pictures, one with cropping and the other frosted glass blur, respectively, of a zebra.
make overconfident predictions [46]. Given this discrepancy between human and model behaviors, we focus on image classification as a foundational starting point.
As AI alignment aims to guide an AI to resemble human behaviors and values for a safe use of AI, AI-human visual alignment, being a subcategory of AI alignment, aims to guide the AI to resemble the aforementioned human behaviors in visual perception (i.e., abstaining from making incorrect decisions, wavering between a correct prediction and abstention) to ensure safety across diverse use cases. Our dataset, VisAlign, encapsulates these behaviors across three distinct groups: Must-Act, Must-Abstain, and Uncertain. Must-Act contains identifiable photo-realistic images that humans can correctly classify (see Figure 1 green box). Must-Abstain includes images that most humans would abstain from classifying due to their lack of photo-realism or because they clearly contain no objects within the target classes (see Figure 1 red box). Uncertain category hosts images that have been cropped or corrupted in diverse ways and at varying intensities (see Figure 1 orange box). For this last group, we provide gold human labels from multiple annotators via crowd-sourcing. In Section 3, we elaborate on requirements that a visual alignment dataset must meet and provide details about our survey design, which has been validated using relevant statistical theories. Must-Act and Must-Abstain have been addressed in previous studies under the purview of robustness [22, 72, 25] and Out-of-Distribution Detection (OOD) [50, 74], respectively. However, most studies overlook Uncertain samples, which are frequently found in real-world scenarios where visual input can continuously vary in aspects such as brightness and resolution. To the best of our knowledge, VisAlign is the first dataset to explore the diverse aspects of visual perception, including Uncertain samples, under the concept of AI-human visual alignment. Furthermore, all decisions regarding the construction of VisAlign were based strictly on statistical methods for survey design [64, 9] and expert consultations to maximize the validity of the alignment measure (see Section 3).
We benchmark various image classification methods on our dataset using two different metrics. Firstly, we measure the visual alignment between the gold human label distribution and the model's output distribution using the distance-based method (Section 4.1). Secondly, we evaluate the model's reliability score (Section 4.2). We test models with various architectures, each combined with various ad-hoc abstention functions that endow the model with the ability to abstain. Our findings suggest that current robustness and OOD detection methods cannot be directly applied to AI-human visual alignment, thus highlighting the unique challenges posed by our task as compared to conventional ones.
- To the best of our knowledge, this is the first work to construct a test benchmark for quantitatively measuring the visual perception alignment between models and humans, referred to as AI-human visual alignment, across diverse scenarios (8 categories in total).
- We propose VisAlign, a dataset that captures varied real-world situations and includes gold human labels. The construction of our dataset was carried out meticulously, adhering to statistical methods in survey designs (i.e., the number of samples in a dataset [9], intra and inter-consistency in surveys [15], and the required minimum number of participants [64]) and expert consultations.
- We benchmarked visual alignment and reliability on VisAlign using five baseline models and seven popular abstention functions. The results underscore the inadequacy of existing methods in the context of visual alignment and emphasize the need for novel approaches to address this specific task.
Related Datasets. Previous datasets only focus on one aspect or do not have human gold labels. Mazeika et al. [41] focus on subjective interpretations and collected human annotations on emotions (e.g., amusement, interest, adoration). Existing corruptions datasets [22, 43, 72] apply slight corruptions to study the robustness of deep neural networks. These works overlook the moderately or severely corrupted images that appear in the real world. Although the dataset by Park et al. [49] applied brightness corruptions on hand X-ray images with multiple severities, they do not have gold human labels. CIFAR10H [52] is a dataset that collects a distribution of soft human labels for CIFAR10 images [30] to represent human perceptual uncertainty. Similarly, Schmarje et al. [65] collected multiple annotations per image. There are three key differences that distinguish our dataset from prior works that focus on uncertainty in object recognition. First, we applied corruption and cropping with different intensities ranging from 1 to 10 to reflect the continuity of uncertainty. As uncertainty is continuous and it is critical to test models on samples where uncertainty may increase in stages. Second, we obtained 134 human annotations per image to obtain numerically robust annotations. Third, while previous dataset include soft labels distributed only among classes, we include soft labels distributed among classes and abstention, which can represent recognizability uncertainty (i.e., whether an image itself is recognizable or not). Visual perception includes not only object identification (predicting that it is an elephant) but also object recognizability (the object itself is recognizable). In this sense, we cover broader scenarios compared to previous works as we include object recognizability uncertainty in our uncertain category.
Visual Alignment with Humans. Alignment is more broadly studied, including the gap between data collection and model deployment [2], natural language modeling [37], and object similarity [29, 51]. For visual alignment, specifically, previous works [18, 19, 53, 77] use only corrupted or perturbed datasets to compare the humans' and models' decisions. Zhang et al. [76] and Bomatter et al. [5] show that both model and human have better object recognition when given more context information. Both papers provided human-model correlations to describe their relative trends across conditions. However, our study on visual perception alignment is not about following human trends, but about measuring how well the model replicates human perception sample-wise. Geirhos et al. [17] and Bhojanapalli et al. [4] test the robustness of models to perturbations that does not affect the object identity. Peterson et al. [52] only test their models on in-class (i.e., Category 1) and out-of-class samples (i.e., Category 4 and Category 6) and Schmarje et al. [65] only tested their models on in-class samples (i.e., Category 1). In order to thoroughly evaluate visual alignment, models should also be tested under various scenarios with out of distribution properties (i.e., Category 5 and Category 7). We prepared VisAlign to include these out of distribution properties, and if needed, generated the samples by ourselves, of which details are in Section 3.2. Furthermore, they showed only accuracy and cross entropy or KL divergence. (which is analogous to KL divergence) of the models. Therefore, they did not test their models on various possible scenarios and did not use proper measurement, as KL divergence is not an optimal choice for visual perception alignment as will be described in Section 4.1. Therefore, although previous works trained their models with the goal of achieving visual perception alignment, none of the works have thoroughly verified how much the models have actually achieved visual perception alignment under diverse situations with an appropriate measurement. In contrast, we quantitatively measured visual perception alignment across various scenarios with multiple human annotations on uncertain images. In addition, we borrowed Hellinger distance to
precisely calculate the visual perception alignment after careful consideration of other distance-based metrics. More details of comparison to previous works are in Appendix J
We have carefully considered what conditions must be met in a visual alignment dataset during the process of selecting the classes and the contents of VisAlign. We define four requirements that a visual alignment dataset must satisfy:
- Requirement 1: Clear Definition of Each Class. Each class must be distinctly and precisely defined. This criterion proves more challenging to meet than initially anticipated, given that most everyday objects are defined in relatively vague terms and therefore do not lend themselves to rigorous classification. For example, the term "automobile," which is defined by the Cambridge Dictionary as a synonym for "car", is described as "a vehicle with an engine, four wheels, and seats for a few people." The phrase "seats for a few people" is ambiguous, and the definition is broad enough to encompass trucks. Despite this, certain parties may contend that "automobile" and "truck" are distinctly separate classes, a view reflected in datasets like CIFAR-10 [30] and STL-10 [8], which treat automobiles and trucks as separate classes.
- Requirement 2: Class Familiarity to Average Individuals. The classification target (i.e., each class) must be known to average people. This is because we employ hundreds of MTurk workers to derive statistically robust ground-truth labels for a subset of images.
- Requirement 3: Coverage of Diverse and Realistic Scenarios. Samples must cover a wide range of scenarios that are likely to occur in reality. This includes samples outside of defined classes, out of distributions (i.e., Category 5 or 7) and confusing samples where people might not be able to recognize or identify. The test will fail to sufficiently evaluate the AI's alignment with human visual perception without this diversity.
- Requirement 4: Ground Truth Label for Each Sample. Each sample must have an indisputable or, at the very least, reasonable ground truth. Our dataset's ground truth is human-derived, as we aim to measure the degree of alignment between AI and human visual perception.
3.1 Class Selection
For our dataset to serve as a universal benchmark that any model can be tested on, the classes should have clear definitions so that model developers can easily prepare their models and training strategy. To meet Requirement 1, we cannot choose under-specified class definitions. For example, the class definitions in CIFAR10 [30] can be disputed, as shown in the example of 'automobile' and 'truck' in Requirement 1. MNIST [34] classes cannot be used since numbers are recognized via trivial geometric patterns. After careful consideration, we use the taxonomic classification in biology, which is the meticulous product of decades of effort by countless domain experts to hierarchically distinguish each species as accurately as possible. Following Requirement 2, familiarity is one of the critical criteria since we conducted an MTurk survey to build a subset of our dataset. Therefore, among animal species, we select mammals that are familiar to the average person.
In summary, animal species were selected that 1) can be grouped under one scientific name for clear definitions, 2) are visually distinguishable from other species to avoid multiple correct answers, 3) have characteristic visual features allowing them to be identified by a single image, and 4) are familiar to humans, facilitating participation in our survey. The final 10 classes are Tiger, Rhinoceros, Camel, Giraffe, Elephant, Zebra, Gorilla, Kangaroo, Bear, and Human. This selection was revised and verified by two zoologists according to the aforementioned criteria. The scientific names and subspecies for each class can be found in Table 4 of Appendix C.
3.2 Sample Categories
Our dataset, depicted in Figure 1, is partitioned into three groups: Must-Act, Must-Abstain, and Uncertain. To avoid misclassifications due to background objects, all samples exclusively contain one object. The authors manually scrutinized all test samples to ensure this. In line with Requirement 3, these three groups are further subdivided into eight categories to account for as many real-world
<https://dictionary.cambridge.org/dictionary/english/car>
scenarios as possible. Each category comprises 100 samples, with the exception of Category 8 comprising  $200^2$ , totaling 900 samples. To establish the reliability of the dataset as a valid benchmark, Cronbach's alpha [9] was used, a metric that evaluates the reliability of tests. The dataset was deemed reliable, with a minimum of 100 samples per category. The complete calculation for Cronbach's alpha is detailed in Appendix D.1.
- MUST-ACT contains clearly identifiable photo-realistic samples belonging to only one of the 10 classes. We intentionally restricted our dataset to photo-realistic samples to avoid ambiguous boundaries between in-class and out-of-class, such as abstract paintings or sculptures (e.g., claiming that a box with four sticks at the bottom and a sinusoidal line on the side is an elephant). Individuals with no visual impairments and familiarity with the 10 mammals can consistently classify these images correctly.
- Category 1: Unaltered samples from the designated classes are included. This category serves as the most basic step required for visual perception alignment. We sourced images from ImageNet1K [60] and images.cv<sup>3</sup>.
- Category 2: Image classification models have been known to sometimes base decisions based on unrelated features, such as the background of an image [25, 57]. We aim to challenge the models by testing them with samples that feature incongruous backgrounds, i.e., images of animals in environments where they are not commonly seen. Well-aligned models should accurately classify objects regardless of the changes in the background. Samples were generated using Stable Diffusion [59]. Examples of text prompts used for generating samples are provided in the Appendix D.2.
- Category 3: Another case of images that humans can easily identify but models cannot are perturbed images used for adversarial attacks [20, 31]. Well-aligned models would not be influenced by noise or adversarial attacks intentionally designed to deceive them. Here we include Category 1 samples with adversarial perturbation to test such cases. We use Fast Gradient Sign Method (FGSM) [20] to inject adversarial perturbations. The gradients are produced by pre-trained image classifiers available in PyTorch<sup>4</sup>.
- MUST-ABSTAIN are images that qualified individuals always abstain from classifying.
- Category 4: This category includes images that do not belong to any one of VisAlign's 10 mammals. Examples might include other animal species (e.g., birds, cats, dogs), textures (e.g., bubbly, banded), or objects (e.g., truck, inline skate, guitar). This category tests the model's ability to abstain from classifying objects outside its defined scope. Well-aligned models should be able to disregard infinitely diverse objects outside the target classes. The space of Category 4 is inexhaustible; thus, the authors use their best efforts to include as diverse samples as possible to represent this space. Samples were collected from ImageNet1K [60], Describable Textures Dataset [7], and Caltech 10 [14].
- Category 5: While Category 2 tests whether models focus on relevant features of the class definition, it is also important to assess if a model evaluates the object as a whole, rather than focusing on specific portions of a sample. Thus, we included images of creatures that incorporate features from two different animals (e.g., a creature with the head and two limbs of an elephant but the body of a zebra). Recent advances in text-to-image models [55, 56, 63] enable us to rapidly and easily generate images of objects that do not naturally exist. We used Stable Diffusion [59] to create these images. Details of prompts are in Appendix D.2.
- Category 6: An image may contain an object that does not belong to the target class but has features closely resembling those of the target classes. Given the challenging nature of these near-miss cases, we include Category 6, featuring mammals that are biologically close to the 10 target mammals according to scientific taxonomy (e.g., donkeys are close to zebras). The primary purpose of Category 6 is to test the model abstention ability on seemingly similar yet different samples. This category can be considered a more challenging version of Category 4. We have set aside this category as these samples can check the model visual alignment on samples near the natural evolutionary boundary. Samples are collected from ImageNet21K [58].
As category 8 contains a diverse set of cropings and corruptions of varying intensities, we double the number of samples for more reliable evaluation.
3https://images.cy/
<sup>4</sup>https://pytorch.org/
- Category 7: This category includes images in styles other than photo-realistic (e.g., a drawing of an elephant, a sculpture of a giraffe). Considering that MUST-ACT samples are photo-realistic images confirmed by humans, well-aligned models should be able to discern styles that deviate from photo-realism. The images were collected from DomainNet [50] and ImageNet-R [24].
- UNCERTAIN includes images that are cropped or corrupted in various styles in different intensities
- Category 8: This category includes images that are either cropped at varying sizes and regions or corrupted using one of the 15 corruption types<sup>5</sup>. The original samples were collected from ImageNet21K [58]. Well-aligned models should be able to correctly classify slightly corrupted images while abstaining from making decisions on indistinguishably corrupted images. The corruption process follows the approach outlined in ImageNet-C [22], with corruption intensities varying from 1 to 10.
3.3 Uncertain Group Label Generation
One challenging aspect of the Uncertain group is the variability of these samples' gold standard labels, which fluctuates depending on corruption types and intensities. For instance, it would be optimal to correctly classify images with slight corruptions when identifiable. However, given a severely darkened image, the object might resemble a tiger, a jaguar, or be entirely unrecognizable. In such scenarios, determining whether a human observer would classify it as a tiger or abstain from decision-making becomes challenging. Therefore, we derive a gold human ratio (i.e., the distribution over classes provided by human annotators), rather than assigning one label per image as in Must-Act and Must-Abstain, because human perception of an image can vary, and approximating the ratio for each image offers the best test of alignment<sup>6</sup>. To derive the gold ratio across the 11 classes (10 mammals + abstention), we employ MTurk workers to classify images in the Uncertain group.
Every MTurk worker is asked to classify 35 images, including Category 4 images corrupted with a severity between 1 to 10, with 10 being distractors. This is to minimize MTurk workers' potential biases; e.g., a severely dark image can be perceived as anything other than the 10 mammals. After reviewing the task description and image samples for each class, MTurk workers select either one of the 10 mammals or an option labeled "None of the 10 mammals, uncertain, or recognizable", which is equivalent to abstention. To ensure the quality of samples, we disregard MTurk results where anything other than abstention was chosen for the distractor images.