SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Muxi Chen\*, Yu Li\*, Qiang Xu
† The Chinese University of Hong Kong
† Harbin Institute of Technology, Shenzhen
{mxchen21, qxu}@cse.cuhk.edu.hk; li.yu@hit.edu.cn
Machine learning models can frequently produce systematic errors on critical subsets (or slices) of data that share common attributes. Discovering and explaining such model bugs is crucial for reliable model deployment. However, existing bug discovery and interpretation methods usually involve heavy human intervention and annotation, which can be cumbersome and have low bug coverage.
In this paper, we propose HiBug, an automated framework for interpretable model debugging. Our approach utilizes large pre-trained models, such as chatGPT, to suggest human-understandable attributes that are related to the targeted computer vision tasks. By leveraging pre-trained vision-language models, we can efficiently identify common visual attributes of underperforming data slices using human-understandable terms. This enables us to uncover rare cases in the training data, identify spurious correlations in the model, and use the interpretable debug results to select or generate new training data for model improvement. Experimental results demonstrate the efficacy of the HiBug framework. Code is available at: https://github.com/cure-lab/HiBug.
Machine learning models for computer vision tasks have made significant advances in recent years. However, they are still prone to systematic failures on critical subsets, or "slices," of data with similar visual attributes. These bugs are largely due to the dataset used for training, such as rare data in certain categories or biased training data leading to spurious correlations, causing potentially catastrophic consequences for models used in safety-critical applications, such as autonomous driving [Fujiyoshi et al., 2019] and medical image processing [Giger, 2018].
Various model debugging techniques are proposed in the literature. Some focus on discovering failure instances. For example, test input prioritization techniques aim to identify as many failure instances as possible under a given labeling budget [Li et al., 2021, Wang et al., 2021]. Other solutions seek to identify the visual attributes that cause model failures. For example, [Singla et al., 2021] proposes a visualization technique (e.g., saliency maps), through which humans can identify bug-related visual attributes. The recent Domino [Eyuboglu et al., 2022] and Spotlight [d'Eon et al., 2022] techniques try to cluster failures in the embedding space for common bug attribute identification. While interesting, the visual attributes in the embedding space are often entangled, making it hard to identify the true bug-related attributes. Adavision [Gao et al., 2022] proposes to describe model bugs with natural languages and refine the description interactively. Such a method requires heavy human intervention and cannot guarantee bug coverage.
In this paper, we propose HiBug, an automated framework for interpretable model debugging. HiBug leverages the recent large pre-trained models to identify common visual attributes of underperforming
Figure 1: Visualization of the CLIP's embedding space. As we can observe, the visually similar images are distributed across the embedding space (yellow dots). This is because the embedding space is entangled and hence cannot accurately reflect visual similarity.
Gender: Female
Age: Young
Hair Color: Blonde
Skin: White
Emotion: Serious
Beard: No
Makeup: No
data slices. Specifically, given the targeted computer vision task, we first employ a pre-trained large language model such as chatGPT to collect a corpus of visual attributes that are related to this task. Next, we attach these attributes to all the images, leveraging pre-trained vision-language models such as BLIP. Afterwards, we perform data slicing based on these attributes and conduct statistical analysis to identify those underperforming slices with common visual attributes. Such a bug identification procedure is naturally human-interpretable. We further conduct failure reasoning and use the debug results to select or generate training data for model repair.
To sum up, the main contributions of this paper include:
- HiBug is a highly automated and interpretable model debugging framework. It achieves this by introducing a new debugging flow to identify failure instances and describe the visual attributes causing failures using natural language.
- We propose to leverage the capability of the powerful pre-trained models for interpretable model debugging, which efficiently identifies common visual attributes of underperforming data slices using human-understandable terms.
- Based on the discovered failure attributes, we further conduct failure reasoning, i.e., classifying if the failure is caused by spurious correlation or lack of training data. Moreover, we show that our framework can be directly used to guide the selection and generation of more images with failure attributes, which facilitate model repair.
Our experiments on dcbench [Eyuboglu et al., 2022] show that HiBug can identify up to  $85\%$  correlation errors and  $81\%$  rare cases. We also use three different tasks to show that HiBug benefits data selection and data generation for model improvement. The rest of this paper is organized as follows. Section 2 presents the related work. Section 3 illustrates our method and Section 4 shows the effectiveness of our method. Section 5 concludes this paper.
Test input prioritization. Test input prioritization techniques are representative model bug discovery methods [Feng et al., 2020, Kim et al., 2019, Li et al., 2021]. Specifically, they aim to identify "high-quality" test instances from a large amount of unlabeled data, which can reveal more model failures with less labeling effort. DeepGini, as proposed by [Feng et al., 2020], rank unlabeled test instances by calculating confidence scores based on the model's output probabilities. Test cases that exhibit nearly equal probabilities across all output classes are considered less confident and are thus more likely to uncover model failures. Follow-up methods include considering the diversity of the selected instances or proposing more accurate metrics to evaluate the failure probability for one test instance. Despite the effectiveness in reducing the debug cost, these methods do not provide further explanation on why this failure happens.
Interpretable bug discovery. Instead of discovering more failures, interpretable bug discovery methods aim to find human-understandable visual attributes that cause the failures.
To this end, [Singla et al., 2021] present a sample-specific visualization technique based on the embedding space of the deep learning model. Specifically, for each failure sample, this method can
Figure 2: The overview of HiBug.
visualize the features that the model sees when making the decision, and thus provide some extent of interpretability. Spotlight [d'Eon et al., 2022] proposes to find the high failure rate area in the model's embedding space. The failures in that area exhibit some visual similarity. AdaVision [Gao et al., 2022] proposes a human-in-the-loop testing procedure. AdaVision first generates a set of potential captions with the help of the user and language model. Then, it matches data to the captions and iteratively selects captions with a high error rate in data. Next, it asks the user to add more details to refine the captions and continue the iteration. However, these methods involve a large amount of human effort. The work proposed by [Singla et al., 2021] and [d'Eon et al., 2022] requires humans to summarize the visual features that cause the failures, and AdaVision requires humans to refine the caption. Such heavy human intervention procedures are cumbersome and cannot guarantee high bug coverage.
Another line of work proposes to automatically summarize failure-related visual attributes using natural language. They cluster failure by mixture models [Eyuboglu et al., 2022] or a linear classifier [Radford et al., 2021]. They then generate a set of candidate captions by combining words and templates from a large, predefined corpus. Finally, they use CLIP [Radford et al., 2021] to match each data group with a caption.
However, the CLIP's embedding space often contains entangled visual attributes. As shown in Figure 1, this can result in semantically coherent images being far away in the embedding space. Furthermore, the candidate captions generated from the corpus might contain irrelevant or contradicting words, such as "a photo of setup by banana" and "a photo of skiing at sandal" [Gao et al., 2022]. Meanwhile, the corpus of candidate words is manually designed and thus requires intensive human effort. Finally, the generated captions only describe the property of failure data, failing to imply why the model goes wrong, i.e., because of rare training cases or suspicious dependence.
HiBug focuses on interpretable bug discovery and addresses the above challenges in several ways. First, unlike existing efforts that seek to cluster failures, HiBug follows a different debugging flow. We first group the data according to their attributes and then judge if there are underperforming ones. This way, underperforming slices must share a similar concept, leading to more accurate explanations. Second, HiBug eliminates the need for a manual-intensive external corpus by leveraging state-of-the-art large language models to propose task-specific attributes based on user-provided task descriptions. Lastly, HiBug supports the root cause discovery of the failures.
The overview of HiBug is presented in Figure 2. HiBug aims to systematically identify and interpret consistent bugs in trained models. As an illustrative example, we consider a model that classifies instances of lipstick usage. HiBug initiates this process by employing large, pre-existing language models, such as ChatGPT, to propose visual attributes pertinent to the task under investigation. Subsequently, for each image in the dataset, corresponding attribute values are assigned by leveraging vision-language models. We consider this step as the attribute proposal and assignment step. Upon
establishing these attributes for each image, HiBug can detect low-performing data slices, and provide natural language descriptions and root causes for the identified bugs. We consider this step as the bug discovery and interpretation step.
In the context of the lipstick-use classification model, HiBug might uncover a correlation between the model's predictions and gender, i.e., predict 'lipstick usage' for images depicting females. A retrospective examination of the issue reveals that the training dataset exhibits an inherent bias, with females frequently appearing with lipsticks. This results in the model developing spurious correlations, which in turn lead to erroneous predictions.
Lastly, HiBug demonstrates its efficacy by selectively generating or selecting novel data for retraining the model, thereby enhancing its overall performance. This approach facilitates the rectification of identified spurious correlations and biases that emerged during the bug discovery phase.
Please note that some components in HiBug, such as ChatGPT and BLIP, are selected mainly because they are easy to use and provide good solutions. HiBug is highly flexible and modular. Therefore, designers can update any component in HiBug with a better counterpart, if needed.
3.1 Attribute proposal and assignment
Images are distinguished by unique visual features. For example, an image might be described as 'A young woman with brown hair'. In this context, 'young', 'woman', and 'brown hair' constitute elements of visual semantics. However, these features aren't inherently known for each image, complicating the task of identifying interpretable bugs.
To articulate model bugs in a form understandable to humans, such as natural language, it is essential to establish a set of task-related visual attributes, denoted as  $A$ . These attributes serve to characterize images within a specific classification task. Using the previous example, task attributes,  $A$ , might include 'gender', 'age', and 'hair color'. Each attribute,  $a \in A$ , can assume several potential values, represented as  $V(a)$ . For instance, the attribute 'gender' could include values such as 'male', 'female', and 'non-binary'.
Hence, with a pre-defined set of attributes,  $A$ , and their corresponding values,  $V(a)$ , characterizing an image becomes a more structured process. However, determining these visual attributes and corresponding values is challenging. This difficulty stems from the absence of inherent knowledge about each image's visual features, which, if known in advance, would simplify the identification of interpretable bugs.
Attribute name proposal. Recent research indicates that large language models, like ChatGPT, possess extensive general knowledge and are highly effective at answering questions [Wei et al., 2022, Touvron et al., 2023]. We aim to utilize these capabilities to propose task-related visual attributes.
In this process, users are required to provide ChatGPT with a simple task description. For instance, in the lipstick-usage problem where the task is to identify lipstick on a human face, we pose the following question to ChatGPT: 'Can you list some attributes to describe human appearance?' ChatGPT then returns a list of potential attributes. An example of such a question-and-answer interaction with ChatGPT is illustrated in Figure 2. Upon receiving the responses, we compile each attribute  $a \in A$  to form an attribute list  $A$ . It should be noted that the question should be task-specific, necessitating user input for its design. However, barring this step, the entire process is fully automated.
Attribute value proposal. Upon generating the attribute list  $A$  for each task, we face the challenge of identifying corresponding attribute values. Although ChatGPT can offer some suggestions, its limited access to dataset images may lead to mismatches with the actual dataset distribution. To address this, we utilize a visual question-answering (VQA) model, which augments attribute value accuracy by analyzing dataset images and providing corresponding attribute value responses.
A straightforward approach to accumulating attribute values using the VQA model would involve posing a query for each image in the form of:
$$
^ {\prime \prime} W h a t i s t h e ^ {*} 1 o f t h e ^ {*} 2 i n t h e i m a g e? ^ {\prime \prime} \tag {1}
$$
In this equation, *1 and *2 represent attributes and task-related elements collected previously. The VQA model's response serves as the attribute value for the image. However, this approach proved time-consuming due to the volume of images.
Therefore, we propose a more efficient approach: selecting representative images from the dataset for attribute value proposal. The rationale is that not every image needs to be queried for their attribute value types, as similar images tend to share similar attribute value types. For example, attribute values like female', male', and 'non-binary' can be gathered by querying just a few images. Querying additional images would likely not yield more diverse choices, but would merely consume more time. Hence, we can use representative images for attribute value proposals to reduce redundancy.
To select these representative images, we employ the pre-trained vision-language model, BLIP [Li et al., 2022], for feature extraction. The representative samples are selected by clustering the features. Then we query the VQA model (BLIP) with the representative samples for attribute value proposals. Finally, we aggregate all attribute values to form a value choice list  $A(a)$  for each attribute name  $a \in A$ .
Attribute assignment. Our process begins with a set of attribute names, denoted as  $A$ , with each attribute name  $a \in A$  having corresponding attribute values represented as  $A(a)$ . The goal is to assign an appropriate attribute value from  $A(a)$  for each attribute  $a \in A$  to every image  $x$ .
To this end, we consider value from  $A(a)$  as labels and apply zero-shot classification on images with the BLIP model. The end result is the formation of an attribute matrix  $X_{A}$ , an illustrative example of which can be found in Figure 2.
3.2 Interpretable bug discovery and classification
Interpretable bug discovery. After the attribute matrix  $X_{A}$  is obtained, we conduct bug discovery based on the matrix. The easiest way is to analyze the attributes separately (e.g., analysis if the 'young' attribute value leads to under-performing slices). Assuming the number of attributes is  $m = |A|$ , and the number of choices for each attribute is  $n_i$ ,  $i \in \{1..m\}$ , then the total number of attribute choices is  $N = \sum n_i$ ,  $i \in \{1..m\}$ . For each attribute choice, we group images with similar attributes and test their accuracy. In total, we can have  $N$  groups and  $N$  test accuracies.
To take a step further, we can analyze the slices defined by combinations of the attributes (e.g., analysis if the 'young woman with brown hair' attribute value combinations leads to under-performing slices). Specifically, we group data with the same attribute value combinations into one data slice. The number of data slices is  $\min(\prod n_i, i \in \{1..m\}, D)$ , where  $D$  is the number of data in the dataset.
The bug slices are slices with an accuracy lower than a given threshold, i.e., the model's overall accuracy. For each under-performing group, i.e., bug slices, the attribute choice can be used as the bug description (e.g., young people tend to have low accuracy).
Bug classification. To facilitate bug understanding, we categorize the bugs into two groups according to their root cause, namely rare case and spurious correlations. After classification, HiBug will give feedback to the user on the discovered bugs.
- Rare case. This type of bug often originates from insufficient training data [Eyuboglu et al., 2022]. To determine if a description is a rare case, HiBug first selects data from the validation set that matches the given description. Next, if these data have a high error rate and low frequency in the training set, we consider the bug a rare case bug.
- Spurious correlation. The training set we collect for model training can be biased. Therefore, the model might learn a dependence on some features that are not casual to the target. For example, in our lipstick-wearing classification task, the model learns to predict "not wearing lipstick" when the gender of people in the image is male. HiBug finds this correlation by calculating the conditional probability between the model's predictions and data description via the validation set. The correlation threshold  $\alpha$  is a hyper-parameter.
3.3 Model repair
To improve the model, we propose a data selection approach and a data generation approach. For data selection, we first select data slices with the top error rate in the validation set. Then, we distribute the selection budget to the data slices based on the number of failures inside the slices. Finally, we select data matching the attribute values of the data slices from the unlabeled pool. The detailed algorithm is in the appendix. For data generation, we combine the class name with the attribute values of top error
Table 1: The percentage of problems that HiBug correctly identify the feature related to model's prediction and describe the correlation behavior of the model.
Threshold	Number of Problems	Find Attr.	Find Des.
0.7	457	84.9%	72.6%
0.8	352	77.8%	64.2%
rate data slices to create prompts. We then use these prompts as input for the generative model to generate new data. It is worth noting that the selected or generated data can also be a countermeasure for discovered bugs since attribute values related to bugs often appear in high error rate data slices.
In this section, we validate our framework by (1) Can HiBug discover bugs? (2) What is the quality of discovered bugs? and (3) Can we improve the model with the discovered bugs? And How? We also present an ablation study to evaluate key factors of our method.
4.1 Can HiBug discover bugs?
Experiment setup. As discussed in section3.2, the model's failure on a data slice can be attributed to bugs such as insufficient training samples on rare cases or erroneous dependence on some feature. HiBug is the first framework that supports interpretable bug discovery. In this part, we evaluate our methods on the correlation discovery and rare case discovery tasks of the dcbench dataset proposed by domino [Eyuboglu et al., 2022].
The correlation discovery task in dcbench consists of 880 problems. Each problem contains common materials used in image classification, such as a model checkpoint, the model's predictions on a validation set, and labels for the validation set. Notably, each problem also includes a description of the erroneous correlation, indicating the name of the feature that the model's prediction is correlated with. For example, the prediction of a human-related classification model can be correlated with gender. The rare case discovery task in dcbench contains 118 problems. Apart from the common materials, each problem also has the name of a rare case.
Correlation discovery. The linear correlation coefficients between features and a model's predictions can differ in correlation discovery problems. Therefore, we evaluate HiBug using different correlation detection thresholds: 0.7 and 0.8. We only test on problems that have a correlation coefficient above the detection threshold  $\alpha$ . In our experiment, we evaluate whether HiBug can correctly identify the attribute value (feature) related to the model's prediction and describe the correlation behavior of the model. This is a challenging task, as HiBug not only needs to cover the feature in the attribute proposal and assignment process but also identify the correlation.
In Table1, we present two results of our methods. "Find Des." counts the cases where our method finds the exact name of a feature or a WordNet synonym [Miller, 1998]. "Find Attr." counts the cases where our method finds the correct feature or features in the same attribute. For example, when the model's prediction is correlated with "male", our method finds that the model's prediction is correlated with "female". "male" and "female" are both within the scope of the attribute "gender". It is worth noting that when the threshold is 0.7, HiBug accurately discovers a  $72.6\%$  correlation. This indicates that HiBug can effectively identify the model's erroneous correlation with the feature.
Previous methods have not explored interpretable bug discovery, so they cannot be used for direct comparison. Domino [Eyuboglu et al., 2022] is the state-of-the-art slice discovery method. As introduced in Section 2, it clusters data into slices and provides a description for each slice. Domino evaluates its clustering and describes abilities separately. In the correlation discovery task, Domino recovers ground truth bug slices with a mean precision-at-10 of  $50\%$ . It provides the correct description for a data slice  $41\%$  of the time. For bug identification, the probability that Domino provides correct feedback to the user can be the product of the above two probabilities. It is worth noting that Domino only produces a general description of the data in a bug slice, rather than identifying the bug itself.
Table 2: Basic information of three experiment settings. Split denotes the number of data in train set: validation set: test set: unlabeled pool.
Dataset	Lipstick	RPC	ImageNet10
Number of classes	2	200	10
Split	80k:10k:10k:80k	54k:25k:74k:49k	10k:28k:2k:10k
All Validation Acc. < Model Acc.
(a) Lipstick
Figure 3: Number of bug slices uncovered by HiBug.
Validation Acc. < Model Acc. - 0.1
(b) RPC
Validation Acc. < Model Acc. -0.2
(c) ImageNet10