SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Existing research has examined various methods to establish human trust in and replicate the predictions of machine learning models. One such method is LIME, a black-box feature importance technique, which was employed to select examples for evaluation by crowdworkers to determine the superior model among two options [67, 45]. However, their selection strategy disregards the human predictor, and their approach merely presents the examples without further action. In the context of visual question answering, Chandrasekaran et al. [18] manually selected seven examples to educate crowdworkers about the AI's capabilities, leading to an enhanced ability to identify instances where the AI failed. Feng et al. [25], in the domain of Quizbowl question answering, emphasize the significance of incorporating the human expert's skill level when designing explanations. This further justifies our decision to involve the human predictor in the selection of teaching examples. Cai et al. [17] conducted a study involving 21 pathologists to gather guidelines on what clinicians desired to know about an AI system prior to interacting with it. Yin et al. [84] investigated the impact of initial debriefing on stated AI accuracy versus observed AI accuracy during deployment, finding a substantial influence of stated accuracy on trust that diminishes quickly once the model is observed in practical use. This reinforces our approach of building trust through examples that
simulate real-world deployment. Bansal et al. [5] examined the role of the human's mental model of the AI in task accuracy; however, the mental model was developed through interaction during testing rather than during an initial onboarding stage. The most similar work to ours is that of [57] which presents an onboarding scheme based on selecting a set of examples and allows the human to describe the regions where the AI performance is good or bad. Through a user study on passage-based question answering, they show that their onboarding scheme improves performance by  $5\%$ , however, they evaluate without the presence of AI evaluations, with a synthetic AI model and their scheme involves more involvement from the human as they have to describe the regions themselves. Another approach to teaching involves providing humans with guidelines on when to rely on AI systems [3]. Model cards [55] and industry practices such as the IBM AI fact sheet [4] demonstrate direct methods of presenting these guidelines to users, we present humans with a similar form of card but that includes aspects of human performance "human-AI card".
There is a growing and large area of literature on discovering (and auditing) regions of AI error, the following is not meant as an extensive list of related work but captures some of the essence of the literature:
- Adatest allows a user to iteratively discover regions of AI error using LLMs for NLP tasks and then re-train the model on the regions of error [66], it was then extended to vision tasks [27] with a similar procedure in [80].
- Erudite allows users to discover regions of error of NLP models through user interfaces [82], there is a wider literature on dashboards for discovering regions of error [2]
- [37] learns an SVM model from image embeddings to predict model error and then uncover regions of error based on the directions of the SVM model.
- Works have done extensive manual annotation of ImageNet model mistakes[35, 76].
- DOMINO discovers regions of model error using a slice discovery model based on a specialized gaussian mixture model [24], extensions include DRML [87].
- The Spotlight method learns individual regions based on a neighborhood of a learned point [23], our region finding algorithms generalize their procedure by learning weighted distance distance measures and with a different aim of improving gain of the prior.
- SEAL: Interactive Tool for Systematic Error Analysis and Labeling, uses k-means to uncover regions of error and then uses an LLM to describe each region [63].
Recent work has emerged on describing sets of images [34, 53] but they don't incorporate a contrastive method as we propose. Helpful tools for describing differences between text and images can be useful for describing regions which future work can incorporate [89, 79, 78].
One of the objectives of explainable machine learning is to enhance humans' ability to assess the accuracy of AI predictions by offering supporting evidence [44, 43, 70, 32, 88, 42, 72, 73, 81, 31]. Nevertheless, these explanations fail to provide guidance to decision makers on how to balance their own predictions against those of the AI or how to integrate the AI's evidence into their final decision [39]. [75] shows that AI explanations can reduce overreliance and improve human-AI team performance, however, their experiments are with simulated AI models and explanations. The central question of our work is not to study the utility of AI explanations, in fact, all our user studies incorporate AI explanations and we aim to improve human-AI performance in their presence.
B Region Finding Algorithm - Details
Regions Requirements. Each region in our algorithm should aim to satisfy the following constraints:
1. Region Size: We want the size of the region to be at least of size  $\beta_{l}$  and at most of size  $\beta_{u}$ .
2. Consistency of takeaway: The examples in each region must agree on what the takeaway is in terms of the integration decision. Specifically, at least  $\alpha \%$  of all points in the region must either be: ignore AI, use AI as is or integrate AI advice.
3. Concise and Distinguishable Theme: Each region must be concisely described in natural language in such a way to differentiate from the overall domain. If a region cannot
be described in natural language, the human may not be able to derive a generalizable recommendation from it. This is a constraint that we implicitly try to satisfy by learning neighborhoods in a natural language embedding space.
4. Minimum Gain: Each region must have a minimum information gain (defined below) of  $\delta$ . This is to ensure that all regions contain sufficient novel information to the human.
The optimization to find each region can be formulated in its non-relaxed form as:
$$
\max  _ {c, \gamma , w, r} \sum_ {i = 1} ^ {n} \mathbb {I} _ {| | w ((e _ {i}, a _ {i}) - c) | | <   \gamma} \cdot \boldsymbol {g} _ {i, r}
$$
$$
\text {s . t .} \quad \sum_ {i = 1} ^ {n} \mathbb {I} _ {| | w ((e _ {i}, a _ {i}) - c) | | <   \gamma} \cdot \mathbb {I} _ {r _ {i} = r} \geq \alpha n
$$
$$
\text {s . t .} \quad n \beta_ {l} \leq \sum_ {i = 1} ^ {n} \mathbb {I} _ {| | w ((e _ {i}, a _ {i}) - c) | | <   \gamma} \leq n \beta_ {u}
$$
And the relaxation we propose is (refer back to Section 4.1):
$$
\max  _ {c, \gamma , w, r} \sum_ {i = 1} ^ {n} \sigma \left(C _ {1} (- \| w \circ ((e _ {i}, a _ {i}) - c) \| + \gamma)\right) \cdot g _ {i, r} - \lambda \max  (\sum_ {i = 1} ^ {n} \sigma \left(C _ {1} (- \| w \circ ((e _ {i}, a _ {i}) - c) \| + \gamma)\right)
$$
$$
\lambda \max  \left(- \sum_ {i = 1} ^ {n} \sigma \left(C _ {1} (- | | w \circ ((e _ {i}, a _ {i}) - c) | | + \gamma)\right) + \beta_ {l} n, 0\right)
$$
We run the optimization for  $r = 0$  and  $r = 1$  and choose the  $r$  with the better objective value. With  $r$  fixed, we optimize with respect to the remaining continuous parameters using AdamW and reduce the learning rate when loss has stopped improving. We initialize with  $\gamma = 0$  and  $w = 1$ . For the centroid  $c$ , we first run  $k$ -medoids clustering on the input data with  $k = \min(\max(100, T), n)$  and randomly select 20 of the resulting centroids. Then with  $c$  initialized as each one of the 20 centroids in turn, we run the optimization for 200 epochs and record the loss, and finally optimize for 2000 epochs with the best initialization for  $c$ . We do these repeated initializations to avoid local minima which are a common failure mode of this type of optimization problem. Note this process is to find one region, to find all regions we repeat this process identically to find regions one by one.
Selection Based Approach. We described in the body of the paper a generative algorithm to find the regions. We now describe a selection based algorithm that finds centroids from points in the dataset  $\widetilde{D} = \{e_i, r_i\}$ . We will also restrict the radius to be the distance between the centroid to another data point in  $\widetilde{D}$ . We proceed with a sequential search, at round  $i$  we perform the following search:
$$
c _ {i}, \gamma_ {i} = \arg \max  _ {i \in \widetilde {D}, \gamma} G \left(N _ {i, \gamma}, \hat {R} ^ {*}, R _ {N _ {1: i - 1}}\right), \tag {5}
$$
$$
s. t. \exists k \in [ n ] s. t. \gamma = d \left(e _ {i}, e _ {k}\right), \tag {6}
$$
$$
\text {a n d} \frac {\sum_ {j \in [ n ] , d \left(e _ {i} , e _ {j}\right) <   \gamma} \mathbb {I} _ {r _ {j} = r _ {i}}}{| \{j \in [ n ], d \left(e _ {i} , e _ {j}\right) <   \gamma \} |} \geq \alpha \tag {7}
$$
$$
\text {a n d} \beta_ {l} \leq | \{j \in [ n ], d \left(e _ {i}, e _ {j}\right) <   \gamma \} | \leq \beta_ {u} \tag {8}
$$
$$
\text {a n d} G \left(N _ {i, \gamma}, \hat {R} ^ {*}, R _ {N _ {1: i - 1}}\right) > \delta \tag {9}
$$
Note that with the selection-based procedure, we have to define a fixed distance measure  $d$ , and we cannot optimize over the deferral decision  $r$  of the region as we inherit from the point  $i$  found. The naive algorithm to solve the above search is as follows. We first compute the distance matrix between all data points in  $\widetilde{D}$ , call this matrix  $K$ . We then at each round do the following for each point  $i$  in  $D\widetilde{D}$ : sort the points by their distance to  $i$ , iteratively grow the region around the point  $i$  to satisfy the
constraints, and then keep track of the maximum gain radius. Each time we grow the region by one point, we check if the constraints are satisfied. The search is parallelized across multiple instances to make it faster. Finally, we compare the gain of all feasible points and pick the highest one. This algorithm extends the approach of [57] to incorporate additional constraints and has significant speed ups over their approach.
Aggregating Regions Across Different Embedding Spaces. We can run our region finding algorithm above and find different regions across multiple embedding spaces. The question is how do we aggregate these regions. Say we found a set of  $T$  regions  $N_{1},\dots ,N_{T}$ . We then run the following meta-algorithm: do a greedy sequential search to add regions one at a time while making sure the minimum gain requirement is satisfied, stop when it is not.
C Region Description Algorithm - Details
The LLM call inside Algorithm 1 is accomplished by building the prompt with  $S^{+}$  being the inside set (positives) and  $S^{-}$  being the outside set (negatives) as follows:
def get_prompt positives, negatives):
    prompt = pre Instruction + "\n"
    prompt += "inside the region: \n"
    counter = 1
    for p in positives:
        prompt += p[0] + ", \n"
        counter += 1
    if len(negatives) > 0:
        prompt += ".\n not in the region: \n"
        counter = 1
        for p in negatives:
            prompt += p[0] + ",\n"
            counter += 1
    prompt += post Instruction
    return prompt
For the experiments in Section 6 we use the following pre instruction:
I will provide you with a set of descriptions of points that belong to a region and a set of descriptions of point that do not belong to the region. Your task is to summarize the points inside the region in a concise and precise short sentence while making sure the summary contrasts to points outside the region. Your one sentence summary should be able to allow a person to distinguish between points inside and outside the region while describing the region well. The summary should not be a single word, it should be accurate, concise, distinguishing, and precise.
Inside the region:
- two cows and two sheep grazing in a pasture.
- the sheep is standing near a tree.
Not in the region:
- the cows are lying on the grass beside the water.
summary: sheep.
End of Example
While the post-instruction is simply "summary:"
For the ablation without contrasting, the pre instruction we use is:
I will provide you with a set of descriptions of points that belong to a region. Your task is to summarize the points inside the region in a concise and precise short sentence. Your one sentence summary should be able to allow a person to distinguish points inside the region while describing the region well. The summary should be a single word, it should be accurate, concise, distinguishing and precise.
inside the region: - two cows and two sheep grazing in a pasture.
-the sheep is standing near a tree.
For the user studies, we had used an earlier instruction that works slightly worse:
"summarize the points inside the region in a concise and precise short sentence while making sure the summary contrasts to points outside the region"
We recommend the use of the following pre instruction:
I will provide you with a set of descriptions of points that belong to a region and a set of descriptions of points that do not belong to the region. Your task is to summarize the points inside the region in a concise and precise short sentence while making sure the summary contrasts to points outside the region. Your one sentence summary should be able to allow a person to distinguish between points inside and outside the region while describing the region well. The summary should be no more than 20 words, it should be accurate, concise, distinguishing and precise.
inside the region:
outside the region:
summary: The region consists of descriptions that have sheep in them outside in nature, it could have cows but must have sheep.
D Onboarding and Recommendations to Promote Rules - Details
Introductory Phase. To facilitate a smooth onboarding process for individuals working with an AI assistant, we introduce the Human-AI Card. This card provides detailed insights into the AI's capabilities, training, and performance. Below is a presentation of the core details of the card inspired by the work in [17]:
Table 6: Human-AI-Card presented to the human as part of onboarding
Information	Description
AI Input	What the AI uses to make its prediction
AI Output	What the AI provides as output (predictions, explanations, ...)
Source of Training Data for AI	Description of data used to train the AI
Source of Pre-Training Data of AI	Description of pre-training data of model AI is based on what the AI is trying to achieve (minimize classification error, detect objects, next word prediction)
Training Objective of AI
Average AI Performance
Average Human Performance
Additionally, we provide a breakdown of AI and Human performance on different subgroups of data.
We take an example of the Berkeley Deep Driving dataset, where a subgroup might comprise images taken during the night, in rainy weather, or on a highway. We compute the model's error for each possible subgroup and then perform a paired t-test comparing the subgroup model error to the average model error over the entire data. For the purpose of our user studies, we highlight subgroups
defined by a single metadata category that show statistically significant differences  $(p \leq 0.05)$ . It's important to note that, for rigorous analysis, one should apply corrections for multiple hypothesis testing. However, considering the vast number of metadata categories, many results might become insignificant. Therefore, for simplicity, we adopt this heuristic approach.
E Method Evaluation - Details
In Table 7 we report the details on the datasets we use in our method evaluation. We normalize all datasets using  $l_{\infty}$  normalization and run our algorithms for 2000 epochs with a learning rate of 0.001 using AdamW. We use a constant  $C_1 = 20$ ,  $\delta = 1$ ,  $\beta_u = 0.1$  ( $\beta_u = 0.5$  for Aim 1),  $\alpha = 0.5$  and  $\beta_l = 0.01$  for all experiments. We use a random prior  $R_0$  that has equal probability for 1 and 0 uniformly at random. All experiments are run on a GeForce GTX 1080 Ti.
Table 7: Datasets for "Learning Accurate Integrators (Aim 1)". We note the total number of samples  $n$ , the target set size  $|\mathcal{V}|$ , the human expert finally the model of the AI. When we note human "XX% accurate", this indicates a synthetic human model that is accurate uniformly at random with probability XX%. For DynaSent, the AI model is a pre-trained sentiment analysis roBERTa-base model [8] on Twitter data and achieves 75% accuracy. For both BDD and MS-COCO we blur the images using a Gaussian Blur with scale 21 and variance 5.
Dataset	n	|Y|	Human	AI
Berkeley Drive [83, 85]	Deep (BDD)	10k	2	80% accurate	faster rcnn r50 fpn 1x4 - Gaussian blur with scale 21 and variance 5
MS-COCO [48]	5k	2 (presence of person in image)	70% accurate	faster rcnn R 50 FPN
Massive Multi-task Language Understanding (MMLU) [33]	14k	4 (MCQ)	50% accurate	flan-t5x1 [20]
Dynamic Sentiment Analysis (DynaSent) [61]	6.5k	3	leave-one-out annotator	a pre-trained sentiment analysis roBERTa-base model [8]
Figure 4: Test Error  $(\downarrow)$  of the human-AI system when following the decisions of the different integrators baselines as we vary the number of regions maximally allowed for each integrator on the MS-COCO dataset.
Figure 5: Test Error  $(\downarrow)$  of the human-AI system when following the decisions of the different integrators baselines as we vary the number of regions maximally allowed for each integrator on the MMLU dataset.
Figure 6: Test Error  $(\downarrow)$  of the human-AI system when following the decisions of the different integrators baselines as we vary the number of regions maximally allowed for each integrator on the Dyanaset dataset.
For Aim 2, we create synthetic AI and human models as follows:
- BDD: AI and Human model each have non-overlapping two to four regions defined as the intersection of two metadata categories, each region is either a good region where the AI/Human have  $95\%$  accuracy or bad region with  $60\%$  accuracy (equal number of good and bad regions). If an example does not belong to any region the AI/Human have  $75\%$  accuracy. Each region is of size at least 0.01 and at most 0.2 in terms of fraction of data points in the dataset. An example region is "AI is good at: weather: clear and scene: highway" or "Human is bad at: weather: snowy and timeofday: night".
- MMLU: same setup as BDD, but there is only one metadata category in the dataset, so each region belongs to a certain metadata category. An example region is "AI is good at: subject: professional psychology".
- MS-COCO: same setup as BDD. An example region is ""Human is bad at: cow: present".
For Aim 3, an obstacle to quantitative results for region descriptions is that they are often complex even when regions are synthetically defined in terms of metadata and captioning metrics are not informative. For a region defined on BDD as images of "scenes of highways during the night with no cars", our algorithm finds the description "highway during the night with various weather conditions and not congested with many cars" while the SEAL [63] describes it as "Highway Nighttime Weather Conditions". The SEAL description surprisingly has higher captioning metric scores (BLEU, METEOR) [50] while being less informative.
For the MS-COCO evaluation, we only select objects that have at least 50 examples present in the evaluation set which leads to only 73 objects over which we evaluate the different region description algorithms.
F User Studies - Details
F.1 BDD Study
Task. The images from BDD are blurred using a Gaussian Blur with a scale of 21 and a variance of 5. The AI model is a trained faster rcnn model that achieves  $84\%$  accuracy without blurring which decreases to  $78\%$  accuracy on the test set. We use the bounding boxes from the model and output them on the image (allowing the user to either hide or show them). To get a confidence score, we take the maximum score for the prediction of a traffic light in the image.
Initial Data Collection. The BDD dataset was split 70-30 where the  $70\%$  split was used to get the initial human predictions and find the regions and the  $30\%$  split was used only to get testing examples for the final user study. As mentioned, we obtained data on 400 examples with both human predictions and prior AI-integration decisions. We use these examples to build Random Forrest models that predict both the human predictions and AI-integration decisions from the embeddings, labels, and AI predictions (AI predictions only for predicting AI integration decisions). We use these predictions from the RF models to label the entire dataset. We ensure that the predictions of the RF models are calibrated (if the human is  $80\%$  accurate, the model is also  $80\%$  accurate) with the human predictions and integration decisions by modifying the threshold on model probability used to make predictions (e.g. from the usual 0.5 threshold to the value that makes the models calibrated). Each participant is evaluated on a randomized set of examples, we create 40 different sets of 20 images that get assigned randomly to each participant.
Attention Checks. In each condition, we insert 3 attention checks for every 20 examples where the images are unblurred and we keep the AI prediction. We only retain responses where participants don't get all attention checks wrong. The class balance of the dataset is close to  $51\% -49\%$ . The attention checks are used as part of the study results as they only modify the blur of the images.
Regions Found. The regions found by our procedure are shown in Table 8.
Table 8: Region descriptions found by IntegrAI for the BDD user study.
Region ID	Description
1	depict various city streets and highways during the daytime with different weather conditions, containing pedestrians, cars, trucks, traffic lights, and signs,.
2	depict various types of roads and streets during the daytime with different weather conditions, containing cars, pedestrians, traffic lights and signs, while the outside examples include scenes with fewer objects or in less common locations such as a parking lot,.
3	depict various scenarios of streets and highways with moderate to heavy traffic flow during the day or night, with different weather conditions, along with traffic signs and lights, cars, trucks, buses, and pedestrians.
4	depict various outdoor scenes during the daytime or nighttime, containing multiple cars, traffic lights, and traffic signs
5	depict various city and residential scenes during the daytime with different weather conditions and contain a variety of vehicles, pedestrians, traffic lights, and signs, while the outside examples depict specific limited scenarios with fewer elements,.
6	depict various traffic scenes, mostly highways at night and during clear weather, with a mix of cars, trucks, buses, traffic lights, and traffic signs
7	depict various city streets and highways with clear weather and a moderate amount of traffic, including cars, signs, and occasionally pedestrians, bicycles, and trucks
8	depict various urban and residential scenes during different times of day and weather conditions, with a diverse range of vehicles, pedestrians, traffic signs, and traffic lights present
9	depict various scenes of city streets and highways with typical traffic conditions and without severe weather conditions
10	depict various scenes of city streets and highways with varying weather condi-tions, traffic, and signage.
F.2 MMLU Study
Task. We rely on the MMLU dataset [33] where participants are shown a question, four possible answers and have to pick the best answer (see screenshots in Figure below for examples). We use ChatGPT, also known as GPT 3.5 turbo as our AI model [59]. We obtain the predictions of ChatGPT
on the MMLU dataset following the approach in the official repo of MMLU  $^{5}$ . We also ask ChatGPT to explain it's answer by using the prompt "Please explain your answer in one sentence". Both the AI answer and the explanation are shown.