SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Illustrative Example. Suppose we want to describe a region consisting of images of highways during the night, with no cars present (see Figure 1 for images of BDD [83]). Our method's initial description is "The highway during the night with clear, rainy or snowy weather", not mentioning that the highway has no cars, particularly because the captions of examples  $t_i$  only mention the presence of cars and not their absence. In the second round, the algorithm finds the counterexample  $s^-$  as 'city street during the night with clear weather with a lot cars' and counterexample  $s^+$  "highway during the night with clear weather". The new description  $T_k^1$  becomes "clear highway during the night with various weather conditions, while outside the region are busy city street at night with clear weather". After one more round, the description  $T_k^2$  becomes "driving on a clear and uncongested highway during the night in various weather conditions". We now proceed in the next section to describe how we onboard the human decision maker using the regions.
5 Onboarding and Recommendations to Promote Rules
Once rules for integration have been learned as described in the previous section, the task is to teach these rules to the human with an onboarding stage and encourage their use at test time. We accomplish this through an onboarding process followed by test-time recommendations, as described next.
Human-AI Card. The onboarding process accordingly consists of an introductory phase and a teaching phase. In the introductory phase, the user is first asked to complete a few practice examples of the task on their own to gain familiarity. The user is then presented with general information about the AI model in the form of a human-AI card, akin to a model card [55], that includes AI model inputs and outputs, training data, training objectives, overall human and AI performance along with AI performance on subgroups that have performance that deviate from average performance (details in Appendix D).
The teaching phase is structured as a sequence of "lessons", each corresponding to a region  $N_{k}$  resulting from the algorithm of Section 4.1. The specific steps in each lesson are as follows:
Step 1: Human predicts on example. A representative from the region is selected at random that has an optimal integration decision identical to that of the region. The human is asked to perform the task for the chosen representative and is shown the AI's output along with the option to use the AI's response.
Step 2: Human receives feedback. After submitting a response, the user is told whether the response is correct and whether the AI is correct.
Step 3: From Example to Region Learning. The user is informed that the representative belongs to a larger region  $N_{k}$  and is provided with the associated recommendation  $r_k$ , textual description  $t_k$ , and AI and human performance in the region as well as the raw examples from the region in a gallery viewer.
After completing all lessons as above, a second pass is done where the user is re-shown all lessons for which their response was incorrect. This serves to reinforce these lessons and is similar to online learning applications such as Duolingo for language learning [60]. Our teaching approach is motivated by literature showing that humans learn through examples and employ a nearest neighbor type of mechanism to make decisions on future examples [12, 68, 29]. We follow this literature by showing concrete data examples in the belief that it effectively teaches humans how to interact with AI. We improve on the approach in [57] by incorporating the introductory phase, showing pre-defined region descriptions more clearly established, and iterating on misclassified examples.
Recommendations in AI Dashboard. At test time, we can check whether an example  $x$  and its corresponding AI output  $a$  fall into one of the learned regions  $N_{k}$ . If they do, then our dashboard shows the associated recommendation  $r_{k}$  and description  $t_{k}$  alongside the AI output  $a$ .
6 Method Evaluation
Objective. In this experimental section $^2$ , we evaluate the ability of our algorithms to achieve three aims: (Aim 1) is learning an integration function that leads to a human-AI team with low error; (Aim 2) is discovering regions of the data space that correspond to the underlying regions where human vs AI performance is different; and, (Aim 3) is evaluating the ability of our region description algorithm to come up with accurate descriptions of the underlying regions. Full experimental details are in Appendix E.
Datasets and AI Models. The experiments are performed on two image object detection datasets and two text datasets. The image datasets include Berkeley Deep Drive (BDD) [83] where the task is to detect the presence of traffic lights from blurred images of the validation dataset (10k), and the validation set of MS-COCO (5k) where the task whether a person is present in the image [48]. The text-based validation datasets comprise of Massive Multi-task Language Understanding (MMLU) [33], and Dynamic Sentiment Analysis Dataset (DynaSent) [61]. The pretrained Faster R-CNN models [65] are considered for the BDD and MS-COCO. For MMLU, a pre-trained flan-t5 model [20] is utilized, whereas a pre-trained sentiment analysis roBERTa-base model is used for DynaSent [8]. Each dataset is split into 70-30 ratio for training and testing five different times so as to obtain error bars of predictions. We obtain embeddings for the text datasets using a sentence transformer [64] and CLIP for the image datasets [62]
Figure 2: Test Error  $(\downarrow)$  of the human-AI system when following the decisions of the different integrators baselines as we vary the number of regions maximally allowed for each integrator on the BDD dataset.
Baselines. We benchmark our algorithm with different baseline methods that can find regions of the space. The baselines include: (a) DOMINO [24] which is a slice-discovery method for AI errors, (b) K-Means following the approach of [63], and (c) the double-greedy algorithm from
$^{2}$ Code is available in https://github.com/clinicalml/onboarding_human.ai.
3We extend this to detecting presence of any object.
[57] that finds regions for Human-AI onboarding. For the regions obtained from these baselines, we compute the optimal integration decision that results in minimal loss. For our method, we set  $\beta_{u} = 0.5$ ,  $\beta_{l} = 0.01$ ,  $\alpha = 0.0$  for Aim 1 and  $\beta_{u} = 0.1$ ,  $\beta_{l} = 0.01$ ,  $\alpha = 0.5$  for Aim 2 and random prior decisions (50-50 for 0 and 1). In the context of the region-description algorithms, we compare to the SEAL approach [63], a simple baseline that picks the best representative description from the existing dataset (best-caption) and ablations of our method.
Table 1: Error  $(\downarrow)$  on the test set (in %) (region discovery) of the human-AI system when following the integrator resulting from the different methods at region size 10 on the different non-synthetic datasets.
BDD	MMLU	DynaSent	MS-COCO
IntegrAI (ours)	17.8 ± 0.2	45.3 ± 0.3	20.2 ± 0.3	22.6 ± 0.4
DOMINO [24]	18.9 ± 0.4	48.1 ± 0.2	20.0 ± 0.2	22.7 ± 0.4
K-MEANS [63]	19.0 ± 0.5	45.3 ± 0.3	20.0 ± 0.2	23.2 ± 0.1
DoubleGreedy [57]	18.9 ± 0.1	46.1 ± 0.6	20.0 ± 0.2	23.8 ± 0.4
Table 2: Clustering metrics (Adjusted Rand index [69] ↑, Fowlkes-Mallows index [26] ↑) of the regions (10 regions) found by the different methods on the synthetic dataset setup.
BDD	MMLU	MS-COCO
IntegrAI (ours)	(0.06,0.40)	(0.41,0.84)	(0.08,0.53)
DOMINO [24]	(0.02,0.25)	(0.17,0.65)	(0.04,0.43)
K-MEANS [63]	(0.04,0.27)	(0.08,0.49)	(0.08,0.49)
DoubleGreedy [57]	(0.02,0.26)	(0.10,0.52)	(0.02,0.40)
Learning Accurate Integrators ( Aim 1). The goal is to measure the ability of our method in learning integration functions that lead to low Human-AI team error (the loss  $L(H, M)$ ). This can be well represented by measuring the errors on the training set (discovering regions of error) and the test set (generalization ability). In Table 1, we show the results of our method and the baselines at learning integrators and find that our method can find regions that are more informative with respect to Human vs AI performance on the test data and significantly better on the training data. Figure 2 shows that on BDD our method can find an integrator that leads to lower loss at test times than the baselines with a minimal number of regions.
Recovering Ground truth Regions (Aim 2). We just established that the regions discovered by our algorithm result in a Human-AI team with lower error than human or AI alone. However, it still needs to be verified if the regions denote any meaningful and consistent regions of space. We utilize a synthetic setup by simulating the AI model and the human responses such that there exist (randomized) regions in the data space where either the human or the AI are accurate/inaccurate. These regions are defined in terms of metadata. As an example on the BDD dataset, we can define the AI to be good at daytime images and bad at images of highways, whereas the human to be good at nighttime images and bad at images of city streets. We employ our algorithm and the baselines to discover the regions and compare them with the ground truth region corresponding to the partition of the data, which is essentially a clustering task with ground truth clusters. Results are shown in Table 2 indicating that the ground truth regions can be recovered to an extent.
Describing Regions (Aim 3). We conduct an ablation study where we evaluate the power of contrasting and self-correcting ability of Algorithm 1 against baselines. On the MS-COCO dataset, we take regions defined in terms of the presence of a singular object (e.g., 'apple') and try to uncover a single-word description of the region from the image captions. We use standard captioning metrics that compare descriptions from the algorithms to the object name, we include a metric called "sent-sim" that simply measures cosine similarity with respect to a sentence transformer [64]. We compare to ablations of Algorithm 1 with  $m \in \{0, 5, 10\}$  (rounds of iteration) and without having examples outside the region (IntegrAI,  $S^{-} = \emptyset$ ). Results are in Table 3 and show that including examples outside the region improves all metrics while increasing iterations ( $m$ ) only slightly improves results.
Table 3: Evaluation of our region description algorithm (Algorithm 1) on a synthetic evaluation of MS-COCO where the different algorithms try to describe a set of points that contain the presence of an object. For example, a region is defined if it contains the object "apple". Then the different algorithms try to describe the region and we compare it to the description "apple".
best-caption	SEAL	IntegrAI (m=5)	IntegrAI (m=10)	IntegrAI (S- = ∅)	IntegrAI (m=0)
METEOR	12.9 ± 1.9	9.16 ± 1.89	26.1 ± 3.3	25.4 ± 3.3	24.3 ± 3.3	25.4 ± 3.2
sent-sim	39.8 ± 1.9	44.1 ± 2.5	66.0 ± 3.2	68.0 ± 3.3	65.1 ± 3.2	67.0 ± 3.1
ROUGE	5.81 ± 1.2	0.0 ± 0.0	27.9 ± 5.1	35.6 ± 5.5	25.6 ± 4.9	32.6 ± 5.4
SPICE	12.7 ± 1.9	7.53 ± 2.3	45.2 ± 5.8	45.2 ± 5.8	41.1 ± 5.8	43.8 ± 5.8
(a) BDD
(b) MMLU
Figure 3: (a) Interface for humans to predict a traffic light on images from BDD dataset in the presence of AI's prediction, confidence score, and bounding box and (b) interface for humans to answer multiple choice questions from MMLU dataset with AI's prediction and explanation.
For the apple example, the ablated method finds the description to be "fruit" whereas our method finds it as "apple", thereby eliminating any possibility of confounding effects. More details are provided in the appendix.
7 User Studies to Evaluate Onboarding Effect
Tasks. We perform user studies on two tasks: 1) predicting the presence of a traffic light in road images from the BDD dataset [83] and 2) answering multiple-choice questions from the MMLU [33] dataset. For BDD, we blur the images with Gaussian blur to make them difficult for humans and use the faster r-cnn model for the AI. Participants can see the AI's prediction, bounding box on the image as an explanation and the model's confidence score. For MMLU, participants are shown a question, four possible answers and have to pick the best answer and the prediction of GPT-3.5-turbo [59]. We also obtain an explanation by using the prompt "Please explain your answer in one sentence", both the AI answer and the explanation are shown. GPT-3.5 obtains an accuracy of  $69\%$  during our evaluation and we restrict our attention to specific subjects within the MMLU dataset. Specifically, we sample 5 subjects (out of the 57 in MMLU) where ChatGPT has significantly better performance than average, 5 where it's significantly worse, and 4 subjects where performance is similar to average performance. We sample 150 questions from each subject and additionally sample 150 questions from OpenBookQA dataset [54] to use as attention checks. We show the prediction interfaces in Figure 3. Details are in Appendix F.
Participants. We submitted an IRB application and the IRB declared it exempt as is, all participants agreed to a consent form for sharing study data. We recruited participants from the crowdsourcing website Prolific [1] from an international pool filtering for those fluent in English, have above  $98\%$  approval rating, have more than 60 previous submissions, and have not completed any of our studies before. For BDD, participants are compensated  $3 per 20 examples in the study and then some receive a bonus of$ 2 for good performance. For MMLU, we pay participants $3 for every 15 questions. We collected information about participants' age, gender (52% identify as Female), knowledge of AI, and other task-specific questions. Participants have to correctly answer three initial images without blur (questions from OpenBookQA for MMLU) and encounter attention checks throughout their studies to further filter them, we exclude participants who fail all attention checks.
Experimental Conditions. For BDD, we initially collect responses from 25 participants that predict without the AI and then predict with the help of AI (but no onboarding), we use this data to collect the dataset  $D$  of prior human integration decisions and human predictions to find 10 regions using IntegrAI. We then run four different experimental conditions with 50 unique participants in each where they predict on 20 examples: (1) human predicts alone (H) and human predicts with the help of AI (H-AI) but no onboarding and random order between with and without AI, (2) human
Metric	AI only	Human	Human+AI	Onboard(ours)+Rec	Onboard(ours)	Onboard(baseline)	Rec
Accuracy (%)	79.0 ± 0.7	78.5 ± 1.7	77.2 ± 1.4	79.9 ± 1.4	82.6 ± 1.3	80.4 ± 1.4	81.4 ± 1.8
Test vs H-AI	0.272, 1.211	0.455, 0.752	N/A	0.268, 1.352	0.042, 2.747	0.261, 1.525	0.206, 1.839
AI reliance (%)	N/A	N/A	16.5 ± 3.1	66.5 ± 2.4	25.5 ± 3.4	24.4 ± 4.4	21.4 ± 3.2
Time/example (s)	N/A	5.408 ± 0.289	7.78 ± 0.517	7.622 ± 0.371	5.936 ± 0.288	6.841 ± 0.543	8.717 ± 0.516
Table 4: Results from our user studies. For accuracy, time per example, and AI-reliance we report mean and standard error across participants. The "Test vs H-AI" row reports the adjusted p-value and t-test for a two-sample t-test between the human+AI condition and the other conditions (columns).
Metric	AI only	Human	Human+AI	Onboard(ours)+Rec	Onboard(ours)	Rec
Accuracy (%)	72.9 ± 0.6	52.8 ± 2.2	75.0 ± 1.7	73.7 ± 1.8	74.4 ± 1.7	69.8 ± 1.8
Test vs H-AI	0.230, -1.488	0.0, -7.899	N/A	0.747, -0.53	0.792, -0.265	0.101, -2.08
AI reliance (%)	N/A	N/A	40.0 ± 3.6	34.5 ± 3.7	40.6 ± 3.8	34.2 ± 2.9
Time/example (s)	N/A	30.608 ± 2.109	23.623 ± 1.66	22.917 ± 1.362	20.977 ± 1.509	29.535 ± 1.883
Table 5: Results from our user studies for MMLU.
receives onboarding using our method and then in a random order also receives recommendations (Onboard(ours)  $^+$  Rec) or no recommendations (Onboard(ours)), (3) human goes through only a modified onboarding procedure that only uses step 1 and step 2 from section 5 and uses regions from DOMINO [24] (Onboard(baseline)) and finally (4) human does not receive onboarding but receives the AI-integration recommendations (Rec). For MMLU, participants are tested on 15 examples per condition and onboarding goes through 7 regions found by our algorithm. We run IntegrAI on both the dataset embeddings, metadata (subject name), and an embedding of ChatGPT explanations separately. We find 10 regions based on the metadata and 2 regions based on the ChatGPT explanations. Due to budget constraints, we only run conditions 1-2-4. Note that all participants receive the introduction phase of the onboarding (AI model information) regardless if they go through onboarding or not.
Results. In Table 4 and Table 5 we display various results from the user studies for BDD and MMLU respectively across all experimental conditions. We note the average accuracy (± standard error) across all participants for their final prediction, how often they pressed the "Use AI Answer" button that we call AI-reliance, the time it took for them to make a prediction on average per example (we filter any time period of more than 2 minutes). Finally, we compute using a two-sample independent t-test (* a paired t-test only between Human and Human-AI conditions) the p-value and t-test when comparing each condition (the columns) to the Human-AI condition where the human receives no onboarding. Since we perform multiple tests, we need to correct for multiple hypothesis testing so we rely on the Benjamini/Hochberg method.
Analysis. For BDD, we first observe that human and AI performance is very comparable at around  $79\%$ , which reduces slightly to  $77.2\%$  when the human collaborates with the AI without onboarding. Participants who go through onboarding have a significantly higher task accuracy compared to those who didn't go through onboarding (corrected p-value of 0.042) with a  $5.4\%$  increase. The onboarding baseline fails to significantly increase task accuracy, showcasing that the increase is not due to just task familiarity but possibly due to insights gained from regions showcased. Displaying recommendations in addition to onboarding (Onboard(ours)+Rec) does not improve performance but adds time to the decision-making process (7.6s compared to 5.9s without). For MMLU, we note that there is a  $20\%$  gap between human and AI performance, but human+AI with and without onboarding can obtain an accuracy of around  $75\%$  which is slightly higher than AI alone; onboarding had no additional effect. Interestingly, we find a weakly significant negative effect of only showing AI-integration recommendations which decrease accuracy by  $5\%$  and adds 6 seconds of time per example.
Discussion. We believe that for MMLU, due to the wide gap between human and AI accuracy and convincing ChatGPT explanations, onboarding did not improve performance. Moreover, it is clear that in their current form, displaying the AI-integration recommendation is not an effective strategy and that onboarding on its own is sufficient. Finally, note that even the Human-AI baseline benefits from the human-AI card which might explain that the team is at least as good as it's components.
Limitations. Onboarding and recommendations can significantly affect human decision making. If the recommendations are inaccurate, they could lead to drops in performance and thus require safeguarding. Onboarding and recommendations can be tailored to the specific human by leveraging their characteristics to few-shot learn their prior integrator and prediction abilities.
We thank Hunter Lang and Arvind Satyanarayan for feedback on early stages of this work. HM is thankful for the support of the MIT-IBM Watson AI Lab.
[79] Qingzhong Wang, Jiuniu Wang, Antoni B Chan, Siyu Huang, Haoyi Xiong, Xingjian Li, and Dejing Dou. Neighbours matter: Image captioning with similar images. In BMVC, 2020.
[80] Olivia Wiles, Isabela Albuquerque, and Sven Gowal. Discovering bugs in vision models using off-the-shelf image generation and captioning. arXiv preprint arXiv:2208.08831, 2022.
[81] Jennifer Wortman Vaughan and Hanna Wallach. A human-centered agenda for intelligible machine learning. This is a draft version of a chapter in a book to be published in the 2020 - 21 timeframe., May 2021. URL https://www.microsoft.com/en-us/research/publication/a-human-centered-agenda-for-intelligible-machine-learning/.
[82] Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel S Weld. Errudite: Scalable, reproducible, and testable error analysis. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 747-763, 2019.
[83] Huazhe Xu, Yang Gao, Fisher Yu, and Trevor Darrell. End-to-end learning of driving models from large-scale video datasets. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2174-2182, 2017.
[84] Ming Yin, Jennifer Wortman Vaughan, and Hanna Wallach. Understanding the effect of accuracy on trust in machine learning models. In Proceedings of the 2019 chi conference on human factors in computing systems, pages 1-12, 2019.
[85] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[86] Peter Zhang. Taking advice from chatgpt. arXiv preprint arXiv:2305.11888, 2023.
[87] Yuhui Zhang, Jeff Z HaoChen, Shih-Cheng Huang, Kuan-Chieh Wang, James Zou, and Serena Yeung. Drml: Diagnosing and rectifying vision models using language. In NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications.
[88] Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 295-305, 2020.
[89] Ruiqi Zhong, Charlie Snell, Dan Klein, and Jacob Steinhardt. Describing differences between text distributions with natural language. In International Conference on Machine Learning, pages 27099-27116. PMLR, 2022.
A Extended Related Work 17
B Region Finding Algorithm - Details 18
C Region Description Algorithm - Details 20
D Onboarding and Recommendations to Promote Rules - Details 21
E Method Evaluation - Details 22
F User Studies - Details 24
F.1 BDD Study 24
F.2 MMLU Study 25
F.3 Screenshot of User Study Interface for BDD 26
F.4 Screenshot of User Study Interface for MMLU 32
Reference [47] proposes a method for human-AI collaboration via conditional delegation rules that the human can write down. Our framework enables the automated learning of such conditional delegation rules for more general forms of data that can also depend on the AI output. [77] proposes to modify the confidence displayed by the AI model to appropriately encourage and discourage reliance on the AI model. However, this technique deliberately misleads the human on the AI model ability, our methodology incorporates similar ideas by learning the human prior function of reliance on the AI and then improving on it with the learned integration recommendations, however, we display these recommendations in a separate dashboard without modifying the AI model output. A related approach to our methodology by [51] is to adaptively display or hide the AI model prediction and display the estimated confidence level of the human and the AI on a task of predicting whether a person's income exceeds a certain level. They show that displaying the confidence of the human and the AI to the human improves performance. Our method is able to learn the confidence level of the human and the AI, but also incorporates how the human utilizes the AI and describes the regions where AI vs human performance is different. [16] presents a similar approach to our AI recommendations, however, they use simulated and faked AI models and descriptions of behavior while we are able to obtain automated generation of these descriptions of AI behavior.