Effective Human-AI Teams via Learned Natural Language Rules and Onboarding
Authors: Hussein Mozannar
Jimin J Lee
Dennis Wei
Prasanna Sattigeri
Subhro Das
David Sontag
Venue: NeurIPS 2023 spotlight (2023)
Keywords: human-ai
collaboration
onboarding
region-discovery
LLM
data description
TL;DR: We introduce a novel method for teaching humans how to effectively collaborate with AI agents through natural language rules learned from data and evaluate on user studies.

Abstract: People are relying on AI agents to assist them with various tasks. The human must know when to rely on the agent, collaborate with the agent, or ignore its suggestions. In this work, we propose to learn rules grounded in data regions and described in natural language that illustrate how the human should collaborate with the AI. Our novel region discovery algorithm finds local regions in the data as neighborhoods in an embedding space that corrects the human prior. Each region is then described using an iterative and contrastive procedure where a large language model describes the region. We then teach these rules to the human via an onboarding stage. Through user studies on object detection and question-answering tasks, we show that our method can lead to more accurate human-AI teams. We also evaluate our region discovery and description algorithms separately.

Effective Human-AI Teams via Learned Natural Language Rules and Onboarding
Hussein Mozannar Jimin Leel.2
Dennis Wei
Prasanna Sattigeri'.3 Subhro Das David Sontag
MIITIIBM Watson AI Lab, Cambridge, MA
PCSAIL and IMES, Massachusetts Institute of Technology, Cambridge, MA
'IBM Research, Cambridge, MA
{mozannar: gmminnout @mit edu, {dwei; psattig)@us. ibm subhro das@ibm com, dsontagemit edu
Abstract
People are relying on Al agents to assist them with various tasks. The human must know when to rely on the agent, collaborate with the agent, or ignore its suggestions. In this work, we propose to learn rules grounded in data regions and described in natural language that illustrate how the human should collaborate with the AI. Our novel region discovery algorithm finds local regions in the data as neighborhoods in an embedding space that corrects the human prior. Each region is then described using iterative and contrastive procedure where a large language model describes the region. We then teach these rules the human via an onboarding stage. Through user studies on object detection and arestion-answere tasks, we show that our method can lead to more accurate human-Al teams. We also evaluate our region discovery and description algorithms separately.
Step 1: Human Data Collection
Step 2: Discover Regions
Understand Human performance and prior reliance
Find regions the data space where human collaborates incorrectly with Al
IntegrAI- Rogion Discovery
No
COO
(dataeet)
24 (task), Bi (ansuer)
Light?
N; 00
o, (AI). h, (bumon)
Yiu
Use All Arower
You score
Frecicss
& (text description), es (embeditng)
00
0
000
(d)
(used Al)
Step 3: Describe Regions
Step 4: Teach Regions to Human In an onboarding stage, teach humans each region
Describe regions in text with an LLM
IntegrAI- Region Description
Region 1:
Ignore when it's
"highway during the night"
Region 2:
VS
Use Al when "city during the day"
Region k:
Large Language Model (GPT-3.5-turbo)
Ignore Al when it's "highway during the night"
Figure I: The proposed onboarding approach with the IntegrAI algorithm.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
As artificial intelligence (AI) becomes more ubiquitous and sophisticated, humans are increasingly working alongside AI systems to accomplish various tasks, ranging from medical diagnosis [9. 28], content moderation [30] to writing [21] and programming [22]. One of the promises of AI is to enhance human performance and efficiency by providing fast and accurate solutions. However, the literature on human-Al collaboration has revealed that humans often underperform expectations when working with AI systems [36, 49, 40. 77]. Moreover, studies have shown that providing explanations for the AI's predictions often does not yield additional performance gains and, in fact, can make things worse [49, 15, 32J. These negative results of human-Al performance may be attributed to two possible reasons. First, humans have miscalibrated expectations about Al's ability, which can lead to over or under-reliance [5]. Second, the cost of verifying the Al's answer with explanations is often higher than that of solving the task without the AI some explanations don't ton't help in verifying the Al answer [75].
The central question remains: how can we collaborate better with Al models? In this work, we propose an intuitive framework for thinking about human-Al collaboration where the human first decides each example whether they should rely on the AI, ignore the AI, collaborate with the AI to solve the task. We refer to these three actions as the Al-integration decisions. The human-Al team will perform optimally if the human knows which action is best on a task-by-task basis. We propose IntegrAI (Figure 1), an algorithm that leverages data from baseline human interactions with the AI to learn near-optimal integration decisions, in the form of natural language rules that are easily understandable. These rules are then taught to the human through an onboarding stage, analogous an onboarding class that humans might take before operating machines and equipment. Onboarding additionally calibrates the human's 's expectations about Al performance. We further investigate surfacing the Al-integration decisions found by IntegrAI AI recommendations to the human within an Al dashboard used after onhoarding. The hope is that onboarding and the dashboard help the human know which action they should take, thereby leading effective Al adoption for enhanced decision-making.
Learning AI-integration rules requires dataset of paired examples and human predictions (Figure 1 Step 1). Each rule is defined as a bounded local region centered around a learned point in potentially mudti-modal embedding space spanning the task space and natural language (Figure 1 Step 2). For example, CLIP embeddings [62] connect image and text spaces for tasks involving images, and typical text embeddings [64] are used for natural language tasks such question answering. The regions are obtained with a novel region discovery algorithm. Then, a text description of the region is generated, resulting in rule that indicates whether the human should ignore, rely on, or collaborate with the AI. We obtain descriptions using a novel procedure that connects the summarization ability of large language model (LLM) [14] with the retrieval ability of embedding search find similar and dissimilar examples. The procedure first queries the LLM to describe points inside the region (Figure Step 3). The embedding space is then leveraged find counterexamples inside and outside the region to refine the description.
We first evaluate the ability of our region finding and region description algorithms to find regions that will aid the human-Al team in several real-world datasets with image and text modalities. We then investigate the efficacy of both algorithms in synthetic scenarios where know the ground truth regions. Finally, we conduct user studies on tasks with real-world AI models to evaluate our onboarding and Al-integration recommendation methodology. Our main task is detecting traffic lights in noisy images [83] from the perspective of a road car, motivated by applications to self-driving cars. The user study reveals that our methods significantly improve the accuracy of the human-Al team by 5.2% compared to without onboarding. We investigate a second task of multiple choice question answering using the MMLU dataset [33] and find that onboarding has no effect on performance and that only displaying AI Al-integration recommendations has negative effect. To summarize, the key contributions of this paper are as follows:
We propose a novel region-finding and -egion-deerrpttion algorithm, IntegrAI, that outper- forms prior work to derive rules that can guide humans in collaborating with Al models for shared decision-making
We demonstrate the effect of onboarding and displaying Al-integration recommendations on two real-world tasks, and find that onboarding has a significant positive effect on one task and that integration recommendations are not useful.
2

---

2 Related Work
A growing literature of empirical studies Al-assisted decision making has revealed that human-Al teams do not perform better than the maximum performance of the human or Al alone even with Al explanations [7. 71, 49]. Recent work has explored modifying AI explanations depend on human behavior [77, 51] and showed positive results with synthetic AI models. A growing literature exists on onboarding humans work on Al models [67. 17, 45, 57, 16, 41J, however, our work enables the automated creation of onboarding without any human in the loop. We compare to a representative from work on discovering regions of errors in AI models [24, 10, 66, 82, 87, 80, 23, 63, 37], our work instead focuses on regions of disparate performance between human and AI. Learning to defer methods learn models that decide using secondary AI model who between the human and the Al classifier should predict [52, 56], this paper focuses on the reverse setting where the human makes all decisions but utilizes some of the thinking from that literature. Our Al-integration recommendations are also related to personalized policies [11]. Our MMLU experiments share similarities with recent work [13, 86, 74, 38]. Further comparison to prior work can be found in Appendix A.
3 AI Assisted Decision Making
Setting, We consider a setting where a human is making decisions with the help of an Al agent who provides advice to complete task. Formally, the human has to make decision Y E ) given access to information about the context as Z E 2 and the Al's advice E A. We denote the human as a potentially randomized function HZ,, A: 0%.) with parameters 0% which other hand, the AI agent provides advice based on its viewpoint of the context as X E X, which might be different from the humans viewpoint& Z as the human may have side information. The Al model provides advice according to (X; 0m) == A E A, the advice always includes a candidate decision Y and possibly an explanation of the decision. We assume that the observed tasks are drawn from an underlying distribution, IP P,,,,,, over the contexts of Al and human, and the ground truth. For simplicity of exposition, we will assume that X Z when describing our methods.
unobservable. On the
Task Metries. The human wants to make decision that optimizes various metries of interest. Given ground truth decision Y and a chosen decision Y. the loss is given by ((Y, y x R. We denote the loss L(H, MM ) of the Human-AI team over the entire domain as:
L(H, == Ey.., PI (y,, H (z, M mm));
In our example, this could be the 0-1 loss =y= We are further interested in metrics that convey efforts undertaken by the human during the process. Particularly, we focus on time to make a decision, which can be measured when the human makes decisions.
Human-Al I team. The decision of the Auman-Al team is represented by the function H A; 0z). The human without the AI is denoted by H H(Z, 0: 0,.) == HZ: 0s), which is obtained by setting the AI advice to the empty set, i.e. advice. In theory, we expect the human with advice to perform least as well as without advice, simply because the human can always ignore the advice. However, the literature on Human-Al teams has clearly demonstrated that this is often not the case [46]. An effective Human-AI team is one where the human with the Al's advice achieves a more favorable trade-off in terms of the metrics than without the advice.
Framework for Cooperation. Our insight into forming an effective team is to explicitly recommend when the human should consider the advice and how to incorporate it into their decision. We propose a two-stage framework for the human to cooperate with the AI: the human first decides whether to ignore the AI advice, use the Al's decision integrate the AI advice make decision with explicit cooperation. Each of these three cases provides a clear path to the second stage of making the final output decision.
Definition The Al-integration function R(Z, A; 0r), also referred to as integrator, formalizes framework for the human to cooperate with the Al:
0
H H Ow)
(ignore AI)
(use Al decision as is)
H(Z, A; 0,)
(collaborate with Al)
In this work, we only consider the actions of ignoring or using the Al decision: E (0, 1) and leave the action of R. = 2 for future work.
3

---

The integration function can be thought of as specific formalization of the human mental model of the AI [5, 6]. Given an integration function R and human H, we can define hypothetical human decision maker Hn who first computes R and then follows its recommendation to make final decision. Similarly, for each human (Z, A: fn), we can associate an integration function Ræ, such that HH H. By fixing (ZZ; 0,), one can try to minimize the loss L RR. M) over all possible choices of R. an optimal point of such an integration function is denoted R Following the recommendations of the optimal Al-integration function leads to an effective Human-Al team.
Learning Rules and Onboarding. There are two problems that need to be solved to achieve this vision: how do we learn such an R and how can we ensure that the human follows the recommendations of this R'? In the next section, we outline how we approximate the optimal integration function: this is fundamentally a machine-learning problem with its own challenges that we tackle in Section 4. The second obstacle is that the human should know to follow the recommendations of R To ensure this, we propose an onboarding stage where the human learns about the Al and the optimal integration function and additionally display the recommendations as part of an AI dashboard. In this onboarding stage, we will help the human shape their internal parameters (0n, 0,) to improve performance. This is a human-computer interaction (HCI) problem that we tackle in Section 5.
4 Learning Rules for Human-Al Cooperation: IntegrAI
In this section, we discuss how to learn an integrator function R: X x A- {0, 1} to approximate an optimal integrator while being understandable to the human. We first deseribe the ingredients for this learning (integrator as a set of regions, objective function, dataset) before detailing how we learn regions and describe them in Sections 4.1 and 4.2 respectively.
Integrator as a Set of Regions. Since the integrator / will be used to both onboard the human and provide test-time recommendations as part of an Al dashboard, it should be easily understandable to humans. If the goal [56, 19, 58]. To address this requirement, we propose to parameterize the integrator in terms of a set of local data regions, each with its own integration decision label as well as a natural language description. More specifically, we aim to learn a variable number of regions N1, N2. ... as functions of (X, A), the observable context and the Al's advice. Each region N₂ consists of the following: 1) an indicator function IN, : X' x A + {0, 1} that indicates membership in the region, 2) a textual description of the region TR and 3) a takeaway for the region consisting of an integration decision in r(N,) E 13. We additionally want these regions to satisfy set of constraints so that they are informative for the human and suitable for onboarding.
to build the most accurate integrator, could use work on learning to defer
Maximizing Human's 's Performance Gain. Since we are working with human decision-makers, we have to account for the fact that the human implicitly has a prior integration function Ro, which represents how the human would act without onboarding. Thus, in learning integrator regions, our goal is to maximize the human performance gain relative to their prior. The performance gain is defined as follows for points in a region N:
G(N, R. Ro) = Eler, HR. (xi, - Uyr, H (=(,
U1,, IEN
where / is the loss defined in Section 3.
Note that the notion of a human's 's prior mental model was also discussed by [57] but we expand on the notion and are able to learn priors as we discuss below.
Dataset with Human Decisions. We assume we have access to a dataset 1) = {z4, Vr. h., Fos ) sampled from IP, where x; the Al-observable context, ys the optimal decision on example i, h; is a human-only decision, defined in Section 3 as H (z,;; 0%), and ros E {0. 1) is an indicator of whether the human relied on AI on example We thus regard the samples froi as proxy for prior human integration function Ry. The prior integration decisions of the human Fos are collected through a data collection study where the human predicts with the AI without onboarding. When the human presses on the "Use AI" button (see Figure 3) we record ro: = 1, and 0 otherwise. The human predictions h; are collected through a secondary data collection study where the human makes predictions without the AI We also assume that we given an AI model M, from which we obtain

---

Al decisions Ds a₂ from the Al advice = M (2;; 6m) Given the dataset D, Al decisions Ús-- and loss ((., .. we can define optimal per-example integration decisions rf by comparing human and Al losses on the example: r? = (1(v,, h;) >> L(v, ii)).
4.1 Region Discovery Algorithm
Representation Space. In this subsection, describe sequential algorithm that starts with the prior integration function Rg and adds regions N₂ one at a time. The domain X' for the task may consist of images, natural language, or other modalities that possess interpretable representation space. We follow a similar procedure for all domains. The first step is to map the domain onto a potentially cross-modal embedding space using mapper E(.), where language. The motivation is that such an embedding space will have local regions that share similar natural language descriptions. enabling us to learn understandable rules. For example, for natural images, we use embeddings from the CLIP model [62]. The result of this step is to transform the dataset ) into a dataset of embeddings {ex ) where e; E εx S Rdd.
of the modes is natural
Region Parameterization. We define region N₂ in terms of a centroid point Ck. a scaled Euclidean neighborhood around it, and integration label r {0, 1}. The neighborhood is in turn defined by a radius E IR and element-wise scaling vector wk. Both Cj and try are in & x A, the concatenation of the embedding space and the Al advice space. The indicator of belonging the region is then where o is the Hadamard (element-wise) product.
TN, (ei, a.) = (())))
Region Constraints. We add the following constraints on each region Ng to make them useful to the human during onboarding. First, the region size in terms fraction of points contained must be bounded from below by 3t and above by B., Second, the examples in each region must have high agreement in terms of their optimal per-example integration decisions r,. Specifically, at least a fraction of the points in region must have the same value of rf
IntegrAI-Discover In round k we add kth region to the integrator RE-1 from the previous round (with k - regions) to yield Rg. After rounds, the updated integrator Rt, is defined as follows: Given point (ar, a), it does not belong to any of the regions N1,..., NN, then we fall back on the prior. Otherwise, we take majority vote over all regions to which (x, a) belongs:
Ro(x, a) majority (fr(N)) k s.t. INz (x, = =1)
if IN. (I. = 0, 1=1....,7 T otherwise.
RT a) =
(4)
In round k, we compute the potential performance gain for each point if we were to take decision on the point as gr. = l(y;, R (2;, a.)) ((y, r). The optimization problem to find the optimal regions is non-differentiable due to the discontinuous nature of the region indicators. To make this
optimization problem differentiable, we relax the constraints as penalties with a multiplier À and replace the indicators with sigmoids scaled by large constant C₁:
max. Σx (C1(-11 (le,,,)
e)]l + 7)) gir À Amax(E.. (Ci(-1luu ((G.a)) c)ll + +7))
-
À max Co(Gi(-llur ((E,, -)ll +7)) Buni,
The lower bound constraint with & can be added in similar fashion. We find the value of with an exhaustive search and use gradient based procedure for the other optimization variables. The full optimization details are in Appendix B.
4.2
Region Description Algorithm
We now describe our region description algorithm aimed at making the rules for integration human understandable. Natural language descriptions are good match for this objective. Specifically, we would like to find contrastive textual description T3 of each region N₂ that describes a way to distinguish it from the rest of the data space.
Textual Descriptions for Regions: The first step is to have textual description t, for each example in our dataset D based on v.. If textual descriptions are not available, we can obtain them by utilizing
Moreover, we assume the Al was not trained on the dataset D so that we can use D to obtain an unbiased measurement of Al performance; this is crucial, as otherwise, might overestimate its performance.
5

---

Algorithm IntegrAI-Describe
Input: Dataset D, region N₂
1: S < 15 random examples from NR. S + random examples outside N₁:
2: Initial Region Description: T t- Ö(s+ ,5)) (LLM call)
3: for i 1,..., m do
4: Find Counterexample outside region. s- arg ma sim sim (T(()) .eeg))ee))))) ez) (most similar
outside region)
5: Find Counterexample inside region. s+ = arg minjeN, (T((E)T), ez) (least similar inside region)
6: Update Inside and Outside Sets.
S f- 5 Ul,- and St < S Ulst
7: Get New Region Description: T + (S5t 5-) (LLM call)
8: end for
Return: Region description Thm
models that map from the domain X' to natural language such as captioning models for images, summarization models for text, exploiting metadata to construct natural language description. One idea to obtain a region description is to ask an LLM (such as GPT-3.5) to summarize all textual descriptions of points inside the region. However, there are issues with this approach: first, the region may contain thousands of examples so we need an effective way to select which points to include, and second, the obtained region description may contrast with points outside the region. To resolve these issues, we propose an algorithm that iteratively refines region descriptions with repeated calls to an LLM (Ò) in 1 PettegrA--Descriiee description and then finds counterexamples to that description at each round: examples outside the region with high cosine similarity with respect to embeddings (sim) of the region description and examples inside the region with low similarity to the region description. Then we add those counterexamples to our example sets and derive new region description, we use an especially created prompt with an exemplar the LLM to get the region description at each round, the prompt can be found in Appendix C.
The algorithm starts with an initial
Illustrative Example. Suppose we want describe region consisting of images of highways during the night, with no cars present (see Figure 1 for images of BDD [83]). Our method's initial description is "The highway during the night with clear, rainy or snowy weather", not mentioning that the highway has no cars, particularly because the captions of examples i; only mention the presence 'city
of cars and not their absence. the second round, the algorithm finds the counterexample street during the night with clear weather with a lot cars' and counterexample st "highway during the night with clear weather". The new description T becomes "clear highway during the night with various weather conditions, while outside the region are busy city street at night with clear weather". After one more round, the description Tē becomes "driving on a clear and uncongested highway during the night in various weather conditions". We now proceed in the next section to describe how we onboard the human decision maker using the regions.
Onboarding and Recommendations to Promote Rules
Once rules for integration have been learned described in the previous section, the task is to teach these rules to the human with an onboarding stage and encourage their use at test time. We accomplish this through onboarding process followed by test-time recommendations, as described next.
Human-Al Card. The onboarding process accordingly consists of an introductory phase and teaching phase. In the introductory phase, the user is first asked to complete a few practice examples of the task on their own gain familiarity. The user is then presented with general information about the AI model in the form of human-AI card, akin to a model card [55], that includes AI model inputs and outputs, training data, training objectives, overall human and Al performance along with AÍ performance on subgroups that have performance that deviate from average performance (details in Appendix D).
The teaching phase structured as a sequence of "lessons", each corresponding to a region N₂ resulting from the algorithm of Section 4.1. The specific steps in each lesson are as follows:

---

Step 1:
Human predicts on example. A representative from the region is selected at random that has an optimal integration decision identical that of the region. The human is asked perform the task for the chosen representative and is shown the Al's output along with the option to use the Al's response.
Step 2: Human receives feedback. After submitting a response, the user is told whether the response correct and whether the AI is correct.
Step 3: From Example to Region Learning, The user informed that the representative belongs to larger region N& and is provided with the associated recommendation rk, textual description ty- and Al and human performance in the region as well as the raw examples from the region a gallery viewer.
After completing all lessons as above, a second pass is done where the user is re-shown all lessons for which their response was incorrect. This serves to reinforce these lessons and is similar to online learning applications such as Duolingo for language learning [60]. Our teaching approach is motivated by literature showing that humans learn through examples and employ nearest neighbor type of mechanism to make decisions on future examples [12, 68, 29]. We follow this literature by showing concrete data examples in the belief that it effectively teaches humans how to interact with AL We improve on the approach in [57] by incorporating the introductory phase, showing pre-defined region descriptions more clearly established, and iterating on misclassified examples.
Recommendations in AT Dashboard. At test time, we can check whether an example x and its corresponding Al output fall into one of the learned regions N₁. If they do, then our dashboard shows the associated recommendation rı: and description 4 alongside the Al output a.
Method Evaluation
Objective. In this experimental section", evaluate the ability of our algorithms to achieve three aims: (Aim 1) is learning an integration function that leads to a human-Al team with low error; (Aim 2) is discovering regions of the data space that correspond to the underlying regions where human vs Al performance is different; and, (Aim 3) is evaluating the ability of our region description algorithm to come up with accurate descriptions of the underlying regions. Full experimental details are in Appendix E.
Datasets and AI Models. The experiments are per- formed on two image object detection datasets and two text datasets. The image datasets include Berke- ley Deep Drive (BDD) [83] where the task is to detect the presence of traffic lights from blurred images of the validation dataset (10k), and the validation set of MS-COCO (5k) where the task whether person is present in the image* 3 [48]. The text-based validation datasets comprise of Massive Multi-task Language Understanding (MMLU) [33], and Dynamic Senti- ment Analysis Dataset (DynaSent) [61]. The pre- trained Faster R-CNN models [65] are considered for the BDD and MS-COCO. For MMLU, a pre-trained flan-t5 model [20] is utilized, whereas a pre- pre-trained sentiment analysis roBERTa-base model is used for DynaSent [8]. Each dataset is split into 70-30 ratio for training and testing five different times so as to ob- tain error bars of predictions. We obtain embeddings for the text datasets using a sentence transformer [64] and CLIP for the image datasets [62]
Irtep (ore))
0.22
DOMINO RAMEANS
0.21
Bazznn
Test Less
Al
10:0
6.10
Number of Regions
Figure 2: Test Error (L) of the human- Al sys- tem when following the decisions of the differ- ent integrators baselines as we vary the num- ber of regions maximally allowed for each integrator on the BDD dataset.
Baselines. We benchmark our algorithm with different baseline methods that can find regions of the space. The baselines include: (a) DOMINO [24] which is a slice-discovery method for AI errors, (b) K-Means following the approach of [63], and (c) the double-greed) algorithm from
2 "Code is available in https://github http://ciiiiaalaa/ooooraoaaa huw.aannnadnnnnnnaaaaaa
We extend this detecting presence of any object.
7

---

[57] that finds regions for Auman-Al onboarding. For the regions obtained from these baselines, we compute the optimal integration decision that results minimal loss. For our method, we set B. = 0.5, & = 0.01, = 0.0 for Aim 1 and 8., 0.1,P & = 0.01, = 0.5 for Aim 2 and random algorithms, we compare to the SEAL approach [63], simple baseline that picks the best representative description from the existing dataset (best-caption) and ablations of our method.
prior decisions (50-50 for O and 1). In the context of the region-descripen
Table I: Error (L) on the test set (in %) (region dis- covery) of the human-Al system when following the integrator resulting from the different methods at re- gion size 10 the different non-synthetic datasets.
Table 2: Clustering metries (Adjusted Rand
index [69] 1, Fowlkes-Mallows index [26] t)
of the regions (10 regions) found by the dif-
ferent methods on the synthetic dataset setup.
BDD
MMLU
DynaSent
MS-COCO
BDD
MMLUU
MS.COCO
InteurAl
17.8
45.3
0.3
0.4
IntegrAl (ours)
(0.06,0.40)
(0.41, (184)
(0.08.0.53)
DOMINO [24]
18.9 1 0.4
48.1 0.2
200 0.2
22.7 0.4
DOMINO (24]
(0.02.0.25)
(0.17.0.65)
(0.04,0.43)
K-MEANS [63]
1.50
453 0.3
20.0 1.2
0.1
K-MEANS [63]
(0.04,0.27)
(0.08,0.49)
(0.08.0.49)
DoubleGreedy (57)
18.9
46.1 1.66
20.0
2.4
DoubleGreedy [57]
(0.02.0.26)
(0.10.0.52)
(0.02,0.40)
Learning Accurate Integrators ( (Aim 1). The goal to measure the ability of our method in learning integration functions that lead to low Human-Al team error (the loss L(H, MM). This can be well represented by measuring the errors on the training set (discovering regions of error) and the test set (generalization ability). In Table 1, we show the results of our method and the baselines at learning integrators and find that our method can find regions that are more informative with respect to Human Al performance on the test data and significantly better on the training data. Figure shows that on BDD our method can find an integrator that leads to lower loss at test times than the baselines with minimal number of regions.
Recovering Ground truth Regions (Aim 2). We just established that the regions discovered by our algorithm result in a Human-Al team with lower error than human or Al alone. However, it still needs to be verified if the regions denote any meaningful and consistent regions of space. We utilize synthetic setup by simulating the AI model and the human responses such that there exist (randomized) regions in the data space where either the human or the AI are accurate/inaccuratc. These regions are defined in terms of metadata. As an example on the BDD dataset, the Al to be good at daytime images and bad at images of highways, whereas the human to be good nighttime images and bad images of city streets. We employ our algorithm and the baselines to discover the regions and compare them with the ground truth region corresponding to the partition of the data, which is essentially clustering task with ground truth clusters. Results shown in Table 2 indicating that the ground truth regions can be recovered to an extent.
define
Describing Regions (Aim 3). We conduct an ablation study where we evaluate the power of contrasting and self-correcting ability of Algorithm against baselines. On the MS-COCO dataset, we take regions defined in terms of the presence of singular object (e.g., 'apple") ) and try uncover a single-word description of the region from the image captions. We use standard captioning metrics that compare descriptions from the algorithms the object name, include metric called "sent- sim" that simply measures cosine similarity with respect to a sentence transformer [64]. We compare to ablations of Algorithm 1 with E {0, 5, 10) (rounds of iteration) and without having examples outside the region (IntegrAI, 5 = 0). Results are in Table 3 and show that including examples outside the region improves all metrics while increasing iterations (m) only slightly improves results.
Table 3: Evaluation of our region description algorithm (Algorithm 1) on synthetic evaluation of MS-COCO where the different algorithms try to describe a set of points that contain the presence of object. For example, a region is defined if it contains the object "apple". Then the different algorithms try to describe the region and we compare it to the description "apple".
best-caption
SEAL
IntegrAI (m=5)
IntegrAI (m=10)
IntegrAI (S- = IntegrAI (m=0)
METEOR
12.9 1.9
9.16 1.89
26 1 3.3
25.4 3.3 3.3
24.3 4 3.3
25.4 : 3.2
39.8
44.1 2.5
66.0
65.1
3.1
ROUGE
5.81 + 1.2
0.0
5.11
35.6 + 5.5
25.6 4.9
32.6 - 5.4
SPICE
12.7 + 1.9
7.53 2.3
45.2 2 5.8
45.2 + 5.8
41.1 5.8
43.8 2 5.8
8

---

Question: a child Inherits physical trait from his father he will likely receive
Skow Ixtrutiaa
A: tooth
B: favorite color
sense of
D: liking dogs
Alc
actiors
Recommendatio:: lynore
Is there Traffic Light?
wrang when variatt quastions un topis such sigebra, biology, public relations, and statetics
Recommendation: Rely on Al Region: moatly highteeys at night and during char weather with mis cans irucks, buses, trallic lights.
A
Al Predicts A
Tootk bequse phyticl tois, The gin betwees teck soe prinody dsaarind asneiko and soaed dows peotss skibban
B
Yes
Predicts: Yos with score
No
Use All Answer
D
Use All Answer
(a) BDD
(b) MMLU
Figure 3: (a) Interface for humans predict traffic light images from BDD dataset in the presence of Al's prediction, confidence score, and bounding box and (b) interface for humans to answer multiple choice questions from MMLU dataset with Al's prediction and explanation.
For the apple example, the ablated method finds the description to be "fruit" whereas our method finds it "apple", thereby eliminating any possibility of confounding effects. More details are provided in the appendix.
User Studies to Evaluate Onboarding Effect
Tasks. We perform user studies two tasks: 1) predicting the presence of traffic light in road images from the BDD dataset [83] and 2) answering multiple-choice questions from the MMLU [33] dataset. For BDD, blur the images with Gaussian blur to make them difficult for humans and use the faster r-cnn model for the AI. Participants can see the Al's prediction, bounding box on the image as explanation and the model's confidence score. For MMLU, participants are shown question, four possible answers and have to pick the best answer and the prediction of GPT-3.5-tund We also obtain an explanation by using the prompt "Please explain your answer one sentence", both the AI answer and the explanation are shown. GPT-3.5 obtains an accuracy of 69% during our evaluation and we restrict our attention to specific subjects within the MMLU dataset. Specifically, we sample subjects (out of the 57 in MMLU) where ChatGPT has significantly better performance than average, where it's significantly worse, and 4 subjects where performance is similar to average performance. We sample 150 questions from each subject and additionally sample 150 questions from OpenBookQA dataset [54] to use as attention checks. We show the prediction interfaces in Figure 3. Details are in Appendix F.
[59].
Participants. We submitted IRB application and the IRB declared it exempt as is, all participants agreed to a consent form for sharing study data. We recruited participants from the crowdsourcing website Prolific [1] from an international pool filtering for those fluent in English, have above 98% approval rating, have more than 60 previous submissions, and have not completed any of our studies before. For BDD, participants are compensated $3 per 20 examples the study and then some receive bonus of $2 for good performance. For MMLU, we pay participants S3 for every 15 questions. We collected information about participants' age, gender (52% identify as Female), knowledge of AI, and other task-specific questions. Participants have correctly answer three initial images without blur (questions from OpenBookQA for MMLU) and encounter attention checks throughout their studies to further filter them, we exclude participants who fail all attention checks.
Experimental Conditions. For BDD, we initially collect responses from 25 participants that predict without the Al and then predict with the help of Al (but no onboarding), we use this data to collect dataset of prior human integration decisions and human predictions to find regions using IntegrAI. We then run four different experimental conditions with 50 unique participants in each where they predict on examples: (1) human predicts alone (H) and human predicts with the help of AI (H-Al) but onboarding and random order between with and without AI, (2) human

---

Metric
AI only
Human
Human=AI
Onboardious))4Re
Onbcardrours)
Onboardthacline)
Accuracy (36)
0.7
78.5 17
77.2 1.4
79.9 1.4
82.6+113
80 80.1214
81.4 18
Test H-Al
1.21
0.755,0.75
NDA
0,268, 1.152
0.042.
1.525
0,206,
relionce (5)
N/A
16.5
66.5
25.5 3.4
24.4 4.4
21.4
Timelotample (6)
N/A
5,408
+ 0.289
7.78
=
0.517
7.622 # 0.371
5,936 0.288
6.811 = 0.543
8.717 +: 0.516
Table 4: Results from our user studies. For accuracy, time per example, and Al-reliance we report
mean and standard error across participants. The "Test vs H-AI" row reports the adjusted p-value and I-test for a two-sample t-test between the human+Al condition and the other conditions (columns).
Metric
Al only
Human
Human+Al
Onboardkour))+Re
Onbeard((uts)
Rec
Accuracy (%)
72.9 + 0.6
32.8 + 2.2
75.0 1.7 N/A
73.7 1.8
74.4 1.7
69.8 = 1,8
Test vs H-AI
0.230, L4488
0.0, 77.899 N/A
0.747, -0.53
0.792. - 265
0.101, -2.08
Al reliance (%)
N/A
40.0 3.6
315 3.7
4
342 2.9
Timelesample (+)
N/A
30.608 = 20109
23.623 + 1.66
22.917 # 1.362
20.977 + 1.509
29.535 1: 18833
Table 5: Results from our user studies for MMLU.
receives onboarding using our method and then in random order also receives recommendations (Re))))) or no recommendations (Onboard(ours)), (3) human goes through only a modified onboarding procedure that only uses step 1 and step 2 from section 5 and uses regions from DOMINO [24] (Onboard(baseline)) and finally (4) human does not receive onboarding but receives the Al-integration recommendations (Rec). For MMLU, participants are tested on 15 examples per condition and onboarding goes through 7 regions found by our algorithm. We run IntegrAI both the dataset embeddings, metadata (subject name), and an embedding of ChatGPT explanations separately. We find 10 regions based on the metadata and 2 regions based on the ChatGPT explanations. Due to budget constraints, we only run conditions 1-2-4. Note that all participants receive the introduction phase of the onboarding (AI model information) regardless they go through onboarding or not.
Results. In Table 4 and Table we display various results from the user studies for BDD and MMLU respectively across all experimental conditions. We note the average accuracy (#: standard error) across all participants for their final prediction, how often they pressed the "Use AI Answer" button that we call Al-reliance, the time it took for them to make prediction filter any time period of more than 2 minutes). Finally, we compute using a two-sample independent t-test (* a paired t-test only between Human and Human-Al conditions) the p-value and t-test when comparing each condition (the columns) to the Human-Al condition where the human receives no onboarding. Since we perform multiple tests, we need to correct for multiple hypothesis testing so we rely on the Bennjamini/Hochbeeee method.
average per example (we
Analysis. For BDD, we first observe that human and Al performance is very comparable at around 79%, which reduces slightly to 77.2% when the human collaborates with the Al without onboarding. Participants who go through onboarding have a significantly higher task accuracy compared to those who didn't go through onboarding (corrected p-value of 0.042) with 5.4% increase. The onboarding baseline fails to significantly increase task accuracy, showcasing that the increase is not due to just task familiarity but possibly due to insights gained from regions showcased. Displaying recommendations in addition onboarding (Onboard(ours)+Re) Docision-making process (7.6 compared to 5.9s without). For MMLU, we note that there is a 20% gap between human and AI performance, but Al with and without onboarding can obtain an accuracy of around 75% which is slightly higher than AI alone; onboarding had no additional effect. Interestingly, find a weakly significant negative effect of only showing Al-integration recommendations which decrease accuracy by 5% and adds 6 seconds of time per example.
does not improve performance but adds time the
Discussion. We believe that for MMLU, due to the wide gap between human and Al accuracy and convincing ChatGPT explanations, onboarding did not improve performance. Moreover, it is clear that in their current form, displaying the Al-integration recommendation is not an effective strategy and that onboarding on its own is sufficient. Finally, note that even the Human-Al baseline benefits from the human-Al card which might explain that the team is at least good as it's components.
Limitations. Onboarding and recommendations can significantly affect human decision making. If the recommendations are inaccurate, they could lead to drops in performance and thus require safeguarding. Onboarding and recommendations can be tailored to the specific human by leveraging their characteristics to few-shot learn their prior integrator and prediction abilities.
10

---

Acknowledgments
We thank Hunter Lang and Arvind Satyanarayan for feedback on early stages of this work. HM is thankful for the support of the MIT-IBM Watson AI Lab.
References
[1] Prolific. attps://www.prolltt
co/. Accessed on May 17, 2023.
[2] Yongsu Ahn, Yu-Ru Lin, Panpan Xu, and Zeng Dai. Escape: Countering systematic enrors from machine's blind spots via interactive visual analysis. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1-16, 2023.
[3] Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et al. Guidelines for human-ai interaction. In Proceedings of the 2019 chi conference on human famtoracttors in computing systems, pages 1-13, 2019.
[4] Matthew Arnold, Rachel KE Bellamy, Michael Hind, Stephanie Houde, Sameep Mehta, Alek- sandra Mojsilovic, Ravi Nair, K Natesan Ramamurthy, Alexandra Olteanu, David Piorkowski, al. Factsheets: Increasing trust ai services through supplier's declarations of conformity. IBM Journal of Research and Development, 63(4/5):6 2019.
[5] Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S Lasecki, Daniel S Weld, and Eric Horvitz. Beyond accuracy: The role of mental models human-ai team performance. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 7, pages 2-11, 11, 2019.
[6] Gagan Bansal, Besmira Nushi, Ece Kamar, Daniel S Weld, Walter S Lasecki, and Eric Horvitz. Updates human-ai teams: Understanding and addressing the performance/comaatiillly tradeoff. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 2429-2437, 2019.
[7] Gagan Bansal, Tongshuang Wu, Joyce Zhu, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel S Weld. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. arXiv preprint wxXi:2006.114777 2020.
[8] Francesco Barbieri, Jose Kumacho-Colladeo Leonardo Neves, and Luis Espinosa- Anke. Tweet- eval: Unified benchmark and comparative evaluation for tweet classification. arXiv preprint arXiv:2010. /2421, 2020.
[9] Emma Beede, Elizabeth Baylor, Fred Hersch, Anna Iurchenko, Lauren Wilcox, Paisan Ruamvi- boonsuk, and Laura M Vardoulakis. A human-centered evaluation of deep learning system deployed in clinics for the detection of diabetic retinopathy. In Proceedings of the 2020 CHI Conference Human Factors in Computing Systems, pages 1-12, 2020.
[10] Homanga Bharadhwaj. De-An Huang. Chaowei Xiao, Anima Anandkumar, and Animesh Garg. Auditing ai models for verified deployment under semantic specifications. arXiv preprint arXiv:2109.122. 12w.2............. 2021.
[11] Umang Bhatt, Valerie Chen, Katherine M Collins, Parameswaran Kamalaruban, Emma Kallina, Adrian Weller, and Ameet Talwalkar. Learning personalized decision support policies. arXiv preprint 27X 2023.
[12] Aaron M Bornstein, Mel W Khaw, Daphna Shohamy, and Nathaniel D Daw. Reminders of past choices bias decisions for reward in humans. Nature Communications, 8(1):1-9, 2017.
[13] Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamile Lukosuite, Amanda Askell, Andy Jones, Anna Chen, et al. Measuring progress on scalable oversight for large language models. arXiv preprint arXiv:22 033540, 2022.
[14] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901,, 2020.
11

---

[15] Zana Bucinca, Phoebe Lin, Krzysztof Z Gajos, and Elena L Glassman. Proxy tasks and subjective measures can be misleading in evaluating explainable systems. In Proceedings of the 25th International Conference Intelligent User Interfaces, pages 454 2020.
[16] Angel Alexander Cabrera, Adam Perer, and Jason Hong. Improving human-ai collaboration with descriptions of ai behavior. arXiv preprint wrXv:2301..06977. 2023.
[17] Carrie Cai, Samantha Winter, David Steiner, Lauren Wilcox, and Michael Terry. "elllo ai": Uncovering the onboarding needs of medical practitioners for human-ai collaborative decision-making. Proceedings of the ACM on Human-computer Interaction, 3(CSCW):1-24, 2019.
[18) Arjun Chandrasekaran, Viraj Prabhu, Deshraj Yadav, Prithvijit Chattopadhyay, and Devi Parikh. Do explanations make vqa models more predictable to a human? arXiv preprint arXiv: /2366, 2018.
[19] Mohammad-Amin Charusaie, Hussein Mozannar, David Sontag, and Samira Samadi. Sample efficient learning of predictors that complement humans. In International Conference on Machine Learning, pages 2972-3005. PMLR, 2022.
[20] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. Scaling language models. arXiv preprint arXiv:2210.1144 11416, 2022.
[21] Andy Coenen, Luke Davis, Daphne Ippolito, Emily Reif, and Ann Yuan. Worderaft: A human-ai collaborative editor for story writing. arXiv preprint arXiv:2107.07433 2021.
[22] Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, Zhen Ming, et Github copilot ai pair programmer: Asset or liability? arXiv preprint arXiv: 15331, 2022.
[23] Greg d'Eon, Jason d' E'Enn, James R Wright, and Kevin Leyton-Brown The spotlight: A general method for discovering systematic errors in deep learning models. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1962-1981, 2022.
[24] Sabri Eyuboglu, Maya Varma, Khaled Kamal Saab, Jean-Benoit Delbrouck, Christopher Lee- Messer, Jared Dunnmon, James Zou, and Christopher Re Domino: Discovering systematic errors with cross-modal embeddings. International Conference on Leurning Representations, 2021.
[25] Shi Feng and Jordan Boyd-Graber. What can ai do for me? evaluating machine learning interpretations in cooperative play. In Proceedings of the 24th International Conference on Intelligent User Interfaces, pages 229 2999229,770 2019.
[26] Edward B Fowlkes and Colin Mallows. A method for comparing two hierarchical clusterings. Journal of the American statistical association, 78(383):553-560 1983.
[27] Irena Gao, Gabriel Ilharco, Scott Lundberg, and Marco Tulio Ribeiro. Adaptive testing of computer vision models. arXiv preprint 22 2022.
[28] Susanne Gaube, Harini Suresh. Martina Raue, Alexander Merritt, Seth J Berkowitz, Eva Lermer, Joseph F Coughlin, John V Guttag, Errol Colak, and Marzyeh Ghassemi. Do as ai say: susceptibility in deployment of clinical decision-aids. NPJ digital medicine, 4(1):1-8, 2021.
[29] Gyslain Giguère and Bradley C Love. Limits in decision making arise from limits in memory retrieval. Proceedings of the National Academy of Sciences, 110(19):613-76118 2013.
[30] Tarleton Gillespie. Content moderation, ai, and the question of scale. Big Data & Society, 7(2):
2053951720943234 2020.
[31] Ana Valeria Gonzalez, Gagan Bansal, Angela Fan, Robin Jia, Yashar Mehdad, and Srinivasan Iyer. Human evaluation of spoken vs. visual explanations for open-domain qa. arXiv preprint arXiv:2012.15075 2020.
12

---

[32] Peter Hase and Mohit Bansal. Evaluating explainable ai: Which algorithmic explanations help users predict model behavior? arXiv preprint wrXiv:2005.01831, 2020.
[33] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint Stt1:20000033300 2020.
[34] Oded Hupert, Idan Schwartz, and Lior Wolf. Describing sets of images with textual-pea, arXiv preprint vrXiv:2210.12112, 2022.
[35] Badr Youbi Idrissi, Diane Bouchacourt, Randall Balestriero, Ivan Evtimov, Caner Hazir- bas, Nicolas Ballas, Pascal Vincent, Michal Drozdzal, David Lopez-Paz, and Mark Ibrahim. Imagenet-x: Understanding model mistakes with factor of variation annotations. arXiv preprint arXiv:2211.01866 2022.
[36] Maia Jacobs, Melanie F Pradier, Thomas H McCoy, Roy H Perlis, Finale Doshi-Velez, and Krzysztof Z Gajos. How machine-learning recommendations influence clinician treatment selections: the example of antidepressant selection. Translational psychiatry, 11(1):1-9, 2021.
[37] Saachi Jain, Hannah Lawrence, Ankur Moitra, and Aleksander Madry. Distilling model failures as directions latent space. arXiv preprint arXiv:2206. 14754, 2022.
[38] Brihi Joshi, Ziyi Liu, Sahana Ramnath, Aaron Chan, Zhewei Tong, Shaoliang Nie, Qifan Wang, Yejin Choi, and Xiang Ren. Are machine rationales (not) useful to humans? measuring and improving human utility of free-text rationales. arXiv preprint arXiv:2305.07009, 2023.
[39] Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, and Jennifer Wortman Vaughan. Interpreting interpretability: Understanding data scientists' use of inter- pretability tools for machine learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1 1-4 2020.
[40] Anna Kawakami, Venkatesh Sivaraman. Hao-Fei Cheng, Logan Stapleton, Yanghuidi Cheng, Diana Qing, Adam Perer, Zhiwei Steven Wu, Haiyi Zhu, and Kenneth Holstein. Improving human-ai partnerships in child welfare: Understanding worker practices, challenges, and desires for algorithmic decision support. In CHI Conference on Human Factors in Computing Systems, pages 1-18, 2022.
[41] Anna Kawakami, Luke Guerdan, Yanghuidi Cheng, Kate Glazko, Matthew Lee, Scott Carter, Nikos Arechiga, Haiyi Zhu, and Kenneth Holstein. Training towards critical use: Learning to situate ai predictions relative to human knowledge. In Proceedings of The ACM Collective Intelligence Conference, pages 63-78, 2023.
[42] Rafal Kocielnik, Saleema Amershi, and Paul N Bennett. Will you accept imperfect ai? exploring designs for adjusting end-user expectations of ai systems. Proceedings of the 2019 CHI Conference Human Factors in Computing Systems, pages 1-14, 2019.
[43] Isaac Lage, Emily Chen, Jeffrey He, Menaka Narayanan, Been Kim, Sam Gershman, and Finale Doshi-Velez. An evaluation of the human--nterprettbility of explanation. arXiv preprint 2019.
[44] Vivian Lai and Chenhao Tan. On human predictions with explanations and predictions of machine learning models: A case study deception detection. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 29-38, 2019.
[45] Vivian Lai, Han Liu, and Chenhao Tan. why is' deceptivee" towards building model- driven tutorials for humans. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1-13, 2020.
[46] Vivian Lai, Chacha Chen, Q Vera Liao, Alison Smith-Renned and Chenhao Tan. Towards a science of human-ai decision making: survey of empirical studies. arXiv preprint axXiv:2112.11471.200 2021. 2021.
13

---

[47] Vivian Lai, Samuel Carton, Rajat Bhatnagar, Q Vera Liao, Yunfeng Zhang, and Chenhao Tan. Human-ai collaboration via conditional delegation: A case study of content moderation. In CHI Conference on Human Factors in Computing Systems, pages 1-18, 2022.
[48] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[49] Han Liu, Vivian Lai, and Chenhao Tan. Understanding the effect of out-of-distribution examples and interactive explanations on human-ai decision making. Proceedings of the ACM on Human- Computer Interaction, 5(CSCW2): 1 2021.
[50] Gaifang Luo, Lijun Cheng, Chao Jing, Can Zhao, and Guozhu Song. A thorough review of models, evaluation metrics, and datasets on image captioning. IET Image Processing, 16(2): 311-332, 2022.
[51] Shuai Ma, Ying Lei, Xinru Wang, Chengbo Zheng, Chuhan Shi, Ming Yin, and Xiaojuan Ma. Who should trust: Ai or myself? leveraging human and ai correctness likelihood to promote appropriate trust in ai-assisted decision- making. arXiv preprint wrXiv:230.00809, 2023.
[52] David Madras, Toni Pitassi, and Richard Zemel. Predict responsibly: Improving faimess and accuracy by learning defer. In Advances in Neural Information Processing Systems, pages 6150-61 6160, 2018.
[53] Zainy M Malakan, Ghulam Mubashar Hassan, and Ajmal Mian. Vision transformer based model for describing a set of images as a story. In Al 2022: Advances in Artificial Intelligence: 35th Australasian Joint Conference, Al 2022, Perth, WA, Australia, December 5-8. 2022, Proceedings, pages 15-28. Springer, 2022.
[54] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint arXiv: 180%02280,,,279, 2018.
[55] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchin- son, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the conference faimesss accountability, and transparency, pages 220-229,
2019.
[56] Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning, pages 7076-7087. PMLR, 2020.
[57] Hussein Mozannar, Arvind Satyanarayan, and David Sontag. Teaching humans when to defer to classifier via exemplars. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 5323-5331, 2022.
[58] Hussein Mozannar, Hunter Lang, Dennis Wei, Prasanna Sattigeri, Subhro Das, and David Sontag. Who should predict? exact algorithms for learning to defer to humans. In International Conference Artificial Intelligence and Statistics, pages 10520-10545. PMLR, 2023.
[59] OpenAI. Chatgpt: Introducing chatgpt. http:p/:/ooeeaaa..ccoo//oo/aaaaa
2022.
[60] Lucy Portnoff, Erin Gustafson, Joseph Rollinson, and Klinton Bicknell. Methods for language learning assessment at scale: Duolingo case study. International Educational Data Mining Society, 2021.
[61] Christopher Potts, Zhengxuan Wu, Atticus Geiger, and Douwe Kiela. Dynasent: A dynamic benchmark for sentiment analysis. arXiv preprint arXiv:20/2./5345 2020.
[62] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages PMLR, 2021.
14

---

[63] Nazneen Rajani, Weixin Liang, Lingjao Chen, Meg Mitchell, and James Zou, Seal: Interactive tool for systematic error analysis and labeling. arXiv preprint arXiv:2210.05839, 2022.
[64] Nils Reimers and Iryna Gurevych. bert:: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL wwtphht/t///atee org/ abs/ 10084.
[65] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[66] Marco Tulio Ribeiro and Scott Lundherg. Adaptive testing and debugging of nip models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 3253-3267, 2022.
[67] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144, 2016.
[68] Jennifer Richler and Thomas Palmeri. Visual category learning. Wiley Interdisciplinary Reviews: Cognitive Science, 5(1):55 94, 2014.
[69] Jorge M Santos and Mark Embrechts. for evaluating supervised classification. In Artificial Neural Networks-ICANN 2009: 19th International Conference, Limassol, Cyprus, September 14-17, 2009, Proceedings, Part II 19, pages 175-184. Springer, 2009.
On the use of the adjusted rand index as a metric
[70] Alison Smith-Renner, Ron Fan, Melissa Birchfield, Tongshuang Wu, Jordan Boyd-Graber Daniel S Weld, and Leah Findlater. No explainability without accountability: An empirical study of explanations and feedback in interactive ml. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1-13, 2020.
[71] Fabian Sperrle, Mennatallah El-Assady, Grace Guo, Rita Borgo, D Horng Chau, Alex Endert, and Daniel Keim. A survey of human-centered evaluations in human-centered machine learning. In Computer Gruphics Forum, volume 40, pages 543-568. Wiley Online Library, 2021.
[72] Harini Suresh, Natalie Lao, and Ilaria Liccardi. Misplaced trust: Measuring the interference of machine learning in human decision-making arXiv preprint arXiv:2005. 10960, 2020.
[73] Harini Suresh, Kathleen M Lewis, John V Guttag, and Arvind Satyanarayan. Intuitively assessing mi model reliability through example-based explanations and editing model inputs. arXiv preprini arXiv: 202.::24002.844000 2021.
[74] Shengbang Tong, Erik Jones, and Jacob Steinhardt. Mass-producing failures of multimodal systems with language models. arXiv preprini arXiv:2306.. /2105, 2023.
[75] Helena Vasconcelos, Matthew Jörke, Madeleine Grunde-Mel McLauhhiin, Tobias Gerstenberg. Michael S Bernstein, and Ranjay Krishna. Explanations can reduce overreliance on ai systems during decision-making Proceedings of the ACM on Hluman-Compunn Interaction, 7(CCCW1):
1 1-38, 2023.
[76] Vijay Vasudevan, Benjamin Caine, Raphael Gontijo-Lopes, Sara Fridovich- Keil, and Rebecca Roelofs. When does dough become a bagel? analyzing the remaining mistakes on imagenet. arXiv preprint arXiv:2205.045900 2022.
[77] Kailas Vodrahalli, Tobias Gerstenberg, and James Zou. Uncalibrated models can improve human-ai collaboration. arXiv preprint arXiv:2202.0500 05983, 2022.
[78] Jiuniu Wang, Wenjia Xu, Qingzhong Wang, and Antoni B Chan. On distinctive image cap- tioning via comparing and reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
15

---

[79] Qingzhong Wang, Jiuniu Wang, Antoni B Chan, Siyu Huang, Haoyi Xiong. Xingjian Li, and Dejing Dou. Neighbours matter: Image captioning with similar images. In BMVC, 2020.
[80] Olivia Wiles, Isabela Albuquerque, and Sven Gowal. Discovering bugs vision models using off-the-shelf image generation and captioning. arXiv preprini arXiv:2208. 08831, 2022.
[81] Jennifer Wortman Vaughan and Hanna Wallach. A human-centered agenda for intelligible chine learning. This is a draft version of a chapter in a book to be published in the 2020 21 time- frame., May 2021. URL httppps::www...nccoosottotoooo
wtt.auurrreeaarruuuuaiaecee a- human centered -eneared-agedda- for wow.ieeelllggileeeeeeeeeeee mww.iiiiieeemmmieeeeeeee -eaeeeig/aannnn.n
mi www.:/ott.ooooooooomommomoooo... wwt.:/t..tc..oeeeoooeeeeeeeeeeee
[82] Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel S Weld. Errudite: Scalable, reproducible, and testable error analysis. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 747-763, 2019.
[83] Huazhe Xu, Yang Gao, Fisher Yu, and Trevor Darrell. End-to-end learning of driving models from large-scale video datasets. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2174-2182, 2017.
[84] Ming Yin, Jennifer Wortman Vaughan, and Hanna Wallach. Understanding the effect of accuracy on trust in machine learning models. In Proceedings of the 2019 chi conference on human factors in computing systems, pages 1-12, 2019.
[85] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangehen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 2020.
[86] Peter Zhang. Taking advice from chatgpt. arXiv preprint arXiv:2305. / 2023.
[87] Yuhui Zhang, Jeff Z HaoChen, Shih-Cheng Huang, Kuan-Chich Wang, James Zou, and Serena Yeung. Drml: Diagnosing and rectifying vision models using language. In NeurIPS 2022 Workshop Distribution Shifts: Connecting Methods and Applications.
[88] Yunfeng Zhang, Vera Liao, and Rachel KE Bellamy. Effect of confidence and explanation on accuracy and trust calibration ai-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 295-305, 2020.
[89] Ruiqi Zhong, Charlic Snell, Dan Klein, and Jacob Steinhardt. Describing differences between text distributions with natural language. In International Conference on Machine Learning, pages 27099-2777 PMLR, 2022.
16

---

Partl I
Appendix
Table of Contents
A Extended Related Work
17
B Region Finding Algorithm Details
18
Region Description Algorithm Details
20
Ouboarding and Recommendations Promote Rules Details
21
E
Method Evaluation Details
22
F
User Studies Details
24
F.1
BDD Study
24
F2
MMLJU Study
25
E3
Screenshots of User Study Interface for BDD
26
F4
Screenshots of User Study Interface for MMLU
32
A Extended Related Work
Reference [47] proposes method human-AI collaboration via conditional delegation rules that the human can write down. Our framework enables the automated learning of such conditional delegation rules for more general forms of data that can also depend on the Al output. [77] proposes to modify the confidence displayed by the AI model to appropriately encourage and discourage reliance on the AI model. However, this technique deliberately mislcads the human on the Al model ability, methodology incorporates similar ideas by learning the human prior function of reliance on the Al and then improving on with the learned integration recommendations, however, we display these recommendations in separate dashboard without modifying the Al model output. A related approach to our methodology by [51] is to adaptively display or hide the AI model prediction and display the estimated confidence level of the human and the Al a task of predicting whether a person's 's income exceeds a certain level. They show that displaying the confidence and the AI the human improves performance. Our method is able learn the confidence level of the human and the AI, but also incorporates how the human utilizes the AI and describes the regions where AI vs human performance is different. [16] presents similar approach our AI recommendations, however, they use simulated and faked AI models and descriptions of behavior while we are able to obtain automated generation of these descriptions of AI behavior.
the human
Existing research has examined various methods to establish human trust in and replicate the pre- dictions of machine learning models. One such method LIME, a black-box feature importance technique, which was employed to select examples for evaluation by crowdworkers to determine the superior model among options [67, 45). However, their selection strategy disregards the human predictor, and their approach merely presents the examples without further action. In the context of visual question answering, Chandrasekaran educate crowdworkers about the AI's capabilities, leading to an enhanced ability to identify instances where the AI failed. Feng et al. [25], in the domain of Quizbowl question answering, emphasize the significance of incorporating the human expert's 's skill level when designing explanations. This further justifies our decision involve the human predictor in the selection of teaching examples. Cai et al. [17] conducted study involving 21 pathologists to gather guidelines on what clinicians desired to know about an Al system prior to interacting with it. Yin et al. [84] investigated the impact of initial debriefing stated AI accuracy versus observed AI accuracy during deployment,
[18] manually selected seven examples to
finding substantial influence of stated accuracy on trust that diminishes quickly once the model
is observed in practical use. This reinforces our approach of building trust through examples that
17

---

simulate real-worid deployment. Bansal et al. [5] examined the role of the human's 's mental model of the AI in task accuracy; however, the mental model was developed through interaction during testing rather than during an initial onboarding stage. The most similar work to ours is that of [57] which presents an onboarding scheme based on selecting a set of examples and allows the human to describe the regions where the Al performance is good or bad. Through user study on passage-based question answering, they show that their onboarding scheme improves performance by 5%, however, they evaluate without the presence of AI evaluations, with synthetic AI model and their scheme involves more involvement from the human as they have to describe the regions themselves. Another approach to teaching involves providing humans with guidelines [3]. Model cards [55] and industry practices such as the IBM Al fact sheet [4] demonstrate direct methods of presenting these guidelines to users, we present humans with a similar form of card but that includes aspects of human performance "human-Al Al card".
when to rely on AI systems
There is growing and large area of literature on discovering (and auditing) regions of Al error, the following is not meant as an extensive list of related work but captures some of the essence of the literature:
Adatest allows user to iteratively discover regions of Al error using LLMs for NLP tasks and then re-train the model on the regions of error [66], it was then extended to vision tasks [27] with a similar procedure in [80].
Erudite allows users to discover regions of error NLP models through user interfaces [82], there is a wider literature dashboards for discovering regions of error [2]
[37] learns an SVM model from image embeddings to predict model error and then uncover regions of error based on the directions of the SVM model.
Works have done extensive manual annotation of ImageNet model mistakes|35, 76].
DOMINO discovers regions of model error using slice discovery model based on specialized gaussian mixture model [24], extensions include DRML [87].
The Spotlight method learns individual regions based on a neighborhood of a learned point [23], our region finding algorithms generalize their procedure by learning weighted distance distance measures and with a different aim of improving gain of the prior.
SEAL: Interactive Tool for Systematic Error Analysis and Labeling, uses k-means uncover regions of eтror and then uses an LLM to describe each region [63].
Recent work has emerged on describing sets of images [34, 53] but they don't incorporate a contrastive method as we propose. Helpful tools for describing differences between text and images can be useful for describing regions which future work can incorporate [89, 79, 78].
One of the objectives of explainable machine learning is to enhance humans' ability to assess the accuracy of Al predictions by offering supporting evidence [44, 43, 70, 32, 88, 42, 72, 73, 81, 31]. Nevertheless, these explanations fail to provide guidance to decision makers on how to balance their own predictions against those of the Al or how to integrate the Al's evidence into their final decision [39]. [75] shows that AI explanations can reduce overreliance and improve Auman-AI team performance, however, their experiments are with simulated AI models and explanations. The central question of our work is not to study the utility of AI explanations, in fact, all our user studies incorporate Al explanations and we aim to improve Auman-Al performance in their presence.
B Region Finding Algorithm Details
Regions Requirements. straints:
Each region in our algorithm should aim satisfy the following con-
1. Region Size: We want the size of the region to be at least of size 3, and at most of size Bu-
2. Consistency of takeaway: The examples in each region must agree on what the takeaway is in terms of the integration decision. Specifically, at least a% of all points in the region must either be: ignore AI, use AI as is or integrate Al advice.
3. Concise and Distinguishable Theme: Each region must be concisely described nat- ural language in such a way to differentiate from the overall domain. If region cannot
18

---

be described in natural language, the human may not be able to derive a generalizable recommendation from it. This constraint that we implicitly try to satisfy by learning neighborhoods in a natural language embedding space.
4. Minimum Gain: Each region must have minimum information gain (defined below) of 8. This ensure that all regions contain sufficient novel information to the human.
The optimization find each region can be formulated in its non-relaxed form as:
[1(((es,a))==)lll Di.r
67,,05
S.t.
[[(((e,a)) -rr,rr 2 on
1=1
s.t. nB
[u((e).a))))]k nBu
And the relaxation we propose is (refer back to Section 4.1):
maxemwar E (C1(-11l ((e,, a.)
-) + 9ir Amax(_") (Ci(- ((en,;) - c)ll + +2))
(Ir;=r- - an),0) - À max Σ
(Ci(-1l ((e,, c)ll +7)) Bun 0
À Amax
( ((es, 0:) -c)ll 7))
7)) + Ben, 0
We run the optimization for = 0 and r= and choose the r with the better objective value. With r fixed, optimize with respect the remaining continuous parameters using AdamW and reduce the learning rate when loss has stopped improving. We initialize with 7 and w = 1. For the centroid c, we first run k-medoids clustering on the input data with & min (max(100. T) T), and randomly select 20 of the resulting centroids. Then with e initialized as each one of the 20 centroids in turn, run the optimization for 200 epochs and record the loss, and finally optimize for 2000 epochs with the best initialization for c. We do these repeated initializations to avoid local minima which are common failure mode of this type of optimization problem. Note this process is find one region, to find all regions we repeat this process identically to find regions one by one.
Selection Based Approach. We described in the body of the paper a generative algorithm find the regions. We now describe a selection based algorithm that finds centroids from points in the dataset D = (ei, We will also restrict the radius to be the distance between the centroid to another data point in D. We proceed with sequential search, at round we perform the following search:
Co 7/ =arg max G(N,,, R", RNI.-)); iEDiY
(5)
s.t. 3k E [n] s.t. d(e;,
(6)
Eyenndde..)) Ir,= and [ij E d(e,d(e;))) ex) <7> 2
(7)
and 81 S Ii [m), d(e;, es) < <>> $ B
(8)
and G(N,,, Re, RN >
(9)
Note that with the selection bastod procedure, we have to define a fixed distance measure d, and we cannot optimize over the deferral decision of the region
inherit from the point found. The naive algorithm to solve the above search is as follows. We first compute the distance matrix between all data points in D. call this matrix K. We then at each round do the following for each point in DD: sort the points by their distance to iteratively grow the region around the point i to satisfy the
19

---

constraints, and then keep track of the maximum gain radius. Each time we grow the region by one point, we check if the constraints are satisfied. The search is parallelized across multiple instances to make it faster. Finally, we compare the gain of all feasible points and pick the highest one. This algorithm extends the approach of [57] to incorporate additional constraints and has significant speed ups over their approach.
Aggregating Regions Across Different Embedding Spaces We can run our region finding algorithm above and find different regions across multiple embedding spaces. The question is how do we aggregate these regions. Say found a set of T regions ... .N. We then run the following meta-algorithm: do greedy sequential search to add regions one at a time while making sure the minimum gain requirement satisfied, stop when is not.
C Region Description Algorithm Details
The LLM call inside Algorithm is accomplished by building the prompt with S being the inside set (positives) and S- being the outside (negatives) as follows:
def get proppro (positives, negatives) prompt pre_instruction "\n" prompt "inside the region: \n counter
for P in positives:
prompt += p[0] " In"
counter +=
len (negatiives)
prompt
An not the region: \n"
counter =
for P in negatives:
prompt +- p[0] ", counter
prompt + post_instruction
return prompt
For the experiments in Section we use the following pre instruction:
I will provide you with set of descriptions of points that belong to region and a set of descriptions point that do not belong to the region. Your task is summarize the points inside the region in concise and precise short sentence while making sure the summary contrasts to points outside the region. Your one sentence summary should be able allow person to distinguish between points inside and outside the region while describing the region well. The summary should not be single word, it should be accurate, concise, distinguishing. and precise. Example:
Inside the region:
two cows and two sheep grazing in a pasture.
the sheep is standing near a tree.
Not in the region:
the cows are lying on the grass beside the water.
summary: sheep.
End of Example
While the post-instruction is simply "summary:".
For the ablation without contrasting, the pre instruction we use is:
will provide you with set of descriptions of points that belong to region. Your task is to summarize the points inside the region in a concise and precise short sen- tence. Your one sentence summary should be able to allow person to distinguish points inside the region while describing the region well. The summary should be a single word, it should be accurate, concise, distinguishing and precise.
20

---

Example:
inside the region: two cows and two sheep grazing in pasture. -the sheep standing near summary: sheep.
End of Example
For the user studies, we had used earlier instruction that works slightly worse:
"summarize the points inside the region in a concise and precise short sentence while making sure the summary contrasts points outside the region"
We recommend the use of the following pre instruction:
will provide you with set of descriptions of points that belong region and a set of descriptions of points that do not belong to the region. Your task is to summarize the points inside the region in a concise and precise short sentence while making sure the summary contrasts to points outside the region. Your one sentence summary should be able to allow person distinguish between points inside and outside the region while describing the region well. The summary should be no more than 20 words, it should be accurate, concise, distinguishing and precise. Example:
inside the region:
two cows and two sheep grazing in a pasture.
the sheep is standing near a tree.
outside the region:
the cows are lying on the grass beside the water.
summary: The region consists of descriptions that have sheep them outside nature, could have cows but must have sheep.
End of Example
D Onboarding and Recommendations to Promote Rules Details
Introductory Phase. To facilitate a smooth onboarding process for individuals working with an AI assistant, we introduce the Human-AI Card. This card provides detailed insights into the Al's capabilities, training, and performance. Below is presentation of the core details of the card inspired by the work in [17]:
Table 6: Human-Al-Cann presented to the human as part of onboarding
Information
Description
AI Input
What the Al uses make its prediction
AI Output
What the Al provides as output (predictions, explanations,
Source Training Data for AI Source of Pre- Pre-Training Data of Al Training Objective of
Description data used to train AI
Description of pre-training data of model Al is based on what the Al trying to achieve (minimize classification error, detect objects, next word prediction)
Average AI Performance Average Human Performance
Additionally, provide breakdown of AI and Human performance different subgroups of data.
We take an example of the Berkeley Deep Driving dataset, where subgroup might comprise images taken during the night, in rainy weather, on highway. We compute the model's 's enror for each possible subgroup and then perform a paired t-test comparing the subgroup model error to the average model error over the entire data. For the purpose of our user studies, highlight subgroups
21

---

defined by single metadata category that show statistically significant differences (p I 0.05). It's important note that, for rigorous analysis, one should apply corrections for multiple hypothesis testing. However, considering the vast number of metadata categories, many results might become insignificant. Therefore, for simplicity, we adopt this heuristic approach.
E Method Evaluation Details
In Table 7 report the details the datasets datasets using loo normalization and run our algorithms for 2000 epochs with a learning rate 0.001 using AdamW. We constant C1 20, 5 3r, 0.1 (Bu, 0.5 for Aim 1), 0.5 and 31 = 0.01 for all experiments. We use a random prior Ru that has equal probability for and uniformly random. All experiments
in method evaluation. We normalize all
GeForce GTX 1080 Ti.
Table 7: Datasets for "Learning Accurate Integrators Aim 1)". We note the total number of samples n, the target set size ()), the human expert finally the model of the AL When we note human "XX% accurate", this indicates synthetic human model that accurate uniformly at random with probability XX%. For DynaSent, the Al model a pre-trained sentiment analysis roBERTa base model [8] Twitter data and achieves 75% accuracy. For both BDD and MS-COCO blur the images using a Guassian Blur with scale 21 and variance
Dataset
n
|
Human
AI
Berkeley
Deep (BDD)
10k
80% accurate
faster renn r50 fpn Ix Gaussian blur with scale 21 and
Drive
[83, 85]
variance 5
MS-COCO [48]
5k
(pres- ence of person in image)
70% accurate
faster renn R 50 FPN
Massive Multi-task
14k
(MCQ)
50% accurate
flan-t5xl [20]
Language Under-
standing (MMLU)
[33]
Dynamic Sentiment
6.5k
lcave-one-out annotator
a pre-trained sen- timent analysis roBERTa-base model [8]
Analysis Dataset (DynaSent) [61]
22

---

IntegrAl (ours)
0.30
DOMINO
K-MEANS
DoubleGreedy
0.28
Human
Test Loss
0.26
0.24
10
12
14
16
18
20
Number of Regions
Figure Test Error (4) of the human-Al system when following the decisions of the different integrators baselines as we vary the number of regions maximally allowed for each integrator on the MS-COCO dataset.
0.51
0.50
Test Loss
0.49
IntegrAI (ours)
DOMINO
0.48
K-MEANS
DoubleGreedy
Human
0.47
Al
0.46
0.45
14
16
18
20
Number of Regions
Figure 5: Test Error (L) of the human-Al system when following the decisions of the different integrators baselines as we vary the number of regions maximally allowed for each integrator on the MMLU dataset.
23

---

0.204
0.202
Test Loss
0.200
0.198
IntégrAl (ours)
DOMINO
K-MEANS
0.196
DoubleGreedy
Human
10
12
14
16
18
20
Number of Regions
Figure 6: Test Error (L) of the human-Al system when following the decisions of the different integrators baselines as we vary the number of regions maximally allowed for each integrator on the Dyanasent dataset.
For Aim 2, we create synthetic AI and human models follows:
BDD: AI and Human model each have non-overlaping two to four regions defined as the intersection of two metadata categories, each region either a good region where the Al/Human have 95% accuracy or bad region with 60% accuracy (equal number of good and bad regions). an example does not belong to any region the Al/Human have 75% accuracy. Each region is of size at least 0.01 and at most 0.2 in terms of fraction of data points in the dataset. An example region is "AI is good at: weather: clear and scene: highway" or "Human is bad at: weather: snowy and timeofday: night" ".
MMLU: same setup BDD, but there is only metadata category in the dataset, each region belongs to certain metadata category. An example region is "A is good at: subject: professional psychology".
MS-COCO; same setup as BDD, An example region is "'Human is bad at: cow: present".
For Aim an obstacle quantitative results for region descriptions is that they are often complex even when regions are synthetically defined in terms of metadata and captioning metrics are not informative. For region defined on BDD as images of "scenes of highways during the night with no cars", our algorithm finds the description "highway during the night with various weather conditions and congested with many cars" while the SEAL [63] describes as "Highway Nighttime Weather Conditions". The SEAL description surprisingly has higher captioning metric scores (BLEU, METEOR) [50] while being less informative.
For the MS-COCO evaluation, we only select objects that have at least 50 examples present in the evaluation set which leads to only 73 objects over which we evaluate the different region description algorithms.
F
User Studies Details
F.1
BDD Study
Task.
The images from BDD are blurred using Gaussian Blur with a scale of 21 and a variance of The model is trained faster renn model that achieves 84% accuracy without blurring which decreases to 78% accuracy on the test set. We use the bounding boxes from the model and output them on the image (allowing the user either hide or show them). To get confidence score, we take the maximum score for the prediction of traffic light in the image.
24

---

Initial Data Collection. The BDD dataset was split 70-30 where the 70% split was used to get the initial human predictions and find the regions and the 30% split was used only to get testing examples for the final user study. As mentioned, we obtained data on 400 examples with both human predictions and prior Al-integration decisions. We use these examples to build Random Forrest models that predict both the human predictions and A Al-integration decisions from the embeddings, labels, and Al predictions (AI predictions only for predicting Al integration decisions). We use these predictions from the RF models to label the entire dataset. We ensure that the predictions of the RF models are calibrated (if the human is 80% accurate, the model is also 80% accurate) with the human predictions and integration decisions by modifying the threshold predictions (e.g. from the usual 0.5 threshold to the value that makes the models calibrated). Each participant is evaluated on randomized set of examples, we create 40 different sets of 20 images that get assigned randomly to each participant.
model probability used to make
Attention Checks. In each condition, insert attention checks for every 20 examples where the images are unblurred and we keep the Al prediction. We only retain responses where participants don't ton'tgn get all attention checks wrong, The class balance of the dataset is close 51 %-49%. The attention checks are used as part of the study results as they only modify the blur of the images.
Regions Found. The regions found by our procedure are shown in Table
Table 8: Region descriptions found by IntegrAI for the BDD user study.
Region ID
Description
1
depict various city streets and highways during the daytime with different weather conditions, containing pedestrians, cars, trucks, traffic lights, and signs., depict various types of roads and streets during the daytime with different weather conditions, containing cars, pedestrians, traffic lights and signs, while the outside examples include scenes with fewer objects or in less common locations such as a parking lot.,
2
3
depict various scenarios of streets and highways with moderate to heavy traffic flow during the day or night, with different weather conditions, along with traffic signs and lights, cars, trucks, buses, and pedestrians. depict various outdoor scenes during the daytime or nighttime, containing multi- plc cars, traffic lights, and traffic signs depict various city and residential scenes during the daytime with different weather conditions and contain variety of vehicles, pedestrians, traffic lights, and signs, while the outside examples depict specific limited scenarios with fewer elements.,
4
5
6
depict various traffic scenes, mostly highways at night and during clear weather, with mix of cars, trucks, buses, traffic lights, and traffic signs depict various city streets and highways with clear weather and a moderate amount of traffic, including cars, signs, and occasionally pedestrians, bicycles, and trucks
7
8
depict various urban and residential scenes during different times of day and weather conditions, with a diverse range of vehicles, pedestrians, traffic signs, and traffic lights present
9
depict various scenes of city streets and highways with typical traffic conditions and without severe weather conditions depict various scenes of city streets and highways with varying weather condi- tions, traffic, and signage.
10
F22 MMLU Study
Task. We rely on the MMLU dataset [33] where participants are shown question, four possible answers and have to pick the best answer (see screenshots Figure below for examples). We use ChatGPT, also known as GPT 3.5 turbo as our Al model [59]. We obtain the predictions of ChatGPT
25

---

on the MMLU dataset following the approach in the official repo of MMLU We also ask ChatGPT explain it's answer by using the prompt "Please explain your answer in one sentence". Both the Al answer and the explanation are shown.
ChatGPT obtains accuracy of 69% during evaluation and restrict attention specific subjects within the MMLU dataset. Specifically, we sample subjects (out of the 57 in MMLU) where ChatGPT has significantly better perform performance than average, 5 where it's significantly worse, and subjects where performance is similar to average performance. These subjects are listed here:
high school government and politics, marketing, high school psychology. logical fallacies, sociology. public relations, high school computer science, anatomy, business ethics, elementary mathematics, high school statistics, machine learning, moral scenarios, global facts
We sample 150 questions from each subject and additionally sample 150 questions from OpenBookQA dataset [54] to use as attention checks as human performance on OpenBookQA 91%.
We run IntegrAI on both the dataset embeddings, metadata (subject name), and an embedding of ChatGPT explanations separately. We find 10 regions based on the metadata and 2 regions based on the ChatGPT explanations. The regions and their descriptions found by our algorithm are reported in Table
Table 9: Region descriptions found by IntegrAI the MMLU user study.
Region ID
Description
Related to marketing, including pricing strategies, branding, communication, product classification, market segmentation, advertising, and supply chain man- agement.
2
Questions related to psychology and neuroscience.
3
Questions related to global statistics and trends, ranging from military spending to mental health disorders.
4
Questions inside the region cover a variety of topics in the field of public rela- tions, including ethical frameworks, evaluation models, common tactics, and regulations.
5
Mathematical and quantitative questions, involving calculations and problem- solving.
Questions related to sociology, including topics such as social class, symbolic interactionism, bureaucracy, and globalization.
7
Questions and descriptions of logical fallacies and syllogisms. Questions focus on US politics, government, and history. Contains questions related to anatomy and physiology. Various topics including ethics, regulation, consumer rights, and corporate trans- parency.
8
9
10
11
Various questions on diffcrent lopics such as algebra, biology, public relations, and statistics with multiple options to choose from.
F33
Screenshots of User Study Interface for BDD
26

---

Classify Images of Road Scenarios
You wnin bof partidiens a reearch dtudy how people daadly inaget You proeettd with arle iaget and have redeeenen "ou may add Sorallyy acid intoadt with mode and wito until son of llad.
This staly will ackile part diants bver 13 yoant App who fodl contortalle using an onlir into read and reor quartic Englth lnved Image:.
yourur dightt stady Find onl
dering, imagee detwore
tnglis.
"ou hav non alrend conpleted this suree. Fhr ane puiss aus
loudergs wast
What Will My Participation Involve?
Are There Any Risks Me?
We raka patidpatio
than norna actltty
Are There Any Benefits To Me?
Thene no ber toy other tonttr compersation
How Will My Confidentiality Be Protected?
Figure Consent Form
Welcome to our experiment (please read carefully)
you previcusly completed study, suto incligible complete again!
Figure 8: User Information Collection
Instructions
This experiment will be in 3 stages.
shown images any light green or yellow)
scenarios point
car, you will be to respond to check whether image
goal
you by choosing the corrasponding button "Yes" "No' and than pressing 'Submit" button.
Interface: you will have first select
You track many examples remain looking the progress the of the webpage after you next. You will keep trying each example until you get the answer correct
Example (no need to click, only for viewing):
O
there a traffic light in the picture?
Figure 9: Practice Task instructions
27

---

Practice Examples
Is there a traffic light in the picture?
Yes
No
Submit
Figure 10: Prediction without Al interface
Instructions (scroll down for task)
Al MODEL: Images are now blurred and you have access to an Artificial Intelligence (Al) model that provides their prediction
Al Button: A new button now appears called "Use Al Answer" in blue and allows you to select the Al's answer which you can then submit.
Please press the button the Al helped with your ansoer and you want to use it's anower, if you agren with the answer of the but got the
answer without it's help then don't press the button.
If predicts there to be traffic light, will put a box around in the image and score (you can hide or show it with button below the
image)
Teaching Phase: The next examples help teach you more about the through examples where you will receive feeolback, that, there applies.
testing phase where
Figure 11: Instructions for the onboarding phase
give you following information about the Al model help you understand when you should and when
Average Accuracy:
Al Model lapart
very Inage
Outpott
Peed clsn light bounding bos imae showing th's locaton and indicad to coniidee
Al Training Objeetive
Drtec TuTk Lights and other oljects
Now deeper dive into the Al error/accuracy:
more tias mrtffic lights image
60%
4o lights
No Cars
65%
Osytima overcnt wenther
Figure 12: Model card information shown during onboarding.
28

---

Hide answer
Show Instructions
Is there traffic light in the picture?
Yes
Al Predicts: NO with score: 0.0
Use Al Answer
No
Submit
Figure 13: Prediction with AI interface (AI predicts no traffic light)
Predicts: NO with
X
Wrong: You predicted incorrectly, the correct answer was yes. Incorrect: The Al predicted incorrectly.
wio mur plo enladd an fevu dlpati uuseen loalices calive
Asourio 61d
Figure 14: Feedback shown during onboarding phase after human predicts.
Figure 15: Feedback shown during onboarding phase after human predicts (sets of examples from region)
29

---

Hide Al answer Show Instructions
Is there a traffic light in the picture?
Yes
Al Predicts: YES with score: 0,92
Use Al Answer
No
Submit
Figure 16: Prediction with Al interface (Al predicts there a traffic light)
Hide answer
Show Instructions
Predicts: YES score: 0.65
Correct: You predicted correctly!, the correct answer was yes. was Correct: The Al predicted correctly.
Recommandettor Bay Arawer
information Acouery 910
Figure 17: Feedback shown during onboarding phase after human predicts (correct feedback)
Instructoon
task)
Testing
MODEL; Images blurred and you have examples Artificial Intelligence (Al) model that provides their prediction
model.
Button: button appears called "Use Al Answer" blue and allows you select the Al's answer which you then submit.
Please press the button
helped with your answer and you want answer, if you agree with the answer of the got
without it's help then don't press the button.
f the All predicts there to be traffic light, will box around the image and score can hide show with button below the
image)
Try to do your best get things correctly as there is potential bonus of to 25.
Please read instructions secs
Figure 18: Testing phase instructions
Instructioss
tesk)
MODEL: images blurred you
Artificial Intelligence (Al)
provides their prediction
Al Button:
appears called "Use Al Answer" blue and allows you select the Al's which you then submit.
Please press the button the Al helped with answer and you want it's answer, you agree with the answer the Al but got without it's help then don't press the button.
If the Al predicts there to be a traffic light, will put a box around it in the image and score (you can hide or show it with button below the image)
Our Recommendation: Additionally, above the button and the A''s answer, we give you a recommendation whether you should use the A answer on your own. This recommendation based on comparing your estimated personal ability compared to the Al's ability, this recommendation not always correct.
Try do your get things correctly
potential
Please read instructions (14 secs
Figure 19; Testing phase instructions that include Al-integration recommendation
30

---

Hide Al answer
Show Instructions
Is there a traffic light in the picture?
Recommerdiion Rely Al Ansiver
Ragion depktt walous city streets and ghtwys during daytine with different wouther conditions, contaiving padectiane, car.
Yes
Al Predicts: NO with score: 0,0
Use Al Answer
No
Submit
Figure 20: Prediction interface with Al and with Al-integration recommendations.
31

---

F4 Screenshots of User Study Interface for MMLU
We give you the following information about the Al model to help you understand when you should use it's answer and when not to:
Average Maman Accuracy
puster betwren people aitinct))
Medel Impat:
Cussion and the arswor
Oulput
axpisnation
Source Training Dute
Tained on all the insame
Training oijoctiios
Traludd segrage inttiirrio
Now deeper dive into the error/accuracy:
Category:
Accuracy
Managemant(Marraaiigg
00%6
Elementtary Math
67%
Sycdalegy/Soddeegy
dlebal Farts
47%
Logical Fallaciss
70%6
High School Statiettes
Figure 21: Model Card for MMLU study
company's
sutubillto,
65% and expected finandel retum
At SWCT anelyst
Poltticll
Pottfolo analyis.
De Compaitivve wnrlpii.
Al Predicts:
Answer: Portfolia andlysis. It involves evalusting company's product portfolio to determine its strengths and weaknesses, well expected
return. allows the company to allocate its resources more effectively and efficiently.
suitability,
Use Al Answer
D
Submit
Figure 22: Prediction Interface for MMLU study
32

---

Al Predicts: C
Wrong: You predicted incorrectly, the correct answer was C.
Correct: The Al predicted correctly.
This exarple is part largr set carrets with killwing recommendaainn
Recommendation: Rely Al Answer
related to marketing. including pricing stratogies, branding, communication, product classification, market segmentation advertising, and supply chain management.
Region Information:
Al Accuracy; 92.0
Human Accuracy: 40.0
are more exmmlee from this regio you generaize from
prccng!
feeing
Bosing poychological expectations
bexing the prize festume benallss the produst
Tasing the price full production oik the mouil red piott
Figure 23: Feedback shown to user during teaching phase
Show Instructions
Queestion: a inheeits a phynica tmit father Biely receive
tooth
foertts color
uesse fun D: lking doge
Recommendation:
Answer
Al often wrong when various questions different topics algebra, biology, public relations, and statistics with multiple options to choose from.
Predicts:
The answer A. toath gap because physical traits, such the primarily deternined by genetics passed down from parents children.
Use Al Answer
D
Submit
Figure 24: Prediction Interface for MMLU study with Al-integration recommendations.
33


Reviews:
[{"review_id": "sVRbIW7Qs7", "reviewer": "Reviewer_doL5", "summary": "The authors attempt to improve human-AI collaboration through a system that (1,2) identifies similar regions in a dataset (both in terms of the task similarity and in terms of human behavior—usage of AI advice—on the task) and optimizes for how humans should ideally use the AI in the given region, (3) generates contrastive descriptions for the region, and (4) provides recommendations to users based on 2. They synthetically evaluate each component of their system, and empirically test their system with human users across two tasks.", "questions": "1. How important is the embedding choice for the region discovery algorithm?\n2. There seem to be a lot of hyperparameters here. How important are the choices here? For example, the upper and lower bound on fraction of points in a region seem to be an important consideration — were there any experiments to assess what is reasonable for (a) human users and (b) for the generated region description? How many regions is reasonable for onboarding?\n3. For the region parametrization, are the CLIP embeddings normalized?\n4. For region description, (1) the initialization and (2) the embedding could matter a lot. Did you do any analysis of the variation in the final description with respect to these?\n5. What do you use to generate the initial text descriptions for each image? In 4.2 you reference a few possible methods, but don’t indicate which you use.\n6. In Table 1, I am unclear on what the numbers represent. How is the error calculated?\n7. For the empirical studies, is the task order randomized? Also, for the people who did “predicts alone” and “predicts with help of AI”, were the tasks where they predicted alone randomized person to person?", "limitations": "As the authors recognize, it would always be nice to have more empirical studies (e.g., with more users, more datasets, etc.). The authors also recognize the potential negative outcomes of this work (incorrectly identified regions and/or recommendations could incorrectly bias users to under/over-trust the AI). ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Problem setup and optimization solution are straightforward and make sense.\n2. Empirical analysis using humans to assess their system (with multiple datasets).\n3. Generally well-written. See “Weaknesses” for a few suggestions I have to clarify some parts (section 4).\n", "weaknesses": "1. Only testing on one task\n2. The experiment design has an obvious confounder — the raters are only offered a binary choice, while the AI \n3. The setup did not allow distinguishing between ignoring the AI and collaborating with the AI. This should be possible with an adjusted setup — for example, have a 2 stage process for data collection where the person makes an initial decision and subsequently is given the chance to use AI (in section 7, it sounds like you did this, but this was not clear earlier in the paper). Additionally, rather than having people submit a discrete label, you can ask them to submit a continuous value (e.g., something like a confidence). Note that this could be done just for the dataset collection process to approximate $R^*$, and you could still use your original setup for the empirical evaluation. (Side note: if you had recorded a “confidence” for the dataset, optimizing for the regions may be easier as this is a differentiable problem now since the $r$ would be continuous-valued.)\n4. It’s actually unclear how well this setup can distinguish between 0 and  1 — using the AI vs not using the AI. Since everyone sees the AI’s output, they may be biased based on that while still making their own decision. There are a number of studies that have established the biasing effect this can have.\n5. The overall method in 4 (and 4.1, 4.2) is relatively straightforward, but it is a bit difficult to parse initially (some key points are buried in the details). I would recommend a concise description at the start of each section to describe what will be done in that section. (e.g., at the start of 4.1, saying something like “We generate an embedding for each datapoint by concatenating a task embedding with the human’s decision to use AI. We then define an objective function to jointly discover data regions and optimal usage of AI; it seeks to maximize task instance similarity while enforcing similar (optimal) human behavior (usage of the AI) on that task”). In my opinion, this high level overview will make the sections easier to read.\n6. In the appendix, it seems the empirical analysis on the second dataset results in not significant benefits to using the proposed system. This is OK, but I would recommend adding discussion about this in the main paper. It is important to understand when your system may and may not work, and where improvements may be needed. For example, it is possible the disparity in human and AI accuracy is large enough that simply trusting the AI is a better strategy than trying to assess when to trust the AI more.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your detailed review, we appreciate your time and all the important comments. \n\n## Weaknesses:\n\n1. It is true that we test our onboarding method on one user study task in the main body of the paper.  However, the purpose of the user study is to test the behavior of various humans (150 overall) when they undergo onboarding. In the setting studied, we showed that onboarding can have a positive effect on performance, we believe the conclusion would remain the same if the task was different (e.g., classify species of animals) but with the same explanations, and relative human and AI performance individually. We do agree additional tasks would be interesting to see how the effect of onboarding changes with different task parameters, this is left to future work. Most importantly note that we do evaluate the components of our method on four different datasets and different tasks within section 6.\n2. It seems the sentence of this point was cut in the middle, we are more than happy to reply to the full point if re-written during the discussion period.\n3. With this setup, we cannot (as you correctly point out) always identify which integration decision did the human take, we can only do it when the press the \"use AI answer\" button. This is something we clearly acknowledge in lines 146-150, this means we can only learn a proxy version of the human integrator. Let us clarify that our data collection in the user study consists of two parts: 1) human predicts without AI on a set of examples and 2) human predicts with AI using the same interface as the evaluation interface. While it is true that we can devise other forms of data collection schemes, notably collecting human confidence which is something we considered but did not pursue as we believed it would be too subjective and create additional biases, we were mostly interested in figuring out how to learn from feedback data collected from the same interface participants will predict on at test time. \n4. It is correct that it is hard to distinguish which integration decision the human took, the only sound way to do this is to ask the human explicitly after they predict which would be an interesting study design but not as relevant to learning accurate regions which do not require exact knowledge of the human prior.  \n5. Thank you so much for pointing out how we can better improve the presentation of section 4, we will improve the presentation in future iterations.\n6. The MMLU results had only been completed after the full paper deadline due to budget constraints, so in the main paper, we only mentioned performing the study but did not elude to the results (whether positive or negative we had planned to include them in the main paper if they had been done in time). Our hypothesis for why the MMLU onboarding results are inconclusive is that the ChatGPT explanations are very informative about the model’s uncertainty, in fact, ChatGPT frequently responds with an explanation stating “I don’t know” or “there is not enough information”. To test a weak version of this hypothesis, we ran a follow-up study conducted after the Appendix deadline with Human-AI performance without ChatGPT explanation but only the ChatGPT answer. We found the Human-AI performance drops to 70.4% (no explanation) compared to 75.1% (with explanation) (from Appendix table 9), a two-sample t-test finds p =0.045 and t=2.03.  Thus our hypothesis is that onboarding is most effective when the AI explanation do not inform the user when and when not to rely on the AI. \n\n## Questions:\n\n1. The choice of the embedding space for region discovery algorithms has been established by prior work DOMINO [17] and has a significant effect on the ability to recover regions of high error. The authors in [17] found that cross-modal embeddings work best compared to random initialization, uni-modal embeddings, and activations of the model itself. We follow the recommendations of prior literature and use the CLIP cross-modal embeddings.\n2. There are four main hyperparameters for the region-finding algorithm: $\\alpha$ (0.5 used) controls consistency of the region, $\\beta_l$ (0.01 used) and $\\beta_u$ (0.1) control the minimum and maximum size of the region and $\\delta$ (1 used)  minimum gain of the region. The two parameters that have the most impact are $\\alpha$ and $\\beta_u$ which we experimented with quite a bit. Setting $\\beta_u$ to 1 (unconstrained) produces few regions of large size and a large number of tiny regions so that the choice of 10% size leads to most often a dozen good-sized regions. For onboarding, we chose 9 so that they can fit in time within a user study.\n3. We do not normalize the CLIP embeddings.\n4. We did not find that the initialization matters a lot for region description since the method allows one to find counterexamples and quickly fix a bad initialization by adding more points.\n5. We use a different method for each dataset. For BDD, we use the metadata provided with each image to construct a textual description with a template:\n caption =   caption += scene + \" during the \" + time_of_day + \" with \" + weather + \" weather, the scene contains \" + objects_in_scene\nFor MS-COCO we use the captions provided with the dataset.\nFor MMLU we use the question itself as the description. For DynaBench we use the review as the textual description.\n6. Let us first clarify that we report in Table 1 and Table 3 (and in Aim 3) results as mean +- standard deviation. In Table 1, we report the loss (equation 1 with l(.,.) being the 0-1 loss) of the human-AI team when following the recommendations of the AI-integration function learned with the regions. \n7. In all the user study conditions, for each participant, we select a random set of examples so that each participant per each condition sees different examples. For conditions (1) and (2) described in lines 362-368, we randomize which of the two sub-conditions participants see first for each participant.\n\n"}, {"review_id": "qgfOskz5f4", "reviewer": "Reviewer_oGDF", "summary": "This paper presents an approach to allow effective human-AI teaming. The approach describes a process of semantic discovery of regions in an embedding space following by generating natural language descriptions of the regions and then learning refinement of the regions using counterexamples. Once regions and region descriptions are generation, human onboarding is performed alongwith proposals for whether to rely on AI for decision making or not with an explanation. Analysis shows that human-AI team performs better than just AI or humans. User study shows very interesting results (role of onboarding, role of providing recommendations, impact of time per task with/wo AI teaming and recommendations). ", "questions": "- There are other approaches for region discovery such as the deep aligned clustering algorithm, adaptive decision boundary algorithms etc (from the fields of open class learning, semi-supervised representation learning, novel intent discovery, etc) What do the authors think about these other approaches for region discovery. ", "limitations": "- This paper covers a lot of ground and the authors have also captured the current limitation well in the Limitations section of the paper. However, given all the limitations, the paper presents some very promising ideas and directions of further research on human AI collaboration.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper presents an interesting and practical approach for developing a human AI collaboration system with a focus on both machine learning aspects (learning AI integration functions) and the HCI aspects (onboarding + natural language descriptions) of how human-AI teams can really work in balance (staying away from over-reliance on AI and under-reliance on AI). \n- The use of chatGPT as an oracle is also a great workaround for lack of ground truth data \n- Evaluation shows that the learnt AI integration function leads to lowest loss at test time and also minimum number of region proposals (which may not overwhelm the user) \n- Groundtruth cluster analysis of regions on the BDD dataset also shows high correlation compared to other region discovery approaches. \n- the contrastive approach for region description is also very intuitive and gives good results. ", "weaknesses": "- There's a lot covered in the paper in the limited space and the focus seems to be on region discovery and region description algorithms as well as the higher level human-AI teaming and overall system level evaluation study. It seems like one could do more extensive analysis with additional experiments with more datasets on each of the sections. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your review, we appreciate your time to review our paper and the important considerations you raise.\n\nWeaknesses:\n\nWe agree that we can add more experiments on more datasets. We do plan to add more ablation experiments and on more datasets in the appendix (on ImageNet16H, CIFAR-10H and a synthetic gaussian setup) notably. \n\nQuestions:\n\nThank you so much for pointing out two related works that can be applied for region discovery [1,2] that we will add a comparison to and cite in future iterations of our work. \nIn [2] the authors present a novel method for intent classification that leverages prior knowledge of previously known intents and tries to discover new intents (analogy between intent and regions in our work). In our work, there are no known previous regions so we cannot leverage the prior data in that fashion as the authors use to fine-tune the representation. As in [2], we do not know the number of regions a priori, and their approach to filter clusters can be useful. We find that the deep aligned clustering algorithm to be very interesting to apply a baseline in our tasks. One caveat is that it would require changing the cross-modal embeddings (CLIP) we use and it has been shown by DOMINO [17] that they outperform other uni-modal representations such as BERT embeddings for region discovery. Similarly, we also think that the Adaptive Decision Boundary Learning algorithm in [1] can also be applied to discover regions and might work well on the setting. We look forward to applying both algorithms as baselines in future iterations of our work.\n\n\n\n[1] Zhang, Hanlei, Hua Xu, and Ting-En Lin. \"Deep open intent classification with adaptive decision boundary.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 16. 2021.\n[2]  Zhang, Hanlei, et al. \"Discovering new intents with deep aligned clustering.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 16. 2021."}, {"review_id": "p6rANk5qCn", "reviewer": "Reviewer_Ea7z", "summary": "This paper discusses some research problems a very interesting scenario where human and AI need to collaborate to achieve a certain goal. The authors propose to learn rules grounded in data regions and described in natural language, which illustrates how the human should collaborate with the AI agent.\n\nThe paper also proposes a novel algorithm, region discovery algorithm, which uses an iterative procedure where a large LM describes the region while distinguishing it from the rest of the date. Experiments show positive results when following this protocol.", "questions": "Have you considered about the \"cost\" during human AI collaborations? e.g., time. If fully automated pipeline can work well with less cost, that might be our preferred one.", "limitations": "The relatively small scale human evaluation.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. It tries to solve research problems in a very interesting domain.\n2. The user study is extensive and its setting is convincing.", "weaknesses": "1. The presentation needs huge improvement. e.g., some figures are not clear and text is vague.\n2. The study was conducted with 25 human participants, which might not be representative human users.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your detailed review, we appreciate your time and all the important comments. Please read below for our response.\n\n## Weaknesses: \n\n1. We will improve the presentation and clarify of the paper in our next iteration. In particular, we will clarify the methods in section 4 better and figure 1 and others.\n\n2. *\"The study was conducted with 25 human participants, which might not be representative human users.\"*\n\nPlease note that the final user study was conducted with *150* human participants, with 50 humans in each condition. The initial 25 human participants were used for data collection to build the approach, then an additional 150 participants were recruited. This number of participants per condition is comparable to prior work in the literature and was sufficient to establish the statistical significance of our approach.\n\n## Questions:\n\nThank you for an insightful question. We did indeed consider a form of cost of the human-ai interaction, in particular time to make a prediction as you suggest. We found in Table 3 (last row), that Humans without the AI spend 5.4s per example, without onboarding they spend 7.78s per example, however with onboarding this reduces to 5.9s which is a significant decrease (two sample t-test, p=0.0014, t=3.2872).\n\n## Limitations: \n\nPlease note again that the user study evaluation was conducted with 150 participants (50*3) as mentioned in lines 360-362 of section 7 “Experimental Conditions”.\n\nIf our rebuttal changed your view of the paper given the corrected sample size of participants for the studies, please consider adjusting your score accordingly.\n"}, {"review_id": "FzRSFq9iU9", "reviewer": "Reviewer_YAuF", "summary": "The authors propose an HAI system called IntegrAI that should help people make better decisions about when to defer to AI predictions, make predictions themselves, or combine their decisions. \nAs part of their system, the authors perform semantic clustering using LLMs to find areas of disagreement between AI and humans and distill those clusters into simple rules that can be surfaced as insights to users. \n\nThe IntegrAI-describe algorithm can be summarized as:\n1. Create clusters in joint embedding space\n2. Get candidate descriptions of each cluster and embed them into same space\n3. Search for counterexamples\n4. Update clusters and repeat\n\nThe key claims are:\n1. IntegrAI learns optimal integration decisions\n2. Rules generated by the oracle (GPT3.5 in this case) are easily understandable\n3. Onboarding calibrates human expectations about AI performance\n4. Dashboard insights help humans choose which action they should take thereby improving teaming performance.", "questions": "See sections above.", "limitations": "The proposed framework is a 2nd-order framework in the sense that:\n- 0th order is if human or AI individually make decision\n- 1st order is if human sees AI decision and can change mind\n - 2nd order is another AI sits on top of this process and tells human whether they should change their mind. \nFor this type of framework to be useful, there needs to be a complex setting where humans struggle to model their own uncertainty/limitations as well as those of the AI. I think the traffic light setting may not be adequately complex to really let this framework shine and show off its capabilities. I think it would really strengthen the paper and provide evidence for the key claims if the authors could evaluate on a more complex setting (e.g. multi-class, greater ambiguity, etc.)\n\nAlso, the proposed approach requires collecting paired data. This is a potential limitation if a lot of expensive human data is required. It would be great to see a discussion on how much paired data is required and whether this is a major limitation. An additional limitation of the paired data collection procedure the authors use in their particular case study is that integration can't be detected (i.e. currently authors only detect whether the human used the AI answer or didn't).", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "- the IntegrAI framework is novel and very interesting\n- the IntegrAI-describe algorithm is a very clever way to leverage LLMs for semantic clustering", "weaknesses": "1. Several of the main claims (see list in summary above) are unsubstantiated by empirical results. Specifically:\n- claim 2 (rules are easily understandable) is not extensively tested and the results comparing it against SEAL seem inconclusive. \n- experimental results provide only weak evidence for claim 3 and claim 4, and it remains unclear whether a combination of Onboarding + Rec is actually useful (e.g. if looking only at effect size means, then Onboarding alone has both higher performance and lower time than Onboarding+Rec)\n\n2. Additionally, I think there may be some issues in the significance testing for the results in Table 3. Given the large reported standard errors, it seems unlikely that the p-values would be as small as reported. \n\nMinor suggestions:\n- small typo in figure 1: no panel \"d)\"\n- small type on line 57: should be \"compared\" instead of \"compare\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your detailed review, we appreciate your time and all the important comments. \n\n\n**Error Measurements Typo**: Before we delve into each point in your review, we want to clarify a typo in our paper. We report standard deviations instead of standard error  for all results when we write $ XX.X \\pm X.X$. Just to clarify in detail what we mean: given a random variable X (e.g. performance of human on user study), given samples $x_1,\\cdots,x_n$ we can compute the mean $\\mu_n$ (np.mean) and the standard deviation $\\sigma_n$ (np.std as used in our code in the supplement) measures the deviation around the mean. On the other hand, the standard error is computed as standard deviation divided by the square root of sample size: $e_n=\\sigma_n / \\sqrt{n}$ measures the precision of our mean measurement $\\mu_n$ to the true mean $\\mu$.\n\n## Weaknesses:\n\n1. We agree that we should make a better job to make clear that claims 1-4 are hypotheses that we as authors made before developing the experiments and are then either refuted or backed by the experiments, the introduction states these hypotheses which then section 6-7 help rebute or back. \n\n*“claim 2 (rules are easily understandable) is not extensively tested and the results comparing it against SEAL seem inconclusive.”*:\n\n It is true that we did not perform direct quantitative analysis about the rules being interpretable themselves, what we did test instead in the user study is whether the rules presented through onboarding have an effect in improving the human accuracy. Moreover, we have in the appendix show examples of these rules for BDD and for MMLU in Tables 7 and 8 respectively so one can qualitatively evaluate whether they are intepretable. Finally, the results in aim 3 (sec 6) evaluating our method compared to SEAL do show a noticeable effect: with respect to METEOR score, our method achieves 0.32 $\\pm$ 0.04  (standard error: $0.04 = 0.30 / \\sqrt{50} $ where 50 is the total amount of objects tested for Aim 3) while SEAL achieves 0.10 $\\pm$ 0.02.\n\n*‘experimental results provide only weak evidence for claim 3 and claim 4“:*\n\n We believe our user study provides strong evidence for claim 3 in the task studied as we have a statistically significant effect for onboarding compared to baselines. For claim 4, as the results show, that claim is refuted as clearly recommendations increase the time to make a prediction and do not improve performance beyond onboarding. We will make in future iterations of this paper that the claims in the intro are hypotheses that we test and then include the result of our tests from the beginning. \n\n\n\n2.  We hope that the note at the beginning of the rebuttal clarifies the issue with standard errors (we reported standard deviation). There are no issues at all with the significance testing performed. Table 3 with standard errors instead of standard deviation is as follows (simply divide the standard deviation by square root of 50, as 50 is the sample size)\n\n\n| Metric           | AI only        | Human          | Human-AI       | Onboard(ours)+Rec | Onboard(ours)  | Onboard(baseline) | Rec              |\n|------------------|----------------|----------------|----------------|-------------------|----------------|-------------------|------------------|\n| Accuracy (\\%)    | $79.2 \\pm 1.3$ | $78.5 \\pm 1.8$| $77.2 \\pm 1.5$| $81.7 \\pm 1.2$    | $82.9 \\pm 1.2$ | $79.9 \\pm 1.4$    | $81.4 \\pm 1.7$ |\n\n\n\n\n## Limitations:\n\n*“The proposed framework is a 2nd-order framework in the sense that:”*\n\nPractically our interface is a 1st order framework in your terminology and not a 2nd order framework since the human sees the AI decision and then makes the final decision, the AI does not tell the human to change their opinion (interface in Figure 3). However, it can be considered as a pseudo-2nd order framework since during onboarding we use the data of the human relying on the AI and try to correct the human’s perception about the AI. \n\n*“I think it would really strengthen the paper and provide evidence for the key claims if the authors could evaluate on a more complex setting (e.g. multi-class, greater ambiguity, etc.)”*\n \n-> Our user study task on traffic light detection went through multiple iterations to make sure the task is interesting which was accomplished with varying increasing levels of blur to the images to accomplish “greater ambiguity” as you mention.\n\n\n*“approach requires collecting paired data.”*\n\nThis is a major limitation of our work in that to work optimally we need human data of interacting with the system. The approach can be made to function without paired data but performance would decrease. We go around the limitation of limited paired data by building predictors of the human prior from the limited data we have so that we can essentially increase the sample size but might introduce additional bias. We will add this as a limitation in our paper in the next iterations. Finally, while it is true that integration cannot be detected which we fully acknowledge in lines 148-151, human prediction with and without AI can reveal some information of integration.\n"}]

Author rebuttal:
Thank you for the reviews of our paper and the insightful suggestions. Please find under each review our rebuttal. 

We wanted to iterate the main claims and contributions of our paper:

- We propose a novel region discovery algorithm (sec 4.1) and evaluate it  on four different datasets and show that it can outperform baselines (aim 1 and aim 2 in sec 6)
- We propose a novel region description algorithm (sec 4.2) (automatic cluster labeling), we show quantitatively that it can uncover the ground truth cluster label in an ablation study on MS-COCO (aim 3 in sec 6) and show qualitatively that it produces coherent descriptions (table 7 and table 8 in Appendix)
- We propose an onboarding scheme that presents the regions discovered as “lessons” to the human (step 1,2,3 in section 6), we evaluate this strategy with a full user study with 150 participants on the BDD dataset in section 6 and show that onboarding can significantly improve performance (p=0.002). We additionally propose to show the recommendations of the region as an aid to the human (end of section 6), however, we show that they are not as effective as onboarding. 

We are very glad that the reviewers thought that the problem tackled was “in a very interesting domain.” (reviewer Ea7z) and “interesting and practical approach for developing a human AI collaboration system” (reviewer oGDF),  our “framework is novel and very interesting” (reviewer YaUF), the approach to describe regions is clever “IntegrAI-describe algorithm is a very clever way to leverage LLMs for semantic clustering“ (reviewer YaUF) and “use of chatGPT as an oracle is also a great workaround“ (reviewer oGDF) and that the “user study is extensive and its setting is convincing” (reviewer Ea7z). 

We also want to point out a typo in the paper on line 376 “$\pm$ standard error”, every result in the paper is reported as mean +- standard deviation (instead of standard error). One can obtain standard error from standard deviation by dividing by the square root of the sample size. 


Meta-review: The paper presents a new approach to improve human-AI teams, leveraging semantic analysis to identify tasks that humans should rely on AI, and using LLM to generate natural language rules to communicate to humans. The approach is evaluated with human-subject experiments that provided the natural language rules as onboarding methods.

The paper is addressing a timely and important question. The approaches are novel and sound, and the evaluations are overall satisfactory. There are various suggestions from the reviewers, including the presentation and aligning the claims with the results. However, overall, the reviewers all agree that this is a solid work and we recommend acceptance.
Decision: Accept (spotlight)