SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hussein Mozannar $^{1,2}$
Jimin J Lee<sup>1,2</sup>
Dennis Wei $^{1,3}$
Prasanna Sattigeri<sup>1,3</sup>
Subhro Das<sup>1,3</sup>
David Sontag<sup>1,2</sup>
$^{1}$ MIT-IBM Watson AI Lab, Cambridge, MA
$^{2}$ CSAIL and IMES, Massachusetts Institute of Technology, Cambridge, MA
<sup>3</sup>IBM Research, Cambridge, MA
{mozannar;gminnout}@mit.edu，{dwei;psattig}@us.ibm
subhro.das@ibm.com, dsontag@mit.edu
People are relying on AI agents to assist them with various tasks. The human must know when to rely on the agent, collaborate with the agent, or ignore its suggestions. In this work, we propose to learn rules grounded in data regions and described in natural language that illustrate how the human should collaborate with the AI. Our novel region discovery algorithm finds local regions in the data as neighborhoods in an embedding space that corrects the human prior. Each region is then described using an iterative and contrastive procedure where a large language model describes the region. We then teach these rules to the human via an onboarding stage. Through user studies on object detection and question-answering tasks, we show that our method can lead to more accurate human-AI teams. We also evaluate our region discovery and description algorithms separately.
Figure 1: The proposed onboarding approach with the IntegrAI algorithm.
As artificial intelligence (AI) becomes more ubiquitous and sophisticated, humans are increasingly working alongside AI systems to accomplish various tasks, ranging from medical diagnosis [9, 28], content moderation [30] to writing [21] and programming [22]. One of the promises of AI is to enhance human performance and efficiency by providing fast and accurate solutions. However, the literature on human-AI collaboration has revealed that humans often underperform expectations when working with AI systems [36, 49, 40, 77]. Moreover, studies have shown that providing explanations for the AI's predictions often does not yield additional performance gains and, in fact, can make things worse [49, 15, 32]. These negative results of human-AI performance may be attributed to two possible reasons. First, humans have miscalibrated expectations about AI's ability, which can lead to over or under-reliance [5]. Second, the cost of verifying the AI's answer with explanations is often higher than that of solving the task without the AI as some explanations don't help in verifying the AI answer [75].
The central question remains: how can we collaborate better with AI models? In this work, we propose an intuitive framework for thinking about human-AI collaboration where the human first decides on each example whether they should rely on the AI, ignore the AI, or collaborate with the AI to solve the task. We refer to these three actions as the AI-integration decisions. The human-AI team will perform optimally if the human knows which action is best on a task-by-task basis. We propose IntegrAI (Figure 1), an algorithm that leverages data from baseline human interactions with the AI to learn near-optimal integration decisions, in the form of natural language rules that are easily understandable. These rules are then taught to the human through an onboarding stage, analogous to an onboarding class that humans might take before operating machines and equipment. Onboarding additionally calibrates the human's expectations about AI performance. We further investigate surfacing the AI-integration decisions found by IntegrAI as recommendations to the human within an AI dashboard used after onboarding. The hope is that onboarding and the dashboard help the human know which action they should take, thereby leading to effective AI adoption for enhanced decision-making.
Learning AI-integration rules requires a dataset of paired examples and human predictions (Figure 1 Step 1). Each rule is defined as a bounded local region centered around a learned point in a potentially multi-modal embedding space spanning the task space and natural language (Figure 1 Step 2). For example, CLIP embeddings [62] connect image and text spaces for tasks involving images, and typical text embeddings [64] are used for natural language tasks such as question answering. The regions are obtained with a novel region discovery algorithm. Then, a text description of the region is generated, resulting in a rule that indicates whether the human should ignore, rely on, or collaborate with the AI. We obtain descriptions using a novel procedure that connects the summarization ability of a large language model (LLM) [14] with the retrieval ability of embedding search to find similar and dissimilar examples. The procedure first queries the LLM to describe points inside the region (Figure 1 Step 3). The embedding space is then leveraged to find counterexamples inside and outside the region to refine the description.
We first evaluate the ability of our region finding and region description algorithms to find regions that will aid the human-AI team in several real-world datasets with image and text modalities. We then investigate the efficacy of both algorithms in synthetic scenarios where we know the ground truth regions. Finally, we conduct user studies on tasks with real-world AI models to evaluate our onboarding and AI-integration recommendation methodology. Our main task is detecting traffic lights in noisy images [83] from the perspective of a road car, motivated by applications to self-driving cars. The user study reveals that our methods significantly improve the accuracy of the human-AI team by  $5.2\%$  compared to without onboarding. We investigate a second task of multiple choice question answering using the MMLU dataset [33] and find that onboarding has no effect on performance and that only displaying AI-integration recommendations has a negative effect. To summarize, the key contributions of this paper are as follows:
- We propose a novel region-finding and region-description algorithm, IntegrAI, that outperforms prior work to derive rules that can guide humans in collaborating with AI models for shared decision-making.
- We demonstrate the effect of onboarding and displaying AI-integration recommendations on two real-world tasks, and find that onboarding has a significant positive effect on one task and that integration recommendations are not useful.
A growing literature of empirical studies on AI-assisted decision making has revealed that human-AI teams do not perform better than the maximum performance of the human or AI alone even with AI explanations [7, 71, 49]. Recent work has explored modifying AI explanations to depend on human behavior [77, 51] and showed positive results with synthetic AI models. A growing literature exists on onboarding humans to work on AI models [67, 17, 45, 57, 16, 41], however, our work enables the automated creation of onboarding without any human in the loop. We compare to a representative from work on discovering regions of errors in AI models [24, 10, 66, 82, 87, 80, 23, 63, 37], our work instead focuses on regions of disparate performance between human and AI. Learning to defer methods learn models that decide using a secondary AI model who between the human and the AI classifier should predict [52, 56], this paper focuses on the reverse setting where the human makes all decisions but utilizes some of the thinking from that literature. Our AI-integration recommendations are also related to personalized policies [11]. Our MMLU experiments share similarities with recent work [13, 86, 74, 38]. Further comparison to prior work can be found in Appendix A.
3 AI Assisted Decision Making
Setting. We consider a setting where a human is making decisions with the help of an AI agent who provides advice to complete a task. Formally, the human has to make a decision  $Y \in \mathcal{Y}$  given access to information about the context as  $Z \in \mathcal{Z}$  and the AI's advice  $A \in \mathcal{A}$ . We denote the human as a potentially randomized function  $H(Z, A; \theta_h)$  with parameters  $\theta_h$  which are unobservable. On the other hand, the AI agent provides advice based on its viewpoint of the context as  $X \in \mathcal{X}$ , which might be different from the humans viewpoint  $Z$  as the human may have side information. The AI model provides advice according to  $M(X; \theta_m) := A \in \mathcal{A}$ , the advice always includes a candidate decision  $\hat{Y}$  and possibly an explanation of the decision. We assume that the observed tasks are drawn from an underlying distribution,  $\mathbb{P}_{X,Z,Y}$ , over the contexts of AI and human, and the ground truth. For simplicity of exposition, we will assume that  $X = Z$  when describing our methods.
Task Metrics. The human wants to make a decision that optimizes various metrics of interest. Given a ground truth decision  $Y$  and a chosen decision  $\hat{Y}$ , the loss is given by  $l(Y, \hat{Y}) : \mathcal{V} \times \mathcal{V} \to \mathbb{R}^+$ . We denote the loss  $L(H, M)$  of the Human-AI team over the entire domain as:
$$
L (H, M) := \mathbb {E} _ {x, z, y \sim \mathbb {P}} \left[ l \left((y, H (z, M (x; \theta_ {m}); \theta_ {h}))\right) \right] \tag {1}
$$
In our example, this could be the 0-1 loss  $\mathbb{I}_{Y = \hat{y}}$ . We are further interested in metrics that convey efforts undertaken by the human during the process. Particularly, we focus on time to make a decision, which can be measured when the human makes decisions.
Human-AI team. The decision of the Human-AI team is represented by the function  $H(Z, A; \theta_h)$ . The human without the AI is denoted by  $H(Z, \emptyset; \theta_h) \coloneqq H(Z; \theta_h)$ , which is obtained by setting the AI advice to the empty set, i.e. no advice. In theory, we expect the human with advice to perform at least as well as without advice, simply because the human can always ignore the advice. However, the literature on Human-AI teams has clearly demonstrated that this is often not the case [46]. An effective Human-AI team is one where the human with the AI's advice achieves a more favorable trade-off in terms of the metrics than without the advice.
Framework for Cooperation. Our insight into forming an effective team is to explicitly recommend when the human should consider the advice and how to incorporate it into their decision. We propose a two-stage framework for the human to cooperate with the AI: the human first decides whether to ignore the AI advice, use the AI's decision or integrate the AI advice to make a decision with explicit cooperation. Each of these three cases provides a clear path to the second stage of making the final output decision.
Definition 1. The AI-integration function  $R(Z, A; \theta_r)$ , also referred to as integrator, formalizes a framework for the human to cooperate with the AI:
$$
R (Z, A; \theta_ {r}) = \left\{\begin{array}{l l}0 \rightarrow H (Z; \theta_ {h})&\text {(i g n o r e A I)}\\1 \rightarrow \hat {Y}&\text {(u s e A I d e c i s i o n a s i s)}\\2 \rightarrow H (Z, A; \theta_ {h})&\text {(c o l l a b o r a t e w i t h A I)}\end{array}\right. \tag {2}
$$
In this work, we only consider the actions of ignoring or using the AI decision:  $R \in \{0,1\}$  and leave the action of  $R = 2$  for future work.
The integration function can be thought of as a specific formalization of the human mental model of the AI [5, 6]. Given an integration function  $R$  and a human  $H$ , we can define a hypothetical human decision maker  $H_{R}$  who first computes  $R$  and then follows its recommendation to make a final decision. Similarly, for each human  $H(Z,A;\theta_h)$ , we can associate an integration function  $R_{H}$  such that  $H_{R_{H}} = H$ . By fixing  $H(Z;\theta_h)$ , one can try to minimize the loss  $L(H_{R},M)$  over all possible choices of  $R$ , an optimal point of such an integration function is denoted  $R^{*}$ . Following the recommendations of the optimal AI-integration function leads to an effective Human-AI team.
Learning Rules and Onboarding. There are two problems that need to be solved to achieve this vision: how do we learn such an  $R^*$  and how can we ensure that the human follows the recommendations of this  $R^*$ ? In the next section, we outline how we approximate the optimal integration function; this is fundamentally a machine-learning problem with its own challenges that we tackle in Section 4. The second obstacle is that the human should know to follow the recommendations of  $R^*$ . To ensure this, we propose an onboarding stage where the human learns about the AI and the optimal integration function and additionally display the recommendations as part of an AI dashboard. In this onboarding stage, we will help the human shape their internal parameters  $(\theta_h, \theta_r)$  to improve performance. This is a human-computer interaction (HCI) problem that we tackle in Section 5.
4 Learning Rules for Human-AI Cooperation: IntegrAI
In this section, we discuss how to learn an integrator function  $\hat{R}:\mathcal{X}\times \mathcal{A}\to \{0,1\}$  to approximate an optimal integrator while being understandable to the human. We first describe the ingredients for this learning (integrator as a set of regions, objective function, dataset) before detailing how we learn regions and describe them in Sections 4.1 and 4.2 respectively.
Integrator as a Set of Regions. Since the integrator  $\tilde{R}$  will be used to both onboard the human and provide test-time recommendations as part of an AI dashboard, it should be easily understandable to humans. If the goal was to build the most accurate integrator, we could use work on learning to defer [56, 19, 58]. To address this requirement, we propose to parameterize the integrator in terms of a set of local data regions, each with its own integration decision label as well as a natural language description. More specifically, we aim to learn a variable number of regions  $N_{1}, N_{2}, \cdots$  as functions of  $(X, A)$ , the observable context and the AI's advice. Each region  $N_{k}$  consists of the following: 1) an indicator function  $I_{N_{k}}: \mathcal{X} \times \mathcal{A} \rightarrow \{0, 1\}$  that indicates membership in the region, 2) a textual description of the region  $T_{k}$ , and 3) a takeaway for the region consisting of an integration decision in  $r(N_{k}) \in \{0, 1\}$ . We additionally want these regions to satisfy a set of constraints so that they are informative for the human and suitable for onboarding.
Maximizing Human's Performance Gain. Since we are working with human decision-makers, we have to account for the fact that the human implicitly has a prior integration function  $R_0$ , which represents how the human would act without onboarding. Thus, in learning integrator regions, our goal is to maximize the human performance gain relative to their prior. The performance gain is defined as follows for points in a region  $N$ :
$$
G (N, \hat {R}, R _ {0}) = \sum_ {i \in N} l \left(y _ {i}, H _ {R _ {0}} \left(x _ {i}, a _ {i}\right)\right) - l \left(y _ {i}, H _ {\hat {R}} \left(x _ {i}, a _ {i}\right)\right), \tag {3}
$$
where  $l$  is the loss defined in Section 3. Note that the notion of a human's prior mental model was also discussed by [57] but we expand on the notion and are able to learn priors as we discuss below.
Dataset with Human Decisions. We assume we have access to a dataset  $D = \{x_{i},y_{i},h_{i},r_{0i}\}_{i = 1}^{n}$  sampled from  $\mathbb{P}$ , where  $x_{i}$  is the AI-observable context,  $y_{i}$  is the optimal decision on example  $i$ ,  $h_{i}$  is a human-only decision, defined in Section 3 as  $H(x_{i};\theta_{h})$ , and  $r_{0i} \in \{0,1\}$  is an indicator of whether the human relied on AI on example  $i$ . We thus regard the samples  $\{r_{0i}\}_{i = 1}^{n}$  as a proxy for prior human integration function  $R_0$ . The prior integration decisions of the human  $r_{0i}$  are collected through a data collection study where the human predicts with the AI without onboarding. When the human presses on the "Use AI" button (see Figure 3) we record  $r_{0i} = 1$ , and 0 otherwise. The human predictions  $h_i$  are collected through a secondary data collection study where the human makes predictions without the AI. We also assume that we are given an AI model  $M$ , from which we obtain
AI decisions  $\hat{y}_i \in a_i$  from the AI advice  $a_i = M(x_i; \theta_m)^1$ . Given the dataset  $D$ , AI decisions  $\hat{y}_i$ , and loss  $l(.,.)$ , we can define optimal per-example integration decisions  $r_i^*$  by comparing human and AI losses on the example:  $r_i^* = \mathbb{I}\left(l(y_i, h_i) > l(y_i, \hat{y}_i)\right)$ .
4.1 Region Discovery Algorithm
Representation Space. In this subsection, we describe a sequential algorithm that starts with the prior integration function  $R_0$  and adds regions  $N_k$  one at a time. The domain  $\mathcal{X}$  for the task may consist of images, natural language, or other modalities that possess an interpretable representation space. We follow a similar procedure for all domains. The first step is to map the domain onto a potentially cross-modal embedding space using a mapper  $E(\cdot)$ , where one of the modes is natural language. The motivation is that such an embedding space will have local regions that share similar natural language descriptions, enabling us to learn understandable rules. For example, for natural images, we use embeddings from the CLIP model [62]. The result of this step is to transform the dataset  $\{x_i\}_{i=1}^n$  into a dataset of embeddings  $\{e_i\}_{i=1}^n$  where  $e_i \in \mathcal{E}_{\mathcal{X}} \subseteq \mathbb{R}^d$ .
Region Parameterization. We define region  $N_{k}$  in terms of a centroid point  $c_{k}$ , a scaled Euclidean neighborhood around it, and an integration label  $r_k \in \{0,1\}$ . The neighborhood is in turn defined by a radius  $\gamma_{k} \in \mathbb{R}$  and element-wise scaling vector  $w_{k}$ . Both  $c_{k}$  and  $w_{k}$  are in  $\mathcal{E}_{\mathcal{X}} \times \mathcal{A}$ , the concatenation of the embedding space and the AI advice space. The indicator of belonging to the region is then  $I_{N_k}(e_i, a_i) = \mathbb{I}_{||w_k \circ ((e_i, a_i) - c_k)||_2 < \gamma_k}$ , where  $\circ$  is the Hadamard (element-wise) product.
Region Constraints. We add the following constraints on each region  $N_{k}$  to make them useful to the human during onboarding. First, the region size in terms of fraction of points contained must be bounded from below by  $\beta_{l}$  and above by  $\beta_{u}$ . Second, the examples in each region must have high agreement in terms of their optimal per-example integration decisions  $r_{i}^{*}$ . Specifically, at least a fraction  $\alpha$  of the points in a region must have the same value of  $r_{i}^{*}$ .
IntegrAI-Disccover. In round  $k$ , we add a  $k$ th region to the integrator  $R_{k-1}$  from the previous round (with  $k-1$  regions) to yield  $R_k$ . After  $m$  rounds, the updated integrator  $R_m$  is defined as follows: Given a point  $(x, a)$ , if it does not belong to any of the regions  $N_1, \ldots, N_T$ , then we fall back on the prior. Otherwise, we take a majority vote over all regions to which  $(x, a)$  belongs:
$$
R _ {T} (x, a) = \left\{ \begin{array}{l l} R _ {0} (x, a) & \text {i f} I _ {N _ {k}} (x, a) = 0, k = 1, \dots , T \\ \text {m a j o r i t y} \left(\left\{r \left(N _ {k}\right): k \text {s . t .} I _ {N _ {k}} (x, a) = 1 \right\}\right) & \text {o t h e r w i s e .} \end{array} \right. \tag {4}
$$
In round  $k$ , we compute the potential performance gain for each point if we were to take decision  $r$  on the point as  $g_{i,r} = l(y_i, H_{R_{k-1}}(z_i, a_i)) - l(y_i, r)$ . The optimization problem to find the optimal regions is non-differentiable due to the discontinuous nature of the region indicators. To make this optimization problem differentiable, we relax the constraints as penalties with a multiplier  $\lambda$  and replace the indicators with sigmoids scaled by a large constant  $C_1$ :
$$
\max  _ {c, \gamma , w, r} \sum_ {i = 1} ^ {n} \sigma \left(C _ {1} (- | | w \circ ((e _ {i}, a _ {i}) - c) | | + \gamma)\right) \cdot g _ {i, r} - \lambda \max  (\sum_ {i = 1} ^ {n} \sigma \left(C _ {1} (- | | w \circ ((e _ {i}, a _ {i}) - c) | | + \gamma)\right)
$$
$$
\cdot \left(\mathbb {I} _ {r _ {i} ^ {*} = r} - \alpha n), 0\right) - \lambda \max  \left(\sum_ {i = 1} ^ {n} \sigma \left(C _ {1} (- | | w \circ ((e _ {i}, a _ {i}) - c) | | + \gamma)\right) - \beta_ {u} n, 0\right)
$$
The lower bound constraint with  $\beta_{l}$  can be added in a similar fashion. We find the value of  $r$  with an exhaustive search and use a gradient based procedure for the other optimization variables. The full optimization details are in Appendix B.
4.2 Region Description Algorithm
We now describe our region description algorithm aimed at making the rules for integration human understandable. Natural language descriptions are a good match for this objective. Specifically, we would like to find a contrastive textual description  $T_{k}$  of each region  $N_{k}$  that describes it in a way to distinguish it from the rest of the data space.
Textual Descriptions for Regions: The first step is to have a textual description  $t_i$  for each example in our dataset  $D$  based on  $x_i$ . If textual descriptions are not available, we can obtain them by utilizing
Moreover, we assume the AI was not trained on the dataset  $D$  so that we can use  $D$  to obtain an unbiased measurement of AI performance; this is crucial, as otherwise, we might overestimate its performance.
Algorithm 1 IntegrAI-Describe
Input: Dataset  $D$ , region  $N_{k}$
1:  $S^{+} \gets 15$  random examples from  $N_{k}$ ,  $S^{-} \gets 5$  random examples outside  $N_{k}$
2: Initial Region Description:  $T_{k}^{0} \gets \hat{O}(S^{+}, S^{-})$  (LLM call)
3: for  $i = 1,\dots ,m$  do
4: Find Counterexample outside region.  $s^{-} = \arg \max_{j \notin N_k} \sin(E(T_k^i), e_j)$  (most similar outside region)
5: Find Counterexample inside region.  $s^+ = \arg \min_{j \in N_k} \sin(E(T_k^i), e_j)$  (least similar inside region)
6: Update Inside and Outside Sets.  $S^{-} \gets S^{-} \cup t_{s^{-}}$  and  $S^{+} \gets S^{-} \cup t_{s^{+}}$
7: Get New Region Description:  $T_{k}^{i} \gets \hat{O}(S^{+}, S^{-})$  (LLM call)
8: end for
Return: Region description  $T_{k}^{m}$
models that map from the domain  $\mathcal{X}$  to natural language such as captioning models for images, summarization models for text, or exploiting metadata to construct a natural language description. One idea to obtain a region description is to ask an LLM (such as GPT-3.5) to summarize all textual descriptions of points inside the region. However, there are two issues with this approach: first, the region may contain thousands of examples so we need an effective way to select which points to include, and second, the obtained region description may not contrast with points outside the region. To resolve these issues, we propose an algorithm that iteratively refines region descriptions with repeated calls to an LLM  $(\hat{O})$  in 1 (IntegrAI-Describe). The algorithm starts with an initial description and then finds counterexamples to that description at each round: examples outside the region with high cosine similarity with respect to embeddings (sim) of the region description and examples inside the region with low similarity to the region description. Then we add those counterexamples to our example sets and derive a new region description, we use an especially created prompt with an exemplar to the LLM to get the region description at each round, the prompt can be found in Appendix C.