SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Nina L. Corvelo Benz
ninacobe@mpi-sws.org
Manuel Gomez Rodriguez
manuel@mpi-sws.org
Whenever a binary classifier is used to provide decision support, it typically provides both a label prediction and a confidence value. Then, the decision maker is supposed to use the confidence value to calibrate how much to trust the prediction. In this context, it has been often argued that the confidence value should correspond to a well calibrated estimate of the probability that the predicted label matches the ground truth label. However, multiple lines of empirical evidence suggest that decision makers have difficulties at developing a good sense on when to trust a prediction using these confidence values. In this paper, our goal is first to understand why and then investigate how to construct more useful confidence values. We first argue that, for a broad class of utility functions, there exist data distributions for which a rational decision maker is, in general, unlikely to discover the optimal decision policy using the above confidence values—an optimal decision maker would need to sometimes place more (less) trust on predictions with lower (higher) confidence values. However, we then show that, if the confidence values satisfy a natural alignment property with respect to the decision maker's confidence on her own predictions, there always exists an optimal decision policy under which the level of trust the decision maker would need to place on predictions is monotone on the confidence values, facilitating its discoverability. Further, we show that multicalibration with respect to the decision maker's confidence on her own predictions is a sufficient condition for alignment. Experiments on four different AI-assisted decision making tasks where a classifier provides decision support to real human experts validate our theoretical results and suggest that alignment may lead to better decisions.
In recent years, there has been an increasing excitement on the potential of machine learning models to improve decision making in a variety of high-stakes domains such as medicine, education or criminal justice [1-3]. One of the main focus has been binary classification tasks, where a classifier helps a decision maker by predicting a binary label of interest using a set of observable features [4-7]. For example, in medical treatment, the classifier may help a doctor by predicting whether a patient may benefit from a treatment. In college admissions, it may help an admissions committee by predicting whether a candidate may successfully complete an undergraduate program. In loan decisions, it may help a bank by predicting whether a prospective customer may default on a loan. In all these scenarios, the decision maker—the doctor, the committee or the bank—aim to use these predictions, together with their own predictions, to take good decisions that maximize a given utility function. In this context, since the predictions are unlikely to always match the truth, it has been widely agreed that the classifier should also provide a confidence value together with each prediction [8, 9].
While the conventional wisdom is that the confidence value should be a well calibrated estimate of the probability that the predicted label matches the true label [10-16], multiple lines of empirical evidence have recently shown that decision makers have difficulties at developing a good sense on when to trust a prediction using these confidence values [17-19]. Therein, Vodrahali et al. [17] have shown that, in certain scenarios, decision makers take better decisions using uncalibrated probability estimates rather than calibrated ones. However, a theoretical framework explaining this puzzling observation has been missing and it is yet unclear what properties we should be looking for to guarantee that confidence values are useful for AI-assisted decision making. In our work, we aim to bridge this gap.
Our contributions. We start by formally characterizing AI-assisted decision making using a structural causal model (SCM) [20], as seen in Figure 1. Building upon this characterization, we first argue that, if a decision maker is rational, the level of trust she places on predictions will be monotone on the confidence values—she will place more (less) trust on predictions with higher (lower) confidence values. Then, we show that, for a broad class of utility functions, there are data distributions for which a rational decision maker can never take optimal decisions using calibrated estimates of the probability that the predicted label matches the true label as confidence values. However, we further show that, if the confidence values a decision maker uses satisfy a natural alignment property with respect to the confidence she has on her own predictions, which we refer to as human-alignment, then the decision maker can both be rational and take optimal decisions. In addition, we demonstrate that human-alignment can be achieved via multicalibration [11], a statistical notion introduced in the context of algorithmic fairness. In particular, we show that multicalibration with respect to the decision maker's confidence on her own predictions is a sufficient condition for human-alignment. Finally, we validate our theoretical framework using real data from four different AI-assisted decision making tasks where a classifier provides decision support to human decision makers in four different binary classification tasks. Our results suggest that, comparing across tasks, classifiers providing human-aligned confidence values facilitate better decisions than classifiers providing confidence values that are not human-aligned. Moreover, our results also suggest that rational decision makers' trust level increases monotonically with the classifier's provided confidence.
Further related work. Our work builds upon a rapidly increasing literature on AI-assisted decision making (refer to Lai et al. [21] for a recent review). More specifically, it is motivated by several empirical studies showing that decision makers have difficulties at modulating trust using confidence values [17-19], as discussed previously. In this context, it is also worth noting that other empirical studies have analyzed how other factors such as model explanations and accuracy modulate trust [22-26]. However, except for a very recent notable exception [27], theoretical frameworks, which could be used to better understand the mixed findings found by these empirical studies, have been missing. More broadly, our work also relates to a flurry of recent work on reinforcement learning with human feedback [28-30], which aims to better align the outputs of large language models (LLMs) with human preferences. However, our formulation is fundamentally different and our technical contributions are orthogonal to theirs.
2 A Causal Model of AI-Assisted Decision Making
We consider an AI-assisted decision making process where, for each realization of the process, a decision maker first observes a set of features  $(x,v)\in \mathcal{X}\times \mathcal{V}$ , then takes a binary decision  $t\in \{0,1\}$  informed by a classifier's prediction  $\hat{y} = \operatorname{argmax}_y f_y(x)$ , as well as confidence  $f_{\hat{y}}(x)\in [0,1]$ , of a binary label of interest  $y\in \{0,1\}$ , and finally receives a utility  $u(t,y)\in \mathbb{R}$ . Such an AI-assisted decision making process fits a variety of real-world applications. For example, in medical treatment, the features  $(x,v)$  may comprise multiple sources of information regarding a patient's health<sup>1</sup>, the label  $y$  may indicate whether a patient would benefit from a specific treatment, the decision  $t$  may indicate whether the doctor applies the specific treatment to the patient, and the utility  $u(t,y)$  may quantify the trade-off between health benefit to the patient and economic cost to the decision maker.
In what follows, rather than working with both  $\hat{y}$  and  $f_{\hat{y}}(x)$ , we will work with just  $b = f_1(x)$ , which we will refer to as classifier's confidence, without loss of generality<sup>2</sup>. Moreover, we will assume that
<sup>1</sup>Our formulation allows for a subset of the features  $v$  to be available only to the decision maker but not to the classifier.
2We can recover  $\hat{y}$  and  $f_{\hat{y}}(x)$  from  $b$  , i.e., if  $b > 0.5$  , we have that  $\hat{y} = 1$  and  $f_{\hat{y}}(x) = b$  ; if  $b <   0.5$ $\hat{y} = 0$  and  $f_{\hat{y}}(x) = 1 - b$
Figure 1: Our structural causal model  $\mathcal{M}$ . Orange circles represent endogenous random variables and blue boxes represent exogenous random variables. The value of each endogenous variable is given by a function of the values of its ancestors in the structural causal model, as defined by Eqs. 2 and 3. The value of each exogenous variable is sampled independently from a given distribution.
the utility  $u(t,y)$  is greater if the value of  $t$  and  $y$  coincide, i.e.,
$$
u (1, 1) > u (1, 0), \quad u (1, 1) > u (0, 1), \quad u (0, 0) > u (1, 0), \quad \text {a n d} \quad u (0, 0) \geq u (0, 1), \tag {1}
$$
a condition that we think it is natural under an appropriate choice of label and decision values. For example, in medical diagnosis, if  $t = 1$  means the patient is tested early for a disease and  $y = 1$  means the patient suffers the disease, the above condition implies that the utility of either testing a patient who suffers the disease or not testing a patient who does not suffer the disease are greater than the utility of either not testing a patient who suffers the disease or testing a patient who does not suffer the disease. In condition 1, we allow for a non-strict inequality  $u(0,0) \geq u(0,1)$  because, in settings in which the label  $Y$  is only realized whenever the decision  $t = 1$  (e.g., in our previous example on medical treatment, we can only observe if a treatment is eventually beneficial or not if the patient is treated), it has been argued that, whenever  $t = 0$ , any choice of utility must be independent of the label value [4-6], i.e.,  $u(0,0) = u(0,1) = u(0)$ .
Next, we characterize the above AI-assisted decision making process using a structural causal model (SCM) [20], which we denote as  $\mathcal{M}$ . The SCM  $\mathcal{M}$  is defined by a set of assignments, which entail a distribution  $P^{\mathcal{M}}$  and divide naturally into two subsets. One subset comprises the features and the label $^3$ , i.e.,
$$
X = f _ {X} (D) \quad V = f _ {V} (D) \quad \text {a n d} \quad Y = f _ {Y} (D), \tag {2}
$$
where  $D$  is an independent exogenous random variable, often called exogenous noise, characterizing the data generating process and  $f_{X}, f_{V}$  and  $f_{Y}$  are given functions. The second subset comprises the decision maker and the classifier, i.e.,
$$
H = f _ {H} (X, V, Q), \quad B = f _ {B} (X, H), \quad T = \pi (H, B, W) \quad \text {a n d} \quad U = u (T, Y), \tag {3}
$$
where  $f_{H}$  and  $f_{B}$  are given functions, which determine the decision maker's confidence  $H$  and classifier's confidence  $B$  that the value of the label of interest is  $Y = 1$ ,  $\pi$  is a given AI-assisted decision policy, which determines the decision maker's decision  $T$ ,  $u$  is a given utility function, which determines the utility  $U$ , and  $Q$  and  $W$  are independent exogenous variables modeling the decision maker's individual characteristics influencing her own confidence  $H$  and her decision  $T$ , respectively. By distinguishing both sources of noise, we allow for the presence of uncertainty on the decision  $T$  even after conditioning on fixed confidence values  $h$  and  $b$ . This accounts for the fact that, in reality, a decision maker may take different decisions  $T$  for instances with the same confidence values  $h$  and  $b$ . For example, in medical treatment, for two different patients with the same confidence  $h$  and  $b$ , a doctor's decision may differ due to limited resources.
In our SCM  $\mathcal{M}$ , the decision maker's confidence  $H$  refers to the confidence the decision maker has that the label  $Y = 1$  before observing the classifier's confidence  $B$ . Moreover, following previous behavioral studies showing that human's confidence  $H$  is discretized in a few distinct levels [32, 33], we assume  $H$  takes values  $h$  from a totally ordered discrete set  $\mathcal{H}$ . We say that the decision maker's
3We denote random variables with capital letters and realizations of random variables with lower case letters.
4Our model allows both for causal and anticausal features [31].
confidence  $f_{H}$  is monotone (with respect to the probability distribution  $P(Y = 1)$ ) if, for all  $h, h' \in \mathcal{H}$  such that  $h \leq h'$ , it holds that  $P(Y = 1 \mid H = h) \leq P(Y = 1 \mid H = h')$ . Further, we allow the classifier's confidence  $B$  to depend on the decision maker's confidence  $H$  because this will be necessary to achieve human-alignment via multicalibration in Section 5. However, our negative result in Section 3 also holds if the classifier's confidence  $f_{B}(X, H) = f_{B}(X)$  only depends on the features, as usual in most classifiers designed for AI-assisted decision making. In the remainder, we will use  $Z = (X, H)$  and denote the space of features and human confidence values as  $\mathcal{Z} = \mathcal{X} \times \mathcal{H}$ . Figure 1 shows a visual representation of our SCM  $\mathcal{M}$ .
Under this characterization, we argue that, if a rational decision maker has decided  $t$  under confidence values  $b$  and  $h$ , then she would have decided  $t' \geq t$  had the confidence values been  $b' \geq b$  and  $h' \geq h$  while holding "everything else fixed" [20]. For example, in medical treatment, assume a doctor's and a classifier's confidence that a patient would benefit from treatment is  $b = h = 0.7$  and the doctor decides to treat the patient, then we argue that, if the doctor is rational, she would have treated the patient had the doctor's and the classifier's confidence been  $b' = h' = 0.8 > 0.7$ . Further, we say that any AI-assisted decision policy  $\pi$  that satisfies this property is monotone, i.e.,
Definition 1 (Monotone AI-assisted decision policy). An AI-assisted decision policy  $\pi$  is monotone if and only if, for any  $b, b' \in [0,1]$  and  $h, h' \in \mathcal{H}$  such that  $b \leq b'$  and  $h \leq h'$ , it holds that  $\pi(h, b, w) \leq \pi(h', b', w)$  for any  $w \sim P^{\mathcal{M}}(w)$ .
Finally, note that, under any monotone AI-assisted decision policy, it trivially follows that
$$
\mathbb {E} [ T \mid H = h, B = b ] \leq \mathbb {E} [ T \mid H = h ^ {\prime}, B = b ^ {\prime} ], \tag {4}
$$
where the expectation is over the uncertainty on the decision maker's individual characteristics and the data generating process.
3 Impossibility of AI-Assisted Decision Making Under Calibration
In AI-assisted decision making, classifiers are usually demanded to provide calibrated confidence values [10-16]. A confidence function  $f_{B}:\mathcal{Z}\rightarrow [0,1]$  is said to be perfectly calibrated if, for any  $b\in [0,1]$ , it holds that  $P(Y = 1\mid f_B(Z) = b) = b$ . Unfortunately, using finite amounts of (calibration) data, one can only hope to construct approximately calibrated confidence functions. There exist many different notions of approximate calibration, which have been proposed over the years. Here, for concreteness, we adopt the notion of  $\alpha$ -calibration<sup>5</sup> introduced by Hébert-Johnson et al. [11], however, our theoretical results can be easily adapted to other notions of approximate calibration<sup>6</sup>.
Definition 2 (Calibration). A confidence function  $f_{B}:\mathcal{Z}\to [0,1]$  satisfies  $\alpha$ -calibration with respect to  $\mathcal{S}\subseteq \mathcal{Z}$  if there exists some  $\mathcal{S}'\subseteq \mathcal{S}$ , with  $|\mathcal{S}'|\geq (1 - \alpha)|\mathcal{S}|$ , such that, for any  $b\in [0,1]$ , it holds that
$$
\left| P (Y = 1 \mid f _ {B} (Z) = b, Z \in S ^ {\prime}) - b \right| \leq \alpha , \tag {5}
$$
If the decision maker's decision  $T$  only depends on the classifier's confidence  $B$ , i.e.,  $\pi(H, B, W) = \pi(B)$  and  $f_{B}$  satisfies  $\alpha$ -calibration with respect to  $\mathcal{Z}$ , then, it readily follows from previous work that, for any utility function that satisfies Eq. 1, a simple monotone AI-assisted decision policy  $\pi_{B}^{*}$  that takes decisions by thresholding the confidence values is optimal [4-7], i.e.,  $\pi_{B}^{*} = \operatorname{argmax}_{\pi \in \Pi(B)} \mathbb{E}_{\pi}[u(T, Y)]$ , where the expectation is with respect to the probability distribution  $P^{\mathcal{M}}$  and  $\Pi(B)$  denotes the class of AI-assisted decision policies using  $B$ . However, one of the main motivations to favor AI-assisted decision making over fully automated decision making is that the decision maker may have access to additional features  $V$  and may like to weigh the classifier's confidence  $B$  against her own confidence  $H$ . Hence, the decision maker may seek for the optimal decision policy  $\pi^{*}$  over the class  $\Pi(H, B)$  of AI-assisted decision policies using  $H$  and  $B$ , i.e.,  $\pi^{*} = \operatorname{argmax}_{\pi \in \Pi(H, B)} \mathbb{E}_{\pi}[u(T, Y)]$ , since it may offer greater expected utility than  $\pi_{B}^{*}$ .
Unfortunately, the following negative result shows that, in general, a rational decision maker may be unable to discover such an optimal decision policy  $\pi^{*}$  using (perfectly) calibrated confidence values and this is true even if  $f_{H}$  is monotone:
5Note that, if  $\alpha = 0$  and  $S = \mathcal{Z}$  , the confidence function  $f$  is perfectly calibrated.
6 All proofs can be found in Appendix A.
Theorem 3. There exist (infinitely many) AI-assisted decision making processes  $\mathcal{M}$  satisfying Eqs. 2 and 3, with utility functions  $u(T,Y)$  satisfying Eq. 1, such that  $f_{B}$  is perfectly calibrated and  $f_{H}$  is monotone but any AI-assisted decision policy  $\pi \in \Pi(H,B)$  that satisfies monotonicity is suboptimal, i.e.,  $\mathbb{E}_{\pi}[u(T,Y)] < \mathbb{E}_{\pi^{*}}[u(T,Y)]$ .
In the proof of the above result in Appendix A.2, we show that there always exist a perfectly calibrated  $f_{B}(Z) = f_{B}(X,H)$  that depends on both  $X$  and  $H$  for which any monotone AI-assisted decision policy is suboptimal. This is due to the fact that  $f_{B}(Z)$  is calibrated on average over  $H$ , however, it may not be calibrated, nor even monotone, after conditioning on a specific value  $H = h$ . Further, we also show that, even if  $f_{B}(Z) = P^{\mathcal{M}}(Y = 1\mid X)$  matches the true distribution of the label  $Y$  given the features  $X$ , which has been typically the ultimate goal in the machine learning literature, there always exists a monotone  $f_{H}$  for which any monotone AI-assisted decision policy is suboptimal. This is due to the fact that the decision maker's confidence  $H$  can differ across instances with the same value for features  $X$  because it also depends on the features  $V$  and noise  $Q$ . Hence,  $f_{H}$  may not be monotone after conditioning on a specific value  $X = x$ . In both cases, when a rational decision maker compares pairs of confidence values  $h,b$  and  $h^{\prime},b^{\prime}$ , the rate of positive outcomes  $Y = 1$  for each pair may appear contradictory with the magnitude of confidence. In what follows, we will show that, if  $f_{B}$  satisfies a natural alignment property with respect to  $f_{H}$ , which we refer to as human-alignment, there always exists an optimal AI-assisted decision policy that is monotone.
4 AI-Assisted Decision Making Under Human-Aligned Calibration
Intuitively, to avoid that pairs of confidence values  $B$  and  $H$  appear as contradictory to a rational decision maker, we need to make sure that, with high probability, both  $f_{B}$  and  $f_{H}$  are monotone after conditioning on specific values of  $H$  and  $B$ , respectively. Next, we formalize this intuition by means of the following property, which we refer to as  $\alpha$ -alignment:
Definition 4 (Human-alignment). A confidence function  $f_{B}$  satisfies  $\alpha$ -alignment with respect to a confidence function  $f_{H}$  if, for any  $h \in \mathcal{H}$ , there exists some  $\tilde{S}_{h} \subseteq S_{h}$ , with  $S_{h} = \{(x,H) \in \mathcal{Z} \mid H = h\}$  and  $|\tilde{S}_h| \geq (1 - \alpha /2)|S_h|$ , such that, for any  $b^{\prime}, b^{\prime \prime} \in [0,1]$  and  $h^\prime , h^{\prime \prime} \in \mathcal{H}$  such that  $b^{\prime} \leq b^{\prime \prime}$  and  $h^\prime \leq h^{\prime \prime}$ , it holds that
$$
P (Y = 1 \mid f _ {B} (X, H) = b ^ {\prime}, (X, H) \in \tilde {\mathcal {S}} _ {h ^ {\prime}}) - P (Y = 1 \mid f _ {B} (X, H) = b ^ {\prime \prime}, (X, H) \in \tilde {\mathcal {S}} _ {h ^ {\prime \prime}}) \leq \alpha \tag {6}
$$
The above definition just means that, if  $f_{B}$  is  $\alpha$ -aligned with respect to  $f_{H}$  then, for any  $h, h' \in \mathcal{H}$ , we can bound any violation of monotonicity by  $f_{B}$  between at least a  $(1 - \alpha / 2)$  fraction of the subspaces of features  $S_{h}$  and  $S_{h'}$ . Moreover, note that, if  $f_{B}$  is 0-aligned with respect to  $f_{H}$ , then there are no violations of monotonicity, i.e.,  $P(Y = 1 \mid f_{B}(X, H) = b', (X, H) \in \tilde{\mathcal{S}}_{h'}') \leq P(Y = 1 \mid f_{B}(X, H) = b'', (X, H) \in \tilde{\mathcal{S}}_{h''})$ , and we say that  $f_{B}$  is perfectly aligned with respect to  $f_{H}$ .
Given the above definition, we are now ready to state our main result, which shows that human-alignment allows for AI-assisted decision policies that satisfy monotonicity and (near-)optimality:
Theorem 5. Let  $\mathcal{M}$  be any AI-assisted decision making process satisfying Eqs. 2 and 3, with an utility function  $u(T,Y)$  satisfying Eq. 1 If  $f_{B}$  satisfies  $\alpha$ -alignment w.r.t.  $f_{H}$ , then there always exists an AI-assisted decision policy  $\pi \in \Pi(H,B)$  that satisfies monotonicity and is near-optimal, i.e.,
$$
\mathbb {E} _ {\pi^ {*}} [ u (T, Y) ] \leq \mathbb {E} _ {\pi} [ u (T, Y) ] + \alpha \cdot \left[ u (1, 1) - u (0, 1) + \frac {3}{2} (u (0, 0) - u (1, 0)) \right] \tag {7}
$$
where  $\pi^{*} = \operatorname{argmax}_{\pi \in \Pi(H, B)} \mathbb{E}_{\pi}[u(T, Y)]$  is the optimal policy.
Corollary 1. If  $f_{B}$  is perfectly aligned with respect to  $f_{H}$ , then there always exists an AI-assisted decision policy  $\pi \in \Pi (H,B)$  that satisfies monotonicity and is optimal.
Finally, in many high-stakes applications, we may like to make sure that the confidence values provided by  $f_{B}$  are both useful and interpretable [34]. Hence, we may like to seek for confidence functions  $f_{B}$  that satisfy human-aligned calibration, which we define as follows:
Definition 6 (Human-aligned calibration). A confidence function  $f_{B}$  satisfies  $\alpha$ -aligned calibration with respect to a confidence function  $f_{H}$  if and only if  $f_{B}$  satisfies  $\alpha$ -alignment with respect to  $f_{H}$  and it satisfies  $\alpha$ -calibration with respect to  $\mathcal{Z}$ .
In the next section, we will show how to achieve human-alignment and human-aligned calibration via multicalibration, a statistical notion introduced in the context of algorithmic fairness [11].
5 Achieving Human-Aligned Calibration via Multicalibration
Multicalibration was introduced by Hébert-Johnson et al. [11] as a notion to achieve fairness in supervised learning. It strengthens the notion of calibration by requiring that the confidence function is calibrated simultaneously across a large collection of subspaces of features  $\mathcal{C} \subseteq 2^{\mathcal{Z}}$  which may or may not be disjoint. More formally, it is defined as follows:
Definition 7 (Multicalibration). A confidence function  $f_{B}:\mathcal{Z}\to \mathcal{B}$  satisfies  $\alpha$ -multicalibration with respect to  $\mathcal{C}\subseteq 2^{\mathcal{Z}}$  if  $f_{B}$  satisfies  $\alpha$ -calibration with respect to every  $S\in \mathcal{C}$ .