Human-Aligned Calibration for AI-Assisted Decision Making
Authors: Nina L. Corvelo Benz
Manuel Gomez Rodriguez
Venue: NeurIPS 2023 poster (2023)
Keywords: Calibration
Trustworthy Machine Learning
Human-Centric ML
Probabilistic Models and Methods

Abstract: Whenever a binary classifier is used to provide decision support, it typically provides both a label prediction and a confidence value. Then, the decision maker is supposed to use the confidence value to calibrate how much to trust the prediction. In this context, it has been often argued that the confidence value should correspond to a well calibrated estimate of the probability that the predicted label matches the ground truth label. However, multiple lines of empirical evidence suggest that decision makers have difficulties at developing a good sense on when to trust a prediction using these confidence values. In this paper, our goal is first to understand why and then investigate how to construct more useful confidence values. We first argue that, for a broad class of utility functions, there exists data distributions for which a rational decision maker is, in general, unlikely to discover the optimal decision policy using the above confidence values—an optimal decision maker would need to sometimes place more (less) trust on predictions with lower (higher) confidence values. However, we then show that, if the confidence values satisfy a natural alignment property with respect to the decision maker’s confidence on her own predictions, there always exists an optimal decision policy under which the level of trust the decision maker would need to place on predictions is monotone on the confidence values, facilitating its discoverability. Further, we show that multicalibration with respect to the decision maker’s confidence on her own prediction is a sufficient condition for alignment. Experiments on a real AI-assisted decision making scenario where a classifier provides decision support to human decision makers validate our theoretical results and suggest that alignment may lead to better decisions.

Human-Aligned Calibration for AI-Assisted Decision Making
Nina L. Corvelo Benz
Manuel Gomez Rodriguez
Max Planck Institute for Software Systems ETH Zürich
Max Planck Institute for Software Systems manuel manuel@mpi- swss. org
ninacobe@mpiissw..
Abstract
Whenever a binary classifier is used to provide decision support, it typically pro- vides both a label prediction and confidence value. Then, the decision maker is supposed to use the confidence value to calibrate how much to trust the prediction. In this context, has been often argued that the confidence value should correspond to well calibrated estimate of the probability that the predicted label matches the ground truth label. However, multiple lines of empirical evidence suggest that decision makers have difficulties at developing good sense on when to trust pre- diction using these confidence values. In this paper, our goal is first understand why and then investigate how to construct more useful confidence values. We first argue that, for a broad class utility functions, there exist data distributions for which rational decision maker is, in general, unlikely to discover the optimal decision policy using the above confidence values- an optimal decision maker would need sometimes place more (less) trust predictions with lower (higher) confidence values. However, we then show that, the confidence values satisfy natural alignment property with respect the decision maker's s confidence her own predictions, there always exists an optimal decision policy under which the level of trust the decision maker would need to place on predictions is mono- tone on the confidence values, facilitating its discoverability. Further, we show that multicalibration with respect the decision maker's confidence her own predictions a sufficient condition for alignment. Experiments on four different Al-assisted decision making tasks where classifier provides decision support real human experts validate our theoretical results and suggest that alignment may lead better decisions.
Introduction
In recent years, there has been an increasing excitement on the potential of machine learning models to improve decision making in variety of high-stakes domains such as medicine, education or criminal justice [1.3]. One of the main focus has been binary classification tasks, where classifier helps a decision maker by predicting binary label of interest using set of observable features [477] For example, in medical treatment, the classifier may help doctor by predicting whether a patient may benefit from treatment. In college admissions, it may help admissions committee by predicting whether a candidate may successfully complete an undergraduate program. In loan decisions, may help bank by predicting whether prospective customer may default loan. In all these scenarios, the decision maker-the doctor, the committee or the bank-aim to use these predictions, together with their this context, since the predictions are unlikely to always match the truth, it has been widely agreed that the classifier should also provide confidence value together with each prediction 8 9.
predictions, to take good decisions that maximize given utility function. In
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

While the conventional wisdom is that the confidence value should be well calibrated estimate of the probability that the predicted label matches the true label [TO] multiple lines of empirical evidence have recently shown that decision makers have difficulties at developing good sense on when to trust prediction using these confidence values [7.19]. Therein, Vodrahali et al. [7 have shown that, in certain scenarios, decision makers take better decisions using uncalibrated probability estimates rather than calibrated ones. However, a theoretical framework explaining this puzzling observation has been missing and it is yet unclear what properties we should be looking for to guarantee that confidence values are useful for Al-assisted decision making. In our work, we aim to bridge this gap.
Our contributions. We start by formally characterizing Al-assisted decision making using a structural causal model (SCM) [20], as seen in Figure I Building upon this characterization, we first argue that, if decision maker rational, the level of trust she places on predictions will be monotone on the confidence values- she will place more (less) trust on predictions with higher (lower) confidence values. Then, we show that, for broad class of utility functions, there are data distributions for which a rational decision maker can never take optimal decisions using calibrated estimates of the probability that the predicted label matches the true label as confidence values. However, we further show that, if the confidence values a decision maker uses satisfy natural alignment property with respect to the confidence she has on her own predictions, which we refer to as aumannalignment, then the decision maker can both be rational and take optimal decisions. In addition, we demonstrate that human- aamannalignmmnt can be achieved via multicalibration [_. a statistical notion introduced in the context of algorithmic fairness. In particular, we show that multicalibration with respect to the decision maker's s confidence on her own predictions is a sufficient condition for human- aligammellnmmt..... Finally, validate our theoretical framework using real data from four different Al-assisted decision making tasks where a classifier provides decision support to human decision makers in four different binary classification tasks. Our results suggest that, comparing across tasks, classifiers providing human-aligned confidence values facilitate better decisions than classifiers providing confidence values that are not human-aligned. Moreover, our results also suggest that rational decision makers' trust level increases monotonically with the classifie'ss ' provided confidence.
Further related work. Our work builds upon a rapidly increasing literature on Al-assisted decision making (refer to Lai al. 21 for a recent review). More specifically, it is motivated by several empirical studies showing that decision makers have difficulties at modulating trust using confidence values 1719), as discussed previously. In this context, is also worth noting that other empirical studies have analyzed how other factors such as model explanations and accuracy modulate trust [22]- 26). However, except for a very recent notable exception [27], theoretical frameworks, which could be used to better understand the mixed findings found by these empirical studies, have been missing. More broadly, our work also relates to a flurry of recent work on reinforcement learning with human feedback (28-33), which aims to better align the outputs of large language models (LLMs) with human preferences. However, our formulation is fundamentally different and our technical contributions are orthogonal to theirs.
2 A Causal Model of AI-Assisted Decision Making
We consider an Al-assisted decision making process where, for each realization of the process, decision maker first observes a set of features (x, u) E X x V. then takes a binary decision t E {0, 1} informed by classifier's s prediction ý = argmax, fy (ar), well confidence (x())) [0. 1]. of binary label of interest y E {0, 1}. and finally receives utility u(t, y) E R. Such Al-assisted decision making process fits a variety of real-world applications. For example, in medical treatment, the features (x, v) may comprise multiple sources of information regarding a patient's 's health the label may indicate whether a patient would benefit from a specific treatment, the decision i may indicate whether the doctor applies the specific treatment to the patient, and the utility u(1, y) may quantify the trade-off between health benefit to the patient and economic cost to the decision maker.
In what follows, rather than working with both ÿ and fz)), we will work with just b fi (z), which we will refer to as classifier's 's confidence, without loss of generality Moreover, we will assume that
'Our formulation allows for subset of the features to be available only the decision maker but not the classifier.
"e can recover ý and fz(x) from b, f.e., if 6 > 0.5, we have that ğ = 1 and fz(x) = b: if 6 < 0.5, 9 = and fa( 1 b.

---

Al's confidence
Features
B
70% Decision
0
Data
DM's
generating
D
confidence
'mid"
Utility
process
U
Individual
Label
characteristics
W
Figure I: Our structural causal model M. Orange circles represent endogenous random variables and blue boxes represent exogenous random variables. The value each endogenous variable is given by function of the values of its ancestors in the structural causal model, as defined by Eqs. 2 and The value of each exogenous variable is sampled independently from a given distribution.
the utility u(t, y) is greater if the value of i. and y coincide, i.e.,
(I) condition that we think it is natural under an appropriate choice of label and decision values. For example, in medical diagnosis, if 1 means the patient is tested early for a disease and = 1 means the patient suffers the discase, the above condition implies that the utility of either testing patient who suffers the disease or not testing a patient who does not suffer the disease are greater than the utility of either not testing patient who suffers the disease or testing patient who does not suffer the disease. In condition [ we allow for a non-strict inequality u00, 0) 2 u(0, 1) because, in settings in which the label Y is only realized whenever the decision t (e.g.. in our previous example on medical treatment, we can only observe a treatment is eventually beneficial or not if the patient is treated), it has been argued that, whenever O, any choice of utility must be independent of the label value [4 i.e., w0, 0) u(0, 1) = u(O).
u(1, 0), u(1, v(0 1), u(0,0) u(1,0), and u(0, 2 w( 1),
Next, we characterize the above Al-assisted decision making process using structural causal model (SCM) [20), which we denote as M. The SCM M is defined by a set of assignments, which entail a distribution PM and divide naturally into two subsets. One subset comprises the features and the label ie.,
X V= fv and Y= fy(),,
where D is an independent exogenous random variable, often called exogenous noise, characterizing the data generating process and fx, fy and fy are given function The second subset comprises the decision maker and the classifier, i.e.,
(3) where fir and fn are given functions, which determine the decision maker's confidence H and classifier's s confidence B that the value of the label of interest decision policy, which determines the decision maker's decision T, is given utility function, which determines the utility U, and Q and W are independent exogenous variables modeling the decision maker's s individual characteristics influencing her own confidence // and her decision T, respectively. By distinguishing both sources noise, we allow for the presence uncertainty on the decision T even after conditioning fixed confidence values ht and b. This accounts for the fact that, in reality, a decision maker may take different decisions for instances with the same confidence values h and b. For example, in medical treatment, for two different patients with the same confidence and b, a doctor's 's decision may differ due to limited resources.
H = fu fx,
B= HH,,HH T=x(H,B,W) and U= (T,
1, æ is a given Al-assisted
In our SCM M, the decision maker's s confidence H refers to the confidence the decision maker has that the label Y = before observing the classifier's confidence B. Moreover, following previous behavioral studies showing that human', confidence H is discretized in few distinct levels [32 33 we assume takes values h from a totally ordered discrete set H. We say that the decision maker's
P We denote random variables with capital letters and realizations of random variables with lower case letters.
'Our model allows both for causal and anticausal features 3I).
3

---

confidence fH is monotone (with respect to the probability distribution P(Y = 1)) if, for all h, h H. such that h S h', it holds that P(Y = =) P
H h'). Further, we allow the classifier's s confidence B to depend on the decision maker's s confidence H because this will be necessary to achieve human-alignmen via multicalibration in Section [ However, our negative result in Section 3 also holds if the classifier's s confidence fx(X, ,, H) [x only depends on the features, as usual in most classifiers designed for Al-assisted decision making. In the remainder, we will use Z = (X, Hf) and denote the space of features and human confidence values as 2 X H.
Figure T shows a visual representation of our SCM MM.
Under this characterization, we argue that, if a rational decision maker has decided under confidence values b and h. then she would have decided t' 2 t had the confidence values been '' 2 and h' 2 while holding "everything else fixed" [20]. For example, in medical treatment, assume a doctor's s and a classifier's s confidence that a patient would benefit from treatment is decides to treat the patient, then we argue that, if the doctor is rational, she would have treated the patient had the doctor's and the classifier's s confidence been b' h' 0.8 > 0.7. Further, we say that any Al-assiste decision policy π that satisfies this property is monotone, i.e.,
0.7 and the doctor
Definition 1 (Monotone A Al-assisted decision policy). An Al-assisted decision policy x is monotone if and only if. for any b, b' E [0, 1] and h, h E H such that b S and h S h', it holds that x (h, b, w) S π (h', b., ww) for any ~ pM
Finally, note that, under any monotone Al-assisted decision policy, it trivially follows that
B H = h, B = b) ET H = h1, B = b].
(4)
where the expectation is over the uncertainty on the decision maker's individual characteristics and the data generating process.
3 Impossibility of AI-Assisted Decision Making Under Calibration
In Al-assisted decision making, classifiers are usually demanded provide calibrated confidence values [10-16]. A confidence function fB : 2 - [0, 1] is said to be perfectly calibrated if, for any b [0, it holds that P(Y (calibration) data, one can only hope to construct approximately calibrated confidence functions. There exist many different notions of approximate calibration, which have been proposed over the years. Here, for concreteness, we adopt the notion of a-calibratiorf introduced by Hébert-Johnson al. [TT] however, our theoretical results can be easily adapted to other notions of approximate calibration
(z) b) = b. Unfortunately, using finite amounts of
Definition 2 (Calibration). A confidence function fB Z + [0. 1] satisfies a-calibration with respect to S S Z if there exists some S' C S, with S'| 2 (1 a)]S], such that, for any b E [0, 1], it holds that
I(Y = fz(Z) = b, Z S') bl a.
(5)
If the decision maker's s decision T only depends on the classifier's ' confidence B, i.e., x (H, B, W) (B) and fa satisfies a-calibration with respect to Z, then, it readily follows from previous work that, for any utility function that satisfics Eq. [ simple monotone Al-assisted deci- sion policy Tp that takes decisions by thresholding the confidence values is optimal (47). i.e., T argmax (BB) E, [u[T, Y)). where the expectation is with respect to the probability distribu- tion P. and (B) denotes the class of Al-assisted decision policies using B. However, one of the main motivations to favor Al-assisted decision making over fully automated decision making is that the decision maker may have access to additional features V and may like to weigh the classifier's s confidence B against her own confidence H. Hence, the decision maker may seek for the optimal decision policy πt over the class 11(H, B) of Al-assisted decision policies using I and B, i.e., n argmax,eu(HH.) [T,,, Y)). since it may offer greater expected utility than T
Unfortunately, the following negative result shows that, in general. a rational decision maker may be unable to discover such an optimal decision policy n² using (perfectly) calibrated confidence values and this is true even if J is monotone:
$ Note that, if a = 0 and S 2, the confidence function f is perfectly calibrated.
All proof's can be found in Appendix

---

Theorem 3. There exist (infinitely many) Al-assisted decision making processes M satisfying Eqs. 2 3 with utility functions u(T, Y) satisfying Eq. 7 such that fn is perfectly calibrated and fHr is monotone but any Al-assisted decision policy x E HI, B) that satisfies monotonicity is suboptimal, i.e, E. (T][(T,Y))) < TEz-(ui (TT]auTTYY)
In the proof of the above result in Appendix A.2 we show that there always exist a perfectly calibrated fz(Z) fi:=fB(22 (X, H) that depends on both X and H for which any monotone Al-assisted decision policy is suboptimal. This is due to the fact that fo (2) is calibrated on average over H, however, may not be calibrated, nor even monotone, after conditioning on specifie value H h. Further, also show that, even if (2) = PM = 1 X) matches the true distribution of the label Y given the features X, which has been typically the ultimate goal in the machine learning literature, there always exists a monotone fH for which any monotone Al-assisted decision policy is suboptimal. This is due to the fact that the decision maker's s confidence I/ can differ across instances with the same value for features X because it also depends on the features V and noise Q. Hence, I'H may not be monotone after conditioning on a specific value X I . In both cases, when rational decision maker compares pairs of confidence values h, b and h',b', the rate of positive outcomes
Y = 1 for each pair may appear contradictory with the magnitude of confidence. In what follows, we will show that, if fB satisfies a natural alignment property with respect to fH. which we refer to as human- alignmee,, there always exists an optimal Al-assisted decision policy that is monotone.
Al-Assisted Decision Making Under Human-Aligned Calibration
Intuitively, to avoid that pairs of confidence values B and H appear as contradictory to a rational decision maker, we need to make sure that, with high probability, both fn and [n are monotone after conditioning on specific values of H and B, respectively. Next, we formalize this intuition by means of the following property, which we refer to as a-alignment:
Definition 4 ailuman-alignmente A confidence function f' satisfies a-allinment with respect to confidence function fu if. for any h E H. there exists some Sp. C Sh with S₁ = {(xx H) E Z |
H- = h and |S,l > (1 a//2)[Sh), such that, for any V. E [0, 1] and h², h² E H such that b' - D" and h' S h"," it holds that
P(Y = | fa(X, b', (X, E Sw) - PPY = fa(X, = =
6,, (X, E Swre S a (6)
The above definition just means that, if fa is a-aligned with respect to J then, for any h, h' E H. can bound any violation of monotonicity by fn between at least a (1 a/2) fraction of the subspaces of features S₁, and Sy. Moreover, note that, if fa is O-aligned with respect to fu, then there are no violations of monotonicity, i.e., P(Y = 1 fn(X, H) = U/ (X, H)- E Sw) S P(Y = 1 fB(X, H) = 8", H) E S, and we say that fø is perfectly aligned with respect to fH.
now ready to state our main result, which shows that human- alignment allows for Al-assisted decision policies that satisfy monotonicity and (near- joptimmtttml
Given the above definition,
Theorem S. Let M be any Al-assisted decision making process satisfying Eqs. 2 aand 3 with an utility function u(T, Y) satisfying Eq. If fa satisfies a-alignment w.r.t. SH, then there always exists an Al-assisted decision policy E IHH,. B) that satisfies monotonicity and is near-optimal, i.e.
Ex-(u(T, S E. (u(T, a [ulL,)) w(0, 3 (u<0.0) (0.,0))u(1100) (=(0,0)-u(1.00) u(1,0))
where x² = Aargmax.em(E)
E)), is the optimal policy.
Corollary 1. If fa is perfectly aligned with respect to fu, then there always exists an Al-assisted decision policy E I[H,, B) that satisfies monotonicity and is optimal.
Finally, in many high-stakes applications. may like to make sure that the confidence values provided by fa are both useful and interpretable [34]. Hence, we may like to seek for confidence functions In that satisfy human-all human-aligned calibration, which we define follows:
Definition (Human-aligned calibration). A confidence fanction I' satisfies a-aligned calibration with respect to confidence function I' if and only if I'n satisfies a-alignment with respect to J' and it satisfies a-calibration with respect to Z.
In the next section, we will show how to achieve human-all aaigmmmt and numan--ligned calibration via multicalibration. a statistical notion introduced in the context of algorithmic faimess [I]].
5

---

5 Achieving Human--Alined Calibration via Multicalibration
Multicalibration was introduced by Hébert-Johnson et al. [ as a notion to achieve fairness in supervised learning. It strengthens the notion of calibration by requiring that the confidence function is calibrated simultaneously across large collection of subspaces of features C C 2 which may or may not be disjoint. More formally, it is defined as follows:
Definition (Multicalibration). A confidence function fB Z + B satisfies a-multicalibration with respect to C 2 if fs satisfies a- a-calibration with respect to every S C.
Then, we can show that, for an appropriate choice of C, if fs satisfies a-multicalibration with respect C, then it satisfies a-aligned calibration with respect to fH. More specifically, we have the following result:
Theorem 8. fn satisfies (a/2) multiiaaiiaaiinnnoon with respect to (Seh with Sn = {(xx H) e 2 H = hi), then In satisfies a-aligned calibration with respect to fn-
The above theorem suggests that, given a classifier's confidence function fB. we can multicalibrate f' with respect to (Sr hee to achieve a-aligned calibration with respect to fH- To achieve multicalibration guarantees using finite amounts of (calibration) data, multicalibration algorithms need to discretize the range of f'B [9 [[]). In what follows, we briefly revisit two algorithms, which carry out this discretization differently, and discuss their complexity and data requirements with respect to achieving a-aligned calibration.
Multicalibration algorithm via X-discretization. This algorithm. which was introduced by Hébert- Johnson et al. (.). discretizes the range of fB, i.e., the interval [o, 1], into bins of fixed size À with values O0, 1] ( 2 .....,, -
3A
Let = b -b/2 ((a,h) E S, (z) E A(6)), with E A 1]. It iteratively updates the confidence values of function fB for these groups until fa satisfies a discretized notion of a -umuutiicllibration over these groups. The algorithm then returns a discretized confidence function fB.,(Z, h) = E (X, H) [n((X, H) A(b)], with b AO0, 1] such that [n( h) E A(b). which is guaranteed to satisfy (a' + x)-multicalibration Refer to Algorithm 1 in Appendix for a pseudocode of the algorithm.
The algorithm partitions each subspace S, into 1/A groups Sk,x =
Then, as a direct consequence of Theorem 8 we can obtain a (discretized) confidence function fB. that satisfies a-aligned calibration by setting a'
a /4. However, the following proposition shows that, to satisfy just a-alignment, it is enough to set a' = a > a/4 and X = a/4:
The discretized confidence function fB, returned by Algorithm 7 |atiisiiss (2a' + X)- alignment with respect to fH-
Proposition 1.
Finally, it is worth noting that, to implement Algorithm [ we need to compute empirical estimates of the expectations and probabilities above using a calibration set D. In this context, Theorem 2 in Hébert- Johnson et al. [I] shows that, if we use a calibration set of size (log // with ((X, H) E Sx) > 7 for all H. then fB, is guaranteed to satisfy a-multicalibration with probability at least 1 - in time 0(H))- poly(1 (1/a,) 1/7)).
Multicalibration algorithm via uniform mass binning. Uniform mass binning (UMD) 9 has been originally designed to calibrate fo with respect to Z using a calibration set D. However, since the subspaces (Sa hEw disjoint, i.e., S₁ O S& = 0 for every h o h², we multicalibrate ja with respect to (Sx ) NE by just running [P(] instances of UMD, each using the subset of samples UMD to achieve multicalibration because,
D n Sn. Here, would like emphasize that we in our setting, the subspaces {Sa, hEC are disjoint.
Each instance of UMD discretizes the range of fn, i.e., the interval [0, 1J. into N 1/A bins with values A, 1) (P(Y
fn(X, h) [0, (1)))..... p(y 1 fn(X, h) (Âv 11 âx))). where à denotes the (i/N)-th empirical quantile of the confidence values f h) of the samples (z. h) E D n Sr. and P denotes an empirical estimate of the probability using samples from D n Sn. aswell. Here, note that, by construction, the bins have similar probability mass. Then, for each (ac, h) E 2, the corresponding instance of UMD provides the value of the discretized confidence function JT, h) = where b E (0 1] denotes the value of the bin whose corresponding defining interval includes fz h). Finally, we have the following theorem, which guarantees that, as

---

1.0-
Human Condieenc
1.2 Human Cortidencn
1o.0:10.0. 0.25)
lowh J.D. 0.37)
0.8
Irath (8.25, 0.74
(mar: (.P))
00.76,
0.8-
0.4-
04-
0.2-
0.125
0.26
0.570
0.6
3.625
0.76
0.875
1.0
0.125
0.25
0.575
0.5
0.625
0.76
0.876
1.0
Model Cortidence.
Model Conficence,
Art
(b) Sarcasm
1.0
Human Confidenca
1.0
Corfidence,
b0.0 0.25)
'low' JUD. 0.32]
Irist (0.25, 0.69!
'mid: (2.52, 0.64)
00
64, 1.0)
n.K-
0.8
0.4-
0.4
0.2
0.0
0.125
0.26
0.370
0.5
3.625
0.75
0.875
1.0
0.725
0.25
0.570
0.5
0.625
0.76 0.676
1.0
Model Cortidence.
Model Conficence.
(c) Cities
(d) Census
Figure 2: Empirical estimate of the probabilities P(Y =1 (X,Y) E S,,x,)), where b E A[0, 1] and h E {low, mid, high} are the discretized confidence values for the classifiers and human participants, respectively. Error bars represent 90% confidence intervals and hatched bars mark alignment violations between confidence pairs (h, b) with |S,,) 2 30.
long as the calibration set is large enough, the discretized confidence function fB. satisfies a-aligned calibration with respect to fu with high probability:
Theorem 9. The discretized confidence function fB. returned by [H] instances of UMD, one per Sn, satisfies a-aligned calibration with respect fH with probability least - as long as the size of the calibration set 'D =0
[HL log
[H]]
with P((X, H) E Sx)) 2 7
XE
Experiments
In this section, we validate our theoretical results using a dataset with real expert predictions in an Al-assisted decision making scenario comprising four different binary classification tasks]
Data description. We experiment with the publicly available Human-Al Interactions dataset [35 The dataset comprises 34,783 unique predictions from 1,088 different human participants on four different binary prediction tasks ("Art", "Sarcasm", "Cities" and "Census"). Overall, there are approximately 32 different instances per task. In the "Art" task, participants need to determine the art period of a painting given two choices and, overall, there are paintings from four art periods. In the "Sarcasm" task, participants need to detect if sarcasm is present in text snippets from the Reddit sarcasm dataset [36]. In the "Cities" task, participants need to determine which large US city is depicted in image given two choices and, overall, there images of four different US cities. Finally, in the "Census" task, participants need to determine an individual earns more than 50k a year based certain demographic information in tabular form. For "Sarcasm", is representation of the text snippets and we set = if sarcasm is present, for "Art" and "Cities", x a representation of the images and set summarizes demographic information and we set
and y 0 at random for each different instance and, for "Census".
if an individual carns more than 50k a year. In each of the tasks, human participants provide confidence values about their predictions before (h) and after (1441) receiving Al advice from a classifier in form of the classifier's confidence values b The original dataset contains predictions by participants from different, but overlapping, sets
We release the code reproduce our analysis at https://gttubb.coo//eewwwssss/// wtw.://ooss.aaaainnnnuauaaaa aligned
calibration
RRfer to Appendix C for more details the dataset.

---

Table 1: Misalignment, miscalibration and AUC.
Misalignment MAE
Miscalibration ECE
AUC
Task
EAE
MCE
THH
Art
4.5 10
0.058
0.084
0.186
86.7%
72.7%
82.0%
Sarcasm
3.8 10-3
0.224
0.085
0.310
89.9%
82.5%
86.5%
Cities
6.2 10-6
0.013
0.066
0.158
84.4%
79.0%
84.7%
Census
9.0 10
0.298
0.109
0.270
80.0%
77.3%
79.9%
of countries across tasks, who were told the Al advice had different values of accuracy In our experiments, to control for these confounding factors, we focus on participants from the US who were told the Al advice was 80% accurate, resulting in 15,063 unique predictions from 471 different human participants.
Experimental setup and evaluation metrics. For each of the tasks, first measure (i) the degree of misalignment between the classifiers' confidence values b and the participants' confidence values h before receiving AI advice b and (ii) the difference (h+A1 h) between the human participant's s confidence values before and after receiving Al advice Then, we compare the utility achieved by a Al-assisted decision policy H.w that predicts the value of y by thresholding the humans' confidence values h4AT after observing the classifier's confidence values against two baselines: (i) a decision policy æB that predicts the value of y by thresholding the classifier's confidence values and (ii) a decision policy that predicts the value of by thresholding the humans' confidence values before observing the classifier's confidence values.
To measure the degree of misalignment, we discretize the confidence values b and h into bins. For the classifiers' confidence b, we use 8 uniform sized bins per task with (centered) values A(0, 1], where x = 1/8. For the human participants' confidence h before receiving Al advice b, we use three bins per task ('low', 'mid' and 'high'), where set the bin boundaries that each bin contains approximately the same probability mass and set the bin values to the average confidence value within each bin. In what follows, we refer to the pairs of discretized confidence values (h, as cells, where samples (x, y) E Z whose confidence values lie in the cell (h, b) define the group Sh,x()). and note that we choose rather low number of bins for both b and h so that most cells have sufficient data samples to reliable estimate several misalignment metrics, which we describe next.
We use three different misalignment metries: (i) the number of alignment violations between cell pairs, (ii) the expected alignment error (EAE) and (iii) the maximum aligmment error (MAE). There is an alignment violation between cells pairs (h, b) and (h' U), with h S h' and b < V,
P(Y 11(X, Y) Sn,(01) > P(Y = 11(X, Y) E Sw.xn)).
Moreover, we have that:
EAE- Σ
[P(Y- (X,Y) E SnA(N)) P(Y =1 (X,Y) E Sworm))4
lih
MAE = max
P(Y =1 (X,Y) E SA,) P(Y = |(,Y) E SRA)))),
bgh
where N = 11h < W, 8. Here, note that the number of alignment violations tells us how frequently is the left hand side of Eq. possitie across cell pairs given S₁ Sh, and the EAE and MAE quantify the average and maximum value of the left hand side of Eq. across cells violating alignment. To obtain reliable estimates of the above metrics, we only consider cells (h, b) with [Sn,A(0) 30 samples. Moreover, we also report the expected calibration calibration error (MCE) 12 37.. which are natural counterparts to EAE and MAE, respectively.
(ECF) and maximum
As measure of utility, we estimate the true positive rate (TPR) and false positive rate (FPR) of the decision policies TB, и and HH for all possible choices of threshold values, which we summarize using the area under the ROC curve (AUC) and, in Appendix C
we also report ROC curves.
Partiicipants were also either told that the advice is from a "Human" or from an "Al" based on a random assignment of participants to a treatment or control group. Since the actual advice received in both groups was identical for the same instance and the "perceived advice source" is randomized, we use data from both treatment and control groups in the experiments.
8

---

0.6
gl (0cm E Suai
02-
0.4
0.2-
-22
-02-
Humen Confocentss h
Humee Confidencee
0.44
Tow: 0.25) 10.25,
04
Text (0.0, 0.37)
mid:00.37
Night
-08
hight: (0.76, 1.0
0.126
0.25
0.576
0.5
625
0.75
0.675
1.0
0.126
0.25
0.376
0.5
0.620
0.76
0.676
1.0
Modal Contidence
Modal Confidence
Art
(b) Sarcasm
0.6-
0.4-
0.2
0.0-
-02-
Mumse Confictntos I
-02-
Humee Confilenco
0.4-
Tow: (3.0, 0.25) Trieh 10.25,
-044
Tewt (0.0, 0.32)
mith 0.64)
-0.6-
hight 1.0
0.125
0.25
0.576
0.5
0.625
0.75
0.675
1.0
0.126
0.25
0.376
0.5
0.620
0.76
0.676
1.0
Modal Contidence
Model Confidence,
(c) Cities
(d) Census
Figure 3: Empirical estimate of the average difference E[/,AI - h I (X,Y) E Sk,N where E O, and h E [low, mid, high} are the discretized confidence values for the classifier and human participants, respectively. Error bars represent 90% confidence intervals and hatched bars mark alignment violations between confidence pairs (h, b) with (Sn,)()) 2 30.
Results. We start by looking at the empirical estimates of the probabilities P(Y=11 =11 (X,Y) Sp,x(i) ) and of our measures of misalignment (EAE, MAE) and miscalibration (ECE, MCE) in Figure 2 and [left and middle columns). The results show that, for "Cities", the probabilities P(Y
(X,Y) E S),A()) are (approximately) monotonically increasing with respect to the classifier's confidence values More specifically, as shown in Figure 2 there is only one alignment violation between cell pairs and, hence, metrics of misalignment acquire also very low values. In contrast, for "Art", "Sarcasm" and especially "Census", there an increasing number of alignment violations and our misalignment metries acquire higher values, up several orders of magnitude higher for "Census". These results also show that misalignment and miscalibration go hand in hand, however, in terms of miscalibration, "Census" does not stand up strongly.
Next, we look at the difference 14A1 - between the human participant's recorded confidence values before and after receiving AI advice b across samples in each of the subsets Sn.x(u) induced by discretized confidence values used above. Figure summarizes the results, which reveal that the difference h4А1 h increases monotonically with respect the classifier's confidence b. This suggests that participants always expect to reflect the probability of a positive outcome irrespectively of their confidence value h before receiving AI advice, providing support for hypothesis that (rational) decision makers implement monotone Al-assisted decisions policies. Further, this finding also implies that, for "Art", "Sarcasm" and "Census". any policy *H., that predicts the value of the label by thresholding the confidence value h44I will be necessarily suboptimal because the probabilities P(Y
(X S1,)()) are monotone increasing with b.
Finally, we look at the AUC achieved by decision policies TB тH and TH Table I (right columns) summarize the results, which shows that NH outperforms consistently across all tasks but it only outperforms TB in a single task ("Cities") out of four. These findings provide empirical support for Theorem 3 which predicts that, in the presence of human- auman-alignment violations those observed in "Art", "Sarcasm" T and "Census". any monotone Al-assisted decision policy will be suboptimal, and they also provide support for Theorem which predicts that, under human-alignment, there exist near-optimal Al Al-assisted decision policies satisfying monotonicity.
7 Discussion and Limitations
In this section, we discuss the intended scope of our work and identify several limitations of our theoretical and experimental results, which may serve starting points for future work.

---

Decision making setting, We have focused on decision making settings where both decisions and outcomes are binary. However, we think that it may be feasible to extend our theoretical analysis to settings with multi-categorical (or real-valued) outcome variables and decisions. One of the main challenges would he to identify which natural conditions utility functions may satisfy in such settings. Further, we also think that it would be significantly more challenging to extend our theoretical analysis to sequential settings-multienn multiccalibration in sequential settings is an open area of research- but our ideas may still be useful starting point. In addition, our theoretical analysis assumes that the decision makers aim to maximize the average utility of their decisions. However, whenever human decisions consequential to individuals, the decision maker may have faimess desiderata.
Confidence values. In our causal model of Al-assisted decision making, we allow the classifier's confidence values to depend on the decision maker's s confidence values because this is necessary to achieve human-aligeh alignmttt via multicalibration as described in Section [5] However, we would like to clarify that both Theorems 3 and still hold if the classifier's s confidence values do not depend typically the status quo today. Looking into the future, our work questions this status quo by showing that, by allowing the classifier's s confidence values to depend on the decision maker's confidence values, decision maker may end up taking decisions with higher utility. Moreover, we would also like to clarify that, while the motivation behind our work is Al-assisted human decision making, our theoretical results do not depend on who-be it a classifier another human-givee gives advice. As long as the advice comes in the form of confidence values, our results are valid. Finally, while we have shown that human- numan-alignment can be achieved via multicalibration, we hypothesize that algorithms specifically designed to achieve human-alignmen may have lower data and computational requirements than multicalibration algorithms.
on the decision maker's 's confidence,
Experimental results. Our experimental results demonstrate that, across tasks, the average utility achieved by decision makers is relatively higher if the classifier they use satisfies human- aamannaliggnmnt. However, they do not empirically demonstrate that, for a fixed task, there is an improvement in average utility achieved by decision makers if the classifier they use satisfies The reason why we could not demonstrate the latter is because, in our experiments, we used an observational dataset gathered by others (35). Looking into the future, it would be very important to run human subject study to empirically demonstrate the latter and, for now, treat our conclusions with caution.
auman--lignment.
Conclusions
We have introduced a theoretical framework to investigate what properties confidence values should have help decision makers take better decisions. We have shown that there exists data distribution for which rational decision maker using calibrated confidence values will always take suboptimal decisions. However, we have further shown that, the confidence values satisfy natural alignment property, which can be achieved via multicalibration, then rational decision maker using these confidence values can take optimal decisions. Finally, we have illustrated our theoretical results using real human predictions on four Al-assisted decision making tasks.
Acknowledgements.. We would like to thank Nastaran Okati for fruitful discussions at early stage of the project. Gomez-Rodriguez acknowledges support from the European Research Council (ERC) under the European Union's is Horizon 2020 research and innovation programme (grant agreement No. 945719).
References
[1] Wei Jiao, Gurnit Atwal, Paz Polak, Rosa Karlic, Edwin Cuppen, Alexandra Danyi, Jeroen de Ridder, Carla van Herpen, Martijn P Lolkema, Neeltje Steeghs, et al. A deep learning system accurately classifies primary and metastatic cancers using passenger mutation patterns. Nature communications, 11(1::1-12, 2020.
[2] Jacob Whitehill, Kiran Mohan, Daniel Seaton, Yigal Rosen, and Dustin Tingley. Mooc dropout prediction: How to measure accuracy? In Proceedings of the fourth (2017) acm conference on learning@ @ scale, pages 161-164, 2017.
[3] Julia Dressel and Hany Farid. The accuracy, fairness, and limits of predicting recidivism. Science advances, 4():ccaa05580 2018.
10

---

[4] Sam decision making and the cost of fairness. Proceedings of the 23rd aem sigkdd international conference on knowledge discovery and data mining, pages 797-806, 2017.
Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. Algorithmic
[5] Niki Kilbertus, Manuel Gomez Rodriguez, Bernhard Schölkopf, Krikamol Muandet, and Isabel Valera. Fair decisions despite imperfect predictions. In International Conference on Artificial Intelligence and Statistics, pages 277-287. PMLR, 2020.
[6] Isabel Valera, Adish Singla, and Manuel Gomez-Rodriguee Enhancing the accuracy and fairness of human decision making. In Advances in Neural Information Processing Systems, 2018.
[7] Guy N Rothblum and Gal Yona. Decision-making under miscalibration. In Innovations in Theoretical Computer Science, 2023.
[8] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, 2017.
[9] Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Proceedings of the 18th International Conference Machine Learning, 2001.
[10] Tilmann Gneiting, Fadoua Balabdaoui, and Adrian Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 2007.
[11] Ursula Hébert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibra- tion: Calibration for the (computationally-idenniee iidetiiiieie masses. In Proceedings of the 35th International Conference on Machine Learning. 2018.
[12] Chirag Gupta and Aaditya K. Ramdas. Distribution- free calibration guarantees for histogram binning without sample splitting. In Proceedings of the 38th International Conference Machine Learning, 2021.
[13] Roshni Sahoo, Shengjia Zhao, Alyssa Chen, and Stefano Ermon. Reliable decisions with threshold calibration. Advances in Neural Information Processing Systems, 34:1831-1844, 2021.
[14] Shengjia Zhao, Michael Kim, Roshni Sahoo, Tengyu Ma, and Stefano Ermon. Calibrating predictions to decisions: A novel approach to multi-class calibration. In Advances in Neural Information Processing Systems, 2021.
[15] Yingxiang Huang, Wentao Li, Fima Macheret, Rodney A Gabriel, and Lucila Mano-Machado.
A tutorial on calibration measurements and calibration models for clinical prediction models. Journal of the American Medical Informatics Association, 27(4):621-63: 2020.
[16] Lequn Wang, Thorsten Joachims, and Manuel Gomez-Rodriguez, Improving screening pro- cesses via calibrated subset selection. In Proceedings of the 39th International Conference on Machine Learning. 2023.
[17] Kailas Vodrahalli, Tobias Gerstenberg, and James Zou. Uncalibrated models can improve human-ai collaboration. In Advances in Neural Information Processing Systems, 2022.
[18) Gal Yona, Amir Feder, and Itay Laish. Useful confidence measures: Beyond the max score. arXiv preprint arXiv:2210 14070, 2022.
[19] Eleni Straitouri, Lequn Wang, Nastaran Okati, and Manuel Gomez- Gomez-Rodriguez. Improving expert predictions with conformal prediction. In Proceedings of the 40th International Conference on Machine Learning, 2023.
[20] Judea Pearl. Causality. Cambridge university press, 2009.
[21] Vivian Lai, Chacha Chen, Q Vera Liao, Alison Smith-Renner, and Chenhao Tan. Towards science of human-ai decision making: a survey of empirical studies. Proceedings of the ACM Conference Fairness, Accountability, and Transparency, 2023.
11

---

[22] Andrea Papenmeier, Gwenn Englebienne, and Christin Seifert. How model accuracy and explanation fidelity influence user trust. arXiv preprint 17XV:1907.12252, 2019.
[23] Xinru Wang and Ming Yin. Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making. In 26th International Conference Intelligent User Interfaces, pages 318-328, 2021.
[24] Ming Yin, Jennifer Wortman Vaughan, and Hanna Wallach. Understanding the effect of accuracy on trust in machine learning models. In Proceedings of the 2019 chi conference human factors in computing systems, pages 1-12, 2019.
[25] Mahsan Nourani, Joanie T. King, and Eric D. Ragan. The role of domain expertise in user trust and the impact of first impressions with intelligent systems. ArXiv, abs/2008. .0..009100 1000 2020.
[26] Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 295-305, 2020.
[27] Chacha Chen, Shi Feng, Amit Sharma, and Chenhao Tan. Machine explanations and human understanding. Transactions of Machine Learning Research, 2023.
[28] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 2017.
[29] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances Neural Information Processing Systems, 2022.
[30] Banghua Zhu, Jiantao Jiao, and Michael Jordan. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. arXiv preprint arXiv:2301.112700 2023.
[31] Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. In Proceedings of the 29th International Coference on International Conference on Machine Learning, 2012.
[32] Matteo Lisi, Gianluigi Mongillo, Georgia Milne, Tessa Dekker, and Andrei Gorea. Discrete confidence levels revealed by sequential decisions. Nature Human Behaviour, 5(2):273-280, 2021.
[33] Hang Zhang, Nathaniel D) Daw, and Laurence T Maloney. Human representation of visuo-motor uncertainty as mixtures of orthogonal basis distributions. Nature neuroscience, 18(8):1152- 1158, 2015.
[34] Umang Bhatt, Javier Antorán, Yunfeng Zhang, Q Vera Liao, Prasanna Sattigeri, Riccardo Fogliato, Gabrielle Melançon, Ranganath Krishnan, Jason Stanley, Omesh Tickoo, et al. Un- certainty form of transparency: Mcasuring, communicating, and using uncertainty. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 401-413, 2021.
[35] Kailas Vodrahalli, Roxana Daneshjou, Tobias Gerstenberg, and James Zou. Do humans trust advice more if it comes from ai? an analysis of human-ai interactions. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pages 763-777, 2022.
[36] Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli. large self-annotated corpus for sarcasm, arXiv preprint a70iv:1704.000
2017.
[37] Telmo Silva Filho, Hao Song, Miquel Perello-Nieto, Raul Santos-Rodriguez Meelis Kull, and Peter Flach. Classifier calibration: How to assess and improve predicted class probabilities: a survey, arXiv e-prints, pages arXiv-2112 12, 2021.
12

---

A Proofs
A.1
Additional Lemmas
Lemma (Monotonicity). If a utility function u satisfies Eq. 7 then u is monotone with respect to the probability that Y = 1, i.c. for any P. P' P((0, 1}) such that P(Y = 1) S P'(Y 1), holds that Ey [u(1 (1, Y)] & (3((,
Proof. We readily have that
Bym,)u(1,y)) = P(Y =) (1 P(Y -l))0)))))))
S P = 1) u(1,1) (1 P'( = -))))((1,0))
=y [u(1, (ul,,))) Y)],
where, in the above inequality, we use that u11, 1) > u)1,0) and P(Y = P'(Y = 1).
Lemma (Trivial policies are not always optimal). If a utility function satisfies Eq. 7 then there
exist P. P' P({0, such that the trivial policies that either always decide T = or always
decide = 0 are suboptimal. particular, for any P. P' E P((0, 13) such that P(Y 1) < c and
P' = 1) > C, where
u(0,0 u11
E (0,1),
(8)
w(1, - u), w/0, 0) u00,
it holds that
Ey (1,,)u(1, < By (u(0, Y)]
and By (u(1, > Ey (u(0
(9)
Proof. Let be any distribution such that
u(0,0) =(1,0)
P(Y= 11 < u(1, 1) u(1,0) ((0,0) u00
where E (0,1) because, by assumption, satisfies Eq. Now, by rearranging the above inequality, we have that
P(Y = 1) w1,, 1) + PPY 1)) =(1,0) < P(Y 1) mt + (1 P(Y = 1))) u(,,
and, using the definition of the expectation, it immediately follows that
Ey [u(1, 1 Y)] < EEyr(uu(, Y)).
The same argument can be used to show that, for any distribution P' such that P'(Y = 1) > it
holds that Ey (11, Y)] Ey-P [u(0, Y)]. Finally, note that, since (0, 1), know that such distributions P and Pr exist.
A.2 Proof of Theorem
Before proving Theorem we rewrite the expected utility with respect to the probability distribution p.M in terms of confidence H and B by using the law total expectation,
Ez [u[,T Y)] BH. BT..B~MM(HHB))
(T] B]]
Here, to simplify notation, we will write
EH E., (T, H, H]]
where note that, using the law of total expectation, we can write the inner expectation in the above expression in terms of the utilities of the trivial policies, i.e.,
[T( H. B] Elu(L,Y) H. H. PT B) + E|u(0, Y) H, B) PA(T= H) (10)
and we will use to refer to probabilities induced by SCM M, egg. P(H, B) to denote pMM (H, B). Now, we restate and prove Theorem
Theorem 3, There exist (infinitely many) Al-assisted decision making processes M satisfying Eqs, 2 and 3 with utility functions u(T,Y) Y') satisfying Eq. T. such that Sn is perfectly calibrated and S is monotone but any Al-assisted decision policy æ E &I[HH B) that satisfies monotonicity is suboptimal, i.e., E, (T, Y)] < Eg- [u( Y)).
13

---

Proof. To prove the above claim, we construct monotone confidence function fH. perfectly calibrated confidence function fn and distribution PM for which any monotone Al-assisted decision policy E [H, B) achieves strictly lower utility than a carefully constructed non monotone Al-assisted decision policy से E [T, B).
We will present the proof in three parts, First, we will introduce the main building block and idea behind the proof by small construction of fH, f' and PM with [H] = [B] where B S [0. 1] denotes the (discrete) output space of the classifier's s confidence function. We then construct examples
of fH. fB and pM for arbitrary [H[] = k and [B] m with m, E N. m k 2 2. Lastly, we construct examples where B is non non-discrete and [7] k with k 2.
Main building block and small example.
We start by presenting the main idea of the proof using an example with small set of confidence
values H. and B. Let the values of the decision maker's s confidence H be in H. {h1 h2, hz) and the values of the classifier's s confidence B be in B = {bi, b2, b3}. with order h4 < (h; + 1) and b, < (6, + 1) respectively.
Our main building block, consists of two distributions P- Pt E P( 11) with P (Y =) 1) < c
and P+ (Y 1) > c, where с depends on utility as described by Eq. 8 in Lemma 2 We use these distributions for our constructions of fu, fB and p.M, so that for some realizations of H. B distribution P(Y = 1 H, B) is either P or P+. Using Lemma 2 and from Eq. 10 we have that:
(I) For any h4 b, such that P(Y | И = h4 B = 6,) = PT, it holds that
E[u(1, Y) И h; B b,] < E(u(0, Y) H h4, B = bi]. Hence, decreasing (T 1 | H, B) increases B[u(T, Y) H = -4;, B bi].
(II) For any h4 by such that P(Y H h4, B b,) pt. it holds that
E Elu(1,Y) H h, B = b,] E (0 Y) H h4, B = bi].
Hence, increasing PA 1 B) increases E Y( H -hi, B by].
Intuitively, suppose we now have that, for confidence values h22 b2. Y ~ p+ and, for confidence values h3, b2, Y P-, i.e., P(Y H H= h2g B by) P+ and P(Y H = B
Then, any non non-monotone Al-assisted decision policy ă with P4 = 1 = B= P= =1 H- ha, B b2) will have higher expected utility than any monotone Al-assisted decision policy given confidence values f12, by and h1,, b2. Finally, under an appropriate choice of distribution P(H, B), such non-monotone Al-assisted decision policies will offer higher overall utility in expectation.
We formalize this intuition with the following lemma:
Lemma 3. Let M be any Al-assisted decision making process satisfying Eqs. and 3 with utility function u(T, Y) satisfying Eq. L If fH fB and PM are such that there exists confidence values be E B, h4, hy E H. with h2 < h.j. which satisfy
P(H h; B b) 0,
P(H h2j == 0,
(1I)
P(Y H h4 B b) = p+ and P(Y H -h,, B b) P
for some distributions p- p+ with P- = 1) c and p+ p+(Y = =1) > c, where
(0 (00)-u(1,000 0)
(12)
-11,1)-m(1. - u1,00 + u(0. 0) (0
Then, for any monotone Al-assisted decision policy & E IHHH B). there exists an Al-assisted decision policy स E 11 BB which is not monotone and achieves a stricly greater utility than n, i.c. E, [n[ Y)] < Ea (Y)]
Proof. Let æ be a monotone Al-assisted decision policy, then it must hold that PT 1 H h,, B b) (T =1 H h,, B b) (see Eq. 4). Let से be an identical Al-assisted decision
14

---

and policy to т up to the decision for confidence values h b and hg. b. We distinguish between three cases.
Case 1: P+ = = h1 B b) < P+ =1 H = h2, B= =b. b).
Let the probability of I 1 under ने for confidence values ft,, b and hj, 6 be switched compared to T, i.e.,
(T 1 H = h4 B = P,, H = h2 == b), (= = H = h2. B =b6) = PT =1 H = h4; b). Then, is monotone, as Eq. I is not satisfied, and it holds that
(2 H =,,, b) PT H = - 1,, 1=
(T = 1 H - < PT H h 3=
=
As we decreased P(T properties (1) and ( it must hold that the expected utility of म given confidence values h,, 6 and hg. b is higher than the one of π. i.e.,
=hh, = b) and increased P(T = H = h, = b). by
E# (T(
H h; B = b) (T)
h4 B b] and = h,, b].
(13) (14)
Ex (T,
H hy, b (T,,)
Case < (T H h b) PT
H H = b) S 1.
Let the probability of T = under मे for confidence values hy, b be strictly lower compared to æ and be the same as ı for h,; b. Then, मे is not monotone, since by case assumption
P= = H = h4, B = b) Pa (7- = H hj, B = b) P= =1 = hj, B b) and the inequality in Eq. 14 holds by property U
Case 3: (T
H h, B = b) = H=
Let the probability of T under न for confidence values h,, b be strictly higher compared to x and be the same as π for h b. Then, # is not monotone, since by case assumption
PT = H = nn, B = b) (T = H = h, = b) < P+(T= = = h; B = b) and the inequality Eq. 13 holds by property (
As in all three cases at least one of the strict inequalities in Eqs. 13 or 14 holds and से is equivalent to (i.e., it has the same expected conditional utility) given any other pair of confidence values h E H. b' E B, we have that
Ea (T, Y) = E E [T, Y | B] > E[ H) B] = (u(,
Before proceeding further, we would like to note that we may also state Lemma 3 usiig h E H. b,, by E B, with < bj. the proof would follow analogously.
Now, we construct an Al-decision making process M. with H = (h1, b2 h2) and B (b), by by such the decision maker's confidence fy is monotone, the classifier's confidence fø is perfectly calibrated, and the conditions of Lemma 3 are satisfied. First, let fa. fB and P. be such that
3/6 if =1
2/6 if 2
P(fb(Z)=b)
1/6 if j 3
otherwise
if ドj
P(H -H h, B = Bbbb) Px,v h Te(Z)=b))
0 otherwise.
Then, it readily follows that P(H = h4 B = by) = 1/6 for 2 j and P(H = h4 B = -b,) = 0 otherwise. Moreover, for each pair of confidence values (h;, by) with positive probability P(H
h4, B= = by), we set
{ P if or (i = 3 and {1,3})
P(Y =1 H h; B by)
if (j and i 3) or (j and (1,2)),
15

---

b8
P
p+
b9
P
pt
b3
b6
P
P
p+
55
p+
P
b2
p+
P
b4
P
P
b3
p+
P
b2
p+
P
b
P
P
p+
b
P
11
h2
h1
h1
112
hg
h4
hg
h6
Figure 4: Nonzero values of P(Y = 1 -4, B by) and P(H = h; B = bj) for every h; E H
and by E B used in the first (left) and second (right) part of the proof of Theorem 3 In each cell
(hi, bj) in both panels, p+ or p- is the value of P(Y = 1| = hi, B bj) and lighter color means lower value of P(H h4, = bj). where white means P(Y = 1/h h,, B B=
- bj) 0 and P(H, B) = 0. In both panels, the assignment of values is very stylized to facilitate the proof-the classifier's confidence function fe partitions the feature space in a way such that rational decision maker is unable to take decisions that maximize utility for almost all confidence values. However, less stylized examples also satisfy the conditions of Lemma 3 For example, as long as there is one triplet of confidence values b2, f2, hg (or fig, b2, by in the left example) for which rational decision maker is unable to take decisions that maximize utility, Lemma can be applied.
as shown in Figure 4 (left). Then, it readily follows that fH is monotone with respect to the probability that Y = 1, i.e., P(Y =1 H h.) P(Y = H h,+ )), and we have that the classifier's confidence values
bj =
Σ WiZj
P(H = h4 B=
3=bb b,) P(Y = H = h1 B = bj)
2/3. P +133.P P+
if
1/2 P + 1/
if = 2 jf
pt
0
otherwise
are perfectly calibrated and satisfy that b5 < bj+1.
Finally, using Lemma 3 with = b2, h4 = l1z, h2 = 13s we have that any monotone Al-assisted
decision policy is suboptimal for any M with J Jn and P.M. as defined above.
Construction with arbitrary [H] = and [ m, m k: 2 2.
In this second part of the proof, we construct an Al-assisted decision making processes M, with such that k 2 such that the decision maker's ss confidence fu is
['H] & and |B monotone, the classifier's confidence fa is perfectly calibrated and the conditions of Lemma are satisfied.
First, the space of confidence values be H = her and = jemm) with order h; < h and b; < lit1, respectively, and fu, I' and p. be such that P(fn(Z) = by) = 1/m and
if =
m
if = 1,j k
P(H = h4 B= b,) = Px fB(Z) = bb))
if if S k (15)
1
if k otherwise.
0
16

---

Moreover, for each pair of confidence values (h by) with positive probability P(H = h4s B = b,),
we set
P i =
p if 1,j > k
P(Y = H = B=
(16)
pt j = +1j k
p+ = > k
as shown in Figure (right). Further, we set the classifier's confidence values by
by m m-
1-1. .
P
m
m
Then, it holds that by < bi+1 and [n is perfectly calibrated
P(Y =1 B b,) { P(H = hJ P(H = h1
B bb,) b P + P(H h5-1 B BB p+
if k
B B= b P(H h2 B B= bU) p+
if >
and thus, using the definitions of P(H B) and P(Y | H, B), we have that P(Y BB= = by) = by.
To show that fH is monotone with respect to the probability that Y = 1. first note that P(H = h4, B 6.) decreases as increases and P( h4 B b(+1) increases as increases. Moreover, further note that P(Y = 1 H h4 B = b,) P < P(Y = 1 H = h, B bi+1) = Pt. Hence, for any E {2, 12...., 1). it readily follows that
P(Y 1 H =hh p+. P(B b h.) + P P(B b =4) h4)
- P(Y = 1 H = h141).
and, for i 1, it evident that P(Y = H ==h P(Y =1 H h2).
Finally, using Lemma 3 with any choice of confidence values b = bj. h; = h--1 and h2 = h2 with j E (2...., k). we have that any monotone Al-assisted decision policy is suboptimal for any M with |H| = k and [8] = m, m > k 2 2, and fH. fa and PM as defined above. Here, note that, as we do not fix the exact distributions p and p+, the above Lemma applies infinitely many Al-assisted decision making processes M.
Construction with B S [0, 1] and |H| k.
In this last part of the proof, we construct an Al-assisted decision making process M, with |7|| k > 2 and B S [0. 1], such that the decision maker's confidence function fu is monotone, the classifierss s confidence function fB is perfectly calibrated and the conditions of Lemma B are satisfied.
First, 1et the space of confidence values be = (hx Ne(g. with order h4 < h4+1, the feature space X [0, 1]. and f be two strictly monotone increasing functions with
f [0, + () and ft : [0.1] + (c, 1],
(17)
where
" (0,0) - -00,0)u(1100
(18)
w(1 1) u(1,0) u00. 10
Further, let Qk+1 {go, 91,- lk 4x+1) be set of quantiles such that P(X S q;) (/ for all j E {0, 1,... 1s + 1} and thus, we have that, for all j E [k + 1].
for = (01-1,9). it holds that P(X 11)
k+1
Now, let fH and PM be such that
1/2
if fi (j-1
Py = h4 X,X
!O
ifi=j=1o or (i = k and j = k ++1 +1) otherwise.
(19)
For a more general feature space X, we can use mapping á of X to [0, 1]. The proof works analogously by substituting X with o(X).
17

---

X
4
1
F()
2
f*())
4
h
h2
h3
Figure 5: Nonzero values of P(Y 1|X, H = h4 X E Ij) for every h4 E H. with | = 3, and
I (qj-1, qj). with qj E Q4 used in the last part the proof of Theorem 3 Lighter color means lower value of f or f
and let
P(Y =1 1 1 X X X,, (x) if i (i=j=1) (xt(x)
(20)
if = = orr k and j = +
as shown in Figure
Next, we define
(X)
if E 11
= PQ
(+(X)
x E Tu+1 otherwise,
(X)
which, by construction, is perfectly calibrated.
To show that the decision maker's confidence function IH monotone with respect to the probability we have that
that V = 1, we first note that, using Eq. 19
1/2
if < k and j E {i, + 1) and
if j
P(X --h+ hi)
(21)
if and j = + 1 otherwise.
Hence, using Eq. 21 and the law of total probability, for any i ie (2,... Kk - 22), we have that
P(Y = H h2)
P(Y = H xe P(Y = = h, X k+1)l
2
<
f (q)) f((++)))
[f- (inf (,11 + f+ (inf [+22)
< [P(Y 1 H = h X E I411) + P(Y =1 H --h11 E I(42)
2
= P(Y = H = h+41),
where the inequalities follow from the fact that f and 1 are strictly monotone increasing. Corner cases for i and k can be shown analogously by further using that (X) < < (x) for all X.
18

---

Finally, using Lemma 3 with any choice of confidence values h; h h2. j f2. . and fX) with X ,, we have that any monotone Al-assisted decision policy suboptimal for any M with B C [0, 1] and [H] k. k and fH. fB and pM as defined above.
A.3 Proof of Theorem
We prove the statement by contraposition. Let M be an Al-assisted decision making process satisfying Eqs. 2 and with utility function u(T, Y) satisfying Eq. I and let M be such that In satisfies a-alignment with respect to Sи and f' has output space - [0, 1]. Assume there exists no (near-)optimal monotone Al-assisted decision policy for utility u. Thus, there must exist an optimal Al-assisted decision policy E II(H, B3) which is not monotone and has strictly greater expected utility than any monotone policy. However, we show that we can modify decision policy ă E IH, B) with near-optimal expected utility.
monotone Al-assisted
As is not monotone, there must exist confidence values h1, h2 E H. h1 I h2, and bj by B, by < b2, such that
T (hh, bi, w) > (h22
b2, w) for some w W,
(22)
where W denotes the space of noise values. In what follows, let Wol containing any such w and let (T(T,h,bb))
C W denote the set h1.01
Un, (r...12,bb) hub
For any confidence value H', V' E H x [], we modify policy to policy á as follows. Let {5 MEH denote the sets satisfying the a-alignment condition for fB with respect to fH and, given confidence h', let by denote the smallest confidence value of fB. such that there exist h S h' with P(Y 1 bw Z E Sm) 2 C, i.e.
bw == min B P(Y == b, Sn) 2 c for h S h
(23)
Now, we define a new Al-assisted policy from π as follows,
if b 2 by. and w E Un UnSw.belBnad selsooelBrro W((n,11)))
a(h v))
if b' < by and Unzu be bwr) VV(T)sh))) otherwise.
(24)
T U' w)
Next, we show that á is monotone and E, (T, Y)] 2 (u a for some constant a. Proof स is a monotone assisted policy.
To prove that á E ITH, B) is monotone Al-assisted decision policy, we show that, for all h', N² E H, H, 6" E B, with s h", 8 < 6". it holds that (y(*.." "a""""") 0. We distinguish between three cases.
-Case 1: 6' > by and 6" 2 b
Since h/ s h1," b' S 8" and, by definition, bav S bw since h S R2, we have that
U
Vil... C
U
high be [r
lish"bbe [E(bbrr.b
Hence, we can conclude that
* *W, (,, 1 ==="(h" "0",,""' w) for wll
U
(25)
NSN" be [b
Further, for any other w E W Un Be( W(T.)) S W
-W "i(TT."".")
we have that
(h', b' w) = m(h,, b. w) and a(h", 8", w) =hh., 8", w) and, by definition of A 8²) follows that
*(h', U, w) < a(h",", U,," w) for all w E W
U
(26)
NSN" [bAM.ON)
19

---

From Eqs. 25 and 26 it follows that V....""""" (8") 0.
Case 2: b' < by and 6" 2 bjg
By definition of it, we have that
*(h', b', w) = = *(h", 6", w) for all w e
(27)
Wigh" we
[bar,
and
a(h, wx = 0 v # "h"",'' 8", w) for all w E
U
(28)
Analogously to case 1, since the values of below are also in W W/ "o""""""" ("") and is equivalent to π for these values, we have that
*(h',b' w) S *(h", "b" w) for all w W
U
U
W((...))
ASM ASMM,bElBw (6
k BeW,bElu.br
(29)
From Eqs. 27 28 and 29 it follows that (VRA (8")"
Case 3: b' bw and b" < bpr.
Since h' < h², b < 8" and, by definition, Bar S bw since h s h², we have that
U
V C
U
Viln..)))
k2hM,bE (e(
hihh be(o,,
Hence, we can conclude that
* *h', ,,, w) = 0 #(h" ;" w) for all E
U
(30)
k2th/ Saa.bbE be(( (br.
Again analogously to case 1, since the values of w below are also in W WW
and t is
equivalent to for these values, we have that
*('' b', w) S Ah"" b" w) for w W
(31)
N2W bElV bur)
30 and it follows that (R.A""8"
From Eqs.
Note that, we cannot have case where b 2 bw and 6" < bar, as this would imply 6" < b. Since, in all three possible cases, we have shown that W "".""".""" monotone.
'1' = 0. we can conclude that # E II(H, B)
Proof स is near optimal.
First, we rewrite the inner expectation in Eq. as
E, [u[[ Y) H, B = Eju(0)
H, B] + (E(u(1,Y) | H, B]
-E(u(O, Y) H, BJ) P*(T =1 H)
Further, recall that |5a1 N (1-a/2) a/2)[Sal for all h E H and, for all W',h"' e H, h $ h" and all b, E [0, 1), D' b", we have that
P(Y
[n(Z)= = b', 2 E Sw P(Y = fn(Z) 8" 2 S.r) S a
(32)
Now, for any H. 6' E B, we show an upper bound on E, [u[T Y) | Hl = HBB = 8l Ea [TT, Y) H h1, B b'.. We distinguish between three cases.
Case 1: b' > by and P(Y 1 H = h' B 8') 2
20

---

Using Lemma 2 we have that
(B|u(L,Y) H =
= B 8T) 0
(33)
Moreover, 6' 2 bw. the distribution of positive decisions in क may also increases for h', R compared to π (see Eq. 24 i.e.
(T = H -
P = B 50 0
Hence, it follows that
Ex(u(T,Y) H = W/
= B =b] -b][=((((T(T) H = N1 B= =b]
Y)] = Y) H = b(])
(34)
x (P (T =1 H = P1 H = b)) S..
Case 2: 6' 2 by and P(Y = 1 H - - V) <.
Since V/ 2 bi. there exists h, b E H. x B, with h S h1, b S U' such that P(Y = B = b, z
S,) 2 Moreover, using the definition of o-alignment, we have that
P(Y = 1 B = b, Z S,) s P(Y = 1 B = b', Z E Sn) +
(35)
Then, we can use this to lower bound the expected utility of = 1 given B = b' and Z E S as follows:
Y[u(1, B b, Z e 8n] - E[, B B= -,
= u(1,1) (P(( = 11 B = b. (,) P(Y = 1 B =B Si²)
(36)
+
1 === - - - =1 B = b, E S₁))
((..)))).. a.
where the last inequality due to Eq. 35 and the assumption that u(1, 1) - u(1, 0) > 0. Analogously. we can also upper bound the expected utility of T 0 given // B b' and Z E Sp follows:
E[u(O, Y) | = b, Z E 5.] E(w Y) B -3 11 Sw.]
= u(0,1))(PP)) = 11 B = b. S.) - P(Y = 1 B = b', 2 E Si))
(37)
(P(Y =0 =
B Sw) P = B= = E S.))
2 (u(0,1) - =m00, a.
where the last inequality holds due to Eq. 35 and the assumption that u00, -0. 0) < 0.
Now, P(Y = 1
B 6, 7 S.)) by Lemma 2 we have that
E Y
B = b. Z E S. 2 El B = b, Z E Sn
(38)
Combining Eqs. 36
37 and
38
obtain
E| Ylu(L,Y)
B = 8', 2 E Sw] + =u(u(1,))-u((1)))
- u(1,0)
(39)
Y) B = b', Sw.1 + (( 0))
In addition, note that we have following trivial bound for the expectation when H h but Z $ Sw
51 u1, u(0, 1) Y) и h1,
(1, 0) < El) H H N'
(40)
(0
(41)
Moreover, since b' > by, the distribution of positive decisions in म may also increase for h²,8 compared to i.e.. (T
H P4 =1 -
so
Hence, we have that
E, (T(TTYY)))))) Y) H = hi, B -x H
H = - 81), (42)
S (-1) (B]u(L,Y) H= -1
21

---

where the inequality follows since B (1, Y)
H B B - | = = h², B 6) N
by Lemma 2 as P(Y
H
= h', B 8) <.
Finally, combining Eqs.
39
40
and 42 and using the law of total expectation, we obtain
Eg (I,
| H = b [u( H h², =
1 (1 -
B b - Y) B B Swel)
+ P(rr, (B|u Y) H = - H == - b1)
< (1 - (a - + + u0, 1)) + (,,, (u(0, 0) u11, 0)), (43)
where B(tr,)) denotes the probability of Z & Sn given H h1, B 6, i.e. Blac P(z $
Sw = R ===b),) b').
-Case 3: 6' < by:
For all h1, b, with h < lt' b S U, we have that P(Y B = bbb E S,) < c. In particular,
P(Y =1 B v, Z Sw) Thus, by Lemma
Biu(1, Y) B = b1, Z E Sw (l Y) B = U' 2 E Sw
(44)
In this case, since b' < by, the distribution of positive decisions in Ωे may decrease for h b compared to π, i.e.,
0 < Pa
H h, B = b) (T = H h, B
Combining Eqs 44 40 and 41 and using the law of total expectation, we obtain
B, E. Y)
H = h', B = b - Ex(u(T,)) h',
- H = B-)) -1)- 1 - (1) H 1= N', =1,
(1 - Senv ./) ( (B(u(1,) B 5 Y) B b', Z Swl)
+ Benn
(I0,,)) (E/u
H N1, B- 81 (u)
H = h2, W, B1)
S (u (u(1,1)-u((,1))
(45)
where again ((v,))) P(Z & Sw H W B=
Now, for a fixed h E H. since S 2 (1- a //)) we know that 0 S EiES ((wrb)) S aa2. Hence, combining Eqs.
43 and 45 from the three cases above, we have that
[B(En[u(TTY) H h', B b']] (u(T,) H = h B= b']
EB E, (T)
H = - и = hh,
<
+00 1)) (u(0,,)-u(1.))
(u(,,1)-u(0,1))) 1))
2
3
w(0 1) a ()
2 (u(0,0) - u(1,0))).
Finally, since by assumption π is optimal, ie., [=(u(T.Y) we can conclude by the law of total expectation that [M(T (e(H) E.
E. (-- =
E, [u( (T, Y) = En EB Ey, TI [u Y) H.
3 +a- (a(( u(0, 1) + (u(0. u(1,
< E Y)] a-
This concludes the proof.
A.4 Proof of Theorem g
If fB is a/2-mulltccalirrate with respect to {Sh)ner then, by definition, for any h E H. there exists
Sp, C Sn with |S] (1 a/2)]S,l such that, for any b [0, 1]. it holds that
IP(Y 1 fa(Z) = b, Z E S.)) - bl a/2.
22

---

This directly implies that, for any h²'h"' E H and ,', E [0, 1]. we have that
P(Y = 1 fa () = 8', S - b' - P(Y =1 fz(() = ,b" 22 S aa (46)
and, using linearity of expectation, we further have that
P(Y = (z b', Sw) - P(Y =1 fz( = 8", E S.x') S + + 8 b², (47)
showing that, whenever 8' - 8", the a-alignment condition is met. This proves that fo is a-aligned with respect to -n-
Finally, if Ja is iu/2-multicalibrrated with respect to {Sa hEH, then, it is a e2a-calibrrtee with respect to any of the sets S₁. Since 2 Un this implies that fB is a /2acalibbatted with respect 2. This concludes the proof.
A.5 Proof of Proposition
Given discretization parameter x, Algorithm namely (a, A)-multicalibration:
1 works with a discretized notion of a-multicalibration,
Definition 10. Let C 2 be collection of subsets of Z. For any a, À > O, confidence function
fB : 2 + [0, is (a, mumuiiicallbrree with respect to C if, for all S E C, b E A(O, and all
SL (9) such that |Sn,x(a]] 2 Sp), it holds that
[X, H) - P(Y =1 H. | (X, E Sa S a.
(48)
Here, we can analogously define a discretized notion of a-alignment, namely (a, A)-alignment.
Definition 11. For a, À > O, a confidence function f'= Z + [0, 1] is (a, A)-aligned with respect to IH if. for all h² E H. h S ht, and all U E A(0, 1]. R S b", with (Sn))(1)) > a/22 X Swi
and (Snr)( > a/2 X A|S we have
P(Y (X, H) E Sw:))))) P(Y =1 I (x, H) Sp (w0))) aa
(49)
In what follows, we first show that (a, -)m-mtuticaaliirttiin with respect to (Sn)nen implies (2a X, Allaliimment with respect to fH.
Theorem 12. For a > O, if fn is (a, A)-muiiicallbbrueed with respect to {Sh]neH then I' is (2a + X, A)-aligned with respect to Ju.
Proof. If f is (a, A)-multicalibrated with respect hEHH then, by definition, for all h E H. b E A 1]. and all Sn.x(b) such that (S,, 2 a ). it holds that
HX) - P(Y = 11 X, H) (X, E Sn,A S a.
(50)
This directly implies that, for all N., W,h² E H. 8" E A(O, with (Sw.Acol 2 a. A|Sa[| and X(r)) a -|/Spr J. holds that
E (X, H) - P(Y = X, H) (X (pmm)
(51)
E[JB(X,H) - P(Y 1
X, H) (X, E SW(X(1) $ 20
and, using the linearity of expectation, we have that
P(Y 1 (X, H) E SW SW. P(Y = 1 H) E (wi)))))) S2a + B[f (X. E Sw.) Elfa(X, H) I (X, E Swr,.oo).
(52)
Whenever 1' U", due to the X-discretization, we have that
II) I)) E Sw E[fB(X, I (X E Swi, 5
(53)
Hence, we have shown that if fn is a-multicalibratee, then for all N', hh" E H,b,8" E A 1] with
|SN (00) 2 | and (h0)) 2 >|S | we have
P(Y = (X, E Sw,,(r)) - P(Y = | (X, E SMm,)())) S 2a - X.
(54)
Further, note that (2a + 2)/2 A a A 0. This concludes the proof.
23

---

Next, we show that, if fB is (a, A)-alinned, then fB. is a-aligned with respect to fH. Theorem 13. For a, À > if fa is (a,, x)-aligned with respect to fu. then fB. is a-aligned with respect to fH-
Proof. The proof is similar to the proof of Lemma 1 in Hébert-Johnson et al. [T]. Consider all Sk,2 such that (p)) < a. By the x-discretization, there are at most 1/A such sets, thus, the cardinality of their union is at most 1/ 1/Aa\|Sa1 - alSa|. Hence, for all l E H. there exists a subset S₁. C S₁ with | 2 (1 aa]S such that, for all h', h² E H. with h' s h", and all b', 6. E A00 1]. with 6' S 6", it holds that
P(Y = 1 (x, H) E SW n. - P(Y 1 (X E Swn) nw.) sa (55)
The X-discretization sets all values of (z, h)) E Swr, to fB,(
([f fo H)
X(K)). Note that, for (xr, h) E _. f h) E Mb6) and for h) E Sw N000 (,,(a, h) e
X(b"), so it still holds that ([f H)) fB(X, H) E ((6I S E[fe(X, fB(X. H) e
(( Thus, using Eq. 55 we have that
P(Y = fx H) = B(f H) I (X, H) E A(0)), (X. E Sw))
(56)
P(Y 1 | fo H)) ((f | (X, H) X(b"),, (X, Swx)) S
This concludes the proof.
Finally, using Theorems 12 and 13 it readily follows that, given a parameter a', the discretized confidence function fB. returned by Algorithm I satisfies (2a' + X)-aligned calibration with respect to fH-
A.6 Proof Theorem g
We structure the proof in three parts. We first explain the calibration guarantee that UMD provides and how it relates to human- -aliane calibration. Then, we derive a lower bound on the size of the subsets 2 n n& so that the discretized confidence function fB.> satisfies a-aligned calibration with respect fu with high probability. Finally, building on this result, we derive an upper bound on [D] so that f.A satisfies a-aligned calibration with high probability as long as there exists 0 so that P((X, H) E (,)) 2 y for all h E H.
Conditional Calibration implies Human-Aligned Calibration. Running UMD on a dataset D E (Z x y))", where each datapoint is sampled from p.M. guarantees (a, -coondicntii calibration, a PAC-style calibration guarantee [1]]. Given a dataset D, a confidence function I's satisfies (o, E)- conditional calibration if, with probability least 1 over the randomness in D,
vb [0, 1], IP(Y () H) = b) bl S a.
This stands in contrast the definition of a-calibration, which requires only that the confidence fx(X, H) is at most a away from the true probability for 1 - a fraction of Z.
Similarly, using an union bound over all h E H. (a/2, (/1H)) ( calibration of fa on each Sn, h H. implies that, with probability at least { over the randomness in D. fB satisfies that
Vh H, vb vb E 1], |P(Y- 1lf H) b, H h) bl S a/2.
(57)
Hence, analogously to the proof of Theorem 8, this implies that, with probability at least 1 E over the randomness in D, fa also satisfies that
Yh, N' E H.h $ Ri, V E g, < b', P(Y = 1lfB (X,
(58)
PL -,
In summary, from Eqs. 57 and 58 we can conclude that (a/ ///o/i/---onii calibration of fB on each Sr. h H. implies that, with probability at least E, I' satisfies a-aligned calibration, where, for all h E H. have that Sr Sn.
Lower bound on 'DnS, Sn,| to achieve conditional calibration with UMD. Running UMD on each partition D n Sr, of 'D induced by h E H achieves (a/2, -oondiiiao calibration as long as each subset D S of the data is large enough. More specifically, the following lower bound on the size of the subsets D n SSS readily follows from Theorem 3 in Gupta et al. [2].
24

---

Lemma 4. The discretized confidence function fB. returned by [H] instances of UMD. one per Si. (a/2.E//fi))-condittn) calibrated on for any 5 E (0, if
2log
H2 8
-11)
1 A
SDNSAl Sm
Timin
+2
(59)
Proof. Let B denote the number of bins in UMD. Theorem 3 in Gupta et al. [2) states that, if fx(X, H) is absolutely continuous with respect to the Lebesgue measurd 11 and ( n S 2 2B, then the discretized confidence function output by UMD is ( conddiionally calibrated for any g' E (0, 1) and
log(2B/E)
(60)
V 2(L n StI/BJ
Then, for a given o, setting 6 = 02/2, B = [1/1] and g E/1H), we can solve Eq, 60 for the lower bound on (D n S 2 Temin with mi as defined Eq. 59
Upper bound on [D] to achieve conditional calibration with UMD, Suppose P((X, H) E S.)) 2 for all h E H. When H 2 2, we give an upper bound on [D] so that with high probability (DNSA Spl 2 Tmmin for all / E H.
In the process of sampling D E x 2)"" from pM, let Ri!h) denote the event that the i-th datapoint (I; h4, 91) has confidence value h, i.e,, h, = h. Then, we can express (D n Sy in terms of random variable ((h), defined as
[D]
R(O)
R!!)
(61)
t=1
Since I2(") is Bernoulli-disrrbbueed variable with P(R. (n)) P((X, H) Sz), the expected value
of (k) is p(h) [E[ P((X, H) E Sn) [D] N 7 [D].
Let D = 2- [H] (ogg2//)) 1/7 T.min observe that in this case
(( < Amin)) P (R(h) < 2 log(2/E) IPI)
For [7]] 2 and & E (0, 1), we have 1/(2/H 1og(2/E)) E (0, 1) and we can use a variation of the Chernoff bound to show
(M < nmmin
S (((N) <
2/H( log(2/E) m(hh)
S (1b) 21K lonfa/))) )²
c -m(b) + Tpाikeaar (919 Hnglt/e))
3 e M|--nin (1 11111110011110 wawpaaeeezettttaaaaoaaoo (0
where the first and last inequality results from using p(h) > 7 (D]. We can now use a union bound to obtain a lower bound on the probability that for any h E H. [D n.DDa Sy S Pmins i.e.
p(3h e H: |Dn Sy S mmmin) < E |7|
174 main
(62)
One can verify that for [H] > 2 and Mmmin 1, we have P(3h H: 1 < Temin) < f. Hence, [D] = 2 |7| (2/E) 1/7 Rmmin then, for all h E H. ID nS S 71 with probability E/2.
"I fa is not continuous with respect to the Lebesgue measure (or equivalently put, fa does not have probability density function), randomization trick can be used to ensure that the results of the theorem hold.
25

---

Combining this result and Lemma 4 we have that the discretized confidence function fa.x returned by [H] instances of UMD, one per Sh, is (a/2, E ( calibrated on each Sn, with probability at least 1 - e/2 for any & E (0, if
l2g(2/E)
2 log
4
[1
D 2 |H]
+
(63)
a²
Finally, using union bound, conclude that SB, achieves a-aligned calibration with respect to fur with probability at least & from
D = 0 (171)
log([H[/EM) a² A x
samples. This concludes the proof.
26

---

B Multicalibration Algorithm
In this section, we give high-level description of the post-processing algorithm for multicalibration introduced by Hébert-Johnson et al. []]. The algorithm works with a discretization of [0, 1] into
uniform sized bins of size X, for À > O. Formally the X-discretization of [O. 1], is defined as
Definition 14 (X-discretization Ii). Let x The X-discretization of [0. 1]. denoted by A(O, 1] { - }. is the set of 1/X eventy spaced real values over [0, 1]. For b E A 1). let
((b) = b x/2, X/2)
(64)
be the X-interval centered around b (except for the final interval, which will be [1 X.
starts by partitioning each subspace Sn, into 1/A groups Sk, {(x, h) Sn fa( h) X(b)), with b E Ao,, 1]. Then, it repeatedly looks for a large enough group Sp.2(0) such that the absolute difference between the average confidence value [[f H) (X,
Sp,x,s) and the probability P(Y = (X, H) E SH.vo)) is larger than and, if it finds it, it updates the confidence value (z h) of each (z, h) E Sh,a( by this difference. Once the algorithm cannot find any more such a group, returns a discretized confidence function fBA h) = E H) fx(X, H) X(b)), with b A(0, 1] such that (x, h) E X(b), which guaranteed satisfy (a + NJ-ulticallibraiinn
Algorithm provides a pseudocode implementation of the overall algorithm. Within the implementa- tion, it is worth noting that the expectations and probabilities can be estimated with fresh samples from the distribution from fixed dataset using tools from differential privacy and adaptive data analysis, as discussed in Hébert-Johnson et al.
Algorithm Post-processing algorithm for (a + m)muliiicalirraii
1: Input: confidence function fB. parameters a, À > O
2: Output: confidence function fB.
3: repeat
4: updated <- false
5: for Sr E
E A 1] do
6:
SRLAi) + Sn n if P((X, H) E SA,())) < ax. P((X, H) E S,) then
Z T)(x,h) X(b)}
7:
8:
continue
9:
bl,)(B) + E[f H) (X E Sn,
10:
TBLA(B) + P(Y =1 (X, H) Sn,201)
11:
if (h,a(b) b,,, > then
12:
updated < true
13:
for (ar, h) E Sh,)()) do
[n [1, h) + (n h)) (Th,200) bm.2(s)) (project into [0, 1] if necessary)
15:
until updated = false
for E A,0, do
17:
bx(e) + [X,n[X[ H))
18:
(or(x,)) E Z fo(I,h) E X(b) do
19:
[ h) bx())
20:
return fB.A
27

---

C Additional Details about the Experiments
Transformation of confidence values. In the Human-AI Interactions dataset, the Al model is simple statistical model where is just noisy average confidence of an independent set of ca. 50 human labelers on each task instance. Moreover, the confidence values were originally recorded on scale of [-1, IJ, where 1 means complete certainty on the correct true label and -1 means complete certainty on the incorrect label. To better match our theoretical framework, we transform all confidence values to a scale of [0, 1), where means complete certainty that the true label
1 and 0 means complete certainty that the true label is y # 1. More formally, let b, h, hLAI E [-1, be the original confidence values in the dataset, then we obtain b E [o, 1] via the following transformation:
(6 + 1)/2 (6 1)/2
y =
b =
if - 0,
and analogously for h and 14AT-
Comparing decision policies TB, HH and H.. Figure 6 shows the ROC curves for the decision policies B, TH and H.r in each of the four tasks in the Human-Al Interactions dataset.
1.0
1.0
0.8-
0.8
True Positive Rate
True Positive Rate
0.6
0.6
Decision Policies
0.4
Decision Policies
0.2
MH
MT
Th.
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate
False Positive
(a) Art
(b) Sarcasm
1.0
1.0
True Positive Rate
0.8
True Positive Rate
0.8
0.6
0.6
0.4
Decision Policies
Decision Policies
TH
Tl
TH
mH...
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate
False Positive Rate
(c) Cities
(d) Census
Figure ROC curves for the decision policies na, #H and H.
28


Reviews:
[{"review_id": "PklQOOHTXF", "reviewer": "Reviewer_iDbU", "summary": "The paper studies a binary decision-making process augmented by AI assistance. The decision maker optimizes a utility function satisfying certain natural conditions. The decision rule depends on the decision maker's (DM) confidence and a classifier's confidence. It is shown that there is a lack of alignment between DM's and classifier's confidences; the optimal rule might not have a natural property of placing more trust in predictions with higher confidence values. The paper then proves that a (near-)optimal decision rule with such a property exists if both confidences do not appear contradictory, with high probability, to the rational decision maker, a property dubbed \"human-alignment\". The paper connects this property with multicalibration algorithms for the classifier's confidence function. The paper illustrates the approach on real human data.", "questions": "It would be interesting to see how the Authors envision future work in the context of LLM-based systems, multi-categorical or sequential \ndecision-making.", "limitations": "The paper does not have a limitations section. Natural limitations include the setup's simplicity or the experiments' limited scope.\n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper introduces a theoretical framework to study optimal decision-making in binary problems, where the decision rule depends on the DM's and classifier's confidence.  The results like this are needed to understand the decision-making process when interacting with AI, particularly now, when strong AI systems are making their way into everyday lives.\nThe theory was illustrated with a real-life data experiment, where the paper asks questions about misalignment, the influence of AI confidence on human confidence values, and a comparison between decision policies.", "weaknesses": "The setup considered in the paper is very simple. It deals with binary decision problems and the experiment provided in the paper actually does not use any modern ML classifier (see footnote 8). \n\nSection 6 would significantly improved in terms of clarity. In particular,\n* In the data description part, it is not clear how the labels are defined. For instance, for the \"Cities,\" does y is random (line 283), or does it encode the correct answer (in which case, e.g., h='low' would correspond to a human having low confidence in the correct answer). In a similar vein, it would help to explicitly define how are empirical values of $\\mathbb P(Y=1|(X,Y)\\in S_{h,\\lambda(b)})$ estimated (line 319; what exact data are used).\n* Figures 2 and 3 use an \"alignment violations\" term, which is not defined.\n* Figures 2 and 3 should be better explained: each figure contains a set of four plots, each with from 8 to 10 sets of triplets of bars. In particular, the monotonicity in lines 318, 327 should be discussed in a more reader-friendly way.\n* The data used in the experiment were aggregated in various ways. The claims like \"support for our hypothesis that (rational) decision-makers implement monotone AI-assisted decisions policies\" should probably be treated with caution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful and insightful comments, which will help improve our paper. Please, find a point-by-point response below.\n\n**[Very simple setup]** Our theoretical results in Sections 3-4 and the multicalibration algorithms in Section 5 do not make any assumption about the classifier. Therefore, as long as the classifier provides a confidence value, we think our results are applicable to any modern ML classifier (e.g.,  deep neural networks). The reason why we did not use a modern ML classifier in our experiments is because we used an observational dataset gathered in a previous work and this work used the ML classifier described in footnote 8. The only way we see to address this limitation is to run a human subject study, however, this would require significant funding and time and, given the on-going discussions about the potential of human+AI collaboration, we think the NeurIPS community may benefit more from the theoretical framework we introduce in our paper if published early.\n\n**[Label definition]**  For the task “Cities” and “Art”, there are four label values overall but, for each instance, there are only two choices of label values. Therefore, in our experiments, we map each instance’s label values to label values {0, 1} at random. For example, for the “Cities” task, for an instance with label choice “Los Angeles” vs. “San Francisco”, we map “Los Angeles” to label 0 and “San Francisco” to label 1. For this instance, if the true label is “Los Angeles”, we have Y = 0. If h = ’low’, the human decision maker is confident that the true label is not “San Francisco”. Here, note that we did not have control on the way label values were initially assigned since we used observational data gathered in a previous work.\n\n**[Estimation of $P(Y=1 | (X,Y) \\in S_{h, \\lambda(b)})$]** This probability is computed as the empirical average of the outcome variable $Y$ across instances where the human confidence $H = h$ and the discretized classifier's confidence $B = b$,  where we used $8$ or $10$ bins.\n\n**[Definition of alignment violations]** There is an alignment violation between confidence values $h, b$ and $h’, b’$, with $h\\leq h’$ and $b\\leq b’$, whenever $P(Y =1 | (X,Y) \\in S_{h,\\lambda(b)}) - P(Y =1 | (X,Y) \\in S_{h’,\\lambda(b')}) > 0$. We will clarify this in the revised version of the paper.\n\n**[Explanations of Figures 2 and 3]** In the revised version of the paper, we will improve the explanations of Figures 2-3 particularly regarding the monotonicity in lines 318 and 327.\n\n**[Empirical support for monotone AI-assisted decision policies]**  Following the reviewer's advice, in the revised version of the paper, we will emphasize that, since the data we used in the experiment were aggregated in various ways, our conclusions should be treated with caution.\n\n**[Other settings]** We think our theoretical results may be a good starting point to study settings with multi-categorical outcome variables and decisions. However, one of the main challenges would be to identify which natural conditions utility functions may satisfy in such settings. A theoretical analysis of AI-assisted sequential decision making seems significantly more challenging—multicalibration in sequential settings is an open area of research—but our theoretical results may still be a useful starting point. A theoretical analysis of LLM-based assisted decision making is rather open and it would be certainly a landmark result. We will include a discussion about this in the revised version of the paper.\n\n**[Limitations]** In the revised version of the paper, we will include an explicit section to discuss limitations, including the setup's simplicity, the experiments' limited scope and the limitations discussed in lines 347-353."}, {"review_id": "CLZblWBQQO", "reviewer": "Reviewer_oDhU", "summary": "This paper theoretically analyzes human decision making in the presence of AI-provided predictions and confidence estimates. They do this by defining a causal model to characterize AI-assisted decision making along with a sensible assumptions about a rational decision-maker's policy: namely that their probability of making a positive classification is monotone in the confidence values. Under this assumption, the paper shows that the traditionally accepted idea of \"calibrated\" confidence is not enough for a rational decision maker to make optimal decisions. However, a new notion of \"human-alignment\" does provide this guarantee, and in fact is achievable via the well-known idea of multicalibration. The paper also presents some experiments to validate their assumptions and to argue that  human-aligned confidences might lead to better decision making.", "questions": "In several of the experiments, the human performance is worse than that of the model itself. This could be due to several factors, but one reason could be that these humans were not experts at the specified task. In terms of your SCM, they aren't using a lot of latent information V that is hidden to the model. Is there a way to include a \"competence/expertise\" parameter in this SCM to model, and even possibly correct, a non-expert decision maker so that their performance exceeds that of the AI system?", "limitations": "Limitations of the work not discussed. What are the limits of this papers' ideas in terms of both theoretical follow-up and real-world usage?", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "This paper approaches the question of AI-assisted decision making from first principles, allowing them to ground their assumptions explicitly and provide a compelling theoretical analysis under their model. The flavour of results in this paper is quite interesting and their techniques build upon and connect a some rich areas of theoretical research. Lastly, the exposition of the paper is quite easy to read, with a good breakdown of high-level ideas and detailed maths in the appendix.\n\nI believe this work would be of notable interest to several communities at NeurIPS.", "weaknesses": "To me, the weakest part of this paper is the experimental section, which in my opinion doesn't add a huge amount of insight into the results. I like the idea of empirically demonstrating the monotonicity assumption, however the interplay between misalignment and the human decision-makers' ability to perform the task well is mentioned like an afterthought. It would be amazing to see a scientifically valid empirical claim that human alignment leads to better performance with AI-assisted decision making. This is likely a significant task to carry out, and it is fine to claim as beyond the scope of a theoretical paper. However, it would make this an extremely strong submission. \n\nI would also like to see a discussion of the limitations of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful and insightful comments, which will help improve our paper. Please, find a point-by-point response below.\n\n**[Empirical validation that human-alignment leads to higher human performance]** We acknowledge that our experiments do not empirically demonstrate that, **for a fixed task**, there is an improvement in $E_{\\pi}[u(T, Y)]$ before and after alignment. However, they empirically demonstrate that, **across tasks**, the degree of alignment of the AI model correlates with the performance of $\\pi_{H_{+AI}}$ as compared to $\\pi_H$ and $\\pi_B$. As the reviewer may guess, the reason why we could not show the former is because, in our experiments, we used an observational dataset, gathered in a previous work, and this dataset does not contain human predictions before and after multicalibrating $f_{B}$ (i.e., the AI). The only way we see to address this limitation is to run a human subject study, however, this would require significant funding and time and, given the on-going discussions about the potential of human+AI collaboration, we think the NeurIPS community may benefit more from the theoretical framework we introduce in our paper if published early. That being said, we are planning to run such a study, which we think would be very worthy on its own, in future work. To avoid any misunderstanding, we will explicitly discuss this limitation in the revised version of the paper.\n\n**[Limitations]** In addition to the limitation just discussed in the previous answer and the limitations/future work discussed in lines 347-353, we acknowledge that our theoretical results apply only to a simple setting where both the outcome variable and the decision are binary. It would be very interesting to analyze settings with categorical, or even real valued, outcome variables and decisions as well as sequential settings. One of the main challenges would be to identify which natural conditions utility functions may satisfy in such settings. In our revised version of the paper, we will include an explicit section describing the above limitations.\n\n**[Competence/expertise in the SCM]** We agree that, in our experiments, humans were unlikely to be _experts_ at the specified tasks and were not using additional information. However, note that our results do not depend on the human's expertise and, given enough historical data about a human decision maker with any level of expertise, one can align any black box model to that decision maker via multicalibration. In this context, we would also like to clarify that our theoretical and empirical results suggest that, if the classifier is sufficiently human-aligned and the decision task reduces to predicting the value of the outcome, a non-expert decision maker can achieve higher performance after observing the classifier's human-aligned confidence.\n"}, {"review_id": "Ix4EflP99F", "reviewer": "Reviewer_TBZV", "summary": "This paper proposes a causal model of AI-assisted decision-making that considers both the classifier’s and decision maker’s confidence scores. The authors demonstrate theoretically why it may be challenging for the human to identify the optimal decision-making policy even if the classifier is perfectly calibrated. The authors propose a remedy by providing theoretical results using multicalibration. The authors then provide experimental results to measure misalignment and miscalibration in real-world datasets.", "questions": "Major question:\nCould the authors please address the concerns with the empirical results above?\n\nMinor questions:\n- In L121, you state that \"allow the classifier’s confidence to depend on the decision maker’s confidence\". What is a realistic example of a classifier that takes in decision maker confidence?\n- Why is Eq. 4 important to measure?\n- Are there any potential issues that can be introduced if you represent human confidence scores in different ways (e.g., binning into low-med-high or using real numbers like 0.7/0.8)?\n- Can you provide error bars for Table 1?\n- The term \"misalignment\" is introduced in the empirical results and seemed to be an important quantity of evaluation, could the authors clarify why the term was not more formally incorporated in the earlier sections?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper was generally well written and lays out the proposed causal model of AI-assisted decision-making in a step-by-step manner, making it easy to follow\n- The paper covers an important, growing topic on AI-assisted decision-making and highlights the need to consider notions of both human and AI calibration\n", "weaknesses": "My main concern with this paper is the lack of empirical results that demonstrate an *improvement* in $\\mathbb{E}_\\pi [u(T,Y)]$ (i.e., show results that the AUC would improve in the H+AI condition as compared to $\\pi_B$ or $\\pi_H$, which is a commonly used definition of complementarity) as stated as a contribution in L62. While they provide a theoretical result (Thrm 5) to guarantee near-optimality under $\\alpha$-alignment, I do not believe the authors implemented multicalibration (the proposed solution in Section 5) in their empirical results. The ideal result would be to show that without multicalibration, human decision makers more often do not achieve complementary performance, as shown in Table 1 last column, and then this could be corrected by introducing some version of multicalibration. Such an implementation would suggest how the theoretical result could be translated to practice and could be very impactful in the human-AI decision-making literature. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful and insightful comments, which will help improve our paper. Please, find a point-by-point response below.\n\n**[Empirical results]** We acknowledge that our experiments do not empirically demonstrate that, **for a fixed task**, there is an improvement in $E_{\\pi}[u(T, Y)]$ before and after alignment. However, they empirically demonstrate that, **across tasks**, the degree of alignment of the AI model correlates with the performance of $\\pi_{H_{+AI}}$ as compared to $\\pi_H$ and $\\pi_B$. As the reviewer may guess, the reason why we could not show the former is because, in our experiments, we used an observational dataset, gathered in a previous work, and this dataset does not contain human predictions before and after multicalibrating $f_{B}$ (i.e., the AI). The only way we see to address this limitation is to run a human subject study, however, this would require significant funding and time and, given the on-going discussions about the potential of human+AI collaboration, we think the NeurIPS community may benefit more from the theoretical framework we introduce in our paper if published early. That being said, we are planning to run such a study, which we think would be very worthy on its own, in future work. To avoid any misunderstanding, we will rewrite L62 and explicitly discuss this limitation in the revised version of the paper.\n\n**[Realistic example]** In our SCM, we allowed the classifier's confidence to depend on the decision maker's confidence because, to achieve human-alignment via multicalibration as described in Section 5, the classifier needs to have access to the decision maker's confidence. However, Theorems 3 and 5 still hold if the classifier does not depend on the classifier's confidence. Further, we would like to clarify that, as of today, existing classifiers do not (typically) depend on the decision maker's confidence. However, looking into the future, our work questions this status quo by showing that, by allowing the model’s confidence to depend on the decision maker's confidence, a decision maker may end up taking decisions with higher utility. We will add a discussion about this in the revised version of the paper.\n\n**[Eq. 4]**  Eq. 4 is important because it indicates whether the decision maker is actually taking decisions using a monotonic policy. More specifically, if Eq. 4 does not hold, then the policy is not monotonic. \n\n**[Human confidence scores]** As long as the human confidence scores are represented using a discrete ordered set, we cannot think of any algorithmic nor theoretical issues. For example, one may decide to use the set ${0, 0.1, 0.2, …, 0.9, 1.0}$ or any other finite set of real values. That being said, the larger the set of human confidence scores used in the representation, the greater the data and computational requirements to achieve human-alignment via multicalibration, as discussed in lines 247-250 and 266-268.\n\n**[Error bars]** We will provide error bars for Table 1 in the revised version of the paper.\n\n**[Term misalignment]** The misalignment measures EAE and MAE introduced in Section 6 quantify the average and maximum value of the left hand side of Eq. 6 in Section 4 given $\\tilde{S}_h = S_h$, whenever those values are positive.  We will make this connection more explicit in the revised version of the paper."}, {"review_id": "SSPh2hwbJP", "reviewer": "Reviewer_Q29R", "summary": "Communicating model uncertainty to human decision makers is critical for trustworthy deployment of assistive AI systems. Prior work, e.g., Vodrahalli et al 2022, have found that people may respond better to miscalibrated AI confidence scores. What underlies this odd behavior? In this submission, the authors explore this phenomenon, specifically developing a new theoretical framework built on structured causal models. In this framework, the authors find that there indeed are some data distributions wherein calibrated AI confidence scores prohibit humans from uncovering optimal policies by which to make their final decision. The authors then propose to resolve this discrepancy through “human-calibration;” multicalibration is demonstrated as one such method. The authors aim to validate their theoretical insights through empirical studies on the same human data from Vodrahalli et al 2022. ", "questions": "- Many of my questions are listed in the Weaknesses section. In particular, it would be great if the authors could clearly spell out which parts of their experiments (Section 6) validate which parts of their theory. I am open to reconsidering some of my critiques from the Weakness section if this is clarified a bit better. \n- Why 8 or 10 bins? [line 298]\n- Could you expand please on the confounding factors in the data from other counties? I am not entirely sure why you threw away that data? [lines 287-288] My understanding is this was because those participants were told the AI had a different accuracy? This may be worth exploring (perhaps in future work) – how do your theoretical results hold up across wider swaths of data?) \n- How does this relate to RLHF? Please spell out the connection a bit more, or o.w. I actually think it’s unnecessary to mention here. [lines 73-74] \n", "limitations": "The authors do not adequately discuss limitations, nor any potential negative societal impact. I would be keen for the authors to provide their thoughts on these two points during the rebuttal/discussion period. \n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "A substantive assessment of the strengths of the paper, touching on each of the following dimensions: originality, quality, clarity, and significance. We encourage reviewers to be broad in their definitions of originality and significance. For example, originality may arise from a new definition or problem formulation, creative combinations of existing ideas, application to a new domain, or removing limitations from prior results. You can incorporate Markdown and Latex into your review. See /faq.\n\nThis paper has many strengths and was an enjoyable read. The paper was very well-written and incredibly well-motivated. As AI systems increasingly move to real-world applications, it is paramount that they are designed with the human user in mind. The theoretical framework from this paper offers a nice contribution to advance our understanding of the nuances of the way that AI output is communicated to humans to best support decision making. \n\nSome further strengths: \n- The specific problem of looking at monotonicity vs. discrepancy from monotonicty in confidence values is interesting and nicely expands on prior work. \n- The authors make an excellent point on lines 110-114 that human may take diff decisions with same level of confidence; it is good to see that their model incorporates sources of noise ontop of confidence. \n- Nice that their method is claimed to be general to other calibration methods. \n- The mathematics seems sound and steps naturally follow from previous; though while I’ve attempted to check through it, there is a possibility I have missed something (hence my own confidence score of 3).\n", "weaknesses": "My biggest misgivings about the paper fall into two categories: 1) the discretization of human confidence, and 2) the empirical validation in Section 6. I will discuss each in turn. I am also concerned that the authors do not adequately discuss limitations. But note this further in the Limitations section of the review. \n\n(1) Discretization of human confidence: \n- The authors cite two papers from the neuroscience literature suggesting that humans have “discrete confidence”; however, the studies in these papers looks at quite different decision making tasks than are considered here. The tasks are much lower level and rote, e.g., simple perception and motor tasks, at least from my reading. These are quite different from the kinds of rich, high-level cognitive tasks that involve say, coming up with a treatment plan for a patient. As such, I feel the methods here ought to be able to handle finer-grained representations of human uncertainty/confidence rather than too coarse binning.\n- I am very open to discussing this — can the authors shed light on how applicable their theoretical results are to different choices of discretization?\n- Specifically on Section 6, I have concerns on the choice of 3 bins. 3 bins are incredibly coarse. Why this number?  For instance, the “mid” confidence level spans a confidence range of 0.5 (out of the normalized 0-1 scale). I recognize that the binning was done here to ensure approximately equal counts across all bins. However, my sense is that this could yield obfuscate some of the nuance in the results? Have the authors considered a finer-grained decomposition, say to 5 or 7 bins? If nothing else, I think this should be discussed in a limitations sections as I imagine this could be an impactful design choice wrt sensitivity in the empirical results.\n\n(2) Empirical validation\n- I am struggling to understand the design of the validation in Section 6. What are the baselines? What are the authors trying to show here to validate their theoretical insights (and which aspects of the theoretical insights?) It is not clear to me what is being shown here. When the authors say “participants benefit from AI advice” in Table 1 [line 333], how is this different from the original paper from which the data is sourced?  \n- The authors note that the results (which per the above, I’d be keen to hear explicated more – what is the expected behavior of these graphs, relative to what baselines?) are weaker for the Census task. Do the authors have a sense for why this may be? \n- The authors grouped the results of cases where participants were told that an AI system had provided the confidence with those told a human had (footnote 9 on pg 8). Is this valid? Could have influenced participants’ uptake of the knowledge? Have the authors explored how results stand up if they decompose / keep those groups separate? Do trends still hold? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful and insightful comments, which will help improve our paper. Please, find a point-by-point response below.\n\n**[Discretization of human confidence]** As long as the human confidence scores are represented using a discrete ordered set, our theoretical results are applicable. For example, one may decide to use a finer-grained representation of human uncertainty/confidence using a **finite** set of real values ${0, 0.05, 0.1, …, 0.95, 1.0}$. However, the finer-grained the representation of human confidence, the greater the amount of data that is needed for (i) reliably estimating the probabilities depicted in Figure 2 and the level of misalignment, as measured by EAE and MAE, and (ii) running the multicalibration algorithms described in Section 5 with theoretical guarantees, as discussed in lines 247-250 and 266-268. In our experiments, we used $3$ bins to have sufficient data for (i).\n\n**[Why 8 or 10 bins? [line 298]]** For each task, we chose the number of bins based on the amount of available data. To avoid any misunderstanding, we have re-generated all figures and tables using 8 bins for all tasks and did not find any major difference (please, refer to the companion pdf submitted with the rebuttal for the new table and figures of the “Art” and “Sarcasm” task using 8 bins). We will update the figures and table and include error bars, in the revised version of the paper.\n\n**[Section 6: what are the baselines?]** The baselines are (i) decision policies $\\pi_B$ constructed by thresholding the classifier's confidence values and (ii) decision policies $\\pi_H$ constructed by thresholding the humans' confidence values _before_ observing the classifier's confidence values. In our experiments, we compare these baselines with (iii) decision policies $\\pi_{H_{+AI}}$ constructed by thresholding the humans' confidence values _after_ observing the classifier's confidence values.\n\n**[Section 6: what do we show to validate the theoretical insights?]** In the context of the specific dataset we used in our experiments, our results support our theory as follows: \n(i) Figure 2 and Table 1 (left and middle columns) demonstrate that AI models often violate the necessary condition for human-alignment, as defined in Section 4. \n(ii) Figure 3 suggests that, whenever they receive AI advice, humans make decisions following a monotone decision policy. This supports our hypothesis in Section 2.\n(iii) Table 1 (right column) shows that, **across tasks**, the level of misalignment of the AI model is inversely correlated with the performance of AI-assisted human decision makers. This supports Theorems 3 and 5.\n\nThe original paper from which the dataset we used was sourced did not demonstrate any of the above (i-iii). In this context, we would also like to acknowledge that our experiments do not demonstrate that, **for a fixed task**, the degree of misalignment of the AI model is inversely correlated with the performance of AI-assisted human decision makers. The reason being that this pre-existing dataset does not contain human predictions before and after multicalibrating an AI model. The only way we see to address this limitation is to run a human subject study, however, this would require significant funding and time and, given the on-going discussions about the potential of human+AI collaboration, we think the NeurIPS community may benefit more from the theoretical framework we introduce in our paper if published early. In future work, we are planning to run such a study, which we think would be very worthy on its own. \n\n**[Section 6: expected behavior, performance in the Census task weaker?]** It is often assumed that AI-assisted human decision makers ($\\pi_{H_{+AI}}$) should outperform both human decision makers ($\\pi_{H}$) and AI decision makers ($\\pi_{B}$). Our experiments support the former—AI-assisted human decision makers outperform human decision makers consistently across all tasks—but not the latter—AI-assisted human decision makers only outperforms AI decision makers in a single task (\"Cities\") out of four. We cannot really tell why the level of human misalignment of the AI model is so high in the “Census” task. \n\n**[Section 6: AI advice vs human advice]** In the dataset we used in our experiments, a participant was told that the advice came from AI or from humans at random in all tasks, i.e., there was randomization with respect to this factor of variation. Therefore, whenever we compare the level of misalignment, the level of miscalibration and AUC across tasks, we are essentially controlling for this factor of variation. Unfortunately, if we keep the above groups separately, we do not have sufficient data to estimate reliably the level of misalignment and miscalibration. In this context, we would also like to clarify that, while we focus on AI-assisted human decision making, our theoretical results do not depend on who (AI model or another human) gives the _advice_. As long as the advice comes in the form of confidence values, our results are valid. We will include a discussion about this in the revision.\n\n**[Data from other countries]** We did not include data from other countries because participants were told that the model had different accuracies and they only performed two of the tasks (\"Art\" and \"Sarcasm\"). However, we agree with the reviewer that it would be important to replicate our experimental findings using other datasets as well as carry out human subject studies in future work.\n\n**[RLHF]** Since the term human alignment is used widely in RLHF, typically in the context of human preferences, we think it is necessary to clarify that our notion of human-alignment is very different.\n\n**[Limitations & potential negative societal impact]** In the revised version of the paper, we will include an explicit section to discuss limitations, including the experiments' limited scope, the setup's simplicity, and the limitations discussed in lines 347-353."}]

Author rebuttal:
We thank the reviewers for their careful and insightful comments, which will help improve our paper. 

In addition to the point-by-point responses to each individual reviewer below, please, find attached a pdf containing updated figures for the “Art” and “Sarcasm” tasks as well as an updated Table 1 where we have used 8 bins for the discretization of the AI model's confidence in all tasks (in the submitted version, we have used 10 bins for the tasks "Art" and "Sarcasm"). We did not find any major difference in the results based on this change.

We will provide error bars for Table 1 in the revised version of the paper.

Meta-review: The paper provides a new theoretical framework for calibrating predictions of AI models for optimal decision making by downstream agents. There was broad consensus amongst the reviewers on the strengths and weaknesses of the proposed work. On the positive side, all the reviewers appreciated the importance of the problem and the elegance of the proposed approach in addressing it for arbitrary binary classifiers. On the negative side, the reviewers had 2 major remarks: one on the limitation of the proposed framework to settings with binary outcomes/targets, and the second on the limitations of the experiments to simple experimental setups which comes across a missed opportunity for demonstrating the broad importance and utility of the proposed framework. These limitations are understandable for a first work, as also reflected in the fact that the overall recommendation of all reviewers was towards acceptance.
Decision: Accept (poster)