Title: Human-Aligned Calibration for AI-Assisted Decision Making
Venue: NeurIPS 2023
Accepted: yes
Keywords: Calibration

Trustworthy Machine Learning

Human-Centric ML

Probabilistic Models and Methods
Abstract: Whenever a binary classifier is used to provide decision support, it typically provides both a label prediction and a confidence value. Then, the decision maker is supposed to use the confidence value to calibrate how much to trust the prediction. In this context, it has been often argued that the confidence value should correspond to a well calibrated estimate of the probability that the predicted label matches the ground truth label. However, multiple lines of empirical evidence suggest that decision makers have difficulties at developing a good sense on when to trust a prediction using these confidence values. In this paper, our goal is first to understand why and then investigate how to construct more useful confidence values. We first argue that, for a broad class of utility functions, there exists data distributions for which a rational decision maker is, in general, unlikely to discover the optimal decision policy using the above confidence values—an optimal decision maker would need to sometimes place more (less) trust on predictions with lower (higher) confidence values. However, we then show that, if the confidence values satisfy a natural alignment property with respect to the decision maker’s confidence on her own predictions, there always exists an optimal decision policy under which the level of trust the decision maker would need to place on predictions is monotone on the confidence values, facilitating its discoverability. Further, we show that multicalibration with respect to the decision maker’s confidence on her own prediction is a sufficient condition for alignment. Experiments on a real AI-assisted decision making scenario where a classifier provides decision support to human decision makers validate our theoretical results and suggest that alignment may lead to better decisions.

Human-Aligned Calibration for AI-Assisted Decision Making

Abstract

Whenever a binary classifier is used to provide decision support, it typically provides both a label prediction and a confidence value. Then, the decision maker is supposed to use the confidence value to calibrate how much to trust the prediction. In this context, it has been often argued that the confidence value should correspond to a well calibrated estimate of the probability that the predicted label matches the ground truth label. However, multiple lines of empirical evidence suggest that decision makers have difficulties at developing a good sense on when to trust a prediction using these confidence values. In this paper, our goal is first to understand why and then investigate how to construct more useful confidence values. We first argue that, for a broad class of utility functions, there exist data distributions for which a rational decision maker is, in general, unlikely to discover the optimal decision policy using the above confidence values-an optimal decision maker would need to sometimes place more (less) trust on predictions with lower (higher) confidence values. However, we then show that, if the confidence values satisfy a natural alignment property with respect to the decision maker's confidence on her own predictions, there always exists an optimal decision policy under which the level of trust the decision maker would need to place on predictions is monotone on the confidence values, facilitating its discoverability. Further, we show that multicalibration with respect to the decision maker's confidence on her own predictions is a sufficient condition for alignment. Experiments on four different AI-assisted decision making tasks where a classifier provides decision support to real human experts validate our theoretical results and suggest that alignment may lead to better decisions.

1 Introduction

In recent years, there has been an increasing excitement on the potential of machine learning models to improve decision making in a variety of high-stakes domains such as medicine, education or criminal justice [1-3]. One of the main focus has been binary classification tasks, where a classifier helps a decision maker by predicting a binary label of interest using a set of observable features [4-7]. For example, in medical treatment, the classifier may help a doctor by predicting whether a patient may benefit from a treatment. In college admissions, it may help an admissions committee by predicting whether a candidate may successfully complete an undergraduate program. In loan decisions, it may help a bank by predicting whether a prospective customer may default on a loan. In all these scenarios, the decision maker-the doctor, the committee or the bank-aim to use these predictions, together with their own predictions, to take good decisions that maximize a given utility function. In this context, since the predictions are unlikely to always match the truth, it has been widely agreed that the classifier should also provide a confidence value together with each prediction [8, 9].

While the conventional wisdom is that the confidence value should be a well calibrated estimate of the probability that the predicted label matches the true label [10-16], multiple lines of empirical evidence have recently shown that decision makers have difficulties at developing a good sense on when to trust a prediction using these confidence values [17-19]. Therein, Vodrahali et al. [17] have shown that, in certain scenarios, decision makers take better decisions using uncalibrated probability estimates rather than calibrated ones. However, a theoretical framework explaining this puzzling observation has been missing and it is yet unclear what properties we should be looking for to guarantee that confidence values are useful for AI-assisted decision making. In our work, we aim to bridge this gap.

Our contributions. Westart by formally characterizing AI-assisted decision making using a structural causal model (SCM) [20], as seen in Figure 1. Building upon this characterization, we first argue that, if a decision maker is rational, the level of trust she places on predictions will be monotone on the confidence values-she will place more (less) trust on predictions with higher (lower) confidence values. Then, we show that, for a broad class of utility functions, there are data distributions for which a rational decision maker can never take optimal decisions using calibrated estimates of the probability that the predicted label matches the true label as confidence values. However, we further show that, if the confidence values a decision maker uses satisfy a natural alignment property with respect to the confidence she has on her own predictions, which we refer to as human-alignment, then the decision maker can both be rational and take optimal decisions. In addition, we demonstrate that human-alignment can be achieved via multicalibration [11], a statistical notion introduced in the context of algorithmic fairness. In particular, we show that multicalibration with respect to the decision maker's confidence on her own predictions is a sufficient condition for human-alignment. Finally, we validate our theoretical framework using real data from four different AI-assisted decision making tasks where a classifier provides decision support to human decision makers in four different binary classification tasks. Our results suggest that, comparing across tasks, classifiers providing human-aligned confidence values facilitate better decisions than classifiers providing confidence values that are not human-aligned. Moreover, our results also suggest that rational decision makers' trust level increases monotonically with the classifier's provided confidence.

Further related work. Our work builds upon a rapidly increasing literature on AI-assisted decision making (refer to Lai et al. [21] for a recent review). More specifically, it is motivated by several empirical studies showing that decision makers have difficulties at modulating trust using confidence values [17-19], as discussed previously. In this context, it is also worth noting that other empirical studies have analyzed how other factors such as model explanations and accuracy modulate trust [2226]. However, except for a very recent notable exception [27], theoretical frameworks, which could be used to better understand the mixed findings found by these empirical studies, have been missing. More broadly, our work also relates to a flurry of recent work on reinforcement learning with human feedback [28-30], which aims to better align the outputs of large language models (LLMs) with human preferences. However, our formulation is fundamentally different and our technical contributions are orthogonal to theirs.

2 A Causal Model of AI-Assisted Decision Making

We consider an AI-assisted decision making process where, for each realization of the process, a decision maker first observes a set of features ( x, v ) ∈ X ×V , then takes a binary decision t ∈ { 0 , 1 } informed by a classifier's prediction ˆ y = argmax y f y ( x ) , as well as confidence f ˆ y ( x ) ∈ [0 , 1] , of a binary label of interest y ∈ { 0 , 1 } , and finally receives a utility u ( t, y ) ∈ R . Such an AI-assisted decision making process fits a variety of real-world applications. For example, in medical treatment, the features ( x, v ) may comprise multiple sources of information regarding a patient's health 1 , the label y may indicate whether a patient would benefit from a specific treatment, the decision t may indicate whether the doctor applies the specific treatment to the patient, and the utility u ( t, y ) may quantify the trade-off between health benefit to the patient and economic cost to the decision maker.

In what follows, rather than working with both ˆ y and f ˆ y ( x ) , we will work with just b = f 1 ( x ) , which we will refer to as classifier's confidence, without loss of generality 2 . Moreover, we will assume that

X

V

B

H

T

Individual

characteristics

Q

W

DM's

confidence

Decision

70%

'mid'

AI's

confidence

U

Y

Label

Utility

1

/ 0

D

Data

generating

process

Features

the utility u ( t, y ) is greater if the value of t and y coincide, i.e. ,

a condition that we think it is natural under an appropriate choice of label and decision values. For example, in medical diagnosis, if t = 1 means the patient is tested early for a disease and y = 1 means the patient suffers the disease, the above condition implies that the utility of either testing a patient who suffers the disease or not testing a patient who does not suffer the disease are greater than the utility of either not testing a patient who suffers the disease or testing a patient who does not suffer the disease. In condition 1, we allow for a non-strict inequality u (0 , 0) ≥ u (0 , 1) because, in settings in which the label Y is only realized whenever the decision t = 1 ( e.g. , in our previous example on medical treatment, we can only observe if a treatment is eventually beneficial or not if the patient is treated), it has been argued that, whenever t = 0 , any choice of utility must be independent of the label value [4-6], i.e. , u (0 , 0) = u (0 , 1) = u (0) .

Next, we characterize the above AI-assisted decision making process using a structural causal model (SCM) [20], which we denote as M . The SCM M is defined by a set of assignments, which entail a distribution P M and divide naturally into two subsets. One subset comprises the features and the label 3 , i.e. ,

where D is an independent exogenous random variable, often called exogenous noise, characterizing the data generating process and f X , f V and f Y are given functions 4 . The second subset comprises the decision maker and the classifier, i.e. ,

where f H and f B are given functions, which determine the decision maker's confidence H and classifier's confidence B that the value of the label of interest is Y = 1 , π is a given AI-assisted decision policy, which determines the decision maker's decision T , u is a given utility function, which determines the utility U , and Q and W are independent exogenous variables modeling the decision maker's individual characteristics influencing her own confidence H and her decision T , respectively. By distinguishing both sources of noise, we allow for the presence of uncertainty on the decision T even after conditioning on fixed confidence values h and b . This accounts for the fact that, in reality, a decision maker may take different decisions T for instances with the same confidence values h and b . For example, in medical treatment, for two different patients with the same confidence h and b , a doctor's decision may differ due to limited resources.

In our SCM M , the decision maker's confidence H refers to the confidence the decision maker has that the label Y = 1 before observing the classifier's confidence B . Moreover, following previous behavioral studies showing that human's confidence H is discretized in a few distinct levels [32, 33], we assume H takes values h from a totally ordered discrete set H . We say that the decision maker's

confidence f H is monotone (with respect to the probability distribution P ( Y = 1) ) if, for all h, h ′ ∈ H such that h ≤ h ′ , it holds that P ( Y = 1 | H = h ) ≤ P ( Y = 1 | H = h ′ ) . Further, we allow the classifier's confidence B to depend on the decision maker's confidence H because this will be necessary to achieve human-alignment via multicalibration in Section 5. However, our negative result in Section 3 also holds if the classifier's confidence f B ( X,H ) = f B ( X ) only depends on the features, as usual in most classifiers designed for AI-assisted decision making. In the remainder, we will use Z = ( X,H ) and denote the space of features and human confidence values as Z = X × H . Figure 1 shows a visual representation of our SCM M .

Under this characterization, we argue that, if a rational decision maker has decided t under confidence values b and h , then she would have decided t ′ ≥ t had the confidence values been b ′ ≥ b and h ′ ≥ h while holding 'everything else fixed' [20]. For example, in medical treatment, assume a doctor's and a classifier's confidence that a patient would benefit from treatment is b = h = 0 . 7 and the doctor decides to treat the patient, then we argue that, if the doctor is rational, she would have treated the patient had the doctor's and the classifier's confidence been b ′ = h ′ = 0 . 8 > 0 . 7 . Further, we say that any AI-assisted decision policy π that satisfies this property is monotone, i.e. ,

Definition 1 (Monotone AI-assisted decision policy) . An AI-assisted decision policy π is monotone if and only if, for any b, b ′ ∈ [0 , 1] and h, h ′ ∈ H such that b ≤ b ′ and h ≤ h ′ , it holds that π ( h, b, w ) ≤ π ( h ′ , b ′ , w ) for any w ∼ P M ( w ) .

Finally, note that, under any monotone AI-assisted decision policy, it trivially follows that

where the expectation is over the uncertainty on the decision maker's individual characteristics and the data generating process.

3 Impossibility of AI-Assisted Decision Making Under Calibration

In AI-assisted decision making, classifiers are usually demanded to provide calibrated confidence values [10-16]. A confidence function f B : Z → [0 , 1] is said to be perfectly calibrated if, for any b ∈ [0 , 1] , it holds that P ( Y = 1 | f B ( Z ) = b ) = b . Unfortunately, using finite amounts of (calibration) data, one can only hope to construct approximately calibrated confidence functions. There exist many different notions of approximate calibration, which have been proposed over the years. Here, for concreteness, we adopt the notion of α -calibration 5 introduced by Hébert-Johnson et al. [11], however, our theoretical results can be easily adapted to other notions of approximate calibration 6 .

Definition 2 (Calibration) . A confidence function f B : Z → [0 , 1] satisfies α -calibration with respect to S ⊆ Z if there exists some S ′ ⊆ S , with |S ′ | ≥ (1 -α ) |S| , such that, for any b ∈ [0 , 1] , it holds that

If the decision maker's decision T only depends on the classifier's confidence B , i.e. , π ( H,B,W ) = π ( B ) and f B satisfies α -calibration with respect to Z , then, it readily follows from previous work that, for any utility function that satisfies Eq. 1, a simple monotone AI-assisted decision policy π ∗ B that takes decisions by thresholding the confidence values is optimal [4-7], i.e. , π ∗ B = argmax π ∈ Π( B ) E π [ u ( T, Y )] , where the expectation is with respect to the probability distribution P M and Π( B ) denotes the class of AI-assisted decision policies using B . However, one of the main motivations to favor AI-assisted decision making over fully automated decision making is that the decision maker may have access to additional features V and may like to weigh the classifier's confidence B against her own confidence H . Hence, the decision maker may seek for the optimal decision policy π ∗ over the class Π( H,B ) of AI-assisted decision policies using H and B , i.e. , π ∗ = argmax π ∈ Π( H,B ) E π [ u ( T, Y )] , since it may offer greater expected utility than π ∗ B .

Unfortunately, the following negative result shows that, in general, a rational decision maker may be unable to discover such an optimal decision policy π ∗ using (perfectly) calibrated confidence values and this is true even if f H is monotone:

Theorem 3. There exist (infinitely many) AI-assisted decision making processes M satisfying Eqs. 2 and 3, with utility functions u ( T, Y ) satisfying Eq. 1, such that f B is perfectly calibrated and f H is monotone but any AI-assisted decision policy π ∈ Π( H,B ) that satisfies monotonicity is suboptimal, i.e. , E π [ u ( T, Y )] < E π ∗ [ u ( T, Y )] .

In the proof of the above result in Appendix A.2, we show that there always exist a perfectly calibrated f B ( Z ) = f B ( X,H ) that depends on both X and H for which any monotone AI-assisted decision policy is suboptimal. This is due to the fact that f B ( Z ) is calibrated on average over H , however, it may not be calibrated, nor even monotone, after conditioning on a specific value H = h . Further, we also show that, even if f B ( Z ) = P M ( Y = 1 | X ) matches the true distribution of the label Y given the features X , which has been typically the ultimate goal in the machine learning literature, there always exists a monotone f H for which any monotone AI-assisted decision policy is suboptimal. This is due to the fact that the decision maker's confidence H can differ across instances with the same value for features X because it also depends on the features V and noise Q . Hence, f H may not be monotone after conditioning on a specific value X = x . In both cases, when a rational decision maker compares pairs of confidence values h, b and h ′ , b ′ , the rate of positive outcomes Y = 1 for each pair may appear contradictory with the magnitude of confidence. In what follows, we will show that, if f B satisfies a natural alignment property with respect to f H , which we refer to as human-alignment, there always exists an optimal AI-assisted decision policy that is monotone.

4 AI-Assisted Decision Making Under Human-Aligned Calibration

Intuitively, to avoid that pairs of confidence values B and H appear as contradictory to a rational decision maker, we need to make sure that, with high probability, both f B and f H are monotone after conditioning on specific values of H and B , respectively. Next, we formalize this intuition by means of the following property, which we refer to as α -alignment:

Definition 4 (Human-alignment) . A confidence function f B satisfies α -alignment with respect to a confidence function f H if, for any h ∈ H , there exists some ˜ S h ⊆ S h , with S h = { ( x, H ) ∈ Z | H = h } and | ˜ S h | ≥ (1 -α/ 2) |S h | , such that, for any b ′ , b ′′ ∈ [0 , 1] and h ′ , h ′′ ∈ H such that b ′ ≤ b ′′ and h ′ ≤ h ′′ , it holds that

The above definition just means that, if f B is α -aligned with respect to f H then, for any h, h ′ ∈ H , we can bound any violation of monotonicity by f B between at least a (1 -α/ 2) fraction of the subspaces of features S h and S h ′ . Moreover, note that, if f B is 0 -aligned with respect to f H , then there are no violations of monotonicity, i.e. , P ( Y = 1 | f B ( X,H ) = b ′ , ( X,H ) ∈ ˜ S h ′ ) ≤ P ( Y = 1 | f B ( X,H ) = b ′′ , ( X,H ) ∈ ˜ S h ′′ ) , and we say that f B is perfectly aligned with respect to f H .

Given the above definition, we are now ready to state our main result, which shows that humanalignment allows for AI-assisted decision policies that satisfy monotonicity and (near-)optimality:

Theorem 5. Let M be any AI-assisted decision making process satisfying Eqs. 2 and 3, with an utility function u ( T, Y ) satisfying Eq. 1 If f B satisfies α -alignment w.r.t. f H , then there always exists an AI-assisted decision policy π ∈ Π( H,B ) that satisfies monotonicity and is near-optimal, i.e. ,

where π ∗ = argmax π ∈ Π( H,B ) E π [ u ( T, Y )] is the optimal policy.

Corollary 1. If f B is perfectly aligned with respect to f H , then there always exists an AI-assisted decision policy π ∈ Π( H,B ) that satisfies monotonicity and is optimal.

Finally, in many high-stakes applications, we may like to make sure that the confidence values provided by f B are both useful and interpretable [34]. Hence, we may like to seek for confidence functions f B that satisfy human-aligned calibration, which we define as follows:

Definition 6 (Human-aligned calibration) . A confidence function f B satisfies α -aligned calibration with respect to a confidence function f H if and only if f B satisfies α -alignment with respect to f H and it satisfies α -calibration with respect to Z .

In the next section, we will show how to achieve human-alignment and human-aligned calibration via multicalibration, a statistical notion introduced in the context of algorithmic fairness [11].

5 Achieving Human-Aligned Calibration via Multicalibration

Multicalibration was introduced by Hébert-Johnson et al. [11] as a notion to achieve fairness in supervised learning. It strengthens the notion of calibration by requiring that the confidence function is calibrated simultaneously across a large collection of subspaces of features C ⊆ 2 Z which may or may not be disjoint. More formally, it is defined as follows:

Definition 7 (Multicalibration) . A confidence function f B : Z → B satisfies α -multicalibration with respect to C ⊆ 2 Z if f B satisfies α -calibration with respect to every S ∈ C .

Then, we can show that, for an appropriate choice of C , if f B satisfies α -multicalibration with respect to C , then it satisfies α -aligned calibration with respect to f H . More specifically, we have the following result:

Theorem 8. If f B satisfies ( α/ 2) -multicalibration with respect to {S h } h ∈H , with S h = { ( x, H ) ∈ Z | H = h } , then f B satisfies α -aligned calibration with respect to f H .

The above theorem suggests that, given a classifier's confidence function f B , we can multicalibrate f B with respect to {S h } h ∈H to achieve α -aligned calibration with respect to f H . To achieve multicalibration guarantees using finite amounts of (calibration) data, multicalibration algorithms need to discretize the range of f B [9, 11, 12]. In what follows, we briefly revisit two algorithms, which carry out this discretization differently, and discuss their complexity and data requirements with respect to achieving α -aligned calibration.

Multicalibration algorithm via λ -discretization. This algorithm, which was introduced by HébertJohnson et al. [11], discretizes the range of f B , i.e. , the interval [0 , 1] , into bins of fixed size λ > 0 with values Λ[0 , 1] = { λ 2 , 3 λ 2 , . . . , 1 -λ 2 } .

Let λ ( b ) = [ b -λ/ 2 , b + λ/ 2) . The algorithm partitions each subspace S h into 1 /λ groups S h,λ ( b ) = { ( x, h ) ∈ S h | f B ( x, h ) ∈ λ ( b ) } , with b ∈ Λ[0 , 1] . It iteratively updates the confidence values of function f B for these groups until f B satisfies a discretized notion of α ′ -multicalibration over these groups. The algorithm then returns a discretized confidence function f B,λ ( x, h ) = E [ f B ( X,H ) | f B ( X,H ) ∈ λ ( b )] , with b ∈ Λ[0 , 1] such that f B ( x, h ) ∈ λ ( b ) , which is guaranteed to satisfy ( α ′ + λ ) -multicalibration. Refer to Algorithm 1 in Appendix B for a pseudocode of the algorithm.

Then, as a direct consequence of Theorem 8, we can obtain a (discretized) confidence function f B,λ that satisfies α -aligned calibration by setting α ′ = λ = α/ 4 . However, the following proposition shows that, to satisfy just α -alignment, it is enough to set α ′ = 3 8 α > α/ 4 and λ = α/ 4 :

Proposition 1. The discretized confidence function f B,λ returned by Algorithm 1 satisfies (2 α ′ + λ ) -alignment with respect to f H .

Finally, it is worth noting that, to implement Algorithm 1, we need to compute empirical estimates of the expectations and probabilities above using a calibration set D . In this context, Theorem 2 in HébertJohnson et al. [11] shows that, if we use a calibration set of size O (log( |H| / ( αγξ )) /α 11 / 2 γ 3 / 2 ) , with P (( X,H ) ∈ S h ) > γ for all h ∈ H , then f B,λ is guaranteed to satisfy α -multicalibration with probability at least 1 -ξ in time O ( |H| · poly (1 /α, 1 /γ )) .

glyph[negationslash]

Multicalibration algorithm via uniform mass binning. Uniform mass binning (UMD) [9, 12] has been originally designed to calibrate f B with respect to Z using a calibration set D . However, since the subspaces {S h } h ∈H are disjoint, i.e. , S h ∩ S h ′ = ∅ for every h = h ′ , we can multicalibrate f B with respect to {S h } h ∈H by just running |H| instances of UMD, each using the subset of samples D∩S h . Here, we would like to emphasize that we can use UMD to achieve multicalibration because, in our setting, the subspaces {S h } h ∈H are disjoint.

Each instance of UMD discretizes the range of f B , i.e. , the interval [0 , 1] , into N = 1 /λ bins with values Λ h [0 , 1] = { ˆ P ( Y = 1 | f B ( X,h ) ∈ [0 , ˆ q 1 ]) , . . . , ˆ P ( Y = 1 | f B ( X,h ) ∈ [ˆ q N -1 , ˆ q N ]) } , where ˆ q i denotes the ( i/N ) -th empirical quantile of the confidence values f B ( x, h ) of the samples ( x, h ) ∈ D ∩ S h and ˆ P denotes an empirical estimate of the probability using samples from D∩S h , aswell. Here, note that, by construction, the bins have similar probability mass. Then, for each ( x, h ) ∈ Z , the corresponding instance of UMD provides the value of the discretized confidence function f B,λ ( x, h ) = b , where b ∈ Λ h [0 , 1] denotes the value of the bin whose corresponding defining interval includes f B ( x, h ) . Finally, we have the following theorem, which guarantees that, as

0.125

0.25

0.375

0.5

0.625

0.75

0.875

1.0

b

0.0

0.2

0.4

0.6

0.8

1.0

P

(

Y

=1

(

X

,

Y

)

h

,

(

b

)

)

Human Confidence,

h

'low': [0.0, 0.25]

'mid': (0.25, 0.74]

'high': (0.74, 1.0]

(a) Art

0.125

0.25

0.375

0.5

0.625

0.75

0.875

1.0

b

0.0

0.2

0.4

0.6

0.8

1.0

P

(

Y

=1

(

X

,

Y

)

h

,

(

b

)

)

Human Confidence,

h

'low': [0.0, 0.37]

'mid': (0.37, 0.76]

'high': (0.76, 1.0]

(b) Sarcasm

0.125

0.25

0.375

0.5

0.625

0.75

0.875

1.0

b

0.0

0.2

0.4

0.6

0.8

1.0

P

(

Y

=1

(

X

,

Y

)

h

,

(

b

)

)

Human Confidence,

h

'low': [0.0, 0.25]

'mid': (0.25, 0.69]

'high': (0.69, 1.0]

(c) Cities

0.125

0.25

0.375

0.5

0.625

0.75

0.875

1.0

b

0.0

0.2

0.4

0.6

0.8

1.0

P

(

Y

=1

(

X

,

Y

)

h

,

(

b

)

)

Human Confidence,

h

'low': [0.0, 0.32]

'mid': (0.32, 0.64]

'high': (0.64, 1.0]

(d) Census

long as the calibration set is large enough, the discretized confidence function f B,λ satisfies α -aligned calibration with respect to f H with high probability:

Theorem 9. The discretized confidence function f B,λ returned by |H| instances of UMD, one per S h , satisfies α -aligned calibration with respect to f H with probability at least 1 -ξ as long as the size of the calibration set |D| = O ( |H| α 2 λγ log ( |H| λξ )) , with P (( X,H ) ∈ S h ) ≥ γ .

6 Experiments

In this section, we validate our theoretical results using a dataset with real expert predictions in an AI-assisted decision making scenario comprising four different binary classification tasks 7 .

Data description. We experiment with the publicly available Human-AI Interactions dataset [35]. The dataset comprises 34 , 783 unique predictions from 1 , 088 different human participants on four different binary prediction tasks ('Art', 'Sarcasm', 'Cities' and 'Census'). Overall, there are approximately 32 different instances per task. In the 'Art' task, participants need to determine the art period of a painting given two choices and, overall, there are paintings from four art periods. In the 'Sarcasm' task, participants need to detect if sarcasm is present in text snippets from the Reddit sarcasm dataset [36]. In the 'Cities" task, participants need to determine which large US city is depicted in an image given two choices and, overall, there are images of four different US cities. Finally, in the 'Census' task, participants need to determine if an individual earns more than 50 k a year based on certain demographic information in tabular form. For 'Sarcasm', x is a representation of the text snippets and we set y = 1 if sarcasm is present, for 'Art' and 'Cities', x is a representation of the images and we set y = 1 and y = 0 at random for each different instance and, for 'Census', x summarizes demographic information and we set y = 1 if an individual earns more than 50 k a year. In each of the tasks, human participants provide confidence values about their predictions before ( h ) and after ( h +AI) receiving AI advice from a classifier in form of the classifier's confidence values b . 8 The original dataset contains predictions by participants from different, but overlapping, sets

of countries across tasks, who were told the AI advice had different values of accuracy. 9 In our experiments, to control for these confounding factors, we focus on participants from the US who were told the AI advice was 80 %accurate, resulting in 15 , 063 unique predictions from 471 different human participants.

Experimental setup and evaluation metrics. For each of the tasks, we first measure (i) the degree of misalignment between the classifiers' confidence values b and the participants' confidence values h before receiving AI advice b and (ii) the difference ( h +AI -h ) between the human participant's confidence values before and after receiving AI advice b . Then, we compare the utility achieved by a AI-assisted decision policy π H +AI that predicts the value of y by thresholding the humans' confidence values h +AI after observing the classifier's confidence values against two baselines: (i) a decision policy π B that predicts the value of y by thresholding the classifier's confidence values b and (ii) a decision policy π H that predicts the value of y by thresholding the humans' confidence values h before observing the classifier's confidence values.

To measure the degree of misalignment, we discretize the confidence values b and h into bins. For the classifiers' confidence b , we use 8 uniform sized bins per task with (centered) values Λ[0 , 1] , where λ = 1 / 8 . For the human participants' confidence h before receiving AI advice b , we use three bins per task ('low', 'mid' and 'high'), where we set the bin boundaries so that each bin contains approximately the same probability mass and set the bin values to the average confidence value within each bin. In what follows, we refer to the pairs of discretized confidence values ( h, b ) as cells, where samples ( x, y ) ∈ Z whose confidence values lie in the cell ( h, b ) define the group S h,λ ( b ) , and note that we choose a rather low number of bins for both b and h so that most cells have sufficient data samples to reliable estimate several misalignment metrics, which we describe next.

We use three different misalignment metrics: (i) the number of alignment violations between cell pairs, (ii) the expected alignment error (EAE) and (iii) the maximum alignment error (MAE). There is an alignment violation between cells pairs ( h, b ) and ( h ′ , b ′ ) , with h ≤ h ′ and b ≤ b ′ , if

Moreover, we have that:

where N = |{ h ≤ h ′ , b ≤ b ′ }| . Here, note that the number of alignment violations tells us how frequently is the left hand side of Eq. 6 positive across cell pairs given ˜ S h = S h and the EAE and MAE quantify the average and maximum value of the left hand side of Eq. 6 across cells violating alignment. To obtain reliable estimates of the above metrics, we only consider cells ( h, b ) with |S h,λ ( b ) | ≥ 30 samples. Moreover, we also report the expected calibration error (ECE) and maximum calibration error (MCE) [12, 37], which are natural counterparts to EAE and MAE, respectively.

As a measure of utility, we estimate the true positive rate (TPR) and false positive rate (FPR) of the decision policies π B , π H and π H +AI for all possible choices of threshold values, which we summarize using the area under the ROC curve (AUC) and, in Appendix C, we also report ROC curves.

0.125

0.25

0.375

0.5

0.625

0.75

0.875

1.0

b

0.6

0.4

0.2

0.0

0.2

0.4

0.6

E

[

h

+AI

h

(

X

,

Y

)

h

,

(

b

)

]

Human Confidence,

h

'low': [0.0, 0.25]

'mid': (0.25, 0.74]

'high': (0.74, 1.0]

(a) Art

0.125

0.25

0.375

0.5

0.625

0.75

0.875

1.0

b

0.6

0.4

0.2

0.0

0.2

E

[

h

+AI

h

(

X

,

Y

)

h

,

(

b

)

]

Human Confidence,

h

'low': [0.0, 0.37]

'mid': (0.37, 0.76]

'high': (0.76, 1.0]

(b) Sarcasm

0.125

0.25

0.375

0.5

0.625

0.75

0.875

1.0

b

0.6

0.4

0.2

0.0

0.2

0.4

0.6

E

[

h

+AI

h

(

X

,

Y

)

h

,

(

b

)

]

Human Confidence,

h

'low': [0.0, 0.25]

'mid': (0.25, 0.69]

'high': (0.69, 1.0]

0.125

0.25

0.375

0.5

0.625

0.75

0.875

1.0

b

0.6

0.4

0.2

0.0

0.2

0.4

E

[

h

+AI

h

(

X

,

Y

)

h

,

(

b

)

]

Human Confidence,

h

'low': [0.0, 0.32]

'mid': (0.32, 0.64]

'high': (0.64, 1.0]

(c) Cities

(d) Census

Results. We start by looking at the empirical estimates of the probabilities P ( Y = 1 | ( X,Y ) ∈ S h,λ ( b ) ) and of our measures of misalignment (EAE, MAE) and miscalibration (ECE, MCE) in Figure 2 and Table 1 (left and middle columns). The results show that, for 'Cities', the probabilities P ( Y = 1 | ( X,Y ) ∈ S h,λ ( b ) ) are (approximately) monotonically increasing with respect to the classifier's confidence values b . More specifically, as shown in Figure 2, there is only one alignment violation between cell pairs and, hence, our metrics of misalignment acquire also very low values. In contrast, for 'Art', 'Sarcasm' and especially 'Census', there is an increasing number of alignment violations and our misalignment metrics acquire higher values, up to several orders of magnitude higher for 'Census'. These results also show that misalignment and miscalibration go hand in hand, however, in terms of miscalibration, 'Census' does not stand up so strongly.

Next, we look at the difference h +AI -h between the human participant's recorded confidence values before and after receiving AI advice b across samples in each of the subsets S h,λ ( b ) induced by the discretized confidence values used above. Figure 3 summarizes the results, which reveal that the difference h +AI -h increases monotonically with respect to the classifier's confidence b . This suggests that participants always expect b to reflect the probability of a positive outcome irrespectively of their confidence value h before receiving AI advice, providing support for our hypothesis that (rational) decision makers implement monotone AI-assisted decisions policies. Further, this finding also implies that, for 'Art', 'Sarcasm' and 'Census', any policy π H +AI that predicts the value of the label y by thresholding the confidence value h +AI will be necessarily suboptimal because the probabilities P ( Y = 1 | ( X,Y ) ∈ S h,λ ( b ) ) are not monotone increasing with b .

Finally, we look at the AUC achieved by decision policies π B , π H and π H +AI . Table 1 (right columns) summarize the results, which shows that π H +AI outperforms π H consistently across all tasks but it only outperforms π B in a single task ('Cities') out of four. These findings provide empirical support for Theorem 3, which predicts that, in the presence of human-alignment violations as those observed in 'Art', 'Sarcasm' and 'Census', any monotone AI-assisted decision policy will be suboptimal, and they also provide support for Theorem 5, which predicts that, under human-alignment, there exist near-optimal AI-assisted decision policies satisfying monotonicity.

7 Discussion and Limitations

In this section, we discuss the intended scope of our work and identify several limitations of our theoretical and experimental results, which may serve as starting points for future work.

Decision making setting. We have focused on decision making settings where both decisions and outcomes are binary. However, we think that it may be feasible to extend our theoretical analysis to settings with multi-categorical (or real-valued) outcome variables and decisions. One of the main challenges would be to identify which natural conditions utility functions may satisfy in such settings. Further, we also think that it would be significantly more challenging to extend our theoretical analysis to sequential settings-multicalibration in sequential settings is an open area of research-but our ideas may still be a useful starting point. In addition, our theoretical analysis assumes that the decision makers aim to maximize the average utility of their decisions. However, whenever human decisions are consequential to individuals, the decision maker may have fairness desiderata.

Confidence values. In our causal model of AI-assisted decision making, we allow the classifier's confidence values to depend on the decision maker's confidence values because this is necessary to achieve human-alignment via multicalibration as described in Section 5. However, we would like to clarify that both Theorems 3 and 5 still hold if the classifier's confidence values do not depend on the decision maker's confidence, as it is typically the status quo today. Looking into the future, our work questions this status quo by showing that, by allowing the classifier's confidence values to depend on the decision maker's confidence values, a decision maker may end up taking decisions with higher utility. Moreover, we would also like to clarify that, while the motivation behind our work is AI-assisted human decision making, our theoretical results do not depend on who-be it a classifier or another human-gives advice. As long as the advice comes in the form of confidence values, our results are valid. Finally, while we have shown that human-alignment can be achieved via multicalibration, we hypothesize that algorithms specifically designed to achieve human-alignment may have lower data and computational requirements than multicalibration algorithms.

Experimental results. Our experimental results demonstrate that, across tasks , the average utility achieved by decision makers is relatively higher if the classifier they use satisfies human-alignment. However, they do not empirically demonstrate that, for a fixed task , there is an improvement in average utility achieved by decision makers if the classifier they use satisfies human-alignment. The reason why we could not demonstrate the latter is because, in our experiments, we used an observational dataset gathered by others [35]. Looking into the future, it would be very important to run a human subject study to empirically demonstrate the latter and, for now, treat our conclusions with caution.

8 Conclusions

We have introduced a theoretical framework to investigate what properties confidence values should have to help decision makers take better decisions. We have shown that there exists data distribution for which a rational decision maker using calibrated confidence values will always take suboptimal decisions. However, we have further shown that, if the confidence values satisfy a natural alignment property, which can be achieved via multicalibration, then a rational decision maker using these confidence values can take optimal decisions. Finally, we have illustrated our theoretical results using real human predictions on four AI-assisted decision making tasks.

Acknowledgements. We would like to thank Nastaran Okati for fruitful discussions at an early stage of the project. Gomez-Rodriguez acknowledges support from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No. 945719).

References

Wei Jiao, Gurnit Atwal, Paz Polak, Rosa Karlic, Edwin Cuppen, Alexandra Danyi, Jeroen de Ridder, Carla van Herpen, Martijn P Lolkema, Neeltje Steeghs, et al. A deep learning system accurately classifies primary and metastatic cancers using passenger mutation patterns. Nature communications , 11(1):1-12, 2020.

Jacob Whitehill, Kiran Mohan, Daniel Seaton, Yigal Rosen, and Dustin Tingley. Mooc dropout prediction: How to measure accuracy? In Proceedings of the fourth (2017) acm conference on learning@ scale , pages 161-164, 2017.

Julia Dressel and Hany Farid. The accuracy, fairness, and limits of predicting recidivism. Science advances , 4(1):eaao5580, 2018.

Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining , pages 797-806, 2017.

Niki Kilbertus, Manuel Gomez Rodriguez, Bernhard Schölkopf, Krikamol Muandet, and Isabel Valera. Fair decisions despite imperfect predictions. In International Conference on Artificial Intelligence and Statistics , pages 277-287. PMLR, 2020.

Isabel Valera, Adish Singla, and Manuel Gomez-Rodriguez. Enhancing the accuracy and fairness of human decision making. In Advances in Neural Information Processing Systems , 2018.

Guy N Rothblum and Gal Yona. Decision-making under miscalibration. In Innovations in Theoretical Computer Science , 2023.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning , 2017.

Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Proceedings of the 18th International Conference on Machine Learning , 2001.

Tilmann Gneiting, Fadoua Balabdaoui, and Adrian Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 2007.

Ursula Hébert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Calibration for the (computationally-identifiable) masses. In Proceedings of the 35th International Conference on Machine Learning , 2018.

Chirag Gupta and Aaditya K. Ramdas. Distribution-free calibration guarantees for histogram binning without sample splitting. In Proceedings of the 38th International Conference on Machine Learning , 2021.

Roshni Sahoo, Shengjia Zhao, Alyssa Chen, and Stefano Ermon. Reliable decisions with threshold calibration. Advances in Neural Information Processing Systems , 34:1831-1844, 2021.

Shengjia Zhao, Michael P Kim, Roshni Sahoo, Tengyu Ma, and Stefano Ermon. Calibrating predictions to decisions: A novel approach to multi-class calibration. In Advances in Neural Information Processing Systems , 2021.

Yingxiang Huang, Wentao Li, Fima Macheret, Rodney A Gabriel, and Lucila Ohno-Machado. A tutorial on calibration measurements and calibration models for clinical prediction models. Journal of the American Medical Informatics Association , 27(4):621-633, 2020.

Lequn Wang, Thorsten Joachims, and Manuel Gomez-Rodriguez. Improving screening processes via calibrated subset selection. In Proceedings of the 39th International Conference on Machine Learning , 2023.

Kailas Vodrahalli, Tobias Gerstenberg, and James Zou. Uncalibrated models can improve human-ai collaboration. In Advances in Neural Information Processing Systems , 2022.

Gal Yona, Amir Feder, and Itay Laish. Useful confidence measures: Beyond the max score. arXiv preprint arXiv:2210.14070 , 2022.

Eleni Straitouri, Lequn Wang, Nastaran Okati, and Manuel Gomez-Rodriguez. Improving expert predictions with conformal prediction. In Proceedings of the 40th International Conference on Machine Learning , 2023.

Judea Pearl. Causality . Cambridge university press, 2009.

Vivian Lai, Chacha Chen, Q Vera Liao, Alison Smith-Renner, and Chenhao Tan. Towards a science of human-ai decision making: a survey of empirical studies. In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency , 2023.

Andrea Papenmeier, Gwenn Englebienne, and Christin Seifert. How model accuracy and explanation fidelity influence user trust. arXiv preprint arXiv:1907.12652 , 2019.

Xinru Wang and Ming Yin. Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making. In 26th International Conference on Intelligent User Interfaces , pages 318-328, 2021.

Ming Yin, Jennifer Wortman Vaughan, and Hanna Wallach. Understanding the effect of accuracy on trust in machine learning models. In Proceedings of the 2019 chi conference on human factors in computing systems , pages 1-12, 2019.

Mahsan Nourani, Joanie T. King, and Eric D. Ragan. The role of domain expertise in user trust and the impact of first impressions with intelligent systems. ArXiv , abs/2008.09100, 2020.

Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , pages 295-305, 2020.

Chacha Chen, Shi Feng, Amit Sharma, and Chenhao Tan. Machine explanations and human understanding. Transactions of Machine Learning Research , 2023.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 2017.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 2022.

Banghua Zhu, Jiantao Jiao, and Michael Jordan. Principled reinforcement learning with human feedback from pairwise or k -wise comparisons. arXiv preprint arXiv:2301.11270 , 2023.

Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. In Proceedings of the 29th International Coference on International Conference on Machine Learning , 2012.

Matteo Lisi, Gianluigi Mongillo, Georgia Milne, Tessa Dekker, and Andrei Gorea. Discrete confidence levels revealed by sequential decisions. Nature Human Behaviour , 5(2):273-280, 2021.

Hang Zhang, Nathaniel D Daw, and Laurence T Maloney. Human representation of visuo-motor uncertainty as mixtures of orthogonal basis distributions. Nature neuroscience , 18(8):11521158, 2015.

Umang Bhatt, Javier Antorán, Yunfeng Zhang, Q Vera Liao, Prasanna Sattigeri, Riccardo Fogliato, Gabrielle Melançon, Ranganath Krishnan, Jason Stanley, Omesh Tickoo, et al. Uncertainty as a form of transparency: Measuring, communicating, and using uncertainty. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , pages 401-413, 2021.

Kailas Vodrahalli, Roxana Daneshjou, Tobias Gerstenberg, and James Zou. Do humans trust advice more if it comes from ai? an analysis of human-ai interactions. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society , pages 763-777, 2022.

Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli. A large self-annotated corpus for sarcasm. arXiv preprint arXiv:1704.05579 , 2017.

Telmo Silva Filho, Hao Song, Miquel Perello-Nieto, Raul Santos-Rodriguez, Meelis Kull, and Peter Flach. Classifier calibration: How to assess and improve predicted class probabilities: a survey. arXiv e-prints , pages arXiv-2112, 2021.

A Proofs

A.1 Additional Lemmas

Lemma 1 (Monotonicity) . If a utility function u satisfies Eq. 1, then u is monotone with respect to the probability that Y = 1 , i.e. , for any P, P ′ ∈ P ( { 0 , 1 } ) such that P ( Y = 1) ≤ P ′ ( Y = 1) , it holds that E Y ∼ P [ u (1 , Y )] ≤ E Y ∼ P ′ [ u (1 , Y )] .

Proof. We readily have that

where, in the above inequality, we use that u (1 , 1) > u (1 , 0) and P ( Y = 1) ≤ P ′ ( Y = 1) .

Lemma 2 (Trivial policies are not always optimal) . If a utility function u satisfies Eq. 1, then there exist P, P ′ ∈ P ( { 0 , 1 } ) such that the trivial policies π that either always decide T = 1 or always decide T = 0 are suboptimal. In particular, for any P, P ′ ∈ P ( { 0 , 1 } ) such that P ( Y = 1) < c and P ′ ( Y = 1) > c , where

it holds that

Proof. Let P be any distribution such that

where c ∈ (0 , 1) because, by assumption, u satisfies Eq. 1. Now, by rearranging the above inequality, we have that

and, using the definition of the expectation, it immediately follows that

The same argument can be used to show that, for any distribution P ′ such that P ′ ( Y = 1) > c , it holds that E Y ∼ P ′ [ u (1 , Y )] > E Y ∼ P ′ [ u (0 , Y )] . Finally, note that, since c ∈ (0 , 1) , we know that such distributions P and P ′ exist.

A.2 Proof of Theorem 3

Before proving Theorem 3, we rewrite the expected utility with respect to the probability distribution P M in terms of confidence H and B by using the law of total expectation,

Here, to simplify notation, we will write

where note that, using the law of total expectation, we can write the inner expectation in the above expression in terms of the utilities of the trivial policies, i.e. ,

and we will use P to refer to probabilities induced by SCM M , e.g. , P ( H,B ) to denote P M ( H,B ) . Now, we restate and prove Theorem 3.

Theorem 3. There exist (infinitely many) AI-assisted decision making processes M satisfying Eqs. 2 and 3, with utility functions u ( T, Y ) satisfying Eq. 1, such that f B is perfectly calibrated and f H is monotone but any AI-assisted decision policy π ∈ Π( H,B ) that satisfies monotonicity is suboptimal, i.e. , E π [ u ( T, Y )] < E π ∗ [ u ( T, Y )] .

Proof. To prove the above claim, we construct a monotone confidence function f H , perfectly calibrated confidence function f B and distribution P M for which any monotone AI-assisted decision policy π ∈ Π( H,B ) achieves strictly lower utility than a carefully constructed non monotone AI-assisted decision policy ˜ π ∈ Π( H,B ) .

We will present the proof in three parts. First, we will introduce the main building block and idea behind the proof by a small construction of f H , f B and P M with |H| = |B| = 3 , where B ⊆ [0 , 1] denotes the (discrete) output space of the classifier's confidence function. We then construct examples of f H , f B and P M for arbitrary |H| = k and |B| = m with m,k ∈ N , m > k ≥ 2 . Lastly, we construct examples where B is non-discrete and |H| = k with k > 2 .

Main building block and small example.

We start by presenting the main idea of the proof using an example with a small set of confidence values H and B . Let the values of the decision maker's confidence H be in H = { h 1 , h 2 , h 3 } and the values of the classifier's confidence B be in B = { b 1 , b 2 , b 3 } , with order h i < ( h i + 1) and b i < ( b i +1) respectively.

Our main building block, consists of two distributions P -, P + ∈ P ( { 0 , 1 } ) with P -( Y = 1) < c and P + ( Y = 1) > c , where c depends on utility u as described by Eq. 8 in Lemma 2. We use these distributions for our constructions of f H , f B and P M , so that for some realizations of H,B distribution P ( Y = 1 | H,B ) is either P -or P + . Using Lemma 2 and from Eq. 10, we have that:

(I) For any h i , b i such that P ( Y | H = h i , B = b i ) = P -, it holds that

Hence, decreasing P π ( T = 1 | H,B ) increases E [ u ( T, Y ) | H = h i , B = b i ] .

(II) For any h i , b i such that P ( Y | H = h i , B = b i ) = P + , it holds that

Hence, increasing P π ( T = 1 | H,B ) increases E [ u ( T, Y ) | H = h i , B = b i ] .

Intuitively, suppose we now have that, for confidence values h 2 , b 2 , Y ∼ P + and, for confidence values h 3 , b 2 , Y ∼ P -, i.e. , P ( Y | H = h 2 , B = b 2 ) = P + and P ( Y | H = h 3 , B = b 2 ) = P -. Then, any non-monotone AI-assisted decision policy ˜ π with P ˜ π ( T = 1 | H = h 2 , B = b 2 ) > P ˜ π ( T = 1 | H = h 3 , B = b 2 ) will have higher expected utility than any monotone AI-assisted decision policy given confidence values h 2 , b 2 and h 3 , b 2 . Finally, under an appropriate choice of distribution P ( H,B ) , such non-monotone AI-assisted decision policies ˜ π will offer higher overall utility in expectation.

We formalize this intuition with the following lemma:

Lemma 3. Let M be any AI-assisted decision making process satisfying Eqs. 2 and 3, with utility function u ( T, Y ) satisfying Eq. 1. If f H , f B and P M are such that there exists confidence values b ∈ B , h i , h j ∈ H , with h i < h j , which satisfy

for some distributions P -, P + with P -( Y = 1) < c and P + ( Y = 1) > c , where

Then, for any monotone AI-assisted decision policy π ∈ Π( H,B ) , there exists an AI-assisted decision policy ˜ π ∈ Π( H,B ) which is not monotone and achieves a stricly greater utility than π , i.e. , E π [ u ( T, Y )] < E ˜ π [ u ( T, Y )] .

Proof. Let π be a monotone AI-assisted decision policy, then it must hold that P π ( T = 1 | H = h i , B = b ) ≤ P π ( T = 1 | H = h j , B = b ) (see Eq. 4). Let ˜ π be an identical AI-assisted decision

policy to π up to the decision for confidence values h i , b and h j , b . We distinguish between three cases.

Let the probability of T = 1 under ˜ π for confidence values h i , b and h j , b be switched compared to π , i.e. ,

Then, ˜ π is not monotone, as Eq. 4 is not satisfied, and it holds that

As we decreased P ( T = 1 | H = h j , B = b ) and increased P ( T = 1 | H = h i , B = b ) , by properties (I) and (II), it must hold that the expected utility of ˜ π given confidence values h i , b and h j , b is higher than the one of π , i.e. ,

Let the probability of T = 1 under ˜ π for confidence values h j , b be strictly lower compared to π and be the same as π for h i , b . Then, ˜ π is not monotone, since by case assumption

Let the probability of T = 1 under ˜ π for confidence values h i , b be strictly higher compared to π and be the same as π for h j , b . Then, ˜ π is not monotone, since by case assumption

As in all three cases at least one of the strict inequalities in Eqs. 13 or 14 holds and ˜ π is equivalent to π ( i.e. , it has the same expected conditional utility) given any other pair of confidence values h ′ ∈ H , b ′ ∈ B , we have that

Before proceeding further, we would like to note that we may also state Lemma 3 using h ∈ H , b i , b j ∈ B , with b i < b j , the proof would follow analogously.

Now, we construct an AI-decision making process M , with H = { h 1 , h 2 , h 3 } and B = { b 1 , b 2 , b 3 } , such the decision maker's confidence f H is monotone, the classifier's confidence f B is perfectly calibrated, and the conditions of Lemma 3 are satisfied. First, let f H , f B and P M be such that

Then, it readily follows that P ( H = h i , B = b j ) = 1 / 6 for i ≥ j and P ( H = h i , B = b j ) = 0 otherwise. Moreover, for each pair of confidence values ( h i , b j ) with positive probability P ( H = h i , B = b j ) , we set

h

h

h

b

1

b

2

b

3

P

-

P

-

P

-

P

+

P

+

P

+

P

+

P

+

P

+

P

+

P

+

P

+

P

+

P

-

P

-

P

-

P

-

P

-

P

-

P

-

P

-

h

1

h

2

h

3

h

4

h

5

h

6

b

1

b

2

b

3

b

4

b

5

b

6

b

7

b

8

1

2

3

as shown in Figure 4 (left). Then, it readily follows that f H is monotone with respect to the probability that Y = 1 , i.e. , P ( Y = 1 | H = h i ) ≤ P ( Y = 1 | H = h i +1 ) ), and we have that the classifier's confidence values

are perfectly calibrated and satisfy that b j < b j +1 .

Finally, using Lemma 3 with b = b 2 , h i = h 2 , h j = h 3 , we have that any monotone AI-assisted decision policy is suboptimal for any M with f H , f B and P M as defined above.

Construction with arbitrary |H| = k and |B| = m , m>k ≥ 2 .

In this second part of the proof, we construct an AI-assisted decision making processes M , with |H| = k and |B| = m such that m > k ≥ 2 , such that the decision maker's confidence f H is monotone, the classifier's confidence f B is perfectly calibrated and the conditions of Lemma 3 are satisfied.

First, let the space of confidence values be H = { h i } i ∈ [ k ] and B = { b j } j ∈ [ m ] , with order h i < h i +1 and b i < b i +1 , respectively, and f H , f B and P M be such that P ( f B ( Z ) = b j ) = 1 /m and

Moreover, for each pair of confidence values ( h i , b j ) with positive probability P ( H = h i , B = b j ) , we set

as shown in Figure 4 (right). Further, we set the classifier's confidence values b j to

Then, it holds that b j < b j +1 and f B is perfectly calibrated as

and thus, using the definitions of P ( H | B ) and P ( Y | H,B ) , we have that P ( Y | B = b j ) = b j .

To show that f H is monotone with respect to the probability that Y = 1 , first note that P ( H = h i , B = b i ) decreases as i increases and P ( H = h i , B = b i +1 ) increases as i increases. Moreover, further note that P ( Y = 1 | H = h i , B = b i ) = P -< P ( Y = 1 | H = h i , B = b i +1 ) = P + . Hence, for any i ∈ { 2 , . . . , k -1 } , it readily follows that

and, for i = 1 , it is evident that P ( Y = 1 | H = h 1 ) < P ( Y = 1 | H = h 2 ) .

Finally, using Lemma 3 with any choice of confidence values b = b j , h i = h j -1 and h j = h j with j ∈ { 2 , . . . , k } , we have that any monotone AI-assisted decision policy π is suboptimal for any M with |H| = k and |B| = m , m > k ≥ 2 , and f H , f B and P M as defined above. Here, note that, as we do not fix the exact distributions P -and P + , the above Lemma applies to infinitely many AI-assisted decision making processes M .

Construction with B ⊆ [0 , 1] and |H| = k .

In this last part of the proof, we construct an AI-assisted decision making process M , with |H| = k ≥ 2 and B ⊆ [0 , 1] , such that the decision maker's confidence function f H is monotone, the classifier's confidence function f B is perfectly calibrated and the conditions of Lemma 3 are satisfied.

First, let the space of confidence values be H = { h i } i ∈ [ k ] , with order h i < h i +1 , the feature space 10 X = [0 , 1] , and f -, f + be two strictly monotone increasing functions with

where

Further, let Q k +1 = { q 0 , q 1 , . . . q k , q k +1 } be a set of quantiles such that P ( X ≤ q j ) = j/ ( k +1) for all j ∈ { 0 , 1 , . . . , k +1 } and thus, we have that, for all j ∈ [ k +1] ,

Now, let f H and P M be such that

h

1

h

2

h

3

X

1

h

1

h

2

h

3

h

h

h

3

I

1

I

2

I

3

I

4

f

-

(

X

)

f

+

(

X

)

2

and let

as shown in Figure 5. Next, we define

which, by construction, is perfectly calibrated.

To show that the decision maker's confidence function f H is monotone with respect to the probability that Y = 1 , we first note that, using Eq. 19, we have that

Hence, using Eq. 21 and the law of total probability, for any i ∈ { 2 , . . . , k -2 } , we have that

where the inequalities follow from the fact that f -and f + are strictly monotone increasing. Corner cases for i = 1 and i = k -1 can be shown analogously by further using that f -( X ) < c < f + ( X ) for all X .

h

4

h

5

h

6

Finally, using Lemma 3 with any choice of confidence values h i = h j -1 h j = h j , j ∈ { 2 , · · · , k -1 } and b = f B ( X ) with X ∈ I j , we have that any monotone AI-assisted decision policy π is suboptimal for any M with |B| ⊆ [0 , 1] and |H| = k , k ≥ 2 and f H , f B and P M as defined above.

A.3 Proof of Theorem 5

We prove the statement by contraposition. Let M be an AI-assisted decision making process satisfying Eqs. 2 and 3, with a utility function u ( T, Y ) satisfying Eq. 1 and let M be such that f B satisfies α -alignment with respect to f H and f B has output space B ⊆ [0 , 1] . Assume there exists no (near-)optimal monotone AI-assisted decision policy for utility u . Thus, there must exist an optimal AI-assisted decision policy π ∈ Π( H,B ) which is not monotone and has strictly greater expected utility than any monotone policy. However, we show that we can modify π to a monotone AI-assisted decision policy ˆ π ∈ Π( H,B ) with near-optimal expected utility.

As π is not monotone, there must exist confidence values h 1 , h 2 ∈ H , h 1 ≤ h 2 , and b 1 , b 2 ∈ B , b 1 ≤ b 2 , such that

where W denotes the space of noise values. In what follows, let ˜ W ( π,h 2 ,b 2 ) h 1 ,b 1 ⊆ W denote the set containing any such w and let ˜ W ( π,h 2 ,b 2 ) = ⋃ h,b ∈H×B ˜ W ( π,h 2 ,b 2 ) h,b .

For any confidence value h ′ , b ′ ∈ H × [0 , 1] , we modify policy π to a policy ˆ π as follows. Let { ˜ S h } h ∈H denote the sets satisfying the α -alignment condition for f B with respect to f H and, given confidence h ′ , let ˆ b h ′ denote the smallest confidence value of f B , such that there exist h ≤ h ′ with P ( Y = 1 | B = ˆ b h ′ , Z ∈ ˜ S h ) ≥ c , i.e. ,

Now, we define a new AI-assisted policy ˆ π from π as follows,

Next, we show that ˆ π is monotone and E ˆ π [ u ( T, Y )] ≥ E π [ u ( T, Y )] + α · a for some constant a .

Proof ˆ π is a monotone assisted policy.

To prove that ˆ π ∈ Π( H,B ) is a monotone AI-assisted decision policy, we show that, for all h ′ , h ′′ ∈ H , b ′ , b ′′ ∈ B , with h ′ ≤ h ′′ , b ′ ≤ b ′′ , it holds that ˜ W (ˆ π,h ′′ ,b ′′ ) h ′ ,b ′ = ∅ . We distinguish between three cases.

-Case 1 : b ′ ≥ ˆ b h ′ and b ′′ ≥ ˆ b h ′′ .

Since h ′ ≤ h ′′ , b ′ ≤ b ′′ and, by definition, ˆ b h ′′ ≤ ˆ b h ′ since h ′ ≤ h ′′ , we have that

Hence, we can conclude that

Further, for any other w ∈ W -⋃ h ≤ h ′′ ,b ∈ [ ˆ b h ′′ ,b ′′ ] ˜ W ( π,h,b ) ⊆ W -˜ W ( π,h ′′ ,b ′′ ) h ′ ,b ′ , we have that ˆ π ( h ′ , b ′ , w ) = π ( h ′ , b ′ , w ) and ˆ π ( h ′′ , b ′′ , w ) = π ( h ′′ , b ′′ , w ) and, by definition of ˜ W ( π,h ′′ ,b ′′ ) h ′ ,b ′ , it follows that

From Eqs. 25 and 26, it follows that ˜ W (ˆ π,h ′′ ,b ′′ ) h ′ ,b ′ = ∅ .

By definition of ˆ π , we have that

and

Analogously to case 1, since the values of w below are also in W-˜ W ( π,h ′′ ,b ′′ ) h ′ ,b ′ and ˆ π is equivalent to π for these values, we have that

From Eqs. 27 28 and 29, it follows that ˜ W (ˆ π,h ′′ ,b ′′ ) h ′ ,b ′ = ∅ .

Since h ′ ≤ h ′′ , b ′ ≤ b ′′ and, by definition, ˆ b h ′′ ≤ ˆ b h ′ since h ′ ≤ h ′′ , we have that

Hence, we can conclude that

Again analogously to case 1, since the values of w below are also in W -˜ W ( π,h ′′ ,b ′′ ) h ′ ,b ′ and ˆ π is equivalent to π for these values, we have that

From Eqs. 30 and 31, it follows that ˜ W (ˆ π,h ′′ ,b ′′ ) h ′ ,b ′ = ∅ .

Note that, we cannot have a case where b ′ ≥ ˆ b h ′ and b ′′ < ˆ b h ′′ , as this would imply b ′′ < b ′ . Since, in all three possible cases, we have shown that ˜ W (ˆ π,h ′′ ,b ′′ ) h ′ ,b ′ = ∅ , we can conclude that ˆ π ∈ Π( H,B ) is monotone.

Proof ˆ π is near optimal.

First, we rewrite the inner expectation in Eq. 10 as

Further, recall that | ˜ S h | ≥ (1 -α/ 2) |S h | for all h ∈ H and, for all h ′ , h ′′ ∈ H , h ′ ≤ h ′′ and all b ′ , b ′′ ∈ [0 , 1] , b ′ ≤ b ′′ , we have that

Now, for any h ′ ∈ H , b ′ ∈ B , we show an upper bound on E π [ u ( T, Y ) | H = h ′ , B = b ′ ] -E ˆ π [ u ( T, Y ) | H = h ′ , B = b ′ ] . We distinguish between three cases.

Using Lemma 2, we have that

Moreover, as b ′ ≥ ˆ b h ′ , the distribution of positive decisions in ˆ π may also increases for h ′ , b ′ compared to π (see Eq. 24), i.e. ,

Hence, it follows that

Since b ′ ≥ ˆ b h ′ , there exists h, b ∈ H × B , with h ≤ h ′ , b ≤ b ′ , such that P ( Y = 1 | B = b, Z ∈ ˜ S h ) ≥ c . Moreover, using the definition of α -alignment, we have that

Then, we can use this to lower bound the expected utility of T = 1 given B = b ′ and Z ∈ ˜ S h ′ as follows:

where the last inequality due to Eq. 35 and the assumption that u (1 , 1) -u (1 , 0) > 0 . Analogously, we can also upper bound the expected utility of T = 0 given H = h ′ , B = b ′ and Z ∈ ˜ S h ′ as follows:

where the last inequality holds due to Eq. 35 and the assumption that u (0 , 1) -u (0 , 0) < 0 .

Now, as P ( Y = 1 | B = b, Z ∈ ˜ S h ) ≥ c , by Lemma 2, we have that

Combining Eqs. 36, 37 and 38, we obtain

In addition, note that we have following trivial bound for the expectation when H = h ′ but Z / ∈ ˜ S h ′

Moreover, since b ′ ≥ ˆ b h ′ , the distribution of positive decisions in ˆ π may also increase for h ′ , b ′ compared to π , i.e. ,

Hence, we have that

where the inequality follows since E [ u (1 , Y ) | H = h ′ , B = b ′ ] -E [ u (0 , Y ) | H = h ′ , B = b ′ ] ≤ 0 by Lemma 2 as P ( Y = 1 | H = h ′ , B = b ′ ) < c .

Finally, combining Eqs. 39, 40, 41 and 42 and using the law of total expectation, we obtain

where β ( h ′ ,b ′ ) denotes the probability of Z / ∈ ˜ S h ′ given H = h ′ , B = b ′ , i.e. , β ( h ′ ,b ′ ) = P ( Z / ∈ ˜ S h ′ | H = h ′ , B = b ′ ) .

-Case 3 : b ′ < ˆ b h ′ .

For all h, b , with h ≤ h ′ , b ≤ b ′ , we have that P ( Y = 1 | B = b, Z ∈ ˜ S h ) < c . In particular, P ( Y = 1 | B = b ′ , Z ∈ ˜ S h ′ ) < c . Thus, by Lemma 2,

In this case, since b ′ < ˆ b h ′ , the distribution of positive decisions in ˆ π may decrease for h, b compared to π , i.e. ,

Combining Eqs.44, 40 and 41 and using the law of total expectation, we obtain

where again β ( h ′ ,b ′ ) = P ( Z / ∈ ˜ S h ′ | H = h ′ , B = b ′ ) .

Now, for a fixed h ′ ∈ H , since | ˜ S h ′ | ≥ (1 -α/ 2) |S h ′ | , we know that 0 ≤ ∑ b ∈B β ( h ′ ,b ) ≤ α/ 2 . Hence, combining Eqs. 34, 43 and 45 from the three cases above, we have that

Finally, since by assumption π is optimal, i.e. , E π [ u ( T, Y )] = E π ∗ [ u ( T, Y )] = max π ′ ∈ Π( H,B ) E π ′ [ u ( T, Y )] , we can conclude by the law of total expectation that

This concludes the proof.

A.4 Proof of Theorem 8

If f B is α/ 2 -multicalibrated with respect to {S h } h ∈H , then, by definition, for any h ∈ H , there exists ˜ S h ⊂ S h with |S| ≥ (1 -α/ 2) |S h | such that, for any b ∈ [0 , 1] , it holds that

This directly implies that, for any h ′ , h ′′ ∈ H and b ′ , b ′′ ∈ [0 , 1] , we have that

and, using linearity of expectation, we further have that

showing that, whenever b ′ ≤ b ′′ , the α -alignment condition is met. This proves that f B is α -aligned with respect to f H .

Finally, if f B is α/ 2 -multicalibrated with respect to {S h } h ∈H , then, it is α/ 2 -calibrated with respect to any of the sets S h . Since Z = ∪ h ∈H S h , this implies that f B is α/ 2 -calibrated with respect to Z . This concludes the proof.

A.5 Proof of Proposition 1

Given a discretization parameter λ , Algorithm 1 works with a discretized notion of α -multicalibration, namely ( α, λ )-multicalibration:

Definition 10. Let C ⊆ 2 Z be a collection of subsets of Z . For any α, λ > 0 , confidence function f B : Z → [0 , 1] is ( α, λ ) -multicalibrated with respect to C if, for all S ∈ C , b ∈ Λ[0 , 1] , and all S h,λ ( b ) ( g ) such that |S h,λ ( b ) | ≥ αλ |S h | , it holds that

Here, we can analogously define a discretized notion of α -alignment, namely ( α, λ )-alignment.

Definition 11. For α, λ > 0 , a confidence function f B : Z → [0 , 1] is ( α, λ ) -aligned with respect to f H if, for all h ′ , h ′′ ∈ H , h ′ ≤ h ′′ , and all b ′ , b ′′ ∈ Λ[0 , 1] , b ′ ≤ b ′′ , with |S h ′ ,λ ( b ′ ) | > α/ 2 · λ | S h ′ | and |S h ′′ ,λ ( b ′′ ) | > α/ 2 · λ | S h ′′ | , we have

In what follows, we first show that ( α, λ ) -multicalibration with respect to {S h } h ∈H implies (2 α + λ, λ ) -alignment with respect to f H .

Theorem 12. For α, λ > 0 , if f B is ( α, λ ) -multicalibrated with respect to {S h } h ∈H , then f B is (2 α + λ, λ ) -aligned with respect to f H .

Proof. If f B is ( α, λ ) -multicalibrated with respect to { S h } h ∈H , then, by definition, for all h ∈ H , b ∈ Λ[0 , 1] , and all S h,λ ( b ) such that |S h,λ ( b ) | ≥ α · λ |S h | , it holds that

This directly implies that, for all h ′ , h ′′ ∈ H , b ′ , b ′′ ∈ Λ[0 , 1] with |S h ′ ,λ ( b ′ ) | ≥ α · λ | S h ′ | and |S h ′′ ,λ ( b ′′ ) | ≥ α · λ | S h ′′ | , it holds that

and, using the linearity of expectation, we have that

Whenever b ′ ≤ b ′′ , due to the λ -discretization, we have that

Hence, we have shown that if f B is α -multicalibrated, then for all h ′ , h ′′ ∈ H , b ′ , b ′′ ∈ Λ[0 , 1] with |S h ′ ,λ ( b ′ ) | ≥ α · λ | S h ′ | and |S h ′′ ,λ ( b ′′ ) | ≥ α · λ | S h ′′ | , we have

Further, note that (2 α + λ ) / 2 · λ > α · λ as λ > 0 . This concludes the proof.

Next, we show that, if f B is ( α, λ ) -aligned, then f B,λ is α -aligned with respect to f H .

Theorem 13. For α, λ > 0 , if f B is ( α, λ ) -aligned with respect to f H , then f B,λ is α -aligned with respect to f H .

Proof. The proof is similar to the proof of Lemma 1 in Hébert-Johnson et al. [11]. Consider all S h,λ ( b ) such that |S h,λ ( b ) | < αλ |S h | . By the λ -discretization, there are at most 1 /λ such sets, thus, the cardinality of their union is at most 1 /λαλ |S h | = α |S h | . Hence, for all h ∈ H , there exists a subset ˜ S h ⊂ S h with | ˜ S h | ≥ (1 -α ) |S h | such that, for all h ′ , h ′′ ∈ H , with h ′ ≤ h ′′ , and all b ′ , b ′′ ∈ Λ[0 , 1] , with b ′ ≤ b ′′ , it holds that

The λ -discretization sets all values of ( x, h ) ∈ S h ′ ,λ ( b ′ ) to f B,λ ( x, h ) = E [ f B ( X,H ) | f B ( X,H ) ∈ λ ( b ′ )] . Note that, for ( x, h ) ∈ S h ′ ,λ ( b ′ ) , f B,λ ( x, h ) ∈ λ ( b ′ ) and for ( x, h ) ∈ S h ′′ ,λ ( b ′′ ) , f B,λ ( x, h ) ∈ λ ( b ′′ ) , so it still holds that E [ f B ( X,H ) | f B ( X,H ) ∈ λ ( b ′ )] ≤ E [ f B ( X,H ) | f B ( X,H ) ∈ λ ( b ′′ )] . Thus, using Eq. 55, we have that

This concludes the proof.

Finally, using Theorems 12 and 13, it readily follows that, given a parameter α ′ , the discretized confidence function f B,λ returned by Algorithm 1 satisfies ( 2 α ′ + λ )-aligned calibration with respect to f H .

A.6 Proof Theorem 9

We structure the proof in three parts. We first explain the calibration guarantee that UMD provides and how it relates to human-aligned calibration. Then, we derive a lower bound on the size of the subsets D∩S h so that the discretized confidence function f B,λ satisfies α -aligned calibration with respect to f H with high probability. Finally, building on this result, we derive an upper bound on |D| so that f B,λ satisfies α -aligned calibration with high probability as long as there exists γ > 0 so that P (( X,H ) ∈ S h ) ≥ γ for all h ∈ H .

Conditional Calibration implies Human-Aligned Calibration. Running UMD on a dataset D ∈ ( Z×Y ) n , where each datapoint is sampled from P M , guarantees ( α, ξ ) -conditional calibration, a PAC-style calibration guarantee [12]. Given a dataset D , a confidence function f B satisfies ( α, ξ ) -conditional calibration if, with probability at least 1 -ξ over the randomness in D ,

This stands in contrast to the definition of α -calibration, which requires only that the confidence f B ( X,H ) is at most α away from the true probability for 1 -α fraction of Z .

Similarly, using an union bound over all h ∈ H , ( α/ 2 , ξ/ |H| ) -conditional calibration of f B on each S h , h ∈ H , implies that, with probability at least 1 -ξ over the randomness in D , f B satisfies that

Hence, analogously to the proof of Theorem 8, this implies that, with probability at least 1 -ξ over the randomness in D , f B also satisfies that

In summary, from Eqs. 57 and 58, we can conclude that ( α/ 2 , ξ/ |H| ) -conditional calibration of f B on each S h , h ∈ H , implies that, with probability at least 1 -ξ , f B satisfies α -aligned calibration, where, for all h ∈ H , we have that ˜ S h = S h .

Lower bound on |D ∩ S h | to achieve conditional calibration with UMD. Running UMD on each partition D ∩ S h of D induced by h ∈ H achieves ( α/ 2 , ξ/ |H| ) -conditional calibration as long as each subset D∩S h of the data is large enough. More specifically, the following lower bound on the size of the subsets D∩S h readily follows from Theorem 3 in Gupta et al. [12].

Lemma 4. The discretized confidence function f B,λ returned by |H| instances of UMD, one per S h , is ( α/ 2 , ξ/ |H| ) -conditional calibrated on S h for any ξ ∈ (0 , 1) if

Proof. Let B denote the number of bins in UMD. Theorem 3 in Gupta et al. [12] states that, if f B ( X,H ) is absolutely continuous with respect to the Lebesgue measure 11 and |D ∩ S h | ≥ 2 B , then the discretized confidence function output by UMD is ( glyph[epsilon1], ξ ′ ) -conditionally calibrated for any ξ ′ ∈ (0 , 1) and

Then, for a given α , setting glyph[epsilon1] = α/ 2 , B = glyph[ceilingleft] 1 /λ glyph[ceilingright] and ξ ′ = ξ/ |H| , we can solve Eq. 60 for the lower bound on |D ∩ S h | ≥ n min with n min as defined in Eq. 59.

Upper bound on |D| to achieve conditional calibration with UMD. Suppose P (( X,H ) ∈ S h ) ≥ γ for all h ∈ H . When |H| ≥ 2 , we give an upper bound on |D| so that with high probability |D ∩ S h | ≥ n min for all h ∈ H .

In the process of sampling D ∈ ( Z × Y ) n from P M , let R ( h ) i = 1 denote the event that the i -th datapoint ( x i , h i , y i ) has confidence value h , i.e. , h i = h . Then, we can express |D ∩ S h | in terms of random variable R ( h ) , defined as

Since R ( h ) i is a Bernoulli-distributed variable with P ( R ( h ) i ) = P (( X,H ) ∈ S h ) , the expected value of R ( h ) is µ ( h ) := E [ R ( h ) ] = P (( X,H ) ∈ S h ) · |D| ≥ γ · |D| .

Let |D| = 2 · |H| · log(2 /ξ ) · 1 /γ · n min, observe that in this case

For |H| ≥ 2 and ξ ∈ (0 , 1) , we have 1 / (2 |H| · log(2 /ξ )) ∈ (0 , 1) and we can use a variation of the Chernoff bound to show

where the first and last inequality results from using µ ( h ) > γ · |D| . We can now use a union bound to obtain a lower bound on the probability that for any h ∈ H , |D ∩ S h | ≤ n min, i.e. ,

One can verify that for |H| ≥ 2 and n min ≥ 1 , we have P ( ∃ h ∈ H : |D ∩ S h | ≤ n min ) ≤ ξ 2 . Hence, if |D| = 2 · |H| · log(2 /ξ ) · 1 /γ · n min, then, for all h ∈ H , |D ∩ S h | ≤ n min with probability 1 -ξ/ 2 .

Combining this result and Lemma 4, we have that the discretized confidence function f B,λ returned by |H| instances of UMD, one per S h , is ( α/ 2 , ξ/ (2 |H| )) -conditional calibrated on each S h with probability at least 1 -ξ/ 2 for any ξ ∈ (0 , 1) if

Finally, using a union bound, we can conclude that f B,λ achieves α -aligned calibration with respect to f H with probability at least 1 -ξ from

samples. This concludes the proof.

B Multicalibration Algorithm

In this section, we give a high-level description of the post-processing algorithm for multicalibration introduced by Hébert-Johnson et al. [11]. The algorithm works with a discretization of [0 , 1] into uniform sized bins of size λ , for a λ > 0 . Formally the λ -discretization of [0 , 1] , is defined as

Definition 14 ( λ -discretization [11]) . Let λ > 0 . The λ -discretization of [0 , 1] , denoted by Λ[0 , 1] = { λ 2 , 3 λ 2 , . . . , 1 -λ 2 } , is the set of 1 /λ evenly spaced real values over [0 , 1] . For b ∈ Λ[0 , 1] , let

be the λ -interval centered around b (except for the final interval, which will be [1 -λ, 1] ).

It starts by partitioning each subspace S h into 1 /λ groups S h,λ ( b ) = { ( x, h ) ∈ S h | f B ( x, h ) ∈ λ ( b ) } , with b ∈ Λ[0 , 1] . Then, it repeatedly looks for a large enough group S h,λ ( b ) such that the absolute difference between the average confidence value E [ f B ( X,H ) | ( X,H ) ∈ S h,λ ( b ) ] and the probability P ( Y = 1 | ( X,H ) ∈ S h,λ ( b ) ) is larger than α and, if it finds it, it updates the confidence value f B ( x, h ) of each ( x, h ) ∈ S h,λ ( b ) by this difference. Once the algorithm cannot find any more such a group, it returns a discretized confidence function f B,λ ( x, h ) = E [ f B ( X,H ) | f B ( X,H ) ∈ λ ( b )] , with b ∈ Λ[0 , 1] such that f B ( x, h ) ∈ λ ( b ) , which is guaranteed to satisfy ( α + λ ) -multicalibration.

Algorithm 1 provides a pseudocode implementation of the overall algorithm. Within the implementation, it is worth noting that the expectations and probabilities can be estimated with fresh samples from the distribution or from a fixed dataset using tools from differential privacy and adaptive data analysis, as discussed in Hébert-Johnson et al. [11].

Algorithm 1 Post-processing algorithm for ( α + λ ) -multicalibration

C Additional Details about the Experiments

glyph[negationslash]

Transformation of confidence values. In the Human-AI Interactions dataset, the AI model is a simple statistical model where b is just a noisy average confidence h of an independent set of ca. 50 human labelers on each task instance. Moreover, the confidence values were originally recorded on a scale of [ -1 , 1] , where 1 means complete certainty on the correct true label and -1 means complete certainty on the incorrect label. To better match our theoretical framework, we transform all confidence values to a scale of [0 , 1] , where 1 means complete certainty that the true label y = 1 and 0 means complete certainty that the true label is y = 1 . More formally, let ˆ b, ˆ h, ˆ h +AI ∈ [ -1 , 1] be the original confidence values in the dataset, then we obtain b ∈ [0 , 1] via the following transformation:

and analogously for h and h +AI.

Comparing decision policies π B , π H and π H +AI . Figure 6 shows the ROC curves for the decision policies π B , π H and π H +AI in each of the four tasks in the Human-AI Interactions dataset.

0.0

0.2

0.4

0.6

0.8

1.0

False Positive Rate

0.0

0.2

0.4

0.6

0.8

1.0

True Positive Rate

Decision Policies

B

H

H

+AI

(a) Art

0.0

0.2

0.4

0.6

0.8

1.0

False Positive Rate

0.0

0.2

0.4

0.6

0.8

1.0

True Positive Rate

Decision Policies

B

H

H

+AI

(b) Sarcasm

0.0

0.2

0.4

0.6

0.8

1.0

False Positive Rate

0.0

0.2

0.4

0.6

0.8

1.0

True Positive Rate

Decision Policies

B

H

H

+AI

(c) Cities

0.0

0.2

0.4

0.6

0.8

1.0

False Positive Rate

0.0

0.2

0.4

0.6

0.8

1.0

True Positive Rate

Decision Policies

B

H

H

+AI

(d) Census

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCommunicating model uncertainty to human decision makers is critical for trustworthy deployment of assistive AI systems. Prior work, e.g., Vodrahalli et al 2022, have found that people may respond better to miscalibrated AI confidence scores. What underlies this odd behavior? In this submission, the authors explore this phenomenon, specifically developing a new theoretical framework built on structured causal models. In this framework, the authors find that there indeed are some data distributions wherein calibrated AI confidence scores prohibit humans from uncovering optimal policies by which to make their final decision. The authors then propose to resolve this discrepancy through “human-calibration;” multicalibration is demonstrated as one such method. The authors aim to validate their theoretical insights through empirical studies on the same human data from Vodrahalli et al 2022.\n\nSTRENGTHS:\nA substantive assessment of the strengths of the paper, touching on each of the following dimensions: originality, quality, clarity, and significance. We encourage reviewers to be broad in their definitions of originality and significance. For example, originality may arise from a new definition or problem formulation, creative combinations of existing ideas, application to a new domain, or removing limitations from prior results. You can incorporate Markdown and Latex into your review. See /faq.\n\nThis paper has many strengths and was an enjoyable read. The paper was very well-written and incredibly well-motivated. As AI systems increasingly move to real-world applications, it is paramount that they are designed with the human user in mind. The theoretical framework from this paper offers a nice contribution to advance our understanding of the nuances of the way that AI output is communicated to humans to best support decision making. \n\nSome further strengths: \n- The specific problem of looking at monotonicity vs. discrepancy from monotonicty in confidence values is interesting and nicely expands on prior work. \n- The authors make an excellent point on lines 110-114 that human may take diff decisions with same level of confidence; it is good to see that their model incorporates sources of noise ontop of confidence. \n- Nice that their method is claimed to be general to other calibration methods. \n- The mathematics seems sound and steps naturally follow from previous; though while I’ve attempted to check through it, there is a possibility I have missed something (hence my own confidence score of 3).\n\nWEAKNESSES:\nMy biggest misgivings about the paper fall into two categories: 1) the discretization of human confidence, and 2) the empirical validation in Section 6. I will discuss each in turn. I am also concerned that the authors do not adequately discuss limitations. But note this further in the Limitations section of the review. \n\n(1) Discretization of human confidence: \n- The authors cite two papers from the neuroscience literature suggesting that humans have “discrete confidence”; however, the studies in these papers looks at quite different decision making tasks than are considered here. The tasks are much lower level and rote, e.g., simple perception and motor tasks, at least from my reading. These are quite different from the kinds of rich, high-level cognitive tasks that involve say, coming up with a treatment plan for a patient. As such, I feel the methods here ought to be able to handle finer-grained representations of human uncertainty/confidence rather than too coarse binning.\n- I am very open to discussing this — can the authors shed light on how applicable their theoretical results are to different choices of discretization?\n- Specifically on Section 6, I have concerns on the choice of 3 bins. 3 bins are incredibly coarse. Why this number?  For instance, the “mid” confidence level spans a confidence range of 0.5 (out of the normalized 0-1 scale). I recognize that the binning was done here to ensure approximately equal counts across all bins. However, my sense is that this could yield obfuscate some of the nuance in the results? Have the authors considered a finer-grained decomposition, say to 5 or 7 bins? If nothing else, I think this should be discussed in a limitations sections as I imagine this could be an impactful design choice wrt sensitivity in the empirical results.\n\n(2) Empirical validation\n- I am struggling to understand the design of the validation in Section 6. What are the baselines? What are the authors trying to show here to validate their theoretical insights (and which aspects of the theoretical insights?) It is not clear to me what is being shown here. When the authors say “participants benefit from AI advice” in Table 1 [line 333], how is this different from the original paper from which the data is sourced?  \n- The authors note that the results (which per the above, I’d be keen to hear explicated more – what is the expected behavior of these graphs, relative to what baselines?) are weaker for the Census task. Do the authors have a sense for why this may be? \n- The authors grouped the results of cases where participants were told that an AI system had provided the confidence with those told a human had (footnote 9 on pg 8). Is this valid? Could have influenced participants’ uptake of the knowledge? Have the authors explored how results stand up if they decompose / keep those groups separate? Do trends still hold?\n\nQUESTIONS:\n- Many of my questions are listed in the Weaknesses section. In particular, it would be great if the authors could clearly spell out which parts of their experiments (Section 6) validate which parts of their theory. I am open to reconsidering some of my critiques from the Weakness section if this is clarified a bit better. \n- Why 8 or 10 bins? [line 298]\n- Could you expand please on the confounding factors in the data from other counties? I am not entirely sure why you threw away that data? [lines 287-288] My understanding is this was because those participants were told the AI had a different accuracy? This may be worth exploring (perhaps in future work) – how do your theoretical results hold up across wider swaths of data?) \n- How does this relate to RLHF? Please spell out the connection a bit more, or o.w. I actually think it’s unnecessary to mention here. [lines 73-74]\n\nLIMITATIONS:\nThe authors do not adequately discuss limitations, nor any potential negative societal impact. I would be keen for the authors to provide their thoughts on these two points during the rebuttal/discussion period.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a causal model of AI-assisted decision-making that considers both the classifier’s and decision maker’s confidence scores. The authors demonstrate theoretically why it may be challenging for the human to identify the optimal decision-making policy even if the classifier is perfectly calibrated. The authors propose a remedy by providing theoretical results using multicalibration. The authors then provide experimental results to measure misalignment and miscalibration in real-world datasets.\n\nSTRENGTHS:\n- The paper was generally well written and lays out the proposed causal model of AI-assisted decision-making in a step-by-step manner, making it easy to follow\n- The paper covers an important, growing topic on AI-assisted decision-making and highlights the need to consider notions of both human and AI calibration\n\nWEAKNESSES:\nMy main concern with this paper is the lack of empirical results that demonstrate an *improvement* in $\\mathbb{E}_\\pi [u(T,Y)]$ (i.e., show results that the AUC would improve in the H+AI condition as compared to $\\pi_B$ or $\\pi_H$, which is a commonly used definition of complementarity) as stated as a contribution in L62. While they provide a theoretical result (Thrm 5) to guarantee near-optimality under $\\alpha$-alignment, I do not believe the authors implemented multicalibration (the proposed solution in Section 5) in their empirical results. The ideal result would be to show that without multicalibration, human decision makers more often do not achieve complementary performance, as shown in Table 1 last column, and then this could be corrected by introducing some version of multicalibration. Such an implementation would suggest how the theoretical result could be translated to practice and could be very impactful in the human-AI decision-making literature.\n\nQUESTIONS:\nMajor question:\nCould the authors please address the concerns with the empirical results above?\n\nMinor questions:\n- In L121, you state that \"allow the classifier’s confidence to depend on the decision maker’s confidence\". What is a realistic example of a classifier that takes in decision maker confidence?\n- Why is Eq. 4 important to measure?\n- Are there any potential issues that can be introduced if you represent human confidence scores in different ways (e.g., binning into low-med-high or using real numbers like 0.7/0.8)?\n- Can you provide error bars for Table 1?\n- The term \"misalignment\" is introduced in the empirical results and seemed to be an important quantity of evaluation, could the authors clarify why the term was not more formally incorporated in the earlier sections?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper theoretically analyzes human decision making in the presence of AI-provided predictions and confidence estimates. They do this by defining a causal model to characterize AI-assisted decision making along with a sensible assumptions about a rational decision-maker's policy: namely that their probability of making a positive classification is monotone in the confidence values. Under this assumption, the paper shows that the traditionally accepted idea of \"calibrated\" confidence is not enough for a rational decision maker to make optimal decisions. However, a new notion of \"human-alignment\" does provide this guarantee, and in fact is achievable via the well-known idea of multicalibration. The paper also presents some experiments to validate their assumptions and to argue that  human-aligned confidences might lead to better decision making.\n\nSTRENGTHS:\nThis paper approaches the question of AI-assisted decision making from first principles, allowing them to ground their assumptions explicitly and provide a compelling theoretical analysis under their model. The flavour of results in this paper is quite interesting and their techniques build upon and connect a some rich areas of theoretical research. Lastly, the exposition of the paper is quite easy to read, with a good breakdown of high-level ideas and detailed maths in the appendix.\n\nI believe this work would be of notable interest to several communities at NeurIPS.\n\nWEAKNESSES:\nTo me, the weakest part of this paper is the experimental section, which in my opinion doesn't add a huge amount of insight into the results. I like the idea of empirically demonstrating the monotonicity assumption, however the interplay between misalignment and the human decision-makers' ability to perform the task well is mentioned like an afterthought. It would be amazing to see a scientifically valid empirical claim that human alignment leads to better performance with AI-assisted decision making. This is likely a significant task to carry out, and it is fine to claim as beyond the scope of a theoretical paper. However, it would make this an extremely strong submission. \n\nI would also like to see a discussion of the limitations of this work.\n\nQUESTIONS:\nIn several of the experiments, the human performance is worse than that of the model itself. This could be due to several factors, but one reason could be that these humans were not experts at the specified task. In terms of your SCM, they aren't using a lot of latent information V that is hidden to the model. Is there a way to include a \"competence/expertise\" parameter in this SCM to model, and even possibly correct, a non-expert decision maker so that their performance exceeds that of the AI system?\n\nLIMITATIONS:\nLimitations of the work not discussed. What are the limits of this papers' ideas in terms of both theoretical follow-up and real-world usage?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies a binary decision-making process augmented by AI assistance. The decision maker optimizes a utility function satisfying certain natural conditions. The decision rule depends on the decision maker's (DM) confidence and a classifier's confidence. It is shown that there is a lack of alignment between DM's and classifier's confidences; the optimal rule might not have a natural property of placing more trust in predictions with higher confidence values. The paper then proves that a (near-)optimal decision rule with such a property exists if both confidences do not appear contradictory, with high probability, to the rational decision maker, a property dubbed \"human-alignment\". The paper connects this property with multicalibration algorithms for the classifier's confidence function. The paper illustrates the approach on real human data.\n\nSTRENGTHS:\nThe paper introduces a theoretical framework to study optimal decision-making in binary problems, where the decision rule depends on the DM's and classifier's confidence.  The results like this are needed to understand the decision-making process when interacting with AI, particularly now, when strong AI systems are making their way into everyday lives.\nThe theory was illustrated with a real-life data experiment, where the paper asks questions about misalignment, the influence of AI confidence on human confidence values, and a comparison between decision policies.\n\nWEAKNESSES:\nThe setup considered in the paper is very simple. It deals with binary decision problems and the experiment provided in the paper actually does not use any modern ML classifier (see footnote 8). \n\nSection 6 would significantly improved in terms of clarity. In particular,\n* In the data description part, it is not clear how the labels are defined. For instance, for the \"Cities,\" does y is random (line 283), or does it encode the correct answer (in which case, e.g., h='low' would correspond to a human having low confidence in the correct answer). In a similar vein, it would help to explicitly define how are empirical values of $\\mathbb P(Y=1|(X,Y)\\in S_{h,\\lambda(b)})$ estimated (line 319; what exact data are used).\n* Figures 2 and 3 use an \"alignment violations\" term, which is not defined.\n* Figures 2 and 3 should be better explained: each figure contains a set of four plots, each with from 8 to 10 sets of triplets of bars. In particular, the monotonicity in lines 318, 327 should be discussed in a more reader-friendly way.\n* The data used in the experiment were aggregated in various ways. The claims like \"support for our hypothesis that (rational) decision-makers implement monotone AI-assisted decisions policies\" should probably be treated with caution.\n\nQUESTIONS:\nIt would be interesting to see how the Authors envision future work in the context of LLM-based systems, multi-categorical or sequential \ndecision-making.\n\nLIMITATIONS:\nThe paper does not have a limitations section. Natural limitations include the setup's simplicity or the experiments' limited scope.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}