Uncalibrated Models Can Improve Human-AI Collaboration
Authors: Kailas Vodrahalli
Tobias Gerstenberg
James Zou
Venue: NeurIPS 2022 Accept (2022)
Keywords: Human-calibrated AI
Human-in-the-loop AI

Abstract: In many practical applications of AI, an AI model is used as a decision aid for human users. The AI provides advice that a human (sometimes) incorporates into their decision-making process. The AI advice is often presented with some measure of "confidence" that the human can use to calibrate how much they depend on or trust the advice. In this paper, we present an initial exploration that suggests showing AI models as more confident than they actually are, even when the original AI is well-calibrated, can improve human-AI performance (measured as the accuracy and confidence of the human's final prediction after seeing the AI advice). We first train a model to predict human incorporation of AI advice using data from thousands of human-AI interactions. This enables us to explicitly estimate how to transform the AI's prediction confidence, making the AI uncalibrated, in order to improve the final human prediction. We empirically validate our results across four different tasks---dealing with images, text and tabular data---involving hundreds of human participants. We further support our findings with simulation analysis. Our findings suggest the importance of jointly optimizing the human-AI system as opposed to the standard paradigm of optimizing the AI model alone.

Uncalibrated Models Can Improve Human-AI
Collaboration
Kailas Vodrahalli Stanford University kai Failasv@stanfors ord. edu
Tobias Gerstenberg Stanford University
James Zou Stanford University
gerstenberg@stanfordd edu
jamesz@stanford. edu
Abstract
In many practical applications of AI, an Al model is used as decision aid for human users. The Al provides advice that human (sometimes) incorporates into their decision-main making process. The Al advice is often presented with some measure of "confidence" that the human can use to calibrate how much they depend on or trust the advice. In this paper, we present an initial exploration that suggests showing Al models more confident than they actually are, even when the original AI is well-calibrated, can improve human- AI performance (measured as the accuracy and confidence of the human's 's final prediction after seeing the Al advice). We first train a model to predict human incorporation thousands of human-A AI interactions. This enables to explicitly estimate how transform the Al's prediction confidence, making the Al uncalibrated, order to improve the final human prediction. We empirically validate our results across four different tasks tasks- dealing with images, text and tabular data involving hundreds of human participants. We further support findings with simulation analysis. Our findings suggest the importance of jointly optimizing the human-Al system as opposed to the standard paradigm of optimizing the Al model alone.
AI advice using data from
Introduction
In safety-critical settings like medicine, Al is often integrated as interactive feedback, allowing a human decide when and what extent the Al's "advice" is utilized [0)] For example, in medical diagnosis tasks, this places the AI in a similar category as lab tests or other exams a doctor may order
aid diagnosis. This mitigates the often black box nature of AI algorithms that limit a user's trust
and usage of AI 6 24 34
Typically, these algorithms
designed and optimized to minimize loss using standard training objectives independent of the human end user. In this paper, we question this premise and ask whether joint optimization of the entire human-Al system is possible. In particular, the conventional wisdom that models with calibrated confidence enable more accurate transfer of prediction uncertainty between models and/or humans X We propose calibrating model for human use, with the intuition that humans are often unable to accurately integrate calibration information correctly and may benefit from uncalibrated information. This approach is complementary to existing ideas on how to help humans utilize Al advice (e.g., through some form of explanation accompanying the advice or through feedback on model/human performance)
revisit
36
We draw on prior work modeling human predictions 30 and assume black box access to an AI algorithm which provides advice. From data collected on thousands of human interactions, fit model for how humans incorporate (or ignore) Al advice. Using this learned human behavior model, optimize monotonic transformation that modifies the Al's reported confidence maximize human-AI system performance (measured as the human's 's final accuracy). We demonstrate first in simulations and subsequently with real-world data using crowd-soureed human study participants
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

that our modified Al advice results in higher overall performance. These results suggest there is merit to jointly optimizing human-AI systems and that an optimal Al algorithm may not be optimal for human use.
Our contributions We propose and empirically validate a framework for optimizing an Al algorithm with respect to the human end user, resulting in human-calibratte AI that calibrates AI models for human use. We only require black box access the AI. Our framework relies on a model for human behavior, which we create using a large collection of empirical data on human interaction with AI advice. To the best of our knowledge, these are some of the first results using an empirically validated model for human behavior to optimize AI performance in collaborative human-Al system. We would like to emphasize, however, that this these methods are not currently suitable for practical use.
Ethical considerations One of the main conclusions of our study is that human users do not consider confidence in the "optimal" way (i.e., how model confidence in statistics). In an attempt correct for this, we present both empirical and simulation-baser evidence that modifying the presented AI advice improves overall performance. One potential question about our approach is: does modifyyng the AI's confidence constitute misleading the user? We agree in our experiments, the Al can mislead the user. However, our goal here is not to propose method that should be used in practical applications, and rather to highlight the importance of modeling human interaction when designing Al for human use.
Related works The experimental setup we use comes from the advice utilization community. Here, studies often use the JAS framework for measuring and comparing the effect of different forms of advice (88) In many of these studies, algorithmic advice is manipulated to elicit various responses from humans. We take this step further in our paper by manipulating the advice to improve model calibration with respect to the user and improve performance. Previous work has studied the effect of peer vs. expert advice 155, automated vs. human advice p2 IS I1,, task difficulty 77), and advice confidence 25 26 Some work has also sought to characterize the perceptions people have of an AI when making decisions with Al advice and how these perceptions affect trust and utilization of the AI [3 Of particular relevance to our work is 25 which provides evidence that uncalibrated, confident advice more utilized by humans in certain settings. We also leverage a proposed model for human behavior from 30
The conventional approach human-Al collaboration is to ensure model and human calibration g 3 and to make the Al's predictions explainable [3 27 These explanations can affect human understanding of AI confidence 12 isolation (i.e., maximize the AI performance). However, recent work suggests that the objectives used to optimize the AI do not adequately consider the human-AI team. Alternative approaches include learning to defer, where a classifier determines when expert input is required [6 19;
35 56
14]. It is also standard practice to optimize the AI in
learning to complement, where the AI optimized to perform well on only the tasks that humans struggle with B3]; and methods that optimize objectives with costs assigned to requesting AI advice and/or human expert input utilization M. Our work most related to M. which also emphasizes the importance of confident Al predictions for human decisions. Our approach is complementary to these prior works and contains several novel aspects. We model human behavior using empirical data, demonstrate that this human model allows us to optimize the Al advice, and demonstrate both in simulation and empirically that our modified advice improves overall system performance. Furthermore, we only require black-box access to the AI advice. In relation to [.]. we further differentiate our work as the model for human interaction is different - we give advice after the user makes an initial assessment.
2 Experimental setup
Our goal is to augment the performance of a human- AI system by optimizing the AI for use by human. To make this goal tractable, we assume (1) that we have a model for human interaction with Al advice, and (2) that we have black box access to the AI algorithm producing the advice.
Human-Al AI interaction
The Auman-AI interaction involves two stages. First, a human is shown a task and completes Al advice is then presented, and the human is allowed to modify their response. As we measure performance before and after receiving advice, this model is conducive to studying the effect of
advice and so sees common usage in the literature 22 30 17

---

Data collection and tasks
We collected data from crowd-sourced participants a diverse of tasks that all have the same basic design. We focused on binary prediction tasks where humans respond on a continuous scale. The use of the continuous scale allows for more rich analysis of the user's prediction than just the simple binary prediction label would permit.
Participants were shown sliding scale with each end indicating a different binary response. They were instructed to move a pointer on the scale to the location that indicated their confidence and their response, with the midpoint the scale indicating an uncertain response. Participants submitted an initial response; subsequently. the advice was indicated with a visual marker on the scale and participants were allowed to adjust their initial response. Participants were incentivized so that higher confidence in correct and incorrect answers increased and decreased their monetary reward respectively.
We collected data on set of four tasks from diverse data modalities. These tasks were originally proposed in prior work [30] The tasks were designed to be accessible to the general public to allow use of crowd-sourced study participants for obtaining empirical data. See below for a brief description of the tasks we used; examples of task instances from each of the four tasks are shown in Appendix B222 The data collected summarized in Table [ Each task consists of 32 binary questions. For each task, the 32 questions were carefully selected to have a range of difficulty. More details on the distribution of task difficulty is presented in Appendix B.3
All participants received the same set of questions, though the order was randomized for each participant. All participants also received the same advice, although small amount of noise was randomly added for each participant. The advice had an accuracy of ~ 80% for each task; participants were informed of this accuracy. However, no feedback on AI or human accuracy on individual questions was provided during the experiment.
Since the datasets we are using are relatively small and curated primarily for human use rather than AI training, we opted to use aggregated human responses as proxy for an AI model. In particular, the advice was constructed by averaging the responses of prior group of human participants who responded to the questions without receiving any advice. This allowed us to ensure the advice given was "reasonable" in the sense that the advice is generally less confident on difficult tasks and more confident on easy tasks. We checked the calibration of this advice by computing the expected calibration error (ECE) (20]. The ECE for AI advice 0.074, indicating the advice is roughly calibrated (0 indicates perfect calibration).
We recruited 49-79 participants for each task, resulting in several thousand datapoints (each question is one datapoint). Participants are US residents, roughly 50% are female, and the average age is approximately 30 years old. Despite limitations in the age distribution and geographic location of participants, survey question we asked indicated varied range of Al familiarity in our participants.
We followed standard practice in collecting data. Our study protocol was approved by our institution's 's IRB board. The data collection process was low risk, and we ensured samples contained diverse participants across age, sex, and ethnicity. We compensated participants at 2 $10.00 per hour (depending on bonus pay). The total cost was around $600 (Section 2 + $750 (Section 5.2).
We have four types of data with corresponding user tasks. Art dataset (Image data) Contains images of paintings from art periods: Renaissance, Baroque, Romanticism, and Modern Art. Participants were asked to determine the art period a painting is from given binary choice. Cities dataset (Image data) This dataset contains images from 4 major US cities: San Francisco, Los Angeles, Chicago, and New York City. The task is to identify which city image is from given binary choice. Sarcasm dataset (Text data) This dataset is a subset of the Reddit sarcasm dataset 100. which includes text snippets from the discussion forum website, Reddit. Participants to detect whether sarcasm was present in a given text snippet. Census dataset (Tabular data) This dataset comes from US census data 62 The task is to identify individual's income level given some of their demographic information-4e e.g. state of residence, age, education level.
asked
3

---

"Activation" model
"Integration' model
Al advice
Decide to use advice
Use advice
Ougpot: New response
initial response
"Yes"
If "No"
Output Initial
Figure 1: Activation inntergatnnnne model for human behavior.
Table 1: Summary of data used for fitting human behavior model. This data is publicly available at https://githu..com/ waluaaaaaall- al in-ineeraations
Partici- pants
# Observa- tions
Activation Rate
Accuracy (before advice)
Accuracy (after advice)
Task
Art 68
2176
0.523
63.7%
75.5%
Cities
79
2528
0.557
70.7%
77.1%
Sarcasm
49
1567
0.347
72.7%
76.1%
Census
50
1600
0.475
70.1%
73.6%
3 Human behavior model
Table D provides a summary of the data collected for developing our human behavior model. We find that humans actually changed their response after receiving advice roughly 50% of the time, with exact number shown in the "Activation Rate" column of Table I When person does change their response. we call this person "activated." Specifically, person is activated their final response changes by at least small amount.
This observation motivates simple, two-stage model for human behavior: a human first decides whether to modify their response ("activation stage"), and subsequently, if they were activated, they modify their response ("integration stage"). A diagram of this model is shown Figure This two-stage model is consistent with previously proposed models from the psychology and HCI literature
We fit functions for each of the two stages: an activation model that predicts how likely it is that a person activated, and an integration model that predicts how the person modifies their advice, assuming they are activated. Taken together, we can model human behavior in the two-stage human- Al system we use to run our experiments. We define this model's behavior more rigorously in Section 4 Our fitted models will only be used to optimize our advvic-moddifying function and will not be used test-time. As such, we can utilize demographie information about the participants fit the model, though such information may not be available during a real-worid deployment. We opt to fit a small neural network for each these stages, as we require a differentiable model (for easy optimization) with moderately more complexity than a linear model.
3.1 Activation model
Our activation model takes 12 input features extracted from a single person-task instance interaction. It outputs a probability for whether the person will be activated on the given task instance. The input features are based on the person's 's initial response, the advice, and several demographic features including age and sex. We do not encode the initial response AI advice directly as the label (the

---

15
dependence
Ave predicts scivation rate
076
0.58
3.0
028
0.25
8 0.05
0.26
-05
4.58
4.79
-1A
Advicc (negative incorrect label)
(a) Activation model PDP for Al advice
(b) Integration model
Heatmap
Figure 2: Our trained model for human behavior: (a) Partial dependence plot measuring the effect of the advice on activation. (b) Heatmap of the average behaviour of our integration model for human behavior when we vary unmodified AI advice and the human's initial response.
left or right slider location of the correct answer) is randomized for each participant. Instead, we use the response and advice confidence, as well as a binary feature that encodes whether the response and advice agreed on the label. This implies a symmetry in the model's predictions. Please see Appendix D. for a description of all the features used.
We use 3 layer neural network with ReLU non- con-linearities for the activation model. We also attempted use other simple algorithms for modeling the activation behavior of humans, like linear models, but found the neural networks to perform better. We trained the network using binary cross- cross-entropy loss with an Adam optimizer [ and stopped training when validation loss increased, as is standard. The activation model achieves ROC-AUC > 0.81 on held-out test participants, suggesting reasonable performance despite the high variability in our dataset.
In Figure 2 we show a partial dependence plot for the learned activation model. This plot shows the average behavior (across our entire test dataset) of the activation model when fix the AI advice to be a certain value (the x-value) for all data points. The Al advice can be positive (correct advice) or negative (incorrect advice), while the magnitude corresponds the advice confidence. We note that (1) there is a base level of activation around 50% and (2) our activation model predicts people will be most activated when the advice is confidently incorrect. This occurs because (a) most people in our dataset get answers correct, on average, and seeing confident advice of the opposite label tends make people change their initial response (e.g. by decreasing confidence even the predicted label does not change). And because (b) confident advice that agrees with a person's 's initial response (whether correct or incorrect) tends to increase their confidence. We also observe that (3) the activation rate increase when advice is confident and correct for the same reason previously noted (2b). The activation rate lower for correct advice than for incorrect advice as most people in our dataset are, on average correct, which tends to invoke smaller response.
3.2 Integration model
Our integration model similar to our activation model, but optimized for a different function. In particular, we use the same input features, model architecture, and training procedure. The integration model outputs a prediction for sign(r1)(r2 rı). where F1,72 E [-1, 1] are the initial and final responses respectively. Note that 72 probabilities normalized to the [-1, 1] scale. The training minimizes the RMSE between the predicted change and the actual change. The optimization is performed only the subset of the dataset that was actually activated (roughly 50% of the data). Our integration model has a reasonable performance, with an RMSE of 0.25 (the maximum absolute possible is 2) and R of 0.73 on test data.
In Figure 2
we plot heatmap of the fitted integration model's behavior. We plot the average output of our integration model across our test data split when the person's 's initial response (x-axis) and Al advice (y-axis) are fixed certain value for all datapoints. The sign and magnitude of the initial response and AI advice correspond to correctness (negative is incorrect) and confidence. The
5

---

integration model prediction can be positive (increasing confidence in the initial predicted label) or negative (decreasing confidence the initial predicted label / change in the predicted label).
We observe three features in the heatmap: (1) The heat map is symmetric across quadrants 1, 3 and 2, 4. This is by design. As the model does not the initial response and Al advice values directly it uses their magnitudes (confidence) as well as a third, binary term indicating whether they agree on label the integration model can only predict a delta change relative the sign of the initial response. (2) The output has largest magnitude when the advice is confident and opposite the person's 's initial response. (3) If the advice and the person's initial response share the same label, the integration model output is largest when the person's imitial response has low confidence and the model has high confidence. These observations confirm our integration model has reasonable operation behavior.
Modifying the Al advice
16
Adeco 115% eccurayl trunlamm
Adric 125% accuraof burolam
02
idantly furafam Iaaiiinl
TearsformiA Adbvio)
0.00
60
-025
-0.75
-1 00
-05
0.30
0.73
Adviic
Initial
(a) Fitted human behavior model (ga+,8-)
(b) Section
4.2 example
Figure 3: Simulation results. (a) Fitted modified advice vs. original unmodified advice. We show results across synthetically varied average advice accuracy in the training data. The curve labeled "Advice given" is what actually show people in experiments. (b) Optimal human-calibran Al output for the setting described in Section 4.2 Heatmap shows delta change in advice compared to calibrated advice: red and blue values indicate where the AI should be under- and overconfident respectively in order to achieve the best human-A AI system output.
Here discuss how modify the raw Al advice output. First let's establish notation. The notation described below in reference to a single datapoint; later, we will add the subscript to index across datapoints.
Let A E denote the inverse sigmoid of the advice presented to participants (i.e., we present the AI advice of (A)). We will work with A rather than (AA here, as we will aim to replace the transform with an optimized function that improves human-Al performance.
Let g: I- + [0, 1] be function (which we will optimize) that maps the Al advice to probability that the user is shown. Note that for the baseline, "unmodified advice," we set g to be the sigmoid function, (I) T+e T
Let 1et 72 be the initial and final responses from a participant (r2 a function of ri, A, and other features), and let and let x = (r1, A, u) be the input feature vector to factivation and funtggration- our activation and integration models for human behavior models described in Section B To simplify notation, let fintegration denote the function that outputs the predicted r2 for an activated person (rather than the delta from r1 that we optimize for). Finally, let fin represent the entire model for human behavior. We assume all feature preprocessing is hidden inside factivation and fintegration for convenience notation.
{0, 1} denote the true label. Let u represent demographic features of a person,
4.1 Optimizing for human-AI performance
We consider g of the form
g(A) (A) 1 (|)(A)))))+)))*
(1)

---

where 8 E R20. Note that setting (a. 3) = (1.0) results in 91.0 the sigmoid function used produce the unmodified advice. Note that we do not exactly perform linear transformation on A: the |A| and sign(A A) terms are included to ensure that does not change the label recommended it makes the function symmetric around increases with Al confidence, while 3 adjusts the minimum confidence level of the presented advice. For example, 3 1, the presented confidence is Itc > 0.73 (relative to the chosen label). In general, we can use other differentiable functions for g. we chose the form in Equation because is easy to optimize, simple to understand, and is still flexible enough to fit our data.
O. modulates the rate which the presented advice
As shorthand for when modify the features input to factivate and Jintegrates denote g(x) (g(A) (A), r,, u). We will optimize for a; 3 to maximize the expected final accuracy using our human behavior model. fHB. As discussed in Section 3 fHB has the following representation:
fHB(g(x)) f fuu
r1
w.p. 1 factivase (ga, (x))
inttegrnee (
w.p. Snctivate (grr (x))
We then solve the following optimization problem to obtain a, B: (a", ,²) arg min E [L(HHB(gx))), v)l. a.9
(2)
where is the binary cross entropy loss function. The expectation is taken with respect to the randomness in fHB and the data distribution. Given training dataset D, we optimize the following expression:
arg min a.3
Σ
B[L(Jn(((xx)) w:)]
(x., (X.D
arg min a,8
Σ
L(ri, y1) (1 factivate (g(x:))) + [iiitegreee (g(x:)), yi) factivane (g(civa (g(x:))
(x. (
practice, we carry out this optimization using stochastic gradient descent, which is possible as we choose factvate and fintegrate be differentiable functions (see Appendix D.2). For data, we used the empirical data described in Section The full training procedure visualized Figure
Al Advice
Al advice (4)
Bag (A)
Backpropagate through behavior model gap(A)
Human behavlor model
Activation model
Integration model
Output: New response
Person's initial response
Logistic loss
Output: Inilial
response
Figure Overview of optimization procedure for Ja.8- Al advice is input through Ja.s- The transformed advice is then input into the human behavior model. Finally, loss is computed and the gradient with respect to the loss is backpropogated to update a and 3. Note here that the human behavior model was trained previously and is now fixed.
Once we have optimized for a" we can apply it to new data without needing the user demograph- ics u (i.e., we do not personalize the confidence transformation choice of loss function is arbitrary here (we used logistic loss as our task is binary classification). In particular, for high risk settings where mistake is very costly, the a 3 could be selected accordingly using appropriate loss function.
individual users). Note that the

---

Table 2: Simulated performance difference of modified advice ("Advice given" curve in Figure 3 to unmodified advice. Positive difference indicates higher value for modified advice. Results shown across three average advice accuracy levels. The bolded row is the advice used in our human experiments.
Advice Accuracy
Final Accuracy
Correct Confidence
Activation Rate
75%
+0.7%
+0.097
+0.083
79%
+1.9% %
+0.114
+0.091
85%
+3.1% %%
+0.131
+0.111
4.2
When does under/overconfident advice help?
Here describe simple theoretical situations where uncalibrated advice improves performance. We consider a binary classification task and assume we want to minimize the logistic loss of the human-AI classifier. We also assume (1) constrained version of the activation-integration model perfectly models human behavior; in particular, humans either use their initial response or follow the given advice exactly. And (2) we have an oracle for human behavior and know exactly the calibration of the human and AI advice. We will now construct function f(a, r1) that outputs a", the optimal advice to minimize logistic loss of the human-AI system.
To select a", we need to consider two quantities:
Lr, - [pr, log(r1) 1 + (1 - Pr. log(1 r1)l
(3)
Lo [P log(a") + PP) log(1 a'))
(4)
Here, Pa and Pr, are the probabilities the advice and initial response are correct respectively. Then Ly, relates the log-loss for non-activated humans and L.- relates the log-loss of following the modified advice, a".
Now simply select a" to minimize the expected loss incurred by the human-Al system L Ly, + Pactivation Laa
a' = arg min (1 Pactivation
(5)
for Pactivation = Sactivation (a", ,r,, u) (which we assumed we know exactly). For example, consider
setting where
Pactivation = 1// +c<
Pr. = f(r1)
where
2), e E [-0.5, 0.3), and for generic function f [0. 1] - [0, 1]. Here human under/overvalues their response relative to the advice (depending on e), and the human not calibrated (depending on f). In this setting, it is indeed optimal for the advice, a', to be under/overrooffident to correct for human misutilization of advice (depending on the advice and initial response). An example where e 0.1 and f(1)) = rf is shown in Figure 3p. Further discussion included in Appendix
5 Results
5.1 Simulation results
Optimizing Equation 2 results in an optimal a", 3 Using fHB, we can then analyze the expected benefit of f.-.9- -98- over the baseline, 91,0- In Figure 3h, we show .e. We plot three separate curves. The curve labeled "Advice given" is optimized using the previously described dataset, and this is the function we use in our empirical results. We also synthetically modify the dataset by biasing the advice towards the correct label and adding noise the advice respectively increase and decrease the average advice accuracy in the dataset to 75% and 85%. We then fit new (ax, 3) to the modified dataset and plot these curves. Note that f is fit to the unmodified dataset and fixed for optimizing all three of these curves.

---

Table 3: Empirical performance on UK-based participants, Showing the difference in metric between modified advice and baseline (value is > when modified advice resulted in larger value). We recruited 50 participants for each task.
Task
Final Accuracy
Correct Confidence
Activation Rate
Art (90%)
+8.5%
+0.150
+0.101
Cities (87%)
+1.7%
+0.140
+0.169
All (89%)
+5.2 %
+0.149
+0.143
For all three curves, we notice there is large discontinuity at A = 0.5. This is a result of a large f and suggests that advice is not useful humans unless is certain level of confidence (e.g., to make the human activated).
That all three curves follow relatively similar paths suggests the performance of . may be robust to small changes advice accuracy. This is confirmed (in simulation) in Table 2 where we show simulated performance difference between modified and unmodified advice. We see here that the final accuracy, confidence in correct result, and activation rate all increase with the modified advice.
5.2
Human experiments
0.30
0.175
Al Asvice
Medified Saseline
Confrdenncs in Concect Labal Deta
Medified
0:25
0.125
0.20
Acorray
0.15
0.075
0.050
0.255
0.05
0.000
AIt
CB45
Sarcasm
Cersus
0.00
Art
CRiss
Sarcaom
Census
(a) Accuracy
(b) Confidence
Figure 5: Our empirical results: plot the average change between the first (before advice) and second (after advice) responses in (a) accuracy and (b) confidence of the correct label, averaged by participant. These results are consistent with simulation findings of Table The error bar represents +1 standard deviation.
We verify our simulation results across the four previously described tasks (with the exception that we do add noise to the advice, so all participants see the exact same advice now). Adult, US-based participants were recruited through the Prolific crowdworker platform [23]. We recruited
50 participants for each task, and randomly assigned either the unmodified (baseline) modified advice to each person. In Figure 5 we show the average change in (a) accuracy and (b) confidence of the correct label (the average confidence, with correct and incorrect labels receiving positive and negative weight respectively), partitioning by the advice received by participants. Accuracy and confidence increase across all tasks when using advice (p = 3.84 x 10-6) -). Morcover, we can largely confirm simulation results. Accuracy and confidence increased due to the modified advice across nearly all tasks. Additionally, activation rate (not plotted) increased with the modified advice, with an average increase of 3.3% across tasks.
Another interesting finding that tasks with higher advice accuracy (Art and Cities) exhibit larger increases in accuracy and confidence. The AI advice is more accurate in Art and Cities (accuracy: 90% and 87%) compared Sarcasm and Census (accuracy: 78% for both). This intuitively makes sense we are making the advice more confident in its predictions, and so while this benefits all tasks, it disproportionately benefits tasks where the advice more accurate. This effect was predicted by our simulation results.

---

Our primary human experiments involved US-based participants. To show our findings generalize, we took the advice modification learned on the US data and applied it to new participants based in the UK for the Art and Cities tasks. The findings are highly consistent: modified advice increases activation and improves final human accuracy and confidence (Table 3
Combining US and UK
data, performance improvement is significant for both accuracy (p 8.86 x 10-3) and confidence
(p- = 4.43 x 10-11).
Discussion, Limitations, and Future Work
In this paper, we proposed optimizing simple function to better calibrate AI advice for human use in a collaborative human-AI system. To optimize the reported confidence, we trained human behavior model from experimental data and used it simulate the full human-Al system. The modified Al advice are explicitly uncalibrated, but in both simulated and empirical results, resulted in substantially higher final human performance. We further validated our model using new participants from different country. These results highlight the importance of recognizing that AI, when deployed in collaborative systems, should not be optimized in isolation. Our results also suggest that we can view modifying Al advice as a nudge for the human user. If the Al says its confidence is 0.7, the user may not fully absorb the import of 0.7 since humans reported confidence could be a useful nudge to help users to better absorb the information. This view is supported by prior work suggesting humans prefer clean decision boundaries
not great at intuitive probability. Modifying the
Our work has several limitations that would be interesting to address in future studies. In particular, is unclear how reliable such method would be in practice. We argue that our method for creating human-calibratee calibrated A/ can be deployed in practice, but requires an accurate model for human behavior. In particular, if we have access to oracle for an individual's utilization of AI advice, we exactly optimize the output of a calibrated Al model to correct for the individual's 's biases. Note that this may lead to under- or overcalibrated advice depending on the user and the advice confidence. However, as we cannot perfectly model any given individual, the question to ask is whether we can create a robust model for human behavior (c.g.. with confidence intervals etc.) and then optimize with respect to that model while accounting for error. This extension was out-of-scope for this paper, and is an important direction of future research. Other limitations include: considered tasks where advice accuracy generally exceeded individual human performance and did not evaluate expert tasks (e.g.. involving clinicians making clinical decisions). We also did not consider how human interaction with an AI evolves over time- that is, we assumed each individual to be static over the course of the study, while in general, humans may naturally adapt the AI and modifying the Al may affect this process. These are also important directions for future research. Finally, we recognize that the human behavior model we used in this work does not capture all real-world settings. It may be important to consider different models for different applications, and our results will need to be validated in these new settings.
References
[1] Gagan Bansal, Besmira Nushi, Ece Kamar, Eric Horvitz, and Daniel S Weld. Is the most accurate ai the best teammate? optimizing for teamwork. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1 1400-111414 2021.
[2] Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S Lasecki, Daniel S Weld, and Eric Horvitz. Beyond accuracy: The role of mental models in human-ai team performance. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 7, pages 2-11, 2019.
[3] Erin K Chiou and John D Lee. Trusting automation: Designing for responsivity and resilience. Human Factors, page 101999982110099995 2021.
[4] Joshua R De Leeuw. jspsych: A javascript library for creating behavioral experiments in web browser. Behavior research methods, 47(1):1-12, 2015.
[5] Mary T Dzindolet, Linda G Pierce, Hall P Beck, and Lloyd A Dawe. The perceived utility of human and automated aids in a visual detection task. Human Factors, 44(1):77 (11:71)99994)))) 94, 2002."
[6] Robin C Feldman, Ehrik Aldana, and Kara Stein. Artificial intelligence in the health care space: how can trust what we cannot know. Stan. L. & Pol'y y Rev. 30:399, 2019.
10

---

[7] Francesca Gino and Don A Moore. Effects of task difficulty on use of advice. Journal of Behavioral Decision Making. 20(1):21-35. 2007.
[8] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration modern neural networks. In International Conference Machine Learning, pages 1321-1330. PMLR, 2017.
[9] Nigel Harvey and Ilan Fischer. Taking advice: Accepting help, improving judgment, and sharing responsibility. Organizational behavior and human decision processes, 7002):117-133: 1997.
[10] Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli. A large self-annotated corpus for sarcasm, arXiv preprint OrXiv:/704.05: 2017.
[11] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6960 2014.
[12] Vivian Lai and Chenhao Tan. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the conference on fairness, accountability, and transparency, pages 29-38, 2019.
[13] Min Kyung Lee. Understanding perception of algorithmic decisions: Fairness, trust, and emotion in response to algorithmic management. Big Data & Society, 5(1)2053951118855664, 2018.
[14] Zhuoran Lu and Ming Yin. Human reliance on machine learning models when performance feedback limited: Heuristies and risks. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-16, 2021.
[15] Poornima Madhavan and Douglas A Wiegmann. Effects of information source, pedigree, and reliability on operator interaction with decision support systems. Human Factors, 49(5):773- 785, 2007.
[16] David Madras, Toniann Pitassi, and Richard Zemel. Predict responsibly: improving fairness and accuracy by learning to defer. arXiv preprint (7t:::/71.0664 2017.
[17] Neda Mesbah, Christoph Tauchert, and Peter Buxmann. Whose advice counts more-man or machine? an experimental investigation of ai-based advice utilization. In Proceedings of the 54th Hawaii International Conference on System Sciences, page 4083. ScholarSpace, 2021.
[18] Tim Miller Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence, 267:1-38, 2019.
[19] Hussein Mozannar and David Sontag, Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning, pages 7076-7087. PMLR, 2020.
[20] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
[21] Dilek Önkal, Paul Goodwin, Mary Thomson, Sinan Gönül, and Andrew Pollock. The relative influence of advice from human experts and statistical methods on forecast adjustments. Journal of Behavioral Decision Making, 22(4):390-409, 2009.
[22] Andrew Prahl and Lyn Van Swol. Understanding algorithm aversion: When is advice from automation discounted? Journal of Forecasting, 36(6))69111))) 2017.
[23] Prolific academic.
2022.
[24] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1145-114. ACM, 2016.
[25] Sunita Sah, Don A Moore, and Jobert J MacCoun. Cheap talk and credibility: The consequences of confidence and accuracy on advisor credibility and persuasiveness. Organizational Behavior and Human Decision Processes, 121(2):246-555 2013.

---

[26] Thomas Schultze, Anne- Femm-Feennndiee Rakotoarisoa, and Stefan Schulz-Harc Effects of distance between initial estimates and advice on advice utilization. Judgment & Decision Making, 10(2), 2015.
[27] Donghee Shin. The effects of explainability and causability perception, trust, and acceptance: Implications for explainable ai. International Journal of Human- Computer Studies, 146:102551, 2021.
[28] Lyn M Van Swol, Jihyun Esther Paik, and Andrew Prahl. Advice recipients: The psychology of advice utilization. In The Oxford handbook of advice, page 21-41. Oxford University Press, 2018.
[29] Kailas Vodrahalli, Roxana Daneshjou, Roberto Novoa, Albert Chiou, Justin M Ko, and James Zou. Trueimage: A machine learning algorithm to improve the quality of telehealth photos. In BIOCOMPUTING 2021: Proceedings of the Pacific Symposium, pages 220-231. World Scientific, 2020.
[30] Kailas Vodrahalli, Tobias Gerstenberg, and James Zou. Do humans trust advice more if it comes from ai? an analysis human-ai interactions. arXiv preprint arXiv:2107.07015, 2021.
[31] Katharina Weitz, Dominik Schiller, Ruben Schlagowski, Tobias Huber, and Elisabeth André. do you trust me?" increasing user-trust by integrating virtual agents in explainable ai interaction design. In Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents, pages 7-9, 2019.
[32] Alex West and Anusha Praturu. Enhancing the census income prediction dataset. https: // people. ischool. ber.eley. edu/ http//llleeests111_oeeoeme http://wwwoooeessssseeeee aancome_html. 2019. Ac- cessed: 2021-05-15.
[33] Bryan Wilder, Eric Horvitz, and Ece Kamar. Learning to complement humans. arXiv preprint arXi::2005.00588 2020.
[34] Yao Xie, Melody Chen, David Kao, Ge Gao, and Xiang' Anthony' Chen. Chexplain: Enabling physicians to explore and understand data-driven, ai-enabled medical imaging analysis. In Proceedings of the 2020 CHI Conference Human Factors in Computing Systems, pages 1-13.
ACM, 2020.
[35] Ming Yin, Jennifer Wortman Vaughan, and Hanna Wallach. Understanding the effect of accuracy on trust in machine learning models. In Proceedings of the 2019 chi conference on human factors in computing systems, pages 1-12, 2019.
[36] Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making. In Proceedings of the 2020 Conference Fairness, Accountability, and Transparency, pages 295-305, 2020.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] We include instructions and materials to reproduce experimental results in the Supplement.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In addition to details included the paper, we include additional details in the Supplement.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A] Compute resources were minimal (i.e., could be run single laptop in under a minute)
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]]
(b) Did you mention the license of the assets? [No] The assets are all for public use and we cite the references.
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 'r using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] Sarcasm dataset was filtered to omit potentially offensive content.
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [Yes] In appendix
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [Yes]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [Yes]
13


Reviews:
[{"review_id": "sJKnpihOXUz", "reviewer": "Reviewer_4Md1", "summary": "The paper shows that calibrated models are not always the best for human-AI systems. Optimizing the ML model alone with human *uncalibrate* the model, but resulting in better performance. The paper provides rigorous experiments and related details. \n", "questions": "1. The paper shows that it's important to optimize the ML model with humans or potentially a model for human behaviors. It reminds me of the model-based RL problem. How does the reliability of the model for human behaviors affect the final human-AI performance?\n\n\n**Suggestions**\n1. Since the paper is related to human perceptions of the model's confidence, it will be great to have some human studies or interviews", "limitations": "The paper is easy to follow and the idea is well justified. As described in the last section, the experiment settings are limited and do not capture many real-world cases.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n1. The paper provides extensive and detailed experiment results that challenge the common belief of \"better-calibrated models lead to better human-AI systems.\"\n\n\n**Weakness**\n1. The experiments are a little bit toy, with only two actions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "qYMh5pqOGyK", "reviewer": "Reviewer_LVB2", "summary": "The paper proposes a method for adjusting model confidence such that the output model decision/recommendation it is best leveraged by a human. The method is based on the observation that humans may not be able to interpret confidence in the form of probabilistic estimate accurately, and may therefore prefer less calibrated notions of confidence. The authors show this via a set of user studies in 4 different domains. They use the data from these user studies to learn a model that re calibrates confidence depending human behavior, and they show that this benefits final team performance. ", "questions": "My main questions are related to W1 (comparison to baselines) and W4 (what happens when the initial model is not calibrated). \n\nMorevoer, there is an overall question on how these results would change when the model accuracy varies. One can assume that if the model has low accuracy, it will be less often useful to the user and gains in team performance will be lower. There could be a discussion around these in the limitations section.", "limitations": "The most important item to address here is the fact that depending on the cost\\reward function of accurate and non-accurate decisions, uncalibration may lead to over reliance, which in the real world may have implications on safety.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nS1- The paper studies the problem of adjusting AI behavior specifically to improve human ai collaboration. This is a problem that deserves more attention in the community, especially for high-stake domains where the model is intended to augment a human and not operate autonomously.\n\nS2- It verifies the observation on \"uncalibration helping team performance\" on several datasets and user studies. \n\nS3- The method is practical to use and can even be used on an already trained model, assuming that that model is somewhat calibrated.\n\nWeaknesses:\n\nW1- The paper is missing comparisons with some important or obvious baselines. For example, comparing with a simple and pure step function (figure 3a) where the model only reveals its answer, this being an extreme case of uncalibration. Most importantly, there could have been a comparison with related work [1] mentioned in the literature. Technically, this is the analogous version of this paper which attempts to achieve similar results but during model optimization and not post hoc. That said, the relationship of the paper with [1] needs to be made more clear as the goals are really similar. The authors may also want to check the following paper [*] from the same authors, which provides evidence that humans prefer clean, non-stochastic error boundaries. Uncalibration as proposed by this work falls within the same framework, if the error boundaries are defined by using confidence.\n\n[*] Beyond accuracy: The role of mental models in human-AI team performance\n\nW2- Except [1], authors could also discuss post-hoc calibration in related work given that the proposed method is indeed a post-hoc calibration method for HAI. Note that the paper often claims for this to be an optimization approach but since it operates only on model confidences it is not such. This is not necessarily negative (in fact sometimes may be advantageous), but the discussion and framing of the approach needs to be made more precise.\n\nW3- Depending on the utility function, uncalibration can be dangerous, if for example mistakes are associated with a much higher cost than the reward for correct predictions. This limitation needs to be discussed accordingly.\n\nW4- The proposed approach may not work as well when the original confidence scores from the model are not calibrated. In this experiment they are reasonably calibrated since the model output is the majority vote of crowd answers. However, if this is not the case, the post-hoc calibration may not change much the situation or it may make it worse.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4Z58q4bHJ-", "reviewer": "Reviewer_jiKw", "summary": "This paper focuses on optimizing AI advice when AI is used as second opinion for a human decision maker. They propose a model for human interaction with AI that can be learned from data into two parts: an activation model and an integration model. They then provide a strategy to modify AI confidence given this human model to maximize task performance. They validate their results on four experimental tasks where they show that their approach of optimizing AI confidence increases performance compared to showing humans calibrated probabilities. ", "questions": "“One potential question about our approach is: does modifying the AI’s confidence constitute misleading the user? We do not view this as misleading since a human’s assessment of confidence may differ from the statistical notion of calibrated probability.” – This is the most major point about this work. Even though I am positive about this paper, it is absolutely misleading the human and such approaches cannot be ethically implemented in practice as they are intentionally misleading. I believe even lay-people understand the notion \n\nWhy the two-stage interaction process instead of providing the AI advice from the beginning to mimic more real-world deployments?\n\nWhat is baseline human accuracy and AI accuracy on all four tasks? Is there a baseline human-AI team, for example a learning to defer baseline, and results for that? It would be quite significant to show that humans deciding when to use AI advice is better than AI deciding when to use human advice. \n\nWhy weren’t alpha and beta optimized given the user demographics? Was there insufficient data? Furthermore, since alpha and beta weren’t personalized, then why does the activate/integrate functions need to be personalized?\n\nThe fact that “asks with higher advice accuracy (Art and Cities) exhibit larger increases in accuracy and confidence” is well documented in the literature since absolute AI advice accuracy is higher. Is there evidence of a relative increase in human accuracy (rather than absolute)?\n\nRather than misleading the users, what would you expect the effect of an intervention where users are taught to adapt their activation/integration of AI to revise their own internal model, could such an intervention mitigate the need for AI optimization and uncalibrated model?\n\nWhat would you think the effect of telling users that one model is calibrated, and one is uncalibrated? For example, if you randomly split the user base that received the optimized advice, and told one part that the advice is calibrated (lie) and the other that it is uncalibrated (truth), what differences would you see? \n", "limitations": "I think the authors should not encourage practice to adopt uncalibrated models, and should highlight that a limitation of their work, but still highlight that their work is an interesting exploration. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: The learning of the activation and integration models from human data are novel, even if the formulation is not as novel. However, the optimization procedure for AI advice is novel. Moreover, this is one of the first papers to do human-AI optimization directly. \n\nQuality: The AI advice optimization procedure is technically sound. The experimental tasks are not very ecologically valid as they are not representative of tasks people will use AI for. \n\nClarity: The paper is well written. I think the human-AI optimization procedure can be better illustrated with a figure to show how $\\alpha,\\beta$ feed into both the activation and integration models. I would also include Figure 6 from the appendix in the main body. \n\nSignificance: To the best of my knowledge, this paper is one of the first to do AI optimization for AI assisted decision making and produce a positive result. I think this will inspire future work in this area, as I can see many extensions to this work. However, for practical significance, the authors need to strongly argue for why misleading users is okay, I strongly encourage them to reward the language of the intro and abstract as to have this paper be an academic exploration rather than a method AI designers should develop to improve human-AI teams in practice\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "-fJa8Ht9yoP", "reviewer": "Reviewer_k53k", "summary": "In the context of AI-assisted decision-making, the authors propose a method to adjust the confidence of the AI advice, that a human user observes on a given predictive task, with the goal to improve the user's accuracy on that task. For this purpose, they assume a two-stage cognitive mechanism for the user's processing of the reported AI confidence.  In the first stage, the users decide whether to use the advice or not, and if they do so, in the second stage, they decide to what extent they will use it. The authors fit a function for each stage using empirical data that they gathered on four binary classification settings. Then, they leverage these fitted functions to find which AI-confidence modification function--among a parametric family of functions-- minimizes the user's binary cross entropy loss on a specific training data set. Finally, the authors demonstrate  with simulations as well as experiments with real human users, that adjusting the AI confidence with such an optimized modification function, appears to improve not only the user's accuracy but also their confidence on their predictions. ", "questions": "1. In the study the authors used prior aggregated human predictions as a proxy for the AI model, since the four datasets used in the experimental setting are too limited for training actual AI models. However, the sarcasm dataset as well as the census dataset are subsets of much larger datasets, namely the Reddit sarcasm dataset [1] and the Census dataset [2], that are suitable for actual model training.  Definitely, training (and calibrating)  models from scratch for the purposes of the study are beyond the scope of the paper, but given that the existing  state-of-the-art model [3] on the Reddit sarcasm dataset, reports F1 score > 70%, why such an off-the-shelf model, or similar ones were not used  as the AI model giving advice for the correspondent dataset? \n\n2. In figure 2(b)  the initial response and advice are in $[-1,1]$ whereas in figures 3(b), and  9(b) they are in $[0,1]$. The authors do mention in line 196 that $r_1$ and $r_2$ are probabilities normalized in $[-1,1]$ and it would be helpful if the $r_1$ range was consistent across plots, or clearly specified otherwise.\n\n3. The objective introduced in Eq. 2 is conditioned on the vector $\\boldsymbol x$ of initial response $r_1$, advice A and user's features $u$. Also in Appendix C.4 it is mentioned that each advice-initial response pair was optimized separately. The objective between lines 249-250 refers to the expected value of the binary cross entropy loss on a given dataset, not conditioning on x or the advice-initial response pair. Is there an optimal $\\alpha^*$, $\\beta^*$ for each advice-initial response pair? Is there one for each $\\boldsymbol x$ or there is one optimal $\\alpha^*$, $\\beta^*$ for a given dataset? From line 276, it seems that there is one $\\alpha^*$, $\\beta^*$ given a training dataset, though conditioning on x in Eq. (2) and stating in Appendix that optimization happens for each $(A,r_1)$ pair are rather confusing.\n\n4. Minor typos: \n       In line 533 \"to modification\" --> \"no modification\" ?\n       In line 540 \"for modify advice\" --> \"for modified advice\"?\n\n\n\n[1]  Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli. 2017. A large self-annotated corpus for sarcasm. arXiv preprint arXiv:1704.05579 (2017).\n\n[2] Alex West and Anusha Praturu. 2019. Enhancing the Census Income Prediction Dataset. https://people.ischool.berkeley.edu/~alexwest/w210_census_income_html/.\n\n[3] Taha Shangipour ataei, Soroush Javdan, and Behrouz Minaei-Bidgoli. 2020. Applying Transformers and Aspect-based Sentiment Analysis approaches on Sarcasm Detection. In Proceedings of the Second Workshop on Figurative Language Processing, pages 67–71, Online. Association for Computational Linguistics.", "limitations": "One key limitation to the proposed method is the need of accurately modeling human behavior, based on which the authors optimize the Ai's reported confidence with respect to the user's accuracy. The authors do acknowledge this limitation and clarify that computing such a human model  is not within the focus of their work. They also consider as future work a robust human model with which they could account for the model's error. Another limitation, that the authors consider as future work, is the fact that they used only an AI model with an accuracy higher than each user's accuracy, while not considering cases in which users' accuracy would exceed the model's accuracy. Furthermore, the proposed optimization is based on the choice of the training dataset D, which could be addressed as a limitation; it would be helpful if the authors could add some discussion or example of how a distribution shift on the test data could affect the user's performance or clarify if their method should only be applied in absence of such a shift.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": " The authors appear to be the first to make an attempt to manipulate the reported confidence of a decision-aid system, with the goal to maximize the user's accuracy when advised by such a system. Even though the idea that the reported system's confidence does affect the user's trust on the system's advice is not new and has been studied in prior works [1], [2], [3], the paper seems to be the first to formalize the problem of adjusting the reported confidence to increase the user's accuracy, opening an interesting new perspective for future works.\n\nOverall the paper seems mostly well-structured and clearly written. There is a meticulous description on the experimental setting to gather user's data used to fit the activation and integration models as well as a detailed analysis of the models themselves. The formalization of the setting is also detailed, however the optimization objective and procedure could be described more clearly with more details, since for example conditioning on x on Eq.2, but not in equations between lines 249-250 confuses the reader. The authors provide theoretical examples, simulations and experimental results, with adequate analysis, based on which they evaluate their proposed method.  \n\nThe idea of adjusting the reported confidence of an AI decision-aid system, with the goal to benefit the user's accuracy, as well as the experimental results do seem promising and opens different perspectives for future work. Overall, one might talk about an interesting and important first attempt to manipulate a system's reported confidence, to benefit the users.\n\n\n[1] Sunita Sah, Don A Moore, and Robert J MacCoun. Cheap talk and credibility: The consequences of confidence and accuracy on advisor credibility and persuasiveness. Organizational Behavior and Human Decision Processes, 121(2):246–255, 2013.\n\n[2] Zhang, Yunfeng, Q. Vera Liao, and Rachel KE Bellamy. \"Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making.\" In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 295-305., 2020.\n\n[3]Rechkemmer, Amy, and Ming Yin. \"When Confidence Meets Accuracy: Exploring the Effects of Multiple Performance Indicators on Trust in Machine Learning Models.\" In CHI Conference on Human Factors in Computing Systems, pp. 1-14. 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept